1. Project Overview & Quickstart (crowdsecurity/crowdsec) Copy Chapter

File: README.md Copy Chapter

CrowdSec: open-source IDS/IPS, WAF and bot detection

CrowdSec is an open-source and participative security solution offering crowdsourced server detection and protection against malicious IPs. Detect and block with our Security Engine, contribute to the network, and enjoy our real-time community blocklist.

Features & Advantages Copy Chapter

Versatile Security Engine Copy Section

CrowdSec Security Engine is an all-in-one IDS/IPS and WAF.

It detects bad behaviors by analyzing log sources and HTTP requests, and allows active remediation thanks to the Remediation Components.

Detection rules are available on our hub under MIT license.

CrowdSec Community Blocklist Copy Section

Console - Monitoring & Automation of your security stack Copy Section

Multiple Platforms support Copy Section

How CrowdSec compares Copy Chapter

To fail2ban — CrowdSec reads the same log files and bans the same IPs, but
detection and remediation are decoupled: parse logs on one machine, block at the
firewall, reverse proxy or CDN on another. Attacks you report feed a community
blocklist, so an IP that hits one user gets blocked for everyone else before it
reaches them.

To ModSecurity, Coraza and naxsi — the AppSec component is a WAF built on
Coraza, so SecLang rules and the OWASP Core Rule Set load as-is. On top of that
it adds a YAML rule format, and shares the IP reputation and remediation layer
with the log-based engine.

Bot and scraper detection — AppSec can answer a suspicious request with a
JavaScript proof-of-work challenge at a tunable difficulty, which stops headless
browsers and scrapers while letting verified crawlers such as Googlebot through.

Outnumbering hackers all together Copy Chapter

By sharing the threat they faced, all users are protecting each other (hence the name Crowd-Security). Crowdsec is designed for modern infrastructures, with its "Detect Here, Remedy There" approach, letting you analyze logs coming from several sources in one place and block threats at various levels (applicative, system, infrastructural) of your stack.

CrowdSec ships by default with scenarios (brute force, port scan, web scan, etc.) adapted for most contexts, but you can easily extend it by picking more of them from the HUB. It is also easy to adapt an existing one or create one yourself.

Installation Copy Chapter

Follow our documentation to install CrowdSec in a few minutes on Linux, Windows, Docker, OpnSense, Kubernetes, and more.

Resources Copy Chapter

• Console: Supercharge your CrowdSec setup with visualization, management capabilities, extra blocklists and premium features.
• Documentation: Learn how to exploit your CrowdSec setup to deter more attacks.
• Discord: A question or a suggestion? This is the place.
• Hub: Improve your stack protection, find the relevant remediation components for your infrastructure.
• Public Roadmap: Vote for your favorite features and submit ideas to influence what we build next.
• CrowdSec Academy: Learn and grow with our courses.
• Corporate Website: For everything else.

File: AGENTS.md Copy Chapter

Working in this repository

For anyone opening a pull request against crowdsec: human, or LLM.
CLAUDE.md is a symlink to this file.

PRs that ignore the rules below are unlikely to be reviewed.

Keep the change small Copy Chapter

• One concern per PR. A bug fix and a refactor are two PRs.
• No drive-by reformatting, renaming, or restructuring.
• Don't bump dependencies. Dependabot owns go.mod and go.sum.
• If the diff is over ~400 lines, either split it, or say in the first line of the PR body why
  it can't be split.

A human must have tested and reviewed this Copy Chapter

Before you open the PR, a human must have:

• read the diff line by line, and
• run the change, or supervised and independently verified the testing an agent did.

Say which, in the PR body.

Automated checks Copy Section

• While iterating: go test ./pkg/<what you changed>/...
• Full suite: make test. Needs gotestsum installed, and make localstack running in
  another shell, or set TEST_LOCAL_ONLY=1 to skip the tests that need containers.
• If you touched pkg/exprhelpers, use make test.
• Lint while iterating: golangci-lint run ./pkg/<what you changed>/...
• Before pushing: make lint. It lints linux, windows and freebsd, same as CI — so expect
  findings outside your change if it's platform-specific.
• Use golangci-lint v2.13, the version CI pins. Other versions report different things.
• Don't restructure working code to satisfy a linter that's switched off. Check the disable:
  list in .golangci.yml before "fixing" something it never complained about.

Running it for real Copy Section

This is how you can spin a simple test instance:

make build                        # crowdsec, cscli and the notification plugins
scripts/test_env.sh -d ./tests    # working config tree in ./tests
cd ./tests
$EDITOR config/acquis.yaml        # point it at a log file you can write to
./crowdsec -c dev.yaml

scripts/test_env.sh copies the binaries out of the source tree, so make build has to run
first.

Look up the crowdSec-skill to deploy, configure and troubleshoot CrowdSec instances: https://www.skills.sh/crowdsecurity/crowdsec-skill/crowdsec

Tests Copy Chapter

• New behaviour needs a test.
• A test must fail when the behaviour it covers breaks. No tautological tests: don't check that a
  mock returns what you told it to, that a setter sets, or that the code does what it says. More
  test lines is not better; a large suite that misses the obvious case is worse than a small one.
• Test the cases the change exists for, not just the happy path. Don't skip the hard one with a
  comment: a skipped test for the scenario that matters is a missing test.
• Refactoring or replacing an existing component: first cover its current user-visible behaviour
  with tests, and make them pass unchanged before and after. List every intended behaviour
  change in the PR body.
• require over assert — a failed assert keeps going and buries the real error.
• Table-driven, with t.Run subtests.
• Unit tests live next to the code. Functional tests are BATS under test/bats/; see
  test/README.md.

Be concise Copy Chapter

Humans read your output, keep verbosity to minimum.

PR and issue bodies — what changed, why, and how it was tested. Nothing else. No prose
restatement of the diff. Any issue or PR text should be under 5k characters.

Instead of:

Summary Copy Chapter

This PR introduces a comprehensive enhancement to the syslog acquisition module...

Motivation Copy Chapter

Currently, the existing implementation suffers from a limitation whereby...

Implementation Details Copy Chapter

• ✅ Added a new timeout field to the configuration struct
• ✅ Updated the parsing logic to respect the new field

Testing Strategy Copy Chapter

Comprehensive unit tests have been added to ensure correctness...

Write:

syslog acquisition hung forever when the remote stopped sending. Adds a configurable
timeout (default 30s) and closes the connection when it expires.

Tested: go test ./pkg/acquisition/modules/syslog/..., plus a local run against a syslog
source that goes silent — connection now drops after 30s instead of hanging.

Code comments — explain why, not what. If the comment restates the line under it, delete
it. Keep them short and to the point, not prose: a comment should rarely be longer than the code
it describes. A 20-line comment over a 5-line function is noise.

Review replies — answer the question that was asked. Don't re-explain the PR, and don't
paste your agent's output into the thread. Read it, then reply in your own words, in a few
lines.

Don't break these Copy Chapter

Each one is a one-way door — users are already depending on it.

• LAPI/CAPI payloads — change pkg/models/localapi_swagger.yaml or
  pkg/modelscapi/centralapi_swagger.yaml and regenerate. Never edit the generated Go by hand.
• Database schema — your migration lands on existing user databases.
• Config keys in pkg/csconfig — renaming or removing one breaks upgrades.
• cscli -o json and -o raw — bouncers and scripts parse these. -o human is the only
  output you can freely change.

If you have to break one of these, say so explicitly at the top of the PR body.

Commits and PRs Copy Chapter

• Base branch is master.
• Commit titles are loose: area: what changed or type(scope): what changed, both fine.
• Never write (#1234) in a commit title. Squash-merge adds it. Yours will be wrong.
• One kind/* label and one area/* label.

File: azure-pipelines.yml Copy Chapter

trigger:
tags:
include:
- "v*"
exclude:
- "vfreebsd"
branches:
exclude:
- ""
pr: none

pool:
vmImage: windows-2022

stages:

• stage: Build
  jobs:

  • job: Build
    displayName: "Build"
    steps:

    • task: GoTool@0
      displayName: "Install Go"
      inputs:
      version: '1.27.1'

    • pwsh: |
      choco install -y make
      displayName: "Install builds deps"

    • task: PowerShell@2
      inputs:
      targetType: 'inline'
      pwsh: true
      #we are not calling make windows_installer because we want to sign the binaries before they are added to the MSI
      script: |
      make build BUILD_RE2_WASM=1

    • pwsh: |
      $build_version=$env:BUILD_SOURCEBRANCHNAME
      #Override the version if it's set in the pipeline
      if ( ${env:USERBUILDVERSION} -ne "")
      {
      $build_version = ${env:USERBUILDVERSION}
      
      }
      if ($build_version.StartsWith("v"))
      {
      $build_version = $build_version.Substring(1)
      
      }
      if ($build_version.Contains("-"))
      {
      $build_version = $build_version.Substring(0, $build_version.IndexOf("-"))
      
      }
      Write-Host "##vso[task.setvariable variable=BuildVersion;isOutput=true]$build_version"
      displayName: GetCrowdsecVersion
      name: GetCrowdsecVersion

    • pwsh: |
      Get-ChildItem -Path .\cmd -Directory | ForEach-Object {
      $dirName = $.Name
      Get-ChildItem -Path .\cmd$dirName -File -Filter '*.exe' | ForEach-Object {
      $fileName = $.Name
      $destDir = Join-Path $(Build.ArtifactStagingDirectory) cmd$dirName
      New-Item -ItemType Directory -Path $destDir -Force
      Copy-Item -Path .\cmd$dirName$fileName -Destination $destDir
      }
      
      }
      displayName: "Copy binaries to staging directory"

    • task: PublishPipelineArtifact@1
      inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'unsigned_binaries'
      displayName: "Upload binaries artifact"

• stage: Sign
  dependsOn: Build
  variables:

  • group: 'FOSS Build Variables'
  • name: BuildVersion
    value: $[ stageDependencies.Build.Build.outputs['GetCrowdsecVersion.BuildVersion'] ]
    condition: succeeded()
    jobs:
  • job: Sign
    displayName: "Sign"
    steps:
    • download: current
      artifact: unsigned_binaries
      displayName: "Download binaries artifact"
    • task: CopyFiles@2
      inputs:
      SourceFolder: '$(Pipeline.Workspace)/unsigned_binaries'
      TargetFolder: '$(Build.SourcesDirectory)'
      displayName: "Copy binaries to workspace"
    • task: DotNetCoreCLI@2
      displayName: "Install SignTool tool"
      inputs:
      command: 'custom'
      custom: 'tool'
      arguments: install --global sign --version 0.9.0-beta.23127.3
    • task: AzureKeyVault@2
      displayName: "Get signing parameters"
      inputs:
      azureSubscription: "Azure subscription"
      KeyVaultName: "$(KeyVaultName)"
      SecretsFilter: "TenantId,ClientId,ClientSecret,Certificate,KeyVaultUrl"
    • pwsh: |
      sign code azure-key-vault "**/*.exe"
      --base-directory "$(Build.SourcesDirectory)/cmd/" --publisher-name "CrowdSec"
      --description "CrowdSec" --description-url "https://github.com/crowdsecurity/crowdsec"
      --azure-key-vault-tenant-id "$(TenantId)" --azure-key-vault-client-id "$(ClientId)"
      --azure-key-vault-client-secret "$(ClientSecret)" --azure-key-vault-certificate "$(Certificate)"
      --azure-key-vault-url "$(KeyVaultUrl)"
      displayName: "Sign crowdsec binaries"
    • pwsh: |
      .\build\windows\make_installer.ps1 -version '$(BuildVersion)'
      displayName: "Build Crowdsec MSI"
      name: BuildMSI
    • pwsh: |
      .\build\windows\make_chocolatey.ps1 -version '$(BuildVersion)'
      displayName: "Build Chocolatey nupkg"
    • pwsh: |
      sign code azure-key-vault "*.msi"
      --base-directory "$(Build.SourcesDirectory)" --publisher-name "CrowdSec"
      --description "CrowdSec" --description-url "https://github.com/crowdsecurity/crowdsec"
      --azure-key-vault-tenant-id "$(TenantId)" --azure-key-vault-client-id "$(ClientId)"
      --azure-key-vault-client-secret "$(ClientSecret)" --azure-key-vault-certificate "$(Certificate)"
      --azure-key-vault-url "$(KeyVaultUrl)"
      displayName: "Sign MSI package"
    • pwsh: |
      sign code azure-key-vault "*.nupkg"
      --base-directory "$(Build.SourcesDirectory)" --publisher-name "CrowdSec"
      --description "CrowdSec" --description-url "https://github.com/crowdsecurity/crowdsec"
      --azure-key-vault-tenant-id "$(TenantId)" --azure-key-vault-client-id "$(ClientId)"
      --azure-key-vault-client-secret "$(ClientSecret)" --azure-key-vault-certificate "$(Certificate)"
      --azure-key-vault-url "$(KeyVaultUrl)"
      displayName: "Sign nuget package"
    • task: PublishPipelineArtifact@1
      inputs:
      targetPath: '$(Build.SourcesDirectory)/crowdsec_$(BuildVersion).msi'
      artifact: 'signed_msi_package'
      displayName: "Upload signed MSI artifact"
    • task: PublishPipelineArtifact@1
      inputs:
      targetPath: '$(Build.SourcesDirectory)/crowdsec.$(BuildVersion).nupkg'
      artifact: 'signed_nuget_package'
      displayName: "Upload signed nuget artifact"

• stage: Publish
  dependsOn: Sign
  jobs:

  • deployment: "Publish"
    displayName: "Publish to GitHub"
    environment: github
    strategy:
    runOnce:
    deploy:
    steps:
    - bash: |
    tag=$(curl -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/crowdsecurity/crowdsec/releases | jq -r '. | map(select(.prerelease==true)) | sort_by(.created_at) | reverse | .[0].tag_name')
    echo "##vso[task.setvariable variable=LatestPreRelease;isOutput=true]$tag"
    name: GetLatestPrelease
    - task: GitHubRelease@1
    inputs:
    gitHubConnection: "github.com_blotus"
    repositoryName: '$(Build.Repository.Name)'
    action: 'edit'
    tag: '$(GetLatestPrelease.LatestPreRelease)'
    assetUploadMode: 'replace'
    addChangeLog: false
    isPreRelease: true #we force prerelease because the pipeline is invoked on tag creation, which happens when we do a prerelease
    assets: |
    $(Pipeline.Workspace)/signed_msi_package/.msi
    $(Pipeline.Workspace)/signed_nuget_package/.nupkg
    condition: ne(variables['GetLatestPrelease.LatestPreRelease'], '')

File: CONTRIBUTING.md Copy Chapter

Please refer to Contributing to CrowdSec.

File: SECURITY.md Copy Chapter

Security Policy

Scope Copy Chapter

This security policy applies to :

• Crowdsec agent
• Crowdsec Local API
• Crowdsec bouncers developed and maintained by the Crowdsec team [1]

Reports regarding developments of community members that are not part of the crowdsecurity organization will be thoroughly investigated nonetheless.

[1] Projects developed and maintained by the Crowdsec team are under the crowdsecurity github organization. Bouncers developed by community members that are not part of the Crowdsec organization are explicitly excluded.

Reporting a Vulnerability Copy Chapter

We are extremely grateful to security researchers and users that report vulnerabilities regarding the Crowdsec project. All reports are thoroughly investigated by members of the Crowdsec organization.

You can email the private [email protected] list with the security details and the details expected for all Crowdsec bug reports.

You may encrypt your email to this list using the GPG key of the Security team. Encryption using GPG is NOT required to make a disclosure.

When Should I Report a Vulnerability? Copy Chapter

• You think you discovered a potential security vulnerability in Crowdsec
• You are unsure how a vulnerability affects Crowdsec
• You think you discovered a vulnerability in another project that Crowdsec depends on

For projects with their own vulnerability reporting and disclosure process, please report it directly there.

File: .golangci.yml Copy Chapter

version: "2"

To lint the windows sources on a linux host, you need to cross-build and

disable the C-based sqlite driver:

GOOS=windows golangci-lint run --build-tags=windows,sqlite_modernc

In the CI, the linters are run on both linux and windows hosts.

run:
build-tags:
- expr_debug

linters:
default: all
disable:
#
# Redundant
#
- cyclop # revive
- funlen # revive
- gocognit # revive
- gocyclo # revive
- lll # revive
- wsl # wsl_v5

# Disabled atm, likely to be enabled later

# new checks in golangci-lint v2.13, to enable soon
- canonicalheader
- noctx
- recvcheck

- funcorder
- exhaustive
- errname      # Checks that sentinel errors are prefixed with the `Err` and error types are suffixed with the `Error`.
- unparam      # Reports unused function parameters
- errchkjson   # Checks types passed to the json encoding functions. Reports unsupported types and reports occasions, where the check for the returned error can be omitted.
- unqueryvet   # Detects SELECT * in SQL queries and SQL builders, preventing performance issues and encouraging explicit column selection.

# we allow named returns (to improve readability, in the function signatures)
# but disallow implicit (or "naked") returns, to improve readability in the actual code.
#
# Implicit returns are still useful with deferred functions, in which case they can be //nolinted.
# https://go.dev/wiki/CodeReviewComments#named-result-parameters
- nonamedreturns

- dogsled               # Checks assignments with too many blank identifiers (e.g. x, _, _, _, := f())
- gosec                 # (gas): Inspects source code for security problems
- musttag               # enforce field tags in (un)marshaled structs
- promlinter            # Check Prometheus metrics naming via promlint
- tagalign              # check that struct tags are well aligned
- thelper               # thelper detects tests helpers which do not start with t.Helper() method.
- wrapcheck             # Checks that errors returned from external packages are wrapped
- embeddedstructfieldcheck

- ireturn               # Accept Interfaces, Return Concrete Types
- mnd                   # An analyzer to detect magic numbers.
- nilnil                # Checks that there is no simultaneous return of `nil` error and an invalid value.
- dupl                  # Tool for code clone detection
- forcetypeassert       # finds forced type assertions
- godox                 # Tool for detection of FIXME, TODO and other comment keywords
- err113                # Go linter to check the errors handling expressions
- paralleltest          # Detects missing usage of t.Parallel() method in your Go test
- testpackage           # linter that makes you use a separate _test package
- exhaustruct           # Checks if all structure fields are initialized
- exhaustruct_v5        # same, new major version
- gochecknoglobals      # Check that no global variables exist.
- goconst               # Finds repeated strings that could be replaced by a constant
- tagliatelle           # Checks the struct tags.
- varnamelen            # checks that the length of a variable's name matches its scope
- prealloc
- noinlineerr
- iotamixing
- godoclint

#
# Formatting only, useful in IDE but should not be forced on CI
#

- godot      # Check if comments end in a period
- nlreturn   # nlreturn checks for a new line before return and branch statements to increase code clarity
- whitespace # Whitespace is a linter that checks for unnecessary newlines at the start and end of functions, if, for, etc.
- wsl_v5     # add or remove empty lines

settings:

decorder:
  # Required order of `type`, `const`, `var` and `func` declarations inside a file.
  # Default: types before constants before variables before functions.
  dec-order:
    - type
    - const
    - var
    - func
  # If true, underscore vars (vars with "_" as the name) will be ignored at all checks.
  # Default: false (underscore vars are not ignored)
  ignore-underscore-vars: false
  # If true, order of declarations is not checked at all.
  # Default: true (disabled)
  disable-dec-order-check: true
  # If true, `init` func can be anywhere in file (does not have to be declared before all other functions).
  # Default: true (disabled)
  disable-init-func-first-check: true
  # If true, multiple global `type`, `const` and `var` declarations are allowed.
  # Default: true (disabled)
  disable-dec-num-check: true
  # If true, type declarations will be ignored for dec num check.
  # Default: false (type statements are not ignored)
  disable-type-dec-num-check: false
  # If true, const declarations will be ignored for dec num check.
  # Default: false (const statements are not ignored)
  disable-const-dec-num-check: false
  # If true, var declarations will be ignored for dec num check.
  # Default: false (var statements are not ignored)
  disable-var-dec-num-check: false

depguard:
  rules:
    wrap:
      deny:
        - pkg: github.com/pkg/errors
          desc: errors.Wrap() is deprecated in favor of fmt.Errorf()
    yaml:
      files:
        - '!**/pkg/acquisition/modules/loki/timestamp_test.go'
        - '!**/pkg/appsec/appsec.go'
        - '!**/pkg/appsec/loader.go'
        - '!**/pkg/csplugin/broker.go'
        - '!**/pkg/leakybucket/bucketstore_test.go'
        - '!**/pkg/leakybucket/manager_load.go'
        - '!**/pkg/parser/node_config.go'
        - '!**/pkg/parser/node_subnodes_test.go'
        - '!**/pkg/parser/node_test.go'
        - '!**/pkg/parser/parsing_test.go'
        - '!**/pkg/parser/stage.go'
      deny:
        - pkg: gopkg.in/yaml.v2
          desc: yaml.v2 is deprecated for new code in favor of yaml.v3

embeddedstructfieldcheck:
  empty-line: false
  forbid-mutex: true

errcheck:
  # Report about not checking of errors in type assertions: `a := b.(MyStruct)`.
  # Such cases aren't reported by default.
  check-type-assertions: false
  # List of functions to exclude from checking, where each entry is a single function to exclude.
  # See https://github.com/kisielk/errcheck#excluding-functions for details.
  exclude-functions:
    - (*bytes.Buffer).ReadFrom          # TODO
    - golang.org/x/sys/windows.FreeLibrary
    - golang.org/x/sys/windows.CloseHandle
    - golang.org/x/sys/windows.ResetEvent
    - (*golang.org/x/sys/windows/svc/eventlog.Log).Info
    - (*golang.org/x/sys/windows/svc/mgr.Mgr).Disconnect

    - (github.com/bluele/gcache.Cache).Set
    - (github.com/gin-gonic/gin.ResponseWriter).WriteString
    - (*github.com/segmentio/kafka-go.Reader).SetOffsetAt

    - (*github.com/crowdsecurity/crowdsec/pkg/appsec.ReqDumpFilter).FilterArgs
    - (*github.com/crowdsecurity/crowdsec/pkg/appsec.ReqDumpFilter).FilterBody
    - (*github.com/crowdsecurity/crowdsec/pkg/appsec.ReqDumpFilter).FilterHeaders
    - (*github.com/crowdsecurity/crowdsec/pkg/longpollclient.LongPollClient).Stop

errorlint:
  # Check whether fmt.Errorf uses the %w verb for formatting errors.
  # See the https://github.com/polyfloyd/go-errorlint for caveats.
  # Default: true
  errorf: false
  errorf-multi: true
  asserts: true
  comparison: true

exhaustive:
  # Program elements to check for exhaustiveness.
  # Default: [ switch ]
  check:
    - switch
    - map
  # Presence of "default" case in switch statements satisfies exhaustiveness,
  # even if all enum members are not listed.
  # Default: false
  default-signifies-exhaustive: true
  # Enum members matching the supplied regex do not have to be listed in
  # switch statements to satisfy exhaustiveness.
  # Default: ""
  ignore-enum-members: "Example.+"
  # Enum types matching the supplied regex do not have to be listed in
  # switch statements to satisfy exhaustiveness.
  # Default: ""
  ignore-enum-types: "Example.+"
  # Consider enums only in package scopes, not in inner scopes.
  # Default: false
  package-scope-only: true
  # Only run exhaustive check on switches with "//exhaustive:enforce" comment.
  # Default: false
  explicit-exhaustive-switch: true
  # Only run exhaustive check on map literals with "//exhaustive:enforce" comment.
  # Default: false
  explicit-exhaustive-map: true
  # Switch statement requires default case even if exhaustive.
  # Default: false
  default-case-required: true

forbidigo:
  forbid:
  # prefer fprint to print calls, keep the latter for debugging
  # so they're not comitted by mistake.
  - pattern: ^print(ln)?$
  - pattern: ^fmt\.Print.*$
    msg: use the Fprint equivalent in production code

gocritic:
  enable-all: true
  disabled-checks:
    - paramTypeCombine
    - hugeParam
    - commentedOutCode
    - commentedOutImport
    - unnamedResult
    - sloppyReassign
    - appendCombine
    - commentFormatting
    - whyNoLint
    - equalFold               #
    - unnecessaryBlock        #
    - tooManyResultsChecker
    - docStub
    - preferFprint
    - importShadow

  settings:
    ifElseChain:
      minThreshold: 4

gomoddirectives:
  replace-allow-list:
    - golang.org/x/time
    - github.com/corazawaf/coraza/v3

govet:
  disable:
    - fieldalignment
    - inline  # new in v2.13, to enable soon
  enable-all: true

iface:
  enable:
    # Identifies interfaces in the same package that have identical method sets.
    - identical
    # Identifies interfaces that are not used anywhere in the same package where the interface is defined.
    # - unused
    # Identifies functions that return interfaces, but the actual returned value is always a single concrete implementation.
    - opaque
    # Identifies interfaces that are not exported but are used in exported functions or methods.
    # - unexported

interfacebloat:
  max: 7

maintidx:
  # raise this after refactoring
  under: 21

misspell:
  locale: US

modernize:
  disable:
    - any
    # embedlit, errorsastype, slicesbackward and stringscut are new in v2.13, to enable soon
    - embedlit
    - errorsastype
    - forvar
    - omitzero
    - slicesbackward
    - stringsbuilder
    - stringscut
    - waitgroupgo

nestif:
  # lower this after refactoring
  min-complexity: 15

nlreturn:
  block-size: 6

nolintlint:
  # don't require an explanation for nolint directives
  require-explanation: false
  # don't require nolint directives to be specific about which linter is being skipped
  require-specific: true
  # report any unused nolint directives
  allow-unused: false

perfsprint:
  # Enable/disable optimization of integer formatting.
  # Default: true
  integer-format: false
  # Optimizes even if it requires an int or uint type cast.
  # Default: true
  int-conversion: false
  # Enable/disable optimization of error formatting.
  # Default: true
  error-format: true
  # Optimizes into `err.Error()` even if it is only equivalent for non-nil errors.
  # Default: false
  err-error: true
  # Optimizes `fmt.Errorf`.
  # Default: true
  errorf: true
  # Enable/disable optimization of string formatting.
  # Default: true
  string-format: false
  # Optimizes `fmt.Sprintf` with only one argument.
  # Default: true
  sprintf1: true
  # Optimizes into strings concatenation.
  # Default: true
  strconcat: false
  # Enable/disable optimization of bool formatting.
  # Default: true
  bool-format: true
  # Enable/disable optimization of hex formatting.
  # Default: true
  hex-format: false
  # Enable/disable optimization of concat loop.
  # Default: true
  concat-loop: false
  # Optimization of `concat-loop` even with other operations.
  # Default: false
  loop-other-ops: false

recvcheck:
  # Disables the built-in method exclusions:
  # - `MarshalText`
  # - `MarshalJSON`
  # - `MarshalYAML`
  # - `MarshalXML`
  # - `MarshalBinary`
  # - `GobEncode`
  disable-builtin: false
  # User-defined method exclusions.
  # The format is `struct_name.method_name` (ex: `Foo.MethodName`).
  # A wildcard `*` can use as a struct name (ex: `*.MethodName`).
  exclusions:
    - "HCLogAdapter.SetLevel"
    # Unmarshal methods must have a pointer receiver by nature; getters on
    # the same type legitimately use value receivers (same rationale as the
    # built-in Marshal* exclusions, which don't cover Unmarshal*).
    - "*.UnmarshalJSON"

revive:
  severity: error
  enable-all-rules: true
  rules:
    - name: argument-limit
      arguments: [9]
    - name: add-constant
      disabled: true
    - name: cognitive-complexity
      arguments:
        # lower this after refactoring
        - 73
    - name: comment-spacings
      disabled: true
    - name: confusing-results
      disabled: true
    - name: cyclomatic
      arguments:
        # lower this after refactoring
        - 36
    - name: empty-block
      disabled: true
    - name: empty-lines
      disabled: true
    - name: enforce-switch-style
      disabled: true
    - name: error-naming
      disabled: true
    - name: exported
      disabled: true
    - name: flag-parameter
      disabled: true
    - name: function-result-limit
      arguments:
        - 6
    - name: function-length
      arguments:
        # lower this after refactoring
        - 86
        - 174
    - name: get-return
      disabled: true
    - name: identical-ifelseif-branches
      disabled: true
    - name: identical-switch-branches
      disabled: true
    - name: import-alias-naming
      disabled: true
    - name: import-shadowing
      disabled: true
    - name: increment-decrement
      disabled: true
    - name: line-length-limit
      arguments:
        # lower this after refactoring
        - 219
    - name: max-control-nesting
      arguments:
        # lower this after refactoring
        - 5
    - name: max-public-structs
      disabled: true
    - name: nested-structs
      disabled: true
    - name: package-comments
      disabled: true
    # new in v2.13, to enable soon
    - name: package-naming
      disabled: true
    - name: package-directory-mismatch
      disabled: true
    - name: redundant-import-alias
      disabled: true
    - name: redundant-test-main-exit
      disabled: true
    - name: var-naming
      disabled: true
    - name: unchecked-type-assertion
      disabled: true
    - name: unexported-naming
      disabled: true
    - name: unexported-return
      disabled: true
    - name: unhandled-error
      arguments:
        - fmt.Print
        - fmt.Printf
        - fmt.Println
      disabled: true
    - name: unnecessary-stmt
      disabled: true
    - name: unsecure-url-scheme
      disabled: true
    - name: unused-parameter
      disabled: true
    - name: use-any
      disabled: true
    # new in v2.13, to enable soon
    - name: use-slices-sort
      disabled: true
    - name: use-waitgroup-go
      disabled: true
    - name: useless-break
      disabled: true

staticcheck:
  checks:
    - all
    - -ST1003  # should not use underscores in Go names
    - -ST1005  # error strings should not be capitalized
    - -ST1012  # error var ... should have name of the form ...
    - -QF1003  # could use tagged switch on ...
    - -QF1012  # QF1012: Use fmt.Fprintf(...) instead of ...
    - -SA4023  # comparison is always true; new in v2.13, to enable soon

unused:
  # Mark all struct fields that have been written to as used.
  # Default: true
  field-writes-are-uses: false
  # Treat IncDec statement (e.g. `i++` or `i--`) as both read and write operation instead of just write.
  # Default: false
  post-statements-are-reads: false
  # Mark all exported identifiers as used.
  # Default: true
  # exported-is-used: false
  # Mark all exported fields as used.
  # default: true
  exported-fields-are-used: false
  # Mark all function parameters as used.
  # default: true
  parameters-are-used: true
  # Mark all local variables as used.
  # default: true
  local-variables-are-used: false
  # Mark all identifiers inside generated files as used.
  # Default: true
  generated-is-used: false

usestdlibvars:
  # Suggest the use of http.MethodXX.
  # Default: true
  http-method: true
  # Suggest the use of http.StatusXX.
  # Default: true
  http-status-code: true
  # Suggest the use of time.Month in time.Date.
  # Default: false
  time-date-month: false
  # Suggest the use of time.Weekday.String().
  # Default: true
  time-weekday: false
  # Suggest the use of time.Month.String().
  # Default: false
  time-month: false
  # Suggest the use of time.Layout.
  # Default: false
  time-layout: false
  # Suggest the use of crypto.Hash.String().
  # Default: false
  crypto-hash: true
  # Suggest the use of rpc.DefaultXXPath.
  # Default: false
  default-rpc-path: true
  # Suggest the use of sql.LevelXX.String().
  # Default: false
  sql-isolation-level: true
  # Suggest the use of tls.SignatureScheme.String().
  # Default: false
  tls-signature-scheme: true
  # Suggest the use of constant.Kind.String().
  # Default: false
  constant-kind: true

usetesting:
  os-temp-dir: false
  context-background: true
  context-todo: true

exclusions:
presets:
- comments
- common-false-positives
- legacy
- std-error-handling
rules:

# `err` is often shadowed, we may continue to do it

  - linters:
      - govet
    text: 'shadow: declaration of "(err|ctx)" shadows declaration'

  - linters:
      - revive
    path: pkg/leakybucket/manager_load.go
    text: 'confusing-naming: Field ''.*'' differs only by capitalization to other field in the struct type BucketFactory'

  - linters:
      - revive
    path: pkg/exprhelpers/helpers.go
    text: 'confusing-naming: Method ''flatten'' differs only by capitalization to function ''Flatten'' in the same source file'

  - linters:
      - revive
    path: pkg/appsec/query_utils.go
    text: 'confusing-naming: Method ''parseQuery'' differs only by capitalization to function ''ParseQuery'' in the same source file'

  - linters:
      - revive
    path: pkg/acquisition/modules/loki/internal/lokiclient/loki_client.go
    text: 'confusing-naming: Method ''QueryRange'' differs only by capitalization to method ''queryRange'' in the same source file'

  - linters:
      - revive
    path: pkg/acquisition/modules/victorialogs/internal/vlclient/vl_client.go
    text: 'confusing-naming: Method ''QueryRange'' differs only by capitalization to method ''queryRange'' in the same source file'

  # pkg/dnscache lookups are deliberately detached from the request
  # context: their results are shared across requests (singleflight +
  # cache), so one caller's cancellation must not abort the lookup for
  # the others; a fixed timeout bounds them instead.
  - linters:
      - contextcheck
    text: '(lookupPTR|lookupForward)` should pass the context parameter'

  # tolerate complex functions in tests for now
  - linters:
      - maintidx
    path: (.+)_test.go

  # tolerate print in tests for now
  - linters:
      - forbidigo
    path: (.+)_test.go
    text: 'use the Fprint equivalent in production code'

  # tolerate long functions in tests
  - linters:
      - revive
    path: (.+)_test.go
    text: 'function-length: .*'

  # tolerate long lines in tests
  - linters:
      - revive
    path: (.+)_test.go
    text: 'line-length-limit: .*'

  # we use t,ctx instead of ctx,t in tests
  - linters:
      - revive
    path: (.+)_test.go
    text: 'context-as-argument: context.Context should be the first parameter of a function'

  # tolerate deep exit in cobra's OnInitialize, for now
  - linters:
      - revive
    path: cmd/crowdsec-cli/main.go
    text: 'deep-exit: .*'

  - linters:
      - revive
    path: cmd/crowdsec/crowdsec.go
    text: 'deep-exit: .*'

  - linters:
      - revive
    path: cmd/crowdsec/api.go
    text: 'deep-exit: .*'

  - linters:
      - revive
    path: cmd/crowdsec/win_service.go
    text: 'deep-exit: .*'

  - linters:
      - revive
    path: cmd/crowdsec-cli/clisetup/setup/systemd_test.go
    text: 'deep-exit: .*'

  - linters:
      - gocritic
    path: cmd/crowdsec-cli
    text: 'rangeValCopy: .*'

  - linters:
      - gocritic
    path: pkg/(cticlient|hubtest)
    text: 'rangeValCopy: .*'

  - linters:
      - gocritic
    path: (.+)_test.go
    text: 'rangeValCopy: .*'

  - linters:
      - gocritic
    path: pkg/(appsec|acquisition|dumps|alertcontext|leakybucket|exprhelpers)
    text: 'rangeValCopy: .*'

  - linters:
      - usetesting
    path: pkg/apiserver/(.+)_test.go
    text: os.MkdirTemp.* could be replaced by t.TempDir.*

  - linters:
      - usetesting
    path: pkg/apiserver/(.+)_test.go
    text: os.CreateTemp.* could be replaced by os.CreateTemp.*

  - linters:
      - containedctx
    path: cmd/notification-file/main.go
    text: found a struct that contains a context.Context field

  - linters:
      - containedctx
    path: pkg/acquisition/modules/s3/source.go
    text: found a struct that contains a context.Context field

  # migrate over time

  - linters:
      - noctx
    path: pkg/parser/enrich_dns.go
    text: "net.LookupAddr must not be called"

  - linters:
      - noctx
    path: pkg/exprhelpers/helpers.go
    text: "net.LookupHost must not be called"

  - linters:
      - embeddedstructfieldcheck
    text: "embedded fields should be listed before regular fields"

  # TODO: remove when NodeConfig is not embedded anymore
  - linters:
      - staticcheck
    path: pkg/parser/(.+).go
    text: 'QF1008: could remove embedded field "NodeConfig" from selector'

paths:
  - third_party$
  - builtin$
  - examples$

issues:
max-issues-per-linter: 0
max-same-issues: 0

formatters:
settings:
gci:
sections:
- standard
- default
- prefix(github.com/crowdsecurity)
- prefix(github.com/crowdsecurity/crowdsec)

exclusions:
paths:
- third_party$
- builtin$
- examples$

File: cmd/notification-dummy/dummy.yaml Copy Chapter

type: dummy # Don't change
name: dummy_default # Must match the registered plugin in the profile

One of "trace", "debug", "info", "warn", "error", "off"

log_level: info

group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"

group_threshold: # Amount of alerts that triggers a message before has expired, eg "10"

max_retry: # Number of attempts to relay messages to plugins in case of error

timeout: # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------

plugin-specific options

The following template receives a list of models.Alert objects

The output goes in the logs and to a text file, if defined

format: |
{{.|toJson}}

output_file: # notifications will be appended here. optional

type: dummy

name: dummy_second_notification

...

File: cmd/notification-email/email.yaml Copy Chapter

type: email # Don't change
name: email_default # Must match the registered plugin in the profile

One of "trace", "debug", "info", "warn", "error", "off"

log_level: info

group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"

group_threshold: # Amount of alerts that triggers a message before has expired, eg "10"

max_retry: # Number of attempts to relay messages to plugins in case of error

timeout: 20s # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------

plugin-specific options

The following template receives a list of models.Alert objects

The output goes in the email message body

format: |

{{range . -}}
{{$alert := . -}}
{{range .Decisions -}}
{{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}} on machine {{$alert.MachineID}}. CrowdSec CTI
{{end -}}
{{end -}}

smtp_host: # example: smtp.gmail.com
smtp_username: # Replace with your actual username
smtp_password: # Replace with your actual password
smtp_port: # Common values are any of [25, 465, 587, 2525]
auth_type: # Valid choices are "none", "crammd5", "login", "plain"
sender_name: "CrowdSec"
sender_email: # example: [email protected]
email_subject: "CrowdSec Notification"
receiver_emails:

- [email protected]

- [email protected]

One of "ssltls", "starttls", "none"

encryption_type: "ssltls"

If you need to set the HELO hostname:

helo_host: "localhost"

If the email server is hitting the default timeouts (10 seconds), you can increase them here

connect_timeout: 10s

send_timeout: 10s

type: email

name: email_second_notification

...

File: cmd/notification-file/file.yaml Copy Chapter

Don't change this

type: file

name: file_default # this must match with the registered plugin in the profile
log_level: info # Options include: trace, debug, info, warn, error, off

This template render all events as ndjson

format: |
{{range . -}}
{ "time": "{{.StopAt}}", "program": "crowdsec", "alert": {{. | toJson }} }
{{ end -}}

group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s"

group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10"

#Use full path EG /tmp/crowdsec_alerts.json or %TEMP%\crowdsec_alerts.json
log_path: "/tmp/crowdsec_alerts.json"
rotate:
enabled: true # Change to false if you want to handle log rotate on system basis
max_size: 500 # in MB
max_files: 5
max_age: 5
compress: true

File: cmd/notification-http/http.yaml Copy Chapter

type: http # Don't change
name: http_default # Must match the registered plugin in the profile

One of "trace", "debug", "info", "warn", "error", "off"

log_level: info

group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"

group_threshold: # Amount of alerts that triggers a message before has expired, eg "10"

max_retry: # Number of attempts to relay messages to plugins in case of error

timeout: # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------

plugin-specific options

The following template receives a list of models.Alert objects

The output goes in the http request body

format: |
{{.|toJson}}

The plugin will make requests to this url, eg: https://www.example.com/

url:

Any of the http verbs: "POST", "GET", "PUT"...

method: POST

headers:

Authorization: token 0x64312313

skip_tls_verification: # true or false. Default is false

type: http

name: http_second_notification

...

File: cmd/notification-sentinel/sentinel.yaml Copy Chapter

type: sentinel # Don't change
name: sentinel_default # Must match the registered plugin in the profile

One of "trace", "debug", "info", "warn", "error", "off"

log_level: info

group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"

group_threshold: # Amount of alerts that triggers a message before has expired, eg "10"

max_retry: # Number of attempts to relay messages to plugins in case of error

timeout: # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------

plugin-specific options

The following template receives a list of models.Alert objects

The output goes in the http request body

format: |
{{.|toJson}}

customer_id: XXX-XXX
shared_key: XXXXXXX
log_type: crowdsec

File: cmd/notification-slack/slack.yaml Copy Chapter

type: slack # Don't change
name: slack_default # Must match the registered plugin in the profile

One of "trace", "debug", "info", "warn", "error", "off"

log_level: info

group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"

group_threshold: # Amount of alerts that triggers a message before has expired, eg "10"

max_retry: # Number of attempts to relay messages to plugins in case of error

timeout: # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------

plugin-specific options

The following template receives a list of models.Alert objects

The output goes in the slack message

format: |
{{range . -}}
{{$alert := . -}}
{{range .Decisions -}}
{{if $alert.Source.Cn -}}
:flag-{{$alert.Source.Cn}}: https://www.whois.com/whois/{{.Value}}|{{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}} on machine '{{$alert.MachineID}}'. <https://app.crowdsec.net/cti/{{.Value}}|CrowdSec CTI>{{end}}
{{if not $alert.Source.Cn -}}
:pirate_flag: https://www.whois.com/whois/{{.Value}}|{{.Value}} will get {{.Type}} for next {{.Duration}} for triggering {{.Scenario}} on machine '{{$alert.MachineID}}'. <https://app.crowdsec.net/cti/{{.Value}}|CrowdSec CTI>{{end}}
{{end -}}
{{end -}}

webhook:

API request data as defined by the Slack webhook API.

#channel:
#username:
#icon_emoji:
#icon_url:

type: slack

name: slack_second_notification

...

File: cmd/notification-splunk/splunk.yaml Copy Chapter

type: splunk # Don't change
name: splunk_default # Must match the registered plugin in the profile

One of "trace", "debug", "info", "warn", "error", "off"

log_level: info

group_wait: # Time to wait collecting alerts before relaying a message to this plugin, eg "30s"

group_threshold: # Amount of alerts that triggers a message before has expired, eg "10"

max_retry: # Number of attempts to relay messages to plugins in case of error

timeout: # Time to wait for response from the plugin before considering the attempt a failure, eg "10s"

#-------------------------

plugin-specific options

The following template receives a list of models.Alert objects

The output goes in the splunk notification

format: |
{{.|toJson}}

url:
token:

ca_cert_path: # CA certificate to verify the Splunk server, in addition to the system ones

skip_tls_verification: # true or false. Default is false

type: splunk

name: splunk_second_notification

...

File: config/acquis.yaml Copy Chapter

TODO: explain this file, point to docs, recommend acquis.d

- docker

- deb,rpm -> cscli setup

- wizard.sh

- windows

filenames:

- /var/log/nginx/*.log

- ./tests/nginx/nginx.log

#this is not a syslog log, indicate which kind of logs it is

labels:

type: nginx

---

filenames:

- /var/log/auth.log

- /var/log/syslog

labels:

type: syslog

---

filename: /var/log/apache2/*.log

labels:

type: apache2

File: config/acquis_win.yaml Copy Chapter

##RDP
source: wineventlog
event_channel: Security
event_ids:

• 4625
• 4623
  event_level: information
  labels:
  type: eventlog

##Firewall
filenames:

• C:\Windows\System32\LogFiles\Firewall*.log
  labels:
  type: windows-firewall

##SQL Server
source: wineventlog
event_channel: Application
event_ids:

• 18456
  event_level: information
  labels:
  type: eventlog

##IIS
use_time_machine: true
filenames:

• C:\inetpub\logs\LogFiles**.log
  labels:
  type: iis

File: config/config.yaml Copy Chapter

common:
daemonize: true
log_media: file
log_level: info
log_dir: /var/log/
log_max_size: 20
compress_logs: true
log_max_files: 10
config_paths:
config_dir: /etc/crowdsec/
data_dir: /var/lib/crowdsec/data/
simulation_path: /etc/crowdsec/simulation.yaml
hub_dir: /etc/crowdsec/hub/
index_path: /etc/crowdsec/hub/.index.json
notification_dir: /etc/crowdsec/notifications/
plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
#console_context_path: /etc/crowdsec/console/context.yaml
acquisition_path: /etc/crowdsec/acquis.yaml
acquisition_dir: /etc/crowdsec/acquis.d
parser_routines: 1
cscli:
output: human
color: auto
db_config:
log_level: info
type: sqlite
db_path: /var/lib/crowdsec/data/crowdsec.db
#max_open_conns: 100
#user:
#password:
#db_name:
#host:
#port:
flush:
max_items: 5000
max_age: 7d
plugin_config:
user: nobody # plugin process would be ran on behalf of this user
group: nogroup # plugin process would be ran on behalf of this group
api:
client:
insecure_skip_verify: false
credentials_path: /etc/crowdsec/local_api_credentials.yaml
server:
log_level: info
listen_uri: 127.0.0.1:8080
profiles_path: /etc/crowdsec/profiles.yaml
console_path: /etc/crowdsec/console.yaml
online_client: # Central API credentials (to push signals and receive bad IPs)
credentials_path: /etc/crowdsec/online_api_credentials.yaml
trusted_ips: # IP ranges, or IPs which can have admin API access
- 127.0.0.1
- ::1

tls:

cert_file: /etc/crowdsec/ssl/cert.pem

key_file: /etc/crowdsec/ssl/key.pem

prometheus:
enabled: true
level: full
listen_addr: 127.0.0.1
listen_port: 6060

File: config/config_win.yaml Copy Chapter

common:
log_media: file
log_level: info
log_dir: C:\ProgramData\CrowdSec\log
config_paths:
config_dir: C:\ProgramData\CrowdSec\config
data_dir: C:\ProgramData\CrowdSec\data
simulation_path: C:\ProgramData\CrowdSec\config\simulation.yaml
hub_dir: C:\ProgramData\CrowdSec\hub
index_path: C:\ProgramData\CrowdSec\hub.index.json
plugin_dir: C:\ProgramData\CrowdSec\plugins
notification_dir: C:\ProgramData\CrowdSec\config\notifications
crowdsec_service:
#console_context_path: C:\ProgramData\CrowdSec\console\context.yaml
acquisition_path: C:\ProgramData\CrowdSec\config\acquis.yaml
parser_routines: 1
cscli:
output: human
db_config:
log_level: info
type: sqlite
db_path: C:\ProgramData\CrowdSec\data\crowdsec.db
#user:
#password:
#db_name:
#host:
#port:
flush:
max_items: 5000
max_age: 7d
api:
client:
insecure_skip_verify: false
credentials_path: C:\ProgramData\CrowdSec\config\local_api_credentials.yaml
server:
log_level: info
listen_uri: 127.0.0.1:8080
profiles_path: C:\ProgramData\Crowdsec\config\profiles.yaml
online_client: # Crowdsec API credentials (to push signals and receive bad IPs)
credentials_path: C:\ProgramData\CrowdSec\config\online_api_credentials.yaml

tls:

cert_file: /etc/crowdsec/ssl/cert.pem

key_file: /etc/crowdsec/ssl/key.pem

prometheus:
enabled: true
level: full
listen_addr: 127.0.0.1
listen_port: 6060

File: config/config_win_no_lapi.yaml Copy Chapter

common:
log_media: file
log_level: info
log_dir: C:\ProgramData\CrowdSec\log
config_paths:
config_dir: C:\ProgramData\CrowdSec\config
data_dir: C:\ProgramData\CrowdSec\data
simulation_path: C:\ProgramData\CrowdSec\config\simulation.yaml
hub_dir: C:\ProgramData\CrowdSec\hub
index_path: C:\ProgramData\CrowdSec\hub.index.json
plugin_dir: C:\ProgramData\CrowdSec\plugins
notification_dir: C:\ProgramData\CrowdSec\config\notifications
crowdsec_service:
acquisition_path: C:\ProgramData\CrowdSec\config\acquis.yaml
parser_routines: 1
cscli:
output: human
api:
client:
insecure_skip_verify: false
credentials_path: C:\ProgramData\CrowdSec\config\local_api_credentials.yaml
prometheus:
enabled: true
level: full
listen_addr: 127.0.0.1
listen_port: 6060

File: config/console.yaml Copy Chapter

share_manual_decisions: false
share_custom: true
share_tainted: true
share_context: false

File: config/context.yaml Copy Chapter

[Empty file]

File: config/detect.yaml Copy Chapter

Main block defining all service detection rules

Since it's a map, the order of evaluation is not guaranteed.

detect:

Detect apache2, or detect log files directly (deb-based systems)

By default Systemd collects only service related messages, access logs are on file.

apache2-file-apache2:
when:
# Check if apache2 is installed (enabled or not) or the log files exist
- Systemd.UnitInstalled("apache2.service") or len(Path.Glob("/var/log/apache2/.log")) > 0
# A direct way to detect distribution family, usually not necessary.
# full list at https://github.com/shirou/gopsutil/blob/master/host/host_linux.go
# Use Host.Platform to disambiguate "ubuntu" from "debian"
# - Host.PlatformFamily == "debian"
hub_spec:
# If the rules match, install the following items
collections:
- crowdsecurity/apache2
acquisition_spec:
# If the rules match, generate an acquisition file named "setup.apache2.yaml"
filename: apache2.yaml
datasource:
source: file
filenames:
- /var/log/apache2/.log
labels:
type: apache2

Detect apache2, or detect log files directly (rpm-based systems)

apache2-file-httpd:
when:
- Systemd.UnitInstalled("httpd.service") or len(Path.Glob("/var/log/httpd/.log")) > 0
# a direct way to detect distribution family (usually not necessary)
# - Host.PlatformFamily in ['rhel', 'fedora', 'suse']
hub_spec:
collections:
- crowdsecurity/apache2
acquisition_spec:
filename: apache2.yaml
datasource:
source: file
filenames:
- /var/log/httpd/.log
labels:
type: apache2

Asterisk PBX

By default Systemd collects only service related messages, access logs are on file.

asterisk:
when:
- Systemd.UnitInstalled("asterisk.service") or len(Path.Glob("/var/log/asterisk/.log")) > 0
hub_spec:
collections:
- crowdsecurity/asterisk
acquisition_spec:
filename: asterisk.yaml
datasource:
source: file
labels:
type: asterisk
filenames:
- /var/log/asterisk/.log

Caddy web server

By default the caddy packages don't contain a "log" directive in the Caddyfile,

but the deb package has an empty /var/log/caddy directory so we look for files there

(the rpm doesn't have it).

If no log file is found, there could be a bare "log" directive which means

logs go to stderr and are captured by systemd.

It is possible that logs would go to file but no access has been logged yet.

Caddy configuration can be retrieved at http://localhost:2019/config/

caddy-journal:
when:
- Systemd.UnitInstalled("caddy.service")
- len(Path.Glob("/var/log/caddy/*.log")) == 0
hub_spec:
collections:
- crowdsecurity/caddy
acquisition_spec:
filename: caddy.yaml
datasource:
source: journalctl
labels:
type: syslog
journalctl_filter:
- "_SYSTEMD_UNIT=caddy.service"

caddy-file:
when:
- len(Path.Glob("/var/log/caddy/.log")) > 0
hub_spec:
collections:
- crowdsecurity/caddy
acquisition_spec:
filename: caddy.yaml
datasource:
source: file
labels:
type: caddy
filenames:
- /var/log/caddy/.log

Dovecot IMAP/POP3 server

Dovecot configuration can be retrieved with "doveconf -n"

and "doveadm log find"

On deb, this includes /var/log/syslog which is already monitored by the linux collection,

so we don't need an additional acquisition.

Reading from mail.log and mail.err would process the same events twice.

On rpm, we have to monitor maillog or mail.log but we can't use them to detect the program,

because it could be from postfix.

dovecot:
when:
- Systemd.UnitInstalled("dovecot.service")
hub_spec:
collections:
- crowdsecurity/dovecot
acquisition_spec:
filename: dovecot.yaml
datasource:
source: file
labels:
type: syslog
filenames:
- /var/log/maillog

Nginx

nginx:
when:
# if nginx is still detected after removing the "nginx" package,
# remove the nginx-common package too.
- Systemd.UnitInstalled("nginx.service") or len(Path.Glob("/var/log/nginx/.log")) > 0
hub_spec:
collections:
- crowdsecurity/nginx
acquisition_spec:
filename: nginx.yaml
datasource:
source: file
labels:
type: nginx
filenames:
- /var/log/nginx/.log

OpenResty (nginx distribution)

openresty:
when:
- Systemd.UnitInstalled("openresty.service") or len(Path.Glob("/usr/local/openresty/nginx/logs/.log")) > 0
hub_spec:
collections:
- crowdsecurity/nginx
acquisition_spec:
filename: openresty.yaml
datasource:
source: file
labels:
type: nginx
filenames:
- /usr/local/openresty/nginx/logs/.log

PostgreSQL

postgres:
when:
- Systemd.UnitInstalled("postgresql.service") or (len(Path.Glob("/var/log/postgresql/.log")) > 0) or (len(Path.Glob("/var/lib/pgsql/data/log/.log")) > 0)
hub_spec:
collections:
- crowdsecurity/pgsql
acquisition_spec:
filename: pgsql.yaml
datasource:
source: file
labels:
type: postgres
filenames:
# deb uses /var/log
- /var/log/postgresql/.log
# rpm uses /var/lib
- /var/lib/pgsql/data/log/.log

Postfix mail server

Postfix configuration can be queried with

"postconf maillog_file syslog_facility syslog_name"

By default maillog_file = "" and syslog_facility="mail" which

goes to either /var/log/maillog or /var/log/mail.log

postfix:
when:
- Systemd.UnitInstalled("postfix.service")
hub_spec:
collections:
- crowdsecurity/postfix
acquisition_spec:
filename: postfix.yaml
datasource:
source: file
labels:
type: syslog
filenames:
- /var/log/mail.log
- /var/log/maillog

Openssh

on deb-based systems: the name of the unit is ssh.service,

while sshd.service is an alias that can't be used to read the journal

openssh-journal-ssh:
when:
# ubuntu-minimal and recent debians don't install rsyslog by default
- Systemd.UnitInstalled("ssh.service")
- not Path.Exists("/var/log/auth.log")
- not Path.Exists("/var/log/secure")
- not Path.Exists("/var/log/messages")
hub_spec:
collections:
- crowdsecurity/sshd
acquisition_spec:
filename: sshd.yaml
datasource:
source: journalctl
labels:
type: syslog
journalctl_filter:
- "_SYSTEMD_UNIT=ssh.service"

on rpm-based systems and others: the name of the unit is sshd.service

and we have to skip this if ssh.service also exists

openssh-journal-sshd:
when:
- Systemd.UnitInstalled("sshd.service")
- not Systemd.UnitInstalled("ssh.service")
- not Path.Exists("/var/log/auth.log")
- not Path.Exists("/var/log/secure")
- not Path.Exists("/var/log/messages")
hub_spec:
collections:
- crowdsecurity/sshd
acquisition_spec:
filename: sshd.yaml
datasource:
source: journalctl
labels:
type: syslog
journalctl_filter:
- "_SYSTEMD_UNIT=sshd.service"

Acquire rsyslog files if they exist (some distributions have them in /var/log/messages,

already acquired by the linux service below)

openssh-file:
when:
- Path.Exists("/var/log/auth.log") or Path.Exists("/var/log/secure")
hub_spec:
collections:
- crowdsecurity/sshd
acquisition_spec:
filename: sshd.yaml
datasource:
source: file
labels:
type: syslog
filenames:
- /var/log/auth.log
- /var/log/secure

HAProxy load balancer

Logs go to syslog, then

- on deb, they are dispatched to /var/log/haproxy.log (via /etc/rsyslog.d/49-haproxy.conf)

- on rpm, the dispatch is missing (default -> /var/log/messages), but can be added with,

assuming rsyslog:

# /etc/rsyslog.d/10-haproxy.conf

module(load="imudp")

input(type="imudp" port="514")

local2.* /var/log/haproxy.log

& stop

haproxy:
when:
- Systemd.UnitInstalled("haproxy.service") or len(Path.Glob("/var/log/haproxy.log")) > 0
hub_spec:
collections:
- crowdsecurity/haproxy
acquisition_spec:
filename: haproxy.yaml
datasource:
source: file
labels:
type: haproxy
filenames:
- /var/log/haproxy.log

Suricata IDS/IPS

suricata:
when:
- Systemd.UnitInstalled("suricata.service") or Path.Exists("/var/log/suricata/eve.json")
hub_spec:
collections:
- crowdsecurity/suricata
acquisition_spec:
filename: suricata.yaml
datasource:
source: file
labels:
type: suricata-evelogs
filenames:
- /var/log/suricata/eve.json

Samba server

samba:
when:
# deb -> smbd.service
# rpm -> smb.service
- Systemd.UnitInstalled("smbd.service") or Systemd.UnitInstalled("smb.service") or Systemd.UnitInstalled("smb.service") or len(Path.Glob("/var/log/samba/.log")) > 0 or len(Path.Glob("/var/log/samba/log.")) > 0
hub_spec:
collections:
- crowdsecurity/smb
acquisition_spec:
filename: smb.yaml
datasource:
source: file
labels:
type: smb
filenames:
# authentication failures go to syslog (/var/log/syslog, /var/log/messages,
# already acquired by linux collection)
- /var/log/samba/.log
- /var/log/samba/log.

MariaDB server

mariadb:
when:
- Systemd.UnitInstalled("mariadb.service") or Path.Exists("/var/log/mariadb/mariadb.log")
# We can't detect error.log directly because it could come from mysql
hub_spec:
collections:
- crowdsecurity/mariadb
acquisition_spec:
filename: mariadb.yaml
datasource:
source: file
labels:
type: mysql
filenames:
# deb: by default, errors go to syslog and are acquired by the linux collection.
# They also go to the systemd journal (not acquired here to avoid redundancy).
# We add file acquisition just in case, with the location referenced in /etc/mysql/mariadb.conf.d/50-server.cnf
- /var/log/mysql/error.log
# rpm: errors go to file by default
- /var/log/mariadb/mariadb.log

MySQL server

mysql:
when:
# deb: mysql.service
# rpm: mysqld.service
- Systemd.UnitInstalled("mysql.service") or Systemd.UnitInstalled("mysqld.service") or Path.Exists("/var/log/mysql/mysqld.log")
# We can't detect error.log directly because it could come from mariadb
hub_spec:
collections:
- crowdsecurity/mysql
acquisition_spec:
filename: mysql.yaml
datasource:
source: file
labels:
type: mysql
filenames:
# deb
- /var/log/mysql/error.log
# rpm
- /var/log/mysql/mysqld.log

Audit daemon

This should be the same for deb and rpm.

Does not go through syslog.

auditd:
when:
- Systemd.UnitInstalled("auditd.service") or len(Path.Glob("/var/log/audit/.log")) > 0
hub_spec:
collections:
- crowdsecurity/auditd
acquisition_spec:
filename: auditd.yaml
datasource:
source: file
labels:
type: auditd
filenames:
- /var/log/audit/.log

There is no standard systemd unit for telnetd, so

we rely on file detection only.

telnet:
when:
- len(Path.Glob("/var/log/telnetd*.log")) > 0
hub_spec:
scenarios:
- crowdsecurity/telnet-bf
acquisition_spec:
filename: telnet.yaml
datasource:
source: file
labels:
type: telnet
filenames:
- /var/log/telnetd*.log

windows_iis:
when:
- Windows.ServiceEnabled("W3SVC")
hub_spec:
collections:
- crowdsecurity/iis
acquisition_spec:
filename: iis.yaml
datasource:
source: file
use_time_machine: true
filenames:
- C:\inetpub\logs\LogFiles**.log
labels:
type: iis

windows_sqlserver:
when:
- Windows.ServiceEnabled("MSSQLSERVER")
hub_spec:
collections:
- crowdsecurity/mssql
acquisition_spec:
filename: mssql.yaml
datasource:
source: wineventlog
event_channel: Application
event_ids:
- 18456
event_level: information
labels:
type: eventlog

Operating System detection

linux:
when:
- Host.OS == "linux"
hub_spec:
collections:
- crowdsecurity/linux
acquisition_spec:
filename: linux.yaml
datasource:
source: file
labels:
type: syslog
filenames:
# collect events that were not dispatched by facility
# (syslog or messages depending on the distribution)
# rpm
- /var/log/messages
# deb
- /var/log/syslog
- /var/log/kern.log

freebsd:
when:
- Host.OS == "freebsd"
hub_spec:
collections:
- crowdsecurity/freebsd

windows:
when:
- Host.OS == "windows"
hub_spec:
collections:
- crowdsecurity/windows
acquisition_spec:
filename: windows_auth.yaml
datasource:
source: wineventlog
event_channel: Security
event_ids:
- 4625
- 4623
event_level: information
labels:
type: eventlog

anti-lockout

whitelists:
hub_spec:
parsers:
- crowdsecurity/whitelists

------------------------------------------------

The following have not been extensively tested.

Feedback is welcome.

------------------------------------------------

Emby media server Copy Chapter

#emby-systemd:

when:

- Systemd.UnitInstalled("emby-server.service")

hub_spec:

collections:

- LePresidente/emby

acquisition_spec:

filename: emby.yaml

datasource:

source: file

labels:

type: emby

filenames:

- /var/log/embyserver.txt

Endlessh SSH tarpit Copy Chapter

#endlessh-systemd:

when:

- Systemd.UnitInstalled("endlessh.service")

hub_spec:

collections:

- crowdsecurity/endlessh

acquisition_spec:

filename: endlessh.yaml

datasource:

source: journalctl

labels:

type: endlessh

# XXX this? or /var/log/syslog?

journalctl_filter:

- "_SYSTEMD_UNIT=endlessh.service"

Odoo ERP Copy Chapter

#odoo-systemd:

when:

- Systemd.UnitInstalled("odoo.service")

hub_spec:

collections:

- crowdsecurity/odoo

acquisition_spec:

filename: odoo.yaml

datasource:

source: file

labels:

type: odoo

filenames:

- /var/log/odoo/*.log

This only works on deb-based systems. On other distributions, the Copy Chapter

application is run from the release tarball and the log location depends on Copy Chapter

the location it's run from. Copy Chapter

Ombi media request system Copy Chapter

#ombi-systemd:

when:

- Systemd.UnitInstalled("ombi.service")

- Host.PlatformFamily == "debian"

hub_spec:

collections:

- LePresidente/ombi

acquisition_spec:

filename: ombi.yaml

datasource:

source: file

labels:

type: ombi

filenames:

- /var/log/ombi/log-*.txt

#

Gitea Git server Copy Chapter

#gitea-snap:

when:

- Systemd.UnitInstalled("snap.gitea.web.service")

hub_spec:

collections:

- LePresidente/gitea

acquisition_spec:

filename: gitea.yaml

datasource:

source: journalctl

labels:

type: gitea

journalctl_filter:

- "_SYSTEMD_UNIT=gitea.service"

#gitea-journal:

when:

- Systemd.UnitInstalled("gitea.service")

- Systemd.UnitLogsToJournal("gitea.service")

hub_spec:

collections:

- LePresidente/gitea

acquisition_spec:

filename: gitea.yaml

datasource:

source: journalctl

labels:

type: gitea

journalctl_filter:

- "_SYSTEMD_UNIT=gitea.service"

#gitea-file:

when:

- Systemd.UnitInstalled("gitea.service") or Path.Exists("/var/log/gitea.log") or len(Path.Glob("/var/log/gitea/*.log")) > 0

- not Systemd.UnitLogsToJournal("gitea.service")

hub_spec:

collections:

- LePresidente/gitea

acquisition_spec:

filename: gitea.yaml

datasource:

source: file

labels:

type: gitea

filenames:

- /var/log/gitea.log

- /var/log/gitea/*.log

ProFTPD FTP server Copy Chapter

#proftpd-journal:

when:

- Systemd.UnitInstalled("proftpd.service")

- Systemd.UnitLogsToJournal("proftpd.service")

hub_spec:

collections:

- crowdsecurity/proftpd

acquisition_spec:

filename: proftpd.yaml

datasource:

source: journalctl

labels:

type: proftpd

journalctl_filter:

- "_SYSTEMD_UNIT=proftpd.service"

#proftpd-file:

when:

- Systemd.UnitInstalled("proftpd.service") or len(Path.Glob("/var/log/proftpd/*.log")) > 0

- not Systemd.UnitLogsToJournal("proftpd.service")

hub_spec:

collections:

- crowdsecurity/proftpd

acquisition_spec:

filename: proftpd.yaml

datasource:

source: file

labels:

type: proftpd

filenames:

- /var/log/proftpd/*.log

Pure-FTPd server Copy Chapter

#pureftpd-journal:

when:

- Systemd.UnitInstalled("pure-ftpd.service")

- Systemd.UnitLogsToJournal("pure-ftpd.service")

hub_spec:

collections:

- fulljackz/pureftpd

acquisition_spec:

filename: pureftpd.yaml

datasource:

source: journalctl

labels:

type: pure-ftpd

journalctl_filter:

- "_SYSTEMD_UNIT=pure-ftpd.service"

#pureftpd-file:

when:

- Systemd.UnitInstalled("pure-ftpd.service") or len(Path.Glob("/var/log/pure-ftpd/*.log")) > 0

- not Systemd.UnitLogsToJournal("pure-ftpd.service")

hub_spec:

collections:

- fulljackz/pureftpd

acquisition_spec:

filename: pureftpd.yaml

datasource:

source: file

labels:

type: pure-ftpd

filenames:

- /var/log/pure-ftpd/*.log

vsftpd FTP server Copy Chapter

#vsftpd-journal:

when:

- Systemd.UnitInstalled("vsftpd.service")

- Systemd.UnitLogsToJournal("vsftpd.service")

hub_spec:

collections:

- crowdsecurity/vsftpd

acquisition_spec:

filename: vsftpd.yaml

datasource:

source: journalctl

labels:

type: vsftpd

journalctl_filter:

- "_SYSTEMD_UNIT=vsftpd.service"

#vsftpd-file:

when:

- Systemd.UnitInstalled("vsftpd.service") or len(Path.Glob("/var/log/vsftpd/*.log")) > 0

- not Systemd.UnitLogsToJournal("vsftpd.service")

hub_spec:

collections:

- crowdsecurity/vsftpd

acquisition_spec:

filename: vsftpd.yaml

datasource:

source: file

labels:

type: vsftpd

filenames:

- /var/log/vsftpd/*.log

File: config/dev.yaml Copy Chapter

common:
log_media: stdout
log_level: info
config_paths:
config_dir: "$CONFIG_DIR"
data_dir: "$DATA_DIR"
notification_dir: "$CONFIG_DIR/notifications/"
plugin_dir: "$PLUGINS_DIR"
#simulation_path: /etc/crowdsec/config/simulation.yaml
#hub_dir: /etc/crowdsec/hub/
#index_path: ./config/hub/.index.json
crowdsec_service:
acquisition_path: "$CONFIG_DIR/acquis.yaml"
parser_routines: 1
plugin_config:
user: "$USER" # plugin process would be ran on behalf of this user
group: "$USER" # plugin process would be ran on behalf of this group
cscli:
output: human
db_config:
type: sqlite
db_path: "$DATA_DIR/crowdsec.db"
user: root
password: crowdsec
db_name: crowdsec
host: "172.17.0.2"
port: 3306
flush:
#max_items: 10000
#max_age: 168h
api:
client:
credentials_path: "$CONFIG_DIR/local_api_credentials.yaml"
server:
console_path: "$CONFIG_DIR/console.yaml"
#insecure_skip_verify: true
listen_uri: 127.0.0.1:8081
profiles_path: "$CONFIG_DIR/profiles.yaml"
tls:
#cert_file: ./cert.pem
#key_file: ./key.pem
online_client: # Central API
credentials_path: "$CONFIG_DIR/online_api_credentials.yaml"
prometheus:
enabled: true
level: full

File: config/local_api_credentials.yaml Copy Chapter

url: http://127.0.0.1:8080

File: config/online_api_credentials.yaml Copy Chapter

[Empty file]

File: config/profiles.yaml Copy Chapter

name: default_ip_remediation
#debug: true
filters:

• Alert.Remediation == true && Alert.GetScope() == "Ip"
  decisions:
• type: ban
  duration: 4h

#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)

notifications:

- slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this.

- splunk_default # Set the splunk url and token in /etc/crowdsec/notifications/splunk.yaml before enabling this.

- http_default # Set the required http parameters in /etc/crowdsec/notifications/http.yaml before enabling this.

- email_default # Set the required email parameters in /etc/crowdsec/notifications/email.yaml before enabling this.

on_success: break Copy Chapter

name: default_range_remediation
#debug: true
filters:

• Alert.Remediation == true && Alert.GetScope() == "Range"
  decisions:
• type: ban
  duration: 4h

#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)

notifications:

- slack_default # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this.

- splunk_default # Set the splunk url and token in /etc/crowdsec/notifications/splunk.yaml before enabling this.

- http_default # Set the required http parameters in /etc/crowdsec/notifications/http.yaml before enabling this.

- email_default # Set the required email parameters in /etc/crowdsec/notifications/email.yaml before enabling this.

on_success: break

File: config/simulation.yaml Copy Chapter

simulation: false

exclusions:

- crowdsecurity/ssh-bf

File: config/user.yaml Copy Chapter

common:
log_media: stdout
log_level: info
log_dir: /var/log/
config_paths:
config_dir: /etc/crowdsec/
data_dir: /var/lib/crowdsec/data
#simulation_path: /etc/crowdsec/config/simulation.yaml
#hub_dir: /etc/crowdsec/hub/
#index_path: ./config/hub/.index.json
crowdsec_service:
#acquisition_path: ./config/acquis.yaml
parser_routines: 1
cscli:
output: human
db_config:
type: sqlite
db_path: /var/lib/crowdsec/data/crowdsec.db
user: crowdsec
#log_level: info
password: crowdsec
db_name: crowdsec
host: "127.0.0.1"
port: 3306
api:
client:
insecure_skip_verify: false # default true
credentials_path: /etc/crowdsec/local_api_credentials.yaml
server:
#log_level: info
listen_uri: 127.0.0.1:8080
profiles_path: /etc/crowdsec/profiles.yaml
online_client: # Central API
credentials_path: /etc/crowdsec/online_api_credentials.yaml
prometheus:
enabled: true
level: full

File: config/windows-dev.yaml Copy Chapter

common:
log_media: stdout
log_level: info
config_paths:
config_dir: "%CONFIG_DIR%"
data_dir: "%DATA_DIR%"
notification_dir: "%CONFIG_DIR%/notifications/"
plugin_dir: "%PLUGINS_DIR%"
#simulation_path: /etc/crowdsec/config/simulation.yaml
#hub_dir: /etc/crowdsec/hub/
#index_path: ./config/hub/.index.json
crowdsec_service:
acquisition_path: "%CONFIG_DIR%/acquis.yaml"
parser_routines: 1
plugin_config:
user: "%USERNAME%" # plugin process would be ran on behalf of this user
group: "%USERNAME%" # plugin process would be ran on behalf of this group
cscli:
output: human
db_config:
type: sqlite
db_path: "%DATA_DIR%/crowdsec.db"
user: root
password: crowdsec
db_name: crowdsec
host: "172.17.0.2"
port: 3306
flush:
#max_items: 10000
#max_age: 168h
api:
client:
credentials_path: "%CONFIG_DIR%/local_api_credentials.yaml"
server:
console_path: "%CONFIG_DIR%/console.yaml"
#insecure_skip_verify: true
listen_uri: 127.0.0.1:8081
profiles_path: "%CONFIG_DIR%/profiles.yaml"
tls:
#cert_file: ./cert.pem
#key_file: ./key.pem
online_client: # Central API
credentials_path: "%CONFIG_DIR%/online_api_credentials.yaml"
prometheus:
enabled: true
level: full

File: test/README.md Copy Chapter

What is this?

This directory contains scripts for functional testing. They complement
the unit tests and provide an environment that can be used during development
as well.

The tests are run with the bats-core
framework, which is provided as git submodules in the crowdsec repository.

With the addition of the ansible playbooks it is possible
to use VMs to test the binary packages, service management and other CPU
architectures.

How to use it

pre-requisites Copy Chapter

(for building crowdsec)

• go
• pkg-config
• re2

On Ubuntu/Debian, you can install the above with:

$ sudo apt install pkgconf golang-go libre2-dev
...

(for the test suite)

• git submodule init; git submodule update
• bash>=4.4
• curl
• daemonize
• jq
• python3

These can be installed with

$ sudo apt install bash curl daemonize jq python3
...

Additional dependencies are required if you need to test with mysql or postgres
(see below).

Running the tests Copy Chapter

Run make clean bats-all to perform a test build + run.

If a test is failing, you can run only the script that contains it with

$ ./test/run-tests test/bats/<script>.bats
...

After changing the code, rebuild the binaries with make bats-build and re-run
the test to verify.

Most tests are isolated, you don't need to run all the script:
see ./test/run-tests --help for all the options.

Example: ./test/run-tests test/bats/02_nolapi.bats -f "cscli config backup"
(the string is a regexp).
You need to provide a path for a test file or directory (even if it's the full 'test/bats')
to use the -f option.

Keep in mind that some of the files in test/bats are meant as a "test suite" so
the individual tests will fail when run in isolation, or out of order.

Do not attempt to run the functional tests in parallel since they reuse
the same file locations.

Writing tests

In BATS, you write tests in the form of Bash functions that have unique
descriptions (the name of the test). You can do most things that you can
normally do in a shell function. When an error is encountered, the test
fails. A set of functions is provided to implement assertions, and a mechanism
of setup/teardown is provided at the level of individual tests (functions)
or group of tests (files).

The stdout/stderr of the commands within the test function are captured by
bats-core and will only be shown if the test fails. If you want to always print
something to debug your test case, you can redirect the output to the file
descriptor 3:

@test "mytest" {
   echo "hello world!" >&3
   run some-command
   assert_success
   echo "goodbye." >&3
}

If you do that, please remove it once the test development is finished, because
this practice breaks the TAP protocol (unless each line has a '#' as first
character, but really, it's better to avoid unnecessary output when tests succeed).

You can find here the documentation for the main framework and the plugins
we use in this test suite:

• bats-core tutorial
• Writing tests
• bats-assert
• bats-support
• bats-file

setup and teardown

If you have read the bats-core tutorial linked above, you are aware of the
setup and teardown functions.

What you may have overlooked is that the script body outside the functions is
executed multiple times, so we have to be careful of what we put there.

Here we have a look at the execution flow with two tests:

echo "begin" >&3

setup_file() {
        echo "setup_file" >&3
}

teardown_file() {
        echo "teardown_file" >&3
}

setup() {
        echo "setup" >&3
}

teardown() {
        echo "teardown" >&3
}

@test "test 1" {
        echo "test #1" >&3
}

@test "test 2" {
        echo "test #2" >&3
}

echo "end" >&3

The above test suite produces the following output:

begin
end
setup_file
begin
end
 ✓ test 1
setup
test #1
teardown
begin
end
 ✓ test 2
setup
test #2
teardown
teardown_file

See how "begin" and "end" are repeated three times each? The code outside
setup/teardown/test functions is really executed three times (more as you add
more tests). You can put there variables or function definitions, but keep it
to a minimum and don't write anything to the standard
output.
For most things you want to use setup_file() instead.

But.. there is a but. Quoting from the FAQ:

You can simply source .sh files. However, be aware that sourceing files with errors outside of any function (or inside setup_file) will trip
up bats and lead to hard to diagnose errors. Therefore, it is safest to only
source inside setup or the test functions themselves.

This doesn't mean you can't do that, just that you're on your own if the is an error.

Testing crowdsec

Fixtures Copy Chapter

For the purpose of functional tests, crowdsec and its companions (cscli, plugin
notifiers, bouncers) are installed in a local environment, which means tests
should not install or touch anything outside a ./test/local directory. This
includes binaries, configuration files, databases, data downloaded from
internet, logs... The use of /tmp is tolerated, but BATS also provides three
useful
variables:
$BATS_SUITE_TMPDIR, $BATS_FILE_TMPDIR and $BATS_TEST_TMPDIR that let you
ensure your desired level of isolation of temporary files across the tests.

When built with make bats-build, the binaries will look there by default for
their configuration and data needs. So you can run ./local/bin/cscli from
a shell with no need for further parameters.

To set up the installation described above we provide a couple of scripts,
instance-data and instance-crowdsec. They manage fixture and background
processes; they are meant to be used in setup/teardown in several ways,
according to the specific needs of the group of tests in the file.

• instance-data make

  Creates a tar file in ./local-init/init-config-data.tar.
  The file contains all the configuration, hub and database files needed
  to restore crowdsec to a known initial state.
  Things like machines add ..., capi register, hub update, collections install crowdsecurity/linux are executed here so they don't need to be
  repeated for each test or group of tests.

• instance-data load

  Extracts the files created by instance-data make for use by the local
  crowdsec instance. Crowdsec must not be running while this operation is
  performed.

• instance-data lock/unlock

When playing around with a local crowdsec installation, you can run "instance-data lock"
to prevent the bats suite from running, so it won't overwrite your configuration or data.

• instance-crowdsec [ start | stop ]

  Runs (or stops) crowdsec as a background process. PID and lockfiles are
  written in ./local/var/run/.

Here are some ways to use these two scripts.

• case 1: load a fresh crowdsec instance + data for each test (01_base, 10_bouncers, 20_collections...)

  This offers the best isolation, but the tests run slower. More importantly,
  since there is no concept of "grouping" tests in bats-core with the exception
  of files, if you need to perform some setup that is common to two or more
  tests, you will have to repeat the code.

• case 2: load a fresh set of data for each test, but run crowdsec only for
  the tests that need it, possibly after altering the configuration
  (02_nolapi, 03_noagent, 04_nocapi, 40_live-ban)

  This is useful because: 1) you sometimes don't want crowdsec to run at all,
  for example when testing cscli in isolation, or you may want to tweak the
  configuration inside the test function before running the lapi/agent. See
  how we use yq to change the YAML files to that effect.

• case 3: start crowdsec with the initial set of configuration+data once, and keep it
  running for all the tests (50_simulation, 98_ipv4, 98_ipv6)

  This offers no isolation across tests, which over time could break more
  often as result, but you can rely on the test order to test more complex
  scenarios with a reasonable performance and the least amount of code.

status, stdout and stderr Copy Chapter

As we said, if any error occurs inside a test function, the test
fails immediately. You call mycommand, it exits with $? != 0, the test fails.

But how to test the output, then? If we call run mycommand, then $? will be 0
allowing the test to keep running. The real error status is stored in the
$status variable, and the command output and standard error content are put
together in the $output variable. By specifying run --separate-stderr, you
can have separated $output and $stderr variables.

The above is better explained in the bats-core tutorial. If you have not read it
yet, now is a good time.

For convenience, the rune function is an alias for run --separate-stderr, which
can be used in most cases. For example, you don't want extraneous log messages in
the way when you check the output of a command.

The $output variable gets special treatment with the
bats-support and
[bats-assert][https://github.com/bats-core/bats-assert) plugins and can be
checked with assert_* commands. The $stderr variable does not have these,
but we can use run echo "$stderr" and then check $output with asserts.

Remember that run always overwrites the $output variable, so if you consume
it with run jq <(output) you can only do it once, because the second time it
will read the output of the jq command. But you can construct a list of all
the values you want and check them all in a single step.

Note that <(output) is substituted with the file name of a file descriptor,
so mycmd <(output) can become mycmd /dev/fd/23, mycmd /tmp//sh-np.hpc7Zs
or mycmd /proc/self/fd/38 depending on the platform. To have it fed to
standard input, use < <(output).

See the lib/*.sh and bats/*.bats files for other tricks we employ.

file operations Copy Chapter

We included the bats-file plugin to
check the result of file system operations: existence, type/size/ownership checks
on files, symlinks, directories, sockets.

gotchas Copy Chapter

• pay attention to tests that are not run - for example "bats warning: Executed 143
  instead of expected 144 tests". They are especially tricky to debug.

• using the load command in teardown() causes tests to be silently skipped or break in "funny"
  ways. The other functions seem safe.

Testing with MySQL and Postgres

By default, the tests are run with the embedded sqlite database engine. This should be
enough in most cases, since the database operations are abstracted via the ent ORM.

You can however easily test with a different engine.

Postgres Copy Chapter

Run Postgres somewhere, version 10 or above - easy to do in a docker container.

You also need to install a postgresql-client package or equivalent, to provide
recent pg_dump and pg_restore executables (not older than the PG version in the docker container).

$ sudo docker run --detach --name=postgres -p 5432:5432 --env="POSTGRES_PASSWORD=postgres" postgres:latest

The name of the container is not really important.
If you are not using Docker, you may need to adjust the PGHOST/PGPORT/PGPASSWORD/PGUSER variables
(defaults are 127.0.0.1, 5432, postgres, postgres).

An additional user and database both named crowdsec_test will be created.

Now you can build and run the tests:

$ export DB_BACKEND=pgx
$ make clean bats-build bats-fixture bats-test

The value of DB_BACKEND must not change between the build/fixture/test steps.

MySQL/MariaDB Copy Chapter

Same considerations as above, with the following changes:

$ sudo docker run --cap-add=sys_nice --detach --name=mysql -p 3306:3306  --env="MYSQL_ROOT_PASSWORD=password" mysql
[...]
$ export DB_BACKEND=mysql
$ make clean bats-build bats-fixture bats-test

or for MariaDB

$ sudo docker run --cap-add=sys_nice --detach --name=mariadb -p 3306:3306  --env="MYSQL_ROOT_PASSWORD=password" mariadb

A mysql-client package is required as well.

troubleshooting Copy Chapter

• CAPI is disabled, why?
  Most tests don't need it. Helper scripts are provided in test/enable-capi
  and test/disable-capi for interactive use, and two library functions
  config_enable_capi and config_disable_capi to call inside the tests.
  You still need to call cscli capi register after enabling it.

• My tests are hanging forever, why?
  See if you have a jq/yq or similar process waiting for standard input. Hint:
  you can pass a file from the result of the previous run command with
  <(output). This substitutes the expression with a file name, but if you
  really want it in standard input, you have to use < <(output). Bash is
  awesome but the syntax is often weird.

• I can't do X with jq.
  If you prefer you can use yq. It can parse and generate json, and it has a
  different syntax.

• I get "while parsing /tmp/....: yaml: line 5: mapping values are not allowed in this context"
  Check the heredocs (the <<EOT blocks). Each line must start with a hard TAB
  followed by spaces. You are probably missing some tabs.

File: test/ansible/README.md Copy Chapter

Ansible playbooks for functional testing

These playbooks allow you to test crowdsec in a real environment, where the
application is running as root, deployed with the OS package manager, and
uses the standard init system for the distribution (systemd or other).

This way, you can test not only the application's feature but also the packaging
boilerplate, integration scripts, and compatibility with new distribution releases,
operating systems, or architectures.

The ansible hosts should be expendable machines with at least 1GB RAM, do not
expect them to be stable if you use them for anything else after the tests.

Install (or update) the requirements with ansible-galaxy install -r requirements.yml --force.

There are several Ansible playbooks. You can use run-all.yml to configure the
installation and run the tests, or run the playbooks separately to iterate while developing.

• run-all.yml: run the other playbooks in the correct order.

• provision-dependencies.yml: install the bats requirements (bash, cfssl, etc.), compilers, and database.

• provision-test-suite.yml: install the tests scripts and bats environment, and the crowdsec sources if we want to build the crowdsec under test.

• install_binary_package.yml: install the crowdsec under test from a binary package (already released or not).

• prepare-tests.yml: create the test fixture data.

• run-tests.yml: run the functional tests. This is not idempotent and can be run multiple times.

The tasks use the following environment variables. They must be exported or
ansible won't be able to see them.

• TEST_SUITE_GIT (default "https://github.com/crowdsecurity/crowdsec"),
  TEST_SUITE_VERSION (default "master"): repo URL and branch/tag/commit of
  the crowdsec sources containing the test fixture and scripts.

• TEST_SUITE_ZIP: optional, archive of a crowdsecurity/crowdsec repository
  containing the test fixture and scripts. Overrides TEST_SUITE_GIT and
  TEST_SUITE_VERSION. It can be created with zip -r crowdsec.zip . from
  the root directory of the repository.

• DB_BACKEND: Required. Set to "sqlite", "pgx", "mysql", "postgres".
  Postgres is automatically provisioned when required. There is no
  provisioning code for mysql/mariadb yet, but it can be added.

• PACKAGE_TESTING: when set to false or not defined, the crowdsec binaries
  to be tested are built from the sources that come from TEST_SUITE_GIT or
  TEST_SUITE_ZIP. Crowdsec is then run as non-root, in a local directory.
  This is basically a fancy wrapper to run make bats-test in a vm.
  When PACKAGE_TESTING is set to true, however, crowdsec is installed from
  a binary package (see variables below), it is run as root from systemd (or
  equivalent) and uses the system-wide /etc/crowdsec and /var/lib
  directories to store the test data.

• TEST_PACKAGE_VERSION_DEB, TEST_PACKAGE_VERSION_RPM: Optional, the
  version of the package under test (ex. "1.4.0-rc5"), can be in the
  packagecloud "stable" or "testing" repository. Requires
  PACKAGE_TESTING=true. You must set both variables to reuse the same set of
  variables for Debian and RedHat-based distributions, because stable releases
  require a package version suffix in the RPM file names.

• TEST_PACKAGE_FILE: optional, file pointing to the package under test (.deb,
  .rpm, .pkg...). It can be a glob expression but it must match a single file,
  and the pattern works only on the filename. If both TEST_PACKAGE_VERSION_*
  and TEST_PACKAGE_FILE are provided, both are be installed (to test upgrades
  for example). Requires PACKAGE_TESTING=true

• TEST_PACKAGE_DIR: optional (but conflicts with TEST_PACKAGE_FILE), the path
  to a directory containing packages with the following layout:

  For DEB: {{ package_dir }}/{{ ansible_distribution_release }}/crowdsec_*_{{ ansible_architecture.replace('x86_64', 'amd64') }}.deb
  For RPM: {{ package_dir }}/{{ releasever }}/RPMS/{{ ansible_architecture }}/crowdsec-*.{{ releasever }}.{{ ansible_architecture }}.rpm

• TEST_SKIP: optional, comma-separated list of scripts that won't be executed.
  Example: TEST_SKIP=02_nolapi.bats,03_noagent.bats

Running tests with Vagrant + Ansible Copy Chapter

You don't need Vagrant to run the ansible tests, if you can manage your own
vm creation and inventory.

However, to avoid relying on (and paying for..) a public cloud, we wrote vagrant
configuration files for the most common distributions we support.

To test with Vagrant, you need to:

• have a working libvirt environment (if you can use virt-manager to create VMs, you're golden)

• install the vagrant-libvirt plugin (vagrant plugin install vagrant-libvirt.
  If it complains about gem versions, blame Ruby and see if you can remove some
  other conflicting plugin).

• copy one of the ./env/*.sh scripts to environment.sh, edit to your
  needs, and execute it with "source environment.sh"

• cd vagrant/<distro-of-your-choice>

• vagrant up --no-provision; vagrant provision. The first command creates
  the VM, the second installs all the dependencies, test suite and package
  under test, then runs the tests. If you run a plain vagrant up, it does
  everything with a single command, but also destroys the VM in case of test
  failure so you are left with nothing to debug.

• vagrant destroy when you want to remove the VM. If you want to free up the
  space taken by the base VM images, they are in
  /var/lib/libvirt/images/*VAGRANT*

The above steps are automated in the script ./prepare-run (requires bash

=4.4). It takes an environment file, and optionally a list of directories with
vagrant configurations. With a single parameter, it loops over all the
directories in alphabetical order, excluding those in the experimental
directory. Watch out for running VMs if you break the loop by hand.

After this, you will find up to 30GB of base images in /var/lib/libvirt/images,
which you need to remove by hand when you have finished testing or leave them
around for the next time.

You can give more memory or CPU juice to the VMs by editing Vagrantfile.common.

Test Matrix Copy Chapter

Tests fail with unsupported configurations or when the environment is not prepared correctly
due to missing setup/teardown parts in Ansible or functional tests. False positives
are also possible due to timing issues or flaky network connections.

If you have a result that deviates from the following matrix, that's probably a genuine bug or regression.
The data was created with crowdsec v1.4.1.

Note: all tests with local/<database> are expected to pass for pkg/<database> as well.

wip - missing ansible or bats parts, could be fixed in a future release

old-db - the database that ships with the distribution is not supported
(Postgres < 10). Won't fix, feel free to install the DB from an unofficial
repository.

0 - MySQL or MariaDB, depending on distribution defaults

1 - ansible may hang, passes all tests if run by hand

File: test/ansible/debug_tools.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Install debug tools"
  hosts: all
  become: true
  tasks:
  • name: "Install debug tools"
    ansible.builtin.apt:
    pkg:
    - kitty
    - neovim
    - zsh
    - zsh-autosuggestions
    - zsh-syntax-highlighting
    - zsh-theme-powerlevel9k
    - silversearcher-ag
    
    when:
    • ansible_facts.os_family == "Debian"

File: test/ansible/install_binary_package.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Install and set up binary crowdsec package..."
  hosts: all
  gather_facts: true
  tasks:

  • name: "Hardcode master branch for the hub, temporary override before install (config.yaml.local)"
    become: true
    block:

    • name: "Create /etc/crowdsec"
      ansible.builtin.file:
      path: "/etc/crowdsec"
      state: directory
      mode: 0o0755
    • name: "Create /etc/crowdsec/config.yaml.local"
      ansible.builtin.copy:
      dest: "/etc/crowdsec/config.yaml.local"
      content: "{{ config_yaml_local | to_nice_yaml }}"
      mode: 0o600
      vars:
      config_yaml_local:
      cscli:
      hub_branch: master

    when:

    • (package_version_deb | length > 0) or
      (package_version_rpm | length > 0) or
      (package_file | length > 0) or
      (package_dir | length > 0)

  • name: "Install crowdsec binaries from a binary repository"
    ansible.builtin.include_role:
    name: crowdsecurity.testing.install_package_from_repo
    when: (package_version_deb | length > 0) or
    (package_version_rpm | length > 0)

  • name: "Install crowdsec binaries from a package file"
    ansible.builtin.include_role:
    name: crowdsecurity.testing.install_package_from_file
    when: package_file | length > 0

  • name: "Install crowdsec binaries from a package directory"
    ansible.builtin.include_role:
    name: crowdsecurity.testing.install_package_from_pkgdir
    when: package_dir | length > 0

  • name: "Hardcode master branch for the hub, for real this time"
    become: true
    block:

    • name: "Read config.yaml"
      ansible.builtin.slurp:
      path: "/etc/crowdsec/config.yaml"
      register: config_yaml
    • name: "Create fact from config.yaml"
      ansible.builtin.set_fact:
      config_data: "{{ config_yaml['content'] | b64decode | from_yaml }}"
    • name: "Patch dictionary"
      ansible.builtin.set_fact:
      config_data: "{{ config_data | combine(config_patch, recursive=True) }}"
      vars:
      config_patch:
      cscli:
      hub_branch: master
    • name: "Write patched config.yaml"
      ansible.builtin.copy:
      content: '{{ config_data | to_nice_yaml }}'
      dest: "/etc/crowdsec/config.yaml"

      preserve mode to be able to test permissions from package

      mode: preserve
    • name: "Remove config.yaml.local"
      ansible.builtin.file:
      path: "/etc/crowdsec/config.yaml.local"
      state: absent
      when:
    • (package_version_deb | length > 0) or
      (package_version_rpm | length > 0) or
      (package_file | length > 0) or
      (package_dir | length > 0)

  this is required to avoid fatal errors in case systemctl is not working

  (which happens on some aws instances)

  • name: "Override acquis.yaml for package testing"
    become: true
    ansible.builtin.copy:
    dest: "/etc/crowdsec/acquis.yaml"
    content: "{{ acquis_yaml | to_nice_yaml }}"
    mode: preserve
    vars:
    acquis_yaml:
    filenames:
    - /tmp/should-not-exist.log
    labels:
    type: syslog
    force_inotify: true
    
    when:
    • (package_version_deb | length > 0) or
      (package_version_rpm | length > 0) or
      (package_file | length > 0) or
      (package_dir | length > 0)

  vars:
  package_version_deb: >-
  {{ lookup('ansible.builtin.env', 'TEST_PACKAGE_VERSION_DEB') }}
  package_version_rpm: >-
  {{ lookup('ansible.builtin.env', 'TEST_PACKAGE_VERSION_RPM') }}
  package_file: >-
  {{ lookup('ansible.builtin.env', 'TEST_PACKAGE_FILE') }}
  package_dir: >-
  {{ lookup('ansible.builtin.env', 'TEST_PACKAGE_DIR') }}
  binary_package_name: >-
  crowdsec

File: test/ansible/prepare_tests.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Prepare fixture for the functional tests"
  hosts: all
  gather_facts: true
  vars_files:
  • vars/go.yml
  • vars/mysql.yml
  • vars/postgres.yml
    roles:
  • name: make_fixture
    environment:
    PGHOST: 127.0.0.1
    PGPORT: 5432
    PGPASSWORD: "{{ postgresql_users[0].password }}"
    PGUSER: postgres
    MYSQL_HOST: localhost
    MYSQL_PORT: 3306
    MYSQL_PASSWORD: "{{ mysql_root_password }}"
    MYSQL_USER: "root"

File: test/ansible/provision_dependencies.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Fix EOL'd centos Stream 8"
  hosts: all
  tasks:

  • name: "update repositories file"
    ansible.builtin.find:
    paths: /etc/yum.repos.d
    patterns: "*.repo"
    register: "repo_files"
    when:
    • ansible_facts.distribution == "CentOS"
    • ansible_facts.distribution_major_version == '8'
  • name: Replace old text with new text
    become: true
    ansible.builtin.replace:
    path: "{{ item.path }}"
    regexp: 'mirrorlist'
    replace: '#mirrorlist'
    loop: "{{ repo_files.files }}"
    when:
    • ansible_facts.distribution == "CentOS"
    • ansible_facts.distribution_major_version == '8'
    • repo_files.matched > 0
  • name: Replace old text with new text
    become: true
    ansible.builtin.replace:
    path: "{{ item.path }}"
    regexp: '#baseurl=http://mirror.centos.org'
    replace: 'baseurl=https://vault.centos.org'
    loop: "{{ repo_files.files }}"
    when:
    • ansible_facts.distribution == "CentOS"
    • ansible_facts.distribution_major_version == '8'
    • repo_files.matched > 0

• name: "Install required packages"
  hosts: all
  vars_files:

  • vars/go.yml
    roles:
  • crowdsecurity.testing.apt_update
  • crowdsecurity.testing.go
  • crowdsecurity.testing.machine_id
  • crowdsecurity.testing.epel
  • crowdsecurity.testing.git
  • crowdsecurity.testing.gcc
  • crowdsecurity.testing.gnu_make
  • crowdsecurity.testing.pkg_config
  • crowdsecurity.testing.re2
  • crowdsecurity.testing.bats_requirements

• name: "Install recent python"
  hosts: all
  vars_files:

  • vars/python.yml
    tasks:
  • name: role "crowdsecurity.testing.python3"
    ansible.builtin.include_role:
    name: crowdsecurity.testing.python3
    when:
    • ansible_facts.distribution in ['CentOS', 'OracleLinux']
    • ansible_facts.distribution_major_version == '8' or ansible_facts.distribution_major_version == '7'

• name: "Install Postgres"
  hosts: all
  become: true
  vars_files:

  • vars/postgres.yml
    tasks:
  • name: role "geerlingguy.postgresql"
    ansible.builtin.include_role:
    name: geerlingguy.postgresql

    enable this for debugging Copy Chapter

    vars:

    postgres_users_no_log: false

    when:
    • lookup('ansible.builtin.env', 'DB_BACKEND') in ['pgx', 'postgres']

• name: "Install MySQL"
  hosts: all
  become: true
  vars_files:

  • vars/mysql.yml
    tasks:
  • name: role "geerlingguy.mysql"
    ansible.builtin.include_role:
    name: geerlingguy.mysql
    when:
    • lookup('ansible.builtin.env', 'DB_BACKEND') == 'mysql'

File: test/ansible/provision_test_suite.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Fetch the test scripts"
  hosts: all
  tasks:

  • name: "Fetch the sources from a git repository"
    ansible.builtin.include_role:
    name: crowdsecurity.testing.download_sources_from_git
    when: sources_zip | length == 0

  • name: "Extract the sources from a zip archive"
    ansible.builtin.include_role:
    name: crowdsecurity.testing.extract_sources_from_zip
    when: sources_zip | length > 0

  • name: "Create crowdsec test/local dir"
    become: false
    ansible.builtin.file:
    path: "{{ ansible_env.HOME }}/crowdsec/test/local"
    state: directory
    mode: 0o755

  vars:
  sources_dest_dir: "{{ ansible_env.HOME }}/crowdsec"
  sources_git_repo: >-
  {{ lookup('ansible.builtin.env', 'TEST_SUITE_GIT')
  | default('https://github.com/crowdsecurity/crowdsec', True) }}
  sources_git_version: >-
  {{ lookup('ansible.builtin.env', 'TEST_SUITE_VERSION')
  | default('master', True) }}
  sources_zip: >-
  {{ lookup('ansible.builtin.env', 'TEST_SUITE_ZIP') }}

File: test/ansible/requirements.yml Copy Chapter

vim: set ft=yaml.ansible:

roles:

• src: geerlingguy.mysql
• src: https://github.com/crowdsecurity/ansible-role-postgresql
  version: crowdsec
  name: geerlingguy.postgresql

these should be included as dependencies of crowdsecurity.testing, but sometime are not

• src: geerlingguy.repo-epel
• src: gantsign.golang

collections:

• name: ansible.posix
• name: https://github.com/crowdsecurity/ansible-collection-crowdsecurity.testing.git
  type: git
  version: v0.0.7

- name: crowdsecurity.testing

source: ../../../crowdsecurity.testing

type: dir

File: test/ansible/run_all.yml Copy Chapter

vim: set ft=yaml.ansible:

• import_playbook: provision_dependencies.yml
• import_playbook: provision_test_suite.yml
• import_playbook: install_binary_package.yml
• import_playbook: prepare_tests.yml
• import_playbook: run_tests.yml

File: test/ansible/run_tests.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Run functional tests"
  hosts: all
  gather_facts: true
  vars_files:
  • vars/mysql.yml
  • vars/postgres.yml
    roles:
  • name: run_func_tests
    environment:
    PGHOST: 127.0.0.1
    PGPORT: 5432
    PGPASSWORD: "{{ postgresql_users[0].password }}"
    PGUSER: postgres
    MYSQL_HOST: localhost
    MYSQL_PORT: 3306
    MYSQL_PASSWORD: "{{ mysql_root_password }}"
    MYSQL_USER: "root"

File: test/ansible/run_wizard_tests.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Run setup/wizard tests"
  hosts: all
  tasks:

- name: "Build release package"

become: false

ansible.builtin.command:

cmd: "make clean release"

- name: "Run BATS: service detection tests"
  become: true
  ansible.builtin.command:
    chdir: "/home/{{ ansible_user }}"
    cmd: "./crowdsec/test/run-tests crowdsec/test/bats-detect"

File: test/ansible/roles/make_fixture/tasks/main.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Set make_cmd = make (!bsd)"
  ansible.builtin.set_fact:
  make_cmd: make
  when:

  • ansible_facts.system not in ['FreeBSD', 'OpenBSD']

• name: "Set make_cmd = gmake (bsd)"
  ansible.builtin.set_fact:
  make_cmd: gmake
  when:

  • ansible_facts.system in ['FreeBSD', 'OpenBSD']

• name: "Build crowdsec from sources, prepare test environment and fixture"
  become: false
  block:

  • name: "Make bats-build bats-fixture"
    ansible.builtin.command:

    static build and we don't have to mess with LD_LIBRARY_PATH

    cmd: "{{ make_cmd }} bats-build bats-fixture BUILD_STATIC=1"
    chdir: "{{ ansible_env.HOME }}/crowdsec"
    creates: "{{ ansible_env.HOME }}/crowdsec/test/local-init/init-config-data.tar"
    environment:
    DB_BACKEND: "{{ lookup('ansible.builtin.env', 'DB_BACKEND') }}"

    daemonize -> /usr/bin or /usr/local/sbin

    pidof -> /usr/sbin

    bash -> /opt/bash/bin

    PATH: "/opt/bash/bin:{{ ansible_env.PATH }}:{{ golang_install_dir }}/bin/:/usr/sbin:/usr/local/sbin"
    rescue:
  • name: "Read crowdsec.log"
    ansible.builtin.slurp:
    path: "{{ ansible_env.HOME }}/crowdsec/test/local/var/log/crowdsec.log"
    register: crowdsec_log
  • name: "Show crowdsec.log"
    ansible.builtin.fail:
    msg: "{{ crowdsec_log['content'] | b64decode }}"
    when: (package_testing is not defined) or (package_testing in ['', 'false', 'False'])

• name: "Prepare test environment and fixture for binary package"
  become: true
  block:

  • name: "Make bats-environment bats-check-requirements bats-fixture"
    ansible.builtin.command:
    cmd: "{{ make_cmd }} bats-environment bats-check-requirements bats-fixture"
    chdir: "{{ ansible_env.HOME }}/crowdsec"
    creates: "{{ ansible_env.HOME }}/crowdsec/test/local-init/init-config-data.tar"
    environment:
    PACKAGE_TESTING: "{{ package_testing }}"
    DB_BACKEND: "{{ lookup('ansible.builtin.env', 'DB_BACKEND') }}"

    daemonize -> /usr/bin or /usr/local/sbin

    pidof -> /usr/sbin

    bash -> /opt/bash/bin

    PATH: "/opt/bash/bin:{{ ansible_env.PATH }}:{{ golang_install_dir }}/bin/:/usr/sbin:/usr/local/sbin"
    rescue:
  • name: "Read crowdsec.log"
    ansible.builtin.slurp:
    path: "/var/log/crowdsec.log"
    register: crowdsec_log
  • name: "Show crowdsec.log"
    ansible.builtin.fail:
    msg: "{{ crowdsec_log['content'] | b64decode }}"
    when: (package_testing is defined) and (package_testing not in ['', 'false', 'False'])

• name: "Debug - show environment"
  become: false
  block:

  • name: "Look for .environment.sh"
    ansible.builtin.slurp:
    src: "{{ ansible_env.HOME }}/crowdsec/test/.environment.sh"
    changed_when: true
    register: envfile
  • name: "Show .environment.sh"
    ansible.builtin.debug:
    msg: "{{ envfile['content'] | b64decode }}"
  • name: "Show environment variables"
    ansible.builtin.debug:
    msg: "{{ ansible_env | to_nice_yaml }}"

File: test/ansible/roles/make_fixture/vars/main.yml Copy Chapter

vim: set ft=yaml.ansible:

package_testing: "{{ lookup('ansible.builtin.env', 'PACKAGE_TESTING') }}"

File: test/ansible/roles/run_func_tests/tasks/main.yml Copy Chapter

vim: set ft=yaml.ansible:

• name: "Tweak systemd configuration for tests"
  become: true
  block:

  • name: "Create /lib/systemd/system/crowdsec.service.d"
    ansible.builtin.file:
    owner: root
    group: root
    mode: 0o755
    path: /lib/systemd/system/crowdsec.service.d
    state: directory
  • name: "Override StartLimitBurst"
    ansible.builtin.ini_file:
    dest: /lib/systemd/system/crowdsec.service.d/startlimitburst.conf
    owner: root
    group: root
    mode: 0o644
    section: Service
    option: StartLimitBurst
    value: 100
  • name: "Systemctl daemon-reload"
    ansible.builtin.systemd:
    daemon_reload: true
    when:
  • (package_testing is defined) and (package_testing not in ['', 'false', 'False'])
  • ansible_facts.os_family in ["RedHat", "Debian"]

• name: "Debug - show environment.sh"
  become: false
  block:

  • name: "Look for .environment.sh"
    ansible.builtin.slurp:
    src: "{{ ansible_env.HOME }}/crowdsec/test/.environment.sh"
    changed_when: true
    register: envfile
  • name: "Show .environment.sh"
    ansible.builtin.debug:
    msg: "{{ envfile['content'] | b64decode }}"

• name: "Search for test scripts"
  become: false
  ansible.builtin.find:
  paths: "{{ ansible_env.HOME }}/crowdsec/test/bats"
  pattern: "*.bats"
  register: testfiles

• name: "Run BATS tests for source build"
  become: false
  block:

  • name: "Run test scripts"
    ansible.builtin.command:
    cmd: test/run-tests {{ item.path }}
    chdir: "{{ ansible_env.HOME }}/crowdsec"
    with_items: "{{ testfiles.files | sort(attribute='path') }}"
    loop_control:
    label: "{{ item['path'] }}"
    environment:

    daemonize -> /usr/bin or /usr/local/sbin

    pidof -> /usr/sbin

    bash -> /opt/bash/bin

    PATH: "/opt/bash/bin:{{ ansible_env.PATH }}:/usr/sbin:/usr/local/sbin"
    changed_when: true
    when:
    • (item.path | basename) not in skip_tests.split(',')
      rescue:
  • name: "Read crowdsec.log"
    ansible.builtin.slurp:
    path: "{{ ansible_env.HOME }}/crowdsec/test/local/var/log/crowdsec.log"
    register: crowdsec_log
  • name: "Show crowdsec.log"
    ansible.builtin.fail:
    msg: "{{ crowdsec_log['content'] | b64decode }}"
    when:
  • (package_testing is not defined) or (package_testing in ['', 'false', 'False'])

• name: "Run BATS tests for binary package"
  become: true
  block:

  • name: "Run test scripts"
    ansible.builtin.command:
    cmd: test/run-tests {{ item.path }}
    chdir: "{{ ansible_env.HOME }}/crowdsec"
    with_items: "{{ testfiles.files | sort(attribute='path') }}"
    loop_control:
    label: "{{ item['path'] }}"
    environment:

    daemonize -> /usr/bin or /usr/local/sbin

    pidof -> /usr/sbin

    bash -> /opt/bash/bin

    PATH: "/opt/bash/bin:{{ ansible_env.PATH }}:/usr/sbin:/usr/local/sbin"
    changed_when: true
    when:
    • (item.path | basename) not in skip_tests.split(',')
      rescue:
  • name: "Read crowdsec.log"
    ansible.builtin.slurp:
    path: "/var/log/crowdsec.log"
    register: crowdsec_log
  • name: "Show crowdsec.log"
    ansible.builtin.fail:
    msg: "{{ crowdsec_log['content'] | b64decode }}"
    when:
  • (package_testing is defined) and (package_testing not in ['', 'false', 'False'])

File: test/ansible/roles/run_func_tests/vars/main.yml Copy Chapter

vim: set ft=yaml.ansible:

package_testing: "{{ lookup('ansible.builtin.env', 'PACKAGE_TESTING') }}"
skip_tests: "{{ lookup('ansible.builtin.env', 'TEST_SKIP') }}"

File: test/ansible/vagrant/experimental/amazon-linux-2/issues.txt Copy Chapter

The file 70_http_plugin.bats hangs forever when run from ansible on amzn2, but all tests pass when run from ssh.

File: test/ansible/vars/go.yml Copy Chapter

vim: set ft=yaml.ansible:

golang_version: "1.21.4"
golang_install_dir: "/opt/go/{{ golang_version }}"

File: test/ansible/vars/mysql.yml Copy Chapter

vim: set ft=yaml.ansible:

The password is insecure since the db is ephemeral and only listen to localhost.

mysql_root_password: password

File: test/ansible/vars/postgres.yml Copy Chapter

vim: set ft=yaml.ansible:

The password is insecure since the db is ephemeral and only listen to localhost.

postgresql_users:

• name: postgres
  password: postgres

postgresql_hba_entries:

• type: local
  database: all
  user: postgres
  auth_method: peer

• type: local
  database: all
  user: all
  auth_method: peer

• type: host
  database: all
  user: all
  address: "127.0.0.1/32"
  auth_method: md5

• type: host
  database: all
  user: all
  address: "::1/128"
  auth_method: md5

File: test/ansible/vars/python.yml Copy Chapter

python_version: "3.12.3"

File: test/bats/testdata/90_decisions/decisions.json Copy Chapter

[
{
"origin": "cscli",
"scope": "ip",
"value": "1.6.11.16",
"reason": "manual import from csv",
"type": "ban",
"duration": "1h"
},
{
"origin": "cscli",
"scope": "ip",
"value": "2.7.12.17",
"reason": "manual import from csv",
"type": "ban",
"duration": "1h"
},
{
"origin": "cscli",
"scope": "ip",
"value": "3.8.13.18",
"reason": "manual import from csv",
"type": "ban",
"duration": "1h"
},
{
"origin": "cscli",
"scope": "ip",
"value": "4.9.14.19",
"reason": "manual import from csv",
"type": "ban",
"duration": "1h"
},
{
"origin": "cscli",
"scope": "ip",
"value": "5.10.15.20",
"reason": "manual import from csv",
"type": "ban",
"duration": "1h"
}
]

File: test/bats/testdata/cfssl/agent.json Copy Chapter

{
"CN": "localhost",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "agent-ou",
"ST": "France"
}
]
}

File: test/bats/testdata/cfssl/agent_invalid.json Copy Chapter

{
"CN": "localhost",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "this-is-not-the-ou-youre-looking-for",
"ST": "France"
}
]
}

File: test/bats/testdata/cfssl/bouncer.json Copy Chapter

{
"CN": "localhost",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "bouncer-ou",
"ST": "France"
}
]
}

File: test/bats/testdata/cfssl/bouncer_invalid.json Copy Chapter

{
"CN": "localhost",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "this-is-not-the-ou-youre-looking-for",
"ST": "France"
}
]
}

File: test/bats/testdata/cfssl/ca_intermediate.json Copy Chapter

{
"CN": "CrowdSec Test CA Intermediate",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "Crowdsec Intermediate",
"ST": "France"
}
],
"ca": {
"expiry": "42720h"
}
}

File: test/bats/testdata/cfssl/ca_root.json Copy Chapter

{
"CN": "CrowdSec Test CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "Crowdsec",
"ST": "France"
}
]
}

File: test/bats/testdata/cfssl/profiles.json Copy Chapter

{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"intermediate_ca": {
"usages": [
"signing",
"key encipherment",
"cert sign",
"crl sign",
"server auth",
"client auth"
],
"expiry": "8760h",
"ca_constraint": {
"is_ca": true,
"max_path_len": 0,
"max_path_len_zero": true
}
},
"server": {
"usages": [
"server auth"
],
"expiry": "8760h"
},
"client": {
"usages": [
"client auth"
],
"expiry": "8760h"
}
}
}
}

File: test/bats/testdata/cfssl/server.json Copy Chapter

{
"CN": "localhost",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "FR",
"L": "Paris",
"O": "Crowdsec",
"OU": "Crowdsec Server",
"ST": "France"
}
],
"hosts": [
"127.0.0.1",
"localhost"
]
}

File: test/bats/testdata/cscli-setup/detect.yaml Copy Chapter

TODO: windows, use_time_machine, event support (see https://hub.crowdsec.net/author/crowdsecurity/collections/iis)

detect:
apache2:
when:
- System.ProcessRunning("apache2")
hub_spec:
collections:
- crowdsecurity/apache2
acquisition_spec:
filename: apache.yaml
datasource:
source: file
labels:
type: apache2
filenames:
- /var/log/apache2/.log
- /var/log/http/.log
- /var/log/httpd/*.log

apache2-systemd:
when:
- Systemd.UnitInstalled("apache2.service")
- OS.ID != "centos"
hub_spec:
collections:
- crowdsecurity/apache2
acquisition_spec:
filename: apache.yaml
datasource:
source: journalctl
journalctl_filter:
- "_SYSTEMD_UNIT=mock-apache2.service"
labels:
type: apache2

apache2-systemd-centos:
when:
- Systemd.UnitInstalled("httpd.service")
- OS.ID == "centos"
hub_spec:
collections:
- crowdsecurity/apache2
acquisition_spec:
filename: apache.yaml
datasource:
source: journalctl
journalctl_filter:
- "_SYSTEMD_UNIT=httpd.service"

ssh-systemd:
when:
- Systemd.UnitInstalled("ssh.service") or Systemd.UnitInstalled("ssh.socket")
hub_spec:
collections:
- crowdsecurity/apache2
acquisition_spec:
filename: ssh.yaml
datasource:
source: journalctl
journalctl_filter:
- "_SYSTEMD_UNIT=ssh.service"
labels:
type: syslog

linux:
when:
- Host.OS == "linux"
hub_spec:
collections:
- crowdsecurity/linux
acquisition_spec:
filename: linux.yaml
datasource:
source: file
labels:
type: syslog
filenames:
- /var/log/syslog
- /var/log/kern.log
- /var/log/messages

freebsd:
when:
- Host.OS == "freebsd"
hub_spec:
collections:
- crowdsecurity/freebsd

windows:
when:
- Host.OS == "windows"
hub_spec:
collections:
- crowdsecurity/windows

File: test/bats/testdata/explain/explain-log.txt Copy Chapter

line: Sep 19 18:33:22 scw-d95986 sshd[24347]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4
├ s00-raw
| └ 🟢 crowdsecurity/syslog-logs (+12 ~9)
├ s01-parse
| └ 🟢 crowdsecurity/sshd-logs (+8)
├-------- parser success 🟢
├ Scenarios
├ 🟢 crowdsecurity/ssh-bf
├ 🟢 crowdsecurity/ssh-bf_user-enum
├ 🟢 crowdsecurity/ssh-slow-bf
├ 🟢 crowdsecurity/ssh-slow-bf_user-enum
├ 🟢 crowdsecurity/ssh-time-based-bf
└ 🟢 crowdsecurity/ssh-time-based-bf_user-enum

File: test/bats-detect/WARNING.md Copy Chapter

Running the tests in this directory WILL change the system configuration in
unpredictable ways, remove packages and data (with a peculiar appetite for
databases) and possibly bring the system to an unusable state.

They are meant to be run, as root, on temporary VMs. They are only intended to
ease the development of configurations for "cscli setup detect".

File: test/localstack/docker-compose.yml Copy Chapter

services:
localstack:
container_name: localstack_main
image: localstack/localstack:3.0
network_mode: bridge
ports:
- "127.0.0.1:53:53" # only required for Pro (DNS)
- "127.0.0.1:53:53/udp" # only required for Pro (DNS)
- "127.0.0.1:443:443" # only required for Pro (LocalStack HTTPS Edge Proxy)
- "127.0.0.1:4510-4559:4510-4559" # external service port range
- "127.0.0.1:4566:4566" # LocalStack Edge Proxy
environment:
AWS_HOST: localstack
DEBUG: ""
KINESIS_ERROR_PROBABILITY: ""
LOCALSTACK_HOST: "localstack"
AWS_REGION: "us-east-1"

volumes:
  - "${TMPDIR:-/tmp}/localstack:/var/lib/localstack"
  - "/var/run/docker.sock:/var/run/docker.sock"
  - "./scripts/init_script.sh:/etc/localstack/init/ready.d/init_script.sh"

zoo1:
image: confluentinc/cp-zookeeper:7.4.3
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_SERVERS: zoo1:2888:3888

healthcheck:
  test: jps -l | grep zookeeper
  interval: 10s
  retries: 5
  timeout: 5s

kafka1:
image: crowdsecurity/kafka-ssl
ports:
- "9093:9093"
- "9092:9092"
- "9999:9999"
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://127.0.0.1:19092,LISTENER_DOCKER_EXTERNAL://127.0.0.1:9092,LISTENER_DOCKER_EXTERNAL_SSL://127.0.0.1:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL_SSL:SSL
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_JMX_PORT: 9999
KAFKA_JMX_HOSTNAME: "127.0.0.1"
KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
KAFKA_SSL_KEYSTORE_FILENAME: kafka.kafka1.keystore.jks
KAFKA_SSL_KEYSTORE_CREDENTIALS: kafka1_keystore_creds
KAFKA_SSL_KEY_CREDENTIALS: kafka1_sslkey_creds
KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.kafka1.truststore.jks
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kafka1_truststore_creds
KAFKA_SSL_ENABLED_PROTOCOLS: TLSv1.2
KAFKA_SSL_PROTOCOL: TLSv1.2
KAFKA_SSL_CLIENT_AUTH: none
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

healthcheck:
  test: kafka-broker-api-versions --version
  interval: 10s
  retries: 5
  timeout: 10s

loki:
image: grafana/loki:2.9.1
ports:
- "127.0.0.1:3100:3100"

victorialogs:
image: victoriametrics/victoria-logs:v1.5.0-victorialogs
ports:
- "9428:9428"

File: .github/codecov.yml Copy Chapter

we measure coverage but don't enforce it

https://docs.codecov.com/docs/codecov-yaml

codecov:
require_ci_to_pass: false

coverage:
status:
patch:
default:
target: 0%
project:
default:
target: 0%

if a directory is ignored, there is no way to un-ignore files like pkg/models/helpers.go

so we make a full list

ignore:

• "./pkg/database/ent/alert/alert.go"
• "./pkg/database/ent/alert_create.go"
• "./pkg/database/ent/alert_delete.go"
• "./pkg/database/ent/alert.go"
• "./pkg/database/ent/alert_query.go"
• "./pkg/database/ent/alert_update.go"
• "./pkg/database/ent/alert/where.go"
• "./pkg/database/ent/allowlist/allowlist.go"
• "./pkg/database/ent/allowlist_create.go"
• "./pkg/database/ent/allowlist_delete.go"
• "./pkg/database/ent/allowlist.go"
• "./pkg/database/ent/allowlistitem/allowlistitem.go"
• "./pkg/database/ent/allowlistitem_create.go"
• "./pkg/database/ent/allowlistitem_delete.go"
• "./pkg/database/ent/allowlistitem.go"
• "./pkg/database/ent/allowlistitem_query.go"
• "./pkg/database/ent/allowlistitem_update.go"
• "./pkg/database/ent/allowlistitem/where.go"
• "./pkg/database/ent/allowlist_query.go"
• "./pkg/database/ent/allowlist_update.go"
• "./pkg/database/ent/allowlist/where.go"
• "./pkg/database/ent/bouncer/bouncer.go"
• "./pkg/database/ent/bouncer_create.go"
• "./pkg/database/ent/bouncer_delete.go"
• "./pkg/database/ent/bouncer.go"
• "./pkg/database/ent/bouncer_query.go"
• "./pkg/database/ent/bouncer_update.go"
• "./pkg/database/ent/bouncer/where.go"
• "./pkg/database/ent/client.go"
• "./pkg/database/ent/configitem/configitem.go"
• "./pkg/database/ent/configitem_create.go"
• "./pkg/database/ent/configitem_delete.go"
• "./pkg/database/ent/configitem.go"
• "./pkg/database/ent/configitem_query.go"
• "./pkg/database/ent/configitem_update.go"
• "./pkg/database/ent/configitem/where.go"
• "./pkg/database/ent/decision_create.go"
• "./pkg/database/ent/decision/decision.go"
• "./pkg/database/ent/decision_delete.go"
• "./pkg/database/ent/decision.go"
• "./pkg/database/ent/decision_query.go"
• "./pkg/database/ent/decision_update.go"
• "./pkg/database/ent/decision/where.go"
• "./pkg/database/ent/ent.go"
• "./pkg/database/ent/enttest/enttest.go"
• "./pkg/database/ent/event_create.go"
• "./pkg/database/ent/event_delete.go"
• "./pkg/database/ent/event/event.go"
• "./pkg/database/ent/event.go"
• "./pkg/database/ent/event_query.go"
• "./pkg/database/ent/event_update.go"
• "./pkg/database/ent/event/where.go"
• "./pkg/database/ent/hook/hook.go"
• "./pkg/database/ent/lock_create.go"
• "./pkg/database/ent/lock_delete.go"
• "./pkg/database/ent/lock.go"
• "./pkg/database/ent/lock/lock.go"
• "./pkg/database/ent/lock_query.go"
• "./pkg/database/ent/lock_update.go"
• "./pkg/database/ent/lock/where.go"
• "./pkg/database/ent/machine_create.go"
• "./pkg/database/ent/machine_delete.go"
• "./pkg/database/ent/machine.go"
• "./pkg/database/ent/machine/machine.go"
• "./pkg/database/ent/machine_query.go"
• "./pkg/database/ent/machine_update.go"
• "./pkg/database/ent/machine/where.go"
• "./pkg/database/ent/meta_create.go"
• "./pkg/database/ent/meta_delete.go"
• "./pkg/database/ent/meta.go"
• "./pkg/database/ent/meta/meta.go"
• "./pkg/database/ent/meta_query.go"
• "./pkg/database/ent/meta_update.go"
• "./pkg/database/ent/meta/where.go"
• "./pkg/database/ent/metric_create.go"
• "./pkg/database/ent/metric_delete.go"
• "./pkg/database/ent/metric.go"
• "./pkg/database/ent/metric/metric.go"
• "./pkg/database/ent/metric_query.go"
• "./pkg/database/ent/metric_update.go"
• "./pkg/database/ent/metric/where.go"
• "./pkg/database/ent/migrate/migrate.go"
• "./pkg/database/ent/migrate/schema.go"
• "./pkg/database/ent/mutation.go"
• "./pkg/database/ent/predicate/predicate.go"
• "./pkg/database/ent/runtime.go"
• "./pkg/database/ent/runtime/runtime.go"
• "./pkg/database/ent/tx.go"
• "./pkg/models/add_alerts_request.go"
• "./pkg/models/add_alerts_response.go"
• "./pkg/models/add_signals_request.go"
• "./pkg/models/add_signals_request_item_decisions.go"
• "./pkg/models/add_signals_request_item_decisions_item.go"
• "./pkg/models/add_signals_request_item.go"
• "./pkg/models/add_signals_request_item_source.go"
• "./pkg/models/alert.go"
• "./pkg/models/all_metrics.go"
• "./pkg/models/allowlist_item.go"
• "./pkg/models/base_metrics.go"
• "./pkg/models/bulk_check_allowlist_request.go"
• "./pkg/models/bulk_check_allowlist_response.go"
• "./pkg/models/bulk_check_allowlist_result.go"
• "./pkg/modelscapi/add_signals_request.go"
• "./pkg/modelscapi/add_signals_request_item_decisions.go"
• "./pkg/modelscapi/add_signals_request_item_decisions_item.go"
• "./pkg/modelscapi/add_signals_request_item.go"
• "./pkg/modelscapi/add_signals_request_item_source.go"
• "./pkg/modelscapi/allowlist_link.go"
• "./pkg/modelscapi/blocklist_link.go"
• "./pkg/modelscapi/decisions_delete_request.go"
• "./pkg/modelscapi/decisions_delete_request_item.go"
• "./pkg/modelscapi/decisions_sync_request.go"
• "./pkg/modelscapi/decisions_sync_request_item_decisions.go"
• "./pkg/modelscapi/decisions_sync_request_item_decisions_item.go"
• "./pkg/modelscapi/decisions_sync_request_item.go"
• "./pkg/modelscapi/decisions_sync_request_item_source.go"
• "./pkg/modelscapi/enroll_request.go"
• "./pkg/modelscapi/error_response.go"
• "./pkg/modelscapi/get_decisions_stream_response_deleted.go"
• "./pkg/modelscapi/get_decisions_stream_response_deleted_item.go"
• "./pkg/modelscapi/get_decisions_stream_response.go"
• "./pkg/modelscapi/get_decisions_stream_response_links.go"
• "./pkg/modelscapi/get_decisions_stream_response_new.go"
• "./pkg/modelscapi/get_decisions_stream_response_new_item.go"
• "./pkg/modelscapi/login_request.go"
• "./pkg/modelscapi/login_response.go"
• "./pkg/modelscapi/metrics_request_bouncers_item.go"
• "./pkg/modelscapi/metrics_request.go"
• "./pkg/modelscapi/metrics_request_machines_item.go"
• "./pkg/modelscapi/register_request.go"
• "./pkg/modelscapi/reset_password_request.go"
• "./pkg/modelscapi/success_response.go"
• "./pkg/models/check_allowlist_response.go"
• "./pkg/models/console_options.go"
• "./pkg/models/decision.go"
• "./pkg/models/decisions_delete_request.go"
• "./pkg/models/decisions_delete_request_item.go"
• "./pkg/models/decisions_stream_response.go"
• "./pkg/models/delete_alerts_response.go"
• "./pkg/models/delete_decision_response.go"
• "./pkg/models/detailed_metrics.go"
• "./pkg/models/error_response.go"
• "./pkg/models/event.go"
• "./pkg/models/flush_decision_response.go"
• "./pkg/models/get_alerts_response.go"
• "./pkg/models/get_allowlist_response.go"
• "./pkg/models/get_allowlists_response.go"
• "./pkg/models/get_decisions_response.go"
• "./pkg/models/hub_item.go"
• "./pkg/models/hub_items.go"
• "./pkg/models/lapi_metrics.go"
• "./pkg/models/log_processors_metrics.go"
• "./pkg/models/meta.go"
• "./pkg/models/metrics_agent_info.go"
• "./pkg/models/metrics_bouncer_info.go"
• "./pkg/models/metrics_detail_item.go"
• "./pkg/models/metrics.go"
• "./pkg/models/metrics_labels.go"
• "./pkg/models/metrics_meta.go"
• "./pkg/models/o_sversion.go"
• "./pkg/models/remediation_components_metrics.go"
• "./pkg/models/source.go"
• "./pkg/models/success_response.go"
• "./pkg/models/topx_response.go"
• "./pkg/models/watcher_auth_request.go"
• "./pkg/models/watcher_auth_response.go"
• "./pkg/models/watcher_registration_request.go"
• "./pkg/protobufs/notifier_grpc.pb.go"
• "./pkg/protobufs/notifier.pb.go"

File: .github/dependabot.yml Copy Chapter

version: 2
updates:

• package-ecosystem: "github-actions"
  directory: "/"
  schedule:
  interval: "weekly"
  open-pull-requests-limit: 10
  labels:

  • "kind/dependencies"
  • "github-actions"
    groups:
    github-actions:
    update-types:
    • "minor"
    • "patch"

• package-ecosystem: "gomod"
  directory: "/"
  schedule:
  interval: "weekly"
  open-pull-requests-limit: 10
  labels:

  • "kind/dependencies"
  • "go"
    groups:
    gomod:
    update-types:
    • "minor"
    • "patch"

• package-ecosystem: "uv"
  directory: "/build/docker/test"
  schedule:
  interval: "weekly"
  open-pull-requests-limit: 10
  labels:

  • "kind/dependencies"
  • "python"
    groups:
    uv:
    update-types:
    • "minor"
    • "patch"

• package-ecosystem: "docker"
  directories:

  • "/build/docker"
    schedule:
    interval: "weekly"
    open-pull-requests-limit: 10
    labels:
  • "kind/dependencies"
  • "docker"
    groups:
    docker:
    update-types:
    • "minor"
    • "patch"

File: .github/governance.yml Copy Chapter

version: v1

issue:
captures:
- regex: 'version: v*(.+)-[rc*]?'
github_release: true
ignore_case: true
label: 'version/$CAPTURED'

- regex: 'Platform: *(windows?|ms|wins?|microsoft).*'
  label: 'os/win'
  ignore_case: true

- regex: 'Platform: *(freebsd|bsd).*'
  label: 'os/freebsd'
  ignore_case: true

- regex: 'Platform: *(linux|linus|lin).*'
  label: 'os/linux'
  ignore_case: true

- regex: 'Platform: *(macos|mac|apple|macintosh|macbook).*'
  label: 'os/mac'
  ignore_case: true

labels:
- prefix: triage
list: ['accepted']
multiple: false
author_association:
collaborator: true
member: true
owner: true
needs:
comment: |
@$AUTHOR: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

      1. Check [Crowdsec Documentation](https://docs.crowdsec.net/) to see if your issue can be self resolved.
      2. You can also join our [Discord](https://discord.gg/crowdsec).
      3. Check [Releases](https://github.com/crowdsecurity/crowdsec/releases/latest) to make sure your agent is on the latest version.

- prefix: kind
  list: ['feature', 'bug', 'packaging', 'enhancement', 'refactoring']
  multiple: false
  author_association:
    author: true
    collaborator: true
    member: true
    owner: true
  needs:
    comment: |
      @$AUTHOR: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.
      * `/kind feature`
      * `/kind enhancement`
      * `/kind refactoring`
      * `/kind bug`
      * `/kind packaging`

- prefix: os
  list: ['mac', 'win', 'linux', 'freebsd']
  multiple: true

pull_request:
labels:
- prefix: kind
multiple: false
list: [ 'feature', 'enhancement', 'fix', 'chore', 'dependencies', 'refactoring']
needs:
comment: |
@$AUTHOR: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.
* /kind feature
* /kind enhancement
* /kind refactoring
* /kind fix
* /kind chore
* /kind dependencies
status:
context: 'Kind Label'
success: Ready for review & merge.
failure: Missing kind label to generate release automatically.

- prefix: area
  list: [ "agent", "local-api", "cscli", "security", "configuration", "appsec"]
  multiple: true
  needs:
    comment: |
      @$AUTHOR: There are no area labels on this PR. You can add as many areas as you see fit.
      * `/area agent`
      * `/area local-api`
      * `/area cscli`
      * `/area appsec`
      * `/area security`
      * `/area configuration`

- prefix: priority
  multiple: false
  list: [ 'urgent', 'important' ]
  author_association:
    collaborator: true
    member: true
    owner: true

File: .github/PULL_REQUEST_TEMPLATE.md Copy Chapter

What and why Copy Chapter

Fixes #

How it was tested Copy Chapter

Checklist Copy Chapter

• One concern only.
• No break to LAPI/CAPI payloads, database schema, config keys, or cscli -o json|raw output.
• A human has reviewed this diff line by line.
• A human has tested this change, not only an agent.

AI assistance used:

File: .github/release-drafter.yml Copy Chapter

categories:

• title: 'New Features'
  labels:
  • 'new feature'
  • 'kind/feature'
• title: 'Improvements'
  labels:
  • 'kind/enhancement'
  • 'enhancement'
  • 'improvement'
• title: 'Bug Fixes'
  labels:
  • 'kind/fix'
  • 'fix'
  • 'bugfix'
  • 'bug'
• title: 'Documentation'
  labels:
  • 'documentation'
  • 'doc'
• title: 'Chore / Deps'
  labels:
  • 'kind/dependencies'
  • 'kind/chore'
    tag-template: "- $TITLE @$AUTHOR (#$NUMBER)"
    exclude-labels:
• 'skip-changelog'
  template: |

Changes Copy Chapter

$CHANGES

Geolite2 notice Copy Chapter

This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.

Installation Copy Chapter

Take a look at the installation instructions.

File: .github/ISSUE_TEMPLATE/bug_report.yaml Copy Chapter

name: Bug report
labels: kind/bug
body:

• type: textarea
  id: problem
  attributes:
  label: What happened?
  Please provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner.
  If this matter is security related, please disclose it privately to [email protected]
  
  validations:
  required: true

• type: textarea
  id: expected
  attributes:
  label: What did you expect to happen?
  validations:
  required: true

• type: textarea
  id: repro
  attributes:
  label: How can we reproduce it (as minimally and precisely as possible)?
  validations:
  required: true

• type: textarea
  id: additional
  attributes:
  label: Anything else we need to know?

• type: textarea
  id: Version
  attributes:
  label: Crowdsec version
  value: |

  text Copy

  ```console
  $ cscli version
  # paste output here
  ```

  validations:
  required: true

• type: textarea
  id: osVersion
  attributes:
  label: OS version
  value: |

  text Copy

  ```console
  # On Linux:
  $ cat /etc/os-release
  # paste output here
  $ uname -a
  # paste output here
  
  # On Windows:
  C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
  # paste output here
  ```

• type: textarea
  id: collections
  attributes:
  label: Enabled collections and parsers
  value: |

  text Copy

  ```console
  $ cscli hub list -o raw
  # paste output here
  ```

• type: textarea
  id: acquis
  attributes:
  label: Acquisition config
  value: |

  text Copy

  ```console
  # On Linux:
  $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
  # paste output here
  
  # On Windows:
  C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
  # paste output here
  ```

• type: textarea
  id: config
  attributes:
  label: Config show
  value: |

  text Copy

  ```console
  $ cscli config show
  # paste output here
  ```

• type: textarea
  id: metrics
  attributes:
  label: Prometheus metrics
  value: |

  text Copy

  ```console
  $ cscli metrics
  # paste output here
  ```

• type: textarea
  id: customizations
  attributes:
  label: "Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc."
  value: |

File: .github/ISSUE_TEMPLATE/config.yml Copy Chapter

contact_links:

• name: Support Request
  url: https://discourse.crowdsec.net
  about: Support request or question relating to Crowdsec

File: .github/ISSUE_TEMPLATE/feature_request.yaml Copy Chapter

name: Feature request
body:

• type: textarea
  id: feature
  attributes:
  label: What would you like to be added?
  Significant feature requests are unlikely to make progress as issues. Please consider engaging on discord (discord.gg/crowdsec) and forums (https://discourse.crowdsec.net), instead.
  
  value: |
  For feature request please pick a kind label by removing `` that wrap the example lines below

  validations:
  required: true

• type: textarea
  id: rationale
  attributes:
  label: Why is this needed?
  validations:
  required: true

File: .github/workflows/bats-hub.yml Copy Chapter

name: (sub) Bats / Hub

on:
workflow_call:
secrets:
GIST_BADGES_SECRET:
required: true
GIST_BADGES_ID:
required: true

permissions:
contents: read

jobs:
build:

name: "Functional tests"
runs-on: ubuntu-latest
timeout-minutes: 30
steps:

- name: "Force machineid"
  run: |
      sudo chmod +w /etc/machine-id
      echo githubciXXXXXXXXXXXXXXXXXXXXXXXX | sudo tee /etc/machine-id

- name: "Check out CrowdSec repository"
  uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
  with:
    fetch-depth: 0
    submodules: true
    persist-credentials: false

- name: "Set up Go"
  uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
  with:
    go-version-file: go.mod

- name: "Install bats dependencies"
  env:
    GOBIN: /usr/local/bin
  run: |
    sudo apt -qq -y -o=Dpkg::Use-Pty=0 install build-essential daemonize jq libre2-dev

- name: "Build crowdsec and fixture"
  # fetch the hub index from a git clone to avoid cdn-hub rate limiting (403)
  env:
    TEST_HUB_CLONE: "true"
  run: make bats-clean bats-build bats-fixture BUILD_STATIC=1

- name: "Run hub tests"
  run: |
      PATH=$(pwd)/test/local/bin:$PATH
      ./test/instance-data load
      git clone --depth 1 https://github.com/crowdsecurity/hub.git ./hub
      cd ./hub
      cscli hubtest run --all --clean --max-jobs 4

- name: "Collect hub coverage"
  run: ./test/bin/collect-hub-coverage ./hub >> $GITHUB_ENV

- name: "Update coverage badges"
  if: ${{ github.ref == 'refs/heads/master' && github.repository_owner == 'crowdsecurity' }}
  env:
    GH_TOKEN: ${{ secrets.GIST_BADGES_SECRET }}
    GIST_ID: ${{ secrets.GIST_BADGES_ID }}
  run: |
    set -euo pipefail
    # shields.io endpoint badge, written as a gist file
    badge() { jq -nc --arg l "$1" --arg m "$2" --arg c "$3" \
      '{schemaVersion:1,label:$l,message:$m,color:$c}'; }
    jq -n \
      --arg p "$(badge 'Hub Parsers'   "$PARSERS_COV"   "$PARSERS_BADGE_COLOR")" \
      --arg s "$(badge 'Hub Scenarios' "$SCENARIOS_COV" "$SCENARIOS_BADGE_COLOR")" \
      '{files:{"crowdsec_parsers_badge.json":{content:$p},
               "crowdsec_scenarios_badge.json":{content:$s}}}' \
      | gh api --method PATCH "/gists/${GIST_ID}" --input -

File: .github/workflows/bats-mysql.yml Copy Chapter

name: (sub) Bats / MySQL

on:
workflow_call:
inputs:
database_image:
required: true
type: string

permissions:
contents: read

jobs:
build:
name: "Functional tests"
runs-on: ubuntu-latest
timeout-minutes: 30
services:
database:
image: ${{ inputs.database_image }}
env:
MYSQL_ROOT_PASSWORD: "secret"
ports:
- 3306:3306

steps:

- name: "Force machineid"
  run: |
      sudo chmod +w /etc/machine-id
      echo githubciXXXXXXXXXXXXXXXXXXXXXXXX | sudo tee /etc/machine-id

- name: "Check out CrowdSec repository"
  uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
  with:
    fetch-depth: 0
    submodules: true
    persist-credentials: false

- name: "Set up Go"
  uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
  with:
    go-version-file: go.mod

- name: "Install bats dependencies"
  env:
    GOBIN: /usr/local/bin
  run: |
    sudo apt -qq -y -o=Dpkg::Use-Pty=0 install build-essential daemonize jq libre2-dev

- name: "Build crowdsec and fixture"
  run: |
    make clean bats-build bats-fixture BUILD_STATIC=1
  env:
    # fetch the hub index from a git clone to avoid cdn-hub rate limiting (403)
    TEST_HUB_CLONE: "true"
    DB_BACKEND: mysql
    MYSQL_HOST: 127.0.0.1
    MYSQL_PORT: 3306
    MYSQL_PASSWORD: "secret"
    MYSQL_USER: root

- name: "Run tests"
  run: ./test/run-tests ./test/bats --formatter $(pwd)/test/lib/color-formatter
  env:
    DB_BACKEND: mysql
    MYSQL_HOST: 127.0.0.1
    MYSQL_PORT: 3306
    MYSQL_PASSWORD: "secret"
    MYSQL_USER: root

#
# In case you need to inspect the database status after the failure of a given test
#
# - name: "Run specified tests"
#   run: ./test/run-tests test/bats/<filename>.bats -f "<test name>"

- name: Show database dump
  run: ./test/instance-db dump /dev/fd/1
  env:
    DB_BACKEND: mysql
    MYSQL_HOST: 127.0.0.1
    MYSQL_PORT: 3306
    MYSQL_PASSWORD: "secret"
    MYSQL_USER: root
  if: ${{ always() }}

- name: "Show stack traces"
  run: for file in $(find /tmp/crowdsec-crash.*.txt); do echo ">>>>> $file"; cat $file; echo; done
  if: ${{ always() }}

- name: "Show crowdsec logs"
  run: for file in $(find ./test/local/var/log -type f); do echo ">>>>> $file"; cat $file; echo; done
  if: ${{ always() }}

- name: "Show database logs"
  run: docker logs "${{ job.services.database.id }}"
  if: ${{ always() }}

File: .github/workflows/bats-postgres.yml Copy Chapter

name: (sub) Bats / Postgres

on:
workflow_call:

permissions:
contents: read

jobs:
build:
name: "Functional tests"
runs-on: ubuntu-latest
timeout-minutes: 30
services:
database:
image: postgres:16
env:
POSTGRES_PASSWORD: "secret"
ports:
- 5432:5432
options: >-
--health-cmd pg_isready -u postgres
--health-interval 10s
--health-timeout 5s
--health-retries 5

steps:

- name: "Install pg_dump v16"
  # we can remove this when it's released on ubuntu-latest
  run: |
      sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
      wget -qO- https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo tee /etc/apt/trusted.gpg.d/pgdg.asc &>/dev/null
      sudo apt update
      sudo apt -qq -y -o=Dpkg::Use-Pty=0 install postgresql-client-16

- name: "Force machineid"
  run: |
      sudo chmod +w /etc/machine-id
      echo githubciXXXXXXXXXXXXXXXXXXXXXXXX | sudo tee /etc/machine-id

- name: "Check out CrowdSec repository"
  uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
  with:
    fetch-depth: 0
    submodules: true
    persist-credentials: false

- name: "Set up Go"
  uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
  with:
    go-version-file: go.mod

- name: "Install bats dependencies"
  env:
    GOBIN: /usr/local/bin
  run: |
    sudo apt -qq -y -o=Dpkg::Use-Pty=0 install build-essential daemonize jq libre2-dev

- name: "Build crowdsec and fixture (DB_BACKEND: pgx)"
  run: |
    make clean bats-build bats-fixture BUILD_STATIC=1
  env:
    # fetch the hub index from a git clone to avoid cdn-hub rate limiting (403)
    TEST_HUB_CLONE: "true"
    DB_BACKEND: pgx
    PGHOST: 127.0.0.1
    PGPORT: 5432
    PGPASSWORD: "secret"
    PGUSER: postgres

- name: "Run tests (DB_BACKEND: pgx)"
  run: ./test/run-tests ./test/bats --formatter $(pwd)/test/lib/color-formatter
  env:
    DB_BACKEND: pgx
    PGHOST: 127.0.0.1
    PGPORT: 5432
    PGPASSWORD: "secret"
    PGUSER: postgres

- name: "Show stack traces"
  run: for file in $(find /tmp/crowdsec-crash.*.txt); do echo ">>>>> $file"; cat $file; echo; done
  if: ${{ always() }}

- name: "Show crowdsec logs"
  run: for file in $(find ./test/local/var/log -type f); do echo ">>>>> $file"; cat $file; echo; done
  if: ${{ always() }}

- name: "Show database logs"
  run: docker logs "${{ job.services.database.id }}"
  if: ${{ always() }}

File: .github/workflows/bats-sqlite-coverage.yml Copy Chapter

name: (sub) Bats / sqlite + coverage

on:
workflow_call:
secrets:
CODECOV_TOKEN:
required: true

permissions:
contents: read

env:
TEST_COVERAGE: true

jobs:
build:
name: "Functional tests"
runs-on: ubuntu-latest
timeout-minutes: 20

steps:

- name: "Force machineid"
  run: |
      sudo chmod +w /etc/machine-id
      echo githubciXXXXXXXXXXXXXXXXXXXXXXXX | sudo tee /etc/machine-id

- name: "Check out CrowdSec repository"
  uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
  with:
    fetch-depth: 0
    submodules: true
    persist-credentials: false

- name: "Set up Go"
  uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
  with:
    go-version-file: go.mod

- name: "Install bats dependencies"
  env:
    GOBIN: /usr/local/bin
  run: |
    sudo apt -qq -y -o=Dpkg::Use-Pty=0 install build-essential daemonize jq libre2-dev

- name: "Build crowdsec and fixture"
  # fetch the hub index from a git clone to avoid cdn-hub rate limiting (403)
  env:
    TEST_HUB_CLONE: "true"
  run: |
    make clean bats-build bats-fixture BUILD_STATIC=1

- name: Generate codecov configuration
  run: |
      .github/generate-codecov-yml.sh >> .github/codecov.yml

- name: "Run tests"
  run: ./test/run-tests ./test/bats --formatter $(pwd)/test/lib/color-formatter

- name: "Collect coverage data"
  run: |
    go tool covdata textfmt -i test/coverage -o coverage-bats.out

#
# In case you need to inspect the database status after the failure of a given test
#
# - name: "Run specified tests"
#   run: ./test/run-tests test/bats/<filename>.bats -f "<test name>"

- name: "Show database dump"
  run: |
    ./test/instance-crowdsec stop
    sqlite3 ./test/local/var/lib/crowdsec/data/crowdsec.db '.dump'
  if: ${{ always() }}

- name: "Show stack traces"
  run: for file in $(find /tmp/crowdsec-crash.*.txt); do echo ">>>>> $file"; cat $file; echo; done
  if: ${{ always() }}

- name: "Show crowdsec logs"
  run: for file in $(find ./test/local/var/log -type f); do echo ">>>>> $file"; cat $file; echo; done
  if: ${{ always() }}

- name: Upload bats coverage to codecov
  uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
  with:
    files: ./coverage-bats.out
    flags: bats
    token: ${{ secrets.CODECOV_TOKEN }}

File: .github/workflows/bats.yml Copy Chapter

This workflow is actually running

only functional tests, but the

name is used for the badge in README.md

name: Tests

Main workflow for functional tests, it calls all the others through parallel jobs.

https://docs.github.com/en/actions/using-workflows/reusing-workflows

There is no need to merge coverage output because codecov.io should take care of that.

on:
push:
branches:
- master
- releases/**
paths-ignore:
- "README.md"
pull_request:
paths-ignore:
- "README.md"

permissions:
contents: read

jobs:
sqlite:
uses: ./.github/workflows/bats-sqlite-coverage.yml
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

Jobs for Postgres (and sometimes MySQL) can have failing tests on GitHub

CI, but they pass when run on devs' machines or in the release checks. We

disable them here by default. Remove if...false to enable them.

mariadb:
uses: ./.github/workflows/bats-mysql.yml
with:
database_image: mariadb:latest

mysql:
uses: ./.github/workflows/bats-mysql.yml
with:
database_image: mysql:latest

postgres:
uses: ./.github/workflows/bats-postgres.yml

hub:
uses: ./.github/workflows/bats-hub.yml
secrets:
GIST_BADGES_ID: ${{ secrets.GIST_BADGES_ID }}
GIST_BADGES_SECRET: ${{ secrets.GIST_BADGES_SECRET }}

File: .github/workflows/cache-cleanup.yaml Copy Chapter

https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#managing-caches

name: cleanup caches by a branch
on:
pull_request:
types:
- closed

permissions:
actions: write # to delete the branch's Actions caches

jobs:
cleanup:
runs-on: ubuntu-latest
steps:
- name: Cleanup
run: gh cache delete --all --ref "refs/pull/${PR_NUMBER}/merge" --succeed-on-no-caches
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}

File: .github/workflows/ci-windows-build-msi.yml Copy Chapter

name: build-msi (windows)

on:
release:
types:
- prereleased
push:
branches:
- master
- releases/**
paths:
- build/windows/installer/.wxs
- .github/workflows/ci-windows-build-msi.yml
pull_request:
branches:
- master
- releases/*
paths:
- build/windows/installer/*.wxs
- .github/workflows/ci-windows-build-msi.yml

permissions:
contents: read

jobs:
build:
name: Build
runs-on: windows-latest

steps:

- name: Check out code into the Go module directory
  uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
  with:
    fetch-depth: 0
    submodules: false
    persist-credentials: false

- name: "Set up Go"
  uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
  with:
    go-version-file: go.mod

- name: Build
  run: make windows_installer BUILD_RE2_WASM=1
- name: Upload MSI
  uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
  with:
    path: crowdsec*msi
    name: crowdsec.msi

File: .github/workflows/ci_release-drafter.yml Copy Chapter

name: Release Drafter

on:
push:
# branches to consider in the event; optional, defaults to all
branches:
- main
- master
- releases/**

permissions:
contents: read

jobs:
update_release_draft:
permissions:
contents: write
pull-requests: read
runs-on: ubuntu-latest
steps:
# Drafts your next Release notes as Pull Requests are merged into "master"
- uses: release-drafter/release-drafter@978bb0b30c5ee8cdd332ced253758157150d43f0 # v7.10.0
with:
# (Optional) specify config name to use, relative to .github/. Default: release-drafter.yml
config-name: release-drafter.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

File: .github/workflows/codeql-analysis.yml Copy Chapter

yamllint disable rule:comments

For most projects, this workflow file will not need changing; you simply need

to commit it to your repository.

You may wish to alter this file to override the set of languages analyzed,

or to provide custom queries or build logic.

******** NOTE ********

We have attempted to detect the languages in your repository. Please check

the `language` matrix defined below to confirm you have the correct set of

supported CodeQL languages.

name: "CodeQL"

on:
push:
branches:
- master
- releases/**
pull_request:
# The branches below must be a subset of the branches above
branches:
- master
- releases/**
schedule:
- cron: '15 16 * * 2'

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write

strategy:
  fail-fast: false
  matrix:
    language: [ 'go', 'python' ]
    # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]
    # Learn more:
    # https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed

steps:
- name: Checkout repository
  uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
  with:
    # required to pick up tags for BUILD_VERSION
    fetch-depth: 0
    persist-credentials: false

- name: "Set up Go"
  uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
  with:
    go-version-file: go.mod
    cache-dependency-path: "**/go.sum"

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
  uses: github/codeql-action/init@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3
  with:
    languages: ${{ matrix.language }}
    # If you wish to specify custom queries, you can do so here or in a config file.
    # By default, queries listed here will override any specified in a config file.
    # Prefix the list here with "+" to use these queries and those in the config file.
    # queries: ./path/to/local/query, your-org/your-repo/queries@main

# Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
# - name: Autobuild
#   uses: github/codeql-action/autobuild@181d5eefc20863364f96762470ba6f862bdef56b

# ℹ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl

# ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
#    and modify them (or add more) to build your code if your project
#    uses a compiled language

- run: |
   make clean build BUILD_RE2_WASM=1

- name: Perform CodeQL Analysis
  uses: github/codeql-action/analyze@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3

File: .github/workflows/docker-tests.yml Copy Chapter

name: Test Docker images

on:
push:
branches:
- master
- releases/**
paths-ignore:
- "README.md"
pull_request:
branches:
- master
- releases/**
paths-ignore:
- "README.md"

permissions:
contents: read

env:
CROWDSEC_BYPASS_DB_VOLUME_CHECK: "TRUE" # to avoid the db volume check in the docker_start.sh script

jobs:
test_flavor:
strategy:
# we could test all the flavors in a single pytest job,
# but let's split them (and the image build) in multiple runners for performance
matrix:
# can be slim, full or debian (no debian slim).
flavor: ["slim", "debian"]

runs-on: ubuntu-latest
timeout-minutes: 20
steps:
  - name: Check out the repo
    uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
    with:
      fetch-depth: 0
      persist-credentials: false

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@594f3bf4285d9ea8dc53c9a0c9c4092420091003 # v4.4.0
    with:
      buildkitd-config: .github/buildkit.toml

  - name: "Build image"
    uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
    with:
      context: .
      file: ./build/docker/Dockerfile${{ matrix.flavor == 'debian' && '.debian' || '' }}
      tags: crowdsecurity/crowdsec:test${{ matrix.flavor == 'full' && '' || '-' }}${{ matrix.flavor == 'full' && '' || matrix.flavor }}
      target: ${{ matrix.flavor == 'debian' && 'full' || matrix.flavor }}
      platforms: linux/amd64
      load: true
      cache-from: type=gha
      cache-to: type=gha,mode=min

  - name: "Create Docker network"
    run: docker network create net-test

  - name: Install uv
    uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
    with:
      version: 0.5.24
      enable-cache: true
      working-directory: "./build/docker/test"

  - name: "Set up Python"
    uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
    with:
      python-version-file: "./build/docker/test/.python-version"

  # running serially to reduce test flakiness
  - name: Lint and run the tests
    run: |
      cd build/docker/test
      uv sync --all-extras --dev --locked
      uv run ruff check
      uv run basedpyright
      uv run pytest tests -n 1 --durations=0 --color=yes
    env:
      CROWDSEC_TEST_VERSION: test
      CROWDSEC_TEST_FLAVORS: ${{ matrix.flavor }}
      CROWDSEC_TEST_NETWORK: net-test
      CROWDSEC_TEST_TIMEOUT: 90

File: .github/workflows/go-tests-windows.yml Copy Chapter

name: Go tests (windows)

on:
push:
branches:
- master
- releases/**
paths-ignore:
- "README.md"
pull_request:
paths-ignore:
- "README.md"

permissions:
contents: read

env:
CROWDSEC_FEATURE_DISABLE_HTTP_RETRY_BACKOFF: true

jobs:
build:
name: "Build + tests"
runs-on: windows-2025

steps:
  - name: Check out CrowdSec repository
    uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
    with:
      fetch-depth: 0
      submodules: false
      persist-credentials: false

  - name: "Set up Go"
    uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
    with:
      go-version-file: go.mod

  - name: Build
    run: |
      make build BUILD_RE2_WASM=1

  - name: Generate codecov configuration
    run: |
      .github/generate-codecov-yml.sh >> .github/codecov.yml

  - name: Unit tests
    run: |
      go install gotest.tools/[email protected]
      make testcover

  - name: Upload unit coverage to Codecov
    uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
    with:
      files: coverage.out
      flags: unit-windows
      token: ${{ secrets.CODECOV_TOKEN }}

  - name: golangci-lint
    uses: golangci/golangci-lint-action@ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a # v9.3.0
    with:
      version: v2.13
      args: --issues-exit-code=1 --timeout 10m
      only-new-issues: false

File: .github/workflows/go-tests.yml Copy Chapter

This workflow is actually running

tests (with localstack) but the

name is used for the badge in README.md

name: Build

on:
push:
branches:
- master
- releases/**
paths-ignore:
- "README.md"
pull_request:
paths-ignore:
- "README.md"

permissions:
contents: read

these env variables are for localstack, so we can emulate aws services

env:
AWS_HOST: localstack

these are to mimic aws config

AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
AWS_REGION: us-east-1
CROWDSEC_FEATURE_DISABLE_HTTP_RETRY_BACKOFF: true

jobs:
build:
name: "Build + tests"
runs-on: ubuntu-latest
services:
localstack:
image: localstack/localstack:3.0
ports:
- 4566:4566 # Localstack exposes all services on the same port
env:
DEBUG: ""
LAMBDA_EXECUTOR: ""
KINESIS_ERROR_PROBABILITY: ""
KINESIS_INITIALIZE_STREAMS: ${{ env.KINESIS_INITIALIZE_STREAMS }}
LOCALSTACK_HOST: ${{ env.AWS_HOST }} # Required so that resource urls are provided properly
# e.g sqs url will get localhost if we don't set this env to map our service
options: >-
--name=localstack
--health-cmd="curl -sS 127.0.0.1:4566 || exit 1"
--health-interval=10s
--health-timeout=5s
--health-retries=3
zoo1:
image: confluentinc/cp-zookeeper:7.4.3
ports:
- "2181:2181"
env:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_SERVERS: zoo1:2888:3888
options: >-
--name=zoo1
--health-cmd "jps -l | grep zookeeper"
--health-interval 10s
--health-timeout 5s
--health-retries 5

kafka1:
    image: crowdsecurity/kafka-ssl
    ports:
      - "9093:9093"
      - "9092:9092"
      - "9999:9999"
    env:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://127.0.0.1:19092,LISTENER_DOCKER_EXTERNAL://127.0.0.1:9092,LISTENER_DOCKER_EXTERNAL_SSL://127.0.0.1:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL_SSL:SSL
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: "127.0.0.1"
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
      KAFKA_SSL_KEYSTORE_FILENAME: kafka.kafka1.keystore.jks
      KAFKA_SSL_KEYSTORE_CREDENTIALS: kafka1_keystore_creds
      KAFKA_SSL_KEY_CREDENTIALS: kafka1_sslkey_creds
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.kafka1.truststore.jks
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kafka1_truststore_creds
      KAFKA_SSL_ENABLED_PROTOCOLS: TLSv1.2
      KAFKA_SSL_PROTOCOL: TLSv1.2
      KAFKA_SSL_CLIENT_AUTH: none
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
    options: >-
      --name=kafka1
      --health-cmd "kafka-broker-api-versions --version"
      --health-interval 10s
      --health-timeout 10s
      --health-retries 5

  loki:
    image: grafana/loki:2.9.1
    ports:
      - "3100:3100"
    options: >-
      --name=loki1
      --health-cmd "wget -q -O - http://localhost:3100/ready | grep 'ready'"
      --health-interval 30s
      --health-timeout 10s
      --health-retries 5
      --health-start-period 30s

  victorialogs:
    image: victoriametrics/victoria-logs:v1.5.0-victorialogs
    ports:
      - "9428:9428"
    options: >-
      --name=victorialogs1
      --health-cmd "wget -q -O - http://0.0.0.0:9428"
      --health-interval 30s
      --health-timeout 10s
      --health-retries 5
      --health-start-period 30s
steps:
  - name: Check out CrowdSec repository
    uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
    with:
      fetch-depth: 0
      submodules: true
      persist-credentials: false

  - name: "Set up Go"
    uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
    with:
      go-version-file: go.mod

  - name: Run "go generate" and check for changes
    run: |
      set -e
      # ensure the version of 'protoc' matches the one that generated the files
      PROTOBUF_VERSION="35.1"
      JAVY_VERSION="9.1.0"
      # sha256 of the release assets, verified before use
      PROTOBUF_SHA256="6930ebf62bd4ea607b98fff052596c6ee564b9835b4ce172c75a3f53ae9d91b7"
      JAVY_SHA256="a68b122d48eb3dfc1b801d4e14c39271fde3638243d3272d206e376ac9189e39"
      # don't pollute the repo
      pushd $HOME
      curl --fail --location --proto '=https' --tlsv1.2 -o protoc.zip https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-linux-x86_64.zip
      echo "${PROTOBUF_SHA256}  protoc.zip" | sha256sum -c -
      unzip protoc.zip -d $HOME/.protoc
      curl --fail --location --proto '=https' --tlsv1.2 -o javy.gz https://github.com/bytecodealliance/javy/releases/download/v${JAVY_VERSION}/javy-x86_64-linux-v${JAVY_VERSION}.gz
      echo "${JAVY_SHA256}  javy.gz" | sha256sum -c -
      mkdir -p $HOME/.javy
      gzip -dc javy.gz > $HOME/.javy/javy
      chmod +x $HOME/.javy/javy
      popd
      export PATH="$HOME/.protoc/bin:$HOME/.javy/:$PATH"
      go install google.golang.org/protobuf/cmd/[email protected]
      go install google.golang.org/grpc/cmd/[email protected]
      go generate ./...
      protoc --version
      if [[ $(git status --porcelain) ]]; then
        echo "Error: Uncommitted changes found after running 'go generate'. Please commit all generated code."
        git diff
        exit 1
      else
        echo "No changes detected after running 'go generate'."
      fi

  - name: Create localstack streams
    run: |
      aws --endpoint-url=http://127.0.0.1:4566 --region us-east-1 kinesis create-stream --stream-name stream-1-shard --shard-count 1
      aws --endpoint-url=http://127.0.0.1:4566 --region us-east-1 kinesis create-stream --stream-name stream-2-shards --shard-count 2

  - name: Generate codecov configuration
    run: |
      .github/generate-codecov-yml.sh > .github/codecov.yml

  - name: Ensure we can do a dynamic build
    run: |
      sudo apt -qq -y -o=Dpkg::Use-Pty=0 install build-essential libre2-dev
      make build

  - name: Ensure we can do a static build
    run: |
      make clean build BUILD_STATIC=1

  - name: Unit tests
    run: |
      go install gotest.tools/[email protected]
      # make the repo read-only, with the exception of coverage output
      touch coverage.out
      chmod -R a-w .
      chmod u+w coverage.out
      make testcover
      # ignore changes to codecov.yml
      if [[ $(git status --porcelain -- . ":(exclude).github/codecov.yml") ]]; then
        echo "Error: Unit tests should not create or alter files inside the repository. Please use the appropriate testing helpers or otherwise temporary locations."
        git diff --name-only
        exit 1
      fi
      chmod -R u+w .

  # check if some component stubs are missing
  - name: "Build profile: minimal"
    run: |
      make build BUILD_PROFILE=minimal

  # check that the freebsd build is not broken
  - name: "Build target: freebsd"
    run: |
      make build BUILD_STATIC=1 BUILD_RE2_WASM=1 BUILD_SQLITE=modernc GOOS=freebsd

  - name: Upload unit coverage to Codecov
    uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1
    with:
      files: coverage.out
      flags: unit-linux
      token: ${{ secrets.CODECOV_TOKEN }}

  - name: golangci-lint
    uses: golangci/golangci-lint-action@ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a # v9.3.0
    with:
      version: v2.13
      args: --issues-exit-code=1 --timeout 10m
      only-new-issues: false

File: .github/workflows/governance-bot.yaml Copy Chapter

.github/workflow/governance.yml

on:
pull_request_target:
types: [ synchronize, opened, labeled, unlabeled ]
issues:
types: [ opened, labeled, unlabeled ]
issue_comment:
types: [ created ]

You can use permissions to modify the default permissions granted to the GITHUB_TOKEN,

adding or removing access as required, so that you only allow the minimum required access.

permissions:
contents: read
issues: write
pull-requests: write
statuses: write
checks: write

jobs:
governance:
name: Governance
runs-on: ubuntu-latest
steps:
# Semantic versioning, lock to different version: v2, v2.0 or a commit hash.
- uses: BirthdayResearch/oss-governance-bot@3abd2d1fd2376ba9990fbc795e7a4c54254e9c61 # v4.0.0
with:
# You can use a PAT to post a comment/label/status so that it shows up as a user instead of github-actions
github-token: ${{secrets.GITHUB_TOKEN}} # optional, default to '${{ github.token }}'
config-path: .github/governance.yml # optional, default to '.github/governance.yml'

File: .github/workflows/publish-docker-master.yml Copy Chapter

name: (push-master) Publish latest Docker images

on:
push:
branches: [ master ]
paths:
- 'pkg/'
- 'cmd/'
- 'build/mk/**'
- 'build/docker/Dockerfile'
- 'build/docker/Dockerfile.debian'
- 'build/docker/docker_start.sh'
- 'build/docker/config.yaml'
- '.github/workflows/publish-docker-master.yml'
- '.github/workflows/publish-docker.yml'
- 'go.mod'
- 'go.sum'
- 'Makefile'

permissions:
contents: read
packages: write # push images to ghcr.io

jobs:
dev-alpine:
uses: ./.github/workflows/publish-docker.yml
# inherit so the reusable job's docker environment secrets resolve
secrets: inherit
with:
platform: linux/amd64
crowdsec_version: ""
image_version: dev
latest: false
push: true
slim: false
debian: false

dev-debian:
uses: ./.github/workflows/publish-docker.yml
secrets: inherit
with:
platform: linux/amd64
crowdsec_version: ""
image_version: dev
latest: false
push: true
slim: false
debian: true

File: .github/workflows/publish-docker-release.yml Copy Chapter

name: (manual) Publish Docker images

on:
workflow_dispatch:
inputs:
image_version:
required: true
crowdsec_version:
required: true
latest:
default: false
required: true
push:
default: false
required: true

permissions:
contents: read
packages: write # push images to ghcr.io

jobs:
alpine:
uses: ./.github/workflows/publish-docker.yml
# inherit so the reusable job's docker environment secrets resolve
secrets: inherit
with:
image_version: ${{ github.event.inputs.image_version }}
crowdsec_version: ${{ github.event.inputs.crowdsec_version }}
latest: ${{ github.event.inputs.latest == 'true' }}
push: ${{ github.event.inputs.push == 'true' }}
slim: true
debian: false
platform: "linux/amd64,linux/386,linux/arm64,linux/arm/v7,linux/arm/v6"

debian:
uses: ./.github/workflows/publish-docker.yml
secrets: inherit
with:
image_version: ${{ github.event.inputs.image_version }}
crowdsec_version: ${{ github.event.inputs.crowdsec_version }}
latest: ${{ github.event.inputs.latest == 'true' }}
push: ${{ github.event.inputs.push == 'true' }}
slim: false
debian: true
platform: "linux/amd64,linux/386,linux/arm64"

File: .github/workflows/publish-docker.yml Copy Chapter

name: (sub) Publish Docker images

on:
workflow_call:
inputs:
environment:
required: false
type: string
default: docker
platform:
required: true
type: string
image_version:
required: true
type: string
crowdsec_version:
required: true
type: string
latest:
required: true
type: boolean
push:
required: true
type: boolean
slim:
required: true
type: boolean
debian:
required: true
type: boolean

permissions:
contents: read
packages: write # push images to ghcr.io

jobs:
push_to_registry:
name: Push Docker image to registries
runs-on: crowdsec-large-runner
environment: ${{ inputs.environment }}
steps:
- name: Check out the repo
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
persist-credentials: false

- name: Set up QEMU
    uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@594f3bf4285d9ea8dc53c9a0c9c4092420091003 # v4.4.0
    with:
      buildkitd-config: .github/buildkit.toml

  - name: Login to DockerHub
    uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
    with:
      username: ${{ secrets.DOCKER_USERNAME }}
      password: ${{ secrets.DOCKER_PASSWORD }}

  - name: Login to GitHub Container Registry
    uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
    with:
      registry: ghcr.io
      username: ${{ github.repository_owner }}
      password: ${{ secrets.GITHUB_TOKEN }}

  - name: Prepare (slim)
    if: ${{ inputs.slim }}
    id: slim
    run: |
      DOCKERHUB_IMAGE=${{ secrets.DOCKER_USERNAME }}/crowdsec
      GHCR_IMAGE=ghcr.io/${{ github.repository_owner }}/crowdsec
      VERSION=${{ inputs.image_version }}
      DEBIAN=${{ inputs.debian && '-debian' || '' }}
      TAGS="${DOCKERHUB_IMAGE}:${VERSION}-slim${DEBIAN},${GHCR_IMAGE}:${VERSION}-slim${DEBIAN}"
      if [[ ${{ inputs.latest }} == true ]]; then
        TAGS=$TAGS,${DOCKERHUB_IMAGE}:slim${DEBIAN},${GHCR_IMAGE}:slim${DEBIAN}
      fi
      echo "tags=${TAGS}" >> $GITHUB_OUTPUT
      echo "created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT

  - name: Prepare (full)
    id: full
    run: |
      DOCKERHUB_IMAGE=${{ secrets.DOCKER_USERNAME }}/crowdsec
      GHCR_IMAGE=ghcr.io/${{ github.repository_owner }}/crowdsec
      VERSION=${{ inputs.image_version }}
      DEBIAN=${{ inputs.debian && '-debian' || '' }}
      TAGS="${DOCKERHUB_IMAGE}:${VERSION}${DEBIAN},${GHCR_IMAGE}:${VERSION}${DEBIAN}"
      if [[ ${{ inputs.latest }} == true ]]; then
        TAGS=$TAGS,${DOCKERHUB_IMAGE}:latest${DEBIAN},${GHCR_IMAGE}:latest${DEBIAN}
      fi
      echo "tags=${TAGS}" >> $GITHUB_OUTPUT
      echo "created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT

  - name: Build and push image (slim)
    if: ${{ inputs.slim }}
    uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
    with:
      context: .
      file: ./build/docker/Dockerfile${{ inputs.debian && '.debian' || '' }}
      push: ${{ inputs.push }}
      tags: ${{ steps.slim.outputs.tags }}
      target: slim
      platforms: ${{ inputs.platform }}
      labels: |
        org.opencontainers.image.source=${{ github.event.repository.html_url }}
        org.opencontainers.image.created=${{ steps.slim.outputs.created }}
        org.opencontainers.image.revision=${{ github.sha }}
      build-args: |
        BUILD_VERSION=${{ inputs.crowdsec_version }}

  - name: Build and push image (full)
    uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
    with:
      context: .
      file: ./build/docker/Dockerfile${{ inputs.debian && '.debian' || '' }}
      push: ${{ inputs.push }}
      tags: ${{ steps.full.outputs.tags }}
      target: full
      platforms: ${{ inputs.platform }}
      labels: |
        org.opencontainers.image.source=${{ github.event.repository.html_url }}
        org.opencontainers.image.created=${{ steps.full.outputs.created }}
        org.opencontainers.image.revision=${{ github.sha }}
      build-args: |
        BUILD_VERSION=${{ inputs.crowdsec_version }}

File: .github/workflows/publish-tarball-release.yml Copy Chapter

name: Release

on:
release:
types:
- prereleased

permissions:

Use write for: gh release upload

contents: write

jobs:
build:
name: Build and upload binary package
runs-on: ubuntu-latest
steps:

- name: Check out code into the Go module directory
    uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
    with:
      fetch-depth: 0
      submodules: false
      persist-credentials: false

  - name: "Set up Go"
    uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
    with:
      go-version-file: go.mod

  - name: Build the binaries
    run: |
      sudo apt -qq -y -o=Dpkg::Use-Pty=0 install build-essential libre2-dev
      make vendor release BUILD_STATIC=1

  - name: Upload to release
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    run: |
      tag_name="${GITHUB_REF##*/}"
      gh release upload "$tag_name" crowdsec-release.tgz vendor.tgz *-vendor.tar.xz

File: .github/workflows/update_docker_hub_doc.yml Copy Chapter

name: (push-master) Update Docker Hub README

on:
push:
branches:
- master
paths:
- 'build/docker/README.md'

permissions:
contents: read

jobs:
update-docker-hub-readme:
runs-on: ubuntu-latest
steps:

-
    name: Check out the repo
    uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
    if: ${{ github.repository_owner == 'crowdsecurity' }}
    with:
      persist-credentials: false

  -
    name: Update docker hub README
    uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
    if: ${{ github.repository_owner == 'crowdsecurity' }}
    with:
      username: ${{ secrets.DOCKER_USERNAME }}
      password: ${{ secrets.DOCKER_PASSWORD }}
      short-description: "Crowdsec - An open-source, lightweight agent to detect and respond to bad behaviours."
      readme-filepath: "./build/docker/README.md"

File: .github/workflows/version.yml Copy Chapter

name: Deploy to Cloudflare Pages (Direct Upload)

on:
release:
types: [published]
workflow_dispatch:
inputs:
version:
required: false
type: string
date:
required: false
type: string

permissions:
contents: read

jobs:
deploy:
runs-on: ubuntu-latest
if: |
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'release' && github.event.release.prerelease == false)
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
    with:
      node-version: 20

  - name: Pick CrowdSec version
    id: pick_version
    env:
      INPUT_VERSION: ${{ inputs.version }}
      INPUT_DATE: ${{ inputs.date }}
      GH_TOKEN: ${{ github.token }}
    run: |
      set -euo pipefail

      if [ -n "${INPUT_VERSION:-}" ] && [ -z "${INPUT_DATE:-}" ]; then
        echo "Error: If version is provided, date must also be provided." >&2
        exit 1
      fi

      if [ -n "${INPUT_VERSION:-}" ]; then
        TAG="${INPUT_VERSION}"
        DATE="${INPUT_DATE}"
      else
        # latest non-prerelease tag
        JSON_CONTENT="$(gh release list --json tagName,publishedAt --exclude-pre-releases --limit 1)"
        TAG="$(echo "$JSON_CONTENT" | jq -r '.[0].tagName')"
        DATE="$(echo "$JSON_CONTENT" | jq -r '.[0].publishedAt')"
      fi

      echo "tag=$TAG" >> "$GITHUB_OUTPUT"
      echo "date=$DATE" >> "$GITHUB_OUTPUT"

  - name: Write dist artifact
    env:
      TAG: ${{ steps.pick_version.outputs.tag }}
      DATE: ${{ steps.pick_version.outputs.date }}
    run: |
      set -euo pipefail
      mkdir -p ./dist
      printf '{"name":"%s", "tag_name":"%s", "published_at":"%s"}\n' "$TAG" "$TAG" "$DATE" > ./dist/latest
      cat ./dist/latest

  # Deploy the build output directory (adjust ./dist to your framework output)
  - name: Publish to Cloudflare Pages
    uses: cloudflare/wrangler-action@953926a2e2182532811c01a25e53647d93bf07c0 # v4.1.3
    with:
      apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
      accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
      command: pages deploy ./dist --project-name version --branch main

================================================ SYMLINK: .github/workflows/.yamllint -> .yamllint

2. Official Technical Reference & Guides (crowdsecurity/crowdsec-docs) Copy Chapter

File: README.md Copy Chapter

Crowdsec documentation Copy Chapter

This repository holds the documentation for the crowdsec project.

Online version of this documentation is available here: https://doc.crowdsec.net/

The documentation is built using Docusaurus, a modern static website generator.

Pre-requisites Copy Chapter

• nvm (Node Version Manager)
• Node.js stable (install via nvm: nvm install stable && nvm use stable)
• VSCode with recommended extensions installed (see .vscode/extensions.json)

Installation Copy Chapter

nvm use stable
cd crowdsec-docs
npm install

Local Development Copy Chapter

nvm use stable
cd crowdsec-docs
npm start
# or equivalently
npx docusaurus start

This command starts a local development server at http://localhost:3000. Most changes are reflected live without having to restart the server.

Build Copy Chapter

npm run build

This command generates static content into the build directory and can be served using any static contents hosting service.
you can also use the npm run serve command to test the build locally.

Linting and Formatting Copy Chapter

This project uses Biomes for linting and formatting. You can run the following commands:

npm run lint
npm run format

File: amplify.yml Copy Chapter

version: 1
frontend:
phases:
build:
commands:
- nvm use 20
- cd crowdsec-docs
- npm install
- npm run build
artifacts:
baseDirectory: crowdsec-docs/build
files:
- "/*"
cache:
paths:
- crowdsec-docs/node_modules//*

File: biome.json Copy Chapter

{
"root": true,
"$schema": "https://biomejs.dev/schemas/2.3.2/schema.json",
"vcs": { "enabled": false, "clientKind": "git", "useIgnoreFile": false },
"files": {
"ignoreUnknown": false,
"includes": [
"crowdsec-docs//*.ts",
"crowdsec-docs//.tsx",
"crowdsec-docs/**/.css",
"!/node_modules",
"!crowdsec-docs/plugins",
"!crowdsec-docs/build",
"!/.docusaurus"
]
},
"formatter": {
"enabled": true,
"formatWithErrors": false,
"indentStyle": "tab",
"indentWidth": 4,
"lineEnding": "lf",
"lineWidth": 140,
"attributePosition": "auto",
"bracketSameLine": false,
"bracketSpacing": true,
"expand": "auto",
"useEditorconfig": true
},
"linter": {
"enabled": true,
"rules": {
"recommended": true,
"style": {
"useImportType": "off"
},
"complexity": {
"noImportantStyles": "off"
},
"suspicious": {
"noUnknownAtRules": "off"
}
}
},
"javascript": {
"formatter": {
"jsxQuoteStyle": "double",
"quoteProperties": "asNeeded",
"trailingCommas": "es5",
"semicolons": "always",
"arrowParentheses": "always",
"bracketSameLine": false,
"quoteStyle": "double",
"attributePosition": "auto",
"bracketSpacing": true
}
},
"html": { "formatter": { "selfCloseVoidElements": "always" } },
"css": {
"parser": {
"tailwindDirectives": true
},
"formatter": {
"enabled": true,
"indentStyle": "tab",
"indentWidth": 4,
"lineEnding": "lf",
"lineWidth": 140
}
},
"assist": {
"enabled": true,
"actions": { "source": { "organizeImports": "on" } }
}
}

File: customHttp.yml Copy Chapter

Markdown copies exist for LLM tools (llms.txt); keep them out of search indexes so they don't compete with the HTML pages.

Headers defined here replace any custom headers configured in the Amplify console.

customHeaders:

• pattern: "**/*.md"
  headers:
  • key: X-Robots-Tag
    value: noindex
• pattern: /llms.txt
  headers:
  • key: X-Robots-Tag
    value: noindex
• pattern: /llms-full.txt
  headers:
  • key: X-Robots-Tag
    value: noindex

File: crowdsec-docs/components.json Copy Chapter

{
"$schema": "https://ui.shadcn.com/schema.json",
"tailwind": {
"config": "tailwind.config.js",
"css": "src/css/custom.css",
"baseColor": "gray",
"prefix": "",
"cssVariables": true
},
"rsc": true,
"aliases": {
"utils": "@site/src/utils",
"components": "@site/src/components",
"ui": "@site/src/ui",
"hooks": "@site/src/hooks",
"lib": "@site/src/lib"
},
"style": "default",
"iconLibrary": "lucide"
}

File: crowdsec-docs/package.json Copy Chapter

{
"name": "crowdsec-docs",
"version": "0.0.0",
"private": true,
"scripts": {
"docusaurus": "docusaurus",
"start": "docusaurus start",
"build": "docusaurus build",
"swizzle": "docusaurus swizzle",
"deploy": "docusaurus deploy",
"clear": "docusaurus clear",
"serve": "docusaurus serve",
"lint": "biome check",
"typecheck": "tsc",
"format": "biome format",
"migrate": "biome migrate --write",
"write-translations": "docusaurus write-translations",
"write-heading-ids": "docusaurus write-heading-ids"
},
"dependencies": {
"@coreui/icons": "^3.0.1",
"@coreui/icons-react": "2.3.0",
"@docusaurus/core": "^3.9.2",
"@docusaurus/faster": "^3.9.2",
"@docusaurus/plugin-client-redirects": "^3.9.2",
"@docusaurus/preset-classic": "^3.9.2",
"@docusaurus/theme-common": "^3.9.2",
"@docusaurus/theme-mermaid": "^3.9.2",
"@docusaurus/theme-search-algolia": "^3.9.2",
"@emotion/react": "^11.14.0",
"@emotion/styled": "^11.14.0",
"@mdx-js/react": "^3.1.1",
"@mui/icons-material": "^7.3.4",
"@mui/material": "^7.3.4",
"@mui/x-date-pickers": "^8.16.0",
"@radix-ui/react-popover": "^1.1.15",
"@radix-ui/react-slot": "^1.2.3",
"@radix-ui/react-tooltip": "^1.2.8",
"@signalwire/docusaurus-plugin-llms-txt": "^1.2.2",
"animate.css": "^4.1.1",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"docusaurus-plugin-zooming": "^1.0.0",
"lucide-react": "^0.552.0",
"material-react-table": "^3.2.1",
"prism-react-renderer": "^2.4.1",
"react": "^19.2.1",
"react-dom": "^19.2.1",
"swagger-ui-react": "^5.30.3",
"tailwind-merge": "^3.3.1",
"tailwindcss-animate": "^1.0.7"
},
"devDependencies": {
"@biomejs/biome": "^2.3.2",
"@docusaurus/module-type-aliases": "^3.9.2",
"@docusaurus/tsconfig": "^3.9.2",
"@docusaurus/types": "^3.9.2",
"autoprefixer": "^10.4.21",
"postcss": "^8.5.1",
"tailwindcss": "^3.4.18",
"typescript": "^5.9.3"
},
"resolutions": {
"@types/react": "^19.2.0"
},
"browserslist": {
"production": [
">0.5%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}

File: crowdsec-docs/tsconfig.json Copy Chapter

{
"extends": "@docusaurus/tsconfig",
"compilerOptions": {
"baseUrl": "."
},
"exclude": [".docusaurus", "build"]
}

File: crowdsec-docs/versions.json Copy Chapter

[
"v1.8",
"v1.7",
"v1.6"
]

File: crowdsec-docs/blog/2024-06-11-cscli_dashboard_deprecation.md Copy Chapter

slug: cscli_dashboard_deprecation title: Cscli dashboard deprecation authors: crowdsec tags: [dashboard, Metabase] Copy Chapter

This blog article will outline the steps being taken to deprecate the cscli dashboard command out of the cscli binary. The cscli dashboard command was used to setup a containerized Metabase dashboard.

Why deprecate `cscli dashboard` ? Copy Chapter

There were multiple factors which led to the decision to deprecate the cscli dashboard command:

The cscli dashboard command was initially offered a way to visualize the data collected by your local CrowdSec instance. However, we found the tools to be too limited and not user-friendly enough. So we started working on the CrowdSec Console which is a more powerful and user-friendly tool to visualize your data.

Some users using cscli dashboard command did not understand what was happening "under the hood" and were not proactively maintaining or securing their dashboards. This led to recent CVE's not being patched in a timely manner since the version was tied to the cscli binary and users were publicly exposing their dashboards.

What will happen to the `cscli dashboard` command ? Copy Chapter

The cscli dashboard command will be deprecated starting from version 1.7.0 of the CrowdSec package. Moving forward, users will have the following alternatives for viewing metrics and insights:

• CrowdSec Console (officially supported)
• Prometheus with Grafana
• VictoriaMetrics with Grafana

Please note that only the CrowdSec Console is officially supported by the CrowdSec team. The other options are provided for flexibility but are community-driven and require self-management.

What should I do now? Copy Chapter

Currently we recommend that you firstly check which version of Metabase that is currently deployed on your system to ensure that you are not vulnerable to any CVE's. You can do this by running the following command:

docker ps -a

This will show you the running containers on your system. Look for the Metabase container and check its version. If you are running a version of Metabase that is vulnerable to a CVE, we recommend updating to the latest version of CrowdSec and re-running the cscli dashboard command to refresh the Metabase dashboard.

If you are already on version 1.7.0, you will need to manually remove the container, as the cscli dashboard command is no longer available.

File: crowdsec-docs/blog/authors.yml Copy Chapter

sbz:
name: Sofian Brabez
url: https://github.com/sbz

crowdsec:
name: Crowdsec Team
url: https://github.com/crowdsecurity
image_url: /img/crowdsec_logo.png

File: crowdsec-docs/docs/concepts.md Copy Chapter

id: concepts title: Concepts sidebar_position: 1 Copy Chapter

Global overview Copy Chapter

This page defines the core CrowdSec concepts and how the components interact. It focuses on how detection is done, where data is stored, and where remediation is enforced.

Security Engine Copy Chapter

The Security Engine is the CrowdSec software you install (an IDS: Intrusion Detection System).
It detects threats and produces alerts. The Local API can turn alerts into decisions using profiles.
Threat prevention (blocking) is enforced by Remediation Components (bouncers).
Internally, the Security Engine is made of two main components: the Log Processor (detection) and the Local API (storage and decision distribution).

A Security Engine can run standalone (Log Processor + Local API on the same host) or in a distributed setup (multiple Log Processors sending alerts to a shared Local API). This lets you adapt CrowdSec to your infrastructure and constraints.

In a distributed setup, detection content (collections/parsers/scenarios) runs on the Log Processor machines, while the Local API focuses on storing data and serving decisions. See the multi-server setup guide for a concrete deployment pattern.

Log Processor (LP) Copy Section

The Log Processor is the part of the Security Engine in charge of detecting malicious behavior based on your logs and HTTP traffic.

The Log Processor (abbreviated as LP) detects malicious behavior in two main ways:

• It acquires logs, parses and enriches events, then matches them against scenarios.
• It receives HTTP requests and matches them against AppSec rules.

When a scenario or an AppSec rule is triggered, the Log Processor sends an alert to the LAPI.

Related documentation:

• Installation and updates of detection content: Collections, Hub management
• Add context to alerts (visible in cscli/Console): Alert context
• AppSec request inspection and compatible integrations: AppSec

Local API (LAPI) Copy Section

The Local API is the part of the Security Engine that stores alerts/decisions and acts as the middleman between Log Processors, Remediation Components, and the Central API.

The Local API (abbreviated as LAPI) has several roles:

• Receive alerts from Log Processors and (optionally) create decisions based on configured profiles.
• Expose decisions to Remediation Components so they can enforce them.
• Interact with the Central API to share signals and receive community blocklists.

For implementation details, see:

• Local API configuration for distributed setups: Local API configuration
• How bouncers consume decisions: Bouncers API
• Authentication methods: Local API authentication
• Notifications: Notification plugins
• Storage: Databases
• Controlling exemptions: AllowLists

Glossary Copy Chapter

Quick definitions of terms used throughout the documentation and in tools like cscli:

• Collections: bundles of parsers, scenarios, and other items installed together. See Collections and Hub management.
• Scenarios: behavior detections evaluated by the Log Processor. See Scenarios.
• AppSec rules: WAF rules evaluated by the AppSec component. See AppSec.
• Alerts: records created when a scenario/AppSec rule triggers; stored in the Local API. See cscli alerts.
• Decisions: remediation instructions (for example ban, sometimes other types depending on your setup) created by Local API profiles or manually via cscli; consumed by bouncers. See cscli decisions and Bouncers API.

Remediation Components (Bouncers) Copy Chapter

The Remediation Components (also called Bouncers) are external components in charge of enforcing decisions.

Remediation Components rely on the Local API to receive decisions about malicious IPs to be blocked (or other remediation types such as CAPTCHA, supported by some bouncers).
Note that they also support CrowdSec's Blocklist as a Service.

Those decisions can come from detections made by the LP or from blocklists.

Remediation Components leverage existing parts of your infrastructure to block malicious IPs where it matters most (firewall, reverse proxy, web server, ...). You can find them on our Remediation Components Hub.

Central API (CAPI) Copy Chapter

The Central API (CAPI) serves as the gateway for network participants to connect and communicate with CrowdSec's network.

The Central API (abbreviated as CAPI) receives attack signals from participating Security Engines and signal partners. It then redistributes curated community decisions (the Community Blocklist).

The Central API is also at the heart of CrowdSec centralized blocklist services.

For details about what data is sent (and when), see the Central API introduction.
If you want to disable sharing to the Central API, see how to disable the Central API.

Console Copy Chapter

The CrowdSec Console is a web-based interface for reporting, alerting, and management across your CrowdSec products (from your fleet of Security Engines to CTI-related actions).

The Console allows you to:

• Manage alerts from your security stack.
• Manage decisions in real time.
• View and use blocklists and integrations.
• Manage your API keys (CTI API, Service API).

To connect an instance to the Console, see Console enrollment and the cscli command reference: cscli console enroll.

Example: from a log line to a block Copy Chapter

This is a typical flow for a log-based scenario (for example, SSH brute-force):

0. Install detection content: you typically install a collection from the Hub (parsers + scenarios) using cscli (see Hub management).
1. Acquire: the Log Processor reads your service logs via an acquisition configuration (for example, a file tail on /var/log/auth.log).
2. Parse + enrich: parsers extract fields (source IP, service, status, ...) and enrichers add context (GeoIP/ASN, ...).
3. Detect: a scenario correlates events over time (for example, many failed logins from the same IP) and triggers an alert.
4. Store + decide: the Log Processor sends the alert to the LAPI. The LAPI applies your profiles to create a decision (for example, ban for a given duration).
5. Enforce: a Remediation Component (bouncer) pulls decisions from the LAPI and enforces them where it matters (firewall, reverse proxy, web server, ...).

File: crowdsec-docs/docs/contact_team.md Copy Chapter

id: contact_team title: Contact the team sidebar_position: 10 Copy Chapter

If you want to contact us using non-public media, you can contact us on support AT crowdsec DOT net with the following gpg-key :

-----BEGIN PGP PUBLIC KEY BLOCK-----

mDMEaq0NzRYJKwYBBAHaRw8BAQdAcCMFX9xPVyIdLrgttvro55FYqJHQo2K2TX60
/NUl7zW0J0Nyb3dkU2VjIHN1cHBvcnQgPHN1cHBvcnRAY3Jvd2RzZWMubmV0PoiW
BBMWCgA+AhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAFiEEPgZxlkMs0krP8A0U
Cd1SadZuRsMFAmqtDroFCQeEzu0ACgkQCd1SadZuRsMPoAEAl50Xc84qjq+zQOls
q3fqcEbuS4sGa4NRC3yaFV432mIBALrwhFjATB4bIzeeyo+eKsuYYFR6JiFE87Ix
DqwfT78OuDgEaq0NzRIKKwYBBAGXVQEFAQEHQPoVA0kxEtye+tIf0J7oO3AiKsh8
6rqp+IazF4OjoclGAwEIB4h+BBgWCgAmAhsMFiEEPgZxlkMs0krP8A0UCd1SadZu
RsMFAmqtDtsFCQeEzw4ACgkQCd1SadZuRsNtiAD/YJqMIUA2qgVgoLFuxWjuqwdR
Tt/TyfWZxiLwOIjtE8QA/RcuILMZj5IXc+PlCFt5ebKrlDpQwgbPYjqTna0iFMEA
=EtLP
-----END PGP PUBLIC KEY BLOCK-----

File: crowdsec-docs/docs/intro.mdx Copy Chapter

title: Introduction id: intro Copy Chapter

import useBaseUrl from "@docusaurus/useBaseUrl"
import AcademyPromo from "@site/src/components/academy-promo"

Security Engine Overview

The CrowdSec Security Engine is an open-source, lightweight security engine that detects and blocks malicious actors. It analyzes logs and HTTP requests using behavior-based patterns called scenarios.

CrowdSec is modular: it provides behavior-based detection, including AppSec rules, and optional Remediation Components that enforce blocks.

CrowdSec is crowdsourced: when you participate, you share the attacks you detect and block. In return, the Security Engine automatically downloads a curated list of validated attackers (the community blocklist), so you can take action sooner against known threats.

Main Features Copy Chapter

In addition to the core "detect and react" mechanism, CrowdSec is committed to several other key aspects:

• Easy Installation: Get started quickly on all supported platforms.
• Simplified Daily Operations: Manage and maintain your setup from the CrowdSec Console (Web UI) or with the cscli command-line tool.
• Reproducibility: Analyze live logs and cold logs to validate detections, run forensic analysis, or generate reports.
• Versatile: Protect your perimeter by analyzing system logs and HTTP requests.
• Observability: Providing valuable insights into the system's activity:
  • View and manage alerts in the Console.
  • Expose detailed Prometheus metrics.
  • Use the cscli CLI for administration.
• API-Centric: All components communicate via an HTTP API, facilitating multi-machine setups.

Architecture Copy Chapter

Under the hood, the Security Engine has various components:

• The Log Processor handles detection. It analyzes logs from various data sources and HTTP requests from compatible web servers.
• The Appsec feature is part of the Log Processor. It filters HTTP requests from compatible web servers.
• The Local API acts as a middleman:
  • Between the Log Processors and the Remediation Components which are in charge of enforcing decisions.
  • And with the Central API to share alerts and receive blocklists.
• The Remediation Components (also called bouncers) block malicious IPs at your chosen level: IpTables, firewalls, web servers, or reverse proxies. See the full list on the CrowdSec Hub.

Deployment options Copy Chapter

This architecture supports simple standalone setups and more distributed deployments:

• Single machine: Follow the getting started guide.
• Multiple machines: Use the distributed setup guide.
• Centralized logs (rsyslog, Loki, ...): Run CrowdSec next to your log pipeline, not on production workloads.
• Kubernetes: See our Helm chart.
• Containers: Use the Docker data source.
• WAF only: Start with the AppSec introduction.

Distributed architecture example:

File: crowdsec-docs/docs/appsec/advanced_deployments.mdx Copy Chapter

id: advanced_deployments title: WAF Deployment Strategies sidebar_position: 6 Copy Chapter

WAF Deployment Strategies

This guide covers advanced CrowdSec WAF deployment strategies for organizations looking to gradually enhance their web application security posture. Learn how to progressively improve your WAF configuration from basic virtual patching to comprehensive multi-layer protection.

:::info Prerequisites
This guide assumes you have completed the General Setup and have a functional basic WAF deployment.
:::

You can read more about the CrowdSec WAF CRS support in the dedicated documentation

Security Enhancement Path Copy Chapter

CrowdSec WAF supports multiple deployment strategies that can be implemented progressively:

1. Basic Virtual Patching (Quickstart) Copy Section

Current State: Blocking protection against known CVEs

• Collections: crowdsecurity/appsec-virtual-patching
• Mode: In-band (blocking)
• Coverage: Known vulnerabilities only
• False Positives: Minimal

2. Enhanced Detection (Out-of-band CRS) Copy Section

Next Step: Add comprehensive attack detection without performance impact

• Add: crowdsecurity/appsec-crs (out-of-band) alongside existing virtual patching
• Mode: Non-blocking analysis + behavioral banning
• Coverage: OWASP Top 10 + comprehensive attack patterns + specific CVE protection
• Performance: No latency impact ⚡
• Security: Layered approach - virtual patching + generic attack detection

3. Maximum Protection (In-band CRS) Copy Section

Advanced: Full blocking protection with comprehensive coverage

• Modify: Use crowdsecurity/appsec-crs-inband for blocking CRS while keeping virtual patching
• Mode: Immediate blocking of all detected attacks (both generic and CVE-specific)
• Coverage: Maximum protection with instant response 🛡️
• Security: Dual-layer blocking - virtual patching handles specific vulnerabilities, CRS covers generic attack patterns
• Consideration: Might require tuning to minimize false positives

You will most likely also need to configure the CRS themselves to tune out potential false positives.
You can do so by using either official or custom CRS plugins. Refer to this documentation for a general plugin overview and this page for some examples.

Implementation Guide Copy Chapter

Adding Out-of-band CRS Copy Section

Enhance your existing virtual patching deployment by adding comprehensive attack detection as an additional security layer:

sudo cscli collections install crowdsecurity/appsec-crs

The crowdsecurity/appsec-crs collection includes:

• crowdsecurity/crs: AppSec config that loads CRS rules in out-of-band mode
• crowdsecurity/crowdsec-appsec-outofband: Scenario that bans IPs after 5+ out-of-band rule violations

Update your WAF acquisition configuration to include both rule sets:

appsec_configs:
  - crowdsecurity/appsec-default # Virtual patching rules (in-band blocking)
  - crowdsecurity/crs # OWASP CRS rules (out-of-band detection)
labels:
  type: appsec
listen_addr: 127.0.0.1:7422
source: appsec
name: myAppSecComponent

sudo systemctl restart crowdsec

Benefits of this layered configuration:

• Layer 1: Immediate protection against known vulnerabilities (virtual patching)
• Layer 2: Comprehensive attack pattern detection (CRS out-of-band)
• Complementary Coverage: Virtual patching rules catch specific CVEs that CRS generic rules might miss
• Behavioral analysis and repeat offender banning
• No performance impact on legitimate traffic

How to Test Out-of-band CRS Detection

After implementing the layered configuration, verify both protection layers are working correctly:

Test 1: Virtual Patching Layer (Immediate Blocking)

Test that virtual patching rules block requests immediately by trying to access sensitive files:

# Test .env file access (common vulnerability)
curl -v "http://your-app.com/.env"
curl -v "http://your-app.com/api/../.env"

Expected result: These requests should be immediately blocked with HTTP 403 Forbidden.

Test 2: CRS Out-of-band Detection Layer

The crowdsecurity/crs collection brings general detection for OWASP top10 attacks, which can be tested:

# Replace with your application URL
TARGET="http://your-app.com"

# SQL injection attempts (trigger multiple CRS rules)
curl "$TARGET/?id=1'+OR+'1'='1"

Expected results:

• Detailed Alert is created (see cscli alerts list)

  cscli alerts list output

+-------+--------------------+--------------------------------------------------------------+---------+------------------------------+-----------+----------------------+
|   ID  |        value       |                            reason                            | country |              as              | decisions |      created_at      |
+-------+--------------------+--------------------------------------------------------------+---------+------------------------------+-----------+----------------------+
| 62419 | Ip:xxx.xx.xx.xx    | anomaly score out-of-band: sql_injection: 10, anomaly: 10,   | FR      | 5410 Bouygues Telecom SA     |           | 2025-09-09T14:41:07Z |
...

• Detailed Alert is visible in console

  Alert Console view

Test 3: Verify Scenario Behavior

The crowdsecurity/crowdsec-appsec-outofband scenario will ban IPs triggering the CRS on more than 5 distinct requests on a short period, which can be tested:

for i in {1..6}; do curl "$TARGET/?id=1'+OR+'1'='1"; done

Expected results:

• Alerts are created for each request

• Decision is created by the 6th request

  cscli alerts list output

───────┬────────────────────┬──────────────────────────────────────────────────────────────┬─────────┬──────────────────────────────┬───────────┬──────────────────────╮
│   ID  │        value       │                            reason                            │ country │              as              │ decisions │      created_at      │
├───────┼────────────────────┼──────────────────────────────────────────────────────────────┼─────────┼──────────────────────────────┼───────────┼──────────────────────┤
│ 62427 │ Ip:xxx.xx.xx.xx    │ crowdsecurity/crowdsec-appsec-outofband                      │ FR      │ 5410 Bouygues Telecom SA     │ ban:1     │ 2025-09-09T14:51:11Z │
│ 62426 │ Ip:xxx.xx.xx.xx    │ anomaly score out-of-band: sql_injection: 10, anomaly: 10,   │ FR      │ 5410 Bouygues Telecom SA     │           │ 2025-09-09T14:51:12Z │
│ 62425 │ Ip:xxx.xx.xx.xx    │ anomaly score out-of-band: sql_injection: 10, anomaly: 10,   │ FR      │ 5410 Bouygues Telecom SA     │           │ 2025-09-09T14:51:11Z │
│ 62424 │ Ip:xxx.xx.xx.xx    │ anomaly score out-of-band: sql_injection: 10, anomaly: 10,   │ FR      │ 5410 Bouygues Telecom SA     │           │ 2025-09-09T14:51:11Z │
│ 62423 │ Ip:xxx.xx.xx.xx    │ anomaly score out-of-band: sql_injection: 10, anomaly: 10,   │ FR      │ 5410 Bouygues Telecom SA     │           │ 2025-09-09T14:51:11Z │
│ 62422 │ Ip:xxx.xx.xx.xx    │ anomaly score out-of-band: sql_injection: 10, anomaly: 10,   │ FR      │ 5410 Bouygues Telecom SA     │           │ 2025-09-09T14:51:11Z │
│ 62421 │ Ip:xxx.xx.xx.xx    │ anomaly score out-of-band: sql_injection: 10, anomaly: 10,   │ FR      │ 5410 Bouygues Telecom SA     │           │ 2025-09-09T14:51:11Z │

CRS In-band (Blocking Mode) Copy Section

For organizations requiring maximum protection, configure CRS rules to block requests immediately by installing the in-band CRS collection:

sudo cscli collections install crowdsecurity/appsec-crs-inband

:::info

In order to avoid potential false positives, you can also install a CRS exclusion rules plugin from the hub or write a custom one.

Refer to this documentation for a general plugin overview and this page for instructions on how to write a custom plugin.

:::

Update Acquisition Configuration

Modify your acquisition to use the in-band CRS configuration:

appsec_configs:
  - crowdsecurity/appsec-default # Virtual patching rules (in-band blocking)
  - crowdsecurity/crs-inband # OWASP CRS rules (in-band blocking)
labels:
  type: appsec
listen_addr: 127.0.0.1:7422
source: appsec
name: myAppSecComponent

sudo systemctl restart crowdsec

How to Test CRS In-band Blocking

After configuring CRS for in-band (blocking) mode, test that both virtual patching and CRS rules provide immediate blocking:

Test 1: Virtual Patching Layer (Still Blocking)

Verify virtual patching continues to work:

# These should still be immediately blocked
curl -v "http://your-app.com/.env"

Expected result: HTTP 403 Forbidden immediately.

Test 2: CRS In-band Blocking

Test that CRS rules now block requests immediately (no more out-of-band delay):

TARGET="http://your-app.com"

# SQL injection - should be blocked immediately
curl -v "$TARGET/?id=1' OR '1'='1"

# XSS - should be blocked immediately
curl -v "$TARGET/?q=<script>alert('xss')</script>"

# Command injection - should be blocked immediately
curl -v "$TARGET/?cmd=; cat /etc/passwd"

# Path traversal - should be blocked immediately
curl -v "$TARGET/?file=../../../etc/passwd"

Expected behavior:

• All requests above: Immediately blocked with HTTP 403 Forbidden
• No delay: Unlike out-of-band mode, blocking is instant
• Dual protection: Both virtual patching AND CRS rules provide immediate blocking

:::warning Important Considerations
In-band CRS blocking provides maximum protection but requires:

• Thorough testing in a staging environment
• Gradual rollout to production traffic
• Monitoring and tuning to prevent blocking legitimate requests
• Whitelisting capabilities for false positives
  :::

Next Steps Copy Chapter

Once you've implemented advanced deployments:

• Configure Custom Rules for application-specific protection
• Set up Hooks for custom response actions
• Explore Configuration Options for fine-tuning
• Review Troubleshooting Guide for operational issues

File: crowdsec-docs/docs/appsec/alerts_and_scenarios.md Copy Chapter

id: alerts_and_scenarios title: "WAF Alerts & Scenarios" sidebar_label: "Alerts & Scenarios" sidebar_position: 5 Copy Chapter

Generated Events Layout Copy Chapter

HTTP requests that trigger In-Band or Out-Of-Band AppSec/WAF rules generate events. These events can trigger scenarios that ban or alert when rules are matched.

The crowdsecurity/appsec-logs parser is designed as a general-purpose tool to convert events into a format that is easier to process with scenarios.

The generated event looks like this:

• evt.Meta.service is set to appsec
• evt.Meta.log_type:
  • appsec-block for blocked requests (In-Band rule matched, for example)
  • appsec-info for requests that triggered Out-Of-Band rule (not blocked)
• evt.Meta.source_ip is set to the source (client) IP
• evt.Meta.target_host is set to the FQDN if present (Host header in the HTTP request)
• evt.Meta.target_uri is set to the complete URI of the HTTP request
• evt.Meta.rule_name is set to the name of the triggered rule
• evt.Meta.remediation_cmpt_ip is set to the IP of the Remediation Component (bouncer) that sent the HTTP request.

:::info
The crowdsecurity/appsec-logs parser is already part of the generic AppSec/WAF collections and doesn't have to be manually installed.
:::

Creating Scenario Based on AppSec/WAF Events Copy Chapter

Triggering on _In-Band_ Rules Copy Section

A simple example is the crowdsecurity/appsec-vpatch scenario that will ban IPs triggering two distinct In-Band rules:

type: leaky
name: crowdsecurity/appsec-vpatch
filter: "evt.Meta.log_type == 'appsec-block'"
distinct: evt.Meta.rule_name
leakspeed: "60s"
capacity: 1
groupby: evt.Meta.source_ip
...

:::info
The crowdsecurity/appsec-vpatch scenario is already part of the generic AppSec/WAF collections, and doesn't have to be manually installed.
:::

Triggering on Out-Of-Band Rules Copy Section

Let's try to solve an imaginary scenario:

We aim to prevent users from enumerating certain URLs (specifically, those that begin with /foobar/*) when a particular HTTP header is present (something: *test*). However, we want to impose this restriction only on users attempting to access two or more distinct /foobar/* URLs while this header is set.

:::info
Keep in mind that Out-Of-Band rules generate an event instead of blocking the HTTP request.
:::

The AppSec/WAF Rule

This is our AppSec/WAF rule:

name: crowdsecurity/foobar-access
description: "Detect access to foobar files with the something header set"
rules:
  - and:
    - zones:
      - URI
      transform:
      - lowercase
      match:
        type: startsWith
        value: /foobar/
    - zones:
      - HEADERS
      variables:
      - something
      transform:
      - lowercase
      match:
        type: contains
        value: test

Let's ensure it's loaded as an Out-Of-Band rule by creating a new AppSec config:

name: crowdsecurity/appsec-oob
default_remediation: ban
#Let's add our rule as an out-of-band rule
outofband_rules:
 - crowdsecurity/foobar-access

And then make sure this appsec-config is loaded:

appsec_configs:
 - crowdsecurity/appsec-default
 - crowdsecurity/appsec-oob
labels:
  type: appsec
listen_addr: 127.0.0.1:7422
source: appsec

The Scenario

We can now create a scenario that will trigger when a single IP triggers this rule on distinct URLs:

type: leaky
format: 3.0
name: crowdsecurity/foobar-enum
description: "Ban IPs repeatedly triggering out of band rules"
filter: "evt.Meta.log_type == 'appsec-info' && evt.Meta.rule_name == 'crowdsecurity/foobar-access'"
distinct: evt.Meta.target_uri
leakspeed: "60s"
capacity: 1
groupby: evt.Meta.source_ip
blackhole: 1m
labels:
  remediation: true

:::info
The filter ensures only Out-Of-Band events generated by our scenario are picked up, while the capacity: 1 and distinct: evt.Meta.target_uri will ensure that the IP has to trigger the rule on at least 2 distinct URLs to trigger the scenario.
:::

Testing

Let's now test our setup:

$ curl -I localhost/foobar/1 -H 'something: test'
HTTP/1.1 404 Not Found

$ curl -I localhost/foobar/2 -H 'something: test'
HTTP/1.1 404 Not Found

$ curl -I localhost/foobar/3 -H 'something: test'
HTTP/1.1 403 Forbidden

And CrowdSec logs will show:

INFO[2024-12-02T15:28:16+01:00] Ip ::1 performed 'crowdsecurity/foobar-enum' (2 events over 4.780233613s) at 2024-12-02 14:28:16.858419797 +0000 UTC 
INFO[2024-12-02T15:28:17+01:00] (test/crowdsec) crowdsecurity/foobar-enum by ip ::1 (/0) : 4h ban on Ip ::1

As expected, the first two requests were processed without being blocked. The second one triggered the scenario, resulting in the third request being blocked by the bouncer.

File: crowdsec-docs/docs/appsec/api_validation.md Copy Chapter

id: api_validation title: OpenAPI Schema Validation sidebar_position: 5 Copy Chapter

The Application Security Component can validate incoming HTTP requests against an OpenAPI 3 schema you provide. Requests that do not conform to the schema (unknown route, unexpected method, missing or malformed parameters, invalid request body, missing/invalid authentication credentials, …) can be rejected before they ever reach the protected application.

This is a positive-security model layered on top of the negative-security model implemented by the WAF rules: instead of describing what an attacker looks like, you describe what a valid client looks like and reject everything else.

How it works Copy Chapter

Schema validation is exposed through the hooks system:

• An on_load hook loads one or more OpenAPI schemas at startup, each under a short string ref.
• A pre_eval hook calls ValidateRequestWithSchema(ref) to validate the current request. The function returns true when the request is valid, false otherwise.
• When validation fails, structured details about the failure are published to hook_vars so the same hook (or a later one) can build a meaningful drop reason, enrich an event, etc.

Storing schemas Copy Chapter

Schemas are loaded from the schemas/ subdirectory of the CrowdSec data_dir (typically /var/lib/crowdsec/data/schemas/).

Filenames passed to the loader must be relative to that directory.

/var/lib/crowdsec/data/schemas/
├── users-api.yaml
└── billing-api.yaml

OpenAPI 3.0 and Swagger schemas in YAML or JSON are both accepted.

Loading schemas (`on_load`) Copy Chapter

Loading is done from an on_load hook using one of two helpers:

ref is an arbitrary string you choose; you will use it later in pre_eval to refer to this schema. A schema name cannot be loaded twice.

name: custom/my-appsec-config
inband_rules:
  - crowdsecurity/base-config
on_load:
  - apply:
      - LoadAPISchemaWithName("users_api", "users-api.yaml")
      - LoadAPISchemaWithName("billing_api", "billing-api.yaml")

If the schema file is missing, malformed, or not a valid OpenAPI 3 document, the datasource will fail to start and log the underlying error.

Schema options Copy Section

LoadAPISchemaWithOptions accepts the following keys, all strings:

drop (the default) treats the unmatched route as a validation failure — ValidateRequestWithSchema returns false and the validation error is surfaced via hook_vars. ignore lets the request through the validator without inspection (the function returns true), which is useful when your schema only covers a subset of your API.

on_load:
  - apply:
      - >
        LoadAPISchemaWithOptions("public_api", "public-api.yaml", {
          "on_route_not_found": "ignore",
          "on_method_not_allowed": "drop",
        })

Body decoders Copy Section

The validator uses the request Content-Type to pick a decoder for the body. By default, only the following Content-Types are decoded:

• application/json and the JSON variants application/json-patch+json, application/merge-patch+json, application/ld+json, application/hal+json, application/vnd.api+json, application/problem+json
• application/x-www-form-urlencoded
• multipart/form-data

A request whose Content-Type is not in this list will fail validation if the matching operation in the schema declares a request body.

To enable validation of additional Content-Types, register a decoder from on_load:

on_load:
  - apply:
      - RegisterAPISchemaBodyDecoder("application/yaml", "yaml")
      - RegisterAPISchemaBodyDecoder("text/csv", "csv")

Available decoder names:

:::warning
Body decoders are registered process-wide. If you run several AppSec datasources in the same CrowdSec process, they share the same set of registered decoders.
:::

Validating requests (`pre_eval`) Copy Chapter

In a pre_eval hook, call ValidateRequestWithSchema(ref) with the ref you used at load time. It returns true if the request matches the schema, false otherwise.

A typical pattern is to fail closed — on validation failure, drop the request and use the failure details to build a human-readable reason:

name: custom/my-appsec-config
on_load:
  - apply:
      - LoadAPISchemaWithName("users_api", "users-api.yaml")
inband:
  pre_eval:
    - filter: req.URL.Path startsWith "/users" && !ValidateRequestWithSchema("users_api")
      apply:
        - |
          DropRequest("schema validation failed: " + hook_vars.validation_error_message)

You can also use the result to pick a softer remediation, send a custom event, etc.

Validation result variables Copy Section

When ValidateRequestWithSchema returns false, the following keys are set on hook_vars. They are available to the apply block of the same hook, to later hooks in the same request, and to on_match / post_eval hooks. The same keys are also propagated to the resulting CrowdSec event.

On success these keys are absent.

Authentication Copy Chapter

If your OpenAPI schema declares a security requirement on an operation, the validator enforces it as part of validation. Failure to satisfy the security requirement is reported as a security reason in hook_vars.

The validator only verifies that the credential is present and well-formed — it does not verify the credential against any backing store.

End-to-end example Copy Chapter

/var/lib/crowdsec/data/schemas/users-api.yaml:

openapi: 3.0.0
info:
  title: Users API
  version: "1.0.0"
paths:
  /users:
    post:
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [username, email]
              additionalProperties: false
              properties:
                username:
                  type: string
                  minLength: 3
                  maxLength: 20
                email:
                  type: string
                  format: email
      responses:
        "201":
          description: created

AppSec configuration:

name: custom/my-appsec-config
on_load:
  - apply:
      - LoadAPISchemaWithName("users_api", "users-api.yaml")
inband:
  pre_eval:
    - filter: req.URL.Path startsWith "/users" && !ValidateRequestWithSchema("users_api")
      apply:
        - |
          DropRequest("API schema violation on '" + hook_vars.validation_error_field + "': " + hook_vars.validation_error_message)

With this configuration:

• POST /users with {"username": "ab", "email": "x"} is dropped (username too short, email malformed).
• POST /users with a valid body passes validation and is then evaluated by the WAF rules as usual.
• GET /users is dropped with reason method_not_allowed (default policy).
• POST /admin is not validated by this hook because the filter only matches /users paths.

Metrics Copy Chapter

Two Prometheus counters are exposed:

reason values match validation_error_reason: parameter, request_body, security, route_not_found, method_not_allowed, internal.

File: crowdsec-docs/docs/appsec/benchmark.md Copy Chapter

id: benchmark title: WAF Component Benchmark toc_max_heading_level: 4 sidebar_position: 80 Copy Chapter

Basic Benchmark Copy Chapter

The Application Security Component benchmarks were run on an AWS EC2 instance t2.medium (2vCPU/4GiB RAM).

All benchmarks were run with a single routine configured for the Application Security Component.

The benchmarks cover the following tests:

• Basic GET request:
  • 5 concurrent connections / 1000 requests
  • 15 concurrent connections / 1000 requests

Each test was run with multiple cases:

• Application Security Component enabled but without any rules
• Application Security Component enabled with 100 vpatch rules (in-band)
• Application Security Component enabled with all the CRS (in-band)

On the system, we deployed:

• Openresty 1.21.4.3
• CrowdSec v1.6.0
• cs-openresty-bouncer v1.0.1

Basic GET request Copy Section

5 concurrent connections / 1000 requests

15 concurrent connections / 1000 requests

Stress Test Copy Chapter

This test was run on a c5a.4xlarge EC2 instance (16CPU/32GiB RAM).

Tested versions are:

• Openresty v1.27.1.2
• CrowdSec v1.7.0
• cs-openresty-bouncer v1.1.2

Openresty was configured to not log anything and forward requests to a Go backend that always returns 200, to improve raw throughput and avoid disk access limits.

CrowdSec WAF was configured with 16 routines to make use of as much CPU as possible.

All tests were simulating 400 concurrent users, making requests as quickly as possible during 1 minute.

Except for the baseline, all values in the tables are shown as a delta from the baseline performance.

Baseline Copy Section

This test was run without loading the Openresty bouncer to get a baseline throughput of the system.

GET Requests

10% POST Requests

Virtual Patching Rules Copy Section

GET Requests - 10% malicious - InBand

Realistic Traffic - 70% GET - 25% POST - 5% malicious - Inband

CRS Copy Section

GET Requests - 10% malicious - InBand

POST Requests - 10% malicious - InBand

Realistic Traffic - 70% GET - 25% POST - 5% malicious - Inband

Virtual Patching Inband + CRS Out-of-band Copy Section

Realistic Traffic - 70% GET - 25% POST - 5% malicious

File: crowdsec-docs/docs/appsec/configuration.md Copy Chapter

id: configuration title: "AppSec Configuration Syntax" sidebar_label: "Syntax" sidebar_position: 1 Copy Chapter

AppSec Configuration Files Copy Chapter

AppSec configuration files define which rules are loaded, how they run, and how the WAF responds.

They are loaded by the AppSec acquisition datasource via appsec_configs (see the AppSec datasource).

Below is a minimal example followed by the full key reference.

name: custom/my-appsec-config
inband_rules:
  - crowdsecurity/base-config
default_remediation: ban

Each AppSec configuration file controls how rules are loaded and processed.
You can create custom configuration files in /etc/crowdsec/appsec-configs/.

Configuration File Format Copy Chapter

Configuration files share a common structure:

• a name (required)
• optional rule lists such as inband_rules and outofband_rules
• optional behavior keys like default_remediation and default_pass_action
• HTTP response codes (for example, blocked_http_code)
• optional performance settings (inband_options, outofband_options)
• optional hooks, scoped per phase (inband, outofband), plus on_load
• legacy top-level hooks (pre_eval, post_eval, on_match)
• optional logging (log_level)

name: custom/my-appsec-config
inband_rules:
  - crowdsecurity/base-config
outofband_rules:
  - crowdsecurity/crs
default_remediation: ban
default_pass_action: allow
blocked_http_code: 403
passed_http_code: 200
log_level: info

Configuration Structure Copy Chapter

name Copy Section

string

Unique identifier for the AppSec configuration, used for logging and referencing.

name: custom/my-appsec-config

inband_rules Copy Section

array of strings

List of rule patterns to load as in-band rules. See in-band rule processing.

inband_rules:
  - crowdsecurity/base-config
  - crowdsecurity/vpatch-*

outofband_rules Copy Section

array of strings

List of rule patterns to load as out-of-band rules. See out-of-band rule processing.

outofband_rules:
  - crowdsecurity/crs
  - custom/detection-rules

default_remediation Copy Section

string

Default action for in-band rules that match. The special value allow disables blocking.

Common values include ban (block) and captcha (challenge), depending on what your remediation component supports.

:::info
When using multiple AppSec configs, the last declared one takes precedence for this property.
:::

default_remediation: ban

default_pass_action Copy Section

string

Action for requests that do not match any rules, or match rules with pass actions.

:::info
When using multiple AppSec configs, the last declared one takes precedence for this property.
:::

default_pass_action: allow

blocked_http_code Copy Section

integer

HTTP status code returned to the remediation component when a request is blocked.

blocked_http_code: 403

passed_http_code Copy Section

integer

HTTP status code returned to the remediation component when a request is allowed.

passed_http_code: 200

user_blocked_http_code Copy Section

integer

HTTP status code returned to the end user when a request is blocked.

user_blocked_http_code: 403

user_passed_http_code Copy Section

integer

HTTP status code returned to the end user when a request is allowed.

user_passed_http_code: 200

inband_options Copy Section

object

Performance tuning options for in-band rule processing.

• disable_body_inspection (bool): Skip HTTP body inspection.
• request_body_in_memory_limit (integer): Max body size in memory (bytes, default: 1048576).

inband_options:
  disable_body_inspection: false
  request_body_in_memory_limit: 1048576

:::note
request_body_in_memory_limit is a Coraza-level setting. It is distinct from the engine's overall maximum body size, which bounds how much of the body CrowdSec buffers before any rule runs (defaults to 10MB). See Request body size handling to tune it.
:::

outofband_options Copy Section

object

Performance tuning options for out-of-band rule processing.

• disable_body_inspection (bool): Skip HTTP body inspection.
• request_body_in_memory_limit (integer): Max body size in memory (bytes, default: 1048576).

outofband_options:
  disable_body_inspection: false
  request_body_in_memory_limit: 1048576

log_level Copy Section

string

Logging verbosity for this configuration. Available levels: debug, info, warn, error.

log_level: info

inband Copy Section

object

Rules, hooks and options scoped to the in-band pass. Accepted keys:

• rules (array of strings): same as inband_rules; both lists are merged.
• pre_eval, post_eval, on_match (arrays): hooks that run only during the in-band pass. See AppSec Hooks.
• on_challenge, on_challenge_submit (arrays): challenge hooks, which only exist in-band. See Bot detection hooks.
• options (object): same as inband_options.
• variables_tracking (array of strings): tracked variables. They are global, so scoping them to a phase has no effect.

inband:
  rules:
    - crowdsecurity/base-config
  pre_eval:
    - filter: req.RemoteAddr == "192.168.1.100"
      apply:
        - RemoveInBandRuleByName("strict-rule")
  options:
    disable_body_inspection: false

outofband Copy Section

object

Same keys as inband, applied to the out-of-band pass, except on_challenge and on_challenge_submit: challenges are in-band only.

on_load Copy Section

array

Executed when the configuration is loaded. Typically used for global rule changes. Not phase-scoped: it runs once at startup, so it has no inband / outofband equivalent.

on_load:
  - apply:
      - RemoveInBandRuleByName("problematic-rule")

pre_eval Copy Section

array

Executed before rule evaluation for each request. Declared at the top level, it runs in both passes, hence the IsInBand filter below. Use inband / outofband to target a single pass.

pre_eval:
  - filter: IsInBand && req.RemoteAddr == "192.168.1.100"
    apply:
      - RemoveInBandRuleByName("strict-rule")

post_eval Copy Section

array

Executed after rule evaluation. Useful for debugging and analysis. Runs in both passes, like pre_eval.

post_eval:
  - filter: IsInBand
    apply:
      - DumpRequest().WithBody().ToJSON()

on_match Copy Section

array

Executed when rules match. Used to adjust remediation or generate custom alerts. Runs in both passes, like pre_eval.

on_match:
  - filter: req.Host == "staging.example.com"
    apply:
      - SetRemediation("allow")
      - CancelAlert()

For complete hook documentation, see AppSec Hooks.

data Copy Section

array

Declares datafiles the appsec-config needs. When the config is installed from the hub, cscli downloads each source_url into dest_file (relative to the data directory). The bot-detection exclude-configs use this to ship the known-bot definitions consulted by MatchKnownBot, with type: bots.

data:
  - source_url: https://hub-data.crowdsec.net/whitelists/benign_bots/legit_bots/gptbot.json
    dest_file: legit_bots/gptbot.json
    type: bots

File: crowdsec-docs/docs/appsec/configuration_creation_testing.md Copy Chapter

id: configuration_creation_testing title: "AppSec Configuration Creation & Testing" sidebar_label: "Creation & Testing" sidebar_position: 3 Copy Chapter

AppSec Acquisition Copy Chapter

The acquisition file is used to:

• Specify the address and port where AppSec-enabled remediation components forward requests.
• Specify one or more AppSec configuration files that define which rules to apply and how.

Details can be found in the AppSec Datasource page.

Defining Multiple AppSec Configurations Copy Section

Often you will want multiple AppSec configurations to define groups of rules that are handled the same way.

Use the appsec_configs parameter to load multiple configurations that work together.

In the following example we have two configurations:

• One with CrowdSec default AppSec rules ↗️ running in in-band mode
• The other for the CRS rules ↗️ that run in out-of-band mode by default

appsec_configs:
  - crowdsecurity/appsec-default    # In-band virtual patching
  - crowdsecurity/crs               # Out-of-band detection based on ModSec CRS - from crowdsecurity/appsec-crs collection
labels:
  type: appsec
listen_addr: 127.0.0.1:7422
source: appsec

:::info
CrowdSec AppSec collections are available on CrowdSec Hub ↗️ and kept up to date.

For example the CRS collection: sudo cscli collections install crowdsecurity/appsec-crs.
This collection installs OWASP CRS in out-of-band and adds a scenario to ban IPs triggering multiple rules.
:::

Using Custom Configurations Copy Section

If you want to alter default configuration files, create a new configuration file instead of modifying hub configurations.
Modifying hub configurations will make them tainted and prevent automatic updates.

For example, if you want to change the default vpatch rules config, create your own and use it instead in the acquisition file.

appsec_configs:
  - crowdsecurity/appsec-default
  - custom/my_vpatch_rules
labels:
  type: appsec
listen_addr: 127.0.0.1:7422
source: appsec

A custom configuration file could look like this:

name: custom/my_vpatch_rules
default_remediation: ban
inband_rules:
  - custom/custom-vpatch-*
# Add custom hooks as needed

AppSec Configuration Files Copy Chapter

AppSec configuration files declare which rules to load in in-band (blocking) and/or out-of-band (non-blocking) mode, define how matches are handled (for example, default remediation), and let you tweak processing via hooks like on_load, pre_eval, post_eval, and on_match.

For the full list of keys, see Configuration Syntax.

:::info
When loading multiple AppSec configs, hooks and appsec rules are appended, and for conflicting options (for example, default_remediation), the last one takes precedence.
:::

Configuration Processing Order Copy Section

When multiple AppSec configurations are loaded, they are processed in the order specified in the appsec_configs list. For details on how in-band and out-of-band rules work, see the AppSec Introduction.

Multi-Config Rule Evaluation Copy Section

1. All inband_rules from all configurations are combined and evaluated together
2. All outofband_rules from all configurations are combined and evaluated together
3. Hooks from all configurations are executed in the order the configurations are listed
4. For conflicting configuration options (like default_remediation), the last configuration's value takes precedence

Testing Changes Copy Chapter

After updating AppSec configuration files:

1. Reload CrowdSec so it picks up the new configuration.
2. Validate behavior with your usual test traffic, or use the generic AppSec test rule.
3. Inspect results in logs or via cscli metrics show appsec.

For more troubleshooting guidance, see AppSec Troubleshooting.

File: crowdsec-docs/docs/appsec/configuration_rule_management.md Copy Chapter

id: configuration_rule_management title: Allowlisting and Rule Overrides sidebar_position: 3 Copy Chapter

Disabling Rules at Runtime Copy Chapter

You can disable rules at runtime, either globally (for all requests) or based on specific conditions (source IP, URI, ...).

You can disable rules by:

• Name with RemoveInBandRuleByName: For CrowdSec rules (name as seen in cscli appsec-rules list)
• ID with RemoveInBandRuleByID: For seclang/ModSecurity rules by numeric ID
• Tag with RemoveInBandRuleByTag: For seclang/ModSecurity rules by tag

The same functions exist for out-of-band rules, prefixed with RemoveOutBandRuleBy...

To disable a rule, create a new AppSec config to avoid tainting the configuration from the hub (or update your existing custom configuration).

name: custom/my_config
on_load:
 - apply:
    - RemoveInBandRuleByName("crowdsecurity/vpatch-env-access")
inband:
  pre_eval:
    - filter: req.URL.Path startsWith "/bar/"
      apply:
        - RemoveInBandRuleByName("crowdsecurity/generic-wordpress-uploads-php")

This example uses hooks to modify the configuration in 2 places:

• on_load: Expressions here will be applied when CrowdSec loads the configuration, effectively disabling the rule crowdsecurity/vpatch-env-access globally.
• inband.pre_eval: Expressions here will be applied only if the provided filter matches, and only during the in-band pass. In this example, we are disabling the rule crowdsecurity/generic-wordpress-uploads-php only if the request URI starts with /bar/.

You can also disable native (seclang) rules by providing their ID with the RemoveInBandRuleByID helper. See the hooks documentation for a list of available helpers.

Also note that we are not loading any rules in our custom config: the rules are loaded by the crowdsecurity/appsec-default config, and we are just modifying the runtime behavior with this config.

Finally, add your new config to the acquisition configuration:

appsec_configs:
  - crowdsecurity/appsec-default
  - custom/my_config
labels:
  type: appsec
listen_addr: 127.0.0.1:7422
source: appsec

Allowlisting Copy Chapter

Fully allow a specific IP or range Copy Section

If you want to ignore all rule matches for a specific IP or range, you can use a centralized allowlist.

The request bypasses AppSec entirely: no in-band or out-of-band rule is evaluated for that IP.

Disable specific rules for a specific IP/range Copy Section

If you want to disable rule(s) for a specific IP (or range), you will need to use the pre_eval hook (refer to the section above for more details):

name: custom/my_config
pre_eval:
 - filter: req.RemoteAddr == "1.2.3.4"
   apply:
    - RemoveInBandRuleByName("crowdsecurity/generic-wordpress-uploads-php")

Disable appsec for a specific FQDN Copy Section

If your reverse-proxy forwards all requests to CrowdSec regardless of the FQDN, you can disable AppSec for specific domains with a custom AppSec config (the request will always be allowed):

name: custom/my_config
on_match:
 - filter: req.Host == "foo.com"
   apply:
    - CancelEvent()
    - CancelAlert()
    - SetRemediation("allow")

With this config, the rules will still be evaluated, but if a rule matches no alert or event will be generated, and the remediation will be set to allow (i.e., instruct the bouncer to let the request through).

File: crowdsec-docs/docs/appsec/create_rules.md Copy Chapter

id: create_rules title: "Writing & Testing WAF Rules" sidebar_label: "Creation & Testing" sidebar_position: 3 Copy Chapter

Foreword Copy Chapter

This documentation assumes you're creating an AppSec rule to submit to the Hub, along with its functional tests. Building the test first guides the process and makes it easier.

We're going to create a rule for an imaginary vulnerability: a shell injection in the user_id parameter that can be triggered because the x-foobar-bypass header confuses the application.

The exploit looks like this:
curl localhost -H "x-foobar-bypass: 1" -d "user_id=123;cat /etc/password&do=yes"

And results in an HTTP request that looks like this:

POST / HTTP/1.1
...
x-foobar-bypass: 1
Content-Length: 36
Content-Type: application/x-www-form-urlencoded

user_id=123;cat /etc/password&do=yes

Prerequisites Copy Chapter

1. Create a local test environment or have CrowdSec (>= 1.6.0) installed locally
2. Have Docker installed locally to run the test web server
3. Have Nuclei installed (go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest)
4. Clone the hub

git clone https://github.com/crowdsecurity/hub.git

Create our test Copy Chapter

From the root of the hub repository:

▶ cscli hubtest create my-vuln --appsec

  Test name                   :  my-vuln
  Test path                   :  /home/bui/github/crowdsec/hub/.appsec-tests/my-vuln
  Config File                 :  /home/bui/github/crowdsec/hub/.appsec-tests/my-vuln/config.yaml
  Nuclei Template             :  /home/bui/github/crowdsec/hub/.appsec-tests/my-vuln/my-vuln.yaml

If you are creating a virtual-patching rule, please name your test CVE-YYYY-XYZ and your rule vpatch-CVE-YYYY-XYZ.

Configure our test Copy Chapter

Let's incorporate our AppSec rule into the configuration file.

.appsec-tests/my-vuln/config.yaml

appsec-rules:
  - ./appsec-rules/crowdsecurity/my-vuln.yaml
nuclei_template: my-vuln.yaml

Note: Since our custom AppSec rule has not been added to the Hub yet, we need to define its path relative to the root of the Hub directory.

The hubtest create command created a boilerplate Nuclei template that we can edit to add our HTTP request:

.appsec-tests/my-vuln/my-vuln.yaml

id: my-vuln
info:
  name: my-vuln
  author: crowdsec
  severity: info
  description: my-vuln testing
  tags: appsec-testing
http:
  #this is a dummy request, edit the request(s) to match your needs
  - raw:
      - |
        POST / HTTP/1.1
        Host: {{Hostname}}
        x-foobar-bypass: 1
        Content-Length: 36
        Content-Type: application/x-www-form-urlencoded

        user_id=123;cat /etc/password&do=yes
    cookie-reuse: true
    #test will fail because we won't match http status
    matchers:
      - type: status
        status:
          - 403

A few notes about the nuclei template:

• The template expects to get a 403 from the server. It's how hubtest can tell if the request was caught.
• To capture the HTTP request, running your exploit against a nc running locally can be useful:

curl localhost:4245 -H "x-foobar-bypass: 1" -d "user_id=123;cat /etc/password&do=yes"

▶ nc -lvp 4245
Listening on 0.0.0.0 4245
Connection received on localhost 50408
POST / HTTP/1.1
Host: localhost:4245
User-Agent: curl/7.68.0
Accept: */*
x-foobar-bypass: 1
Content-Length: 36
Content-Type: application/x-www-form-urlencoded

user_id=123;cat /etc/password&do=yes

Rule creation Copy Chapter

Let's now create our AppSec rule:

appsec-rules/crowdsecurity/my-vuln.yaml

name: crowdsecurity/my-vuln
description: "Imaginary RCE (CVE-YYYY-XYZ)"
rules:
  - and:
      - zones:
          - METHOD
        match:
          type: equals
          value: POST
      - zones:
          - HEADERS_NAMES
        transform:
          - lowercase
        match:
          type: equals
          value: x-foobar-bypass
      - zones:
          - BODY_ARGS
        variables:
          - user_id
        match:
          type: regex
          value: "[^a-zA-Z0-9_]"
labels:
  type: exploit
  service: http
  confidence: 3
  spoofable: 0
  behavior: "http:exploit"
  label: "Citrix RCE"
  classification:
    - cve.CVE-YYYY-XYZ
    - attack.T1595
    - attack.T1190
    - cwe.CWE-284

Let's get over the relevant parts:

• name is how the alert will appear to users (in cscli or the console)
• description is how your scenario will appear in the hub
• labels section is used both in the hub and the console. It must follow rules described here
• rules describe what we want to match:
  • a METHOD equal to POST
  • the presence of a header (HEADERS_NAME) with a name that once transformed to lowercase, is x-foobar-bypass
  • a post parameter (BODY_ARGS), user_id that contains something else than a-z A-Z or 0-9 or _

:::info
You can find detailed rules syntax here.
:::

Testing Copy Chapter