# Technical Documentation: getsentry/sentry
> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/getsentry/sentry)
### Overview (README.md)
Users and logs provide clues. Sentry provides answers.
# What's Sentry?
Sentry is the debugging platform that helps every developer detect, trace, and fix issues. Code breaks, fix it faster.
## Official Sentry SDKs
- [JavaScript](https://github.com/getsentry/sentry-javascript)
- [Electron](https://github.com/getsentry/sentry-electron/)
- [React-Native](https://github.com/getsentry/sentry-react-native)
- [Python](https://github.com/getsentry/sentry-python)
- [Ruby](https://github.com/getsentry/sentry-ruby)
- [PHP](https://github.com/getsentry/sentry-php)
- [Laravel](https://github.com/getsentry/sentry-laravel)
- [Go](https://github.com/getsentry/sentry-go)
- [Rust](https://github.com/getsentry/sentry-rust)
- [Java/Kotlin](https://github.com/getsentry/sentry-java)
- [Objective-C/Swift](https://github.com/getsentry/sentry-cocoa)
- [C\#/F\#](https://github.com/getsentry/sentry-dotnet)
- [C/C++](https://github.com/getsentry/sentry-native)
- [Dart/Flutter](https://github.com/getsentry/sentry-dart)
- [Perl](https://github.com/getsentry/perl-raven)
- [Clojure](https://github.com/getsentry/sentry-clj/)
- [Elixir](https://github.com/getsentry/sentry-elixir)
- [Unity](https://github.com/getsentry/sentry-unity)
- [Unreal Engine](https://github.com/getsentry/sentry-unreal)
- [Godot Engine](https://github.com/getsentry/sentry-godot)
- [PowerShell](https://github.com/getsentry/sentry-powershell)
# Resources
- [Documentation](https://docs.sentry.io/)
- [Discussions](https://github.com/getsentry/sentry/discussions) (Bugs, feature requests,
general questions)
- [Discord](https://discord.gg/PXa5Apfe7K)
- [Contributing](https://docs.sentry.io/internal/contributing/)
- [Bug Tracker](https://github.com/getsentry/sentry/issues)
- [Code](https://github.com/getsentry/sentry)
- [Transifex](https://explore.transifex.com/getsentry/sentry/) (Translate
Sentry\!)
---
### Src Sentry Flags Docs Api (`src/sentry/flags/docs/api.md`)
# Flags API
Host: https://sentry.io/api/0
**Authors.**
@cmanallen
**How to read this document.**
This document is structured by resource with each resource having actions that can be performed against it. Every action that either accepts a request or returns a response **must** document the full interchange format.
## Flag Logs [/organizations//flags/logs/]
- Parameters
- flag (optional, string) - The flag name to filter the result by. Can be specified multiple times.
- start (optional, string) - ISO 8601 format (`YYYY-MM-DDTHH:mm:ss.sssZ`)
- end (optional, string) - ISO 8601 format. Required if `start` is set.
- statsPeriod (optional, string) - A positive integer suffixed with a unit type.
- cursor (optional, string)
- per_page (optional, number)
Default: 10
- offset (optional, number)
Default: 0
### Browse Flag Logs [GET]
Retrieve a collection of flag logs.
**Attributes**
| Column | Type | Description |
| ------------- | ---------------- | ------------------------------------------------------------- |
| action | string | Enum of `created`, `updated`, or `deleted`. |
| createdAt | string | ISO-8601 timestamp of when the flag was changed. |
| createdBy | optional[string] | The user responsible for the change. |
| createdByType | optional[string] | Enum of `email`, `id`, or `name`. |
| flag | string | The name of the flag changed. Maps to flag_log_id in the URI. |
| id | number | A unique identifier for the log entry. |
| tags | object | A collection of provider-specified scoping metadata. |
- Response 200
```json
{
"data": [
{
"action": "created",
"createdAt": "2024-01-01T05:12:33",
"createdBy": "2552",
"createdByType": "id",
"flag": "my-flag-name",
"id": 1,
"tags": {
"environment": "production"
}
}
]
}
```
## Flag Log [/organizations//flags/logs//]
### Fetch Flag Log [GET]
Retrieve a single flag log instance.
- Response 200
```json
{
"data": {
"action": "updated",
"createdAt": "2024-11-19T19:12:55",
"createdBy": "user@site.com",
"createdByType": "email",
"flag": "new-flag-name",
"id": 1,
"tags": {
"environment": "development"
}
}
}
```
## Signing Secrets [/organizations//flags/signing-secrets/]
- Parameters
- cursor (optional, string)
- per_page (optional, number)
Default: 10
- offset (optional, number)
Default: 0
### Browse Signing Secrets [GET]
Browse a list of signing secrets. Secrets are unique per provider. Secrets only show the first six characters; the remainder are redacted.
**Attributes**
| Column | Type | Description |
| --------- | ------ | ---------------------------------------------------------------------- |
| createdAt | string | ISO-8601 timestamp of when the secret was added. |
| createdBy | string | The user responsible for adding the secret. |
| id | number | A unique identifier for the secret entry. |
| provider | string | The provider this secret applies to. |
| secret | string | A secret value which allows us to verify the signature of the request. |
- Response 200
```json
{
"data": [
{
"createdAt": "2024-12-12T00:00:00+00:00",
"createdBy": 12345,
"id": 123,
"provider": "launchdarkly",
"secret": "abc123**********"
}
]
}
```
### Create Signing Secret [POST]
Requests from web hook providers can be signed. We use the signing secret to verify the webhook's origin is authentic.
- Request (application/json)
```json
{
"provider": "launchdarkly",
"secret": "d41d7d1adced450d9e2eb7f76dde6a04"
}
```
- Response 201
## Signing Secret [/organizations//flags/signing-secrets//]
### Delete Signing Secret [DELETE]
Delete a signing secret.
- Response 204
## Webhooks [/organizations//flags/hooks/provider//]
### Create Generic Flag Log [POST]
A flag log event must be emitted after every flag definition change which influences a flag's evaluation. Updates to a flag that do not change a flag's evaluation logic do not need to be emitted to this endpoint. We are only concerned with changes which could have influenced behavior.
Sentry does not currently have a concept of disambiguating flag changes by project or environment. Everything is done at the organization level. Flag changes that are duplicated across projects, environments, or other groupings within the provider, must be de-duplicated. To support this, the posted payload sets a "change_id" field for idempotency. In the presence of duplicate ids, only one audit-log record is written in Sentry.
**Data Attributes**
| Column | Type | Description |
| --------------- | ------ | -------------------------------------------------------------- |
| action | string | Enum of `created`, `updated`, or `deleted`. |
| change_id | number | A 64-bit idempotency token representing a unique change group. |
| created_at | string | String formatted UTC date time: YYYY-MM-DDTHH:MM:SS. |
| created_by | object | Created-by object. |
| created_by.id | string | User identifier which made the change. |
| created_by.type | string | Enum of `email`, `id`, or `name`. |
| flag | string | The name of the flag changed. |
**Meta Attributes**
| Column | Type | Description |
| ------- | ---- | --------------------- |
| version | int | The protocol version. |
- Request (application/json)
```json
{
"data": [
{
"action": "created",
"created_at": "2024-12-12T00:02:00+00:00",
"created_by": {
"id": "first.last@company.com",
"type": "email"
},
"flag": "hello.world"
}
],
"meta": {
"version": 1
}
}
```
- Response 201
### Create Provider-Specific Flag Log [POST]
The shape of the request object varies by provider. The `` URI parameter informs the server of the shape of the request and it is on the server to handle the provider. The following providers are supported: LaunchDarkly.
Webhooks are signed by their provider. The provider handler must use the secret stored in Sentry to verify the signature of the payload. Failure to do so could lead to unauthorized access.
Any request content-type is acceptable (JSON, XML, binary-formats) so long as the server is capable of decoding the request and mapping it to our object model.
- Request
- Response 201
---
### Src Sentry Workflow Engine Docs Adding Detectors (`src/sentry/workflow_engine/docs/adding-detectors.md`)
# Adding a Workflow Engine Detector
This guide is a code-oriented checklist for adding a detector that evaluates typed data
and emits Issue Platform occurrences or resolutions. Read the
[architecture overview](../README.md), [data model](data-model.md), and
[condition contract](conditions.md), then use the [execution guide](execution.md) to
trace the surrounding pipelines.
Detector implementations normally live in the product module that owns the data and
Issue Platform group type. Only shared abstractions, processors, and generic source or
condition handling belong in `sentry.workflow_engine`.
Before coding, identify:
- The owning product module
- The packet payload type
- The source of packet ordering
- Whether one packet contains one value or grouped values
- Trigger and recovery semantics
- The stable issue fingerprint
- The API creation/update requirements
- Expected packet volume and state cardinality
## Choose a Handler Abstraction
```mermaid
flowchart TD
Start[New detector] --> State{Needs persisted priority, dedupe, thresholds, or resolution?}
State -->|Yes| Stateful[Inherit StatefulDetectorHandler]
State -->|No| Default{Does the default stateless evaluation fit?}
Default -->|Yes| Handler[Inherit DetectorHandler]
Default -->|No| Custom[Inherit DetectorHandler and override evaluate]
Stateful --> Hooks[Implement extract_value, extract_dedupe_value, and create_occurrence]
Handler --> Minimal[Implement extract_value and create_occurrence]
Custom --> Minimal
```
### Handler hierarchy
| Class | Role |
| ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`BaseDetectorHandler`](../handlers/detector/base.py) | Abstract interface: `_evaluate`, `evaluate`, `extract_value`, and `create_occurrence`. Shared occurrence identity, fingerprint, evidence, and event data helpers. |
| [`DetectorHandler`](../handlers/detector/condition.py) | Data condition implementation: condition-group loading, evaluation metrics, and a default stateless `evaluate`. The base for new detectors. |
| [`StatefulDetectorHandler`](../handlers/detector/stateful.py) | `DetectorHandler` plus dedupe, priority thresholds, durable state, and resolution. Replaces the default `evaluate`. |
### `BaseDetectorHandler`
[`BaseDetectorHandler`](../handlers/detector/base.py) is the abstract interface. Do not inherit this directly.
The exception is [`ErrorDetectorHandler`](../../grouping/grouptype.py), a placeholder
that associates error issues with a detector. It does not get evaluated in the normal detector evaluation pipeline and its methods are all stubs.
### `DetectorHandler`
[`DetectorHandler`](../handlers/detector/condition.py) is the base class for every detector
that does not need persisted state. It loads the detector's trigger condition group and
provides a default stateless `evaluate`. A concrete subclass implements `extract_value`
and `create_occurrence`.
#### Default stateless evaluation
```mermaid
flowchart TD
Packet[Receive DataPacket] --> Extract[extract_value]
Extract --> Group[Normalize to group key and value pairs]
Group --> Conditions[evaluate_conditions for each group]
Conditions --> Missing{Condition group missing?}
Missing -->|Yes| Skip[Skip the group]
Missing -->|No| Priority{Selected priority is OK?}
Priority -->|Yes| None[No Issue Platform output]
Priority -->|No| Occurrence[create_occurrence and build IssueOccurrence]
```
Override these hooks to adjust the default evaluation:
| Hook | Default |
| ----------------------- | ------------------------------------------------------------------------------ |
| `evaluate_conditions` | Evaluate the trigger condition group and select the highest triggered priority |
| `get_issue_fingerprint` | `detector:`, plus `:` for grouped detectors |
| `get_event_id` | `event_id` from the event data, otherwise a random UUID |
| `get_occurrence_id` | UUID5 of the detector ID, group key, and event ID |
Override `evaluate` itself when the detector needs a different flow. See
[`PreprodSizeAnalysisDetectorHandler`](../../preprod/size_analysis/grouptype.py) for how to achieve this
### `StatefulDetectorHandler`
Prefer [`StatefulDetectorHandler`](../handlers/detector/stateful.py) for detectors that
evaluate ordered samples and transition among `OK`, `LOW`, `MEDIUM`, and `HIGH`.
It provides:
- Bulk state loading for grouped values
- Out-of-order and duplicate packet suppression
- Condition-group evaluation
- Priority threshold counters
- Durable priority transitions
- Occurrence creation for non-OK transitions
- Resolution messages for transitions to `OK`
- Shared metrics and evidence data
Uptime and metric issues are canonical examples.
## Design the Packet and Evaluation Value
[`DataPacket[T]`](../models/data_source.py) separates source identity from payload:
```python
@dataclass
class ExamplePacket:
value: float
sequence: int
packet = DataPacket(
source_id=str(source.id),
packet=ExamplePacket(value=42.0, sequence=123),
)
```
The payload type is what the producer knows. The evaluation value is what detector
conditions consume. They can differ.
Every handler implements `extract_value`. A stateful handler also implements
`extract_dedupe_value`:
```python
def extract_value(
self, data_packet: DataPacket[ExamplePacket]
) -> float | DetectorGroupValues[float]:
return data_packet.packet.value
def extract_dedupe_value(self, data_packet: DataPacket[ExamplePacket]) -> int:
return data_packet.packet.sequence
```
`extract_value` can return one value, or `DetectorGroupValues` mapping group key to value.
Group keys create independent Issue Platform fingerprints for one detector, and independent
detector state for stateful handlers. Any other return value, including a plain `dict`, is
one ungrouped value.
`extract_dedupe_value` must return a positive integer whose ordering matches the source
stream. Missing Redis state is read as zero, so a first value of zero or less is skipped.
Good values include a source sequence or integer timestamp with sufficient resolution.
Do not use processing time when the producer can deliver older packets later. The Redis
watermark expires after seven days.
The dedupe value is extracted once per packet and applied to every group key. A fresh
value is committed even when no condition matches. Detectors that need independent
ordering per group or replay of previously non-matching packets require custom
orchestration.
### Group-key requirements
A `DetectorGroupKey` is `str | None`:
- Return one value for an ungrouped detector; the state key is `None`.
- Return `DetectorGroupValues` for independent substreams.
- Use non-empty strings no longer than 200 characters. Do not use `""`: state-key
construction and the database uniqueness constraint treat it like ungrouped `None`.
- Keep keys stable across trigger and recovery packets.
- Keep key cardinality bounded. Each key can create durable and Redis state plus an
Issue Platform group.
- Do not put unbounded IDs or raw user-controlled values into group keys without a
product-level retention and cardinality plan.
- Return an empty `DetectorGroupValues` to mean "no groups"; nothing is evaluated.
## Implement Occurrence Creation
Every handler implements `create_occurrence` with this contract:
```python
def create_occurrence(
self,
evaluation: DataConditionGroupEvaluation,
data_packet: DataPacket[ExamplePacket],
priority: DetectorPriorityLevel,
) -> tuple[DetectorOccurrence, EventData]:
...
```
The [`DetectorOccurrence`](../handlers/detector/base.py) describes the Issue Platform
occurrence. `EventData` supplies event fields associated with it.
An abbreviated implementation looks like:
```python
def create_occurrence(
self,
evaluation: DataConditionGroupEvaluation,
data_packet: DataPacket[ExamplePacket],
priority: DetectorPriorityLevel,
) -> tuple[DetectorOccurrence, EventData]:
occurrence = DetectorOccurrence(
issue_title=f"{self.detector.name} triggered",
subtitle="The monitored value crossed its threshold",
evidence_data={"value": data_packet.packet.value},
evidence_display=[],
type=ExampleGroupType,
level="error",
culprit="",
priority=priority,
)
event_data: EventData = {"platform": "other", "tags": {}}
return occurrence, event_data
```
Use an existing product implementation for the exact event shape. The snippet is
illustrative and omits product-specific evidence. The returned dictionary must be
mutable.
The default evaluations supply defaults for `environment`, `platform`, `received`, and `tags`
The default evaluations overwrite `timestamp` and `project_id`.
Stateless and stateful handlers differ on the `event_id` field:
- The default `DetectorHandler.evaluate` uses a random or derived `event_id` and then generates an occurrence ID from it.
- `StatefulDetectorHandler` overwrites `event_id` with a random ID that is also used as
the occurrence ID.
### Evidence
Evidence serves two audiences:
- `evidence_data` stores structured data used by APIs, grouping context, and issue
rendering.
- `evidence_display` stores the human-readable evidence shown on the issue.
The stateful handler adds standard detector and condition evidence. Product-specific
evidence should be serializable, bounded in size, and avoid sensitive data.
### Fingerprints
#### Default evaluation
The default evaluation uses `get_issue_fingerprint(group_key)`:
```text
detector:
detector::
```
Overriding `get_issue_fingerprint` replaces the whole fingerprint.
#### Stateful evaluation
The stateful implementation always includes its detector state key:
```text
detector:
detector::
```
Override `build_issue_fingerprint` only when the product needs additional stable
grouping components. The engine appends the detector state key to trigger and resolution
fingerprints. Currently occurrence creation passes `group_key` to this override while
resolution calls it without that argument. Do not make custom components depend on the
argument unless resolution is updated and trigger/resolution tests cover the behavior.
Fingerprint changes can split or merge production issues and can break recovery.
Test the exact trigger and resolution fingerprints. For grouped detectors, explicitly
test that one key cannot resolve another key's issue.
## Configure Threshold Semantics
`StatefulDetectorHandler.thresholds` maps priority to the number of qualifying
evaluations required before transition:
```python
@property
def thresholds(self) -> DetectorThresholds:
return {
DetectorPriorityLevel.: self.detector.config["failure_threshold"],
DetectorPriorityLevel.OK: self.detector.config["recovery_threshold"],
}
```
The default threshold is one. For a detectors evaluation, the threshold will say we needs
to see X number of a non-OK evaluations to trigger the detector. An OK
evaluation clears non-OK counters before advancing the OK counter. An evaluation equal
to the current durable state clears accumulated counters. These rules are not simply an
independent consecutive-sample counter for every priority.
A detector that can trigger but never produce `OK` evaluations will not resolve through
the stateful handler.
For the StatefulDetectorHandler implementation, failing a non-OK condition does not resolve
the detector. To have the detector be considered resolved, the trigger group must resolve
to `DetectorPriorityLevel.OK`. This normally uses a passing condition with an `OK` result,
but an empty or passing `NONE` group can also trigger with no priority-bearing result and
use the default `OK` priority. A missing detector trigger group is invalid and produces no
state transition.
**NOTE**: `create_occurrence` does not receive the current group key. The base handler still adds
the key to `DetectorEvaluation.data` and the engine fingerprint. If product evidence
must contain the key, include it in the extracted evaluation value or implement and
test the required custom orchestration.
### Example Threshold Evaluation
Here's an illustration of how a threshold works; specifically how the `OK` result will
reset the threshold counts. Note that if it the priority is de-escalating, it does _not_
reset the threshold counts. See
#### Detector Threshold Settings
```python
@property
def thresholds(self) -> DetectorThresholds:
return {
DetectorPriorityLevel.HIGH: 2,
DetectorPriorityLevel.LOW: 2,
DetectorPriorityLevel.OK: 1
}
```
#### Example Evaluation Results
##### Example 1 - Consecutive Triggers
```python
- DetectorPriorityLevel.HIGH
- DetectorPriorityLevel.HIGH
===> Detector is triggered at HIGH
```
##### Example 2 - OK resets thresholds
```python
- DetectorPriorityLevel.HIGH
- DetectorPriorityLevel.OK
- DetectorPriorityLevel.HIGH # Note that the Detector does *not* trigger here
- DetectorPriorityLevel.OK
- DetectorPriorityLevel.HIGH
- DetectorPriorityLevel.HIGH
===> Detector is triggered at HIGH
```
##### Example 3 - De-escalation does _not_ reset thresholds
```python
- DetectorPriorityLevel.OK
- DetectorPriorityLevel.HIGH
- DetectorPriorityLevel.LOW
- DetectorPriorityLevel.HIGH
===> Detector is triggered at HIGH
```
##### Example 4 - Escalating detectors do _not_ reset thresholds
```python
- DetectorPriorityLevel.LOW
- DetectorPriorityLevel.HIGH
- DetectorPriorityLevel.LOW
===> Detector is triggered at LOW
- DetectorPriorityLevel.HIGH
===> Detector is triggered at HIGH
```
## Define the Issue and Detector Type
Create or extend a concrete [`GroupType`](../../issues/grouptype.py) in the owning
product's `grouptype.py`:
```python
@dataclass(frozen=True)
class ExampleGroupType(GroupType):
type_id = 12345 # Allocate the real Issue Platform type ID.
slug = "example_detector"
description = "Example detector issue"
category = GroupCategory.ERROR.value # Choose the applicable category.
released = False
default_priority = PriorityLevel.HIGH
```
Use the Issue Platform documentation and existing group types to choose the type ID,
category, release controls, priority, auto-resolution, escalation behavior, and
notification configuration.
### Register detector settings
The group type describes the issue. The detector components live in a
[`DetectorSettings`](../types.py) subclass registered in
[`detector_settings_registry`](../registry.py) under the group type's slug:
```python
from sentry.workflow_engine.registry import detector_settings_registry
from sentry.workflow_engine.types import DetectorAPIOperation, DetectorSettings, FeatureGate
@detector_settings_registry.register(ExampleGroupType.slug)
class ExampleDetectorSettings(DetectorSettings):
handler = ExampleDetectorHandler
validator = ExampleDetectorValidator
config_schema = {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"failure_threshold": {"type": "integer", "minimum": 1},
"recovery_threshold": {"type": "integer", "minimum": 1},
},
"required": ["failure_threshold", "recovery_threshold"],
"additionalProperties": False,
}
```
Register the class, not an instance. The settings can also live in their own module, as long as that module is
imported during application startup.
[`DetectorSettings`](../types.py) fields are:
| Field | Purpose |
| ------------------ | -------------------------------------------------------------------------------------- |
| `api_availability` | Optional operation-to-`bool`/`FeatureGate` overrides; inherits the all-enabled default |
| `handler` | Runtime `DetectorHandler` class |
| `validator` | Native detector API validator |
| `config_schema` | Save-time JSON schema for `Detector.config` |
| `filter` | Optional `Q` filter controlling user-visible detector rows of this type |
Settings classes inherit `DetectorSettings.DEFAULT_API_AVAILABILITY` unless they override
`api_availability`. This shared mapping is immutable and sets each `DetectorAPIOperation`
member to `True`, so ordinary detector implementations need no availability declaration.
Missing entries in custom maps still default to `True`. The supported enum members are
`LIST`, `GET`, `POST`, `PUT`, and `DELETE`.
Each gate is either `True`, `False`, or `FeatureGate("organizations:feature-name")`.
`FeatureGate` checks a registered organization-scoped feature at request time using the
authorized organization and actor. Feature names remain strings, but raw strings are not
valid gate values.
To hide a type from the list and its counts without disabling other operations:
```python
api_availability = {
**DetectorSettings.DEFAULT_API_AVAILABILITY,
DetectorAPIOperation.LIST: False,
}
```
For a rollout across every detector-platform API, assign the same feature gate to all
operations:
```python
api_availability = {
operation: FeatureGate("organizations:example-detector-api")
for operation in DetectorAPIOperation
}
```
`LIST` additionally requires `GET`, so a denied `GET` gate hides a type from both listing
and counts even when `LIST` is explicitly allowed. Gate checks short-circuit in that order:
implied `GET` for `LIST`, then the requested operation.
Generic detector-platform APIs evaluate these gates through `get_excluded_detector_types`,
which reads registered settings without fetching detector rows. List filtering happens
before pagination and hit counting, including requests that filter by detector ID or type.
Counts use the same `LIST` policy. `GET` also controls detail retrieval and type discovery;
`LIST` alone does not restrict those other reads. Gates are independent of the group type's
`released` state and the detector's `enabled` state.
Product-specific APIs, such as anomaly-data retrieval, do not inherit these platform API
exclusions.
Individual requests for a detector excluded from `GET`, `PUT`, or `DELETE` return 404.
Creation of a type excluded from `POST`, or an update targeting a type excluded from
`PUT`, returns a 400 type-field validation error before saving.
Bulk updates and deletions filter excluded types before mutation. Query/project selections
operate on the remaining types; explicit ID selections return 400 without changing any
detectors if an excluded ID was requested.
Read exclusions do not restrict creation, updates, or deletion. Write exclusions do not
affect listing, and API exclusions never change detector evaluation.
Both registrations happen at import time. The `GroupType` subclass registers itself
with the global Issue Platform registry when the class is created. The settings class
registers itself when its decorator runs. Sentry startup imports the root
`grouptype.py` of every installed Django app through
[`import_grouptype`](../../issues/grouptype.py). Put both classes there, or import a
nested implementation from that file, as
[`sentry.preprod.grouptype`](../../preprod/grouptype.py) does. Add a test that checks
both the group type and its settings resolve by slug after normal application startup.
## Implement API Validation
Subclass
[`BaseDetectorTypeValidator`](../endpoints/validators/base/detector.py) in the owning
product or under `workflow_engine/endpoints/validators` when the behavior is shared.
The base validator provides:
- Common detector fields
- Condition-result priority validation
- Detector config-schema validation
- Transactional creation and update
- Workflow connections
- Audit entries
- Scheduled deletion
Detector-specific validators commonly provide:
- A `data_conditions` validator
- Product-specific data-source fields and creators
- Cross-field validation
- Quota calculation
- Detector-specific cleanup
If for some reason there's not a consistent data_source or you need to opt-out of the data_source
to detector connections, then set `data_source_required = False`. This should only be when the detector
is intentionally created and resolved outside the generic data-source graph. Set `enforce_single_datasource = True`
when multiple sources would make the handler ambiguous.
_Do not create the detector graph piecemeal in an endpoint._ The base validator's
transaction coordinates condition groups, detector config validation, source mappings,
workflow connections, and audit logging.
## Connect a Data Source
Skip this section only when the validator intentionally sets
`data_source_required = False` and the producer has another reliable detector lookup.
### Reuse or register a source type
First determine whether an existing source type represents the producer object. If not,
define a source type constant and implement
[`DataSourceTypeHandler`](../types.py):
- `bulk_get_query_object`
- `related_model`
- `get_instance_limit`
- `get_current_instance_count`
- `get_relocation_model_name`
Register the handler with [`data_source_type_registry`](../registry.py). Update
`DataSource.__relocation_dependencies__` when the source references a model not already
listed. Add deletion, quota, relocation, and unknown-type tests appropriate to the new
source.
### Create the source graph
Native API creation expects validated source data to contain a product-specific creator.
The base validator calls that creator, creates the generic `DataSource`, and creates the
`DataSourceDetector` mapping in the detector transaction.
The resulting source identity must match the producer call:
```text
DataSource.type == query_type passed to process_data_packet
DataSource.source_id == DataPacket.source_id
```
Test this match end to end. A valid detector with a mismatched source type or stringified
ID will never be selected.
## Produce Packets
Use [`process_data_packet`](../processors/data_packet.py) when the producer has a generic
source mapping:
```python
process_data_packet(
DataPacket(source_id=str(source.id), packet=payload),
DATA_SOURCE_EXAMPLE,
)
```
Use [`process_detectors`](../processors/detector.py) directly only when the product path
already resolves the exact detector set and a generic lookup would add no value. Explain
and test that ownership boundary in the producer module. `process_detectors` does not
filter `enabled` or lifecycle status. Direct callers must define and test both
eligibility checks rather than assuming the generic source path's filtering applies.
Producer requirements:
- Preserve the source's natural ordering value.
- Expect retries and duplicate delivery.
- Avoid one database query per detector or group key.
- Keep packet payloads bounded.
- Record enough product metrics to distinguish missing input from detector rejection.
- Decide whether detector exceptions should fail, retry, or isolate one item in a batch.
## Configure Trigger Conditions
Detector trigger conditions return `DetectorPriorityLevel` values. The base API
validator rejects results that are not valid priority integers.
Prefer built-in operators when conditions compare the extracted value directly. Add a
[`DataConditionHandler`](../types.py) only when evaluation needs a richer comparison or
event context.
The complete shared evaluation contract and condition-handler extension path live in
[Conditions](conditions.md). In particular, handler-group placement is not enforced by
generic backend validation, and stateful Detector triggers do not participate in the
Workflow batched data-acquisition phase.
## Rollout and Operational Safety
Before enabling production issue creation:
- Keep a new `GroupType` unreleased until Issue Platform ingest, post-process, and UI
behavior are intentionally enabled.
- Add a feature flag or runtime option when product rollout needs control beyond the
Issue Platform release controls.
- Estimate packet throughput and group-key cardinality.
- Estimate PostgreSQL `DetectorState` growth and Redis counter usage.
- Verify occurrence and event payload sizes.
- Ensure fingerprints cannot collide with another detector or an experiment.
- Add metrics with bounded-cardinality tags.
- Verify deletion and missing-record behavior for queued work.
- Document whether old packets can arrive after configuration updates.
- Confirm recovery behavior before triggering real issues.
Detector rollout and Issue Platform type rollout are separate controls. A detector can
run without its resulting type being fully exposed, and a released group type does not
cause a producer to run.
## Implementation Checklist
### Design
- [ ] Confirm Workflow Engine is the correct detection system.
- [ ] Define the packet payload and evaluation value types.
- [ ] Choose the handler: `StatefulDetectorHandler` or `DetectorHandler`.
- [ ] For stateful detectors, choose an ordered, integer dedupe value.
- [ ] Decide whether results are grouped and bound group-key cardinality.
- [ ] Specify trigger, escalation, and recovery transitions.
- [ ] Specify stable trigger and resolution fingerprints.
### Issue type and handler
- [ ] Add or update the product `GroupType`.
- [ ] Register a `DetectorSettings` subclass in `detector_settings_registry` with
handler, validator, and strict config schema.
- [ ] Implement the selected detector handler abstraction.
- [ ] Implement occurrence evidence and event data.
- [ ] Implement thresholds and recovery for stateful detectors.
- [ ] Verify registry discovery at startup.
### API and persistence
- [ ] Implement the detector-specific validator.
- [ ] Validate condition comparisons and detector config.
- [ ] Reuse or register a data source type.
- [ ] Create `DataSource` and `DataSourceDetector` through the validator path.
- [ ] Confirm cache invalidation covers every new processing relationship.
- [ ] Confirm deletion and relocation behavior.
### Producer
- [ ] Build the typed `DataPacket`.
- [ ] Call `process_data_packet` or deliberately resolve and call `process_detectors`.
- [ ] Match packet source identity to the persisted `DataSource`.
- [ ] Add bounded metrics and exception handling.
### Rollout
- [ ] Configure Issue Platform release controls.
- [ ] Add product flags or options if needed.
- [ ] Validate throughput, state cardinality, and payload sizes.
- [ ] Plan trigger and recovery monitoring.
## Test Matrix
| Area | Required cases |
| ---------------- | --------------------------------------------------------------------------- |
| Value extraction | Normal packet, malformed or missing product data |
| Dedupe | Stateful: new, duplicate, and out-of-order packets |
| Conditions | Each configured priority and no-match/OK result |
| Thresholds | Stateful: below threshold, transition at threshold, counter reset, recovery |
| Grouping | Independent state and fingerprints for at least two group keys |
| Occurrence | Title, type, priority, evidence, event data, fingerprint |
| Resolution | Stateful: same issue identity as trigger and correct status change |
| Registry | Group type, detector settings, source, and conditions resolve after startup |
| API | Create, update, invalid config, invalid condition, delete, audit, rollback |
| Data source | Correct mapping, disabled detector exclusion, cache invalidation |
| Producer | Correct packet/source identity and exception behavior |
| Integration | Issue Platform payload produced from representative product input |
| Lifecycle | Deleted detector/source and queued-work races |
Use existing factory methods rather than direct model creation in tests. The most useful
shared references are:
- [`test_base.py`](../../../../tests/sentry/workflow_engine/handlers/detector/test_base.py)
for the default stateless evaluation
- [`test_stateful.py`](../../../../tests/sentry/workflow_engine/handlers/detector/test_stateful.py)
- [`test_data_packet.py`](../../../../tests/sentry/workflow_engine/processors/test_data_packet.py)
- [`test_detector.py`](../../../../tests/sentry/workflow_engine/processors/test_detector.py)
- [`test_organization_project_detector_index.py`](../../../../tests/sentry/workflow_engine/endpoints/test_organization_project_detector_index.py)
- [`test_integration.py`](../../../../tests/sentry/workflow_engine/test_integration.py)
## Recommended Implementations
| Implementation | Why to read it |
| ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| [`UptimeDetectorHandler`](../../uptime/grouptype.py) | Stateful trigger/recovery thresholds, source data, evidence, and API config |
| [`MetricIssueDetectorHandler`](../../incidents/grouptype.py) | Stateful metric thresholds, anomaly payloads, and metric evidence |
| Processing error handlers in [`processing_errors/grouptype.py`](../../processing_errors/grouptype.py) | Customized state transitions and asymmetric resolution |
| [`PreprodSizeAnalysisDetectorHandler`](../../preprod/size_analysis/grouptype.py) | Custom `DetectorHandler.evaluate` override |
Copy architecture, not product assumptions. Each example has specialized producer,
state, and lifecycle behavior.
---
### Src Sentry Workflow Engine Docs Conditions (`src/sentry/workflow_engine/docs/conditions.md`)
# Conditions in the Workflow Engine
Detectors and Workflows both store [`DataConditionGroup`](../models/data_condition_group.py)
and [`DataCondition`](../models/data_condition.py) records. Their immediate paths use
[`process_data_condition_group`](../processors/data_condition_group.py); delayed
Workflow processing evaluates slow conditions with
[`evaluate_data_conditions`](../processors/data_condition_group.py). They do not pass
the same values into that shared layer or interpret its results the same way.
Read this document before adding or changing a condition. It defines the shared
evaluation contract, the Detector and Workflow-specific behavior around it, and the
complete path for adding a condition type and handler.
## Architecture
```mermaid
flowchart TD
subgraph Shared[Shared condition layer]
Group[DataConditionGroup]
Conditions[DataCondition records]
Process[process_data_condition_group]
Delayed[evaluate_data_conditions for delayed values]
Evaluation[DataConditionGroupEvaluation]
Group --> Conditions
Conditions --> Process
Process --> Evaluation
Conditions --> Delayed
Delayed --> Evaluation
end
Packet[DataPacket] --> Extract[DetectorHandler extracts product value]
Extract --> Process
Evaluation --> DetectorResult[Detector interprets priority results]
DetectorResult --> Issue[IssueOccurrence or StatusChangeMessage]
Event[WorkflowEventData] --> Process
Evaluation --> When[Workflow interprets WHEN as a gate]
When --> If[Workflow evaluates IF groups]
If --> Actions[Eligible actions]
```
The paths converge only in the shared condition layer. Detector output later reaches
Workflow processing through Issue Platform ingestion and issue post-processing; a
Detector evaluation does not invoke a Workflow directly. See
[`process_detectors`](../processors/detector.py) and
[`process_workflows`](../processors/workflow.py).
## Shared Condition Layer
### Persistent records
[`DataCondition`](../models/data_condition.py) stores:
| Field | Contract |
| ------------------ | --------------------------------------------------------------------------- |
| `type` | A string represented by the [`Condition`](../models/data_condition.py) enum |
| `comparison` | JSON passed to an operator or registered handler |
| `condition_result` | JSON used when a boolean comparison matches |
| `condition_group` | Required parent `DataConditionGroup` |
[`DataConditionGroup`](../models/data_condition_group.py) stores an organization and a
logic type. Detector trigger groups, Workflow WHEN groups, and Workflow IF groups all
use this same model.
Those are intended roles, not exclusive database types. The schema independently
constrains each relation, but does not prevent one group from being referenced through
more than one role-bearing relation. Code that creates groups must preserve role
separation by convention.
### Condition dispatch
[`DataCondition.evaluate_value`](../models/data_condition.py) evaluates one condition:
1. `eq`, `gte`, `gt`, `lte`, `lt`, and `ne` use the built-in operators in
`CONDITION_OPS`.
2. Every other `Condition` looks up a class in
[`condition_handler_registry`](../registry.py) and calls
`handler.evaluate_value(value, comparison)`.
3. A boolean handler/operator result is interpreted as a match:
- `False` becomes `result=None` and `triggered=False`.
- `True` is replaced by `get_condition_result()`, which converts a valid integer
priority to `DetectorPriorityLevel`, preserves floats and booleans, leaves other
integers unchanged, and rejects unsupported JSON shapes with `ConditionError`.
4. A non-boolean handler result passes through unchanged and does not receive this
stored-result normalization.
5. The final evaluation is triggered when `result is not None`, not when the result is
truthy.
The last two rules are important. A matched condition whose stored `condition_result` is
`False` has `triggered=True` and `result=False`. A Detector handler that directly returns
the raw integer `1` does not produce `DetectorPriorityLevel.LOW` and can be ignored by
the stateful priority filter. Consumers must use `.triggered` rather than `bool(result)`.
The return value is a
[`DataConditionEvaluation`](../processors/evaluations/condition.py). It carries the
condition, input data, result, triggered flag, and optional `ConditionError`.
### Group evaluation contract
The shared entry point is:
```python
def process_data_condition_group(
group: DataConditionGroup,
value: T,
data_conditions_for_group: list[DataCondition] | None = None,
) -> tuple[DataConditionGroupEvaluation, list[DataCondition]]:
...
```
The first return item is the current group evaluation. The second is the list of slow
conditions that still require evaluation. It is not a list of every condition skipped
by short-circuiting.
If `data_conditions_for_group` is omitted, the processor uses prefetched conditions or
loads them through `get_data_conditions_for_group`. Bulk callers should pass already
loaded conditions to avoid per-group queries.
[`DataConditionGroup.Type`](../models/data_condition_group.py) has four values:
| Logic type | Behavior |
| ------------------- | ----------------------------------------------------------------------------- |
| `ANY` | Evaluate all supplied conditions; pass when at least one is triggered |
| `ANY_SHORT_CIRCUIT` | Stop and pass on the first triggered condition |
| `ALL` | Evaluate all supplied conditions; pass only when every condition is triggered |
| `NONE` | Stop and fail on the first triggered condition; pass when none trigger |
An existing group with no conditions returns a clean, triggered evaluation for every
logic type. A passing `NONE` group has no passing condition evaluations in its result
data. An invalid stored logic type returns a non-triggered evaluation with a
`ConditionError`.
These behaviors are implemented by
[`evaluate_data_conditions`](../processors/data_condition_group.py) and
[`evaluate_condition_group_results`](../processors/data_condition_group.py).
### Inline and batched condition data
[`SLOW_CONDITIONS`](../models/data_condition.py) is a manual list of six event-frequency
and session condition types. A slow condition has the same `DataConditionHandler` and
evaluation-result contract as any other condition. "Slow" describes how its input data
is acquired: the required query is too expensive to run inline for each Workflow, so the
engine batches equivalent queries and gathers the data once for evaluation across the
buffered Workflows.
The classification is not inferred from `DataConditionHandler.Group`, subgroup, or
handler behavior. It only controls the split between inline evaluation and the batched
data-acquisition phase.
`process_data_condition_group` splits fast and slow conditions:
1. A group containing only slow conditions returns a provisional non-triggered result
and all slow conditions.
2. Otherwise, fast conditions run first.
3. A fast result is conclusive when:
- `ANY` or `ANY_SHORT_CIRCUIT` passes, or
- `ALL` or `NONE` fails.
4. A conclusive result discards the remaining slow list.
5. A non-conclusive result returns the remaining slow conditions to the caller.
For event-driven Workflows, the delayed processor batches query generation, obtains the
query values, and calls the same `evaluate_data_conditions` function used by inline
group processing. This is a scheduling/data-fetch difference, not a second condition
semantics system. Activity-driven Workflows and stateful Detectors do not enqueue this
batched phase.
### Errors and taint
[`BaseWorkflowEngineEvaluation`](../processors/evaluations/base.py) keeps `result`,
`triggered`, and `error` separate. An evaluation with an error is _tainted_: the answer
may have been affected by a failed condition.
The `any`, `all`, and `none` combinators propagate errors only when the failed input
could affect the answer. For example, one clean true input makes `ANY` clean even when
another input failed; an `ANY` result of false remains tainted if any input failed.
The condition error boundary is intentionally narrow:
- Operator `TypeError`, missing handler registration, invalid stored result data, and a
handler-raised `DataConditionEvaluationException` become tainted, non-triggered
condition evaluations.
- An invalid stored condition-type string raises
`DataConditionEvaluationException` from `DataCondition.evaluate_value`.
- Arbitrary exceptions raised by a handler are not converted to `ConditionError`.
An arbitrary exception therefore escapes the shared layer. Whether it is retried depends
on the Detector producer or Workflow task entrypoint; the condition framework itself
does not retry handler evaluation.
Neither Detector nor Workflow processing automatically suppresses every tainted result.
A tainted group whose `.triggered` value is decisive can still cause a state transition
or select actions. Handler code should raise `DataConditionEvaluationException` only for
known evaluation failures and must not rely on taint as a retry mechanism.
### Validation is layered
[`BaseDataConditionValidator`](../endpoints/validators/base/data_condition.py) performs
the complete API validation path:
- Validates the enum value and handler registration
- Validates comparison and result JSON against handler schemas
- Calls `handler.validate_comparison(comparison, organization)` when organization
context is available
The model `pre_save` receiver in
[`receivers/data_condition.py`](../receivers/data_condition.py) is weaker. It validates
only comparison JSON for registered, non-operator handlers. It does not enforce result
schemas, contextual validation, or handler placement, and `bulk_create` bypasses the
signal.
Use the existing validators for API/model graph creation. Do not treat direct model
saves as equivalent validation.
## Detector and Workflow Touchpoints
| Concern | Detector trigger | Workflow WHEN | Workflow IF/action filter |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Owning relation | `Detector.workflow_condition_group` | `Workflow.when_condition_group` | `WorkflowDataConditionGroup.condition_group` |
| Input passed to shared layer | Detector-specific value extracted from a `DataPacket` | `WorkflowEventData` | `WorkflowEventData` |
| Intended handler group | `DETECTOR_TRIGGER` | `WORKFLOW_TRIGGER` | `ACTION_FILTER` |
| Placement enforced by generic backend validation | No | No | No |
| Expected result | `DetectorPriorityLevel` | Gate, conventionally stored `True` | Gate, conventionally stored `True` |
| Consumer | Detector handler selects priority/state transition | Decides whether Workflow continues | Selects an attached action group |
| Group not configured | Stateful handler treats it as invalid; no transition | `when_condition_group_id=None` passes | No IF groups means no actions are selected |
| Configured group row cannot be loaded | Stateful handler treats it as invalid; no transition | Tainted and non-triggered | Missing group rows are skipped by lookup/evaluation paths |
| Existing empty group | Shared layer passes with no condition results; the stateful default priority is `OK`, so an already-triggered Detector can count toward recovery | Passes | Passes and selects attached actions |
| Batched input acquisition | No batched phase; remaining conditions are logged and discarded | Event-driven processing batches data, then uses the shared evaluator | Event-driven processing batches data, then uses the shared evaluator |
| Activity behavior | Not applicable to detector evaluation | No environment; no delayed phase | No environment; no delayed phase |
`DataConditionHandler.Group` drives the top-level control placement exposed through
[`OrganizationDataConditionIndexEndpoint`](../endpoints/organization_data_condition_index.py).
The intended conceptual split is:
- `group` identifies whether the control belongs to Detector or Workflow configuration.
- `subgroup` organizes the UI control within that subsystem, such as trigger or action
filter placement.
The current enum shape does not represent that hierarchy literally. `Group` currently
combines subsystem and control phase through `DETECTOR_TRIGGER`, `WORKFLOW_TRIGGER`, and
`ACTION_FILTER`. `Subgroup` currently refines UI presentation with
`ISSUE_ATTRIBUTES`, `EVENT_ATTRIBUTES`, and `FREQUENCY`. The availability API serializes
those current values as `handlerGroup` and `handlerSubgroup`. Subgroup does not affect
evaluation.
Neither field determines the runtime input type or prevents a caller from storing a
handler in the wrong role. Matching the handler's input and result contract to its
placement is a convention-only invariant.
## Detectors
A [`Detector`](../models/detector.py) represents configured detection. A runtime handler
is selected through the [`DetectorSettings`](../types.py) registered for the detector
type in [`detector_settings_registry`](../registry.py).
Handlers that inherit `DetectorHandler` without overriding `evaluate` use the
[default stateless evaluation](adding-detectors.md#default-stateless-evaluation). It
selects a priority the same way as the stateful path below and discards slow conditions
the same way, but it has no dedupe or thresholds and produces no output for `OK`.
The common stateful path is
[`StatefulDetectorHandler.evaluate`](../handlers/detector/stateful.py):
1. Extract one packet-wide positive integer dedupe value.
2. Extract one evaluation value or a mapping of `DetectorGroupKey` to value.
3. Load PostgreSQL detector state and Redis dedupe/counter state.
4. Skip a group key when the packet dedupe value is not newer.
5. Enqueue the new dedupe watermark _before_ condition evaluation.
6. Pass each extracted value to `process_data_condition_group`.
7. From a triggered group, keep triggered results that are actual
`DetectorPriorityLevel` values and choose the maximum.
8. Apply priority threshold rules and persist state.
9. Produce an `IssueOccurrence` for a non-OK transition or a `StatusChangeMessage` for
a transition to `OK`.
Detector-specific consequences:
- A fresh packet advances the Redis watermark even when no condition matches.
- One dedupe value applies to every group key extracted from that packet. Independent
per-key ordering requires custom orchestration.
- A passing condition with `condition_result=DetectorPriorityLevel.OK` is the normal
recovery signal. Failing all non-OK conditions does not imply recovery.
- Stateful Detector processing has no batched data-acquisition phase. Conditions that
require it are logged as remaining work and discarded, so Detector trigger inputs must
be available inline.
- A tainted triggered result can still update counters and produce a transition.
- PostgreSQL state, Redis state, and Issue Platform publication are not one transaction.
`DetectorStateManager.commit_state_updates` writes PostgreSQL before Redis, and
`process_detectors` publishes only after handler evaluation returns. A failure after
state commit but before publication can cause a retry to be skipped by dedupe.
The default `on_complete` callback enters Issue Platform through
[`IssuePlatformOutcomeHandler`](../handlers/detector_outcome/issue_platform.py). Workflows
run later from the resulting issue event; they are not part of the detector state transaction.
## Workflows
A [`Workflow`](../models/workflow.py) is connected to detectors through
[`DetectorWorkflow`](../models/detector_workflow.py). Its conditions have two stages:
1. The optional WHEN group decides whether processing continues.
2. Each IF group is evaluated independently and selects the actions attached to that
group through [`DataConditionGroupAction`](../models/data_condition_group_action.py).
Both stages pass equivalent [`WorkflowEventData`](../types.py) values to the shared
layer. Each stage creates a dataclass copy with its Workflow environment; the copies
share the event-local cache. The value contains a `GroupEvent` or `Activity`, the issue
group, optional state and escalation data, and an optional workflow environment.
[`process_workflows`](../processors/workflow.py) performs the surrounding lifecycle:
1. Resolve the event-specific and issue-stream detectors.
2. Load enabled Workflows connected to those detectors for the event environment.
3. Evaluate WHEN groups.
4. Evaluate IF groups for Workflows whose WHEN passed or remains pending slow work.
5. Buffer unresolved slow work for eligible `GroupEvent` processing.
6. Resolve attached actions for passing IF groups.
7. Hand passing action groups to the Workflow frequency, history, active-action,
deduplication, and asynchronous dispatch path documented in [Execution](execution.md).
### Event and activity differences
| Input | `GroupEvent` | `Activity` |
| ----------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Task entry | `process_workflows_event` | Activity handler resolves a concrete detector, then calls `process_workflow_activity` |
| Environment | Event environment or global | Global only (`workflow_env=None`) |
| Slow WHEN/IF conditions | Can be buffered for delayed processing | Not buffered |
| Preferred detector | Occurrence evidence or default error detector, then issue stream fallback | Concrete detector supplied by the activity path |
These differences are implemented in [`tasks/workflows.py`](../tasks/workflows.py),
[`get_environment_by_event`](../processors/workflow.py), and
[`get_preferred_detector`](../processors/detector.py).
## Adding a Condition Type and Handler
This walkthrough is verified against
[`FirstSeenEventConditionHandler`](../handlers/condition/first_seen_event_handler.py) and
[`test_first_seen_event_handler.py`](../../../../tests/sentry/workflow_engine/handlers/condition/test_first_seen_event_handler.py).
The example adds a fast Workflow WHEN condition. Differences for Detector triggers and
Workflow IF conditions are called out explicitly.
### 1. Choose the placement and contract
Before editing, decide:
- **Detector trigger:** What detector-specific value will the handler receive? Should it
return a priority directly, or a boolean that maps to the stored priority result?
- **Workflow WHEN:** What property of `WorkflowEventData` gates the entire Workflow?
- **Workflow IF:** What property of `WorkflowEventData` gates one action group?
- **Batched Workflow WHEN or IF:** Does acquiring the evaluation input require an
expensive query? If so, the condition handler itself is unchanged, but the delayed
data-acquisition path must know how to batch and supply that input.
Do not make one handler accept unrelated Detector and Workflow input shapes. The generic
base class cannot enforce the runtime input type.
### 2. Add the enum value
Edit [`models/data_condition.py`](../models/data_condition.py):
```python
class Condition(StrEnum):
# ...
MY_NEW_CONDITION = "my_new_condition"
```
The enum feeds model choices and API validation. Storage is a `CharField`, not a database
enum. Run the repository's migration check rather than assuming whether a model-state
migration is required.
### 3. Implement and register the handler
For a fast Workflow WHEN condition, add
`handlers/condition/my_new_condition_handler.py`:
```python
from typing import Any
from sentry.workflow_engine.models.data_condition import Condition
from sentry.workflow_engine.registry import condition_handler_registry
from sentry.workflow_engine.types import DataConditionHandler, WorkflowEventData
@condition_handler_registry.register(Condition.MY_NEW_CONDITION)
class MyNewConditionHandler(DataConditionHandler[WorkflowEventData]):
group = DataConditionHandler.Group.WORKFLOW_TRIGGER
comparison_json_schema = {"type": "boolean"}
condition_result_schema = {"type": "boolean"}
label_template = "The new condition matches"
@staticmethod
def evaluate_value(event_data: WorkflowEventData, comparison: Any) -> bool:
# `group.some_field` is illustrative; use a real Group property.
return comparison and event_data.group.some_field
```
Required handler contract from [`DataConditionHandler`](../types.py):
| Member | Requirement |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `group` | Set the top-level Detector/Workflow control placement exposed to the UI |
| `evaluate_value(value, comparison)` | Return a `DataConditionResult`; raise `DataConditionEvaluationException` for a known evaluation failure |
| `comparison_json_schema` | Define accepted stored/API comparison JSON |
| `condition_result_schema` | Define accepted stored/API result JSON; the inherited `{}` is permissive |
| `subgroup` | Optional UI-only categorization within the top-level group; it does not affect evaluation |
| `label_template` or `render_label` | Needed when the condition must render through legacy rule-shaped APIs |
| `validate_comparison` | Override for organization-scoped or cross-field validation that JSON Schema cannot express |
For a Detector trigger, specialize the generic with the actual detector value type. A
boolean return maps to the stored `DetectorPriorityLevel`; a direct non-boolean
`DetectorPriorityLevel` return passes through. Detector API validation requires stored
condition results to be integers convertible to `DetectorPriorityLevel`.
For a Workflow IF condition, use `DataConditionHandler.Group.ACTION_FILTER`, continue to
accept `WorkflowEventData`, and set a subgroup when the automation UI needs to place it
under issue attributes, event attributes, or frequency.
### 4. Make the registration run at startup
Edit [`handlers/condition/__init__.py`](../handlers/condition/__init__.py):
```python
__all__ = [
# ...
"MyNewConditionHandler",
]
from .my_new_condition_handler import MyNewConditionHandler
```
This import is functional. [`Config.ready`](../apps.py) imports the condition package;
the package import executes the registration decorator. An enum and handler class that
are not imported remain unavailable at runtime.
A product-specific condition can live outside this package, but its owning app startup
must import the decorated module. Anomaly detection demonstrates this path through
[`incidents/handlers/condition`](../../incidents/handlers/condition) and
[`incidents/grouptype.py`](../../incidents/grouptype.py).
### 5. Decide whether manual lists apply
Registration does not update these lists automatically:
| List | Add the condition when |
| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| [`SLOW_CONDITIONS`](../models/data_condition.py) | It uses the delayed Workflow query path; delayed query generation and result handling must also support it |
| `PERCENT_CONDITIONS` | It is a delayed percentage comparison |
| `LEGACY_CONDITIONS` | It must be hidden from the condition availability endpoint |
A condition whose data must be acquired in the batched phase additionally needs a query
handler registered in
`slow_condition_query_handler_registry`. Implement the
[`BaseEventFrequencyQueryHandler`](../handlers/condition/event_frequency_query_handlers.py)
data-acquisition contract and register the condition in that module. This handler does
not replace `DataConditionHandler`; it gathers the values that are later passed through
the same condition evaluator. Without this registration,
[`generate_unique_queries`](../processors/delayed_workflow.py) cannot build a query and
the buffered condition cannot fire.
### 6. Verify API exposure and serialization
Once registered, the handler appears in
[`OrganizationDataConditionIndexEndpoint`](../endpoints/organization_data_condition_index.py)
under its `group`, unless it is in `LEGACY_CONDITIONS`. The endpoint exposes the type,
group, optional subgroup, and comparison schema.
Stored conditions serialize through
[`DataConditionSerializer`](../endpoints/serializers/data_condition_serializer.py).
Comparison dictionary keys are converted to the API's camelCase representation.
If `validate_comparison` resolves a user, team, integration, or other organization-owned
object, scope the lookup by the supplied organization. The generic validator does not
make an unscoped handler lookup safe.
### 7. Add focused backend tests
Add `tests/sentry/workflow_engine/handlers/condition/test_my_new_condition_handler.py`,
using [`ConditionTestCase`](../../../../tests/sentry/workflow_engine/handlers/condition/test_base.py).
Cover:
- Passing and non-passing values
- Exact result and `.triggered` behavior
- Missing or malformed runtime data
- Comparison JSON Schema acceptance and rejection
- Known failure conversion when using `DataConditionEvaluationException`
- Event versus activity behavior when relevant
- Organization scoping when `validate_comparison` or `render_label` loads objects
- Registry discovery through normal application startup
Add an explicit registration assertion without importing the implementation module in
the test:
```python
handler = condition_handler_registry.get(Condition.MY_NEW_CONDITION)
assert handler.__name__ == "MyNewConditionHandler"
```
Then test the placement:
- **Detector trigger:** create a detector condition with a priority result and assert the
detector transition or `DetectorEvaluation` in
[`test_detector.py`](../../../../tests/sentry/workflow_engine/processors/test_detector.py)
or the owning product's handler test. If the condition needs a new extracted value,
update and test the owning `DetectorHandler.extract_value` implementation.
- **Workflow WHEN:** assert the Workflow stops or proceeds in
[`test_workflow.py`](../../../../tests/sentry/workflow_engine/processors/test_workflow.py).
- **Workflow IF:** in the same processor test, create the
`WorkflowDataConditionGroup`/`DataConditionGroupAction` graph and assert only actions
attached to passing groups are selected.
- **Slow Workflow WHEN or IF:** extend
[`test_delayed_workflow.py`](../../../../tests/sentry/workflow_engine/processors/test_delayed_workflow.py)
for query generation, WHEN gating or IF selection, preservation of passing IF groups,
result combination, retry behavior, and cleanup.
Detector-specific comparison validation belongs in the owning subclass of
[`BaseDetectorTypeValidator`](../endpoints/validators/base/detector.py). The generic
condition handler alone cannot add detector packet data or detector-specific API fields.
### 8. Add optional compatibility and frontend support
These are separate decisions, not part of backend registration.
Production code no longer translates legacy issue-alert conditions. Test factories still do,
through `data_condition_translator_mapping` in
[`testutils/helpers/issue_alert_conditions.py`](../../testutils/helpers/issue_alert_conditions.py).
For the automation builder, add the matching frontend enum and node implementation in a
separate frontend PR:
- [`static/app/types/workflowEngine/dataConditions.tsx`](../../../../static/app/types/workflowEngine/dataConditions.tsx)
- [`static/app/views/automations/components/dataConditionNodes.tsx`](../../../../static/app/views/automations/components/dataConditionNodes.tsx)
The backend comparison schema does not generate the complete editor; current conditions
have explicit frontend nodes, defaults, details, and validation.
## File Map
| Change | Files to inspect or edit |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| Add condition enum | `models/data_condition.py:Condition` |
| Add shared handler | `handlers/condition/_handler.py` |
| Register at startup | Handler decorator and `handlers/condition/__init__.py` |
| Add product-owned handler | Product handler package and owning app's startup import |
| Change evaluation semantics | `models/data_condition.py`, `processors/data_condition_group.py`, evaluation classes |
| Change Detector consumption | `handlers/detector/`, especially `stateful.py` |
| Change Workflow WHEN/IF consumption | `models/workflow.py`, `processors/workflow.py` |
| Add slow condition | `SLOW_CONDITIONS`, `event_frequency_query_handlers.py`, delayed query/result code, delayed processor tests |
| Change API validation | `endpoints/validators/base/data_condition.py` and group/Detector/Workflow validators |
| Change availability metadata | `endpoints/organization_data_condition_index.py` and handler serializer |
| Add legacy test translation | `src/sentry/testutils/helpers/issue_alert_conditions.py` |
| Add automation UI | Frontend condition enum, node registry, editor/details components, frontend tests |
## Conventions and Invariants
| Rule | Enforcement |
| ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| Enum value, decorator key, stored `type`, and frontend enum use the same snake_case string | Convention only across backend/frontend |
| A handler is imported before validation or evaluation | Enforced only by startup wiring and tests |
| Handler placement matches `handler.group` | Convention/UI discovery only; generic backend validation does not enforce it |
| Handler `subgroup` changes only UI organization | UI serialization/rendering; evaluation does not inspect it |
| Detector conditions produce valid priorities | Enforced by Detector API validators; not fully enforced by direct model saves |
| Workflow conditions use gate-like results, conventionally stored `True` | Convention; runtime checks non-`None`, not boolean type |
| Detector trigger conditions are fast | Convention required by `DetectorHandler` evaluation; remaining slow work is discarded |
| Comparison and result schemas reject unknown shapes | Enforced only when handlers define strict schemas and callers use API validation |
| Organization-owned IDs are scoped in `validate_comparison` and `render_label` | Handler responsibility; not inferred by the framework |
| Group-role records are not reused across Detector, WHEN, and IF relations | Convention; no cross-role database constraint |
| Slow/percent/trigger/legacy behavior is added to the corresponding manual list | Convention plus focused tests |
| Actions are interpreted through `.triggered`, not result truthiness | Runtime contract in evaluation objects |
| Handler generic input matches the value passed at its placement | Static annotation and convention; runtime dispatch does not check it |
| Direct non-boolean handler results use the expected semantic type | Static annotation and focused tests; runtime accepts the broader result union |
---
### Src Sentry Workflow Engine Docs Data Model (`src/sentry/workflow_engine/docs/data-model.md`)
# Workflow Engine Data Model
This document describes the persistent graph used by the Workflow Engine. It focuses
on relationships, ownership, and invariants rather than repeating every Django field.
For runtime control flow, see [Execution](execution.md).
## Conceptual Graph
```mermaid
erDiagram
ORGANIZATION ||--o{ DATA_SOURCE : owns
PROJECT |o--o{ DETECTOR : owns
DATA_SOURCE ||--o{ DATA_SOURCE_DETECTOR : maps
DETECTOR ||--o{ DATA_SOURCE_DETECTOR : evaluates
DETECTOR ||--o| DETECTOR_TRIGGER_GROUP : owns
DETECTOR_TRIGGER_GROUP ||--o{ DETECTOR_CONDITION : contains
DETECTOR ||--o{ DETECTOR_WORKFLOW : connects
WORKFLOW ||--o{ DETECTOR_WORKFLOW : connects
ORGANIZATION ||--o{ WORKFLOW : owns
WORKFLOW ||--o| WHEN_GROUP : owns
WHEN_GROUP ||--o{ WHEN_CONDITION : contains
WORKFLOW ||--o{ WORKFLOW_DATA_CONDITION_GROUP : has
IF_GROUP ||--o| WORKFLOW_DATA_CONDITION_GROUP : connects
IF_GROUP ||--o{ IF_CONDITION : contains
IF_GROUP ||--o{ DATA_CONDITION_GROUP_ACTION : gates
ACTION ||--o{ DATA_CONDITION_GROUP_ACTION : executes
DETECTOR ||--o{ DETECTOR_STATE : tracks
DETECTOR ||--o{ DETECTOR_GROUP : creates
GROUP ||--o| DETECTOR_GROUP : associates
WORKFLOW ||--o{ WORKFLOW_ACTION_GROUP_STATUS : throttles
ACTION ||--o{ WORKFLOW_ACTION_GROUP_STATUS : throttles
GROUP ||--o{ WORKFLOW_ACTION_GROUP_STATUS : throttles
WORKFLOW ||--o{ WORKFLOW_FIRE_HISTORY : records
GROUP ||--o{ WORKFLOW_FIRE_HISTORY : records
DETECTOR |o--o{ WORKFLOW_FIRE_HISTORY : attributes
```
The diagram is conceptual. Some relationships use join models rather than direct
foreign keys, and compatibility models are omitted from the main graph. The abbreviated
names are condition roles, not model names: `WHEN_GROUP`/`WHEN_CONDITION` correspond to
the `DataConditionGroup`/`DataCondition` on `Workflow.when_condition_group`;
`IF_GROUP`/`IF_CONDITION` to those on `WorkflowDataConditionGroup.condition_group`;
`DETECTOR_TRIGGER_GROUP` to `Detector.workflow_condition_group`.
## Detection Graph
### `DataSource`
[`DataSource`](../models/data_source.py) belongs to an organization and generically
references a product model that emits detector input. The pair `(type, source_id)` is
globally unique; the organization is not part of that database constraint.
`type` selects a registered
[`DataSourceTypeHandler`](../types.py). The handler provides behavior that cannot be
expressed by the generic reference alone:
- Bulk-loading the referenced product objects
- Deletion relationships
- Organization quota calculation
- Relocation model naming
`source_id` is text because source models do not share a primary-key type. Callers must
use the registered handler instead of assuming which model it references.
Known examples include Snuba query subscriptions, uptime subscriptions, and cron
monitors. Adding a source type requires both registration and startup import coverage.
### `DataSourceDetector`
[`DataSourceDetector`](../models/data_source_detector.py) is the many-to-many mapping
between sources and detectors. `(data_source, detector)` is unique.
One source can feed multiple detectors. One detector can also be connected to source
records according to the owning product's model. Detector lookup by source is cached,
so mutations must preserve the receiver-driven invalidation path.
### `Detector`
[`Detector`](../models/detector.py) is the configured unit of detection. Important
relationships and fields are:
| Field or relationship | Meaning |
| -------------------------- | ------------------------------------------------------------------------------------------ |
| `project` | Owning project for normal detectors; nullable for organization-wide issue-stream detectors |
| `type` | Slug of an Issue Platform `GroupType` |
| `workflow_condition_group` | Trigger conditions that map input to priority levels |
| `config` | Detector-type-specific settings validated against `DetectorSettings.config_schema` |
| `enabled` | User-controlled active/snoozed state |
| `status` | Lifecycle state, including pending deletion and plan-level disabling |
The detector type is registered outside this Django app, in two parts. A concrete
[`GroupType`](../../issues/grouptype.py) defines the issue type. A
[`DetectorSettings`](../types.py) subclass, registered in
[`detector_settings_registry`](../registry.py) under the group type's slug, can define:
- Runtime handler
- API validator
- Configuration JSON schema
- Query filter for detector visibility
`GroupType.detector_settings` and `Detector.settings` read from the registry.
`Detector` does not enforce a database uniqueness constraint for project and type.
Default detector creation uses short distributed locks and idempotent lookups instead.
### Detector trigger conditions
The detector's `workflow_condition_group` points to a
[`DataConditionGroup`](../models/data_condition_group.py). The relation is nullable and
unique independently of the Workflow condition relations.
The field name is historical: this group belongs to detector evaluation even though it
is named `workflow_condition_group`.
## Conditions
### `DataCondition`
[`DataCondition`](../models/data_condition.py) contains:
| Field | Meaning |
| ------------------ | ---------------------------------- |
| `type` | Stored condition type |
| `comparison` | JSON comparison/configuration |
| `condition_result` | JSON result configured for a match |
| `condition_group` | Required parent group |
### `DataConditionGroup`
[`DataConditionGroup`](../models/data_condition_group.py) owns an organization and a
stored logic type. Groups have three intended roles:
- Detector trigger group
- Workflow WHEN group
- Workflow IF/action-filter group
Role exclusivity is not enforced across these relations, and generic validators do not
enforce handler placement. Creation code must keep the roles separate by convention.
See [Conditions](conditions.md) for the canonical evaluation, validation, and placement
contracts.
## Workflow Graph
### `DetectorWorkflow`
[`DetectorWorkflow`](../models/detector_workflow.py) connects detectors to workflows.
The connection determines which workflows are candidates for an issue event associated
with that detector.
An issue event can select more than one detector for lookup:
- Its specific detector, when known
- The project's issue-stream detector
- An optional organization-wide all-projects issue-stream detector
This allows workflows to target a product-specific detector or the broader issue stream.
### `Workflow`
[`Workflow`](../models/workflow.py) is organization-scoped. Important fields are:
| Field or relationship | Meaning |
| ---------------------- | ------------------------------------------------------- |
| `when_condition_group` | Optional trigger group evaluated before action filters |
| `environment` | `NULL` for all environments or one specific environment |
| `config.frequency` | Repeat suppression interval in minutes |
| `enabled` | Whether the workflow is eligible to run |
### `WorkflowDataConditionGroup`
[`WorkflowDataConditionGroup`](../models/workflow_data_condition_group.py) connects one
workflow to one IF/action-filter condition group. A condition group can belong to only
one workflow through this relationship.
### `DataConditionGroupAction` and `Action`
[`DataConditionGroupAction`](../models/data_condition_group_action.py) connects actions
to IF groups. [`Action`](../models/action.py) stores the action type and its handler-
specific configuration and data.
The action type selects a handler from
[`action_handler_registry`](../registry.py). Handlers are commonly implemented in the
owning integration or notification package rather than in `workflow_engine`.
An action can be reachable from more than one Workflow. The data model does not define
action execution order.
## Runtime State and History
### `DetectorState`
[`DetectorState`](../models/detector_state.py) stores durable detector status for a
detector and optional [`DetectorGroupKey`](../types.py). The group key lets one detector
track independent state for multiple values in a packet, such as separate endpoints.
The unique constraint treats a null group key as an empty value so that a non-grouped
detector has one state row.
Detector state is split across stores:
- PostgreSQL stores the durable priority and triggered status.
- Redis stores dedupe watermarks and priority threshold counters with a TTL.
The split is part of the runtime contract. For `StatefulDetectorHandler`, dedupe values
must be positive integers that increase with source order. The watermark defaults to
zero when absent and expires from Redis after seven days, so duplicate suppression is
not durable beyond that TTL.
### `DetectorGroup`
[`DetectorGroup`](../models/detector_group.py) associates an Issue Platform
[`Group`](../../models/group.py) with the detector that produced or owns it. The detector
foreign key can become null after deletion so existing issue history remains valid.
The association is normally created during Issue Platform ingestion. Compatibility
paths can backfill missing associations. Callers resolving a detector from an issue must
handle deleted or missing detector records.
### `WorkflowActionGroupStatus`
[`WorkflowActionGroupStatus`](../models/workflow_action_group_status.py) records when a
`(workflow, action, group)` tuple last passed repeat-frequency filtering. It is updated
before event-level deduplication and task execution, so it does not prove that an
external action completed.
### `WorkflowFireHistory`
[`WorkflowFireHistory`](../models/workflow_fire_history.py) stores a Workflow, issue
group, event ID, notification UUID, and optional detector. It has no action or delivery
outcome field. Rows are created before inactive-action filtering and event-level
deduplication, so they record pre-dispatch intent rather than a scheduled or successfully
delivered action.
## Compatibility Models
Compatibility association models map legacy issue and metric alert records to Detector,
Workflow, condition, action, and Issue Platform records. They are not part of normal
Detector or Workflow evaluation. See [Legacy alert API compatibility](legacy_backport.md)
for the model list and current mixed read/write behavior.
## Lifecycle and Invalidation
### Creation
Detector model API creation is coordinated by
[`BaseDetectorTypeValidator`](../endpoints/validators/base/detector.py). In one
transaction it creates or connects the detector's condition graph, product-specific
source object, generic data source, source mapping, and workflows.
Default detectors and workflows are created by [`defaults/`](../defaults) from project
and organization signal receivers. These functions must remain idempotent.
### Updates
Updates to detectors, workflows, conditions, source mappings, or action mappings can
invalidate cached processing graphs. Receivers in [`receivers/`](../receivers) generally
schedule invalidation with `transaction.on_commit()` so a cache cannot be repopulated
from uncommitted data.
Bypassing validators or relationship models can skip schema validation, audit logging,
or cache invalidation. Prefer the existing validator and service paths.
### Deletion
Detector API deletion marks the detector pending deletion and schedules a cell deletion
task. Related Issue Platform groups and historical fire records may outlive the detector.
Runtime lookup must therefore tolerate missing records.
Action cleanup for control-silo integration changes goes through the hybrid-cloud
[`ActionService`](../service/action/service.py).
## Scope and Relocation
Workflow Engine models live in the cell silo. Models participate in Sentry's relocation
framework according to their declared scopes. `DataSource` requires special
normalization because its generic `source_id` must be remapped using the registered
source handler's model name.
Do not infer organization ownership only through an arbitrary related ID. API lookups
must be explicitly scoped to the organization and permitted projects.
## Recommended Tests
- [`test_organization_project_detector_index.py`](../../../../tests/sentry/workflow_engine/endpoints/test_organization_project_detector_index.py)
demonstrates transactional detector graph creation.
- [`test_detectors.py`](../../../../tests/sentry/workflow_engine/defaults/test_detectors.py)
covers default creation and locking.
- [`test_workflows.py`](../../../../tests/sentry/workflow_engine/defaults/test_workflows.py)
covers default workflow graphs.
- [`receivers/`](../../../../tests/sentry/workflow_engine/receivers) covers validation and
cache invalidation.
- [`project_transfer.py`](../processors/project_transfer.py) contains project-transfer
cloning and reconnection helpers; these currently have no dedicated processor test.
---
### Src Sentry Workflow Engine Docs Execution (`src/sentry/workflow_engine/docs/execution.md`)
# Workflow Engine Execution
This document traces data through the Workflow Engine processors and asynchronous
boundaries. For persistent relationships, see [Data model](data-model.md). For an
implementation checklist, see [Adding detectors](adding-detectors.md).
## Two Pipelines
Detection and workflow processing are connected through Issue Platform, not by a direct
function call. Detector output is published to Issue Platform; the resulting issue event
later enters Workflow processing through post-processing or an activity. The overview
diagram is in the [module README](../README.md).
Existing issue events can enter at post-processing without first running a Workflow
Engine detector. This is how error and issue-stream workflows participate in the same
workflow pipeline.
## Detection Pipeline
### 1. Build a packet
A producer creates a typed [`DataPacket`](../models/data_source.py):
```python
DataPacket(source_id=str(source.id), packet=product_payload)
```
The packet's payload remains product-specific. The producer also supplies the source
type used to resolve `source_id`.
### 2. Resolve detectors
[`process_data_packet`](../processors/data_packet.py) is the generic entry point. It
calls [`process_data_source`](../processors/data_source.py), which resolves:
```text
(source type, source_id)
-> DataSource
-> DataSourceDetector
-> Detector rows with enabled=True through the default manager
```
The lookup uses the detector-by-source cache and preloads trigger conditions. Cache
invalidation is connected to source, detector, and mapping updates.
The default manager excludes pending-deletion and deletion-in-progress rows, but this
path does not exclude every non-active status such as plan-level `DISABLED`.
Not every producer uses this generic lookup. Uptime, processing errors, and preprod
paths can identify detectors through product-specific data and call
[`process_detectors`](../processors/detector.py) directly. The handler contract is the
same after detector selection.
### 3. Evaluate each detector
`process_detectors` obtains `detector.detector_handler` from the `DetectorSettings`
registered for the detector's type in the `detector_settings_registry`. Then, it calls the `_evaluate` method on the handler,
which delegates to the default or overridden `evaluate` methods that contain most of the detector's logic.
One packet can produce:
- No evaluation result
- One result for an ungrouped detector
- Multiple results keyed by `DetectorGroupKey`
A handler returns a mapping of group keys to `DetectorEvaluation` objects. An evaluation
can contain an `IssueOccurrence`, a `StatusChangeMessage`, or `None`. `process_detectors`
routes non-null results according to the handler's `outcome`.
### 4. Detector orchestration
Every detector handler inherits [`DetectorHandler`](../handlers/detector/condition.py). Its
default `evaluate` is stateless; `StatefulDetectorHandler` replaces it with durable
state and thresholds.
#### Stateless (default)
```mermaid
flowchart TD
Packet[Receive DataPacket] --> Extract[Extract evaluation values]
Extract --> Group[Normalize to group key and value pairs]
Group --> Conditions[Evaluate detector condition group]
Conditions --> Priority[Select highest triggered priority]
Priority --> Ok{Priority is OK?}
Ok -->|Yes| Stop[No Issue Platform output]
Ok -->|No| Occurrence[Build IssueOccurrence]
```
Important semantics:
- Each packet is evaluated independently. There is no dedupe, threshold, or durable
state.
- `OK` produces no status-change message; the default path never resolves issues.
#### Stateful
Most threshold-based detectors inherit
[`StatefulDetectorHandler`](../handlers/detector/stateful.py):
```mermaid
flowchart TD
Packet[Receive DataPacket] --> Extract[Extract dedupe value and evaluation values]
Extract --> Group[Normalize to group key and value pairs]
Group --> Load[Bulk-load durable and Redis state]
Load --> Newer{Dedupe value is newer?}
Newer -->|No| Skip[Skip stale value]
Newer -->|Yes| Conditions[Evaluate detector condition group]
Conditions --> Priority[Select highest passing priority]
Priority --> Counters[Update threshold counters]
Counters --> Transition{Priority state changed?}
Transition -->|Non-OK| Occurrence[Build IssueOccurrence]
Transition -->|OK| Resolution[Build StatusChangeMessage]
Transition -->|No| Persist[No Issue Platform output]
Occurrence --> Persist[Persist state, counters, and dedupe]
Resolution --> Persist
```
The handler performs state I/O in bulk for grouped values. The exact transaction and
Redis pipeline behavior is implemented by `DetectorStateManager` in the same module.
Important semantics:
- Dedupe values must be positive integers that increase in event order. Equal or smaller
values are ignored while the Redis watermark exists. The watermark defaults to zero
when absent and expires after seven days.
- State is independent for each detector group key.
- Reaching a higher priority also increments applicable lower-priority counters.
- Thresholds describe how many qualifying evaluations, according to the priority
counter rules, are needed before transition.
- A transition to a non-OK priority creates an occurrence.
- A transition to `OK` creates a status-change message.
- No state transition means no Issue Platform message.
Failing every non-OK condition does not infer recovery because a non-triggered group
produces no transition. Recovery requires a triggered group whose selected priority
remains `OK`. This normally comes from a passing condition with an `OK` result, but an
empty or passing `NONE` group can also trigger without a priority-bearing result and
therefore use the stateful handler's default `OK` priority. A missing trigger group is
invalid and produces no transition.
Detectors that need a different flow inherit `DetectorHandler` and override `evaluate`,
but then they own this orchestration. `BaseDetectorHandler` is only the abstract
interface and is not a base for new detectors.
### 5. Produce detector output
`process_detectors` invokes the detector handler's `on_complete` lifecycle callback with
the detector and each evaluation. The default callback is
`DetectorOutcome.ISSUE_PLATFORM.dispatch`; it resolves the registered outcome handler,
whose `handle` method ignores evaluations without a result and passes the others to Issue
Platform via [`produce_occurrence_to_kafka`](../../issues/producer.py). A detector can
assign another supported outcome dispatcher or override `on_complete` for custom processing.
Supported outcome implementations register their `DetectorOutcomeHandler` class with
`@detector_outcome_registry.add(DetectorOutcome.)`. The closed, typed set of keys
lives in `supported_outcomes.py`; the registry maps each key to its handler implementation.
Issue Platform ingestion creates or updates a group. The detector ID in occurrence
evidence allows ingestion to create a
[`DetectorGroup`](../models/detector_group.py) association. Resolution uses the stable
issue fingerprint to find the same group.
The Kafka boundary means detector evaluation succeeding does not imply that group
creation, workflow processing, or action dispatch has completed.
## Workflow Ingress
### Event post-processing
[`process_workflow_engine`](../../tasks/post_process.py) schedules the
[`process_workflows_event`](../tasks/workflows.py) task only for non-reprocessed jobs
with an event whose group is currently unresolved. The task rebuilds a
[`WorkflowEventData`](../types.py) from EventStore and Issue Platform data using
[`tasks/utils.py`](../tasks/utils.py). Resolution-driven processing enters through
supported issue activities instead.
### Issue activities
Issue activities invoke handlers through
[`invoke_workflow_activity_handlers`](../handlers/registry/invoke_activities.py).
Registered handlers receive an optional detector ID. A handler that schedules
[`process_workflow_activity`](../tasks/workflows.py) must first resolve a concrete
detector and pass its required detector ID with the activity and group.
Event and activity paths use `WorkflowEventData`, which contains:
- A `GroupEvent` or `Activity`
- The issue `Group`
- Optional group state and escalation information
- Optional workflow environment
- A per-event local cache for repeated lookups
They do not have identical scheduling. Activity processing has no workflow environment,
so it considers only global workflows. Activities do not enter batched condition data
acquisition; an activity whose WHEN or IF result still needs a slow query does not
continue through delayed processing.
## Workflow Pipeline
[`process_workflows`](../processors/workflow.py) is the main synchronous orchestrator
inside the workflow task.
```mermaid
flowchart TD
Event[WorkflowEventData] --> Detectors[Resolve event and issue-stream detectors]
Detectors --> Environment[Resolve event environment]
Environment --> Workflows[Load connected enabled workflows]
Workflows --> When[Evaluate WHEN groups]
When -->|False| Done[Record non-triggered evaluation]
When -->|True or slow result pending| Filters[Load and evaluate IF groups]
Filters -->|WHEN or IF slow conditions remain| Delay[Buffer delayed workflow]
Filters -->|WHEN true and IF groups pass| Select[Select attached actions]
Select --> Frequency[Apply repeat frequency]
Frequency --> History[Create fire history]
History --> Dedup[Filter inactive and deduplicate by get_dedup_key]
Dedup --> Tasks[Dispatch action tasks]
```
### 1. Resolve event detectors
[`get_detectors_for_event_data`](../processors/detector.py) builds `EventDetectors`:
- For a `GroupEvent` with an occurrence, `event_detector` is resolved only from the
occurrence's `detector_id` evidence; a missing or deleted ID yields no event detector.
For an ordinary error event with no occurrence, the project's default error detector
is used.
- `issue_stream_detectors` contains the project issue-stream detector and, when enabled,
the organization all-projects detector.
- `preferred_detector` is the event detector when available, otherwise the first
issue-stream detector.
The preferred detector is later supplied to actions that need detector context.
Detector deletion and older groups can leave associations missing, so lookup paths are
expected to degrade safely.
### 2. Resolve environment and workflows
The processor resolves the event environment, then
[`get_workflows_by_detectors`](../caches/workflow.py) loads workflows connected to any
selected detector. Eligible workflows must be enabled and either global
(`environment=None`) or connected to the event's environment.
The workflow cache is keyed by detector and environment. Relationship receivers
invalidate it after committed changes.
### 3. Evaluate WHEN conditions
[`evaluate_workflow_triggers`](../processors/workflow.py) evaluates each workflow's WHEN
group. A Workflow with no group passes. A conclusively false WHEN stops processing and
reports metrics/context; the returned per-Workflow result map can remain empty.
### 4. Evaluate IF groups
[`evaluate_workflows_action_filters`](../processors/workflow.py) loads IF groups from the
action-filter cache for workflows whose WHEN group passed or remains pending slow
evaluation. Fast IF results and remaining slow IF groups are retained with a pending
WHEN result so the combined work can be buffered.
Passing groups contribute their attached actions. A Workflow can therefore select
actions from multiple groups. A Workflow with no IF groups selects no actions.
The shared evaluation, fast/slow split, empty-group behavior, taint contract, and
Detector differences are documented once in [Conditions](conditions.md).
## Delayed Workflow Processing
The delayed path batches expensive conditions across workflows and issue groups.
```mermaid
sequenceDiagram
participant Immediate as Immediate workflow processor
participant Redis as Delayed workflow Redis
participant Scheduler as Cohort scheduler
participant Task as Delayed workflow task
participant Snuba as Snuba
participant Action as Action processor
Immediate->>Redis: Store event and remaining condition IDs
Scheduler->>Redis: Select due project cohorts
Scheduler->>Task: Schedule project or project batch
Task->>Redis: Load buffered items
Task->>Snuba: Execute bulk queries
Snuba-->>Task: Results by issue group
Task->>Action: Fire passing groups
Task->>Redis: Delete successfully processed items
Note over Task,Snuba: Equivalent condition queries are collapsed into bulk queries;
results are combined with fast results before firing.
```
### Buffering
[`DelayedWorkflowClient`](../buffer/batch_client.py) stores project IDs in sharded sorted
sets and each project's work in a hash. A hash field is keyed by workflow, issue group,
delayed WHEN and IF groups, and IF groups that already passed. Its value retains the
event or occurrence ID and original evaluation timestamp. Re-enqueuing the same field
overwrites that value, coalescing equivalent pending work rather than preserving every
event.
### Scheduling
The [`schedule_delayed_workflows`](../tasks/workflows.py) task acquires a global
scheduler lock around [`process_buffered_workflows`](../processors/schedule.py), which
divides projects into cohorts and schedules due work. Large project hashes are split into
batches so asynchronous task messages contain identifiers rather than large payloads.
### Evaluation
[`process_delayed_workflows`](../processors/delayed_workflow.py):
1. Parses buffered items.
2. Drops references to deleted workflows.
3. Loads remaining conditions.
4. Combines equivalent conditions into bulk queries.
5. Evaluates query results for each issue group.
6. Combines delayed results with fast results retained in the buffer key.
7. Reconstructs representative group events.
8. Sends passing action groups through the normal action path.
9. Deletes successfully processed Redis items.
Snuba rate limits are retried by the task. A missing per-group query result becomes a
tainted non-triggered evaluation. If a slow condition has no registered query handler,
no query is generated; processing can complete and remove the buffered item without
constructing that tainted result.
Malformed hash entries are excluded from the parsed event-key set and are not removed by
normal successful cleanup. They remain until expiry or manual cleanup.
## Action Selection and Dispatch
[`processors/action.py`](../processors/action.py) applies two different controls:
### Repeat frequency
[`WorkflowActionGroupStatus`](../models/workflow_action_group_status.py) tracks when each
workflow, action, and issue group tuple last passed the frequency check. It is updated
before event-level deduplication and task execution. A tuple is suppressed when it
passed within `Workflow.config.frequency` minutes, even if a later equivalent action was
deduplicated or an external handler failed.
### Event-level deduplication
`get_unique_active_actions` first removes actions whose status is not active, then keeps
one concrete action for each `Action.get_dedup_key()`. This is an implementation key,
not a guarantee that every semantically equivalent external side effect is recognized.
Invocation attribution comes from the concrete action retained for the key and does not
imply Workflow execution order.
### History and task dispatch
The processor updates frequency statuses and creates
[`WorkflowFireHistory`](../models/workflow_fire_history.py) rows before `fire_actions`
removes inactive actions and deduplicates by key. It then schedules
[`trigger_action`](../tasks/actions.py) for retained actions. Fire-history rows describe
scheduling, not successful external delivery, and can exist for actions that are later
filtered or deduplicated.
The action task:
1. Requires exactly one event ID or activity ID.
2. Rebuilds `WorkflowEventData`.
3. Loads the action and preferred detector.
4. Creates an [`ActionInvocation`](../types.py).
5. Calls the handler registered for the action type.
External integration calls therefore run outside the workflow evaluation task. Action
handlers must handle deleted integrations and other expected external-state changes.
## Processor Reference
| Processor | Input | Output | Side effects | Async boundary |
| ------------------------------ | -------------------------- | ------------------------------ | --------------------------------------------------------- | -------------------------- |
| `process_data_packet` | Packet and source type | Detector evaluations | Source cache reads, detector state, Kafka payloads | Called by product producer |
| `process_data_source` | Packet and source type | Packet and detectors | Detector-source cache reads | None |
| `process_detectors` | Packet and detector list | Detector evaluations | PostgreSQL/Redis state and Issue Platform Kafka | Kafka after evaluation |
| `process_data_condition_group` | Group and event/value | Evaluation and slow conditions | Metrics and logs | None |
| `process_workflows` | `WorkflowEventData` | Workflow evaluations | Buffer writes, fire history, status updates, action tasks | Asynchronous action tasks |
| `process_buffered_workflows` | Delayed client | None | Cohort metadata and delayed tasks | Asynchronous delayed tasks |
| `process_delayed_workflows` | Project and optional batch | None | Snuba queries, action state/history, buffer cleanup | Snuba and action tasks |
## Failure and Consistency Model
- Database graph mutations and cache invalidation are coordinated with
`transaction.on_commit()` where stale repopulation would be unsafe.
- Detector state and Issue Platform publication are not atomic. PostgreSQL and Redis
state commit before `process_detectors` publishes, so a failure in between can cause a
retry to be skipped by the committed dedupe watermark. See
[Conditions](conditions.md) for the commit ordering.
- Issue Platform publication is asynchronous. There is no transaction spanning detector
state, Kafka, group creation, workflow evaluation, and external actions.
- Workflow action frequency and deduplication reduce repeated side effects but do not
provide a distributed exactly-once guarantee for an external provider.
- Delayed items are removed after successful processing so task retries can recover
buffered work.
- Models referenced by queued work can be deleted before a task runs. Tasks and
handlers must treat expected missing records as normal lifecycle races.
## Caches
| Cache | Purpose | Approximate TTL |
| ------------------------------------------------------------- | ------------------------------------------- | --------------- |
| [`caches/detector.py`](../caches/detector.py) | Detectors connected to a data source | 20 minutes |
| [`caches/workflow.py`](../caches/workflow.py) | Workflows connected to detector/environment | 1 minute |
| [`caches/action_filters.py`](../caches/action_filters.py) | IF groups and actions for workflows | 5 minutes |
| Default detector cache in [`Detector`](../models/detector.py) | Default project/type lookup | 10 minutes |
Treat receivers as part of the model mutation contract. A new relationship that affects
processing usually needs explicit invalidation coverage.
Receivers invalidate supported current relationship keys after commit. Updating a
`DataSource` type/source ID or a Workflow environment in place does not retain and
invalidate every old cache key; stale entries can therefore survive until TTL expiry.
## Observability and Debugging
The engine emits metrics for detector lookup/evaluation, condition evaluation, workflow
processing, delayed scheduling, caches, and actions. Logging helpers in
[`utils/log_context.py`](../utils/log_context.py) attach workflow and event context.
[`emit_evaluations`](../processors/evaluations/tracking.py) records detector and workflow
evaluations through each `EvaluationEmitter`, which filters artifacts for its destination.
Runtime options and feature flags control sampling and direct logging.
When `organizations:workflow-engine-evaluation-artifacts-eap` is enabled,
[`EAPEmitter`](../processors/evaluations/eap.py) publishes artifacts to
`SNUBA_ITEMS` as `TRACE_ITEM_TYPE_WORKFLOW_ENGINE_EVALUATION` trace items. These are not
log trace items and do not include log body or severity attributes.
Delivery is best-effort: the producer does not track futures for task completion or apply
backpressure. Serialization and delivery failures are logged without retrying workflow
tasks. Artifacts can be dropped when Kafka is unavailable or its producer queue is full.
The sanitized `trigger_evaluation`, `filter_evaluations`, and `delayed` attributes are
JSON strings because EAP ingestion discards nested protobuf key/value lists. Scalar
attributes and primitive arrays, such as `triggered_action_ids`, retain their native types.
When tracing a missing action, check the boundaries in order:
1. Was the source mapped to an enabled detector?
2. Did detector state transition and publish an Issue Platform message?
3. Was the resulting issue associated with the expected detector?
4. Did post-processing or activity handling schedule workflow evaluation?
5. Was the workflow connected, enabled, and valid for the environment?
6. Did WHEN and at least one IF group pass?
7. Was a slow condition buffered and later processed?
8. Was the action suppressed by frequency or event-level deduplication?
9. Was fire history created and the action task scheduled?
10. Did the action handler reject missing or invalid external configuration?
## Recommended Tests
- [`test_integration.py`](../../../../tests/sentry/workflow_engine/test_integration.py)
covers the full detector and workflow paths.
- [`test_stateful.py`](../../../../tests/sentry/workflow_engine/handlers/detector/test_stateful.py)
covers state transitions and Redis behavior.
- [`test_detector.py`](../../../../tests/sentry/workflow_engine/processors/test_detector.py)
covers detector output and event detector selection.
- [`test_workflow.py`](../../../../tests/sentry/workflow_engine/processors/test_workflow.py)
covers workflow evaluation.
- [`test_data_condition_group.py`](../../../../tests/sentry/workflow_engine/processors/test_data_condition_group.py)
covers condition logic and fast/slow splitting.
- [`test_delayed_workflow.py`](../../../../tests/sentry/workflow_engine/processors/test_delayed_workflow.py)
covers delayed queries and action firing.
- [`test_schedule.py`](../../../../tests/sentry/workflow_engine/processors/test_schedule.py)
covers cohort scheduling and batches.
- [`test_actions.py`](../../../../tests/sentry/workflow_engine/tasks/test_actions.py)
covers action task reconstruction and dispatch.
---
### Src Sentry Workflow Engine Docs Legacy Backport (`src/sentry/workflow_engine/docs/legacy_backport.md`)
# Legacy Alert API Compatibility
> This document covers remaining legacy alert ID compatibility with Workflow Engine
> models after the legacy alert APIs were retired. Start with
> the [Workflow Engine overview](../README.md), [data model](data-model.md), and
> [execution guide](execution.md) for current architecture.
## Current Boundary
The deprecated legacy alert API routes were removed in
[sentry#121879](https://github.com/getsentry/sentry/pull/121879). Clients must use
the detector and workflow APIs to manage alerts. The serializer and issue-alert
POST/PUT rollout flags no longer have consumers and have been retired.
Some compatibility code remains: lookup endpoints expose associations between legacy and
Workflow Engine models, and notification code still uses legacy-shaped data. Endpoint retirement does
not imply that the association tables, serializers, or ID helpers are all unused.
## Handling IDs
### Dual-written data
Data that was created by the legacy system and migrated (or is being written to both systems simultaneously). Association tables bridge the two:
- `AlertRuleDetector` — despite the name, this maps either a metric alert (`alert_rule_id`) or an issue alert (`rule_id`) to a `Detector`. Each row carries exactly one of the two (enforced by a check constraint).
- `AlertRuleWorkflow` — same pattern, mapping `alert_rule_id` or `rule_id` to a `Workflow`.
- `IncidentGroupOpenPeriod` — maps a legacy `incident_id` / `incident_identifier` to a `GroupOpenPeriod`.
When the workflow engine path receives a real legacy ID (e.g. an `alertRule` query param), it resolves the corresponding workflow engine object via these tables.
### Single-written data
Data created exclusively by the workflow engine with no legacy counterpart. These objects have no rows in the association tables. To maintain API compatibility, they are exposed with manufactured IDs. The helpers for this live in `src/sentry/incidents/endpoints/serializers/utils.py`:
- `get_fake_id_from_object_id(obj_id)` — used by serializers to manufacture an ID for API responses
- `get_object_id_from_fake_id(fake_id)` — used by endpoints to recover the real object ID from an incoming parameter. If non-positive, the input wasn't a valid manufactured ID.
Endpoints that accept IDs as input must handle both real legacy IDs (via association tables) and manufactured IDs (via `get_object_id_from_fake_id`).
## Compatibility Endpoints
Check the registered routes and their implementations when changing compatibility
code. The association lookup routes (`alert-rule-workflow`, `alert-rule-detector`, and
`incident-groupopenperiod`) live in `src/sentry/workflow_engine/endpoints/urls.py`.
The endpoint-tracking decorator was removed with the deprecated routes.
## Unsupported legacy features
Some legacy features can't or won't be supported in workflow engine models (e.g. `AlertRule` snapshots). Acknowledge these explicitly in code and tests where appropriate to make it clear which differences are known and intentional and which may be bugs.
## Testing
- **Delta tests** compare old and new serializer output for dual-written data, with an explicit `known_differences` set documenting expected divergences.
- **Single-write tests** verify that workflow engine-only data (no legacy counterpart) is returned correctly.
- **Filter tests** verify ID-based filters work for both real legacy IDs (via association tables) and manufactured IDs (via extraction).
---
--- METRICS ---
- Files Extracted: 7
- Estimated Token Budget: ~29824 tokens
- Canonical Reference: https://codewiki.google/github.com/getsentry/sentry