{"owner":"stakater","repo":"Reloader","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md"],"skills":{"CLAUDE.md":"# Stakater Reloader Project Memory\n\n## Project Purpose\n\nReloader is a Kubernetes operator that automatically triggers rolling restarts of workloads when the ConfigMaps or Secrets they reference are updated. Without it, Kubernetes does not restart pods when configuration changes — operators must do it manually or rely on GitOps pipelines.\n\n**What it watches**: ConfigMaps, Secrets, Namespaces, and (optionally) `SecretProviderClassPodStatus` (CSI-mounted secrets).\n\n**Workload types it can reload**: Deployment, StatefulSet, DaemonSet, CronJob, Job, Argo Rollout, and OpenShift DeploymentConfig.\n\n**How restarts are triggered**: Two strategies (selected via `--reload-strategy`):\n1. **env-vars** (default) — injects an environment variable (`STAKATER_{NAME}_{TYPE}`) into every container with the SHA1 hash of the resource's data. A change in data changes the env var value, causing Kubernetes to restart pods.\n2. **annotations** — writes the SHA1 hash into the pod template's annotations, which also forces a rollout.\n\n**The core problem it solves**: ConfigMaps and Secrets are decoupled from pod lifecycle in Kubernetes. Applications reading config at startup see stale data after a config update unless pods are restarted. Reloader closes that gap automatically and selectively.\n\n**Potential improvements observed**:\n- **Duplicate reload suppression**: If a workload references both a ConfigMap and a Secret that are updated in the same controller reconcile cycle, it may get reloaded twice. Could be solved with a per-workload debounce map keyed by namespace/name/resourceVersion, flushed after a short TTL.\n- **CronJob/Job reload is destructive**: Jobs are deleted and recreated on change, which loses run history. Could instead only annotate the CronJob template without spawning a new Job.\n- **No per-resource reload rate limiting**: A rapid-fire ConfigMap update (e.g., from a CI pipeline) can trigger many restarts. A cooldown window per resource would help.\n- **CSI integration gap**: CSI volumes are watched at the `SecretProviderClassPodStatus` level, but the link back to the workload is indirect and may miss edge cases. Needs a direct map from SecretProviderClass → workloads that mount it.\n\n---\n\n## Repo Map\n\n| Path | Owns | Inspect when |\n|---|---|---|\n| `main.go` | Entry point, delegates to `app.Run()` | Never needs changes |\n| `internal/pkg/app/` | `Run()` bootstrap, Cobra command wiring | Startup sequence changes |\n| `internal/pkg/cmd/` | CLI flags parsing, `startReloader()`, controller/HA wiring | Adding new flags or startup behavior |\n| `internal/pkg/controller/` | Informer/queue per resource type, event handlers (Add/Update/Delete) | Watching new resource types, queue tuning |\n| `internal/pkg/handler/` | Per-event handlers (create, update, delete), `doRollingUpgrade()`, pause deployment | Core reload logic changes |\n| `internal/pkg/callbacks/` | Workload-specific get/list/update/patch functions, `RollingUpgradeFuncs` struct | Adding new workload types |\n| `internal/pkg/options/` | All CLI flag variables, defaults, `ArgoRolloutStrategy` type | Adding or renaming flags |\n| `internal/pkg/constants/` | Constants: env var postfixes, annotation prefix, strategy names, HA lock name | Renaming global identifiers |\n| `internal/pkg/metrics/` | Prometheus `Collectors` struct, all metric registration and recording helpers | Adding metrics |\n| `internal/pkg/alerts/` | Slack/Teams/GChat/raw webhook alerting, env var config | Alert sink changes |\n| `internal/pkg/util/` | SHA generation via `crypto/sha.go`, env var name conversion, namespace/label utilities | Utility/hash changes |\n| `internal/pkg/crypto/` | `GenerateSHA(data)` — SHA1 hex digest | Hash algorithm changes |\n| `internal/pkg/leadership/` | Leader election via Kubernetes Lease, HA stop/start of controllers | HA behavior changes |\n| `internal/pkg/testutil/` | Fake Kubernetes objects for unit tests | Writing new tests |\n| `pkg/common/` | `ReloadCheckResult`, `ReloaderOptions`, `ShouldReload()` logic, `Config` struct | Reload decision logic, annotation precedence |\n| `pkg/kube/` | `Clients` struct (k8s + OpenShift + Argo + CSI), `GetKubernetesClient()`, `ResourceMap` | Client initialization, new CRD clients |\n| `deployments/` | Helm chart (`deployments/kubernetes/chart/reloader/`), Kustomize manifests | Helm values, RBAC, deployment config |\n| `docs/` | User-facing annotation documentation, architecture notes | Writing docs or confirming annotation behavior |\n| `scripts/` | Shell scripts used by CI and Makefile | Build/release pipeline |\n| `test/loadtest/` | Load test CLI (`cmd/loadtest`), 13 scenarios (S1–S13), Kind cluster setup | Performance testing, regression benchmarks |\n| `.github/` | CI workflows: lint, test, Kind e2e, multi-arch Docker build, release | CI changes |\n\n---\n\n## Core Runtime Flow\n\n**1. Entry** — `main.go:10` calls `app.Run()`.\n\n**2. CLI Init** — `internal/pkg/app/app.go` calls `cmd.NewReloaderCommand()` which registers all Cobra flags from `options/flags.go` and runs `startReloader()`.\n\n**3. Client Setup** — `pkg/kube/client.go`: builds `kube.Clients` with:\n- `kubernetes.Interface` — standard k8s client\n- `appsclient.Interface` — OpenShift client (auto-detected by probing `deploymentconfigs`)\n- `argorollout.Interface` — if `--is-Argo-Rollouts=true`\n- `csiclient.Interface` — if `--enable-csi-integration`\n\n**4. Controller Creation** — `startReloader()` iterates `kube.ResourceMap` (configmaps, secrets, namespaces, and optionally secretproviderclasspodstatuses) and calls `controller.NewController()` for each resource in each watched namespace.\n\n**5. Informer/Queue** — `controller.NewController()`:\n- Creates a `cache.NewFilteredListWatchFromClient` with label/field selectors.\n- Registers `Add`, `Update`, `Delete` event handlers.\n- Creates a `workqueue.TypedRateLimitingQueue` for async processing.\n\n**6. Event Detection**:\n- `Add` — enqueues only if `ReloadOnCreate` is enabled (skips during initial sync unless `SyncAfterRestart`).\n- `Update` — compares SHA of old vs new object data; enqueues only on real changes.\n- `Delete` — enqueues only if `ReloadOnDelete` is enabled.\n- Namespace events update `selectedNamespacesCache` for namespace-selector filtering.\n\n**7. Handler Dispatch** — The queue worker calls `handler.Handle()` on the dequeued item. Three handler types:\n- `ResourceCreatedHandler` (`create.go`) — fires `doRollingUpgrade` or sends webhook.\n- `ResourceUpdatedHandler` (`update.go`) — fires `doRollingUpgrade` or sends webhook.\n- `ResourceDeleteHandler` (`delete.go`) — calls `invokeDeleteStrategy` (removes env vars or clears annotation).\n\n**8. Workload Discovery** — `doRollingUpgrade()` (`upgrade.go:181`) calls `rollingUpgrade()` for each workload type. For each type, `ItemsFunc` lists all workloads in the namespace, then `pkg/common.ShouldReload()` checks annotations to decide which ones need reloading.\n\n**9. Reload Execution** — `invokeReloadStrategy()` either:\n- **env-vars**: mutates container env vars; uses JSON patch if `SupportsPatch=true`, full update otherwise.\n- **annotations**: writes SHA to pod template annotations; same patch/update split.\n\n**10. Post-reload** — optionally pauses the Deployment via `pause_deployment.go`, records Kubernetes Events via `recorder`, updates Prometheus metrics, sends alert webhooks.\n\n**HA Mode**: if `--enable-ha`, `internal/pkg/leadership/` runs Kubernetes Lease-based leader election. Only the leader runs controllers; losing leadership stops them and marks the pod unhealthy.\n\n**HTTP Server**: port `:9090` serves `/metrics` (Prometheus) and liveness/readiness probes.\n\n---\n\n## Reload Behavior And Annotations\n\nAll annotation names are configurable via CLI flags; the values below are defaults.\n\n### Trigger Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any** ConfigMap or Secret referenced by the workload (via envFrom, env valueFrom, or volumes) |\n| `configmap.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced ConfigMap** only |\n| `secret.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced Secret** only |\n| `secretproviderclass.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced SecretProviderClass** only |\n| `configmap.reloader.stakater.com/reload` | `\"cm1,cm2\"` | Reload only when the **named ConfigMaps** change (regex supported) |\n| `secret.reloader.stakater.com/reload` | `\"sec1,sec2\"` | Reload only when the **named Secrets** change (regex supported) |\n| `secretproviderclass.reloader.stakater.com/reload` | `\"spc1\"` | Reload only when the **named SecretProviderClass** changes |\n| `reloader.stakater.com/search` | `\"true\"` | Reload when any ConfigMap/Secret tagged with `reloader.stakater.com/match: \"true\"` changes |\n\n### Exclude Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/ignore` | `\"true\"` | Skip this workload entirely |\n| `configmaps.exclude.reloader.stakater.com/reload` | `\"cm1,cm2\"` | Exclude these named ConfigMaps from triggering reload |\n| `secrets.exclude.reloader.stakater.com/reload` | `\"sec1,sec2\"` | Exclude these named Secrets |\n| `secretproviderclasses.exclude.reloader.stakater.com/reload` | `\"spc1\"` | Exclude these named SecretProviderClasses |\n\n### Behavior Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/rollout-strategy` | `\"restart\"` or `\"rollout\"` | For Argo Rollouts: `\"restart\"` uses restartAt, `\"rollout\"` (default) uses full rollout update |\n| `deployment.reloader.stakater.com/pause-period` | Go duration e.g. `\"30s\"` | Pause Deployment for this duration after reload |\n| `deployment.reloader.stakater.com/paused-at` | RFC3339 timestamp | Set by Reloader to track pause start time; do not set manually |\n\n### Search/Match Pattern\n\nThe `reloader.stakater.com/search` annotation on a workload pairs with `reloader.stakater.com/match: \"true\"` on a ConfigMap or Secret. Any workload with `search: true` will reload when any `match: true` resource changes.\n\n### Global Flag Overrides\n\n- `--auto-reload-all` — reload all workloads on any ConfigMap/Secret change; annotation not required.\n- `--resources-to-ignore=configMaps` or `=secrets` — skip one type entirely.\n- `--ignored-workload-types=jobs,cronjobs` — skip Job and CronJob reload.\n- `--namespaces-to-ignore` — comma-separated namespace names to skip.\n- `--namespace-selector` — only watch namespaces with matching labels.\n- `--resource-label-selector` — only watch ConfigMaps/Secrets with matching labels.\n\n### Precedence Rules\n\n1. `reloader.stakater.com/ignore: \"true\"` wins everything — workload is skipped.\n2. Exclude annotations override include annotations for specific named resources.\n3. Named annotations (`.../reload`) are checked before auto annotations.\n4. `--auto-reload-all` is the lowest-priority fallback (only applies if no annotation matches).\n5. Annotations are checked on both the workload and its pod template (pod template takes precedence in some paths — verify in `pkg/common/common.go:ShouldReload()`).\n\n---\n\n## Workload Support\n\n| Workload | SupportsPatch | Update Mechanism | Key files |\n|---|---|---|---|\n| **Deployment** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:38` |\n| **StatefulSet** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:109` |\n| **DaemonSet** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:91` |\n| **CronJob** | No | Creates a new Job from CronJob spec (adds `cronjob.kubernetes.io/instantiate: manual`) | `callbacks.CreateJobFromCronjob`, `handler/upgrade.go:55` |\n| **Job** | No | Deletes old Job, creates new one (strips ResourceVersion, UID, Status, controller labels) | `callbacks.ReCreateJobFromjob`, `handler/upgrade.go:73` |\n| **Argo Rollout** | No | Full update via Argo Rollouts client | `callbacks.UpdateRollout`, `handler/upgrade.go:127`; requires `--is-Argo-Rollouts=true` |\n| **DeploymentConfig** | Yes | OpenShift DeploymentConfigs API | `callbacks/rolling_upgrade.go`; auto-detected by probing `deploymentconfigs` |\n\n**Reload flow per workload**: `doRollingUpgrade()` → `rollingUpgrade()` per type → `ItemsFunc` lists workloads → `ShouldReload()` filters → `invokeReloadStrategy()` patches or updates → optional pause + metrics + alert.\n\n---\n\n## CSI Support\n\n**Enabled by**: `--enable-csi-integration`\n\n**What is watched**: `SecretProviderClassPodStatus` resources (from `sigs.k8s.io/secrets-store-csi-driver`). Resource name constant: `constants.SecretProviderClassController = \"secretproviderclasspodstatuses\"`.\n\n**How it works**:\n1. The CSI driver injects secrets into pods as volume mounts and tracks injection state via `SecretProviderClassPodStatus` objects.\n2. Reloader watches these objects for version changes.\n3. When a version change is detected, it computes a SHA of the object's IDs and versions.\n4. It then looks up the referenced `SecretProviderClass` and treats the event like a Secret update, triggering workload reloads.\n\n**Workload annotation**: `secretproviderclass.reloader.stakater.com/reload: \"my-spc\"` or `secretproviderclass.reloader.stakater.com/auto: \"true\"`.\n\n**Required**: CSI CRDs must be installed in the cluster. Reloader auto-detects their presence at startup.\n\n**Env var postfix**: `STAKATER_{NAME}_SECRETPROVIDERCLASS`.\n\n**Known limitations**:\n- Only works for secrets mounted as volumes via CSI, not env-var-based CSI injection.\n- The link from `SecretProviderClassPodStatus` → workload is indirect; edge cases may be missed.\n- Requires the CSI driver CRDs to be pre-installed; Reloader won't start CSI controller if CRDs are absent.\n\n---\n\n## Build, Test, And Run Commands\n\n**Go version**: `go 1.26.2` (from `go.mod`)\n\n| Purpose | Command |\n|---|---|\n| Run locally | `go run ./main.go` |\n| Build binary | `make build` → `go build -o Reloader` |\n| Unit tests | `make test` → `go test -timeout 1800s -v ./...` |\n| Lint | `make lint` → `golangci-lint run ./...` (v2.6.1) |\n| Docker build (single arch) | `make build-image ARCH=amd64` |\n| Docker push | `make push` |\n| Full release (build+push+manifest) | `make release ARCH=amd64` |\n| Multi-arch release | `make release-all` |\n| Generate k8s manifests | `make k8s-manifests` (Kustomize v5.3.0) |\n| Load test (quick) | `make loadtest-quick LOADTEST_OLD_IMAGE=... LOADTEST_NEW_IMAGE=...` (runs S1, S4, S6) |\n| Load test (full) | `make loadtest-full LOADTEST_OLD_IMAGE=... LOADTEST_NEW_IMAGE=...` |\n| Load test (custom) | `make loadtest LOADTEST_SCENARIOS=S1,S3 LOADTEST_DURATION=120` |\n\n**Docker image**: `ghcr.io/stakater/reloader` — multi-arch (amd64, arm64, arm), distroless nonroot base.\n\n**Helm chart**: `deployments/kubernetes/chart/reloader/` — install via Helm or `kubectl apply -f deployments/kubernetes/reloader.yaml`.\n\n---\n\n## Coding Conventions\n\n**Package boundaries**: Each `internal/pkg/<name>` package has a single clear responsibility. Cross-package access goes through exported types/functions only.\n\n**Error handling**: `logrus.Errorf(...)` for non-fatal, `logrus.Fatalf(...)` for startup failures. Errors are returned up the call stack and logged at the point of action, not at every layer. Retry uses `k8s.io/client-go/util/retry.RetryOnConflict`.\n\n**Logging**: `logrus` with structured fields. Format controlled by `--log-format=json` flag. Log level controlled by `--log-level`. Messages follow the pattern: `\"Changes detected in '%s' of type '%s' in namespace '%s'\"`.\n\n**Kubernetes client patterns**: All k8s operations go through the `kube.Clients` struct. Use `context.TODO()` for context (no request-scoped contexts). List/watch via informers, not polling.\n\n**Callback pattern**: Workload-specific logic is encapsulated in `callbacks.RollingUpgradeFuncs` structs returned by `handler.Get*RollingUpgradeFuncs()`. Adding a new workload type = add a new `RollingUpgradeFuncs` factory function and call it in `doRollingUpgrade()`.\n\n**Test style**: Standard `testing.T`, `testify/assert`. Fake k8s objects via `testutil/kube.go`. Tests live alongside source in the same package. Large integration-style tests in `handler/upgrade_test.go`.\n\n**Naming patterns**:\n- Annotation variables: `XxxUpdateOnChangeAnnotation`, `XxxReloaderAutoAnnotation`\n- Callback funcs: `GetXxxItem`, `GetXxxItems`, `UpdateXxx`, `PatchXxx`\n- Handler factories: `GetXxxRollingUpgradeFuncs()`\n\n**Adding new behavior**: Add flag to `options/flags.go` + `common.ReloaderOptions` struct → wire in `cmd/reloader.go` → implement logic in `handler/` or `callbacks/` → add metrics recording → write tests in `*_test.go`.\n\n---\n\n## Gotchas And Risks\n\n**Duplicate reloads**: If a workload references multiple ConfigMaps/Secrets and all change simultaneously, each change event fires a separate reload. No deduplication exists within a reconcile window. This can cause unnecessary rolling restarts.\n\n**Controller init guard**: `secretControllerInitialized` and `configmapControllerInitialized` booleans in `controller/controller.go` prevent processing Add events during the initial list/sync (to avoid reloading everything on startup). If `--sync-after-restart` is set, both are pre-set to `true`, bypassing the guard. Be careful when this interacts with `--reload-on-create`.\n\n**Namespace filtering**: `--namespaces-to-ignore` does a name match; `--namespace-selector` watches namespaces by label and caches them in `selectedNamespacesCache`. The cache is updated on Namespace Add/Update/Delete events. A race between cache population and first ConfigMap event could cause missed reloads on startup in label-selected deployments.\n\n**RBAC**: Reloader requires get/list/watch on secrets and configmaps, and get/list/watch/update/patch on all workload types it manages. Missing RBAC silently causes no reloads (not an error — just empty lists). Check ClusterRole in `deployments/kubernetes/chart/reloader/templates/`.\n\n**GitOps drift**: If a GitOps tool (Flux, ArgoCD) manages the same Deployments, annotation or env var changes made by Reloader will be detected as drift and reverted. Use `--reload-strategy=annotations` with care in GitOps setups; `env-vars` strategy is generally safer since it modifies the pod template rather than workload-level annotations.\n\n**Annotation precedence edge case**: Annotations are checked first on the workload object, then on the pod template. If both are set to conflicting values, the behavior depends on which path `ShouldReload()` hits first. Verify in `pkg/common/common.go`.\n\n**CronJob/Job destructive reload**: Job recreation deletes the old Job. Any in-flight pod from that Job will be terminated. This is intentional but surprising. There is no protection for long-running jobs.\n\n**OpenShift DeploymentConfig**: Auto-detected by probing for the `deploymentconfigs` resource. If the probe fails at startup, OpenShift support is silently disabled. Check `pkg/kube/client.go`.\n\n**Argo Rollouts**: Must be explicitly enabled via `--is-Argo-Rollouts=true`. Without it, Rollout objects are never listed. The `SupportsPatch=false` means full object updates are used — be aware of potential conflicts with Argo's own controller.\n\n**CSI rotation behavior**: `SecretProviderClassPodStatus` is updated by the CSI driver when secrets rotate. Reloader reacts to those updates. However, if the CSI driver updates the status in a way that doesn't change the versions Reloader tracks, the reload will be missed.\n\n**Backward compatibility**: Annotation names are configurable, so changing defaults would break existing clusters. Never change default annotation values without a migration path.\n\n**Tests to update for risky changes**: `handler/upgrade_test.go` (large suite covering all workload types), `controller/controller_test.go` (event handling), `pkg/common/common_test.go` (reload decision logic).\n\n---\n\n## Open Questions\n\n- **Exact `ShouldReload()` precedence**: The code in `pkg/common/common.go` checks annotations in a specific order. The exact tie-breaking when both workload-level and pod-template-level annotations are set should be verified by reading that function fully before making annotation behavior changes.\n- **CSI → workload mapping**: How exactly does Reloader map a `SecretProviderClassPodStatus` change back to workloads? Is it via the SecretProviderClass name matching an annotation on the workload, or via volume reference scanning? Needs confirmation before adding CSI-related features.\n- **`ContainerPatchPathFunc` field**: `RollingUpgradeFuncs` has a `ContainerPatchPathFunc` field, but it is not documented — unclear if/how it differs from `ContainersFunc` in patch scenarios.\n- **Webhook vs alert**: `--webhook-url` replaces reloading with a POST request. `ALERT_WEBHOOK_URL` env var sends an alert *after* reloading. These are two different mechanisms; the naming is confusing and easy to conflate.\n- **Load test scenarios S7–S13**: Only S1, S4, and S6 are confirmed from CI. The behavior and coverage of the remaining scenarios is unknown without reading `test/loadtest/` in full.\n- **`SyncAfterRestart` semantics**: Flag docs say it \"syncs add events after restart\" but only if `ReloadOnCreate` is also true. The interaction between these two flags in HA mode (where controllers restart on leader change) needs verification.\n\n---\n\n## Important Files\n\n| File | Description |\n|---|---|\n| `internal/pkg/cmd/reloader.go` | `startReloader()` — main wiring of clients, controllers, HA, and HTTP server |\n| `internal/pkg/handler/upgrade.go` | `doRollingUpgrade()` + all `Get*RollingUpgradeFuncs()` factories |\n| `internal/pkg/callbacks/rolling_upgrade.go` | All workload-specific get/update/patch implementations |\n| `pkg/common/common.go` | `ShouldReload()` — the annotation decision tree |\n| `internal/pkg/options/flags.go` | Every configurable option with defaults |\n| `internal/pkg/controller/controller.go` | Informer setup, queue, event handlers |\n| `pkg/kube/client.go` | Multi-client initialization and OpenShift/CSI detection |\n| `internal/pkg/handler/pause_deployment.go` | Pause/resume deployment logic with timers |\n| `internal/pkg/leadership/leadership.go` | HA leader election |\n| `internal/pkg/metrics/prometheus.go` | All Prometheus collector definitions |\n| `internal/pkg/alerts/alert.go` | Slack/Teams/GChat alerting |\n| `internal/pkg/constants/constants.go` | Global constants (env var prefixes, annotation prefix, strategy names) |\n| `deployments/kubernetes/chart/reloader/values.yaml` | Helm chart defaults — source of truth for production config |\n| `handler/upgrade_test.go` | Largest test suite; must be updated for any reload logic change |\n| `Makefile` | All build/test/release/loadtest commands |\n"},"files":{"CLAUDE.md":"# Stakater Reloader Project Memory\n\n## Project Purpose\n\nReloader is a Kubernetes operator that automatically triggers rolling restarts of workloads when the ConfigMaps or Secrets they reference are updated. Without it, Kubernetes does not restart pods when configuration changes — operators must do it manually or rely on GitOps pipelines.\n\n**What it watches**: ConfigMaps, Secrets, Namespaces, and (optionally) `SecretProviderClassPodStatus` (CSI-mounted secrets).\n\n**Workload types it can reload**: Deployment, StatefulSet, DaemonSet, CronJob, Job, Argo Rollout, and OpenShift DeploymentConfig.\n\n**How restarts are triggered**: Two strategies (selected via `--reload-strategy`):\n1. **env-vars** (default) — injects an environment variable (`STAKATER_{NAME}_{TYPE}`) into every container with the SHA1 hash of the resource's data. A change in data changes the env var value, causing Kubernetes to restart pods.\n2. **annotations** — writes the SHA1 hash into the pod template's annotations, which also forces a rollout.\n\n**The core problem it solves**: ConfigMaps and Secrets are decoupled from pod lifecycle in Kubernetes. Applications reading config at startup see stale data after a config update unless pods are restarted. Reloader closes that gap automatically and selectively.\n\n**Potential improvements observed**:\n- **Duplicate reload suppression**: If a workload references both a ConfigMap and a Secret that are updated in the same controller reconcile cycle, it may get reloaded twice. Could be solved with a per-workload debounce map keyed by namespace/name/resourceVersion, flushed after a short TTL.\n- **CronJob/Job reload is destructive**: Jobs are deleted and recreated on change, which loses run history. Could instead only annotate the CronJob template without spawning a new Job.\n- **No per-resource reload rate limiting**: A rapid-fire ConfigMap update (e.g., from a CI pipeline) can trigger many restarts. A cooldown window per resource would help.\n- **CSI integration gap**: CSI volumes are watched at the `SecretProviderClassPodStatus` level, but the link back to the workload is indirect and may miss edge cases. Needs a direct map from SecretProviderClass → workloads that mount it.\n\n---\n\n## Repo Map\n\n| Path | Owns | Inspect when |\n|---|---|---|\n| `main.go` | Entry point, delegates to `app.Run()` | Never needs changes |\n| `internal/pkg/app/` | `Run()` bootstrap, Cobra command wiring | Startup sequence changes |\n| `internal/pkg/cmd/` | CLI flags parsing, `startReloader()`, controller/HA wiring | Adding new flags or startup behavior |\n| `internal/pkg/controller/` | Informer/queue per resource type, event handlers (Add/Update/Delete) | Watching new resource types, queue tuning |\n| `internal/pkg/handler/` | Per-event handlers (create, update, delete), `doRollingUpgrade()`, pause deployment | Core reload logic changes |\n| `internal/pkg/callbacks/` | Workload-specific get/list/update/patch functions, `RollingUpgradeFuncs` struct | Adding new workload types |\n| `internal/pkg/options/` | All CLI flag variables, defaults, `ArgoRolloutStrategy` type | Adding or renaming flags |\n| `internal/pkg/constants/` | Constants: env var postfixes, annotation prefix, strategy names, HA lock name | Renaming global identifiers |\n| `internal/pkg/metrics/` | Prometheus `Collectors` struct, all metric registration and recording helpers | Adding metrics |\n| `internal/pkg/alerts/` | Slack/Teams/GChat/raw webhook alerting, env var config | Alert sink changes |\n| `internal/pkg/util/` | SHA generation via `crypto/sha.go`, env var name conversion, namespace/label utilities | Utility/hash changes |\n| `internal/pkg/crypto/` | `GenerateSHA(data)` — SHA1 hex digest | Hash algorithm changes |\n| `internal/pkg/leadership/` | Leader election via Kubernetes Lease, HA stop/start of controllers | HA behavior changes |\n| `internal/pkg/testutil/` | Fake Kubernetes objects for unit tests | Writing new tests |\n| `pkg/common/` | `ReloadCheckResult`, `ReloaderOptions`, `ShouldReload()` logic, `Config` struct | Reload decision logic, annotation precedence |\n| `pkg/kube/` | `Clients` struct (k8s + OpenShift + Argo + CSI), `GetKubernetesClient()`, `ResourceMap` | Client initialization, new CRD clients |\n| `deployments/` | Helm chart (`deployments/kubernetes/chart/reloader/`), Kustomize manifests | Helm values, RBAC, deployment config |\n| `docs/` | User-facing annotation documentation, architecture notes | Writing docs or confirming annotation behavior |\n| `scripts/` | Shell scripts used by CI and Makefile | Build/release pipeline |\n| `test/loadtest/` | Load test CLI (`cmd/loadtest`), 13 scenarios (S1–S13), Kind cluster setup | Performance testing, regression benchmarks |\n| `.github/` | CI workflows: lint, test, Kind e2e, multi-arch Docker build, release | CI changes |\n\n---\n\n## Core Runtime Flow\n\n**1. Entry** — `main.go:10` calls `app.Run()`.\n\n**2. CLI Init** — `internal/pkg/app/app.go` calls `cmd.NewReloaderCommand()` which registers all Cobra flags from `options/flags.go` and runs `startReloader()`.\n\n**3. Client Setup** — `pkg/kube/client.go`: builds `kube.Clients` with:\n- `kubernetes.Interface` — standard k8s client\n- `appsclient.Interface` — OpenShift client (auto-detected by probing `deploymentconfigs`)\n- `argorollout.Interface` — if `--is-Argo-Rollouts=true`\n- `csiclient.Interface` — if `--enable-csi-integration`\n\n**4. Controller Creation** — `startReloader()` iterates `kube.ResourceMap` (configmaps, secrets, namespaces, and optionally secretproviderclasspodstatuses) and calls `controller.NewController()` for each resource in each watched namespace.\n\n**5. Informer/Queue** — `controller.NewController()`:\n- Creates a `cache.NewFilteredListWatchFromClient` with label/field selectors.\n- Registers `Add`, `Update`, `Delete` event handlers.\n- Creates a `workqueue.TypedRateLimitingQueue` for async processing.\n\n**6. Event Detection**:\n- `Add` — enqueues only if `ReloadOnCreate` is enabled (skips during initial sync unless `SyncAfterRestart`).\n- `Update` — compares SHA of old vs new object data; enqueues only on real changes.\n- `Delete` — enqueues only if `ReloadOnDelete` is enabled.\n- Namespace events update `selectedNamespacesCache` for namespace-selector filtering.\n\n**7. Handler Dispatch** — The queue worker calls `handler.Handle()` on the dequeued item. Three handler types:\n- `ResourceCreatedHandler` (`create.go`) — fires `doRollingUpgrade` or sends webhook.\n- `ResourceUpdatedHandler` (`update.go`) — fires `doRollingUpgrade` or sends webhook.\n- `ResourceDeleteHandler` (`delete.go`) — calls `invokeDeleteStrategy` (removes env vars or clears annotation).\n\n**8. Workload Discovery** — `doRollingUpgrade()` (`upgrade.go:181`) calls `rollingUpgrade()` for each workload type. For each type, `ItemsFunc` lists all workloads in the namespace, then `pkg/common.ShouldReload()` checks annotations to decide which ones need reloading.\n\n**9. Reload Execution** — `invokeReloadStrategy()` either:\n- **env-vars**: mutates container env vars; uses JSON patch if `SupportsPatch=true`, full update otherwise.\n- **annotations**: writes SHA to pod template annotations; same patch/update split.\n\n**10. Post-reload** — optionally pauses the Deployment via `pause_deployment.go`, records Kubernetes Events via `recorder`, updates Prometheus metrics, sends alert webhooks.\n\n**HA Mode**: if `--enable-ha`, `internal/pkg/leadership/` runs Kubernetes Lease-based leader election. Only the leader runs controllers; losing leadership stops them and marks the pod unhealthy.\n\n**HTTP Server**: port `:9090` serves `/metrics` (Prometheus) and liveness/readiness probes.\n\n---\n\n## Reload Behavior And Annotations\n\nAll annotation names are configurable via CLI flags; the values below are defaults.\n\n### Trigger Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any** ConfigMap or Secret referenced by the workload (via envFrom, env valueFrom, or volumes) |\n| `configmap.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced ConfigMap** only |\n| `secret.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced Secret** only |\n| `secretproviderclass.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced SecretProviderClass** only |\n| `configmap.reloader.stakater.com/reload` | `\"cm1,cm2\"` | Reload only when the **named ConfigMaps** change (regex supported) |\n| `secret.reloader.stakater.com/reload` | `\"sec1,sec2\"` | Reload only when the **named Secrets** change (regex supported) |\n| `secretproviderclass.reloader.stakater.com/reload` | `\"spc1\"` | Reload only when the **named SecretProviderClass** changes |\n| `reloader.stakater.com/search` | `\"true\"` | Reload when any ConfigMap/Secret tagged with `reloader.stakater.com/match: \"true\"` changes |\n\n### Exclude Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/ignore` | `\"true\"` | Skip this workload entirely |\n| `configmaps.exclude.reloader.stakater.com/reload` | `\"cm1,cm2\"` | Exclude these named ConfigMaps from triggering reload |\n| `secrets.exclude.reloader.stakater.com/reload` | `\"sec1,sec2\"` | Exclude these named Secrets |\n| `secretproviderclasses.exclude.reloader.stakater.com/reload` | `\"spc1\"` | Exclude these named SecretProviderClasses |\n\n### Behavior Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/rollout-strategy` | `\"restart\"` or `\"rollout\"` | For Argo Rollouts: `\"restart\"` uses restartAt, `\"rollout\"` (default) uses full rollout update |\n| `deployment.reloader.stakater.com/pause-period` | Go duration e.g. `\"30s\"` | Pause Deployment for this duration after reload |\n| `deployment.reloader.stakater.com/paused-at` | RFC3339 timestamp | Set by Reloader to track pause start time; do not set manually |\n\n### Search/Match Pattern\n\nThe `reloader.stakater.com/search` annotation on a workload pairs with `reloader.stakater.com/match: \"true\"` on a ConfigMap or Secret. Any workload with `search: true` will reload when any `match: true` resource changes.\n\n### Global Flag Overrides\n\n- `--auto-reload-all` — reload all workloads on any ConfigMap/Secret change; annotation not required.\n- `--resources-to-ignore=configMaps` or `=secrets` — skip one type entirely.\n- `--ignored-workload-types=jobs,cronjobs` — skip Job and CronJob reload.\n- `--namespaces-to-ignore` — comma-separated namespace names to skip.\n- `--namespace-selector` — only watch namespaces with matching labels.\n- `--resource-label-selector` — only watch ConfigMaps/Secrets with matching labels.\n\n### Precedence Rules\n\n1. `reloader.stakater.com/ignore: \"true\"` wins everything — workload is skipped.\n2. Exclude annotations override include annotations for specific named resources.\n3. Named annotations (`.../reload`) are checked before auto annotations.\n4. `--auto-reload-all` is the lowest-priority fallback (only applies if no annotation matches).\n5. Annotations are checked on both the workload and its pod template (pod template takes precedence in some paths — verify in `pkg/common/common.go:ShouldReload()`).\n\n---\n\n## Workload Support\n\n| Workload | SupportsPatch | Update Mechanism | Key files |\n|---|---|---|---|\n| **Deployment** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:38` |\n| **StatefulSet** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:109` |\n| **DaemonSet** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:91` |\n| **CronJob** | No | Creates a new Job from CronJob spec (adds `cronjob.kubernetes.io/instantiate: manual`) | `callbacks.CreateJobFromCronjob`, `handler/upgrade.go:55` |\n| **Job** | No | Deletes old Job, creates new one (strips ResourceVersion, UID, Status, controller labels) | `callbacks.ReCreateJobFromjob`, `handler/upgrade.go:73` |\n| **Argo Rollout** | No | Full update via Argo Rollouts client | `callbacks.UpdateRollout`, `handler/upgrade.go:127`; requires `--is-Argo-Rollouts=true` |\n| **DeploymentConfig** | Yes | OpenShift DeploymentConfigs API | `callbacks/rolling_upgrade.go`; auto-detected by probing `deploymentconfigs` |\n\n**Reload flow per workload**: `doRollingUpgrade()` → `rollingUpgrade()` per type → `ItemsFunc` lists workloads → `ShouldReload()` filters → `invokeReloadStrategy()` patches or updates → optional pause + metrics + alert.\n\n---\n\n## CSI Support\n\n**Enabled by**: `--enable-csi-integration`\n\n**What is watched**: `SecretProviderClassPodStatus` resources (from `sigs.k8s.io/secrets-store-csi-driver`). Resource name constant: `constants.SecretProviderClassController = \"secretproviderclasspodstatuses\"`.\n\n**How it works**:\n1. The CSI driver injects secrets into pods as volume mounts and tracks injection state via `SecretProviderClassPodStatus` objects.\n2. Reloader watches these objects for version changes.\n3. When a version change is detected, it computes a SHA of the object's IDs and versions.\n4. It then looks up the referenced `SecretProviderClass` and treats the event like a Secret update, triggering workload reloads.\n\n**Workload annotation**: `secretproviderclass.reloader.stakater.com/reload: \"my-spc\"` or `secretproviderclass.reloader.stakater.com/auto: \"true\"`.\n\n**Required**: CSI CRDs must be installed in the cluster. Reloader auto-detects their presence at startup.\n\n**Env var postfix**: `STAKATER_{NAME}_SECRETPROVIDERCLASS`.\n\n**Known limitations**:\n- Only works for secrets mounted as volumes via CSI, not env-var-based CSI injection.\n- The link from `SecretProviderClassPodStatus` → workload is indirect; edge cases may be missed.\n- Requires the CSI driver CRDs to be pre-installed; Reloader won't start CSI controller if CRDs are absent.\n\n---\n\n## Build, Test, And Run Commands\n\n**Go version**: `go 1.26.2` (from `go.mod`)\n\n| Purpose | Command |\n|---|---|\n| Run locally | `go run ./main.go` |\n| Build binary | `make build` → `go build -o Reloader` |\n| Unit tests | `make test` → `go test -timeout 1800s -v ./...` |\n| Lint | `make lint` → `golangci-lint run ./...` (v2.6.1) |\n| Docker build (single arch) | `make build-image ARCH=amd64` |\n| Docker push | `make push` |\n| Full release (build+push+manifest) | `make release ARCH=amd64` |\n| Multi-arch release | `make release-all` |\n| Generate k8s manifests | `make k8s-manifests` (Kustomize v5.3.0) |\n| Load test (quick) | `make loadtest-quick LOADTEST_OLD_IMAGE=... LOADTEST_NEW_IMAGE=...` (runs S1, S4, S6) |\n| Load test (full) | `make loadtest-full LOADTEST_OLD_IMAGE=... LOADTEST_NEW_IMAGE=...` |\n| Load test (custom) | `make loadtest LOADTEST_SCENARIOS=S1,S3 LOADTEST_DURATION=120` |\n\n**Docker image**: `ghcr.io/stakater/reloader` — multi-arch (amd64, arm64, arm), distroless nonroot base.\n\n**Helm chart**: `deployments/kubernetes/chart/reloader/` — install via Helm or `kubectl apply -f deployments/kubernetes/reloader.yaml`.\n\n---\n\n## Coding Conventions\n\n**Package boundaries**: Each `internal/pkg/<name>` package has a single clear responsibility. Cross-package access goes through exported types/functions only.\n\n**Error handling**: `logrus.Errorf(...)` for non-fatal, `logrus.Fatalf(...)` for startup failures. Errors are returned up the call stack and logged at the point of action, not at every layer. Retry uses `k8s.io/client-go/util/retry.RetryOnConflict`.\n\n**Logging**: `logrus` with structured fields. Format controlled by `--log-format=json` flag. Log level controlled by `--log-level`. Messages follow the pattern: `\"Changes detected in '%s' of type '%s' in namespace '%s'\"`.\n\n**Kubernetes client patterns**: All k8s operations go through the `kube.Clients` struct. Use `context.TODO()` for context (no request-scoped contexts). List/watch via informers, not polling.\n\n**Callback pattern**: Workload-specific logic is encapsulated in `callbacks.RollingUpgradeFuncs` structs returned by `handler.Get*RollingUpgradeFuncs()`. Adding a new workload type = add a new `RollingUpgradeFuncs` factory function and call it in `doRollingUpgrade()`.\n\n**Test style**: Standard `testing.T`, `testify/assert`. Fake k8s objects via `testutil/kube.go`. Tests live alongside source in the same package. Large integration-style tests in `handler/upgrade_test.go`.\n\n**Naming patterns**:\n- Annotation variables: `XxxUpdateOnChangeAnnotation`, `XxxReloaderAutoAnnotation`\n- Callback funcs: `GetXxxItem`, `GetXxxItems`, `UpdateXxx`, `PatchXxx`\n- Handler factories: `GetXxxRollingUpgradeFuncs()`\n\n**Adding new behavior**: Add flag to `options/flags.go` + `common.ReloaderOptions` struct → wire in `cmd/reloader.go` → implement logic in `handler/` or `callbacks/` → add metrics recording → write tests in `*_test.go`.\n\n---\n\n## Gotchas And Risks\n\n**Duplicate reloads**: If a workload references multiple ConfigMaps/Secrets and all change simultaneously, each change event fires a separate reload. No deduplication exists within a reconcile window. This can cause unnecessary rolling restarts.\n\n**Controller init guard**: `secretControllerInitialized` and `configmapControllerInitialized` booleans in `controller/controller.go` prevent processing Add events during the initial list/sync (to avoid reloading everything on startup). If `--sync-after-restart` is set, both are pre-set to `true`, bypassing the guard. Be careful when this interacts with `--reload-on-create`.\n\n**Namespace filtering**: `--namespaces-to-ignore` does a name match; `--namespace-selector` watches namespaces by label and caches them in `selectedNamespacesCache`. The cache is updated on Namespace Add/Update/Delete events. A race between cache population and first ConfigMap event could cause missed reloads on startup in label-selected deployments.\n\n**RBAC**: Reloader requires get/list/watch on secrets and configmaps, and get/list/watch/update/patch on all workload types it manages. Missing RBAC silently causes no reloads (not an error — just empty lists). Check ClusterRole in `deployments/kubernetes/chart/reloader/templates/`.\n\n**GitOps drift**: If a GitOps tool (Flux, ArgoCD) manages the same Deployments, annotation or env var changes made by Reloader will be detected as drift and reverted. Use `--reload-strategy=annotations` with care in GitOps setups; `env-vars` strategy is generally safer since it modifies the pod template rather than workload-level annotations.\n\n**Annotation precedence edge case**: Annotations are checked first on the workload object, then on the pod template. If both are set to conflicting values, the behavior depends on which path `ShouldReload()` hits first. Verify in `pkg/common/common.go`.\n\n**CronJob/Job destructive reload**: Job recreation deletes the old Job. Any in-flight pod from that Job will be terminated. This is intentional but surprising. There is no protection for long-running jobs.\n\n**OpenShift DeploymentConfig**: Auto-detected by probing for the `deploymentconfigs` resource. If the probe fails at startup, OpenShift support is silently disabled. Check `pkg/kube/client.go`.\n\n**Argo Rollouts**: Must be explicitly enabled via `--is-Argo-Rollouts=true`. Without it, Rollout objects are never listed. The `SupportsPatch=false` means full object updates are used — be aware of potential conflicts with Argo's own controller.\n\n**CSI rotation behavior**: `SecretProviderClassPodStatus` is updated by the CSI driver when secrets rotate. Reloader reacts to those updates. However, if the CSI driver updates the status in a way that doesn't change the versions Reloader tracks, the reload will be missed.\n\n**Backward compatibility**: Annotation names are configurable, so changing defaults would break existing clusters. Never change default annotation values without a migration path.\n\n**Tests to update for risky changes**: `handler/upgrade_test.go` (large suite covering all workload types), `controller/controller_test.go` (event handling), `pkg/common/common_test.go` (reload decision logic).\n\n---\n\n## Open Questions\n\n- **Exact `ShouldReload()` precedence**: The code in `pkg/common/common.go` checks annotations in a specific order. The exact tie-breaking when both workload-level and pod-template-level annotations are set should be verified by reading that function fully before making annotation behavior changes.\n- **CSI → workload mapping**: How exactly does Reloader map a `SecretProviderClassPodStatus` change back to workloads? Is it via the SecretProviderClass name matching an annotation on the workload, or via volume reference scanning? Needs confirmation before adding CSI-related features.\n- **`ContainerPatchPathFunc` field**: `RollingUpgradeFuncs` has a `ContainerPatchPathFunc` field, but it is not documented — unclear if/how it differs from `ContainersFunc` in patch scenarios.\n- **Webhook vs alert**: `--webhook-url` replaces reloading with a POST request. `ALERT_WEBHOOK_URL` env var sends an alert *after* reloading. These are two different mechanisms; the naming is confusing and easy to conflate.\n- **Load test scenarios S7–S13**: Only S1, S4, and S6 are confirmed from CI. The behavior and coverage of the remaining scenarios is unknown without reading `test/loadtest/` in full.\n- **`SyncAfterRestart` semantics**: Flag docs say it \"syncs add events after restart\" but only if `ReloadOnCreate` is also true. The interaction between these two flags in HA mode (where controllers restart on leader change) needs verification.\n\n---\n\n## Important Files\n\n| File | Description |\n|---|---|\n| `internal/pkg/cmd/reloader.go` | `startReloader()` — main wiring of clients, controllers, HA, and HTTP server |\n| `internal/pkg/handler/upgrade.go` | `doRollingUpgrade()` + all `Get*RollingUpgradeFuncs()` factories |\n| `internal/pkg/callbacks/rolling_upgrade.go` | All workload-specific get/update/patch implementations |\n| `pkg/common/common.go` | `ShouldReload()` — the annotation decision tree |\n| `internal/pkg/options/flags.go` | Every configurable option with defaults |\n| `internal/pkg/controller/controller.go` | Informer setup, queue, event handlers |\n| `pkg/kube/client.go` | Multi-client initialization and OpenShift/CSI detection |\n| `internal/pkg/handler/pause_deployment.go` | Pause/resume deployment logic with timers |\n| `internal/pkg/leadership/leadership.go` | HA leader election |\n| `internal/pkg/metrics/prometheus.go` | All Prometheus collector definitions |\n| `internal/pkg/alerts/alert.go` | Slack/Teams/GChat alerting |\n| `internal/pkg/constants/constants.go` | Global constants (env var prefixes, annotation prefix, strategy names) |\n| `deployments/kubernetes/chart/reloader/values.yaml` | Helm chart defaults — source of truth for production config |\n| `handler/upgrade_test.go` | Largest test suite; must be updated for any reload logic change |\n| `Makefile` | All build/test/release/loadtest commands |\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# Stakater Reloader Project Memory\n\n## Project Purpose\n\nReloader is a Kubernetes operator that automatically triggers rolling restarts of workloads when the ConfigMaps or Secrets they reference are updated. Without it, Kubernetes does not restart pods when configuration changes — operators must do it manually or rely on GitOps pipelines.\n\n**What it watches**: ConfigMaps, Secrets, Namespaces, and (optionally) `SecretProviderClassPodStatus` (CSI-mounted secrets).\n\n**Workload types it can reload**: Deployment, StatefulSet, DaemonSet, CronJob, Job, Argo Rollout, and OpenShift DeploymentConfig.\n\n**How restarts are triggered**: Two strategies (selected via `--reload-strategy`):\n1. **env-vars** (default) — injects an environment variable (`STAKATER_{NAME}_{TYPE}`) into every container with the SHA1 hash of the resource's data. A change in data changes the env var value, causing Kubernetes to restart pods.\n2. **annotations** — writes the SHA1 hash into the pod template's annotations, which also forces a rollout.\n\n**The core problem it solves**: ConfigMaps and Secrets are decoupled from pod lifecycle in Kubernetes. Applications reading config at startup see stale data after a config update unless pods are restarted. Reloader closes that gap automatically and selectively.\n\n**Potential improvements observed**:\n- **Duplicate reload suppression**: If a workload references both a ConfigMap and a Secret that are updated in the same controller reconcile cycle, it may get reloaded twice. Could be solved with a per-workload debounce map keyed by namespace/name/resourceVersion, flushed after a short TTL.\n- **CronJob/Job reload is destructive**: Jobs are deleted and recreated on change, which loses run history. Could instead only annotate the CronJob template without spawning a new Job.\n- **No per-resource reload rate limiting**: A rapid-fire ConfigMap update (e.g., from a CI pipeline) can trigger many restarts. A cooldown window per resource would help.\n- **CSI integration gap**: CSI volumes are watched at the `SecretProviderClassPodStatus` level, but the link back to the workload is indirect and may miss edge cases. Needs a direct map from SecretProviderClass → workloads that mount it.\n\n---\n\n## Repo Map\n\n| Path | Owns | Inspect when |\n|---|---|---|\n| `main.go` | Entry point, delegates to `app.Run()` | Never needs changes |\n| `internal/pkg/app/` | `Run()` bootstrap, Cobra command wiring | Startup sequence changes |\n| `internal/pkg/cmd/` | CLI flags parsing, `startReloader()`, controller/HA wiring | Adding new flags or startup behavior |\n| `internal/pkg/controller/` | Informer/queue per resource type, event handlers (Add/Update/Delete) | Watching new resource types, queue tuning |\n| `internal/pkg/handler/` | Per-event handlers (create, update, delete), `doRollingUpgrade()`, pause deployment | Core reload logic changes |\n| `internal/pkg/callbacks/` | Workload-specific get/list/update/patch functions, `RollingUpgradeFuncs` struct | Adding new workload types |\n| `internal/pkg/options/` | All CLI flag variables, defaults, `ArgoRolloutStrategy` type | Adding or renaming flags |\n| `internal/pkg/constants/` | Constants: env var postfixes, annotation prefix, strategy names, HA lock name | Renaming global identifiers |\n| `internal/pkg/metrics/` | Prometheus `Collectors` struct, all metric registration and recording helpers | Adding metrics |\n| `internal/pkg/alerts/` | Slack/Teams/GChat/raw webhook alerting, env var config | Alert sink changes |\n| `internal/pkg/util/` | SHA generation via `crypto/sha.go`, env var name conversion, namespace/label utilities | Utility/hash changes |\n| `internal/pkg/crypto/` | `GenerateSHA(data)` — SHA1 hex digest | Hash algorithm changes |\n| `internal/pkg/leadership/` | Leader election via Kubernetes Lease, HA stop/start of controllers | HA behavior changes |\n| `internal/pkg/testutil/` | Fake Kubernetes objects for unit tests | Writing new tests |\n| `pkg/common/` | `ReloadCheckResult`, `ReloaderOptions`, `ShouldReload()` logic, `Config` struct | Reload decision logic, annotation precedence |\n| `pkg/kube/` | `Clients` struct (k8s + OpenShift + Argo + CSI), `GetKubernetesClient()`, `ResourceMap` | Client initialization, new CRD clients |\n| `deployments/` | Helm chart (`deployments/kubernetes/chart/reloader/`), Kustomize manifests | Helm values, RBAC, deployment config |\n| `docs/` | User-facing annotation documentation, architecture notes | Writing docs or confirming annotation behavior |\n| `scripts/` | Shell scripts used by CI and Makefile | Build/release pipeline |\n| `test/loadtest/` | Load test CLI (`cmd/loadtest`), 13 scenarios (S1–S13), Kind cluster setup | Performance testing, regression benchmarks |\n| `.github/` | CI workflows: lint, test, Kind e2e, multi-arch Docker build, release | CI changes |\n\n---\n\n## Core Runtime Flow\n\n**1. Entry** — `main.go:10` calls `app.Run()`.\n\n**2. CLI Init** — `internal/pkg/app/app.go` calls `cmd.NewReloaderCommand()` which registers all Cobra flags from `options/flags.go` and runs `startReloader()`.\n\n**3. Client Setup** — `pkg/kube/client.go`: builds `kube.Clients` with:\n- `kubernetes.Interface` — standard k8s client\n- `appsclient.Interface` — OpenShift client (auto-detected by probing `deploymentconfigs`)\n- `argorollout.Interface` — if `--is-Argo-Rollouts=true`\n- `csiclient.Interface` — if `--enable-csi-integration`\n\n**4. Controller Creation** — `startReloader()` iterates `kube.ResourceMap` (configmaps, secrets, namespaces, and optionally secretproviderclasspodstatuses) and calls `controller.NewController()` for each resource in each watched namespace.\n\n**5. Informer/Queue** — `controller.NewController()`:\n- Creates a `cache.NewFilteredListWatchFromClient` with label/field selectors.\n- Registers `Add`, `Update`, `Delete` event handlers.\n- Creates a `workqueue.TypedRateLimitingQueue` for async processing.\n\n**6. Event Detection**:\n- `Add` — enqueues only if `ReloadOnCreate` is enabled (skips during initial sync unless `SyncAfterRestart`).\n- `Update` — compares SHA of old vs new object data; enqueues only on real changes.\n- `Delete` — enqueues only if `ReloadOnDelete` is enabled.\n- Namespace events update `selectedNamespacesCache` for namespace-selector filtering.\n\n**7. Handler Dispatch** — The queue worker calls `handler.Handle()` on the dequeued item. Three handler types:\n- `ResourceCreatedHandler` (`create.go`) — fires `doRollingUpgrade` or sends webhook.\n- `ResourceUpdatedHandler` (`update.go`) — fires `doRollingUpgrade` or sends webhook.\n- `ResourceDeleteHandler` (`delete.go`) — calls `invokeDeleteStrategy` (removes env vars or clears annotation).\n\n**8. Workload Discovery** — `doRollingUpgrade()` (`upgrade.go:181`) calls `rollingUpgrade()` for each workload type. For each type, `ItemsFunc` lists all workloads in the namespace, then `pkg/common.ShouldReload()` checks annotations to decide which ones need reloading.\n\n**9. Reload Execution** — `invokeReloadStrategy()` either:\n- **env-vars**: mutates container env vars; uses JSON patch if `SupportsPatch=true`, full update otherwise.\n- **annotations**: writes SHA to pod template annotations; same patch/update split.\n\n**10. Post-reload** — optionally pauses the Deployment via `pause_deployment.go`, records Kubernetes Events via `recorder`, updates Prometheus metrics, sends alert webhooks.\n\n**HA Mode**: if `--enable-ha`, `internal/pkg/leadership/` runs Kubernetes Lease-based leader election. Only the leader runs controllers; losing leadership stops them and marks the pod unhealthy.\n\n**HTTP Server**: port `:9090` serves `/metrics` (Prometheus) and liveness/readiness probes.\n\n---\n\n## Reload Behavior And Annotations\n\nAll annotation names are configurable via CLI flags; the values below are defaults.\n\n### Trigger Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any** ConfigMap or Secret referenced by the workload (via envFrom, env valueFrom, or volumes) |\n| `configmap.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced ConfigMap** only |\n| `secret.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced Secret** only |\n| `secretproviderclass.reloader.stakater.com/auto` | `\"true\"` | Reload on change to **any referenced SecretProviderClass** only |\n| `configmap.reloader.stakater.com/reload` | `\"cm1,cm2\"` | Reload only when the **named ConfigMaps** change (regex supported) |\n| `secret.reloader.stakater.com/reload` | `\"sec1,sec2\"` | Reload only when the **named Secrets** change (regex supported) |\n| `secretproviderclass.reloader.stakater.com/reload` | `\"spc1\"` | Reload only when the **named SecretProviderClass** changes |\n| `reloader.stakater.com/search` | `\"true\"` | Reload when any ConfigMap/Secret tagged with `reloader.stakater.com/match: \"true\"` changes |\n\n### Exclude Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/ignore` | `\"true\"` | Skip this workload entirely |\n| `configmaps.exclude.reloader.stakater.com/reload` | `\"cm1,cm2\"` | Exclude these named ConfigMaps from triggering reload |\n| `secrets.exclude.reloader.stakater.com/reload` | `\"sec1,sec2\"` | Exclude these named Secrets |\n| `secretproviderclasses.exclude.reloader.stakater.com/reload` | `\"spc1\"` | Exclude these named SecretProviderClasses |\n\n### Behavior Annotations (on workloads)\n\n| Annotation | Value | Behavior |\n|---|---|---|\n| `reloader.stakater.com/rollout-strategy` | `\"restart\"` or `\"rollout\"` | For Argo Rollouts: `\"restart\"` uses restartAt, `\"rollout\"` (default) uses full rollout update |\n| `deployment.reloader.stakater.com/pause-period` | Go duration e.g. `\"30s\"` | Pause Deployment for this duration after reload |\n| `deployment.reloader.stakater.com/paused-at` | RFC3339 timestamp | Set by Reloader to track pause start time; do not set manually |\n\n### Search/Match Pattern\n\nThe `reloader.stakater.com/search` annotation on a workload pairs with `reloader.stakater.com/match: \"true\"` on a ConfigMap or Secret. Any workload with `search: true` will reload when any `match: true` resource changes.\n\n### Global Flag Overrides\n\n- `--auto-reload-all` — reload all workloads on any ConfigMap/Secret change; annotation not required.\n- `--resources-to-ignore=configMaps` or `=secrets` — skip one type entirely.\n- `--ignored-workload-types=jobs,cronjobs` — skip Job and CronJob reload.\n- `--namespaces-to-ignore` — comma-separated namespace names to skip.\n- `--namespace-selector` — only watch namespaces with matching labels.\n- `--resource-label-selector` — only watch ConfigMaps/Secrets with matching labels.\n\n### Precedence Rules\n\n1. `reloader.stakater.com/ignore: \"true\"` wins everything — workload is skipped.\n2. Exclude annotations override include annotations for specific named resources.\n3. Named annotations (`.../reload`) are checked before auto annotations.\n4. `--auto-reload-all` is the lowest-priority fallback (only applies if no annotation matches).\n5. Annotations are checked on both the workload and its pod template (pod template takes precedence in some paths — verify in `pkg/common/common.go:ShouldReload()`).\n\n---\n\n## Workload Support\n\n| Workload | SupportsPatch | Update Mechanism | Key files |\n|---|---|---|---|\n| **Deployment** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:38` |\n| **StatefulSet** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:109` |\n| **DaemonSet** | Yes | JSON patch or full update | `callbacks/rolling_upgrade.go`, `handler/upgrade.go:91` |\n| **CronJob** | No | Creates a new Job from CronJob spec (adds `cronjob.kubernetes.io/instantiate: manual`) | `callbacks.CreateJobFromCronjob`, `handler/upgrade.go:55` |\n| **Job** | No | Deletes old Job, creates new one (strips ResourceVersion, UID, Status, controller labels) | `callbacks.ReCreateJobFromjob`, `handler/upgrade.go:73` |\n| **Argo Rollout** | No | Full update via Argo Rollouts client | `callbacks.UpdateRollout`, `handler/upgrade.go:127`; requires `--is-Argo-Rollouts=true` |\n| **DeploymentConfig** | Yes | OpenShift DeploymentConfigs API | `callbacks/rolling_upgrade.go`; auto-detected by probing `deploymentconfigs` |\n\n**Reload flow per workload**: `doRollingUpgrade()` → `rollingUpgrade()` per type → `ItemsFunc` lists workloads → `ShouldReload()` filters → `invokeReloadStrategy()` patches or updates → optional pause + metrics + alert.\n\n---\n\n## CSI Support\n\n**Enabled by**: `--enable-csi-integration`\n\n**What is watched**: `SecretProviderClassPodStatus` resources (from `sigs.k8s.io/secrets-store-csi-driver`). Resource name constant: `constants.SecretProviderClassController = \"secretproviderclasspodstatuses\"`.\n\n**How it works**:\n1. The CSI driver injects secrets into pods as volume mounts and tracks injection state via `SecretProviderClassPodStatus` objects.\n2. Reloader watches these objects for version changes.\n3. When a version change is detected, it computes a SHA of the object's IDs and versions.\n4. It then looks up the referenced `SecretProviderClass` and treats the event like a Secret update, triggering workload reloads.\n\n**Workload annotation**: `secretproviderclass.reloader.stakater.com/reload: \"my-spc\"` or `secretproviderclass.reloader.stakater.com/auto: \"true\"`.\n\n**Required**: CSI CRDs must be installed in the cluster. Reloader auto-detects their presence at startup.\n\n**Env var postfix**: `STAKATER_{NAME}_SECRETPROVIDERCLASS`.\n\n**Known limitations**:\n- Only works for secrets mounted as volumes via CSI, not env-var-based CSI injection.\n- The link from `SecretProviderClassPodStatus` → workload is indirect; edge cases may be missed.\n- Requires the CSI driver CRDs to be pre-installed; Reloader won't start CSI controller if CRDs are absent.\n\n---\n\n## Build, Test, And Run Commands\n\n**Go version**: `go 1.26.2` (from `go.mod`)\n\n| Purpose | Command |\n|---|---|\n| Run locally | `go run ./main.go` |\n| Build binary | `make build` → `go build -o Reloader` |\n| Unit tests | `make test` → `go test -timeout 1800s -v ./...` |\n| Lint | `make lint` → `golangci-lint run ./...` (v2.6.1) |\n| Docker build (single arch) | `make build-image ARCH=amd64` |\n| Docker push | `make push` |\n| Full release (build+push+manifest) | `make release ARCH=amd64` |\n| Multi-arch release | `make release-all` |\n| Generate k8s manifests | `make k8s-manifests` (Kustomize v5.3.0) |\n| Load test (quick) | `make loadtest-quick LOADTEST_OLD_IMAGE=... LOADTEST_NEW_IMAGE=...` (runs S1, S4, S6) |\n| Load test (full) | `make loadtest-full LOADTEST_OLD_IMAGE=... LOADTEST_NEW_IMAGE=...` |\n| Load test (custom) | `make loadtest LOADTEST_SCENARIOS=S1,S3 LOADTEST_DURATION=120` |\n\n**Docker image**: `ghcr.io/stakater/reloader` — multi-arch (amd64, arm64, arm), distroless nonroot base.\n\n**Helm chart**: `deployments/kubernetes/chart/reloader/` — install via Helm or `kubectl apply -f deployments/kubernetes/reloader.yaml`.\n\n---\n\n## Coding Conventions\n\n**Package boundaries**: Each `internal/pkg/<name>` package has a single clear responsibility. Cross-package access goes through exported types/functions only.\n\n**Error handling**: `logrus.Errorf(...)` for non-fatal, `logrus.Fatalf(...)` for startup failures. Errors are returned up the call stack and logged at the point of action, not at every layer. Retry uses `k8s.io/client-go/util/retry.RetryOnConflict`.\n\n**Logging**: `logrus` with structured fields. Format controlled by `--log-format=json` flag. Log level controlled by `--log-level`. Messages follow the pattern: `\"Changes detected in '%s' of type '%s' in namespace '%s'\"`.\n\n**Kubernetes client patterns**: All k8s operations go through the `kube.Clients` struct. Use `context.TODO()` for context (no request-scoped contexts). List/watch via informers, not polling.\n\n**Callback pattern**: Workload-specific logic is encapsulated in `callbacks.RollingUpgradeFuncs` structs returned by `handler.Get*RollingUpgradeFuncs()`. Adding a new workload type = add a new `RollingUpgradeFuncs` factory function and call it in `doRollingUpgrade()`.\n\n**Test style**: Standard `testing.T`, `testify/assert`. Fake k8s objects via `testutil/kube.go`. Tests live alongside source in the same package. Large integration-style tests in `handler/upgrade_test.go`.\n\n**Naming patterns**:\n- Annotation variables: `XxxUpdateOnChangeAnnotation`, `XxxReloaderAutoAnnotation`\n- Callback funcs: `GetXxxItem`, `GetXxxItems`, `UpdateXxx`, `PatchXxx`\n- Handler factories: `GetXxxRollingUpgradeFuncs()`\n\n**Adding new behavior**: Add flag to `options/flags.go` + `common.ReloaderOptions` struct → wire in `cmd/reloader.go` → implement logic in `handler/` or `callbacks/` → add metrics recording → write tests in `*_test.go`.\n\n---\n\n## Gotchas And Risks\n\n**Duplicate reloads**: If a workload references multiple ConfigMaps/Secrets and all change simultaneously, each change event fires a separate reload. No deduplication exists within a reconcile window. This can cause unnecessary rolling restarts.\n\n**Controller init guard**: `secretControllerInitialized` and `configmapControllerInitialized` booleans in `controller/controller.go` prevent processing Add events during the initial list/sync (to avoid reloading everything on startup). If `--sync-after-restart` is set, both are pre-set to `true`, bypassing the guard. Be careful when this interacts with `--reload-on-create`.\n\n**Namespace filtering**: `--namespaces-to-ignore` does a name match; `--namespace-selector` watches namespaces by label and caches them in `selectedNamespacesCache`. The cache is updated on Namespace Add/Update/Delete events. A race between cache population and first ConfigMap event could cause missed reloads on startup in label-selected deployments.\n\n**RBAC**: Reloader requires get/list/watch on secrets and configmaps, and get/list/watch/update/patch on all workload types it manages. Missing RBAC silently causes no reloads (not an error — just empty lists). Check ClusterRole in `deployments/kubernetes/chart/reloader/templates/`.\n\n**GitOps drift**: If a GitOps tool (Flux, ArgoCD) manages the same Deployments, annotation or env var changes made by Reloader will be detected as drift and reverted. Use `--reload-strategy=annotations` with care in GitOps setups; `env-vars` strategy is generally safer since it modifies the pod template rather than workload-level annotations.\n\n**Annotation precedence edge case**: Annotations are checked first on the workload object, then on the pod template. If both are set to conflicting values, the behavior depends on which path `ShouldReload()` hits first. Verify in `pkg/common/common.go`.\n\n**CronJob/Job destructive reload**: Job recreation deletes the old Job. Any in-flight pod from that Job will be terminated. This is intentional but surprising. There is no protection for long-running jobs.\n\n**OpenShift DeploymentConfig**: Auto-detected by probing for the `deploymentconfigs` resource. If the probe fails at startup, OpenShift support is silently disabled. Check `pkg/kube/client.go`.\n\n**Argo Rollouts**: Must be explicitly enabled via `--is-Argo-Rollouts=true`. Without it, Rollout objects are never listed. The `SupportsPatch=false` means full object updates are used — be aware of potential conflicts with Argo's own controller.\n\n**CSI rotation behavior**: `SecretProviderClassPodStatus` is updated by the CSI driver when secrets rotate. Reloader reacts to those updates. However, if the CSI driver updates the status in a way that doesn't change the versions Reloader tracks, the reload will be missed.\n\n**Backward compatibility**: Annotation names are configurable, so changing defaults would break existing clusters. Never change default annotation values without a migration path.\n\n**Tests to update for risky changes**: `handler/upgrade_test.go` (large suite covering all workload types), `controller/controller_test.go` (event handling), `pkg/common/common_test.go` (reload decision logic).\n\n---\n\n## Open Questions\n\n- **Exact `ShouldReload()` precedence**: The code in `pkg/common/common.go` checks annotations in a specific order. The exact tie-breaking when both workload-level and pod-template-level annotations are set should be verified by reading that function fully before making annotation behavior changes.\n- **CSI → workload mapping**: How exactly does Reloader map a `SecretProviderClassPodStatus` change back to workloads? Is it via the SecretProviderClass name matching an annotation on the workload, or via volume reference scanning? Needs confirmation before adding CSI-related features.\n- **`ContainerPatchPathFunc` field**: `RollingUpgradeFuncs` has a `ContainerPatchPathFunc` field, but it is not documented — unclear if/how it differs from `ContainersFunc` in patch scenarios.\n- **Webhook vs alert**: `--webhook-url` replaces reloading with a POST request. `ALERT_WEBHOOK_URL` env var sends an alert *after* reloading. These are two different mechanisms; the naming is confusing and easy to conflate.\n- **Load test scenarios S7–S13**: Only S1, S4, and S6 are confirmed from CI. The behavior and coverage of the remaining scenarios is unknown without reading `test/loadtest/` in full.\n- **`SyncAfterRestart` semantics**: Flag docs say it \"syncs add events after restart\" but only if `ReloadOnCreate` is also true. The interaction between these two flags in HA mode (where controllers restart on leader change) needs verification.\n\n---\n\n## Important Files\n\n| File | Description |\n|---|---|\n| `internal/pkg/cmd/reloader.go` | `startReloader()` — main wiring of clients, controllers, HA, and HTTP server |\n| `internal/pkg/handler/upgrade.go` | `doRollingUpgrade()` + all `Get*RollingUpgradeFuncs()` factories |\n| `internal/pkg/callbacks/rolling_upgrade.go` | All workload-specific get/update/patch implementations |\n| `pkg/common/common.go` | `ShouldReload()` — the annotation decision tree |\n| `internal/pkg/options/flags.go` | Every configurable option with defaults |\n| `internal/pkg/controller/controller.go` | Informer setup, queue, event handlers |\n| `pkg/kube/client.go` | Multi-client initialization and OpenShift/CSI detection |\n| `internal/pkg/handler/pause_deployment.go` | Pause/resume deployment logic with timers |\n| `internal/pkg/leadership/leadership.go` | HA leader election |\n| `internal/pkg/metrics/prometheus.go` | All Prometheus collector definitions |\n| `internal/pkg/alerts/alert.go` | Slack/Teams/GChat alerting |\n| `internal/pkg/constants/constants.go` | Global constants (env var prefixes, annotation prefix, strategy names) |\n| `deployments/kubernetes/chart/reloader/values.yaml` | Helm chart defaults — source of truth for production config |\n| `handler/upgrade_test.go` | Largest test suite; must be updated for any reload logic change |\n| `Makefile` | All build/test/release/loadtest commands |\n","category":"root","tokens":5687}]}