## File: README.md

# Pulse



Pulse is a self-hosted monitoring workspace for Proxmox, Docker, Kubernetes,
TrueNAS, physical and virtual machines, and early-access VMware vSphere
environments. It combines live infrastructure state, history, alerts, recovery
visibility, and scheduled health checks without requiring a conventional
enterprise monitoring stack.



## Why Pulse

- **It watches between visits.** Alerts and Pulse Patrol find failed backups,
  capacity pressure, restart loops, unhealthy containers, clock drift, and
  other problems that dashboards cannot surface when nobody is looking.
- **It keeps each platform familiar.** Proxmox, Docker, Kubernetes, TrueNAS,
  vSphere, and machines have dedicated views, backed by one shared resource
  model for search, alerts, history, and investigation.
- **It stays operator-controlled.** Credentials are encrypted at rest, API
  tokens are scoped, agent commands are disabled by default, and governed fixes
  require the configured policy and approval path.

## Platform coverage

| Platform | Coverage |
|---|---|
| Proxmox VE, PBS, and PMG | Nodes, guests, storage, backups, replication, Ceph, mail gateways, and alerts |
| Docker and Podman | Hosts, containers, Compose projects, Swarm services, health, images, and updates |
| Kubernetes | Clusters, nodes, workloads, pods, services, storage, and events through the unified agent |
| TrueNAS SCALE and CORE | Pools, datasets, disks, snapshots, replication tasks, apps, VMs, and alerts |
| Linux, Windows, and macOS machines | Host health, filesystems, networking, temperatures, RAID, and availability through the unified agent |
| VMware vSphere | Early-access inventory, hosts, clusters, VMs, datastores, networks, snapshots, and recovery context; validate against your own vCenter before production use |

Platform pages keep storage and recovery information beside the infrastructure
it belongs to. Alerts, Actions, and Patrol remain cross-platform views.

## Patrol: monitoring that does rounds

Pulse Patrol runs scheduled checks across the current state and recent history
of your infrastructure. Community installations can use a local model or their
own AI provider for watch-only analysis. Pulse Pro adds investigation and
policy-bound fixes with approval, verification, and an audit trail.



Pulse also includes an interactive Assistant and an MCP adapter for external
clients such as Claude Code and OpenCode. Both sit on top of the same scoped
inventory, metrics, alert, storage, and governed-action contracts.

## Quick start

Choose an exact version from the [latest release](https://github.com/rcourtman/Pulse/releases/latest)
and keep that version pinned during installation.

### Docker

```bash
docker run -d \
  --name pulse \
  -p 7655:7655 \
  -v pulse_data:/data \
  -e PULSE_DEPLOYMENT_METHOD=docker_run \
  --restart unless-stopped \
  rcourtman/pulse:vX.Y.Z
```

Open `http://<your-ip>:7655` and follow the bootstrap-token setup. Docker host
monitoring is provided by the unified agent; the Pulse server container does
not need the Docker socket.

### Proxmox LXC, Linux, and Kubernetes

- [Signed Proxmox LXC and Linux installation](docs/INSTALL.md#quick-start-recommended)
- [Docker Compose](docs/INSTALL.md#docker-compose)
- [Kubernetes and Helm](docs/KUBERNETES.md)

The installer is signed. Verify `install.sh` against the pinned
`pulse-installer` key before running it:

```bash
export PULSE_VERSION=vX.Y.Z
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh"
curl -fsSLO "https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}/install.sh.sshsig"
ssh-keygen -Y verify \
  -f <(printf '%s\n' 'pulse-installer namespaces="pulse-install" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMZd/DaH+BldzOkq1A8KVTcFk73nAyrE8aJOyf7i00jm pulse-installer') \
  -I pulse-installer \
  -n pulse-install \
  -s install.sh.sshsig < install.sh
bash install.sh --version "${PULSE_VERSION}"
rm -f install.sh install.sh.sshsig
```

The GitHub installer installs the Pulse server. Install and upgrade agents
(including v5-to-v6 agent upgrades) with the per-host command generated under
**Settings → Infrastructure → Install on a host**.

### Do you need an agent?

Often not. Proxmox VE, PBS, and PMG are monitored through the Proxmox API
with a read-only token, so nothing runs on the host and the generated setup
script creates a privilege-separated monitoring user. Install the unified
agent only where you want data the platform API cannot provide, such as host
SMART health, temperatures, Docker hosts, or standalone machines.

The agent is operator-controlled by design. Command execution is off by
default, the local listener binds to localhost, and generated systemd units
ship hardened. On Linux, the installer's `--least-privilege` profile runs the
service as a dedicated non-root user, with optional scoped sudo grants for
the few collectors that need elevation. The
[agent security model](docs/AGENT_SECURITY.md) documents exactly what the
agent can and cannot do at each privilege level.

> [!IMPORTANT]
> GitHub release assets and `rcourtman/pulse` images are Community builds.
> Relay, Pro, and eligible legacy customers should use the private image or
> Linux archive provided by the [Pulse download portal](https://pulserelay.pro/download.html).
> Replacing a private Pro runtime with a public Community build removes its
> private runtime hooks.

## Editions

- **Community** — self-hosted monitoring, seven days of metric history, core
  SSO, update alerts, and Patrol with your own provider or local model.
- **Relay** — Community plus secure remote web access, Pulse Mobile pairing,
  push notifications, and fourteen days of history.
- **Pro** — Relay plus Patrol investigation, governed fixes, ninety days of
  history, centralized agent profiles, RBAC, audit logging, and reporting.
- **MSP** — for managed service providers: one Pulse Account running many
  client workspaces, each with an isolated Pulse runtime — separate
  dashboards, alerts, users, audit history, and reports. Free sixty-day
  two-client evaluation at [Pulse for MSPs](https://pulserelay.pro/msp.html).

Core self-hosted monitoring is not gated by monitored-system or child-resource
volume. See the [runtime-aligned capability reference](docs/PULSE_PRO.md) and
[current plans](https://pulserelay.pro) for details.

## Documentation

- [Install and deployment](docs/INSTALL.md)
- [Production deployment and security](docs/PRODUCTION_SECURITY.md)
- [Upgrade from Pulse v5](docs/UPGRADE_v6.md)
- [Configuration](docs/CONFIGURATION.md)
- [Platform and agent guides](docs/README.md#platforms-and-agents)
- [Pulse Intelligence](docs/AI.md)
- [Security](SECURITY.md) and [privacy](docs/PRIVACY.md)
- [Code signing policy](docs/CODE_SIGNING_POLICY.md)
- [Troubleshooting](docs/TROUBLESHOOTING.md)
- [API reference](docs/API.md) and [architecture](ARCHITECTURE.md)

Localized getting started guides:
[Deutsch](docs/i18n/de/README.md) · [Español](docs/i18n/es/README.md)

## Development

Pulse uses Go 1.26 and a SolidJS/TypeScript frontend. The managed development
runtime starts the frontend at `http://127.0.0.1:5173` and proxies API and
WebSocket traffic to the backend on port `7655`.

```bash
npm ci
npm --prefix frontend-modern ci
npm run dev
```

Useful checks:

```bash
go test ./...
npm --prefix frontend-modern test
npm --prefix frontend-modern run type-check
python3 scripts/check_public_docs.py
```

See [CONTRIBUTING.md](CONTRIBUTING.md) before investing in a code change. Pulse
uses an issue-first contribution process and does not normally accept
unsolicited pull requests.

## Community and support

- Ask questions in [GitHub Discussions](https://github.com/rcourtman/Pulse/discussions).
- Report reproducible bugs through [GitHub Issues](https://github.com/rcourtman/Pulse/issues).
- Home Assistant users can use the community-maintained
  [Pulse add-ons](https://github.com/Kosztyk/homeassistant-addons).
- If Pulse is useful to you, support its development through
  [GitHub Sponsors](https://github.com/sponsors/rcourtman) or
  [Ko-fi](https://ko-fi.com/rcourtman).

## License

Everything in this repository is licensed under the [MIT License](LICENSE).
There is no dual licensing and no commercial or source-available code here.

Pulse Pro is a separate commercial product, built from private sources and
distributed through [pulserelay.pro](https://pulserelay.pro). Its use is
governed by the [Terms of Service](TERMS.md), which apply to that product
only, not to anything in this repository.

---

## File: deploy/helm/pulse/README.md

# pulse

  

Helm chart for deploying the Pulse hub and optional Docker, Kubernetes, or OpenShift monitoring agent.

**Homepage:** <https://github.com/rcourtman/Pulse>

## Maintainers

| Name | Email | Url |
| ---- | ------ | --- |
| Pulse Maintainers |  |  |

## Source Code

* <https://github.com/rcourtman/Pulse>

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| agent.affinity | object | `{}` |  |
| agent.args[0] | string | `"--enable-docker"` |  |
| agent.args[1] | string | `"--enable-host=false"` |  |
| agent.command[0] | string | `"/usr/local/bin/pulse-agent"` |  |
| agent.dockerSocket.enabled | bool | `true` |  |
| agent.dockerSocket.hostPathType | string | `"Socket"` |  |
| agent.dockerSocket.path | string | `"/var/run/docker.sock"` |  |
| agent.enabled | bool | `false` |  |
| agent.envFrom | list | `[]` |  |
| agent.env[0].name | string | `"PULSE_URL"` |  |
| agent.env[0].value | string | `"http://pulse:7655"` |  |
| agent.extraEnv | list | `[]` |  |
| agent.extraEnvFrom | list | `[]` |  |
| agent.extraVolumeMounts | list | `[]` |  |
| agent.extraVolumes | list | `[]` |  |
| agent.healthPort | int | `9191` |  |
| agent.image.pullPolicy | string | `"IfNotPresent"` |  |
| agent.image.repository | string | `"rcourtman/pulse"` |  |
| agent.image.tag | string | `""` |  |
| agent.kind | string | `"DaemonSet"` |  |
| agent.livenessProbe.enabled | bool | `true` |  |
| agent.livenessProbe.failureThreshold | int | `3` |  |
| agent.livenessProbe.initialDelaySeconds | int | `5` |  |
| agent.livenessProbe.path | string | `"/healthz"` |  |
| agent.livenessProbe.periodSeconds | int | `10` |  |
| agent.livenessProbe.timeoutSeconds | int | `3` |  |
| agent.nodeSelector | object | `{}` |  |
| agent.podAnnotations | object | `{}` |  |
| agent.podLabels | object | `{}` |  |
| agent.podSecurityContext | object | `{}` |  |
| agent.rbac.create | bool | `false` | Create the read-only Kubernetes collector ClusterRole and binding. |
| agent.rbac.name | string | `""` | Override the Kubernetes collector ClusterRole name. |
| agent.readinessProbe.enabled | bool | `true` |  |
| agent.readinessProbe.failureThreshold | int | `3` |  |
| agent.readinessProbe.initialDelaySeconds | int | `5` |  |
| agent.readinessProbe.path | string | `"/readyz"` |  |
| agent.readinessProbe.periodSeconds | int | `5` |  |
| agent.readinessProbe.timeoutSeconds | int | `3` |  |
| agent.replicaCount | int | `1` |  |
| agent.resources | object | `{}` |  |
| agent.secretEnv.create | bool | `false` |  |
| agent.secretEnv.data | object | `{}` |  |
| agent.secretEnv.keys | list | `[]` |  |
| agent.secretEnv.name | string | `""` |  |
| agent.securityContext.privileged | bool | `false` |  |
| agent.securityContext.runAsGroup | int | `0` |  |
| agent.securityContext.runAsUser | int | `0` |  |
| agent.serviceAccount.annotations | object | `{}` |  |
| agent.serviceAccount.create | bool | `false` |  |
| agent.serviceAccount.name | string | `""` |  |
| agent.tolerations | list | `[]` |  |
| containerSecurityContext.enabled | bool | `true` |  |
| containerSecurityContext.runAsGroup | int | `1000` |  |
| containerSecurityContext.runAsNonRoot | bool | `true` |  |
| containerSecurityContext.runAsUser | int | `1000` |  |
| fullnameOverride | string | `""` |  |
| image.pullPolicy | string | `"IfNotPresent"` |  |
| image.repository | string | `"rcourtman/pulse"` |  |
| image.tag | string | `""` |  |
| imagePullSecrets | list | `[]` |  |
| ingress.annotations | object | `{}` |  |
| ingress.className | string | `""` |  |
| ingress.enabled | bool | `false` |  |
| ingress.hosts[0].host | string | `"pulse.local"` |  |
| ingress.hosts[0].paths[0].path | string | `"/"` |  |
| ingress.hosts[0].paths[0].pathType | string | `"Prefix"` |  |
| ingress.tls | list | `[]` |  |
| monitoring.serviceMonitor.enabled | bool | `false` |  |
| monitoring.serviceMonitor.interval | string | `"30s"` |  |
| monitoring.serviceMonitor.labels | object | `{}` |  |
| monitoring.serviceMonitor.metricRelabelings | list | `[]` |  |
| monitoring.serviceMonitor.path | string | `"/metrics"` |  |
| monitoring.serviceMonitor.relabelings | list | `[]` |  |
| monitoring.serviceMonitor.scrapeTimeout | string | `"10s"` |  |
| nameOverride | string | `""` |  |
| openShift.enabled | bool | `false` | Use SCC-compatible server and agent security defaults. |
| openShift.kubernetesAgent.clusterID | string | `""` | Stable logical agent ID; defaults to the release fullname plus `-openshift`. |
| openShift.kubernetesAgent.enabled | bool | `false` | Deploy the socket-free OpenShift Kubernetes collector. |
| persistence.accessModes[0] | string | `"ReadWriteOnce"` |  |
| persistence.annotations | object | `{}` |  |
| persistence.enabled | bool | `true` |  |
| persistence.existingClaim | string | `""` |  |
| persistence.size | string | `"8Gi"` |  |
| persistence.storageClass | string | `""` |  |
| podAnnotations | object | `{}` |  |
| podLabels | object | `{}` |  |
| podSecurityContext.enabled | bool | `true` |  |
| podSecurityContext.fsGroup | int | `1000` |  |
| replicaCount | int | `1` |  |
| server.affinity | object | `{}` |  |
| server.envFrom | list | `[]` |  |
| server.env[0].name | string | `"TZ"` |  |
| server.env[0].value | string | `"UTC"` |  |
| server.extraEnv | list | `[]` |  |
| server.extraEnvFrom | list | `[]` |  |
| server.extraVolumeMounts | list | `[]` |  |
| server.extraVolumes | list | `[]` |  |
| server.livenessProbe.enabled | bool | `true` |  |
| server.livenessProbe.failureThreshold | int | `3` |  |
| server.livenessProbe.initialDelaySeconds | int | `20` |  |
| server.livenessProbe.path | string | `"/"` |  |
| server.livenessProbe.periodSeconds | int | `30` |  |
| server.livenessProbe.timeoutSeconds | int | `5` |  |
| server.nodeSelector | object | `{}` |  |
| server.podAnnotations | object | `{}` |  |
| server.podLabels | object | `{}` |  |
| server.podSecurityContext | object | `{}` |  |
| server.readinessProbe.enabled | bool | `true` |  |
| server.readinessProbe.failureThreshold | int | `3` |  |
| server.readinessProbe.initialDelaySeconds | int | `10` |  |
| server.readinessProbe.path | string | `"/"` |  |
| server.readinessProbe.periodSeconds | int | `10` |  |
| server.readinessProbe.timeoutSeconds | int | `5` |  |
| server.resources | object | `{}` |  |
| server.secretEnv.create | bool | `false` |  |
| server.secretEnv.data | object | `{}` |  |
| server.secretEnv.keys | list | `[]` |  |
| server.secretEnv.name | string | `""` |  |
| server.securityContext | object | `{}` |  |
| server.tolerations | list | `[]` |  |
| service.annotations | object | `{}` |  |
| service.externalTrafficPolicy | string | `"Cluster"` |  |
| service.loadBalancerIP | string | `""` |  |
| service.port | int | `7655` |  |
| service.type | string | `"ClusterIP"` |  |
| serviceAccount.annotations | object | `{}` |  |
| serviceAccount.create | bool | `true` |  |
| serviceAccount.name | string | `""` |  |
| strategy.type | string | `"RollingUpdate"` |  |

----------------------------------------------
Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)

---

## File: docs/api/SCHEDULER_HEALTH.md

# 🩺 Scheduler Health API

**Endpoint**: `GET /api/monitoring/scheduler/health`
**Auth**: Required (`Authorization: Bearer <token>`, `X-API-Token`, or session cookie)

Returns a real-time snapshot of the adaptive scheduler, including queue state, circuit breakers, and dead-letter tasks.

## 📦 Response Format

```json
{
  "updatedAt": "2025-10-20T13:05:42Z",
  "enabled": true,
  "queue": {
    "depth": 7,
    "dueWithinSeconds": 2,
    "perType": { "pve": 4, "pbs": 2 }
  },
  "deadLetter": {
    "count": 1,
    "tasks": [
      {
        "instance": "pbs-main",
        "type": "pbs",
        "nextRun": "2025-10-20T13:06:40Z",
        "lastError": "connection timeout",
        "failures": 5
      }
    ]
  },
  "breakers": [
    {
      "instance": "pve-a",
      "type": "pve",
      "state": "half_open",
      "failures": 3,
      "retryAt": "2025-10-20T13:06:15Z"
    }
  ],
  "staleness": [
    {
      "instance": "pve-a",
      "type": "pve",
      "lastSuccess": "2025-10-20T13:05:10Z",
      "stalenessSeconds": 32,
      "stalenessScore": 0.12
    }
  ],
  "instances": [
    {
      "key": "pve::pve-a",
      "type": "pve",
      "displayName": "Pulse PVE Cluster",
      "instance": "pve-a",
      "connection": "https://pve-a:8006",
      "pollStatus": {
        "lastSuccess": "2025-10-20T13:05:10Z",
        "lastError": {
          "at": "2025-10-20T13:05:40Z",
          "message": "connection timeout",
          "category": "transient"
        },
        "consecutiveFailures": 2,
        "firstFailureAt": "2025-10-20T13:05:20Z"
      },
      "breaker": {
        "state": "half_open", // closed, open, half_open
        "retryAt": "2025-10-20T13:06:15Z",
        "failureCount": 3,
        "since": "2025-10-20T12:58:10Z",
        "lastTransition": "2025-10-20T13:05:40Z"
      },
      "deadLetter": {
        "present": false,
        "reason": "",
        "retryCount": 0
      }
    }
  ]
}
```

## 🔍 Key Fields

### Instances (`instances`)
The authoritative source for per-instance health.

*   **`pollStatus`**: `lastSuccess` timestamp, `lastError` details, `consecutiveFailures` count.
*   **`breaker`**: `state` (`closed`/`open`/`half_open`), `retryAt` next retry, `since` state start, `lastTransition` timestamp.
*   **`deadLetter`**: `present` flag, `reason` (e.g., `permanent_failure`), `retryCount`, `nextRetry` if scheduled.

### Top-Level Queue and DLQ
*   **`queue`**: Snapshot of the active task queue (depth + per-type counts).
*   **`deadLetter`**: Aggregate DLQ summary plus up to 25 queued tasks.

### Optional Summaries
*   **`breakers`**: Only breakers that are not in default `closed`/zero-failure state.
*   **`staleness`**: Snapshot of staleness scores (if the tracker is enabled).

## 🛠️ Common Queries (jq)

**Find Failing Instances:**
```bash
curl -s http://HOST:7655/api/monitoring/scheduler/health | \
jq '.instances[] | select(.pollStatus.consecutiveFailures > 0) | {key, failures: .pollStatus.consecutiveFailures}'
```

**Check Dead Letter Queue:**
```bash
curl -s http://HOST:7655/api/monitoring/scheduler/health | \
jq '.instances[] | select(.deadLetter.present) | {key, reason: .deadLetter.reason}'
```

**Find Open Breakers:**
```bash
curl -s http://HOST:7655/api/monitoring/scheduler/health | \
jq '.instances[] | select(.breaker.state != "closed") | {key, state: .breaker.state}'
```

---

## File: docs/monitoring/ADAPTIVE_POLLING.md

# 📉 Adaptive Polling

Pulse uses an adaptive scheduler to optimize polling based on instance health and activity.

## 🧠 Architecture
- **Scheduler**: Calculates intervals based on health/staleness.
- **Priority Queue**: Min-heap keyed by `NextRun`.
- **Circuit Breaker**: Prevents hot loops on failing instances using success/failure counters.
- **Backoff**: Exponential retry delays (5s min to 5m max).
- **Worker Pool**: One worker per configured instance (PVE/PBS/PMG), capped at 10.
- **Global Concurrency Cap**: At most 2 polling cycles run at once to avoid resource spikes.

## 🔬 Implementation Details (Developer Info)

### Staleness Scoring
The `AdaptiveScheduler` (`internal/monitoring/scheduler.go`) relies on the `StalenessTracker` to compute a `StalenessScore` (0.0 to 1.0) based on **how long it has been since the last successful poll**.

- `0.0` = fresh (recent success)
- `1.0` = very stale or never succeeded

The staleness score is normalized against `AdaptivePollingMaxInterval` (default 5 minutes).

The scheduler applies **Exponential Smoothing** (alpha 0.6) and a small jitter (5%) to avoid oscillation.

Additional influences:
- **Error penalty**: retries tighten the interval based on the error count.
- **Queue stretch**: large queues gently stretch intervals to avoid overload.

### Circuit Breaker Recovery
The `circuitBreaker` (`internal/monitoring/circuit_breaker.go`) follows a standard state machine:
1. **Closing the Circuit**: One successful poll moves *Half-Open* → *Closed* and resets failure count.
2. **Backoff Calculation**: Retries use exponential backoff starting at 5s (multiplier 2, jitter 0.2) capped at 5m.
3. **Transient vs. Permanent**:
   - **Transient** errors (retryable) are retried up to 5 times before moving to the Dead Letter Queue.
   - **Permanent** errors move directly to the Dead Letter Queue.

**Note:** When `AdaptivePollingMaxInterval` is set to 15 seconds or less, the retry backoff is shortened (750ms initial, 4s max) to keep fast feedback loops during tight polling windows.

## ⚙️ Configuration
Adaptive polling is **disabled by default**.

### UI
There is currently no dedicated UI for adaptive polling. Use environment variables or `system.json`.

### Environment Variables

| Variable | Default | Description |
| :--- | :--- | :--- |
| `ADAPTIVE_POLLING_ENABLED` | `false` | Enable/disable. |
| `ADAPTIVE_POLLING_BASE_INTERVAL` | `10s` | Healthy poll rate. |
| `ADAPTIVE_POLLING_MIN_INTERVAL` | `5s` | Active/busy rate. |
| `ADAPTIVE_POLLING_MAX_INTERVAL` | `5m` | Idle/backoff rate. |

### system.json
You can also set `adaptivePollingEnabled` (and related interval fields) in `system.json` and restart Pulse.

## 📊 Metrics
Exposed at `:9091/metrics`.

| Metric | Type | Description |
| :--- | :--- | :--- |
| `pulse_monitor_poll_total` | Counter | Total poll attempts. |
| `pulse_monitor_poll_duration_seconds` | Histogram | Poll latency. |
| `pulse_monitor_poll_staleness_seconds` | Gauge | Age since last success. |
| `pulse_monitor_poll_queue_depth` | Gauge | Queue size. |
| `pulse_monitor_poll_errors_total` | Counter | Error counts by category. |
| `pulse_scheduler_queue_due_soon` | Gauge | Tasks due in the next 12 seconds. |

## ⚡ Circuit Breaker

| State | Trigger | Recovery |
| :--- | :--- | :--- |
| **Closed** | Normal operation. | — |
| **Open** | ≥3 failures. | Backoff (max 5m). |
| **Half-open** | Retry window elapsed. | Success = Closed; Fail = Open. |

**Dead Letter Queue**: After 5 transient or 1 permanent failure, tasks move to DLQ (30m retry).

## 🩺 Health API
`GET /api/monitoring/scheduler/health` (Auth required)

Returns:
- Queue depth & breakdown.
- Dead-letter tasks.
- Circuit breaker states.
- Per-instance staleness.

---

## File: docs/monitoring/METRICS_DATA_FLOW.md

# Metrics Data Flow (Sparklines vs History)

## Quick summary
- Sparklines/trends toggle: client ring buffer + short-term in-memory server history via `/api/charts` (fast, not durable).
- Guest History tab: persistent SQLite metrics store via `/api/metrics-store/history` (durable, long-range, downsampled).

## Path A: Sparklines ("Trends" toggle)
1. Server polling writes to in-memory history: `monitor.go` -> `metricsHistory.AddGuestMetric/AddNodeMetric`.
2. `/api/charts` (`handleCharts`) reads from `metricsHistory` via `monitor.GetGuestMetrics/GetNodeMetrics`.
3. Client toggles to sparklines: `metricsViewMode.ts` -> `seedFromBackend()` -> `ChartsAPI.getCharts()` -> ring buffer in `metricsHistory.ts`.
4. While in sparklines mode, `metricsSampler.ts` samples websocket state every 30s and appends to the ring buffer; localStorage saves periodically.

## Path B: Guest drawer History tab
1. Server polling writes to SQLite store: `monitor.go` -> `metricsStore.Write(resourceType, ...)`.
2. `/api/metrics-store/history` (`handleMetricsHistory`) queries `metrics.Store` (`Query/QueryAll`) with tiered downsampling and license gating.
3. `GuestDrawer` History charts call `ChartsAPI.getMetricsHistory()` for CPU/memory/disk and ranges `24h/7d/30d/90d`.

## Architecture notes
- In-memory retention is `NewMetricsHistory(1000, 24h)` (`monitor.go`). At 10s polling, 1000 points covers ~2.8h of data.
- `/api/charts` uses a two-tier strategy: ranges ≤ 2h are served from the in-memory buffer; longer ranges (4h, 8h, 24h, 7d, 30d) fall back to the SQLite persistent store with LTTB downsampling to ~500 points per metric.
- Frontend sparkline ring buffer keeps up to 8h locally (`metricsHistory.ts`).
- Docker resource keys differ: in-memory uses `docker:`, persistent store uses `resourceType=dockerContainer`. The `GetGuestMetricsForChart` method maps between these automatically.
- History charts in the guest drawer use `/api/metrics-store/history` (SQLite) for ranges `24h/7d/30d/90d`.

---

## File: docs/monitoring/PROMETHEUS_METRICS.md

# 📊 Prometheus Metrics

Pulse exposes metrics at `/metrics` (default port `9091`).

Example scrape target:

- `http://127.0.0.1:9091/metrics`

This listener is separate from the main UI/API port (`7655`) and binds to loopback by default. Use `PULSE_METRICS_BIND_ADDRESS` only when a scraper must reach Pulse from another host.

When `PULSE_METRICS_TOKEN` is set, Pulse refuses to serve that bearer token on a non-loopback plaintext listener unless `PULSE_METRICS_ALLOW_INSECURE_REMOTE=true` is explicitly set. Prefer a local Prometheus agent, SSH tunnel, VPN-private scrape path, or TLS/mTLS reverse proxy for remote scraping.

| Variable | Description | Default |
| :--- | :--- | :--- |
| `PULSE_METRICS_PORT` | Metrics listener port. | `9091` |
| `PULSE_METRICS_BIND_ADDRESS` | Metrics listener bind address. Keep loopback unless a protected remote scraper needs direct access. | `127.0.0.1` |
| `PULSE_METRICS_TOKEN` | Optional bearer token required for `/metrics`. | *(empty)* |
| `PULSE_METRICS_ALLOW_INSECURE_REMOTE` | Explicit opt-in to serve `PULSE_METRICS_TOKEN` on a non-loopback plaintext listener. Use only behind a trusted private network or TLS-terminating proxy. | `false` |

In Docker and Kubernetes you must expose `9091` explicitly if you want to scrape it from outside the container/pod, and the container/pod should set `PULSE_METRICS_BIND_ADDRESS` to the intended listener address.

**Helm note:** the current chart exposes only port `7655`, so Prometheus scraping requires an additional Service that targets `9091` (and a matching ServiceMonitor).

## 🌐 HTTP Ingress

| Metric | Type | Description |
| :--- | :--- | :--- |
| `pulse_http_request_duration_seconds` | Histogram | Latency buckets by `method`, `route`, `status`. |
| `pulse_http_requests_total` | Counter | Total requests by `method`, `route`, `status`. |
| `pulse_http_request_errors_total` | Counter | Error totals by `method`, `route`, `status_class` (`client_error`, `server_error`, `none`). |

## 🔄 Polling & Nodes

| Metric | Type | Description |
| :--- | :--- | :--- |
| `pulse_monitor_poll_duration_seconds` | Histogram | Per-instance poll latency. |
| `pulse_monitor_poll_total` | Counter | Success/error counts per instance (`result` label). |
| `pulse_monitor_poll_errors_total` | Counter | Poll failures by `error_type`. |
| `pulse_monitor_poll_last_success_timestamp` | Gauge | Unix timestamp of last success. |
| `pulse_monitor_poll_staleness_seconds` | Gauge | Seconds since last success (`-1` if never succeeded). |
| `pulse_monitor_poll_queue_depth` | Gauge | Global queue depth. |
| `pulse_monitor_poll_inflight` | Gauge | In-flight polls by `instance_type`. |
| `pulse_monitor_node_poll_duration_seconds` | Histogram | Per-node poll latency. |
| `pulse_monitor_node_poll_total` | Counter | Success/error counts per node (`result` label). |
| `pulse_monitor_node_poll_errors_total` | Counter | Node poll failures by `error_type`. |
| `pulse_monitor_node_poll_last_success_timestamp` | Gauge | Unix timestamp of last node success. |
| `pulse_monitor_node_poll_staleness_seconds` | Gauge | Seconds since last node success (`-1` if never succeeded). |

## 🧠 Scheduler Health

| Metric | Type | Description |
| :--- | :--- | :--- |
| `pulse_scheduler_queue_due_soon` | Gauge | Tasks due within the next 12 seconds. |
| `pulse_scheduler_queue_depth` | Gauge | Queue depth per `instance_type`. |
| `pulse_scheduler_queue_wait_seconds` | Histogram | Wait time between task readiness and execution. |
| `pulse_scheduler_dead_letter_depth` | Gauge | DLQ depth by `instance_type` and `instance`. |
| `pulse_scheduler_breaker_state` | Gauge | `0`=Closed, `1`=Half-Open, `2`=Open, `-1`=Unknown. |
| `pulse_scheduler_breaker_failure_count` | Gauge | Consecutive failure count. |
| `pulse_scheduler_breaker_retry_seconds` | Gauge | Seconds until next retry allowed. |

## ⚡ Diagnostics Cache

| Metric | Type | Description |
| :--- | :--- | :--- |
| `pulse_diagnostics_cache_hits_total` | Counter | Cache hits. |
| `pulse_diagnostics_cache_misses_total` | Counter | Cache misses. |
| `pulse_diagnostics_refresh_duration_seconds` | Histogram | Refresh latency. |

## 🚨 Alert Lifecycle

| Metric | Type | Description |
| :--- | :--- | :--- |
| `pulse_alerts_active` | Gauge | Active alerts by `level` and `type`. |
| `pulse_alerts_fired_total` | Counter | Total alerts fired by `level` and `type`. |
| `pulse_alerts_resolved_total` | Counter | Total alerts resolved by `type`. |
| `pulse_alerts_acknowledged_total` | Counter | Total alerts acknowledged. |
| `pulse_alerts_suppressed_total` | Counter | Alerts suppressed by `reason` (quiet_hours, rate_limit, duplicate, etc.). |
| `pulse_alerts_rate_limited_total` | Counter | Alerts suppressed due to rate limiting. |
| `pulse_alert_duration_seconds` | Histogram | Time from alert fire to resolve (by `type`). |

## Operational Trust

Operational Trust metrics use closed low-cardinality labels. They never label
series with resource, operational-record, evidence, actor, provider-instance,
or notification-destination IDs.

| Metric | Type | Description |
| :--- | :--- | :--- |
| `pulse_operational_trust_observation_to_open_seconds` | Histogram | Time from the first canonical observation to an open operational record. |
| `pulse_operational_trust_open_to_notification_enqueue_seconds` | Histogram | Time from an open transition to durable notification enqueue. |
| `pulse_operational_trust_evidence_observations_total` | Counter | Evidence observations by bounded `source` and state (`current`, `stale`, `unknown`, `partial`, `unavailable`, `partial_permission`, `denied`, or `other`). |
| `pulse_operational_trust_identity_correlations_total` | Counter | Resource correlation outcomes (`attached`, `standalone`, `ambiguous`, `unresolved`, or `other`). |
| `pulse_operational_trust_protection_posture_evaluations_total` | Counter | Protection posture evaluations by state. |
| `pulse_operational_trust_protection_posture_evaluation_failures_total` | Counter | Posture evaluation failures by bounded reason. |
| `pulse_operational_trust_notification_delivery_total` | Counter | Transition-linked delivery outcomes (`queued`, `retry`, `sent`, `failed`, `dead_letter`, `cancelled`, or `other`). |
| `pulse_operational_trust_active_count_mismatch_total` | Counter | Detected disagreement between the canonical active records and attention projection. |
| `pulse_operational_trust_action_offers_total` | Counter | Action-offer projections by eligibility. |
| `pulse_operational_trust_action_verification_total` | Counter | Verification outcomes, including confirmed, contradicted, inconclusive, timed out, and not attempted. |

Example rollout alerts:

- `increase(pulse_operational_trust_active_count_mismatch_total[15m]) > 0`
- `increase(pulse_operational_trust_notification_delivery_total{outcome="dead_letter"}[15m]) > 0`
- `increase(pulse_operational_trust_action_verification_total{outcome=~"contradicted|timed_out"}[30m]) > 0`

## 🚨 Alerting Examples
- **High Error Rate**: `rate(pulse_http_request_errors_total[5m]) > 0.05`
- **Stale Node**: `pulse_monitor_node_poll_staleness_seconds > 300`
- **Breaker Open**: `pulse_scheduler_breaker_state == 2`

---

## File: docs/operations/ADAPTIVE_POLLING_ROLLOUT.md

# 🚀 Adaptive Polling Rollout

Safely enable dynamic scheduling.

## 📋 Pre-Flight
1.  **Snapshot Health**:
    ```bash
    curl -s -H "X-API-Token: $TOKEN" http://localhost:7655/api/monitoring/scheduler/health | jq .
    ```
2.  **Check Metrics**: Ensure `pulse_monitor_poll_queue_depth` is stable.

## 🟢 Enable
Choose one method:
- **UI**: Not currently exposed in the UI (use CLI or env vars).
- **CLI**:
  - systemd/LXC: `jq '.adaptivePollingEnabled=true' /etc/pulse/system.json > /tmp/system.json && sudo mv /tmp/system.json /etc/pulse/system.json`
  - Docker/Kubernetes: edit `/data/system.json` in the volume and restart the container/pod
- **Env**: `ADAPTIVE_POLLING_ENABLED=true` (Docker/K8s).

## 🔍 Monitor (First 15m)
Watch for stability:
```bash
watch -n 5 'curl -s http://localhost:9091/metrics | grep pulse_monitor_poll_queue_depth'
```
- **Success**: Queue depth < 50, no permanent errors.
- **Failure**: High queue depth, open breakers.

## ↩️ Rollback
If instability occurs > 10m:
1.  **Disable**: Remove the env var override or set `adaptivePollingEnabled=false` in `system.json`.
2.  **Restart**: Required if using Env/CLI overrides.
3.  **Verify**: Confirm queue drains.

---

## File: docs/operations/AUTO_UPDATE.md

# 🔄 Automatic Updates
Manage Pulse auto-updates on host-mode installations.

> **Note**: Docker/Kubernetes users should manage updates via their orchestrator.

## ⚙️ Components

| File | Purpose |
| :--- | :--- |
| `pulse-update.timer` | Daily check (02:00 + jitter). |
| `pulse-update.service` | Runs the update script. |
| `pulse-auto-update.sh` | Fetches release & restarts Pulse (`/opt/pulse/scripts/pulse-auto-update.sh`). |

**Release channel note:** the systemd timer script tracks GitHub `releases/latest` (stable). The internal `rc` channel only affects manual and in-app pre-release selection. `stable` is the only recommended channel for paid or production environments.

## 🚀 Enable/Disable

### Via UI (Recommended)
**Settings → System → Updates**.

### Via CLI
```bash
# Enable
sudo jq '.autoUpdateEnabled=true' /etc/pulse/system.json > /tmp/system.json && sudo mv /tmp/system.json /etc/pulse/system.json
sudo systemctl enable --now pulse-update.timer

# Disable
sudo jq '.autoUpdateEnabled=false' /etc/pulse/system.json > /tmp/system.json && sudo mv /tmp/system.json /etc/pulse/system.json
sudo systemctl disable --now pulse-update.timer
```

## 🧪 Manual Run
Test the update process:
```bash
sudo systemctl start pulse-update.service
journalctl -u pulse-update -f
```

## 🔍 Observability
*   **History**: in-app updates are tracked via `GET /api/updates/history` (admin auth required) and stored in `update-history.jsonl` under `/etc/pulse` or `/data`. The systemd timer script does not record update history entries.
*   **Logs**: `journalctl -u pulse-update -f` or `journalctl -t pulse-auto-update -f` for timer runs. In-app updates write detailed logs to `/var/log/pulse/update-<timestamp>.log`.

## ↩️ Rollback
If an update fails:
1.  Check logs: `journalctl -u pulse-update -f` or `journalctl -t pulse-auto-update -f`.
2.  The timer script keeps a temporary backup under `/tmp/pulse-backup-<timestamp>` during the update; failures auto-restore from that backup and then clean it up.
3.  If you need to pin a specific version, re-run the installer with a version:
    ```bash
    sudo /bin/update --version vX.Y.Z
    ```
    `/bin/update` is installed by the supported Pulse server installer. If your host does not have it yet, use the signed server-installer flow in [INSTALL.md](../INSTALL.md). Agent updates still use the `/install.sh` command generated in **Settings → Infrastructure → Install on a host**.