# Technical Documentation: dkron-io/dkron

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/dkron-io/dkron)

### Overview (`README.md`)

<p align="center">
<img width="400" src="docs/images/DKRON_STICKER_OK_CMYK_RGB_CONV_300.png" alt="Dkron" title="Dkron" />
</p>

# Dkron - Distributed, fault tolerant job scheduling system for cloud native environments [](https://godoc.org/github.com/distribworks/dkron) [](https://github.com/distribworks/dkron/actions) [](https://gitter.im/distribworks/dkron) [](https://gurubase.io/g/dkron)

Website: http://dkron.io/

Dkron is a distributed cron service, easy to setup and fault tolerant with focus in:

- Easy: Easy to use with a great UI
- Reliable: Completely fault tolerant
- Highly scalable: Able to handle high volumes of scheduled jobs and thousands of nodes

Dkron is written in Go and leverage the power of the Raft protocol and Serf for providing fault tolerance, reliability
and scalability while keeping simple and easily installable.

Dkron is inspired by the google
whitepaper [Reliable Cron across the Planet](https://queue.acm.org/detail.cfm?id=2745840) and by Airbnb Chronos
borrowing the same features from it.

Dkron runs on Linux, OSX and Windows. It can be used to run scheduled commands on a server cluster using any combination
of servers for each job. It has no single points of failure due to the use of the Gossip protocol and fault tolerant
distributed databases.

You can use Dkron to run the most important part of your company, scheduled jobs.

## Installation

[Installation instructions](https://dkron.io/docs/basics/installation)

Full, comprehensive documentation is accessible on the [Dkron website](http://dkron.io)

## Quickstart

### Deploying Dkron using Docker

The best way to test and develop dkron is using docker, you will need [Docker](https://www.docker.com/) with Docker
compose installed before proceeding.

```bash
docker compose up -d
```

The UI should be available on http://localhost:8080/ui.

### Using Dkron

To add jobs to the system read the [API docs](https://dkron.io/api/).

### Scaling the cluster

To add more Dkron instances to the cluster:

```bash
docker compose up -d --scale dkron-server=4
docker compose up -d --scale dkron-agent=10
```

## Development

To develop Dkron, you can deploy the cluster with local changes applied with the following steps:

1. Clone the repository.

2. Run the `docker compose`:

    ```bash
    docker compose -f docker-compose.dev.yml up
    ```

### Email Testing

For testing email notifications during development, Mailpit is included in the development docker-compose setup. Mailpit provides a local SMTP server that captures outgoing emails without sending them to real recipients.

Start Mailpit with the development environment:

```bash
docker compose -f docker-compose.dev.yml up mailpit
```

Or run it standalone:

```bash
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit
```

View captured emails at: http://localhost:8025

For more information, see [docs/EMAIL_TESTING.md](docs/EMAIL_TESTING.md).

### Testing CI Locally

To validate that your changes will pass in GitHub Actions before pushing:

```bash
./scripts/test-ci-locally.sh
```

This script:
- Starts Mailpit (simulating the CI service container)
- Runs tests with the same configuration as GitHub Actions
- Provides clear pass/fail results
- Allows you to inspect emails in the Mailpit UI

See [.github/TESTING.md](.github/TESTING.md) for more information about CI testing.

### Frontend development

Dkron dashboard is built using [React Admin](https://marmelab.com/react-admin/) as a single page application.

To start developing the dashboard enter the `ui` directory and run `npm install` to get the frontend dependencies and
then start the local server with `npm start` it should start a new local web server and open a new browser window
serving de web ui.

Make your changes to the code, then run `make ui` to generate assets files. This is a method of embedding resources in
Go applications.

### Resources

Chef cookbook
https://supermarket.chef.io/cookbooks/dkron

Python Client Library
https://github.com/oldmantaiter/pydkron

Ruby client
https://github.com/jobandtalent/dkron-rb

PHP client
https://github.com/gromo/dkron-php-adapter

Terraform provider
https://github.com/bozerkins/terraform-provider-dkron

Manage and run jobs in Dkron from your django project
https://github.com/surface-security/django-dkron

## Contributors

<a href="https://github.com/distribworks/dkron/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=distribworks/dkron" />
</a>

Made with [contrib.rocks](https://contrib.rocks).

## Get in touch

- Twitter: [@dkronio](https://twitter.com/dkronio)
- Chat: https://gitter.im/distribworks/dkron
- Email: support at dkron.io

---

### Overview (`client/README.md`)

# Dkron Go client 

## Installation

```bash
go get github.com/distribworks/dkron/v4/client
```

## Usage

```go
package main

import (
	"context"

	"github.com/distribworks/dkron/v4/client"
	log "github.com/sirupsen/logrus"
)

func main() {
	newClient, err := client.NewClient("http://localhost:9090")
	if err != nil {
		log.Fatal(err)
	}

	// Create a new job
	params := &client.CreateOrUpdateJobParams{
		Runoncreate: nil,
	}
	body := client.CreateOrUpdateJobJSONRequestBody{
		Name:       "testjob1",
		Timezone:   "Europe/Paris",
		Schedule:   "@every 1m",
		Owner:      "test",
		OwnerEmail: "",
		Tags: map[string]string{
			"tag1": "value1",
			"tag2": "value2",
		},
		Retries:  3,
		Executor: "Shell",
		ExecutorConfig: map[string]string{
			"command": "echo hello world!",
		},
		Displayname: "Job",
		Ephemeral:   false,
	}

	_, err = newClient.CreateOrUpdateJob(context.Background(), params, body)
	if err != nil {
		log.WithError(err).Error("Failed to create job")
	}
}

```

---

### Overview (`ui/README.md`)

# ui

## Installation

Install the application dependencies by running:

```sh
yarn
```

## Development

Start the application in development mode by running:

```sh
yarn dev
```

## Production

Build the application in production mode by running:

```sh
yarn build
```

## DataProvider

The included data provider use [ra-data-simple-rest](https://github.com/marmelab/react-admin/tree/master/packages/ra-data-simple-rest). It fits REST APIs using simple GET parameters for filters and sorting. This is the dialect used for instance in [FakeRest](https://github.com/marmelab/FakeRest).

You'll find an `.env` file at the project root that includes a `VITE_JSON_SERVER_URL` variable. Set it to the URL of your backend.

---

### Overview (`website/README.md`)

# Website

This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.

### Installation

```
$ yarn
```

### Local Development

```
$ yarn start
```

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

### Build

```
$ yarn build
```

This command generates static content into the `build` directory and can be served using any static contents hosting service.

### Deployment

Using SSH:

```
$ USE_SSH=true yarn deploy
```

Not using SSH:

```
$ GIT_USER=<Your GitHub username> yarn deploy
```

If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.

---

### Overview (`builtin/bins/dkron-executor-rabbitmq/README.md`)

### RabbitMQ Executor

#### Executor Configuration

The options names are inherited from the [RabbitMQ Publishers](https://www.rabbitmq.com/docs/publishers)

| Option                | Required | Description                   | Default    |
|-----------------------|----------|-------------------------------|------------|
| url                   | yes      | RabbitMQ connection string    | -          |
| exchange              | no       | RabbitMQ Exchange             | ""         |
| queue.name            | yes      | Queue name to send message to | -          |
| queue.create          | no       | Create queue if not exists    | false      |
| queue.durable         | no       | Durable queue                 | false      |
| queue.auto_delete     | no       | Auto delete queue             | false      |
| queue.exclusive       | no       | Exclusive queue               | false      |
| message.content_type  | no       | Message content type          | text/plain |
| message.delivery_mode | no       | Message delivery mode         | 0          |
| message.messageId     | no       | Message id                    | ""         |
| message.body          | yes      | Message body                  | -          |
| message.base64Body    | yes      | Base64 encoded message body   | -          |

#### Example

```shell
curl localhost:8080/v1/jobs -XPOST -d '{
  "name": "job1",
  "schedule": "@every 10s",
  "timezone": "Europe/Berlin",
  "owner": "Platform Team",
  "owner_email": "platform@example.com",
  "disabled": false,
  "tags": {
    "server": "true:1"
  },
  "metadata": {
    "user": "12345"
  },
  "concurrency": "allow",
  "executor": "rabbitmq",
  "executor_config": {
  		"url": "amqp://guest:guest@localhost:5672/",
      "exchange": "amq.default",
  		"queue.name": "test",
  		"queue.create": "true",
  		"queue.durable": "true",
  		"queue.auto_delete": "false",
  		"queue.exclusive": "false",
  		"message.content_type": "application/json",
  		"message.delivery_mode": "2",
  		"message.messageId": "4373732772",
  		"message.body": "{\"key\":\"value\"}"
  }
}'
```

---

### Intro (`website/docs/intro.md`)

---
sidebar_position: 1
---
# Intro

## Dkron - Distributed, fault tolerant job scheduling system

Welcome to the Dkron documentation! This is the reference guide on how to use Dkron. If you want a getting started guide refer to the [getting started guide](/docs/basics/getting-started).

## What is Dkron

Dkron is a distributed system to run scheduled jobs against a server or a group of servers of any size. One of the machines is the leader and the others will be followers. If the leader fails or becomes unreachable, any other one will take over and reschedule all jobs to keep the system healthy.

In case the old leader becomes alive again, it'll become a follower.

Dkron is a distributed cron drop-in replacement, easy to setup and fault tolerant with focus in:

- Easy: Easy to use with a great UI
- Reliable: Completely fault tolerant
- Highly scalable: Able to handle high volumes of scheduled jobs and thousands of nodes

Dkron is written in Go and leverages the power of distributed key value stores and [Serf](https://www.serfdom.io/) for providing fault tolerance, reliability and scalability while remaining simple and easily installable.

Dkron is inspired by the google whitepaper [Reliable Cron across the Planet](https://queue.acm.org/detail.cfm?id=2745840)

Dkron runs on Linux, OSX and Windows. It can be used to run scheduled commands on a server cluster using any combination of servers for each job. It has no single points of failure due to the use of the fault tolerant distributed databases and can work at large scale thanks to the efficient and lightweight gossip protocol.

Dkron uses the efficient and lightweight [gossip protocol](https://www.serfdom.io/docs/internals/gossip.html) underneath to communicate with nodes. Failure notification and task handling are run efficiently across an entire cluster of any size.

For a detailed overview of Dkron's internal architecture, components, and how they work together, see the [Architecture documentation](/docs/architecture).

## Key Concepts

### Jobs

Jobs are the core entity in Dkron. A job consists of:

- **Name**: Unique identifier for the job
- **Schedule**: When to run the job (cron expression)
- **Command**: What to run
- **Executor**: How to run the command (shell, HTTP, etc.)
- **Processors**: How to process the output
- **Tags**: Key-value pairs for node selection
- **Concurrency**: Options to control concurrent execution
- **Dependent Jobs**: Jobs that should run after this job completes

### Tags and Node Selection

Dkron uses tags to control which nodes execute specific jobs:

- **Node Tags**: Assigned to nodes during startup (`--tag key=value`)
- **Job Tags**: Specified in job definitions (`"tags": {"role": "web"}`)
- **Tag Matching**: Jobs run on nodes where all job tags match node tags

### Concurrency Options

Dkron provides several options to control job concurrency:

- **Concurrency**: Allow (or disallow) concurrent executions of the same job

### Status Codes and Retries

Jobs can be configured with:

- **Retries**: Number of times to retry a failed execution

### Job Dependencies

Dkron supports job dependencies for complex workflows:

- **Parent-Child Relationships**: Jobs can depend on other jobs
- **Status Checking**: Child jobs run only if parent jobs succeed
- **Chained Execution**: Create multi-step job pipelines

## Web UI



The Dkron web UI provides an easy-to-use interface for:

1. Creating and editing jobs
2. Viewing execution history and logs
3. Monitoring cluster status
4. Running jobs manually
5. Managing job dependencies

## Design Philosophy

Dkron is designed to solve one problem well: executing commands at given intervals. Following the Unix philosophy of doing one thing and doing it well (like the battle-tested cron), but with the addition of being designed for the cloud era, removing single points of failure in environments where scheduled jobs need to be run across multiple servers.

---

### Basics Getting Started (`website/docs/basics/getting-started.md`)

---
sidebar_position: 1
---
# Getting started

## Introduction

Dkron is a distributed job scheduling system that runs across multiple nodes. Understanding its architecture will help you deploy it effectively.

### Node Types

Dkron nodes can work in two modes:

- **Agents**: Cluster members that handle job executions, run scripts, and return the output to servers.
- **Servers**: Cluster members that send job execution queries and also handle scheduling, data storage, and leader election.

### Leadership and Job Execution

- A single server node is elected as the **leader** in each Dkron cluster
- The leader is responsible for initiating job execution queries across the cluster
- Any Dkron agent or server can run scheduled jobs
- By default, all nodes execute each job, but this is configurable

### Job Targeting

You can control which nodes run a job by:
- Specifying tags on nodes
- Setting a count of target nodes with specific tags in your job definition

This provides flexibility in running jobs across clusters of any size with precise node targeting.

### Execution Flow

1. The leader schedules a job based on its timing configuration
2. Target nodes are selected based on tags and count
3. Selected nodes execute the job
4. Execution responses are collected by the scheduler
5. Results are stored in the database

## State Storage

Dkron deployment is just a single binary. It stores state in an internal BoltDB instance and replicates all changes between server nodes using the Raft protocol. No external storage system is required.

## Installation

See the [installation](installation.md).

## Configuration

See the [configuration](configuration.md).

## Usage

By default Dkron uses the following ports:

- `8946` for serf layer between agents
- `8080` for HTTP for the API and Dashboard
- `6868` for gRPC and raft layer comunication between agents.

:::info
Be sure you have opened this ports (or the ones that you configured) in your firewall or AWS security groups.
:::

### Starting a single node

Works out of the box, good for non HA installations.

- System service: If no changes are done to the default config files, dkron will start as a service in single mode.
- Command line: Running a single node with default config can be done by running:

```
dkron agent --server --bootstrap-expect=1
```

Check your server is working: `curl localhost:8080/v1`

:::info
For a full list of configuration parameters and its description, see the [CLI docs](/docs/cli/dkron_agent)
:::

### Create a Job

:::info
This job will only run in just one `server` node due to the node count in the tag. Refer to the [target node spec](/docs/usage/target-nodes-spec) for details.
:::

```bash
curl localhost:8080/v1/jobs -XPOST -d '{
  "name": "job1",
  "schedule": "@every 10s",
  "timezone": "Europe/Berlin",
  "owner": "Platform Team",
  "owner_email": "platform@example.com",
  "disabled": false,
  "tags": {
    "server": "true:1"
  },
  "metadata": {
    "user": "12345"
  },
  "concurrency": "allow",
  "executor": "shell",
  "executor_config": {
    "command": "date"
  }
}'
```

For full Job params description refer to the Job model in the [API guide](/api)

That's it!

#### To start configuring an HA installation of Dkron follow the [clustering guide](/docs/usage/clustering)

---

### Pro  Index (`website/docs/pro/_index.md`)

---
title: Dkron Pro
weight: 40
---

{{% children style="li" depth="999" %}}

---

### Usage Executors Index (`website/docs/usage/executors/index.md`)

# Executors

Executors are the core plugins that perform actual job tasks in Dkron. They implement different execution strategies and allow Dkron to interact with various systems and services.

## Overview

Executors define how jobs are executed on target nodes. When a job is scheduled to run, Dkron uses the specified executor to perform the actual work. Each executor has its own configuration options and capabilities.

:::info
If you only plan to use the built-in executors, `http` and `shell` you can use the Dkron Light edition that only includes a single binary as the plugins are built-in.
:::

## Built-in Executors

Dkron includes the following built-in executors:

| Executor | Description | Use Case |
|----------|-------------|----------|
| [Shell](/docs/usage/executors/shell) | Executes shell commands on target nodes | Running scripts, system commands |
| [HTTP](/docs/usage/executors/http) | Makes HTTP requests to specified endpoints | API integrations, web hooks |

## Extended Executors (Open Source)

These executors are available as separate plugins in the open source version:

| Executor | Description | Use Case |
|----------|-------------|----------|
| [Kafka](/docs/usage/executors/kafka) | Produces messages to Kafka topics | Event-driven architectures, data pipelines |
| [NATS](/docs/usage/executors/nats) | Publishes messages to NATS servers | Microservices communication, event publishing |
| [RabbitMQ](/docs/usage/executors/rabbitmq) | Sends messages to RabbitMQ exchanges | Message queuing, work distribution |
| [gRPC](/docs/usage/executors/grpc) | Makes calls to gRPC services | Microservices integration, efficient RPC |
| [Google Pub/Sub](/docs/usage/executors/gcp-pubsub) | Publishes messages to Google Pub/Sub topics | Cloud-native event publishing |

## Pro Executors

[Dkron Pro](/pro) includes additional commercially supported executors:

| Executor | Description | Use Case |
|----------|-------------|----------|
| Docker | Runs containers and commands within containers | Isolated execution environments |
| AWS ECS | Runs tasks on Amazon ECS clusters | Serverless job execution in AWS |

## Executor Architecture

Executors follow a plugin architecture in Dkron:

```mermaid
flowchart TD
    DkronServer["Dkron Server"] -->|Job Assignment| ExecutorPlugin["Executor Plugin"]
    ExecutorPlugin -->|Performs Action| TargetSystem["Target System"]
```

1. The Dkron server schedules a job and assigns it to a node
2. The node loads the specified executor plugin
3. The executor performs its action on the target system
4. Results are returned to Dkron for processing

## Common Executor Configuration Patterns

Most executors share some common configuration concepts:

1. **Timeouts**: Limit how long a job can run before being terminated
2. **Environment Variables**: Pass context to the execution environment
3. **Credentials**: Securely provide authentication information
4. **Retry Logic**: Define behavior when execution fails

## Executor Selection Guide

When choosing an executor for your job, consider:

1. **Target System**: What system or service needs to be interacted with?
2. **Isolation Needs**: Does the job need to run in an isolated environment?
3. **Resource Requirements**: How much CPU/memory is needed?
4. **Security Considerations**: What permissions or credentials are required?

## Custom Executors

Dkron supports custom executors through its plugin system. You can develop your own executors to meet specific requirements not covered by existing plugins.

To develop a custom executor:

1. Implement the `Executor` interface from the Dkron plugin package
2. Follow the [Go plugin system](https://golang.org/pkg/plugin/) requirements
3. Build and deploy your executor to the Dkron plugins directory

See the [plugin development guide](/docs/usage/plugins/develop) for more details.

## Best Practices

1. **Use the Simplest Executor**: Choose the simplest executor that meets your requirements
2. **Set Appropriate Timeouts**: Always configure timeouts to prevent runaway jobs
3. **Validate Output**: Use processors to validate and act on executor output
4. **Monitor Resource Usage**: Pay attention to the resource consumption of jobs
5. **Security First**: Follow the principle of least privilege when configuring executor permissions

---

### Usage Plugins Index (`website/docs/usage/plugins/index.md`)

# Plugins

## Intro

Plugins in Dkron allow you to add funcionality that integrates with the workflow of the job execution in Dkron. It's a powerful system that allows you to extend and adapt Dkron to your special needs.

This page documents the basics of how the plugin system in Dkron works, and how to setup a basic development environment for plugin development if you're writing a Dkron plugin.

## How it Works

Dkron execution execution processors are provided via plugins. Each plugin exposes functionality for modifying the execution. Plugins are executed as a separate process and communicate with the main Dkron binary over an RPC interface.

The code within the binaries must adhere to certain interfaces. The network communication and RPC is handled automatically by higher-level libraries. The exact interface to implement is documented in its respective documentation section.

## Installing a Plugin

Dkron searches for plugins at startup, to install a plugin just drop the binary in one of the following locations:

1. /etc/dkron/plugins
2. Dkron executable directory

---

### Usage Processors Index (`website/docs/usage/processors/index.md`)

# Processors

## Execution Processors

Processor plugins are called when an execution response has been received. They are passed the resulting execution data and configuration parameters, this plugins can perform a variety of operations with the execution and it's very flexible and per Job, examples of operations this plugins can do:

* Execution output storage, forwarding or redirection.
* Notification
* Monitoring

For example, Processor plugins can be used to redirect the output of a job execution to different targets.

Currently Dkron provides you with some built-in plugins but the list keeps growing. Some of the features previously implemented in the application will be progessively moved to plugins.

### Built-in processors

Dkron provides the following built-in processors:

0. not specified - Store the output in the key value store (Slow performance, good for testing, default method)
0. log - Output the execution log to Dkron stdout (Good performance, needs parsing)
0. syslog - Output to the syslog (Good performance, needs parsing)
0. files - Output to multiple files (Good performance, needs parsing)

[Dkron Pro](/pro/) provides you with several more processors.

All plugins accepts one configuration option: `forward` Indicated if the plugin must forward the original execution output. This allows for chaining plugins and sending output to different targets at the same time.

---

### EMAIL TESTING (`docs/EMAIL_TESTING.md`)

# Email Testing with Mailpit

This document describes how to test email notifications in Dkron using Mailpit.

## What is Mailpit?

[Mailpit](https://github.com/axllent/mailpit) is a modern email testing tool for developers. It runs a fake SMTP server to capture outgoing emails instead of sending them to real recipients. This allows you to:

- Test email functionality without sending real emails
- Inspect email content, headers, and formatting via a web UI
- Verify emails are being sent correctly during development and testing
- View emails in a modern, responsive interface with search capabilities

## Why Mailpit?

Mailpit offers several advantages over older tools:

- **Actively Maintained**: Regular updates and bug fixes
- **Modern UI**: Clean, responsive interface with better UX
- **Fast**: Written in Go with excellent performance
- **Feature-Rich**: Search, filtering, tagging, and more
- **Small Footprint**: Lightweight Docker image (~15MB compressed)
- **API Support**: RESTful API for automated testing

## Running Mailpit

### Using Docker (Recommended)

The easiest way to run Mailpit is using Docker:

```bash
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit
```

This exposes:
- **Port 1025**: SMTP server (for your application to send emails)
- **Port 8025**: Web UI (to view captured emails)

### Using Docker Compose

Mailpit is included in the development docker-compose configuration. Start it with:

```bash
docker compose -f docker-compose.dev.yml up mailpit
```

Or start the entire development environment including Mailpit:

```bash
docker compose -f docker-compose.dev.yml up
```

### Standalone Installation

You can also install Mailpit directly on your system. See the [Mailpit documentation](https://github.com/axllent/mailpit#installation) for installation instructions.

## Configuring Dkron to Use Mailpit

To configure Dkron to send emails through Mailpit, use the following SMTP settings:

```yaml
MailHost: localhost
MailPort: 1025
MailFrom: dkron@example.com
```

**Note**: Mailpit does not require authentication, so you don't need to set `MailUsername` or `MailPassword`.

## Running Email Tests

The email notification tests in `dkron/notifier_test.go` are configured to use Mailpit by default. The tests automatically verify that emails are correctly sent and received using Mailpit's REST API.

To run the email notification test:

```bash
# Make sure Mailpit is running first
docker run -d -p 8025:8025 -p 1025:1025 axllent/mailpit

# Run the test
go test -v -run TestNotifier_sendExecutionEmail ./dkron
```

Or use the convenient Makefile target:

```bash
make test-email
```

## Viewing Captured Emails

1. Open your browser and navigate to: http://localhost:8025
2. The Mailpit web UI will show all emails sent during testing
3. Click on any email to view its details, including:
   - Subject line
   - Recipients (To, Cc, Bcc)
   - Email body (HTML and plain text)
   - Attachments
   - Raw MIME content
   - Headers
   - Source code

### Web UI Features

Mailpit's web interface includes:
- **Search**: Full-text search across all emails
- **Filtering**: Filter by sender, recipient, subject
- **Responsive Design**: Works on desktop and mobile
- **Dark Mode**: Toggle between light and dark themes
- **Real-time Updates**: Emails appear instantly via WebSocket

## Test Configuration

The test in `dkron/notifier_test.go` uses the following configuration:

```go
c := &Config{
    MailHost:          "localhost",
    MailPort:          1025,
    MailFrom:          "dkron@dkron.io",
    MailSubjectPrefix: "[Test]",
}
```

This configuration:
- Connects to Mailpit's SMTP server at `localhost:1025`
- Sets the sender address to `dkron@dkron.io`
- Adds a `[Test]` prefix to all email subjects
- Uses Mailpit API at `http://localhost:8025` to verify email delivery

### API Verification

The test automatically verifies email delivery using Mailpit's REST API:

```go
// The test performs the following verifications:
// 1. Checks that email was received
// 2. Verifies From address is correct
// 3. Verifies To address matches recipient
// 4. Confirms subject contains expected text
// 5. Validates email body contains execution output
```

This ensures that emails are not just sent, but actually captured correctly by Mailpit.

## Troubleshooting

### Connection Refused Errors

If you see connection errors when running tests:

1. Verify Mailpit is running:
   ```bash
   docker ps | grep mailpit
   ```

2. Check that port 1025 is accessible:
   ```bash
   nc -zv localhost 1025
   ```

3. If using Docker, ensure the ports are properly mapped

### Emails Not Appearing in Web UI

1. Check the Mailpit web UI at http://localhost:8025
2. Verify the test completed successfully without errors
3. Check Mailpit logs for any issues:
   ```bash
   docker logs <mailpit-container-id>
   ```

### Port Already in Use

If port 1025 or 8025 is already in use:

```bash
# Find what's using the port
lsof -i :1025
lsof -i :8025

# Or use different ports
docker run -p 8026:8025 -p 1026:1025 axllent/mailpit
# Then update test configuration to use port 1026
```

## GitHub Actions CI/CD Integration

Mailpit is automatically configured in the GitHub Actions test workflow. The workflow includes Mailpit as a service container, making it available during test execution.

### How It Works

The `.github/workflows/test.yml` file includes Mailpit as a service:

```yaml
services:
    mailpit:
        image: axllent/mailpit
        ports:
            - 1025:1025
            - 8025:8025
```

When tests run in GitHub Actions:
1. Mailpit starts automatically as a service container
2. The SMTP server is available at `localhost:1025`
3. Tests can send emails without any additional configuration
4. No secrets or credentials are needed

### Viewing Emails in CI

Since GitHub Actions runners don't expose web UIs, you cannot view the Mailpit web interface during CI runs. However, you can:

1. Verify emails are sent successfully by checking test results
2. Tests automatically use Mailpit's API to verify email content
3. Email assertions include subject, sender, recipient, and body content

### Testing Locally Before CI

To ensure your tests will pass in GitHub Actions:

```bash
# Start Mailpit locally with the same configuration as CI
docker run -p 1025:1025 -p 8025:8025 axllent/mailpit

# Run the full test suite
go test -v -timeout 200s ./...
```

Or use the validation script:

```bash
./scripts/test-ci-locally.sh
```

## Advantages Over External Services

Mailpit provides several advantages for local development and testing:

- **Free and Open Source**: No account required, runs completely locally
- **No External Dependencies**: Doesn't require internet connection
- **Simple Setup**: Single Docker command to get started
- **Fast**: No network latency from external services
- **Privacy**: All emails stay on your local machine
- **CI/CD Friendly**: Easy to integrate into automated testing pipelines
- **No Secrets Required**: No API keys or credentials needed in CI

## Production Email Configuration

Remember that Mailpit is for **testing only**. For production, configure Dkron with your actual SMTP server credentials:

```yaml
MailHost: smtp.your-provider.com
MailPort: 587
MailUsername: your-username
MailPassword: your-password
MailFrom: noreply@yourdomain.com
```

## Using Mailpit API in Tests

The email notification tests demonstrate how to use Mailpit's API for verification:

### Get All Messages

```go
resp, err := http.Get("http://localhost:8025/api/v1/messages")
```

### Get Specific Message

```go
resp, err := http.Get(fmt.Sprintf("http://localhost:8025/api/v1/message/%s", messageID))
```

### Delete All Messages

```go
req, err := http.NewRequest("DELETE", "http://localhost:8025/api/v1/messages", nil)
resp, err := http.DefaultClient.Do(req)
```

See `dkron/notifier_test.go` for a complete example of API usage in tests.

## Additional Resources

- [Mailpit GitHub Repository](https://github.com/axllent/mailpit)
- [Mailpit Documentation](https://mailpit.axllent.org/)
- [API Documentation](https://mailpit.axllent.org/docs/api-v1/)
- [GitHub Actions Integration Guide](GITHUB_ACTIONS_MAILPIT.md)
- [CI Testing Guide](../.github/TESTING.md)

---

### GITHUB ACTIONS MAILPIT (`docs/GITHUB_ACTIONS_MAILPIT.md`)

# Mailpit Integration with GitHub Actions

This document provides a comprehensive guide on how Mailpit is integrated with GitHub Actions for automated email testing in the Dkron project.

## Overview

Mailpit is used in GitHub Actions as a service container to enable automated testing of email notifications without requiring external email services or credentials. This provides a reliable, fast, and secure way to test email functionality in CI/CD pipelines.

## Why Mailpit for GitHub Actions?

### Advantages

1. **No Secrets Required**: Mailpit doesn't need authentication, eliminating the need to store SMTP credentials in GitHub Secrets
2. **Fast**: Runs locally within the GitHub Actions runner with minimal overhead
3. **Reliable**: No dependency on external services that might have downtime or rate limits
4. **Free**: No costs associated with email testing in CI
5. **Consistent**: Same behavior in local development and CI environments
6. **Secure**: Emails never leave the CI runner, no risk of accidentally sending test emails
7. **Modern**: Active maintenance and regular updates
8. **Lightweight**: Small Docker image (~15MB) for faster CI runs

## GitHub Actions Configuration

### Service Container Setup

In `.github/workflows/test.yml`, Mailpit is configured as a service container:

```yaml
jobs:
    test:
        name: Test
        runs-on: ubuntu-latest
        services:
            mailpit:
                image: axllent/mailpit
                ports:
                    - 1025:1025  # SMTP port
                    - 8025:8025  # Web UI port (not accessible in CI)
        steps:
            # ... test steps
```

### How It Works

1. **Container Start**: When the workflow runs, GitHub Actions automatically starts the Mailpit container
2. **Port Mapping**: Ports 1025 (SMTP) and 8025 (Web UI) are mapped to the runner
3. **Network Access**: Tests can access Mailpit at `localhost:1025`
4. **Automatic Cleanup**: The container is automatically removed when the workflow completes

## Test Configuration

### Email Test Setup

The email notification tests in `dkron/notifier_test.go` are configured to use Mailpit:

```go
func TestNotifier_sendExecutionEmail(t *testing.T) {
    mailHost := "localhost"
    mailPort := 1025  // Mailpit SMTP port
    
    // Check if Mailpit is available
    checkMailpitAvailable(t, mailHost, mailPort)
    
    c := &Config{
        MailHost:          mailHost,
        MailPort:          uint16(mailPort),
        MailFrom:          "dkron@dkron.io",
        MailSubjectPrefix: "[Test]",
    }
    
    // ... test implementation
}
```

### Availability Check

The `checkMailpitAvailable` helper function:
- Attempts to connect to Mailpit on the specified port
- **In CI**: Mailpit is always available, tests run normally
- **Locally**: If Mailpit isn't running, test is skipped (not failed)

```go
func checkMailpitAvailable(t *testing.T, host string, port int) {
    t.Helper()
    address := fmt.Sprintf("%s:%d", host, port)
    conn, err := net.DialTimeout("tcp", address, 2*time.Second)
    if err != nil {
        t.Skipf("Mailpit is not available at %s. Start it with: docker run -p 8025:8025 -p 1025:1025 axllent/mailpit", address)
        return
    }
    conn.Close()
}
```

## Workflow Execution Flow

### 1. Workflow Trigger

Tests run on:
- Every push to any branch
- Every pull request (excluding documentation-only changes)

### 2. Environment Setup

```
┌─────────────────────────────────────┐
│   GitHub Actions Runner (Ubuntu)    │
│                                      │
│  ┌────────────────────────────────┐ │
│  │   Mailpit Container            │ │
│  │   - SMTP: localhost:1025       │ │
│  │   - Web UI: localhost:8025     │ │
│  │   - Image: ~15MB               │ │
│  └────────────────────────────────┘ │
│                                      │
│  ┌────────────────────────────────┐ │
│  │   Go Test Process              │ │
│  │   - Connects to localhost:1025 │ │
│  │   - Sends test emails          │ │
│  └────────────────────────────────┘ │
└─────────────────────────────────────┘
```

### 3. Test Execution

1. Mailpit container starts automatically
2. Go sets up the test environment
3. Tests run with 200-second timeout
4. Email tests connect to Mailpit
5. Emails are captured (but not viewable in CI)
6. Tests verify email sending succeeded
7. Coverage report generated
8. Results uploaded to Codecov

### 4. Cleanup

- Mailpit container automatically stopped and removed
- No manual cleanup required

## Running Tests Locally to Match CI

### Option 1: Using the Validation Script

```bash
./scripts/test-ci-locally.sh
```

This script:
- ✅ Starts Mailpit with the same configuration as CI
- ✅ Runs tests with the same timeout (200s)
- ✅ Generates coverage report
- ✅ Provides interactive access to Mailpit web UI
- ✅ Automatically cleans up after completion

### Option 2: Manual Setup

```bash
# 1. Start Mailpit
docker run -d --rm --name mailpit -p 1025:1025 -p 8025:8025 axllent/mailpit

# 2. Run tests (matching CI configuration)
go test -v -timeout 200s -coverprofile=coverage.txt ./...

# 3. View emails (optional)
open http://localhost:8025

# 4. Cleanup
docker stop mailpit
```

### Option 3: Using Make

```bash
make test-email
```

## Viewing Email in CI

### Limitations

The Mailpit web UI (port 8025) is not accessible from outside the GitHub Actions runner. You cannot view the web interface during CI runs.

### API-Based Verification

The email notification tests automatically use Mailpit's REST API to verify email delivery. This provides comprehensive validation without requiring web UI access:

**Automatic Verification in Tests:**
1. ✅ Email was successfully received by Mailpit
2. ✅ From address matches expected sender
3. ✅ To address matches expected recipient
4. ✅ Subject line contains expected text
5. ✅ Email body contains execution output and details

**Example from `dkron/notifier_test.go`:**

```go
// Get all messages from Mailpit API
messages := getMailpitMessages(t, "http://localhost:8025")

// Find and verify the test email
for _, msg := range messages {
    if strings.Contains(msg.Subject, "[Test]") {
        // Get full message details
        detail := getMailpitMessage(t, apiURL, msg.ID)
        
        // Verify content
        assert.Equal(t, "dkron@dkron.io", detail.From.Address)
        assert.Contains(t, detail.Text, "test-output")
    }
}
```

This approach ensures reliable email testing in CI without manual inspection.

## Troubleshooting

### Tests Fail in CI But Pass Locally

**Possible causes:**
1. Different Go versions
2. Environment-specific configurations
3. Race conditions or timing issues

**Solutions:**
```bash
# Run with the same Go version as CI
go version  # Check against .github/workflows/test.yml

# Run with the same timeout as CI
go test -v -timeout 200s ./...

# Run race detector
go test -race ./...
```

### Mailpit Service Not Starting

**Error**: Tests skip with "Mailpit is not available"

**Diagnosis:**
1. Check workflow file has `services.mailpit` section
2. Verify port mappings are correct (`1025:1025`)
3. Review GitHub Actions logs for container startup errors

**Fix**:
Ensure `.github/workflows/test.yml` contains:
```yaml
services:
    mailpit:
        image: axllent/mailpit
        ports:
            - 1025:1025
            - 8025:8025
```

### Port Conflicts in CI

**Unlikely scenario**: Port 1025 or 8025 already in use on runner

**Solutions:**
- Use different ports in workflow configuration
- Update test configuration to match

### Coverage Upload Fails

**Note**: Coverage upload failures don't fail the build

**Common causes:**
- `CODECOV_TOKEN` secret not set
- Codecov service temporarily unavailable
- Network issues

**Fix**:
1. Verify secret exists in repository settings
2. Check Codecov status page
3. Update codecov-action to latest version

## Best Practices

### 1. Keep Tests Fast

✅ **Do:**
```go
func TestEmail(t *testing.T) {
    // Quick email send test
    err := sendEmail()
    assert.NoError(t, err)
}
```

❌ **Don't:**
```go
func TestEmail(t *testing.T) {
    // Don't poll or wait unnecessarily
    time.Sleep(30 * time.Second)
    // ...
}
```

### 2. Use Helper Functions

```go
// Reusable helper for email testing
func setupEmailTest(t *testing.T) *Config {
    t.Helper()
    checkMailpitAvailable(t, "localhost", 1025)
    return &Config{
        MailHost: "localhost",
        MailPort: 1025,
        // ...
    }
}
```

### 3. Clear Error Messages

```go
if err != nil {
    t.Fatalf("Failed to send email: %v\nCheck that Mailpit is running", err)
}
```

### 4. Skip Gracefully

```go
// Skip test if Mailpit unavailable (local dev)
// Test runs normally in CI (Mailpit always available)
checkMailpitAvailable(t, host, port)
```

## Monitoring and Maintenance

### Workflow Health

Monitor test runs in the GitHub Actions tab:
- Check for consistent pass rates
- Watch for timeout increases
- Monitor for flaky tests

### Mailpit Updates

The `axllent/mailpit` image is pulled automatically by GitHub Actions:
- No manual updates needed
- GitHub caches images for performance
- Updates happen when new workflow runs pull newer images

### Go Version Updates

When updating Go version in workflow:

1. Update `.github/workflows/test.yml`:
   ```yaml
   - uses: actions/setup-go@v6
     with:
       go-version: '1.26.1'  # Update this
   ```

2. Test locally with the same version:
   ```bash
   go version
   ```

3. Update documentation if needed

## Mailpit Features in CI

### Performance Benefits

- **Fast Startup**: Container starts in ~1-2 seconds
- **Low Memory**: Minimal memory footprint during tests
- **Quick Processing**: Emails processed instantly
- **Efficient Storage**: In-memory storage for test duration

### API Usage in Tests

The Dkron tests use Mailpit's API for comprehensive email verification. This is the recommended approach for CI testing:

**Messages API:**
```go
// List all messages
resp, err := http.Get("http://localhost:8025/api/v1/messages")

// Response includes: total count, message list with ID, From, To, Subject
var result mailpitMessagesResponse
json.NewDecoder(resp.Body).Decode(&result)
```

**Message Detail API:**
```go
// Get specific message with full content
resp, err := http.Get("http://localhost:8025/api/v1/message/MESSAGE_ID")

// Response includes: full Text/HTML content, headers, attachments
var detail mailpitMessageDetail
json.NewDecoder(resp.Body).Decode(&detail)
```

**Cleanup API:**
```go
// Delete all messages (useful for test isolation)
req, err := http.NewRequest("DELETE", "http://localhost:8025/api/v1/messages", nil)
http.DefaultClient.Do(req)
```

See [Mailpit API documentation](https://mailpit.axllent.org/docs/api-v1/) for complete API reference.

**Real Example:** Check `dkron/notifier_test.go` for a complete implementation showing how to verify email delivery, content, and metadata using the API.

## Additional Resources

- [GitHub Actions Documentation](https://docs.github.com/en/actions)
- [Service Containers Guide](https://docs.github.com/en/actions/using-containerized-services/about-service-containers)
- [Mailpit GitHub Repository](https://github.com/axllent/mailpit)
- [Mailpit Documentation](https://mailpit.axllent.org/)
- [Email Testing Guide](EMAIL_TESTING.md)
- [CI Testing Guide](../.github/TESTING.md)

## Support

If you encounter issues with Mailpit in GitHub Actions:

1. Check this documentation first
2. Review the [troubleshooting section](#troubleshooting)
3. Run tests locally with `./scripts/test-ci-locally.sh`
4. Check GitHub Actions logs for detailed error messages
5. Review recent workflow changes
6. Open an issue with reproduction steps

## Summary

Mailpit integration with GitHub Actions provides:

- ✅ **Automated email testing** without external dependencies
- ✅ **No secrets or credentials** required in CI
- ✅ **Fast and reliable** test execution with modern tooling
- ✅ **Consistent behavior** between local and CI environments
- ✅ **Zero cost** for email testing
- ✅ **Simple setup** with minimal configuration
- ✅ **Active maintenance** ensuring long-term reliability
- ✅ **Better performance** with smaller Docker image and faster processing

The integration is production-ready and has been tested to work reliably in the Dkron CI/CD pipeline.

---

### MAILPIT API EXAMPLES (`docs/MAILPIT_API_EXAMPLES.md`)

# Mailpit API Usage Examples

This document provides practical examples of using Mailpit's REST API in tests to verify email delivery.

## Overview

Mailpit provides a comprehensive REST API at `http://localhost:8025/api/v1/` that allows programmatic access to captured emails. This is essential for automated testing in CI/CD environments where the web UI is not accessible.

## Base URL

```
http://localhost:8025/api/v1/
```

## API Response Types

### Message Summary

```go
type mailpitMessage struct {
    ID      string           `json:"ID"`
    From    mailpitAddress   `json:"From"`
    To      []mailpitAddress `json:"To"`
    Subject string           `json:"Subject"`
    Snippet string           `json:"Snippet"`
    Created time.Time        `json:"Created"`
}

type mailpitAddress struct {
    Name    string `json:"Name"`
    Address string `json:"Address"`
}

type mailpitMessagesResponse struct {
    Total    int              `json:"total"`
    Messages []mailpitMessage `json:"messages"`
}
```

### Message Detail

```go
type mailpitMessageDetail struct {
    ID      string           `json:"ID"`
    From    mailpitAddress   `json:"From"`
    To      []mailpitAddress `json:"To"`
    Cc      []mailpitAddress `json:"Cc"`
    Bcc     []mailpitAddress `json:"Bcc"`
    Subject string           `json:"Subject"`
    Text    string           `json:"Text"`
    HTML    string           `json:"HTML"`
    Created time.Time        `json:"Created"`
}
```

## Common Operations

### 1. Get All Messages

Retrieve a list of all captured messages.

```go
func getMailpitMessages(t *testing.T, apiURL string) []mailpitMessage {
    t.Helper()

    resp, err := http.Get(apiURL + "/api/v1/messages")
    require.NoError(t, err, "Failed to call Mailpit API")
    defer resp.Body.Close()

    require.Equal(t, http.StatusOK, resp.StatusCode)

    var result mailpitMessagesResponse
    err = json.NewDecoder(resp.Body).Decode(&result)
    require.NoError(t, err, "Failed to decode response")

    return result.Messages
}
```

**Usage:**
```go
messages := getMailpitMessages(t, "http://localhost:8025")
assert.NotEmpty(t, messages, "Expected at least one message")
```

### 2. Get Message Details

Retrieve full details of a specific message including content.

```go
func getMailpitMessage(t *testing.T, apiURL, messageID string) *mailpitMessageDetail {
    t.Helper()

    url := fmt.Sprintf("%s/api/v1/message/%s", apiURL, messageID)
    resp, err := http.Get(url)
    require.NoError(t, err, "Failed to get message detail")
    defer resp.Body.Close()

    require.Equal(t, http.StatusOK, resp.StatusCode)

    var result mailpitMessageDetail
    err = json.NewDecoder(resp.Body).Decode(&result)
    require.NoError(t, err, "Failed to decode message detail")

    return &result
}
```

**Usage:**
```go
detail := getMailpitMessage(t, "http://localhost:8025", messageID)
assert.Contains(t, detail.Text, "expected content")
```

### 3. Search Messages

Search for messages by query string.

```go
func searchMailpitMessages(t *testing.T, apiURL, query string) []mailpitMessage {
    t.Helper()

    url := fmt.Sprintf("%s/api/v1/search?query=%s", apiURL, url.QueryEscape(query))
    resp, err := http.Get(url)
    require.NoError(t, err, "Failed to search messages")
    defer resp.Body.Close()

    require.Equal(t, http.StatusOK, resp.StatusCode)

    var result mailpitMessagesResponse
    err = json.NewDecoder(resp.Body).Decode(&result)
    require.NoError(t, err, "Failed to decode search results")

    return result.Messages
}
```

**Usage:**
```go
// Search by subject
messages := searchMailpitMessages(t, "http://localhost:8025", "subject:test")

// Search by recipient
messages := searchMailpitMessages(t, "http://localhost:8025", "to:user@example.com")
```

### 4. Find Specific Email

Find a specific email by subject or other criteria.

```go
func findEmailBySubject(t *testing.T, apiURL, subjectContains string) *mailpitMessage {
    t.Helper()

    messages := getMailpitMessages(t, apiURL)
    
    for i := range messages {
        if strings.Contains(messages[i].Subject, subjectContains) {
            return &messages[i]
        }
    }
    
    return nil
}
```

**Usage:**
```go
email := findEmailBySubject(t, "http://localhost:8025", "[Test]")
require.NotNil(t, email, "Email with subject containing '[Test]' not found")
```

### 5. Delete All Messages

Clean up all messages (useful for test isolation).

```go
func deleteAllMailpitMessages(t *testing.T, apiURL string) {
    t.Helper()

    req, err := http.NewRequest("DELETE", apiURL+"/api/v1/messages", nil)
    require.NoError(t, err, "Failed to create delete request")

    resp, err := http.DefaultClient.Do(req)
    if err == nil {
        defer resp.Body.Close()
    }
    // Don't fail if delete fails - it's just cleanup
}
```

**Usage:**
```go
// Clean up before test
deleteAllMailpitMessages(t, "http://localhost:8025")

// Run your test
sendTestEmail()

// Clean up after test (optional - can use defer)
defer deleteAllMailpitMessages(t, "http://localhost:8025")
```

### 6. Wait for Email

Wait for an email to arrive (with timeout).

```go
func waitForEmail(t *testing.T, apiURL string, timeout time.Duration) *mailpitMessage {
    t.Helper()

    deadline := time.Now().Add(timeout)
    
    for time.Now().Before(deadline) {
        messages := getMailpitMessages(t, apiURL)
        if len(messages) > 0 {
            return &messages[0]
        }
        time.Sleep(100 * time.Millisecond)
    }
    
    t.Fatal("Timeout waiting for email")
    return nil
}
```

**Usage:**
```go
// Send email
sendTestEmail()

// Wait up to 5 seconds for email to arrive
email := waitForEmail(t, "http://localhost:8025", 5*time.Second)
assert.NotNil(t, email)
```

## Complete Test Example

Here's a complete example showing best practices:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Advanced Examples

### Verify Multiple Recipients

```go
func TestMultipleRecipients(t *testing.T) {
    mailpitAPIURL := "http://localhost:8025"
    deleteAllMailpitMessages(t, mailpitAPIURL)
    
    // Send email to multiple recipients
    err := sendEmail([]string{"user1@example.com", "user2@example.com"}, "Subject", "Body")
    require.NoError(t, err)
    
    time.Sleep(100 * time.Millisecond)
    
    messages := getMailpitMessages(t, mailpitAPIURL)
    require.Len(t, messages, 1, "Expected exactly one email")
    
    detail := getMailpitMessage(t, mailpitAPIURL, messages[0].ID)
    
    // Verify all recipients
    assert.Len(t, detail.To, 2)
    recipients := []string{detail.To[0].Address, detail.To[1].Address}
    assert.Contains(t, recipients, "user1@example.com")
    assert.Contains(t, recipients, "user2@example.com")
}
```

### Verify HTML Email

```go
func TestHTMLEmail(t *testing.T) {
    mailpitAPIURL := "http://localhost:8025"
    deleteAllMailpitMessages(t, mailpitAPIURL)
    
    // Send HTML email
    err := sendHTMLEmail("user@example.com", "Subject", "<h1>Hello</h1><p>World</p>")
    require.NoError(t, err)
    
    time.Sleep(100 * time.Millisecond)
    
    messages := getMailpitMessages(t, mailpitAPIURL)
    require.NotEmpty(t, messages)
    
    detail := getMailpitMessage(t, mailpitAPIURL, messages[0].ID)
    
    // Verify HTML content
    assert.Contains(t, detail.HTML, "<h1>Hello</h1>")
    assert.Contains(t, detail.HTML, "<p>World</p>")
    
    // Verify text fallback exists
    assert.NotEmpty(t, detail.Text)
}
```

### Verify Email Ordering

```go
func TestEmailOrdering(t *testing.T) {
    mailpitAPIURL := "http://localhost:8025"
    deleteAllMailpitMessages(t, mailpitAPIURL)
    
    // Send multiple emails
    sendEmail("user@example.com", "First Email", "Body 1")
    time.Sleep(10 * time.Millisecond)
    sendEmail("user@example.com", "Second Email", "Body 2")
    time.Sleep(100 * time.Millisecond)
    
    messages := getMailpitMessages(t, mailpitAPIURL)
    require.Len(t, messages, 2)
    
    // Messages are typically returned newest first
    assert.Contains(t, messages[0].Subject, "Second Email")
    assert.Contains(t, messages[1].Subject, "First Email")
}
```

### Test Email Not Sent

```go
func TestEmailNotSent(t *testing.T) {
    mailpitAPIURL := "http://localhost:8025"
    deleteAllMailpitMessages(t, mailpitAPIURL)
    
    // Perform action that should NOT send email
    err := performActionWithoutEmail()
    require.NoError(t, err)
    
    time.Sleep(100 * time.Millisecond)
    
    // Verify no emails were sent
    messages := getMailpitMessages(t, mailpitAPIURL)
    assert.Empty(t, messages, "Expected no emails to be sent")
}
```

## Helper Utilities

### Check Mailpit Availability

```go
func isMailpitAvailable(t *testing.T, apiURL string) bool {
    t.Helper()
    
    client := &http.Client{Timeout: 2 * time.Second}
    resp, err := client.Get(apiURL + "/api/v1/messages")
    if err != nil {
        return false
    }
    defer resp.Body.Close()
    
    return resp.StatusCode == http.StatusOK
}
```

### Count Messages

```go
func countMessages(t *testing.T, apiURL string) int {
    t.Helper()
    
    messages := getMailpitMessages(t, apiURL)
    return len(messages)
}
```

### Assert Email Count

```go
func assertEmailCount(t *testing.T, apiURL string, expected int) {
    t.Helper()
    
    messages := getMailpitMessages(t, apiURL)
    assert.Len(t, messages, expected, "Unexpected number of emails")
}
```

## Best Practices

### 1. Always Clean Up Before Tests

```go
// Clean up at the start of each test
deleteAllMailpitMessages(t, mailpitAPIURL)
```

This ensures test isolation and prevents false positives from previous test runs.

### 2. Add Small Delays

```go
// Send email
sendEmail()

// Give Mailpit time to process
time.Sleep(100 * time.Millisecond)

// Now verify
messages := getMailpitMessages(t, mailpitAPIURL)
```

Mailpit is fast, but a small delay ensures the email is fully processed.

### 3. Use Helper Functions

Create reusable helper functions for common operations:

```go
func requireEmailWithSubject(t *testing.T, apiURL, subject string) *mailpitMessageDetail {
    t.Helper()
    
    email := findEmailBySubject(t, apiURL, subject)
    require.NotNil(t, email, "Email with subject '%s' not found", subject)
    
    return getMailpitMessage(t, apiURL, email.ID)
}
```

### 4. Log Useful Information

```go
t.Logf("Email verified:")
t.Logf("  ID: %s", detail.ID)
t.Logf("  From: %s", detail.From.Address)
t.Logf("  To: %s", detail.To[0].Address)
t.Logf("  Subject: %s", detail.Subject)
```

This helps with debugging when tests fail.

### 5. Check API Availability

```go
if !isMailpitAvailable(t, mailpitAPIURL) {
    t.Skip("Mailpit not available. Start with: docker run -p 8025:8025 -p 1025:1025 axllent/mailpit")
}
```

Skip tests gracefully when Mailpit is not running locally.

## Troubleshooting

### API Returns 404

- Verify Mailpit is running: `docker ps | grep mailpit`
- Check the correct port (8025 for API, not 1025 for SMTP)
- Ensure URL includes `/api/v1/` path

### No Messages Found

- Add a delay after sending email: `time.Sleep(100 * time.Millisecond)`
- Verify email was actually sent (check for send errors)
- Check Mailpit logs: `docker logs <container-id>`
- Ensure you're not checking before the email arrives

### Message ID Invalid

- Don't reuse message IDs between tests
- Clean up messages at the start of tests
- Message IDs are only valid until messages are deleted

## Resources

- [Mailpit API Documentation](https://mailpit.axllent.org/docs/api-v1/)
- [Real Implementation Example](../dkron/notifier_test.go)
- [Email Testing Guide](EMAIL_TESTING.md)
- [GitHub Actions Integration](GITHUB_ACTIONS_MAILPIT.md)

## See Also

- `dkron/notifier_test.go` - Complete working example in the Dkron codebase
- [Mailpit GitHub](https://github.com/axllent/mailpit) - Official repository
- [REST API Reference](https://mailpit.axllent.org/docs/api-v1/) - Complete API documentation

---

### MAILPIT QUICK START (`docs/MAILPIT_QUICK_START.md`)

# Mailpit Quick Start Guide

Quick reference for getting started with Mailpit email testing in Dkron.

## TL;DR

```bash
# Start Mailpit
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit

# Run tests
make test-email

# View emails
open http://localhost:8025
```

## What is Mailpit?

Mailpit is a modern email testing tool that captures outgoing emails without sending them.

**Key Features:**
- 🚀 Fast and lightweight (~15MB Docker image)
- 🎨 Modern UI with dark mode
- 🔍 Full-text search
- 📱 Responsive design
- ⚡ Real-time updates via WebSocket
- 🔓 No authentication required

## Installation

### Option 1: Docker (Recommended)

```bash
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit
```

### Option 2: Docker Compose

```bash
docker compose -f docker-compose.dev.yml up mailpit
```

### Option 3: Using Make

```bash
make test-email
```

## Ports

- **1025** - SMTP server (send emails here)
- **8025** - Web UI (view emails here)

## Running Tests

### Run Email Tests

```bash
# Ensure Mailpit is running, then:
go test -v -run TestNotifier_sendExecutionEmail ./dkron
```

**Note:** Tests automatically verify email delivery using Mailpit's REST API, checking sender, recipient, subject, and content.

### Run All Tests (CI Mode)

```bash
./scripts/test-ci-locally.sh
```

This simulates GitHub Actions CI environment.

## Web Interface

Open http://localhost:8025 in your browser to:

- View all captured emails
- Search emails by subject, sender, recipient
- Toggle dark mode
- Inspect headers and raw MIME
- Delete or mark emails

**API Access:** Tests use the REST API at `http://localhost:8025/api/v1/` for automated verification.

## Configuration

Dkron is pre-configured to use Mailpit:

```go
MailHost: "localhost"
MailPort: 1025
// No authentication required
```

## GitHub Actions

Mailpit runs automatically in CI. No configuration needed!

The workflow includes Mailpit as a service container:

```yaml
services:
    mailpit:
        image: axllent/mailpit
        ports:
            - 1025:1025
            - 8025:8025
```

## Common Commands

```bash
# Start Mailpit
docker run -d --name mailpit -p 8025:8025 -p 1025:1025 axllent/mailpit

# Stop Mailpit
docker stop mailpit

# Remove Mailpit
docker rm mailpit

# View logs
docker logs mailpit

# Restart Mailpit
docker restart mailpit
```

## Troubleshooting

### Port already in use

```bash
# Find what's using the port
lsof -i :1025
lsof -i :8025

# Use different ports
docker run -p 8026:8025 -p 1026:1025 axllent/mailpit
```

### Mailpit not starting

```bash
# Check if Docker is running
docker ps

# Pull latest image
docker pull axllent/mailpit

# Check container logs
docker logs mailpit
```

### Tests skip with "Mailpit not available"

```bash
# Verify Mailpit is running
docker ps | grep mailpit

# Check connection
curl -I http://localhost:8025
nc -zv localhost 1025
```

## API Usage

Mailpit provides a REST API for automation and verification. The Dkron email tests use this API to verify email delivery:

```bash
# List all messages
curl http://localhost:8025/api/v1/messages

# Search messages
curl http://localhost:8025/api/v1/search?query=test

# Get specific message
curl http://localhost:8025/api/v1/message/{id}

# Delete all messages (cleanup)
curl -X DELETE http://localhost:8025/api/v1/messages
```

**Test Integration:** See `dkron/notifier_test.go` for a complete example of using the API to verify:
- Email was received
- Sender/recipient addresses
- Subject line content
- Email body text

See [API docs](https://mailpit.axllent.org/docs/api-v1/) for complete reference.

## Next Steps

- 📖 Read full guide: [docs/EMAIL_TESTING.md](EMAIL_TESTING.md)
- 🤖 CI integration: [docs/GITHUB_ACTIONS_MAILPIT.md](GITHUB_ACTIONS_MAILPIT.md)
- 🧪 CI testing: [.github/TESTING.md](../.github/TESTING.md)

## Resources

- [Mailpit GitHub](https://github.com/axllent/mailpit)
- [Mailpit Docs](https://mailpit.axllent.org/)
- [API Reference](https://mailpit.axllent.org/docs/api-v1/)

## Support

If you encounter issues:

1. Check this guide
2. Review [troubleshooting](#troubleshooting) section
3. Run `./scripts/test-ci-locally.sh` to verify setup
4. Check [full documentation](EMAIL_TESTING.md)

---

**Quick Help:**

```bash
# Fresh start
docker stop mailpit && docker rm mailpit
docker run -d --name mailpit -p 8025:8025 -p 1025:1025 axllent/mailpit
make test-email
open http://localhost:8025
```

---

### Architecture (`website/docs/architecture.md`)

---
sidebar_position: 2
---

# Architecture

This document provides a comprehensive overview of Dkron's architecture, internal components, and how they work together to provide a distributed, fault-tolerant job scheduling system.

## System Architecture

Dkron utilizes a server-agent architecture where multiple agents can form a cluster for high availability. Here's a high-level overview of how Dkron works:

```mermaid
flowchart TD
    Node1["Dkron Server Node 1<br/>(Leader)"] <--> Node2["Dkron Server Node 2<br/>(Follower)"]
    Node1 <--> Node3["Dkron Server Node 3<br/>(Follower)"]
    Node1 <--> Node4["Dkron Server Node 4<br/>(Follower)"]
    Node2 <--> Node3
    Node2 <--> Node4
    Node3 <--> Node4
```

## Key Components

Dkron's architecture consists of several key components:

1. **Server Nodes**: Nodes running in server mode (with `--server` flag) that participate in leader election and can schedule jobs.

2. **Leader Node**: One server node is elected as leader and is responsible for:
   - Scheduling jobs
   - Assigning job executions to target nodes
   - Maintaining the cluster state

3. **Follower Nodes**: Server nodes that are not the leader. They:
   - Can execute jobs when selected as targets
   - Are ready to become the leader if the current leader fails
   - Maintain a replicated log of all operations

4. **Embedded Data Store**: Dkron uses dual embedded databases:
   - **BuntDB** for application data (job definitions, execution history)
   - **BoltDB** for Raft consensus log (operation replication)

5. **Serf Layer**: Handles cluster membership, failure detection, and messaging between nodes using the gossip protocol.

6. **HTTP API and Web UI**: Provides a RESTful API and web interface for job management.

7. **Executors**: Plugins that handle the actual execution of job commands (shell, HTTP, etc.).

8. **Processors**: Plugins that process the output of job executions (log, file, email, etc.).

## Dkron Scheduling Flow

Here's how job scheduling works in Dkron:

```mermaid
flowchart LR
    A[Job Definition] --> B[Leader Scheduling]
    B --> C[Target Selection]
    C --> D[Execution]
    D --> E[Processing Output]
    E --> F[Results Storage]
```

1. **Job Definition**: Users define jobs with scheduling parameters (cron expression), execution options, and target node tags.
2. **Leader Scheduling**: The leader node tracks job schedules and triggers executions at the appropriate times.
3. **Target Selection**: The leader selects target nodes for job execution based on tags and execution options.
4. **Execution**: Target nodes run the job using the specified executor.
5. **Processing Output**: Job output is processed by configured processors.
6. **Results Storage**: Execution results are stored in the distributed data store.

## Job Execution Flow

This diagram illustrates the detailed function call flow when a job is executed in Dkron. It shows the interaction between three main components: the Agent Server (which receives requests), the Leader Server (which orchestrates scheduling), and the Worker Agent (which executes jobs).

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Component Details

### Agent Server

- Entry point for HTTP/gRPC requests
- Routes job execution requests to the leader
- Can forward requests if not the leader

### Leader Server (Raft Leader)

- Elected via Raft consensus protocol
- Orchestrates all job scheduling via the `Scheduler`
- Selects target nodes based on job tags
- Coordinates execution across worker nodes
- Applies state changes to Raft log for consistency
- Stores execution results in BuntDB

### Worker Agent

- Executes jobs assigned by the leader
- Streams execution progress back to leader in real-time
- Can be any node in the cluster (including the leader)
- Uses executor plugins for actual job execution

### Storage Layer

- **BuntDB**: Embedded key-value store for application data (jobs, executions)
- **BoltDB**: Raft log storage for consensus and replication
- Application data is replicated across all server nodes via Raft
- Fast, embedded storage with no external dependencies

### Executor Plugins

- Pluggable execution backends (shell, HTTP, Kafka, NATS, etc.)
- Implement the actual job execution logic
- Stream output back to the worker agent during execution

### Communication Protocols

- **HTTP/REST API** for external clients
- **gRPC** with bidirectional streaming for inter-node communication
- **Serf** for cluster membership and failure detection
- **Raft** for consensus and state replication

## Leadership and Fault Tolerance

Dkron uses a leader-follower model for high availability:

### Leader Responsibilities

- Monitors leadership status through `monitorLeadership()` 
- When elected, calls `establishLeadership()` to initialize duties
- Starts the `Scheduler` to begin scheduling jobs
- Reconciles cluster members via Serf

### Fault Tolerance

- If the leader fails, Raft automatically elects a new leader from follower nodes
- The new leader takes over scheduling without job loss
- Running jobs continue to completion on their worker nodes
- Serf detects the failure and updates cluster membership
- State is preserved through Raft's replicated log

## Consensus and Replication

Dkron uses the **Raft consensus algorithm** (via HashiCorp Raft library) to ensure data consistency across the cluster:

### How Raft Works in Dkron

1. **Leader Election**: Nodes vote to elect a leader. The leader handles all writes.
2. **Log Replication**: The leader replicates all state changes to follower nodes.
3. **Commitment**: Changes are committed only when a majority of nodes have replicated them.
4. **Safety**: Once committed, changes are durable and will survive failures.

### What Gets Replicated

- Job definitions (create, update, delete)
- Job execution results
- Configuration changes
- System state

### Quorum Requirements

- Minimum 3 server nodes recommended for production (allows 1 failure)
- 5 server nodes allow 2 failures
- Quorum = (N/2) + 1 where N is the number of servers

## Cluster Membership with Serf

**Serf** provides cluster membership and failure detection through a gossip protocol:

### Serf Responsibilities

- **Membership**: Track which nodes are part of the cluster
- **Failure Detection**: Quickly detect when nodes fail or become unreachable
- **Event Propagation**: Broadcast custom events across the cluster
- **Node Metadata**: Share node information (tags, roles, etc.)

### Gossip Protocol Benefits

- Scales to thousands of nodes efficiently
- Eventually consistent membership information
- Low network overhead
- Fast failure detection (typically within seconds)

## Data Storage

### BuntDB (Application Data)

Dkron uses BuntDB as the primary application data store:

- **Key-Value Store**: Simple, fast embedded database
- **ACID Transactions**: Ensures data consistency
- **Indexes**: Fast lookups for jobs and executions
- **Storage Limits**: Keeps last 100 executions per job
- **File-Based**: Data stored in a single file (`dkron.db`)

### BoltDB (Raft Log)

Raft operations are stored in BoltDB:

- **Append-Only Log**: Records all state changes
- **Snapshots**: Periodic snapshots to reduce log size
- **Replication**: Log entries are replicated to all servers
- **Recovery**: Allows nodes to catch up after downtime

## Plugin System

Dkron uses **HashiCorp's go-plugin** framework for a flexible plugin architecture:

### Plugin Types

1. **Executors**: Define how jobs are executed
   - Built-in: shell, http, grpc, kafka, nats, rabbitmq, gcppubsub
   - Custom: Users can write their own executors

2. **Processors**: Define how job output is processed
   - Built-in: log, files, syslog, fluent
   - Custom: Users can write their own processors

### Plugin Communication

- Plugins run as separate processes
- Communication via gRPC
- Language-agnostic (can be written in any language with gRPC support)
- Isolated from main process (plugin crashes don't affect Dkron)

## Network Ports

Default ports used by Dkron:

| Port | Protocol | Purpose |
|------|----------|---------|
| 8080 | HTTP | REST API and Web UI |
| 8946 | TCP/UDP | Serf gossip protocol |
| 8190 | TCP | Raft consensus |
| 6868 | gRPC | Inter-node communication |

## Scalability Considerations

### Horizontal Scaling

- **Server Nodes**: 3-5 servers recommended for most deployments
- **Agent Nodes**: Can scale to hundreds or thousands of nodes
- **Job Distribution**: Use tags to distribute load across agents

### Vertical Scaling

- **Leader Node**: May need more resources as job count increases
- **Memory**: Increases with number of jobs and execution history
- **CPU**: Leader uses CPU for scheduling; agents use CPU for execution

### Performance Limits

- **Jobs**: Can handle thousands of job definitions
- **Execution Frequency**: Avoid sub-second schedules for many jobs
- **Cluster Size**: Tested with hundreds of nodes (agent nodes)
- **Concurrent Executions**: Limited by agent resources, not Dkron itself

## Security Architecture

### Network Security

- TLS support for HTTP API
- Encryption for Serf gossip traffic
- Raft communication can be secured with TLS

### Access Control

- Basic authentication for HTTP API
- ACL system available in Pro version
- API token support

### Data Security

- Encryption at rest (optional, filesystem-level)
- Encryption in transit (TLS)
- Secret management through environment variables or external systems

## High Availability Setup

Recommended HA configuration:

```
┌─────────────────────────────────────────┐
│         Load Balancer (Optional)        │
└─────────────────────────────────────────┘
           │              │              │
    ┌──────┴───┐   ┌─────┴────┐   ┌─────┴────┐
    │ Server 1 │   │ Server 2 │   │ Server 3 │
    │ (Leader) │   │(Follower)│   │(Follower)│
    └──────────┘   └──────────┘   └──────────┘
         │              │              │
    ┌────┴──────────────┴──────────────┴────┐
    │         Serf Gossip Network            │
    └─────────────────────────────────────────┘
         │              │              │
    ┌────┴───┐   ┌─────┴────┐   ┌─────┴────┐
    │Agent 1 │   │ Agent 2  │   │ Agent 3  │
    └────────┘   └──────────┘   └──────────┘
```

### Best Practices

1. Deploy servers across multiple availability zones
2. Use odd number of servers (3 or 5)
3. Monitor leader election events
4. Set up proper backup procedures
5. Use health checks and alerting
6. Plan for disaster recovery

## Design Philosophy

Dkron is designed to solve one problem well: executing commands at given intervals. Following the Unix philosophy of doing one thing and doing it well (like the battle-tested cron), but with the addition of being designed for the cloud era, removing single points of failure in environments where scheduled jobs need to be run across multiple servers.

### Design Principles

- **Simplicity**: Easy to deploy and operate
- **Reliability**: No single points of failure
- **Scalability**: Handle thousands of nodes and jobs
- **Observability**: Built-in metrics and logging
- **Extensibility**: Plugin system for custom functionality

---

### Faq (`website/docs/faq.md`)

# Frequently Asked Questions (FAQ)

This FAQ addresses the most common issues and questions reported by Dkron users based on GitHub issues and community discussions.

## Performance and Scalability

### Why is my Dkron cluster experiencing performance degradation with high-frequency jobs?

**Issue**: Leader node crashes or becomes unresponsive when running many jobs with frequent schedules (e.g., every second or minute).

**Solutions**:

- Avoid scheduling jobs too frequently (overlapping on the second). Consider batching operations or using a queue system for very high-frequency tasks.
- Offload logs to stdout/file using processors.
- Monitor memory usage and ensure adequate resources are allocated to the leader node.
- Distribute jobs across multiple agent nodes using tags to reduce load on the leader.
- Review the number of concurrent jobs and adjust concurrency policies accordingly.
- Consider upgrading to a more powerful server for the leader node in large-scale deployments.
- Consider upgrading to Dkron Pro using Raft fastlog

**Related**: [Issue #1828](https://github.com/distribworks/dkron/issues/1828), [Issue #1842](https://github.com/distribworks/dkron/issues/1842)

### Does Dkron provide performance benchmark data?

Dkron does not currently provide official benchmark data. Performance varies significantly based on:

- Job frequency and complexity
- Cluster size and configuration
- Hardware resources
- Network latency
- Type of executor used

For high-frequency scheduling scenarios, it's recommended to conduct your own benchmarking in an environment similar to your production setup.

**Related**: [Issue #1839](https://github.com/distribworks/dkron/issues/1839)

## Memory Issues

### Why is Dkron consuming excessive memory or experiencing OOM kills?

**Common causes**:

1. **Memory leaks**: Older versions had memory leak issues that have been addressed in recent releases.
2. **High execution count**: Storing too many execution records (Dkron keeps up to 100 executions per job).
3. **Large job output**: Jobs that produce large amounts of output can consume significant memory.
4. **Too many jobs**: Running thousands of jobs simultaneously increases memory usage.

**Solutions**:

- Update to the latest version of Dkron (4.x series has several memory improvements).
- Use the `files` processor to write job output to disk instead of storing it in memory.
- Limit job output size in executor configurations.
- Monitor memory usage with Prometheus metrics.
- Set appropriate memory limits in Docker/Kubernetes deployments.
- Reduce the number of concurrent jobs or distribute them across more agents.

**Related**: [Issue #1753](https://github.com/distribworks/dkron/issues/1753), [Issue #1566](https://github.com/distribworks/dkron/issues/1566), [Issue #1553](https://github.com/distribworks/dkron/issues/1553)

## Clustering and High Availability

### Why is my agent unable to connect to the cluster?

**Common causes**:

- Network connectivity issues
- Firewall blocking required ports (default: 8946 for Serf, 8190 for Raft)
- Incorrect `retry-join` configuration
- DNS resolution problems
- Encryption key mismatch

**Solutions**:

- Verify network connectivity between nodes using `telnet` or `nc`.
- Check firewall rules allow traffic on ports 8946 (Serf) and 8190 (Raft).
- Use IP addresses instead of hostnames in `retry-join` if DNS is unreliable.
- Ensure all nodes use the same encryption key if encryption is enabled.
- Check logs for specific error messages: `level=error ts=<timestamp> caller=agent.go:xxx`

**Related**: [Issue #1727](https://github.com/distribworks/dkron/issues/1727), [Issue #1702](https://github.com/distribworks/dkron/issues/1702)

### Why does retry-join fail after a server restart?

If using DNS names in `retry-join`, Dkron may cache the old IP addresses. Solutions:

- Use static IP addresses in `retry-join` configuration.
- Ensure DNS updates propagate before restarting nodes.
- Set appropriate DNS TTL values for dynamic environments.

**Related**: [Issue #1253](https://github.com/distribworks/dkron/issues/1253), [Issue #1213](https://github.com/distribworks/dkron/issues/1213)

### How do I recover from "no leader" state after restart?

In single-node mode:

```bash
# Check Raft peers
dkron raft list-peers

# If cluster state is corrupted, backup and remove the data directory
cp -r /var/lib/dkron /var/lib/dkron.backup
rm -rf /var/lib/dkron/*
# Restart dkron
```

In multi-node mode:

- Ensure at least `bootstrap-expect` number of servers are running.
- Check that all servers can communicate on the Raft port (8190).
- Verify there's no network partition.
- Review logs for Raft election errors.

**Related**: [Issue #1529](https://github.com/distribworks/dkron/issues/1529), [Issue #1212](https://github.com/distribworks/dkron/issues/1212)

### Why do servers have inconsistent job data?

**Causes**:

- Network partitions causing split-brain scenarios
- Raft replication issues
- Corrupted data on one or more nodes

**Solutions**:

- Ensure stable network connectivity between all server nodes.
- Verify Raft replication is working: `dkron raft list-peers`
- Check logs for Raft errors: `grep -i raft /var/log/dkron.log`
- In extreme cases, restore from backup or rebuild the cluster.

**Related**: [Issue #1371](https://github.com/distribworks/dkron/issues/1371), [Issue #1337](https://github.com/distribworks/dkron/issues/1337)

## Kubernetes and Container Deployments

### Why does one Dkron pod fail with "panic: log not found" in Kubernetes?

This typically occurs during Raft initialization in StatefulSets when persistent volumes are not properly configured.

**Solutions**:

- Ensure persistent volumes are correctly configured and bound.
- Verify `bootstrap-expect` matches the number of server replicas.
- Check that all pods can resolve each other's DNS names.
- Use `StatefulSet` with headless service for stable network identities.
- Ensure proper initialization order with `podManagementPolicy: Parallel` or readiness probes.

**Related**: [Issue #1557](https://github.com/distribworks/dkron/issues/1557)

### How do I configure log rotation in Docker?

Dkron writes logs to stdout/stderr by default in Docker. To manage logs:

**Docker Compose**:

```yaml
services:
    dkron:
        logging:
            driver: "json-file"
            options:
                max-size: "10m"
                max-file: "3"
```

**Docker CLI**:

```bash
docker run --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  distribworks/dkron:latest
```

To mount logs to the host:

```yaml
volumes:
    - ./logs:/var/log/dkron
```

Then configure Dkron to write to files via `log-level` and redirect output.

**Related**: [Issue #1556](https://github.com/distribworks/dkron/issues/1556)

## Job Execution

### Why does my shell job fail to execute?

**Common causes**:

1. Shell not available in the container/environment
2. Incorrect command syntax
3. Missing environment variables
4. Permission issues

**Solutions**:

- Verify shell is available: `which sh` or `which bash`
- Use absolute paths for commands: `/bin/bash -c "command"`
- Set the `shell` option explicitly in executor config:
    ```json
    {
        "executor": "shell",
        "executor_config": {
            "command": "your-command",
            "shell": "true"
        }
    }
    ```
- Check file permissions and ownership
- Review execution logs for specific error messages

**Related**: [Issue #1775](https://github.com/distribworks/dkron/issues/1775)

### Why does "forbid" concurrency policy not prevent concurrent executions?

In distributed environments, there can be a race condition where multiple executions start before the concurrency check completes. This is a known limitation of distributed scheduling.

**Workarounds**:

- Implement application-level locking (e.g., using Redis, etcd, or database locks).
- Increase job duration to reduce the likelihood of overlapping schedules.
- Use idempotent job designs that can handle concurrent executions safely.
- Add a small delay at job start to check for other running instances.

**Related**: [Issue #1569](https://github.com/distribworks/dkron/issues/1569)

### Why does `cmd.Process.Kill()` not clean up child processes?

On Unix systems, killing the parent process doesn't automatically kill child processes. This can leave orphaned processes running.

**Solution**:
Use process groups to ensure all child processes are terminated:

```bash
# In your script, create a process group
set -m  # Enable job control
trap 'kill -- -$$' EXIT INT TERM

# Your commands here
```

Or use the `timeout` command:

```bash
timeout 300 your-long-running-command
```

**Related**: [Issue #1385](https://github.com/distribworks/dkron/issues/1385)

## Job Configuration

### Why does `runoncreate` block the API call?

When `runoncreate` is set to `true`, the job creation API waits for the first execution to complete before returning. This is intentional behavior for validation purposes.

**Workaround**:
If you need immediate API response:

1. Create the job with `runoncreate: false`
2. Make a separate API call to trigger execution: `POST /v1/jobs/{job_name}/run`

**Related**: [Issue #1268](https://github.com/distribworks/dkron/issues/1268)

## Web UI

### Why does the UI show "Empty output" for running jobs?

The UI displays execution output after the job completes. For long-running jobs, output is not streamed in real-time.

**Solutions**:

- Wait for job completion to see output
- Use the `log` or `files` processor to write output to a log aggregation system
- Use the API to fetch execution details: `GET /v1/executions/{execution_id}`
- Check agent logs directly on the executing node

**Related**: [Issue #1742](https://github.com/distribworks/dkron/issues/1742)

### Why does a running job show "failed" status with strange finished_at date?

This is a UI display bug that has been addressed in recent versions. The job is still running, but the UI incorrectly shows it as failed.

**Solution**:

- Update to the latest version (4.x)
- Check actual job status via API: `GET /v1/jobs/{job_name}`
- Monitor execution status: `GET /v1/executions/{execution_id}`

**Related**: [Issue #1817](https://github.com/distribworks/dkron/issues/1817)

### Why is the job ID missing on small screens?

This was a responsive design issue in the Web UI that has been fixed in version 4.x.

**Solution**:
Update to the latest version of Dkron.

**Related**: [Issue #1583](https://github.com/distribworks/dkron/issues/1583)

## Plugins and Executors

### Why doesn't the GRPC executor close connections?

Older versions had connection leak issues. Solutions:

- Update to the latest version
- Implement connection pooling in your gRPC service
- Set appropriate timeout values in executor config

**Related**: [Issue #1236](https://github.com/distribworks/dkron/issues/1236)

## API

### Why does the restore API not work?

The restore endpoint requires proper backup format and permissions.

**Usage**:

```bash
# Create backup
curl http://localhost:8080/v1/backup > backup.json

# Restore backup
curl -X POST http://localhost:8080/v1/restore \
  -H "Content-Type: application/json" \
  -d @backup.json
```

**Note**: Restore should be performed on a clean cluster or during maintenance windows.

**Related**: [Issue #1376](https://github.com/distribworks/dkron/issues/1376)

## Configuration and Setup

### How do I set up Dkron in a dual data center configuration?

Dkron uses Raft which requires a majority of nodes to be available. For multi-datacenter setups:

**Option 1: Single Raft cluster across DCs** (not recommended)

- High latency can cause leader election issues
- Network partitions can cause outages

**Option 2: Separate clusters per DC** (recommended)

- Run independent Dkron clusters in each DC
- Use external synchronization for job definitions
- Configure failover at the application level

**Option 3: Use Pro version features**

- Enhanced replication capabilities
- Better multi-datacenter support

**Related**: [Issue #1363](https://github.com/distribworks/dkron/issues/1363)

### What are the resource requirements for Dkron?

Requirements vary based on:

- Number of jobs
- Job execution frequency
- Cluster size
- Execution history retention

**Minimum recommendations**:

- **Server nodes**: 1-2 CPU cores, 512MB-1GB RAM
- **Agent nodes**: 1 CPU core, 256MB-512MB RAM

**Production recommendations**:

- **Server nodes**: 2-4 CPU cores, 2-4GB RAM
- **Agent nodes**: 2 CPU cores, 1-2GB RAM
- **Storage**: 10GB+ for execution history

Scale vertically for the leader node and horizontally for agents.

**Related**: [Issue #1560](https://github.com/distribworks/dkron/issues/1560)

## Troubleshooting

### How do I debug "invalid memory address or nil pointer dereference" errors?

This panic typically indicates a bug in Dkron or a plugin. Steps to debug:

1. **Update to the latest version** - Many nil pointer issues have been fixed
2. **Check logs** for the full stack trace
3. **Identify the context** - What operation was being performed?
4. **Reproduce** - Can you consistently trigger the issue?
5. **Report** - If it persists, file a GitHub issue with:
    - Dkron version
    - Configuration
    - Steps to reproduce
    - Full logs and stack trace

**Related**: [Issue #1388](https://github.com/distribworks/dkron/issues/1388), [Issue #1265](https://github.com/distribworks/dkron/issues/1265)

### Where can I find more help?

- **Documentation**: https://dkron.io/docs
- **GitHub Issues**: https://github.com/distribworks/dkron/issues
- **Discussions**: https://github.com/distribworks/dkron/discussions
- **Commercial Support**: Available for Dkron Pro - see https://dkron.io/pro

## Best Practices

### General recommendations for production deployments:

1. **High Availability**: Run at least 3 server nodes (odd number for Raft quorum)
2. **Monitoring**: Enable Prometheus metrics and set up alerts
3. **Backups**: Regularly backup job configurations using the backup API
4. **Updates**: Keep Dkron updated to get bug fixes and performance improvements
5. **Resource Limits**: Set appropriate memory/CPU limits in container environments
6. **Job Design**:
    - Use idempotent operations
    - Implement proper error handling
    - Keep job execution time reasonable (< 1 hour)
    - Avoid very high-frequency schedules (< 1 minute)
7. **Security**:
    - Enable TLS for API communication
    - Use firewall rules to restrict access
    - Consider using ACLs (Pro version)
8. **Logging**: Use processors (log, files, syslog) to persist job output
9. **Testing**: Test job configurations in a staging environment first
10. **Documentation**: Document your job configurations and dependencies

---

### Basics Configuration (`website/docs/basics/configuration.md`)

# Configuration
## Configuration sources

Settings can be specified in three ways (in order of precedence): 

1. Command line arguments.
1. Environment variables starting with **`DKRON_`**
1. **`dkron.yml`** config file

:::caution
Dkron sends anonymous usage data to a server with the purpose of elaborating usage statistics, if you want to disable statistics collection, you can disable it in the dkron config file or in the command line using `--disable-usage-stats` parameter
:::

## Config file location

Config file will be loaded from the following paths:

- `/etc/dkron`
- `$HOME/.dkron`
- `./config`

### Config file example

```yaml
# Dkron example configuration file
# server: false
# bootstrap-expect: 3
# data-dir: dkron.data
# log-level: debug
# tags:
#   dc: east
# encrypt: a-valid-key-generated-with-dkron-keygen
# retry-join:
#   - 10.0.0.1
#   - 10.0.0.2
#   - 10.0.0.3
# raft-multiplier: 1
# webhook-url: https://hooks.slack.com/services/XXXXXX/XXXXXXX/XXXXXXXXXXXXXXXXXXXX
# webhook-payload: "payload={\"text\": \"{{.Report}}\", \"channel\": \"#foo\"}"
# webhook-headers: Content-Type:application/x-www-form-urlencoded
# mail-host: email-smtp.eu-west-1.amazonaws.com
# mail-port: 25
# mail-username": mailuser
# mail-password": mailpassword
# mail-from": cron@example.com
# mail-subject_prefix: [Dkron]
# mail-payload: "Job {{.JobName}} finished with status {{.Success}} on node {{.NodeName}}\n\nOutput:\n{{.Output}}"
```

## Important Configuration Parameters

Here's a detailed explanation of the key configuration parameters:

### Core Configuration

| Parameter | Description |
|-----------|-------------|
| `server` | When set to `true`, the agent runs in server mode and participates in leader election. Default: `false` |
| `bootstrap-expect` | Number of expected servers in the cluster. When this many servers are available, Dkron will bootstrap the cluster and start a leader election. This helps automate cluster formation. |
| `data-dir` | Directory for storing server-specific data including the replicated log. Default: `dkron.data` |
| `log-level` | Log verbosity level: `debug`, `info`, `warn`, `error`, `fatal`, `panic`. Default: `info` |

### Networking and Cluster Formation

| Parameter | Description |
|-----------|-------------|
| `bind-addr` | Address to bind for network services. Uses go-sockaddr template format. Default: `{{ GetPrivateIP }}:8946` |
| `http-addr` | Address to bind the UI web server to. Format: `:8080` |
| `advertise-addr` | Address advertised to other nodes. Uses the bind address by default. |
| `advertise-rpc-port` | RPC port advertised to clients. Uses `rpc-port` by default. |
| `rpc-port` | Port used for RPC communication with clients. Default: `6868` |
| `join` | Initial agent to join with. Can be specified multiple times. |
| `retry-join` | Address of agent to join with retries. Can be specified multiple times. |
| `retry-interval` | Time to wait between join attempts. Default: `30s` |
| `retry-max` | Maximum number of join attempts. `0` means retry indefinitely. |
| `encrypt` | Key for encrypting network traffic. Must be a base64-encoded 16-byte key generated by `dkron keygen` |

### Node Configuration

| Parameter | Description |
|-----------|-------------|
| `node-name` | Name of this node. Must be unique in the cluster. Default: system hostname |
| `datacenter` | Data center of the local agent. All members of a datacenter should share a LAN connection. Default: `dc1` |
| `region` | Region the agent belongs to. A region typically maps to a geographic region. Default: `global` |
| `tag` | Attach key/value tag pairs to the node. Format: `key=value`. Can be specified multiple times. |

### Performance and Scaling

| Parameter | Description |
|-----------|-------------|
| `raft-multiplier` | Integer multiplier for Raft timing parameters. Lower values increase sensitivity to failures but require more resources. 1 is high-performance, 10 is max. Default: `1` |
| `profile` | Controls timing profiles. Options: `lan`, `wan`, etc. Default: `lan` |
| `serf-reconnect-timeout` | Time to attempt reconnection to failed nodes before removing them. Default: `24h` |

### Notification Configuration

| Parameter | Description |
|-----------|-------------|
| `webhook-endpoint` | Endpoint URL to call for webhook notifications. |
| `webhook-payload` | Body of the POST request to send when calling the webhook. |
| `webhook-headers` | Headers to use when calling the webhook URL. Can be specified multiple times. |
| `mail-host` | Mail server hostname for sending notifications. |
| `mail-port` | Mail server port. |
| `mail-username` | Username for mail server authentication. |
| `mail-password` | Password for mail server authentication. |
| `mail-from` | From email address to use in notifications. |
| `mail-subject-prefix` | Prefix for the email subject line. Default: `[Dkron]` |
| `mail-payload` | Custom Go template for the email body. If not set, defaults to the execution output. Available template variables: `{{.Report}}`, `{{.JobName}}`, `{{.ReportingNode}}`, `{{.StartTime}}`, `{{.FinishedAt}}`, `{{.Success}}`, `{{.NodeName}}`, `{{.Output}}` |

### Metrics Configuration

| Parameter | Description |
|-----------|-------------|
| `statsd-addr` | Address of a statsd server for metrics collection. |
| `dog-statsd-addr` | Address of a DataDog statsd agent. |
| `dog-statsd-tags` | Tags to include with DataDog metrics. Format: `key:value` |
| `enable-prometheus` | When `true`, enables serving Prometheus metrics at the `/metrics` endpoint. |

## Environment Variables

All configuration options can also be specified as environment variables by:

1. Converting to uppercase
2. Replacing dashes with underscores
3. Prefixing with `DKRON_`

Example: `server: true` becomes `DKRON_SERVER=true`

### SEE ALSO

* [dkron agent](/docs/cli/dkron_agent/)	 - Start a dkron agent
* [dkron doc](/docs/cli/dkron_doc/)	 - Generate Markdown documentation for the Dkron CLI.
* [dkron keygen](/docs/cli/dkron_keygen/)	 - Generates a new encryption key
* [dkron version](/docs/cli/dkron_version/)	 - Show version

---

### Basics Installation (`website/docs/basics/installation.md`)

# Installation

This guide covers different methods of installing Dkron for development, testing, and production environments.

## Quick Start Methods

### Running the binary

Download the packaged archive for your platform from the [downloads page](https://github.com/distribworks/dkron/releases) and extract the package to a shared location in your drive, like /opt/local/bin.

Run Dkron with default setting: `dkron agent --server --bootstrap-expect=1`

Navigate to http://localhost:8080/ui

### Installing from package repositories

#### Debian/Ubuntu

APT repository:

```
deb [trusted=yes] https://repo.dkron.io/apt/ /
```

Setup and install:

```bash
# Add the repository
echo "deb [trusted=yes] https://repo.dkron.io/apt/ /" | sudo tee /etc/apt/sources.list.d/dkron.list

# Update package list
sudo apt-get update

# Install Dkron
sudo apt-get install dkron
```

#### RHEL/CentOS/Fedora

YUM repository:

```
[dkron]
name=Dkron Repository
baseurl=https://repo.dkron.io/yum/
enabled=1
gpgcheck=0
```

Setup and install:

```bash
# Create repo file
sudo tee /etc/yum.repos.d/dkron.repo << EOF
[dkron]
name=Dkron Repository
baseurl=https://repo.dkron.io/yum/
enabled=1
gpgcheck=0
EOF

# Install Dkron
sudo yum install dkron
```

After package installation, Dkron will be started as a system service and an example configuration file will be placed under `/etc/dkron/dkron.yml`.

## Running in Docker

Dkron provides official Docker images via Docker Hub that can be used for deployment on any system running Docker.

:::info
If you only plan to use the built-in executors, `http` and `shell` you can use the Dkron Light edition that only includes a single binary as the plugins are built-in.
:::

### Launching Dkron as a new container

Here's a quick one-liner to get you off the ground (please note, we recommend further configuration for production deployments below):

```
docker run -d -p 8080:8080 --name dkron dkron/dkron agent --server --bootstrap-expect=1 --node-name=node1
```

This will launch a Dkron server on port 8080 by default. You can use `docker logs -f dkron` to follow the rest of the initialization progress. Once the Dkron startup completes you can access the app at localhost:8080

Since Docker containers have their own ports and we just map them to the system ports as needed it's easy to move Dkron onto a different system port if you wish. For example running Dkron on port 12345:

```
docker run -d -p 12345:8080 --name dkron dkron/dkron agent --server --bootstrap-expect=1 --node-name=node1
```

### Docker Image Variants

Dkron offers two Docker image variants:

1. **Standard Image** (`dkron/dkron`): Includes all executors and processors
2. **Light Image** (`dkron/dkron:light`): Contains only the main binary with built-in shell and HTTP executors

### Mounting a mapped file storage volume

Dkron uses the local filesystem for storing the embedded database to store its own application data and the Raft protocol log. The end result is that your Dkron data will be on disk inside your container and lost if you ever remove the container.

To persist your data outside of the container and make it available for use between container launches we can mount a local path inside our container.

```
docker run -d -p 8080:8080 -v ~/dkron.data:/dkron.data --name dkron dkron/dkron agent --server --bootstrap-expect=1 --data-dir=/dkron.data
```

Now when you launch your container we are mounting that folder from our local filesystem into the container.

### Running a Dkron cluster in Docker Compose

For development or simple multi-node setups, Docker Compose offers an easy way to run a Dkron cluster:

```yaml
version: "3"
services:
  dkron-leader:
    image: dkron/dkron
    ports:
      - "8080:8080"
    command: agent --server --bootstrap-expect=3 --node-name=leader --retry-join=dkron-follower-1 --retry-join=dkron-follower-2 --data-dir=/dkron.data
    volumes:
      - leader-data:/dkron.data

  dkron-follower-1:
    image: dkron/dkron
    command: agent --server --node-name=follower1 --retry-join=dkron-leader --retry-join=dkron-follower-2 --data-dir=/dkron.data
    volumes:
      - follower1-data:/dkron.data
    depends_on:
      - dkron-leader

  dkron-follower-2:
    image: dkron/dkron
    command: agent --server --node-name=follower2 --retry-join=dkron-leader --retry-join=dkron-follower-1 --data-dir=/dkron.data
    volumes:
      - follower2-data:/dkron.data
    depends_on:
      - dkron-leader

volumes:
  leader-data:
  follower1-data:
  follower2-data:
```

## Production Deployment Considerations

When deploying Dkron in production, consider these best practices:

### System Requirements

- **CPU**: Minimum 2 cores recommended, more for heavy workloads
- **Memory**: Minimum 1GB RAM, 2GB+ recommended for production
- **Disk**: SSD recommended for better performance, especially for the data directory
- **Network**: Low-latency network between cluster nodes

### Cluster Size

- **Minimum Nodes**: 3 server nodes for high availability
- **Recommended**: Use an odd number of server nodes (3, 5, 7) to avoid split-brain scenarios
- **Region Distribution**: For multi-region deployments, use at least 3 servers per region

### Security Recommendations

1. **Network Security**:
   - Restrict access to Dkron API/UI using firewalls
   - Run behind a reverse proxy with TLS termination
   - Use VPN or private network for inter-node communication

2. **Authentication**:
   - Use Dkron Pro for built-in authentication features
   - Consider implementing a reverse proxy with authentication

3. **Encryption**:
   - Enable network encryption with `--encrypt` parameter using a key generated with `dkron keygen`

### Data Directory Considerations

- Place the data directory on persistent, high-performance storage
- Implement regular backups of this directory
- Monitor disk space usage

### Configuration Example for Production

Example production configuration file (`/etc/dkron/dkron.yml`):

```yaml
# Server settings
server: true
bootstrap-expect: 3
data-dir: /var/lib/dkron
log-level: info

# Network settings
bind-addr: "{{ GetPrivateIP }}:8946"
http-addr: ":8080"
advertise-addr: "{{ GetPublicIP }}"
encrypt: "YOUR_ENCRYPTION_KEY_GENERATED_WITH_DKRON_KEYGEN"

# Node identification
node-name: "dkron-prod-1"
datacenter: "dc1"
region: "us-west"

# Clustering
retry-join:
  - "10.0.1.10"
  - "10.0.1.11"
  - "10.0.1.12"
retry-interval: "30s"
raft-multiplier: 1

# Optional metrics for monitoring
statsd-addr: "localhost:8125"
enable-prometheus: true

# Tags for node selection
tags:
  role: "backend"
  env: "production"
```

### Monitoring and Operations

1. **Health Monitoring**:
   - Set up health checks against the API endpoint
   - Monitor node status and leader elections
   - Configure metrics collection with Prometheus or StatsD

2. **Log Management**:
   - Configure log collection and aggregation
   - Set appropriate log levels (`info` for production, `debug` for troubleshooting)

3. **Backup Strategy**:
   - Regular backups of the data directory
   - Document and test restoration procedures

## Next Steps

After installation, follow the [Getting Started guide](/docs/basics/getting-started) to learn how to create and manage jobs with Dkron.

---

### Cli Dkron (`website/docs/cli/dkron.md`)

---
date: 2022-06-05
title: "dkron"
slug: dkron
url: /cli/dkron/
---
## dkron

Open source distributed job scheduling system

### Synopsis

Dkron is a system service that runs scheduled jobs at given intervals or times,
just like the cron unix service but distributed in several machines in a cluster.
If a machine fails (the leader), a follower will take over and keep running the scheduled jobs without human intervention.

### Options

```
      --config string   config file path
  -h, --help            help for dkron
```

### SEE ALSO

* [dkron agent](/docs/cli/dkron_agent/)	 - Start a dkron agent
* [dkron completion](/docs/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell
* [dkron doc](/docs/cli/dkron_doc/)	 - Generate Markdown documentation for the Dkron CLI.
* [dkron keygen](/docs/cli/dkron_keygen/)	 - Generates a new encryption key
* [dkron leave](/docs/cli/dkron_leave/)	 - Force an agent to leave the cluster
* [dkron raft](/docs/cli/dkron_raft/)	 - Command to perform some raft operations
* [dkron version](/docs/cli/dkron_version/)	 - Show version

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Agent (`website/docs/cli/dkron_agent.md`)

---
date: 2022-06-05
title: "dkron agent"
slug: dkron_agent
url: /cli/dkron_agent/
---
## dkron agent

Start a dkron agent

### Synopsis

Start a dkron agent that schedules jobs, listens for executions and runs executors.
It also runs a web UI.

```
dkron agent [flags]
```

### Options

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron](/docs/cli/dkron/)	 - Open source distributed job scheduling system

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Completion (`website/docs/cli/dkron_completion.md`)

---
date: 2022-06-05
title: "dkron completion"
slug: dkron_completion
url: /cli/dkron_completion/
---
## dkron completion

Generate the autocompletion script for the specified shell

### Synopsis

Generate the autocompletion script for dkron for the specified shell.
See each sub-command's help for details on how to use the generated script.


### Options

```
  -h, --help   help for completion
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron](/docs/cli/dkron/)	 - Open source distributed job scheduling system
* [dkron completion bash](/docs/cli/dkron_completion_bash/)	 - Generate the autocompletion script for bash
* [dkron completion fish](/docs/cli/dkron_completion_fish/)	 - Generate the autocompletion script for fish
* [dkron completion powershell](/docs/cli/dkron_completion_powershell/)	 - Generate the autocompletion script for powershell
* [dkron completion zsh](/docs/cli/dkron_completion_zsh/)	 - Generate the autocompletion script for zsh

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Completion Bash (`website/docs/cli/dkron_completion_bash.md`)

---
date: 2022-06-05
title: "dkron completion bash"
slug: dkron_completion_bash
url: /cli/dkron_completion_bash/
---
## dkron completion bash

Generate the autocompletion script for bash

### Synopsis

Generate the autocompletion script for the bash shell.

This script depends on the 'bash-completion' package.
If it is not installed already, you can install it via your OS's package manager.

To load completions in your current shell session:

```
	source <(dkron completion bash)
```

To load completions for every new session, execute once:

#### Linux:

	dkron completion bash > /etc/bash_completion.d/dkron

#### macOS:

	dkron completion bash > /usr/local/etc/bash_completion.d/dkron

You will need to start a new shell for this setup to take effect.


```
dkron completion bash
```

### Options

```
  -h, --help              help for bash
      --no-descriptions   disable completion descriptions
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron completion](/docs/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Completion Fish (`website/docs/cli/dkron_completion_fish.md`)

---
date: 2022-06-05
title: "dkron completion fish"
slug: dkron_completion_fish
url: /cli/dkron_completion_fish/
---
## dkron completion fish

Generate the autocompletion script for fish

### Synopsis

Generate the autocompletion script for the fish shell.

To load completions in your current shell session:

	dkron completion fish | source

To load completions for every new session, execute once:

	dkron completion fish > ~/.config/fish/completions/dkron.fish

You will need to start a new shell for this setup to take effect.


```
dkron completion fish [flags]
```

### Options

```
  -h, --help              help for fish
      --no-descriptions   disable completion descriptions
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron completion](/docs/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Completion Powershell (`website/docs/cli/dkron_completion_powershell.md`)

---
date: 2022-06-05
title: "dkron completion powershell"
slug: dkron_completion_powershell
url: /cli/dkron_completion_powershell/
---
## dkron completion powershell

Generate the autocompletion script for powershell

### Synopsis

Generate the autocompletion script for powershell.

To load completions in your current shell session:

	dkron completion powershell | Out-String | Invoke-Expression

To load completions for every new session, add the output of the above command
to your powershell profile.


```
dkron completion powershell [flags]
```

### Options

```
  -h, --help              help for powershell
      --no-descriptions   disable completion descriptions
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron completion](/docs/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Completion Zsh (`website/docs/cli/dkron_completion_zsh.md`)

---
date: 2022-06-05
title: "dkron completion zsh"
slug: dkron_completion_zsh
url: /cli/dkron_completion_zsh/
---
## dkron completion zsh

Generate the autocompletion script for zsh

### Synopsis

Generate the autocompletion script for the zsh shell.

If shell completion is not already enabled in your environment you will need
to enable it.  You can execute the following once:

	echo "autoload -U compinit; compinit" >> ~/.zshrc

To load completions for every new session, execute once:

#### Linux:

	dkron completion zsh > "~/.zsh/completion/_dkron"

#### macOS:

	dkron completion zsh > /usr/local/share/zsh/site-functions/_dkron

You will need to start a new shell for this setup to take effect.


```
dkron completion zsh [flags]
```

### Options

```
  -h, --help              help for zsh
      --no-descriptions   disable completion descriptions
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron completion](/docs/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Doc (`website/docs/cli/dkron_doc.md`)

---
date: 2022-06-05
title: "dkron doc"
slug: dkron_doc
url: /cli/dkron_doc/
---
## dkron doc

Generate Markdown documentation for the Dkron CLI.

### Synopsis

Generate Markdown documentation for the Dkron CLI.
This command is, mostly, used to create up-to-date documentation
of Dkron's command-line interface for http://dkron.io/.
It creates one Markdown file per command with front matter suitable
for rendering in Hugo.

```
dkron doc [flags]
```

### Options

```
      --dir string   the directory to write the doc. (default "/tmp/dkrondoc/")
  -h, --help         help for doc
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron](/docs/cli/dkron/)	 - Open source distributed job scheduling system

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Keygen (`website/docs/cli/dkron_keygen.md`)

---
date: 2022-06-05
title: "dkron keygen"
slug: dkron_keygen
url: /cli/dkron_keygen/
---
## dkron keygen

Generates a new encryption key

### Synopsis

Generates a new encryption key that can be used to configure the
  agent to encrypt traffic. The output of this command is already
  in the proper format that the agent expects.

```
dkron keygen [flags]
```

### Options

```
  -h, --help   help for keygen
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron](/docs/cli/dkron/)	 - Open source distributed job scheduling system

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Leave (`website/docs/cli/dkron_leave.md`)

---
date: 2022-06-05
title: "dkron leave"
slug: dkron_leave
url: /cli/dkron_leave/
---
## dkron leave

Force an agent to leave the cluster

### Synopsis

Stop stops an agent, if the agent is a server and is running for election
	stop running for election, if this server was the leader
	this will force the cluster to elect a new leader and start a new scheduler.

```
dkron leave [flags]
```

### Options

```
  -h, --help              help for leave
      --rpc-addr string   gRPC address of the agent (default "{{ GetPrivateIP }}:6868")
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron](/docs/cli/dkron/)	 - Open source distributed job scheduling system

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Raft (`website/docs/cli/dkron_raft.md`)

---
date: 2022-06-05
title: "dkron raft"
slug: dkron_raft
url: /cli/dkron_raft/
---
## dkron raft

Command to perform some raft operations

### Options

```
  -h, --help              help for raft
      --rpc-addr string   gRPC address of the agent. (default "{{ GetPrivateIP }}:6868")
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron](/docs/cli/dkron/)	 - Open source distributed job scheduling system
* [dkron raft list-peers](/docs/cli/dkron_raft_list-peers/)	 - Command to list raft peers
* [dkron raft remove-peer](/docs/cli/dkron_raft_remove-peer/)	 - Command to list raft peers

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Raft List Peers (`website/docs/cli/dkron_raft_list-peers.md`)

---
date: 2022-06-05
title: "dkron raft list-peers"
slug: dkron_raft_list-peers
url: /cli/dkron_raft_list-peers/
---
## dkron raft list-peers

Command to list raft peers

```
dkron raft list-peers [flags]
```

### Options

```
  -h, --help   help for list-peers
```

### Options inherited from parent commands

```
      --config string     config file path
      --rpc-addr string   gRPC address of the agent. (default "{{ GetPrivateIP }}:6868")
```

### SEE ALSO

* [dkron raft](/docs/cli/dkron_raft/)	 - Command to perform some raft operations

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Raft Remove Peer (`website/docs/cli/dkron_raft_remove-peer.md`)

---
date: 2022-06-05
title: "dkron raft remove-peer"
slug: dkron_raft_remove-peer
url: /cli/dkron_raft_remove-peer/
---
## dkron raft remove-peer

Command to list raft peers

```
dkron raft remove-peer [flags]
```

### Options

```
  -h, --help             help for remove-peer
      --peer-id string   Remove a Dkron server with the given ID from the Raft configuration.
```

### Options inherited from parent commands

```
      --config string     config file path
      --rpc-addr string   gRPC address of the agent. (default "{{ GetPrivateIP }}:6868")
```

### SEE ALSO

* [dkron raft](/docs/cli/dkron_raft/)	 - Command to perform some raft operations

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Cli Dkron Version (`website/docs/cli/dkron_version.md`)

---
date: 2022-06-05
title: "dkron version"
slug: dkron_version
url: /cli/dkron_version/
---
## dkron version

Show version

### Synopsis

Show the version

```
dkron version [flags]
```

### Options

```
  -h, --help   help for version
```

### Options inherited from parent commands

```
      --config string   config file path
```

### SEE ALSO

* [dkron](/docs/cli/dkron/)	 - Open source distributed job scheduling system

###### Auto generated by spf13/cobra on 5-Jun-2022

---

### Pro Acls (`website/docs/pro/acls.md`)

# Access Control Lists

Dkron provides an optional Access Control List (ACL) system which can be used to control access to data and APIs. The ACL is capability-based, relying on policies to determine which fine-grained rules can be applied. Dkron's capability-based ACL system is very similar to common ACL systems you are used to.

## ACL System Overview

Dkron's ACL system is implemented with the CNCF [Open Policy Agent](https://www.openpolicyagent.org/) bringing a powerful system to suit your needs.

The ACL system is designed to be easy to use and fast to enforce while providing administrative insight. At the highest level, there are two major components to the ACL system:

* **OPA policy engine.** OPA provides policy decision-making [decoupling](https://www.openpolicyagent.org/docs/latest/philosophy/#policy-decoupling). Dkron integrates OPA as a library and provides default policy rules written in the OPA Policy language. These rules enforce permissions on request parameters to the `/v1` API paths and are ready to use for most cases. You do not need to learn the OPA Policy language to start using Dkron's ACL system, but you can modify the default policy rules to adapt to your use case if needed. Read more in the [OPA docs](https://www.openpolicyagent.org/docs/latest/).

* **ACL Policies.** Dkron's ACL policies are simple JSON documents that define patterns to allow access to resources. You can find below an example ACL policy that works with the default OPA policy. The ACL JSON structure is not rigid; you can adapt it to add new features in combination with the OPA Policy rules.

## Tutorial

### Configuring ACLs

ACLs are not enabled by default and must be enabled. To enable ACLs, set up the `acl` section in your config file:

```yaml
acl:
  enabled: true
```

Below you can find the most basic example of an ACL policy:

Basic example policy:

```json
{
    "path": {
        "/v1": {
            "capabilities": [
                "read"
            ]
        },
        "/v1/**": {
            "capabilities": [
                "create",
                "read",
                "update",
                "delete",
                "list"
            ]
        }
    }
}
```

This policy allows requests to the `/v1` API paths. Paths use glob patterns, and capabilities allow operations on resources.

This is a much more granular policy file used as default policy:

```json
{
    "path": {
        "/v1/members": {
            "capabilities": ["read"]
        },
        "/v1/jobs": {
            "capabilities": [
                "list",
                "read",
                "create",
                "update",
                "delete"
            ]
        },
        "/v1/jobs/*": {
            "capabilities": [
                "create",
                "read",
                "update",
                "delete"
            ]
        },            
        "/v1/jobs/*/run": {
            "capabilities": ["create"]
        },
        "/v1/jobs/*/toggle": {
            "capabilities": ["create"]
        },
        "/v1/jobs/*/executions*": {
            "capabilities": ["read"]
        },
        "/v1/jobs/*/executions/*": {
            "capabilities": ["read"]
        },
        "/v1/leader": {
            "capabilities": ["read"]
        },
        "/v1/isleader": {
            "capabilities": ["read"]
        },
        "/v1/leave": {
            "capabilities": ["create"]
        },
        "/v1/restore": {
            "capabilities": ["create"]
        },
        "/v1/busy": {
            "capabilities": ["read"]
        }
    }
}
```

### Setup

The first step after enabling the ACL system and restarting the service is to bootstrap the management token.

:::note
The `dkron acl bootstrap` command currently expects the `policies/default.json` file to be present in the working directory. Ensure you run this command from the Dkron project root or a directory where the `policies/` folder is available.
:::

Run this command to create the initial management token:

```bash
dkron acl bootstrap
```

:::info
Adjust your `--rpc-addr` param if necessary.
:::

After this step you will obtain the details of the management token, something similar to:
```
Accessor ID: fc4a83e5-4657-4c18-92b0-723d7c5f6c1f
Secret: ca40c646-4a86-425d-ae55-b27fdd99d8c4
Name: bootstrap
Type: management
CreateTime: 2024-10-06 11:03:36.605368 +0000 UTC
Policies: default
```

From this point, you must use the "Secret" to communicate with the Dkron API. Management tokens can administer ACL policies, while client tokens are constrained by their assigned policies.

When making API requests, include the secret in the `Authorization` header:

```bash
curl -H "Authorization: Bearer <Secret>" http://localhost:8080/v1/jobs
```

If you navigate to your Dkron installation, it should show the login page:



Enter the secret and click on "Sign in", after that you should be able to use the UI without limitations.

Check the full documentation on all the available [ACL management commands](cli/dkron_acl).

### Use the readonly policy

```json
{
    "path": {
        "/v1/members": {
            "capabilities": ["read"]
        },
        "/v1/jobs": {
            "capabilities": [
                "list",
                "read"
            ]
        },
        "/v1/jobs/*": {
            "capabilities": [
                "read"
            ]
        },            
        "/v1/jobs/*/executions*": {
            "capabilities": ["read"]
        },
        "/v1/jobs/*/executions/*": {
            "capabilities": ["read"]
        },
        "/v1/leader": {
            "capabilities": ["read"]
        },
        "/v1/isleader": {
            "capabilities": ["read"]
        },
        "/v1/busy": {
            "capabilities": ["read"]
        }
    }
}
```

Save the JSON above to a local file named `readonly.json`. The policy document must have `path` at its top level; do not wrap it in a `rules` object.

Create or update the `readonly` policy from that file. The `--rules-file` flag belongs to `acl policy apply`:

```bash
dkron acl policy apply --name readonly --rules-file ./readonly.json
```


Then create a client token and associate the existing `readonly` policy with it. `acl token create` accepts `--policy`, but it does not accept `--rules-file`:

Clients using this token will be able to perform `GET` requests, but any write attempts will return a `403 Forbidden` error:

```bash
dkron acl token create --name alice --type client --policy readonly
```

Provide the token details to the user securely.

## Inspecting job contents in OPA policies

### Policy input

Dkron supplies the following fields to the OPA policy:

| Field | Contents |
| --- | --- |
| `input.method` | HTTP request method, such as `POST`. |
| `input.path` | Request URL path, without query parameters. |
| `input.token` | Authenticated bearer secret. Do not log this value. |
| `input.rules` | The JSON rules from the token's first associated ACL policy. |
| `input.body` | The decoded job on the job-write routes below; an empty object on other routes. |

Job contents are available on these routes:

- `POST /v1/jobs`
- `PATCH /v1/jobs`
- `PUT /v1/jobs/:job`

There is no `PUT /v1/jobs` route. The default OPA policy does not authorize PATCH. If you extend its method-to-capability mapping to permit PATCH, apply the same content restrictions as for POST and PUT.

`input.body` is the decoded job shared with the API handler, rather than the raw JSON document. It uses canonical job field names and includes defaults and zero values. The owner is the authenticated token's accessor, the ID is the job name, and concurrency defaults to `allow`. Unknown job fields are omitted. Sharing the decoded job ensures that authorization and job handling interpret repeated or differently capitalized JSON fields consistently. Normal job validation still runs after authorization.

With ACL enabled, job submissions must contain a single JSON object and are limited to **10 MiB**. Invalid bodies return **400 Bad Request**, oversized bodies return **413 Request Entity Too Large**, and policy denials return **403 Forbidden**.

Other routes retain `input.body == {}`. Their request streams are not parsed or consumed by the ACL middleware, including multipart restore uploads. Restrict access to `/v1/restore` separately if a token must not import arbitrary jobs.

### Adding content restrictions

For example, a policy can inspect both the executor and its command:

```rego
input.body.executor == "shell"
input.body.executor_config.command == "cronexec alice task"
```

These are conditions to use in a rule, not a complete policy. Apply them to job-write requests while retaining your existing method, path, capability, and token checks. Adding them unconditionally to a general rule would also deny reads and other routes whose body is empty.

:::warning Keep content checks in the authorization path
A separate `allow` rule provides an alternative way to authorize a request. Do not add a content-checking `allow` rule alongside an existing rule that already permits unrestricted job writes. Every rule that can authorize a job write must enforce the required restrictions.
:::

Keep user allow-lists in administrator-controlled ACL policy data, rather than in fields supplied with the job. For example, a custom ACL JSON policy can include an `allowed_users` array alongside `path`; your Rego policy can check a parsed wrapper user against `input.rules.allowed_users`. Associate different ACL policies with tokens that need different allow-lists.

When restricting commands to a wrapper such as `cronexec`, validate its argument grammar and execution settings as well as the command string. A prefix check alone is insufficient: enabling shell interpretation can allow additional commands after the wrapper invocation. Account for quoting, `executor_config.shell`, and environment settings that could change execution behavior.

## Debugging OPA policies

Policy `print()` statements are disabled by default. Enable them in the agent configuration:

```yaml
acl:
  enabled: true
  debug: true
```

Restart Dkron after changing the configuration. Policy print messages are emitted through Dkron's logger at **info** level with `component=acl-policy`, so the configured log level must permit info messages. Raising the log level alone does not enable print statements.

Print selected fields while evaluating a job-write rule, for example:

```rego
print("job executor", input.body.executor)
```

Dkron does not automatically dump policy input. Avoid `print(input)`: it includes the bearer secret, and job configuration may contain credentials. Disable `acl.debug` after troubleshooting.

You can also test Rego policies outside Dkron using the OPA CLI with synthetic input. Such tests validate policy logic but do not change which fields an older Dkron binary supplies.

## Disable ACLs

To disable the ACL system, set the configuration value in your Dkron configuration file to `false`:

```yaml
acl:
  enabled: false
```

Restart your system for the change to take effect. After the restart, you should be able to access the system without any restriction.

---

### Pro Commercial Faq (`website/docs/pro/commercial-faq.md`)

---
title: Commercial FAQ
---

### What is Dkron Pro?

Dkron Pro is a flavor of Dkron which add more functionality and provide additional support options for customers.

### Is there a trial version?

There's no free trial but we do offer a 14 day period with full refund if it does not work for you.

### Can I get a discount?

I'm sure you're very nice but no. Everyone pays the same price.

### What is the license?

See [COMM-LICENSE](https://github.com/distribworks/dkron/blob/master/COMM-LICENSE).

### How does Pro licensing work?

Every organization running Dkron Pro on its own servers must have a license. There's no limit to the number of servers or environments used by that organization.

### What happens if my subscription lapses?

You must have an active subscription to run Dkron Pro. After a one week grace period, you'll lose access to the package repository and priority support. You won't get any more updates or bug fixes and `apt-get install dkron-pro` won't work anymore.

### Can I distribute Dkron Pro to my customers?

This is a common requirement for "on-site installs" or "appliances" sold to large corporations.

The standard license is only appropriate for SaaS usage as it does not allow distribution. Dkron Pro have an Appliance license option which **does** allow you to distribute them. The Appliance license is $7,500/yr for Pro. It allows you to distribute the Pro binaries as part of your application and each of your customers to run Dkron Pro **as part of your application only**. Email [support&#64;dkron.io](mailto:support@dkron.io) to purchase.

### Can you transfer a license?

Licenses are **not** transferrable to another company. We will transfer the license from a user-specific email to a group email address (e.g. john_smith@example.com -> tech@example.com) but only for **the same domain**. It is strongly recommended that you buy the license using a group email address so the license is not attached to any one employee's email address.

### What does the license require me to do?

Your purchase gets you unique access credentials for downloading the Pro packages. The license agreement requires you to keep these access credentials private. If we find your access credentials are ever publicized:

1. We'll send you a warning email with details. You need to remove the content and send a new email address so we can generate new credentials for you. The old credentials will stop working immediately so you'll need to update your apps.
2. If your credentials are publicized a second time, we reserve the right to permanently remove access (but won't unless it's really egregious - sloppy contractors happen).

### Can I get a refund?

Yes, up to two weeks after purchase. Let us know the reason and maybe we can help but either way it's not a problem. Email [support&#64;dkron.io](mailto:support@dkron.io).

### How do I update my credit card info?

If you purchased Dkron Pro ([settings](https://gumroad.com/settings)) with a credit card, log into Gumroad with your email address, click the Settings, enter your card info and hit Save. Follow instructions in Gumroad docs https://help.gumroad.com/how-do-i-update-my-credit-card-information I can't provide support for the Gumroad website and don't have the ability to edit customer info - if you can't log in or change your credit card, you can always let your current subscription expire and purchase a new subscription.

## Can I request a change to the license terms?

Dkron Pro customers purchasing the Appliance license can ask for changes to the terms and conditions.

If you don't need the Appliance license but wants to ask for changes to the terms and conditions you will need to purchase a Dkron Pro standard license per production server.

Email your concerns and we can negotiate something.

## Can I pay via invoice and purchase order?

Dkron Pro is credit card only, no exceptions.

## Contact Info

Distributed Works

All billing/support inquiries: support@dkron.io

Phone: not available

---

### Pro Commercial Support (`website/docs/pro/commercial-support.md`)

---
title: Commercial Support
---

Dkron offers only community support. Dkro Pro offers priority support via email.

## Priority Support

Covers 1 incident per quarter, with a max response time of 2 working days. Scope is limited to Dkron and Dkron Pro features and APIs, not application integration or infrastructure. For support, email support AT dkron.io. Please email using the same domain as the original license email or explain your connection to the licensed company.

---

### Pro Configuration (`website/docs/pro/configuration.md`)

# Configuration

Dkron Pro uses the [same parameters](/docs/basics/configuration) as Dkron OSS and add some extra parameters.

### Command line options

* `--username` - Authentication username
* `--password` - Authentication password
* `--cert-file` - Path to the client server TLS cert file
* `--key-file` - Path to the client server TLS key file
* `--client-crl-file` - Path to the client certificate revocation list file
* `--trusted-ca-file` - Path to the client server TLS trusted CA cert file
* `--client-cert-auth` - Enable client cert authentication
* `--auto-tls` - Client TLS using generated certificates
* `--fast` - Enable Raft Fastlog storage engine

---

### Pro Encryption (`website/docs/pro/encryption.md`)

# Encryption

SSL encryption is used for communicating dkron pro nodes. Also if client auth is enabled. This means that no other software running on your local network will be able to talk to dkron servers.

This ensures that no unexpected usage of the Dkron's store will happen, unless it is another Dkron pro instance.

SSL encryption is enabled by default in Dkron Pro and can not be disabled, you don't need to do nothing to use it.

By default Dkron Pro runs with automatically generated SSL certificates, this is enough for using it in a trusted environment but to have a better grade of confidence, it is recommended to run Dkron Pro with custom SSL certificates.

Follow [this tutorial](https://github.com/coreos/docs/blob/master/os/generate-self-signed-certificates.md) to generate autosigned SSL certificates for your instances.

:::info
You don't need a client certificate for Dkron server, just add "client auth" usage to your server cert.
:::

```yaml
# dkron.yaml
auto-tls: false # Set to false to use custom certs
key-file: server-key.pem
cert-file: server.pem
trusted-ca-file: ca.pem
client-cert-auth: true # Enable it to only allow certs signed by the same CA
```

---

### Pro Failover (`website/docs/pro/failover.md`)

# Cross region failover

:::warning
This feature is experimental and should be handled with care.
:::

Dkron Pro can run federated in failover mode, this allows to have two clusters running in different regions and configure one of the clusters in an active-passive fashion, doing a failover in case the active cluster dies.

---

### Pro Raft Fastlog (`website/docs/pro/raft-fastlog.md`)

# Raft FastLog

Dkron Pro supports the [Raft FastLog](https://github.com/tidwall/raft-fastlog) storage engine for Raft state. FastLog keeps the Raft log in memory for fast access while persisting it to disk, which can reduce Raft write latency and improve throughput compared to the default BoltDB-backed Raft log.

## Overview

FastLog is optimized for Raft's append-heavy write pattern. It is a good fit when the default Raft storage becomes a bottleneck for cluster coordination, scheduling activity, or leader-side state changes.

Typical reasons to evaluate FastLog include:

- **Higher Raft throughput** for write-heavy workloads
- **Lower commit latency** for Raft log operations
- **Faster in-memory access** to recent Raft log entries
- **Durability tuning** through the `--raft-duration` setting

FastLog is most useful when you are running busy production clusters, scheduling a high volume of jobs, or trying to reduce the overhead of Raft log persistence.

## Configuration

Enable FastLog with the `--fast` flag when starting Dkron Pro.

### CLI example

```bash
dkron agent --server --bootstrap-expect=3 --fast --raft-duration=0
```

### Environment variables

```bash
DKRON_FAST=true
DKRON_RAFT_DURATION=0
```

### Config file

```yaml
fast: true
raft-duration: 0
```

## Durability and performance tuning

The `--raft-duration` flag controls how aggressively FastLog flushes Raft log data to disk. Lower values favor performance. Higher values favor durability.

| Value | Mode | Behavior | Trade-off |
|-------|------|----------|-----------|
| `-1` | Low | No explicit fsync by FastLog | Highest performance, highest risk of losing recent writes after a crash or power loss |
| `0` | Mid | Fsync approximately once per second | Default setting and the best starting point for most deployments |
| `1` | High | Fsync on every change | Strongest durability, lowest write throughput |

For most clusters, start with `--raft-duration=0`. Move to `1` if you need the strongest durability guarantees, or test `-1` only if you fully understand the crash-recovery trade-off.

## When to use FastLog

FastLog is a good option when:

- You are running a busy production cluster with frequent scheduling activity.
- Raft log persistence is contributing noticeable latency.
- You want more control over the durability/performance trade-off for Raft writes.
- You are scaling the server side of the cluster and want to reduce Raft storage overhead.

If your cluster is small and Raft storage is not a bottleneck, BoltDB may remain the simpler default choice.

## Operational considerations

### Storage format compatibility

FastLog uses a different on-disk format than BoltDB. The Raft storage files are not interchangeable.

:::warning
Do not point a FastLog-enabled node at an existing BoltDB Raft data directory and expect it to reuse the old log files. Treat the switch as a storage migration.
:::

### Migration from BoltDB

Switching an existing cluster from BoltDB to FastLog should be planned as a maintenance operation.

1. Stop all Dkron server nodes in the cluster.
2. Back up the existing `data-dir` on every node.
3. Remove or archive the existing Raft log data.
4. Restart the cluster with `--fast` enabled on every server node.
5. Verify cluster health, leadership, and job scheduling after the cluster comes back.

Because the storage formats differ, you should not mix old BoltDB Raft state with a FastLog deployment.

### Rollout guidance

- **Test in staging first** using a workload that resembles production.
- **Monitor memory usage** because FastLog keeps the Raft log in memory for fast access.
- **Watch disk latency and Raft health** after rollout, especially if you change `--raft-duration`.
- **Keep backups** of your cluster data before changing Raft storage settings.

## Summary

FastLog gives Dkron Pro operators a faster Raft storage backend with configurable durability. Enable it with `--fast`, tune it with `--raft-duration`, and plan migrations carefully because the underlying storage format is different from BoltDB.

---

### Upgrading From V1 To V2 (`website/docs/upgrading/from_v1_to_v2.md`)

---
title: Upgrade from v1 to v2
---

Dkron v2 brings lots of changes to the previous version. To successfully upgrade from v1 to v2 you need to take care of certain steps.

## Storage

Dkron v2 brings native storage, no external storage engine is needed. This change can be shocking for existing users and should be carefully planned to consider Dkron as a stateful service in contrast of a stateless service as in v1.

You must plan a backup strategy for the data directories, you can configure it using the `--data-dir` parameter, by default `./dkron.data`.

This will ensure that you can recover cluster data in case of an unexpected failure.

## Migrating Jobs

To migrate jobs from v1 to v2, export jobs from the v1 cluster and import them into v2.

A basic script to do that can be found [here](https://gist.github.com/pjz/94f4bd81a0897fd64db44593078e2156)

You can take the opportunity to update your job definitions as explained in the following section:

## Change job name

In v2 dkron doesn't accept jobs with invalid naming, adapt your scripts or api calls if necessary to set the job name with valid characters.

## Changing tags and metadata

Tags behave different in dkron 2. There are several dkron tags that changed names and have become reserved. The reserved tags are:

- dc
- region
- role
- version
- server
- bootstrap
- expect
- rpc_addr
- port

You can not set these tags with the `tags` param. Tags `region` and `dc` can be set using the `region` and `datacenter` params, respectively.

In Dkron 2, the job filtering API now filters on the `metadata` instead of the `tags` field. Jobs have a new param `metadata` that can be used to set any data to classify jobs. These can then be used to filter results returned by the API.

---

### Upgrading From V2 0 To V2 2 (`website/docs/upgrading/from_v2_0_to_v2_2.md`)

---
title: Upgrade from v2.0.x to v2.2.x
---

## Migrating Jobs

To migrate jobs from v2.0.x to v2.2.x, export jobs from the v2.0.x cluster and import them into v2.2.x

Refer to the [backup&restore upgrade guide](/docs/usage/upgrade#backup--restore)

---

### Usage Chaining (`website/docs/usage/chaining.md`)

# Job chaining

You can set some jobs to run after other job is executed. To setup a job that will be executed after any other given job, just set the `parent_job` property when saving the new job.

The dependent job will be executed after the main job finished a successful execution.

Child jobs schedule property will be ignored if it's present.

Take into account that parent jobs must be created before any child job.

Example:

```json
{
  "name": "job1",
  "schedule": "@every 10s",
  "executor": "shell",
  "executor_config": {
    "command": "echo \"Hello from parent\""
  }
}

{
  "name": "child_job",
  "parent_job": "job1",
  "executor": "shell",
  "executor_config": {
    "command": "echo \"Hello from child\""
  }
}
```

---

### Usage Cloud Auto Join (`website/docs/usage/cloud-auto-join.md`)

---
description: |-
  Dkron supports automatic cluster joining using cloud metadata on various providers.
---

# Cloud Auto-join

As of Dkron 2.0.0, `retry-join` accepts a unified interface using the
[go-discover](https://github.com/hashicorp/go-discover) library for doing
automatic cluster joining using cloud metadata. To use retry-join with a
supported cloud provider, specify the configuration on the command line or
configuration file as a `key=value key=value ...` string.

If the values contain spaces, equals, backslashes or double quotes then
they need to be double quoted and the usual escaping rules apply.

```sh
$ dkron agent --retry-join 'provider=my-cloud config=val config2="some other val" ...'
```

or via a configuration file:

```yml
retry-join": ["provider=my-cloud config=val config2=\"some other val\" ..."]
```

The cloud provider-specific configurations are detailed below. This can be
combined with static IP or DNS addresses or even multiple configurations
for different providers.

In order to use discovery behind a proxy, you will need to set
`HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables per
[Golang `net/http` library](https://golang.org/pkg/net/http/#ProxyFromEnvironment).

The following sections give the options specific to each supported cloud
provider.

### Amazon EC2

This returns the first private IP address of all servers in the given
region which have the given `tag_key` and `tag_value`.

```sh
$ dkron agent --retry-join "provider=aws tag_key=... tag_value=..."
```

```yml
retry-join: ["provider=aws tag_key=... tag_value=..."]
```

- `provider` (required) - the name of the provider ("aws" in this case).
- `tag_key` (required) - the key of the tag to auto-join on.
- `tag_value` (required) - the value of the tag to auto-join on.
- `region` (optional) - the AWS region to authenticate in.
- `addr_type` (optional) - the type of address to discover: `private_v4`, `public_v4`, `public_v6`. Default is `private_v4`. (>= 1.0)
- `access_key_id` (optional) - the AWS access key for authentication (see below for more information about authenticating).
- `secret_access_key` (optional) - the AWS secret access key for authentication (see below for more information about authenticating).

#### Authentication & Precedence

- Static credentials `access_key_id=... secret_access_key=...`
- Environment variables (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
- Shared credentials file (`~/.aws/credentials` or the path specified by `AWS_SHARED_CREDENTIALS_FILE`)
- ECS task role metadata (container-specific).
- EC2 instance role metadata.

The only required IAM permission is `ec2:DescribeInstances`, and it is
recommended that you make a dedicated key used only for auto-joining. If the
region is omitted it will be discovered through the local instance's [EC2
metadata
endpoint](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html).

### Microsoft Azure

This returns the first private IP address of all servers in the given region
which have the given `tag_key` and `tag_value` in the tenant and subscription, or in
the given `resource_group` of a `vm_scale_set` for Virtual Machine Scale Sets.

```sh
$ dkron agent --retry-join "provider=azure tag_name=... tag_value=... tenant_id=... client_id=... subscription_id=... secret_access_key=..."
```

```yml
retry-join: ["provider=azure tag_name=... tag_value=... tenant_id=... client_id=... subscription_id=... secret_access_key=..."]
```

- `provider` (required) - the name of the provider ("azure" in this case).
- `tenant_id` (required) - the tenant to join machines in.
- `client_id` (required) - the client to authenticate with.
- `secret_access_key` (required) - the secret client key. **NOTE** This value often may have an equals sign in it's value, especially if generated from the Azure Portal, so is important to wrap in single quotes eg. `secret_acccess_key='fpOfcHQJAQBczjAxiVpeyLmX1M0M0KPBST+GU2GvEN4='`

Variables can also be provided by environmental variables:

* `ARM_SUBSCRIPTION_ID` for subscription
* `ARM_TENANT_ID` for tenant
* `ARM_CLIENT_ID` for client
* `ARM_CLIENT_SECRET` for secret access key

Use these configuration parameters when using tags:

- `tag_name` - the name of the tag to auto-join on.
- `tag_value` - the value of the tag to auto-join on.

Use these configuration parameters (instead of `tag_name` and `tag_value`) when using Virtual Machine Scale Sets (Dkron 1.0.3 and later):

- `resource_group` - the name of the resource group to filter on.
- `vm_scale_set` - the name of the virtual machine scale set to filter on.

When using tags the only permission needed is `Microsoft.Network/networkInterfaces`.

When using Virtual Machine Scale Sets the only role action needed is `Microsoft.Compute/virtualMachineScaleSets/*/read`.

### Google Compute Engine

This returns the first private IP address of all servers in the given
project which have the given `tag_value`.

```sh
$ dkron agent --retry-join "provider=gce project_name=... tag_value=..."
```

```yml
retry_join: ["provider=gce project_name=... tag_value=..."]
```

- `provider` (required) - the name of the provider ("gce" in this case).
- `tag_value` (required) - the value of the tag to auto-join on.
- `project_name` (optional) - the name of the project to auto-join on. Discovered if not set.
- `zone_pattern` (optional) - the list of zones can be restricted through an RE2 compatible regular expression. If omitted, servers in all zones are returned.
- `credentials_file` (optional) - the credentials file for authentication. Note, if you set `-config-dir` do not store the credentials.json file in the configuration directory as it will be parsed as a config file and Dkron will fail to start. See below for more information.

#### Authentication & Precedence

- Use credentials from `credentials_file`, if provided.
- Use JSON file from `GOOGLE_APPLICATION_CREDENTIALS` environment variable.
- Use JSON file in a location known to the gcloud command-line tool.
    - On Windows, this is `%APPDATA%/gcloud/application_default_credentials.json`.
    - On other systems, `$HOME/.config/gcloud/application_default_credentials.json`.
- On Google Compute Engine, use credentials from the metadata
    server. In this final case any provided scopes are ignored.

Discovery requires a [GCE Service
Account](https://cloud.google.com/compute/docs/access/service-accounts).
Credentials are searched using the following paths, in order of precedence.

### IBM SoftLayer

This returns the first private IP address of all servers for the given
datacenter with the given `tag_value`.

```sh
$ dkron agent --retry-join "provider=softlayer datacenter=... tag_value=... username=... api_key=..."
```

```yml
retry-join: ["provider=softlayer datacenter=... tag_value=... username=... api_key=..."]
```

- `provider` (required) - the name of the provider ("softlayer" in this case).
- <a name="sl_datacenter"></a><a href="#sl_datacenter"><code>datacenter</code></a> (required) - the name of the datacenter to auto-join in.
- `tag_value` (required) - the value of the tag to auto-join on.
- `username` (required) - the username to use for auth.
- `api_key` (required) - the api key to use for auth.

### Aliyun (Alibaba Cloud)

This returns the first private IP address of all servers for the given
`region` with the given `tag_key` and `tag_value`.

```sh
$ dkron agent --retry-join "provider=aliyun region=... tag_key=dkron tag_value=... access_key_id=... access_key_secret=..."
```

```yml
retry-join: ["provider=aliyun region=... tag_key=dkron tag_value=... access_key_id=... access_key_secret=..."]
```

- `provider` (required) - the name of the provider ("aliyun" in this case).
- `region` (required) - the name of the region.
- `tag_key` (required) - the key of the tag to auto-join on.
- `tag_value` (required) - the value of the tag to auto-join on.
- `access_key_id` (required) -the access key to use for auth.
- `access_key_secret` (required) - the secret key to use for auth.

The required RAM permission is `ecs:DescribeInstances`.
It is recommended you make a dedicated key used only for auto-joining.

### Digital Ocean

This returns the first private IP address of all servers for the given
`region` with the given `tag_name`.

```sh
$ dkron agent --retry-join "provider=digitalocean region=... tag_name=... api_token=..."
```

```yml
retry-join: ["provider=digitalocean region=... tag_name=... api_token=..."]
```

- `provider` (required) - the name of the provider ("digitalocean" in this case).
- `region` (required) - the name of the region.
- `tag_name` (required) - the value of the tag to auto-join on.
- `api_token` (required) -the token to use for auth.

### Openstack

This returns the first private IP address of all servers for the given
`region` with the given `tag_key` and `tag_value`.

```sh
$ dkron agent --retry-join "provider=os tag_key=dkron tag_value=server username=... password=... auth_url=..."
```

```yml
retry-join: ["provider=os tag_key=dkron tag_value=server username=... password=... auth_url=..."]
```

- `provider` (required) - the name of the provider ("os" in this case).
- `tag_key` (required) - the key of the tag to auto-join on.
- `tag_value` (required) - the value of the tag to auto-join on.
- `project_id` (optional) - the id of the project (tenant id).
- `username` (optional) - the username to use for auth.
- `password` (optional) - the password to use for auth.
- `token` (optional) - the token to use for auth.
- `auth_url` (optional) - the identity endpoint to use for auth.
- `insecure` (optional) - indicates whether the API certificate should not be checked. Any value means `true`.

The configuration can also be provided by environment variables.

### Scaleway

This returns the first private IP address of all servers for the given
`region` with the given `tag_name`.

```sh
$ dkron agent --retry-join "provider=scaleway organization=my-org tag_name=dkron-server token=... region=..."
```

```yml
retry-join: ["provider=scaleway organization=my-org tag_name=dkron-server token=... region=..."]
```

- `provider` (required) - the name of the provider ("scaleway" in this case).
- `region` (required) - the name of the region.
- `tag_name` (required) - the name of the tag to auto-join on.
- `organization` (required) - the organization access key to use for auth (equal to access key).
- `token` (required) - the token to use for auth.

### Joyent Triton

This returns the first PrimaryIP addresses for all servers with the given `tag_key` and `tag_value`.

```sh
$ dkron agent --retry-join "provider=triton account=testaccount url=https://us-sw-1.api.joyentcloud.com key_id=... tag_key=dkron-role tag_value=server"
```

```yml
retry-join: ["provider=triton account=testaccount url=https://us-sw-1.api.joyentcloud.com key_id=... tag_key=dkron-role tag_value=server"]
```

- `provider` (required) - the name of the provider ("triton" in this case).
- `account` (required) - the name of the account.
- `url` (required) - the URL of the Triton api endpoint to use.
- `key_id` (required) - the key id to use.
- `tag_key` (optional) - the instance tag key to use.
- `tag_value` (optional) - the tag value to use.


### vSphere

This returns the first private IP address of all servers for the given region with the given `tag_name` and `category_name`.

```sh
$ dkron agent --retry-join "provider=vsphere category_name=dkron-role tag_name=dkron-server host=... user=... password=... insecure_ssl=[true|false]"
```

```yml
retry-join: ["provider=vsphere category_name=dkron-role tag_name=dkron-server host=... user=... password=... insecure_ssl=[true|false]"]
```

- `provider` (required) -   the name of the provider ("vsphere" is the provider here)
- `tag_name` (required) -    The name of the tag to look up.
- `category_name` (required) - The category of the tag to look up.
- `host` (required) -        The host of the vSphere server to connect to.
- `user` (required) -         The username to connect as.
- `password` (required) -     The password of the user to connect to vSphere as.
- `insecure_ssl` (optional) -  Whether or not to skip SSL certificate validation.
- `timeout` (optional) -     Discovery context timeout (default: 10m)

### Packet

This returns the first private IP address (or the IP address of `address type`) of all servers with the given `project` and `auth_token`.

```sh
$ dkron agent --retry-join "provider=packet auth_token=token project=uuid url=... address_type=..."
```

```yml
retry-join: ["provider=packet auth_token=token project=uuid url=... address_type=..."]
```

- `provider` (required)	-	the name of the provider ("packet" is the provider here)
- `project` (required) 	- 	the UUID of packet project
- `auth_token` (required) -  the authentication token for packet
- `url` (optional) - 		 a REST URL for packet
- `address_type` (optional) - the type of address to check for in this provider  ("private_v4", "public_v4" or "public_v6".                                   Defaults to "private_v4")


### Kubernetes (k8s)

The Kubernetes provider finds the IP addresses of pods with the matching
[label or field selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).
This is useful for non-Kubernetes agents that are joining a server cluster
running within Kubernetes.

The pod IP is used by default, which requires that the agent connecting can
network to the pod IP. The `host_network` boolean can be set to true to use
the host IP instead, but this requires the agent ports (Gossip, RPC, etc.)
to be exported to the host as well.

By default, no port is specified. This causes Dkron to use the default
gossip port (default behavior with all join requests). The pod may specify
the `dkron.hashicorp.com/auto-join-port` annotation to set the port. The value
may be an integer or a named port.

```sh
$ dkron agent --retry-join "provider=k8s label_selector=\"app=dkron,component=server\""
```

```yml
retry-join: ["provider=k8s label_selector=..."]
```

- `provider` (required)	-	the name of the provider ("k8s" is the provider here)
- `kubeconfig` (optional) 	- 	path to the kubeconfig file. If this isn't
  set, then in-cluster auth will be attempted. If that fails, the default
  kubeconfig paths are tried (`$HOME/.kube/config`).
- `namespace` (optional) -  the namespace to search for pods. If this isn't
  set, it defaults to all namespaces.
- `label_selector` (optional) - the label selector for matching pods.
- `field_selector` (optional) - the field selector for matching pods.

---

### Usage Clustering (`website/docs/usage/clustering.md`)

---
title: Clustering
---

# Clustering

## High Availability Clustering

Dkron is designed to operate as a cluster of multiple nodes for fault tolerance and high availability. This guide explains how to set up and manage a Dkron cluster.

```mermaid
flowchart TD
    subgraph DC1[Datacenter 1]
        Server1[Server Node 1\nLeader] <--> Server2[Server Node 2\nFollower]
        Server1 <--> Server3[Server Node 3\nFollower]
        Server1 --- Agent1[Agent 1]
        Server2 --- Agent1
        Server3 --- Agent1
    end
    
    subgraph DC2[Datacenter 2]
        Server4[Server Node 4\nFollower] <--> Server5[Server Node 5\nFollower]
        Server4 --- Agent2[Agent 2]
        Server5 --- Agent2
    end
    
    DC1 <--> DC2
```

## Cluster Formation

### Bootstrap Methods

Dkron supports several methods for nodes to discover and join each other:

1. **Static List**: Explicitly specify other cluster members
2. **Auto-join**: Automatically discover other nodes
3. **Cloud Auto-join**: Discover nodes in cloud environments

### Basic Cluster Setup

To start a basic 3-node server cluster:

1. **Start the first node**:
   ```
   dkron agent --server --bootstrap-expect=3 --node-name=node1 --bind=10.0.0.1 --advertise=10.0.0.1
   ```

2. **Start additional server nodes**:
   ```
   dkron agent --server --bootstrap-expect=3 --node-name=node2 --bind=10.0.0.2 --advertise=10.0.0.2 --join=10.0.0.1
   dkron agent --server --bootstrap-expect=3 --node-name=node3 --bind=10.0.0.3 --advertise=10.0.0.3 --join=10.0.0.1
   ```

3. **Add agent nodes**:
   ```
   dkron agent --node-name=agent1 --bind=10.0.0.4 --join=10.0.0.1
   ```

### Key Configuration Parameters

| Parameter | Description | Example |
|-----------|-------------|---------|
| `--server` | Run in server mode | `--server` |
| `--bootstrap-expect` | Expected number of servers | `--bootstrap-expect=3` |
| `--node-name` | Unique node identifier | `--node-name=node1` |
| `--bind` | Address to bind network services | `--bind=10.0.0.1` |
| `--advertise` | Address to advertise to cluster | `--advertise=10.0.0.1` |
| `--join` | Address of another node to join | `--join=10.0.0.1` |
| `--retry-join` | Auto-retry joining | `--retry-join=10.0.0.1` |
| `--tag` | Node tag for job targeting | `--tag role=web` |

## Multi-Datacenter Configuration

Dkron can run in HA mode, avoiding SPOFs, this mode provides better scalability and better reliability for users that wants a high level of confidence in the cron jobs they need to run.

Manually bootstrapping a Dkron cluster does not rely on additional tooling, but does require operator participation in the cluster formation process. When bootstrapping, Dkron servers and clients must be started and informed with the address of at least one Dkron server.

As you can tell, this creates a chicken-and-egg problem where one server must first be fully bootstrapped and configured before the remaining servers and clients can join the cluster. This requirement can add additional provisioning time as well as ordered dependencies during provisioning.

First, we bootstrap a single Dkron server and capture its IP address. After we have that nodes IP address, we place this address in the configuration.

1. First bootstrap a node with a configuration like this:

```yaml
# dkron.yml
server: true
bootstrap-expect: 1
```

2. Then stop the bootstrapped server and capture the server IP address.

3. To form a cluster, configure server nodes (including the bootstrapped server) with the address of its peers as in the following example:

```yaml
# dkron.yml
server: true
bootstrap-expect: 3
retry-join:
- 10.19.3.9
- 10.19.4.64
- 10.19.7.215
```

## Deployment Table

Below is a table that shows quorum size and failure tolerance for various
cluster sizes. The recommended deployment is either 3 or 5 servers. A single
server deployment is _**highly**_ discouraged as data loss is inevitable in a
failure scenario.

<table>
  <thead>
    <tr>
      <th>Servers</th>
      <th>Quorum Size</th>
      <th>Failure Tolerance</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1</td>
      <td>1</td>
      <td>0</td>
    </tr>
    <tr>
      <td>2</td>
      <td>2</td>
      <td>0</td>
    </tr>
    <tr class="warning">
      <td>3</td>
      <td>2</td>
      <td>1</td>
    </tr>
    <tr>
      <td>4</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr class="warning">
      <td>5</td>
      <td>3</td>
      <td>2</td>
    </tr>
    <tr>
      <td>6</td>
      <td>4</td>
      <td>2</td>
    </tr>
    <tr>
      <td>7</td>
      <td>4</td>
      <td>3</td>
    </tr>
  </tbody>
</table>

## Fault Tolerance and Recovery

Dkron's distributed architecture is designed to maintain operation even when nodes fail.

### Leader Failure Recovery

When a leader node fails:

1. Remaining server nodes detect the failure through gossip protocol
2. A new leader election is automatically triggered
3. A new leader is elected from the available server nodes
4. The new leader takes over scheduling responsibilities
5. All running jobs continue to operate without interruption

### Quorum Requirements

- A cluster must maintain a quorum (majority) of server nodes to operate
- For a cluster of size N, at least (N/2)+1 nodes must be available
- If quorum is lost, the cluster stops scheduling new jobs until quorum is restored
- Refer to the Deployment Table above for specific quorum sizes and failure tolerance

### Node Rejoining

When a previously failed node rejoins the cluster:

1. The node establishes connection with existing cluster members
2. It synchronizes its state with the current leader
3. If it was previously a leader, it joins as a follower
4. The node becomes available for job execution

### Data Recovery

In case of catastrophic failure:

1. Restore from backup if available
2. Bootstrap a new cluster with at least one server node
3. Import job definitions via API
4. New executions will begin according to schedule

## Cluster Maintenance

Proper maintenance ensures your Dkron cluster remains healthy and performant.

### Adding Nodes

To add a new server node to an existing cluster:

1. Install Dkron on the new server
2. Configure with appropriate server settings
3. Set `--join` parameter to point to existing cluster members
4. Start the Dkron process
5. Verify the node joins successfully via the web UI or API

Adding agent nodes follows the same process but without the `--server` flag.

### Removing Nodes

To gracefully remove a node:

1. For agents: simply stop the Dkron service
2. For server nodes:
   - If possible, demote the node to an agent first
   - Ensure you maintain sufficient servers for quorum
   - Stop the Dkron service

### Upgrading Dkron

For minimal disruption during upgrades:

1. **Rolling Upgrade** (recommended):
   - Upgrade one node at a time
   - Start with agent nodes
   - Then upgrade server nodes, leaving the leader for last
   - Allow state synchronization between each node upgrade

2. **Version Compatibility**:
   - Check release notes for compatibility information
   - Ensure all nodes in a cluster run compatible versions
   - Some upgrades may require special procedures

### Monitoring Cluster Health

Regular health checks should include:

1. **Node Status**: Check that all expected nodes are active
2. **Leadership**: Verify leader election is stable
3. **Job Execution**: Monitor successful job execution rates
4. **Resource Usage**: Track CPU, memory, and disk usage
5. **Log Analysis**: Review logs for errors or warnings

### Backup and Recovery

Implement a regular backup strategy:

1. **Data Store Backup**:
   - Stop Dkron service or use backup-friendly commands
   - Copy the data directory to a secure location
   - Restart the service if stopped

2. **Configuration Backup**:
   - Maintain copies of configuration files
   - Document any custom settings

3. **Job Definition Export**:
   - Use the API to export job definitions
   - Store in version control for tracking changes

---

### Usage Concepts (`website/docs/usage/concepts.md`)

# Concepts

## Core Concepts

Dkron's architecture is based on a few key concepts that are essential to understand:

```mermaid
flowchart TD
    Job["Job Definition"] --> Schedule["Schedule"]
    Job --> Executor["Executor"]
    Job --> Processors["Processors"]
    Job --> TargetNodes["Target Nodes"]
    
    Schedule --> Execution["Job Execution"]
    Executor --> Execution
    TargetNodes --> Execution
    
    Execution --> Status["Execution Status"]
    Execution --> Output["Execution Output"]
    
    Status --> Retry["Retry Logic"]
    Status --> Dependent["Dependent Jobs"]
    
    Output --> Processors
```

### Jobs

Jobs are the central entity in Dkron. A job represents a scheduled task with the following properties:

| Property | Description |
|----------|-------------|
| Name | Unique identifier for the job |
| Schedule | When to run (cron expression) |
| Timezone | Reference timezone for the schedule |
| Owner | Owner name for accountability |
| Disabled | Flag to disable the job temporarily |
| Tags | Key-value pairs for node targeting |
| Metadata | Custom user data for the job |
| Concurrency | How to handle overlapping executions |
| Executor | Plugin to execute the job |
| Retry | Configuration for retrying failed jobs |
| Dependent Jobs | Jobs that should run after this one |

### Schedules

Jobs are scheduled using cron expressions. Dkron supports:

- Standard cron expressions (`* * * * *`)
- Predefined schedules (`@hourly`, `@daily`, `@weekly`)
- Interval notation (`@every 1h30m`)
- Custom timezone support

### Executors

Executors are plugins that perform the actual job execution:

- **Shell**: Executes commands on the target node
- **HTTP**: Makes HTTP requests to specified endpoints
- **Kafka**: Produces messages to Kafka topics
- **And more**: Custom executors can be implemented

### Processors

Processors handle the output from job executions:

- **Log**: Records execution output to logs
- **Files**: Saves output to files
- **Custom**: User-implemented processors

### Target Node Selection

Jobs can be targeted to specific nodes using:

- **Tags**: Key-value pairs assigned to nodes
- **Count**: Number of nodes to run the job on
- **Selector**: Logic for choosing nodes

Example tag specification:
```json
"tags": {
  "role": "web",
  "datacenter": "us-east:2"
}
```

### Concurrency Control

Dkron provides several concurrency options:

- **Allow**: Multiple executions can run simultaneously
- **Forbid**: New executions are skipped if one is running
- **Replace**: New executions replace running ones

### Job Dependencies

Jobs can depend on other jobs:

- Parent jobs must complete successfully before dependent jobs run
- Dependencies can form complex workflows
- Status codes determine success/failure

## Advanced Concepts

### Distributed Leadership

```mermaid
flowchart LR
    subgraph Before
        Leader1[Node 1\nLeader] --> Follower1[Node 2\nFollower]
        Leader1 --> Follower2[Node 3\nFollower]
    end
    
    subgraph After[After Node 1 Failure]
        Follower3[Node 2\nNew Leader] --> X[Node 1\nDown]
        Follower3 --> Follower4[Node 3\nFollower]
    end
    
    Before --> After
```

Dkron uses a leader-follower model to ensure high availability:

- One server node is elected as leader
- The leader is responsible for scheduling
- If the leader fails, a new leader is automatically elected
- The cluster continues operating without interruption
- When the old leader rejoins, it becomes a follower

### Status Codes

Jobs return status codes that indicate success or failure:

- By default, a zero exit code means success
- You can customize which codes are considered successful
- Status codes determine whether dependent jobs run
- Processors can take different actions based on status codes

### Storage Backend

Dkron uses an embedded BoltDB database for:

- Storing job definitions
- Recording execution history
- Maintaining cluster state
- Storing processor output

For larger deployments, consider these best practices:

- Regular database backups
- Monitoring storage usage
- Setting appropriate retention policies

### Security Considerations

```mermaid
flowchart TD
    User[User] -->|Authentication| API[API Layer]
    API -->|Authorization| Jobs[Job Execution]
    API -->|TLS| Agent[Agent Communication]
    
    subgraph Security Controls
        Authentication[Authentication]
        Authorization[Authorization]
        TLS[Transport Layer Security]
        Encryption[Data Encryption]
    end
```

Securing your Dkron deployment:

- **Authentication (Pro feature)**: Protect the API with authentication
- **Authorization (Pro feature)**: Control who can manage jobs
- **TLS (Pro feature)**: Encrypt communication between nodes
- **Network Security**: Use firewalls to restrict access
- **Secrets Management**: Carefully handle credentials in jobs

### Failure Recovery

Dkron provides multiple mechanisms for handling failures:

- **Automatic retries**: Configure jobs to retry on failure
- **Dependent jobs with fallbacks**: Create contingency plans
- **Notifications**: Set up alerting for critical failures
- **Health checks**: Monitor node and cluster status

### Performance Optimization

For optimal performance:

- Distribute jobs across nodes based on resource requirements
- Use appropriate concurrency settings
- Monitor execution times and adjust schedules
- Consider time zone impacts on scheduling
- Optimize command execution with appropriate timeouts

---

### Usage Concurrency (`website/docs/usage/concurrency.md`)

---
title: Concurrency
toc: true
---

## Concurrency

Jobs can be configured to allow overlapping executions or forbid them. 

Concurrency property accepts two option: 

* **allow** (default): Allow concurrent job executions.
* **forbid**: If the job is already running don't send the execution, it will skip the executions until the next schedule.

Example:

```json
{
  "name": "job1",
  "schedule": "@every 10s",
  "executor": "shell",
  "executor_config": {
    "command": "echo \"Hello from parent\""
  },
  "concurrency": "forbid"
}
```

---

### Usage Cron Spec (`website/docs/usage/cron-spec.md`)

---
title: Cron Specification
weight: 20
---

# Cron Expression Format

Dkron uses a powerful cron expression format to define job schedules. This guide explains the syntax in detail with practical examples.

## Basic Format

A cron expression consists of 6 space-separated fields:

| Field name   | Mandatory? | Allowed values  | Allowed special characters |
|------------- | ---------- | --------------- | -------------------------- |
| Seconds      | Yes        | 0-59            | * / , - ~                  |
| Minutes      | Yes        | 0-59            | * / , - ~                  |
| Hours        | Yes        | 0-23            | * / , - ~                  |
| Day of month | Yes        | 1-31            | * / , - ? ~                |
| Month        | Yes        | 1-12 or JAN-DEC | * / , - ~                  |
| Day of week  | Yes        | 0-6 or SUN-SAT  | * / , - ? ~                |

Note: Month and Day-of-week field values are case insensitive. "SUN", "Sun", and "sun" are equally accepted.

## Special Characters

### Asterisk ( * )

The asterisk indicates that the cron expression will match for all values of the field.

Example: `* * * * * *` runs every second of every minute of every hour, every day.

### Slash ( / )

Slashes define increments within ranges.

Example: `0 */15 * * * *` runs every 15 minutes (at 0:00, 0:15, 0:30, 0:45).

### Comma ( , )

Commas are used to separate items in a list.

Example: `0 0 0 * * MON,WED,FRI` runs at midnight on Mondays, Wednesdays, and Fridays.

### Hyphen ( - )

Hyphens define ranges.

Example: `0 0 9-17 * * *` runs every hour from 9 AM to 5 PM.

### Question mark ( ? )

Question mark may be used instead of '*' for leaving either day-of-month or day-of-week blank to avoid conflicts.

Example: `0 0 0 1 * ?` runs at midnight on the first day of each month, regardless of the day of the week.

### Tilde ( ~ )

Tilde is replaced by a deterministic numeric value based on the job name, providing an even distribution of load.

Example: `0 ~ * * * *` distributes jobs evenly across different minutes within the hour.

## Predefined Schedules

For convenience, Dkron supports several predefined schedules:

| Entry                  | Description                                | Equivalent To    |
|----------------------- | ------------------------------------------ | ---------------- |
| @yearly (or @annually) | Run once a year at midnight on January 1   | 0 0 0 1 1 *      |
| @monthly               | Run once a month at midnight on first day  | 0 0 0 1 * *      |
| @weekly                | Run once a week at midnight on Sunday      | 0 0 0 * * 0      |
| @daily (or @midnight)  | Run once a day at midnight                 | 0 0 0 * * *      |
| @hourly                | Run once an hour at the beginning of hour  | 0 0 * * * *      |
| @minutely              | Run once a minute at the beginning         | 0 * * * * *      |
| @manually              | Never runs automatically (manual triggers) | N/A              |
| @at <time>             | Run once at specified time                 | N/A              |
| @after <time> <grace>  | Run once with grace period for late jobs   | N/A              |

Example: `@daily` is equivalent to `0 0 0 * * *`

## Fixed Intervals

You can schedule jobs to execute at fixed intervals using:

```
@every <duration>
```

Where "duration" is a string accepted by [time.ParseDuration](http://golang.org/pkg/time/#ParseDuration).

Example: `@every 1h30m10s` runs every 1 hour, 30 minutes, and 10 seconds.

Note: The interval does not account for job runtime. For example, if a job takes 3 minutes to run, and it is scheduled to run every 5 minutes, it will have only 2 minutes of idle time between each run.

## One-time Execution

To schedule a job to be executed just once at a specific time:

```
@at <datetime>
```

Where "datetime" is a string in [RFC3339 format](https://golang.org/pkg/time/#Parse).

Example: `@at 2023-12-31T23:59:00Z` will run the job once on December 31, 2023 at 11:59 PM UTC.

**Important**: If a job is created after the specified time, it will never run.

## One-time Execution with Grace Period

To schedule a job that can still run if created slightly after its intended time (useful for handling network latency or delayed job creation):

```
@after <datetime> <duration>
```

Where:
- "datetime" is a string in [RFC3339 format](https://golang.org/pkg/time/#Parse)
- "duration" is a grace period in [ISO8601 duration format](https://en.wikipedia.org/wiki/ISO_8601#Durations), prefixed with `<`

**Behavior**:
- If the job is created **before** the scheduled time → it runs at the scheduled time
- If the job is created **within the grace period** after the scheduled time → it runs immediately
- If the job is created **after the grace period** → it never runs

**Duration Format**:
The grace period uses ISO8601 duration format: `P[n]Y[n]M[n]DT[n]H[n]M[n]S`
- `P` - Period designator (required)
- `T` - Time designator (required for hours, minutes, seconds)
- Years: `P1Y` = 1 year (365 days approximation)
- Months: `P1M` = 1 month (30 days approximation)
- Days: `P1D` = 1 day
- Hours: `PT1H` = 1 hour
- Minutes: `PT1M` = 1 minute
- Seconds: `PT1S` = 1 second

**Examples**:

```
@after 2023-12-31T23:59:00Z <PT2H
```
Runs at 11:59 PM UTC on December 31, 2023, or immediately if created up to 2 hours late.

```
@after 2024-01-01T09:00:00Z <PT30M
```
Runs at 9:00 AM on January 1, 2024, or immediately if created up to 30 minutes late.

```
@after 2024-01-15T00:00:00Z <P1D
```
Runs at midnight on January 15, 2024, or immediately if created up to 1 day late.

```
@after 2024-02-01T02:00:00Z <P1DT2H30M
```
Runs at 2:00 AM on February 1, 2024, or immediately if created up to 1 day, 2 hours, and 30 minutes late.

**Use Cases**:
- **Network Latency Tolerance**: When creating jobs via API, network delays might cause the job to be created after its intended execution time. The grace period allows it to still run.
- **Flexible Maintenance Windows**: Schedule maintenance tasks that can run anytime within a specific window.
- **Event-Based Execution**: Jobs that should run once after a specific event time but need flexibility for delayed creation.

**Note**: The grace period boundary is inclusive - a job created exactly at the end of the grace period will still run immediately.

## Time Zones

Dkron supports scheduling jobs in specific time zones by specifying the `timezone` parameter in a job definition.

If no time zone is specified:
- All scheduling is done in the server's local time zone
- Jobs scheduled during daylight-savings leap-ahead transitions will not be run

When a `timezone` is specified, the job will be scheduled in the specified time zone, correctly handling daylight saving time transitions.

Example timezone values:
- "America/New_York"
- "Europe/London"
- "Asia/Tokyo"
- "UTC"

## Practical Examples

Here are some common scheduling patterns with explanations:

### Every weekday at 8 AM
```
0 0 8 * * 1-5
```
This runs at 8:00 AM Monday through Friday.

### Every 10 minutes during business hours
```
0 */10 9-17 * * 1-5
```
This runs every 10 minutes from 9 AM to 5 PM, Monday through Friday.

### First day of every month at 3 AM
```
0 0 3 1 * *
```
This runs at 3:00 AM on the first day of each month.

### Every 15 minutes with even load distribution
```
0 ~/15 * * * *
```
This distributes jobs evenly across the hour in 15-minute intervals, with the exact minute determined by the job name.

### Every quarter (Jan, Apr, Jul, Oct) on the first day at midnight
```
0 0 0 1 1,4,7,10 *
```
This runs at midnight on the first day of January, April, July, and October.

### Every weekend at 10 PM
```
0 0 22 * * 0,6
```
This runs at 10:00 PM on Saturdays and Sundays.

### Every day at 8 AM New York time
```
0 0 8 * * *
```
With `timezone` parameter set to "America/New_York" in the job configuration.

## Best Practices

1. **Avoid Running Too Frequently**: Consider resource usage when scheduling frequent jobs. Running jobs every few seconds can put unnecessary load on your system.

2. **Use Even Load Distribution**: For jobs that don't need to run at an exact time, use the tilde (~) character to distribute load evenly.

3. **Consider Time Zones Carefully**: Be explicit about time zones in distributed systems to avoid confusion, especially if your servers are in different locations.

4. **Avoid Overlapping Executions**: For long-running jobs, ensure the schedule interval is longer than the expected execution time, or enable concurrency controls in the job configuration.

5. **Use Descriptive Job Names**: With the tilde (~) feature, job names influence scheduling, so use consistent naming conventions.

6. **Test Complex Expressions**: Use tools like [crontab.guru](https://crontab.guru/) to validate your cron expressions (note: these tools typically use 5-field format, while Dkron uses 6 fields with seconds).

7. **Document Job Schedules**: Maintain documentation about why jobs are scheduled at specific times to help with maintenance and troubleshooting.

---

### Usage Cronitor (`website/docs/usage/cronitor.md`)

# Cronitor Integration

:::info
This feature is available since v3.2.0
:::

Dkron includes tight integration with [Cronitor](https://cronitor.io/) for advanced execution monitoring, check their product page for more information.

To setup the integration add this parameter to your `dkron.yaml` config file, including your private cronitor API endpoint key:

:::caution
Remember: do not check your config file into your source code repository, it contains sensitive information.
:::

```
cronitor-endpoint: https://cronitor.link/p/xxxxxxxxxxxxxxxxxxxxxx
```

You can also use the `--cronitor-endpoint` CLI parameter or the `DKRON_CRONITOR_ENDPOINT` env variable.

Dkron will call Cronitor before and after running a job, sending all necessary information, no further configuration is necessary as Cronitor automagically create a monitor for each job.



You can further configure your job using the Cronitor UI or API after it has run at least once.

Cronitor can be used to notify over different channels using integrations, go to the settings page to integrate it with several popular services:

---

### Usage Ecs (`website/docs/usage/ecs.md`)

---
title: Use with AWS ECS
---

:::tip
[Dkron Pro](/pro) comes with a [native ECS executor](/docs/pro/executors/ecs/) out of the box.
:::

## Use with Amazon ECS

To use Dkron to schedule jobs that run in containers, a wrapper ECS script is needed.

Install the following snippet in the node that will run the call to ECS

<script src="https://gist.github.com/distribworks/3ac4aae9279d7c68c486fecccc2546cc.js"></script>

### Prerequisites

The node that will run the call to ECS will need to have installed

* AWS cli
* jq

### Example

`ecs-run --cluster cron --task-definition cron-taskdef --container-name cron --region us-east-1 --command "rake foo"`

---

### Usage Metatags (`website/docs/usage/metatags.md`)

---
title: Job metadata
---

## Job metadata

Jobs can have an optional extra property field called `metadata` that allows to set arbitrary tags to jobs and query the jobs using the API:

```json
{
    "name": "job_name",
    "command": "/bin/true",
    "schedule": "@every 2m",
    "metadata": {
        "user_id": "12345"
    }
}
```

And then query the API to get only the results needed:

```
$ curl http://localhost:8080/v1/jobs --data-urlencode "metadata[user_id]=12345"`
```

---

### Usage Metrics (`website/docs/usage/metrics.md`)

---
title: Metrics
---

Dkron has the ability to send metrics to Statsd for dashboards and historical reporting or provide prometheus format metrics via the api. It sends job processing metrics, golang, and serf metrics.

## Configuration

### Statsd

Add this in your yaml config file to enable statsd metrics.

```yaml
statsd-addr: "localhost:8125"
# Or for datadog statsd
dog-statsd-addr: "localhost:8125"
```

### Prometheus

Add this to your yaml config file to enable serving prometheus metrics at the endpoint `/metrics`

```yaml
enable-prometheus: true
```

Additionally, in your Prometheus config file (prometheus.yml), add the following to link dkron metric endpoint
```yaml
scrape_configs:
  ... #initial configuration
  
  - job_name: "dkron_metrics"
    # metrics_path defaults to '/metrics'
    static_configs:
      - targets: ["localhost:8080"]
```

## Monitoring Dashboard Setup

Setting up a monitoring dashboard with metrics from Dkron is highly recommended for production deployments. You have several options:

1. **Grafana + Prometheus**: Create dashboards with job execution success rates, execution timing trends, and system health metrics
2. **DataDog**: Use built-in dashboard templates with the DogStatsD integration
3. **Custom StatsD Dashboards**: Configure with any StatsD-compatible visualization tool

Example Grafana dashboard panels for monitoring Dkron might include:
- Job execution success rate over time
- Average execution duration by job
- System resource utilization
- Cluster health status

## Metrics Reference

Dkron emits several categories of metrics to help you monitor both the application and your job executions:

### Agent Event Metrics

These metrics track internal events within Dkron agents:

| Metric | Description |
|--------|-------------|
| `dkron.agent.event_received.query_execution_done` | Count of completed job execution events received by the agent |
| `dkron.agent.event_received.query_run_job` | Count of job run requests received by the agent |

### Network Communication Metrics

These metrics help monitor the health of inter-node communication:

| Metric | Description |
|--------|-------------|
| `dkron.memberlist.gossip` | Count of gossip protocol messages exchanged |
| `dkron.memberlist.probeNode` | Count of node health probe checks |
| `dkron.memberlist.pushPullNode` | Count of anti-entropy sync operations |
| `dkron.memberlist.tcp.accept` | Count of accepted TCP connections |
| `dkron.memberlist.tcp.connect` | Count of initiated TCP connections |
| `dkron.memberlist.tcp.sent` | Count and bytes of TCP packets sent |
| `dkron.memberlist.udp.received` | Count and bytes of UDP packets received |
| `dkron.memberlist.udp.sent` | Count and bytes of UDP packets sent |

### gRPC Service Metrics

These metrics track the internal RPC communication:

| Metric | Description |
|--------|-------------|
| `dkron.grpc.call_execution_done` | Count and timing of execution completion RPC calls |
| `dkron.grpc.call_get_job` | Count and timing of job retrieval RPC calls |
| `dkron.grpc.execution_done` | Count of completed job executions |
| `dkron.grpc.get_job` | Count of job information retrievals |

### Runtime Metrics

These metrics provide insights into the Go runtime health:

| Metric | Description |
|--------|-------------|
| `dkron.runtime.alloc_bytes` | Current bytes allocated by the application |
| `dkron.runtime.free_count` | Count of memory free operations |
| `dkron.runtime.gc_pause_ns` | Duration of the last garbage collection pause in nanoseconds |
| `dkron.runtime.heap_objects` | Number of objects in the heap |
| `dkron.runtime.malloc_count` | Count of memory allocation operations |
| `dkron.runtime.num_goroutines` | Number of goroutines currently running |
| `dkron.runtime.sys_bytes` | Total bytes of memory obtained from the OS |
| `dkron.runtime.total_gc_pause_ns` | Total time spent in GC pauses |
| `dkron.runtime.total_gc_runs` | Total number of completed GC cycles |

### Serf Metrics

These metrics track the cluster membership and failure detection system:

| Metric | Description |
|--------|-------------|
| `dkron.serf.coordinate.adjustment_ms` | Time adjustment for coordinate system in milliseconds |
| `dkron.serf.msgs.received` | Count of messages received |
| `dkron.serf.msgs.sent` | Count of messages sent |
| `dkron.serf.queries` | Count of queries processed |
| `dkron.serf.queries.execution_done` | Count of execution completion queries |
| `dkron.serf.queries.run_job` | Count of job run queries |
| `dkron.serf.query_acks` | Count of query acknowledgments |
| `dkron.serf.query_responses` | Count of responses to queries |
| `dkron.serf.queue.Event` | Count of events in processing queue |
| `dkron.serf.queue.Intent` | Count of intent messages in queue |
| `dkron.serf.queue.Query` | Count of queries in processing queue |

## Alerting on Metrics

To set up effective alerting based on Dkron metrics, consider these recommendations:

1. **Job Failure Alerts**: Monitor `dkron.serf.queries.execution_done` with error status
2. **Cluster Health**: Set alerts on node count changes or consistent gossip failures
3. **Performance Degradation**: Watch for increases in job execution time trends
4. **Resource Constraints**: Monitor `dkron.runtime.gc_pause_ns` and other runtime metrics for signs of resource pressure

For Prometheus users, example alerting rules:

```yaml
groups:
- name: dkron_alerts
  rules:
  - alert: DkronJobFailures
    expr: increase(dkron_job_executions_failed_total[1h]) > 3
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Multiple job failures detected"
      description: "There have been more than 3 job failures in the last hour"
```

---

### Usage Recovery (`website/docs/usage/recovery.md`)

---
title: Outage recovery
---

## Outage Recovery

Don't panic! This is a critical first step.

Depending on your deployment configuration, it may take only a single server failure for cluster unavailability. Recovery requires an operator to intervene, but the process is straightforward.

:::tip
This guide is for recovery from a Dkron outage due to a majority of server nodes in a datacenter being lost. If you are looking to add or remove servers, see the [clustering](/docs/usage/clustering) guide.
:::

## Failure of a Single Server Cluster

If you had only a single server and it has failed, simply restart it. A single server configuration requires the -bootstrap-expect=1 flag. If the server cannot be recovered, you need to bring up a new server. See the [clustering](/docs/usage/clustering) guide for more detail.

In the case of an unrecoverable server failure in a single server cluster, data loss is inevitable since data was not replicated to any other servers. This is why a single server deploy is never recommended.

## Failure of a Server in a Multi-Server Cluster

If you think the failed server is recoverable, the easiest option is to bring it back online and have it rejoin the cluster with the same IP address, returning the cluster to a fully healthy state. Similarly, even if you need to rebuild a new Dkron server to replace the failed node, you may wish to do that immediately. Keep in mind that the rebuilt server needs to have the same IP address as the failed server. Again, once this server is online and has rejoined, the cluster will return to a fully healthy state.

Both of these strategies involve a potentially lengthy time to reboot or rebuild a failed server. If this is impractical or if building a new server with the same IP isn't an option, you need to remove the failed server. Usually, you can issue a `dkron leave` command to remove the failed server if it's still a member of the cluster.

If `dkron leave` isn't able to remove the server, you can use the `dkron raft remove-peer` command to remove the stale peer server on the fly with no downtime.

You can use the `dkron raft list-peers` command to inspect the Raft configuration:

```
$ dkron raft list-peers
Node                   ID               Address          State     Voter
dkron-server01.global  10.10.11.5:4647  10.10.11.5:4647  follower  true
dkron-server02.global  10.10.11.6:4647  10.10.11.6:4647  leader    true
dkron-server03.global  10.10.11.7:4647  10.10.11.7:4647  follower  true
```

## Failure of Multiple Servers in a Multi-Server Cluster

In the event that multiple servers are lost, causing a loss of quorum and a complete outage, partial recovery is possible using data on the remaining servers in the cluster. There may be data loss in this situation because multiple servers were lost, so information about what's committed could be incomplete. The recovery process implicitly commits all outstanding Raft log entries, so it's also possible to commit data that was uncommitted before the failure.

See the section below for details of the recovery procedure. You simply include just the remaining servers in the raft/peers.json recovery file. The cluster should be able to elect a leader once the remaining servers are all restarted with an identical raft/peers.json configuration.

Any new servers you introduce later can be fresh with totally clean data directories.

In extreme cases, it should be possible to recover with just a single remaining server by starting that single server with itself as the only peer in the raft/peers.json recovery file.

The raft/peers.json recovery file is final, and a snapshot is taken after it is ingested, so you are guaranteed to start with your recovered configuration. This does implicitly commit all Raft log entries, so should only be used to recover from an outage, but it should allow recovery from any situation where there's some cluster data available.

## Manual Recovery Using peers.json

To begin, stop all remaining servers. You can attempt a graceful leave, but it will not work in most cases. Do not worry if the leave exits with an error. The cluster is in an unhealthy state, so this is expected.

The peers.json file will be deleted after Dkron starts and ingests this file.

Using raft/peers.json for recovery can cause uncommitted Raft log entries to be implicitly committed, so this should only be used after an outage where no other option is available to recover a lost server. Make sure you don't have any automated processes that will put the peers file in place on a periodic basis.

The next step is to go to the `data-dir` of each Dkron server. Inside that directory, there will be a raft/ sub-directory. We need to create a raft/peers.json file. It should look something like:

```json
[
  {
    "id": "node1",
    "address": "10.1.0.1:4647"
  },
  {
    "id": "node2",
    "address": "10.1.0.2:4647"
  },
  {
    "id": "node3",
    "address": "10.1.0.3:4647"
  }
]
```

Simply create entries for all remaining servers. You must confirm that servers you do not include here have indeed failed and will not later rejoin the cluster. Ensure that this file is the same across all remaining server nodes.

At this point, you can restart all the remaining servers. In Dkron 0.5.5 and later you will see them ingest recovery file:

```
Recovery log placeholder
```

It should be noted that any existing member can be used to rejoin the cluster as the gossip protocol will take care of discovering the server nodes.

At this point, the cluster should be in an operable state again. One of the nodes should claim leadership and emit a log like:

`[INFO] Dkron: cluster leadership acquired`

You can use the `dkron raft list-peers` command to inspect the Raft configuration:

```
$ dkron raft list-peers
Node   ID     Address          State     Voter
node1  node1  10.10.11.5:4647  follower  true
node2  node2  10.10.11.6:4647  leader    true
node3  node3  10.10.11.7:4647  follower  true
```

* id (string: ) - Specifies the node ID of the server. This is the `name` of the node.

* address (string: ) - Specifies the IP and port of the server in ip:port format. The port is the server's gRPC port used for cluster communications, typically `6868`.

---

### Usage Retries (`website/docs/usage/retries.md`)

---
title: Job retries
---

Jobs can be configured to retry in case of failure.

## Configuration

```json
{
  "name": "job1",
  "schedule": "@every 10s",
  "executor": "shell",
  "executor_config": {
    "command": "echo \"Hello from parent\""
  },
  "retries": 5
}
```

In case of failure to run the job in one node, it will try to run the job again in that node until the retries count reaches the limit.

---

### Usage Storage (`website/docs/usage/storage.md`)

---
title: Embedded storage
---

Dkron has an embedded distributed KV store engine based on BuntDB. This works out of the box on each dkron server.

This ensures a dead easy install and setup, basically run dkron and you will have a full working node.

---

### Usage Target Nodes Spec (`website/docs/usage/target-nodes-spec.md`)

---
title: Target nodes spec
weight: 10
---

## Target nodes spec

Default is for all nodes to execute each job. Dkron has the ability to run jobs in specific nodes by leveraging the use of tags. You can choose whether a job is run on a node or group of nodes by specifying tags and a count of target nodes having this tag do you want a job to run.

The target node syntax is:
    
    "tag": "value[:count]"

To achieve this Nodes and Jobs have tags, for example, having a node with the following tags:

```json
{
    "tags": {
        "dc": "dc1",
        "expect": "3",
        "port": "6868",
        "region": "global",
        "role": "dkron",
        "rpc_addr": "10.88.94.129:6868",
        "server": "true",
        "version": "devel",
        "my_role": "web"
    }
}
```

:::tip
You can specify tags for nodes in the dkron config file or in the command line using `--tags` parameter
:::

In case there is no matching nodes with the specified tags, the job will not run.

Following some examples using different tag combinations:

#### Target all nodes with a tag

```json
{
    "name": "job_name",
    "command": "/bin/true",
    "schedule": "@every 2m",
    "tags": {
        "my_role": "web"
    }
}
```

```mermaid
graph LR;
    J("Job tags: #quot;my_role#quot;: #quot;web#quot;") -->|Run Job|N1["Node1 tags: #quot;my_role#quot;: #quot;web#quot;"]
    J -->|Run Job|N2["Node2 tags: #quot;my_role#quot;: #quot;web#quot;"]
    J -->|Run Job|N3["Node2 tags: #quot;my_role#quot;: #quot;web#quot;"]
```

#### Target only one nodes of a group of nodes with a tag

```json
{
    "name": "job_name",
    "command": "/bin/true",
    "schedule": "@every 2m",
    "tags": {
        "my_role": "web:1"
    }
}
```

```mermaid
graph LR;
    J("Job tags: #quot;my_role#quot;: #quot;web:1#quot;") -->|Run Job|N1["Node1 tags: #quot;my_role#quot;: #quot;web#quot;"]
    J -.- N2["Node2 tags: #quot;my_role#quot;: #quot;web#quot;"]
    J -.- N3["Node2 tags: #quot;my_role#quot;: #quot;web#quot;"]
```

Dkron will try to run the job in the amount of nodes indicated by that count having that tag.

### Details and limitations

#### Reserved tags

The `region` tag is reserved for multi-region setup purposes and should not be used for any other purpose. 

Dkron will always run jobs in nodes with the same region tag.

You should use an alternative name.

#### Combinations

Tags specified in a Job are combined using `AND`, therefore a job that specifies several tags like:

```json
{
    "tags": {
        "my_role": "web",
        "role": "dkron"
    }
}
```

Will try to run the job in nodes that have all speciefied tags.

#### Tags Limit

There is no limit in the tags that a job can have but having a Job with several tags with count like:

```json
{
    "tags": {
        "my_role": "web:1",
        "role": "dkron:2"
    }
}
```

Will try to run the job in nodes that have all specified tags and using the lowest count. In the last example, it will run in **one** node having `"my_role": "web"` and `"role": "dkron"` tag, even if there is more than one node with these tags.

---

### Usage Upgrade (`website/docs/usage/upgrade.md`)

---
title: Upgrade methods
---

# Upgrade methods

Use the upgrade method that matches the kind of change you are making. In most cases, a rolling upgrade is the safest option because it lets you replace nodes gradually while keeping the cluster available. Use backup and restore when you are building a fresh cluster, migrating to new infrastructure, or when release notes require a full rebuild instead of mixed-version operation.

## Before you start

Before upgrading any node:

1. Read the release notes for the target version and check whether mixed-version clusters are supported during the transition.
2. Make sure the current cluster is healthy and has quorum.
3. Export the current jobs so you have a recovery point:

```bash
curl -fsS http://localhost:8080/v1/jobs > backup.json
```

4. Inspect the current Raft peers so you know which server is the leader and which peer IDs are registered:

```bash
dkron raft list-peers
```

:::tip
When upgrading server nodes, it is usually best to leave the current leader for last. That reduces unnecessary leader elections while you rotate the rest of the cluster.
:::

## Rolling upgrade

Use a rolling upgrade when you want to keep the cluster online and the target version supports a gradual transition.

### Recommended order

1. Upgrade agent-only nodes first.
2. Upgrade follower server nodes one at a time.
3. Upgrade the leader last.

### Server rotation procedure

Use the following procedure to replace server nodes one at a time:

1. Add a new server running the target version and configure it to join the existing cluster.
2. Wait until the new server has joined successfully and the cluster is healthy.
3. Stop Dkron on one old server.
4. If that server was the leader, wait until a new leader is elected before continuing.
5. List the current peers and identify the old server's peer ID:

```bash
dkron raft list-peers
```

6. Remove the old server from the Raft configuration:

```bash
dkron raft remove-peer --peer-id <peer-id>
```

7. Confirm the cluster is healthy again.
8. Repeat the process until every old server has been replaced.

:::warning
Do not remove multiple server nodes at once. Dkron needs a healthy Raft quorum to continue scheduling jobs.
:::

## Backup and restore

Use backup and restore when you need to recreate the cluster on new infrastructure or when a rolling upgrade is not appropriate.

### Export jobs from the existing cluster

```bash
curl -fsS http://localhost:8080/v1/jobs > backup.json
```

### Restore jobs into the new cluster

After the new cluster is running and has elected a leader, restore the exported jobs file:

```bash
curl -fsS -X POST http://localhost:8080/v1/restore \
  --form 'file=@backup.json'
```

The restore endpoint expects a multipart form field named `file`. If a job in the file already exists in the target cluster, it is overwritten with the definition from the backup.

:::warning
This export and restore flow restores job definitions from the `/v1/jobs` payload. It should not be treated as a full cluster snapshot, and it does not recreate Raft state or execution history.
:::

## After the upgrade

After either method completes:

1. Run `dkron raft list-peers` and confirm the expected server set is present.
2. Verify that one node is leader and the cluster remains stable.
3. Check the UI or API and confirm the expected jobs are present.
4. Watch the next scheduled executions to ensure jobs are still running as expected.
5. Keep the exported `backup.json` until you are confident the upgrade is complete.

---

### Pro Cli Dkron (`website/docs/pro/cli/dkron.md`)

---
date: 2024-10-06
title: "dkron"
slug: dkron
url: /docs/pro/cli/dkron/
---
## dkron

Professional distributed job scheduling system

### Synopsis

Dkron is a system service that runs scheduled jobs at given intervals or times,
just like the cron unix service but distributed in several machines in a cluster.
If a machine fails (the leader), a follower will take over and keep running the scheduled jobs without human intervention.

### Options

```
      --config string   config file (default is /etc/dkron/dkron.yml)
  -h, --help            help for dkron
```

### SEE ALSO

* [dkron acl](/docs/pro/cli/dkron_acl/)	 - Control ACL policies
* [dkron agent](/docs/pro/cli/dkron_agent/)	 - Start a dkron agent
* [dkron completion](/docs/pro/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell
* [dkron doc](/docs/pro/cli/dkron_doc/)	 - Generate Markdown documentation for the Dkron CLI.
* [dkron keygen](/docs/pro/cli/dkron_keygen/)	 - Generates a new encryption key
* [dkron leave](/docs/pro/cli/dkron_leave/)	 - Force an agent to leave the cluster
* [dkron raft](/docs/pro/cli/dkron_raft/)	 - Command to perform some raft operations
* [dkron version](/docs/pro/cli/dkron_version/)	 - Show version

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl (`website/docs/pro/cli/dkron_acl.md`)

---
date: 2024-10-06
title: "dkron acl"
slug: dkron_acl
url: /docs/pro/cli/dkron_acl/
---
## dkron acl

Control ACL policies

### Synopsis

Control ACL policies. This command allows to manage ACL policies.

### Options

```
      --cert-file string         Path to the client server TLS cert file
  -h, --help                     help for acl
      --key-file string          Path to the client server TLS key file
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron](/docs/pro/cli/dkron/)	 - Professional distributed job scheduling system
* [dkron acl bootstrap](/docs/pro/cli/dkron_acl_bootstrap/)	 - Bootstrap ACL policies
* [dkron acl policy](/docs/pro/cli/dkron_acl_policy/)	 - Operate ACL policies
* [dkron acl token](/docs/pro/cli/dkron_acl_token/)	 - Operate ACL tokens

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Bootstrap (`website/docs/pro/cli/dkron_acl_bootstrap.md`)

---
date: 2024-10-06
title: "dkron acl bootstrap"
slug: dkron_acl_bootstrap
url: /docs/pro/cli/dkron_acl_bootstrap/
---
## dkron acl bootstrap

Bootstrap ACL policies

### Synopsis

Bootstrap ACL policies. This command allows to bootstrap ACL policies.

```
dkron acl bootstrap [flags]
```

### Options

```
  -h, --help   help for bootstrap
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron acl](/docs/pro/cli/dkron_acl/)	 - Control ACL policies

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Policy (`website/docs/pro/cli/dkron_acl_policy.md`)

---
date: 2024-10-06
title: "dkron acl policy"
slug: dkron_acl_policy
url: /docs/pro/cli/dkron_acl_policy/
---
## dkron acl policy

Operate ACL policies

### Synopsis

Operate ACL policies. This command allows to manage ACL policies.

### Options

```
  -h, --help   help for policy
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron acl](/docs/pro/cli/dkron_acl/)	 - Control ACL policies
* [dkron acl policy apply](/docs/pro/cli/dkron_acl_policy_apply/)	 - Apply ACL policies
* [dkron acl policy delete](/docs/pro/cli/dkron_acl_policy_delete/)	 - Delete ACL policies
* [dkron acl policy info](/docs/pro/cli/dkron_acl_policy_info/)	 - Info ACL policy
* [dkron acl policy list](/docs/pro/cli/dkron_acl_policy_list/)	 - List ACL policies

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Policy Apply (`website/docs/pro/cli/dkron_acl_policy_apply.md`)

---
date: 2024-10-06
title: "dkron acl policy apply"
slug: dkron_acl_policy_apply
url: /docs/pro/cli/dkron_acl_policy_apply/
---
## dkron acl policy apply

Apply ACL policies

### Synopsis

Apply ACL policies. This command allows to apply ACL policies.

```
dkron acl policy apply [flags]
```

### Options

```
      --cert-file string         Path to the client server TLS cert file
  -h, --help                     help for apply
      --key-file string          Path to the client server TLS key file
  -n, --name string              Policy name
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
  -r, --rules-file string        Path to the file containing the policy rules
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron acl policy](/docs/pro/cli/dkron_acl_policy/)	 - Operate ACL policies

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Policy Delete (`website/docs/pro/cli/dkron_acl_policy_delete.md`)

---
date: 2024-10-06
title: "dkron acl policy delete"
slug: dkron_acl_policy_delete
url: /docs/pro/cli/dkron_acl_policy_delete/
---
## dkron acl policy delete

Delete ACL policies

### Synopsis

Delete ACL policies. This command allows to apply ACL policies.

```
dkron acl policy delete [flags]
```

### Options

```
      --cert-file string         Path to the client server TLS cert file
  -h, --help                     help for delete
      --key-file string          Path to the client server TLS key file
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron acl policy](/docs/pro/cli/dkron_acl_policy/)	 - Operate ACL policies

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Policy Info (`website/docs/pro/cli/dkron_acl_policy_info.md`)

---
date: 2024-10-06
title: "dkron acl policy info"
slug: dkron_acl_policy_info
url: /docs/pro/cli/dkron_acl_policy_info/
---
## dkron acl policy info

Info ACL policy

### Synopsis

Info ACL policy. Print policy details.

```
dkron acl policy info [flags]
```

### Options

```
  -h, --help   help for info
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron acl policy](/docs/pro/cli/dkron_acl_policy/)	 - Operate ACL policies

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Policy List (`website/docs/pro/cli/dkron_acl_policy_list.md`)

---
date: 2024-10-06
title: "dkron acl policy list"
slug: dkron_acl_policy_list
url: /docs/pro/cli/dkron_acl_policy_list/
---
## dkron acl policy list

List ACL policies

### Synopsis

List ACL policies. This command allows to list ACL policies.

```
dkron acl policy list [flags]
```

### Options

```
  -h, --help   help for list
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron acl policy](/docs/pro/cli/dkron_acl_policy/)	 - Operate ACL policies

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Token (`website/docs/pro/cli/dkron_acl_token.md`)

---
date: 2024-10-06
title: "dkron acl token"
slug: dkron_acl_token
url: /docs/pro/cli/dkron_acl_token/
---
## dkron acl token

Operate ACL tokens

### Synopsis

Operate ACL tokens. This command allows to manage ACL tokens.

### Options

```
  -h, --help   help for token
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron acl](/docs/pro/cli/dkron_acl/)	 - Control ACL policies
* [dkron acl token create](/docs/pro/cli/dkron_acl_token_create/)	 - Create a new ACL token
* [dkron acl token info](/docs/pro/cli/dkron_acl_token_info/)	 - Print ACL token details
* [dkron acl token list](/docs/pro/cli/dkron_acl_token_list/)	 - List ACL tokens

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Token Create (`website/docs/pro/cli/dkron_acl_token_create.md`)

---
date: 2024-10-06
title: "dkron acl token create"
slug: dkron_acl_token_create
url: /docs/pro/cli/dkron_acl_token_create/
---
## dkron acl token create

Create a new ACL token

### Synopsis

Create a new ACL token. This command allows to create a new ACL token.

```
dkron acl token create [flags]
```

### Options

```
      --cert-file string         Path to the client server TLS cert file
  -h, --help                     help for create
      --key-file string          Path to the client server TLS key file
  -n, --name string              Sets the human readable name for the ACL token.
  -p, --policy strings           Specifies a policy to associate with the token. Can be specified multiple times, but only with client type tokens. (default [default])
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file
  -t, --type string              Sets the type of token. Must be one of "client" (default), or "management". (default "client")
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron acl token](/docs/pro/cli/dkron_acl_token/)	 - Operate ACL tokens

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Token Info (`website/docs/pro/cli/dkron_acl_token_info.md`)

---
date: 2024-10-06
title: "dkron acl token info"
slug: dkron_acl_token_info
url: /docs/pro/cli/dkron_acl_token_info/
---
## dkron acl token info

Print ACL token details

### Synopsis

Print ACL token details. This command allows to print ACL token details.

```
dkron acl token info [flags]
```

### Options

```
  -h, --help   help for info
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron acl token](/docs/pro/cli/dkron_acl_token/)	 - Operate ACL tokens

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Acl Token List (`website/docs/pro/cli/dkron_acl_token_list.md`)

---
date: 2024-10-06
title: "dkron acl token list"
slug: dkron_acl_token_list
url: /docs/pro/cli/dkron_acl_token_list/
---
## dkron acl token list

List ACL tokens

### Synopsis

List ACL tokens. This command allows to list ACL tokens.

```
dkron acl token list [flags]
```

### Options

```
  -h, --help   help for list
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron acl token](/docs/pro/cli/dkron_acl_token/)	 - Operate ACL tokens

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Agent (`website/docs/pro/cli/dkron_agent.md`)

---
date: 2024-10-06
title: "dkron agent"
slug: dkron_agent
url: /docs/pro/cli/dkron_agent/
---
## dkron agent

Start a dkron agent

### Synopsis

Start a dkron agent that schedule jobs, listen for executions and run executors.
It also runs a web UI.

```
dkron agent [flags]
```

### Options

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron](/docs/pro/cli/dkron/)	 - Professional distributed job scheduling system

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Completion (`website/docs/pro/cli/dkron_completion.md`)

---
date: 2024-10-06
title: "dkron completion"
slug: dkron_completion
url: /docs/pro/cli/dkron_completion/
---
## dkron completion

Generate the autocompletion script for the specified shell

### Synopsis

Generate the autocompletion script for dkron for the specified shell.
See each sub-command's help for details on how to use the generated script.


### Options

```
  -h, --help   help for completion
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron](/docs/pro/cli/dkron/)	 - Professional distributed job scheduling system
* [dkron completion bash](/docs/pro/cli/dkron_completion_bash/)	 - Generate the autocompletion script for bash
* [dkron completion fish](/docs/pro/cli/dkron_completion_fish/)	 - Generate the autocompletion script for fish
* [dkron completion powershell](/docs/pro/cli/dkron_completion_powershell/)	 - Generate the autocompletion script for powershell
* [dkron completion zsh](/docs/pro/cli/dkron_completion_zsh/)	 - Generate the autocompletion script for zsh

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Completion Bash (`website/docs/pro/cli/dkron_completion_bash.md`)

---
date: 2024-10-06
title: "dkron completion bash"
slug: dkron_completion_bash
url: /docs/pro/cli/dkron_completion_bash/
---
## dkron completion bash

Generate the autocompletion script for bash

### Synopsis

Generate the autocompletion script for the bash shell.

This script depends on the 'bash-completion' package.
If it is not installed already, you can install it via your OS's package manager.

To load completions in your current shell session:

```
	source <(dkron completion bash)
```

To load completions for every new session, execute once:

#### Linux:

	dkron completion bash > /etc/bash_completion.d/dkron

#### macOS:

	dkron completion bash > $(brew --prefix)/etc/bash_completion.d/dkron

You will need to start a new shell for this setup to take effect.


```
dkron completion bash
```

### Options

```
  -h, --help              help for bash
      --no-descriptions   disable completion descriptions
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron completion](/docs/pro/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Completion Fish (`website/docs/pro/cli/dkron_completion_fish.md`)

---
date: 2024-10-06
title: "dkron completion fish"
slug: dkron_completion_fish
url: /docs/pro/cli/dkron_completion_fish/
---
## dkron completion fish

Generate the autocompletion script for fish

### Synopsis

Generate the autocompletion script for the fish shell.

To load completions in your current shell session:

	dkron completion fish | source

To load completions for every new session, execute once:

	dkron completion fish > ~/.config/fish/completions/dkron.fish

You will need to start a new shell for this setup to take effect.


```
dkron completion fish [flags]
```

### Options

```
  -h, --help              help for fish
      --no-descriptions   disable completion descriptions
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron completion](/docs/pro/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Completion Powershell (`website/docs/pro/cli/dkron_completion_powershell.md`)

---
date: 2024-10-06
title: "dkron completion powershell"
slug: dkron_completion_powershell
url: /docs/pro/cli/dkron_completion_powershell/
---
## dkron completion powershell

Generate the autocompletion script for powershell

### Synopsis

Generate the autocompletion script for powershell.

To load completions in your current shell session:

	dkron completion powershell | Out-String | Invoke-Expression

To load completions for every new session, add the output of the above command
to your powershell profile.


```
dkron completion powershell [flags]
```

### Options

```
  -h, --help              help for powershell
      --no-descriptions   disable completion descriptions
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron completion](/docs/pro/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Completion Zsh (`website/docs/pro/cli/dkron_completion_zsh.md`)

---
date: 2024-10-06
title: "dkron completion zsh"
slug: dkron_completion_zsh
url: /docs/pro/cli/dkron_completion_zsh/
---
## dkron completion zsh

Generate the autocompletion script for zsh

### Synopsis

Generate the autocompletion script for the zsh shell.

If shell completion is not already enabled in your environment you will need
to enable it.  You can execute the following once:

	echo "autoload -U compinit; compinit" >> ~/.zshrc

To load completions in your current shell session:

```
	source <(dkron completion zsh)
```

To load completions for every new session, execute once:

#### Linux:

	dkron completion zsh > "~/.zsh/completion/_dkron"

#### macOS:

	dkron completion zsh > $(brew --prefix)/share/zsh/site-functions/_dkron

You will need to start a new shell for this setup to take effect.


```
dkron completion zsh [flags]
```

### Options

```
  -h, --help              help for zsh
      --no-descriptions   disable completion descriptions
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron completion](/docs/pro/cli/dkron_completion/)	 - Generate the autocompletion script for the specified shell

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Doc (`website/docs/pro/cli/dkron_doc.md`)

---
date: 2024-10-06
title: "dkron doc"
slug: dkron_doc
url: /docs/pro/cli/dkron_doc/
---
## dkron doc

Generate Markdown documentation for the Dkron CLI.

### Synopsis

Generate Markdown documentation for the Dkron CLI.
This command is, mostly, used to create up-to-date documentation
of Dkron's command-line interface for http://dkron.io/.
It creates one Markdown file per command with front matter suitable
for rendering in Hugo.

```
dkron doc [flags]
```

### Options

```
      --dir string   the directory to write the doc. (default "/tmp/dkrondoc/")
  -h, --help         help for doc
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron](/docs/pro/cli/dkron/)	 - Professional distributed job scheduling system

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Keygen (`website/docs/pro/cli/dkron_keygen.md`)

---
date: 2024-10-06
title: "dkron keygen"
slug: dkron_keygen
url: /docs/pro/cli/dkron_keygen/
---
## dkron keygen

Generates a new encryption key

### Synopsis

Generates a new encryption key that can be used to configure the
  agent to encrypt traffic. The output of this command is already
  in the proper format that the agent expects.

```
dkron keygen [flags]
```

### Options

```
  -h, --help   help for keygen
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron](/docs/pro/cli/dkron/)	 - Professional distributed job scheduling system

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Leave (`website/docs/pro/cli/dkron_leave.md`)

---
date: 2024-10-06
title: "dkron leave"
slug: dkron_leave
url: /docs/pro/cli/dkron_leave/
---
## dkron leave

Force an agent to leave the cluster

### Synopsis

Stop stops an agent, if the agent is a server and is running for election
	stop running for election, if this server was the leader
	this will force the cluster to elect a new leader and start a new scheduler.
	If this is a server and has the scheduler started stop it, ignoring if this server
	was participating in leader election or not (local storage).
	Then actually leave the cluster.

```
dkron leave [flags]
```

### Options

```
      --cert-file string         Path to the client server TLS cert file
  -h, --help                     help for leave
      --key-file string          Path to the client server TLS key file
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron](/docs/pro/cli/dkron/)	 - Professional distributed job scheduling system

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Raft (`website/docs/pro/cli/dkron_raft.md`)

---
date: 2024-10-06
title: "dkron raft"
slug: dkron_raft
url: /docs/pro/cli/dkron_raft/
---
## dkron raft

Command to perform some raft operations

### Options

```
      --cert-file string         Path to the client server TLS cert file
  -h, --help                     help for raft
      --key-file string          Path to the client server TLS key file
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron](/docs/pro/cli/dkron/)	 - Professional distributed job scheduling system
* [dkron raft list-peers](/docs/pro/cli/dkron_raft_list-peers/)	 - Command to list raft peers
* [dkron raft remove-peer](/docs/pro/cli/dkron_raft_remove-peer/)	 - Command to list raft peers

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Raft List Peers (`website/docs/pro/cli/dkron_raft_list-peers.md`)

---
date: 2024-10-06
title: "dkron raft list-peers"
slug: dkron_raft_list-peers
url: /docs/pro/cli/dkron_raft_list-peers/
---
## dkron raft list-peers

Command to list raft peers

```
dkron raft list-peers [flags]
```

### Options

```
  -h, --help   help for list-peers
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron raft](/docs/pro/cli/dkron_raft/)	 - Command to perform some raft operations

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Raft Remove Peer (`website/docs/pro/cli/dkron_raft_remove-peer.md`)

---
date: 2024-10-06
title: "dkron raft remove-peer"
slug: dkron_raft_remove-peer
url: /docs/pro/cli/dkron_raft_remove-peer/
---
## dkron raft remove-peer

Command to list raft peers

```
dkron raft remove-peer [flags]
```

### Options

```
  -h, --help             help for remove-peer
      --peer-id string   Remove a Dkron server with the given ID from the Raft configuration.
```

### Options inherited from parent commands

```
      --config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")
```

### SEE ALSO

* [dkron raft](/docs/pro/cli/dkron_raft/)	 - Command to perform some raft operations

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Cli Dkron Version (`website/docs/pro/cli/dkron_version.md`)

---
date: 2024-10-06
title: "dkron version"
slug: dkron_version
url: /docs/pro/cli/dkron_version/
---
## dkron version

Show version

### Synopsis

Show the version

```
dkron version [flags]
```

### Options

```
  -h, --help   help for version
```

### Options inherited from parent commands

```
      --config string   config file (default is /etc/dkron/dkron.yml)
```

### SEE ALSO

* [dkron](/docs/pro/cli/dkron/)	 - Professional distributed job scheduling system

###### Auto generated by spf13/cobra on 6-Oct-2024

---

### Pro Executors Docker (`website/docs/pro/executors/docker.md`)

# Docker executor

Docker executor can launch docker based cron jobs using the docker command of the target node.

This executor needs a recent version of docker to be available and configured in the target node.

## Configuration

To run a docker job create a job config with the docker executor as in this example:

```json
{
  "executor": "docker",
  "executor_config": {
    "image": "alpine", //docker image to use
    "volumes": "/logs:/var/log/", //comma separated list of volume mappings
    "command": "echo \"Hello from dkron\"", //command to pass to run on container
    "env": "ENVIRONMENT=variable" //environment variables to pass to the container
  }
}
```

---

### Pro Executors Ecs (`website/docs/pro/executors/ecs.md`)

# AWS ECS Executor

The ECS exeutor is capable of launching tasks in ECS clusters, then listen to a stream of CloudWatch Logs and return the output.

To configure a job to be run in ECS, the executor needs a JSON Task definition template or an already defined task in ECS.

To allow the ECS Task runner to run tasks, the machine running Dkron needs to have the appropriate permissions configured in AWS IAM:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1460720941000",
            "Effect": "Allow",
            "Action": [
                "ecs:RunTask",
                "ecs:DescribeTasks",
                "ecs:DescribeTaskDefinition",
                "logs:FilterLogEvents",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
```

To configure a job to be run with the ECS executor:

Example using an existing taskdef

```json
{
  "executor": "ecs",
  "executor_config": {
    "taskdefName": "mytaskdef-family",
    "region": "eu-west-1",
    "cluster": "default",
    "env": "ENVIRONMENT=variable",
    "service": "mycontainer",
    "overrides": "echo,\"Hello from dkron\"",
    "propagateTags": "TASK_DEFINITION"
  }
}
```

Example using a provided taskdef

```json
{
  "executor": "ecs",
  "executor_config": {
    "taskdefBody": "{\"containerDefinitions\": [{\"essential\": true,\"image\": \"hello-world\",\"memory\": 100,\"name\": \"hello-world\"}],\"family\": \"helloworld\"}",
    "region": "eu-west-1",
    "cluster": "default",
    "fargate": "yes",
    "env": "ENVIRONMENT=variable",
    "maxAttempts": "5000"
  }
}
```

This is the complete list of configuration parameters of the plugin:

```
taskdefBody
taskdefName
region
cluster
logGroup
fargate
securityGroup
subnet
env
service
overrides
maxAttempts // Defaults to 2000, will perform a check every 6s * 2000 times waiting a total of 12000s or 3.3h
enableExecuteCommand
```

---

### Pro Processors Elasticsearch (`website/docs/pro/processors/elasticsearch.md`)

# Elasticsearch processor

The Elasticsearch processor can fordward execution logs to an ES cluster. It need an already available Elasticsearch installation that is visible in the same network of the target node.

The output logs of the job execution will be stored in the indicated ES instace.

:::info For Dkron Pro < v3.2.3
:::
## Configuration

```json
{
  "processors": {
    "elasticsearch": {
      "url": "http://localhost:9200", //comma separated list of Elasticsearch hosts urls (default: http://localhost:9200)
      "index": "dkron_logs", //desired index name (default: dkron_logs)
      "forward": "false" //forward logs to the next processor (default: false)
    }
  }
}
```

:::info For Dkron Pro > v3.2.3
:::

## Configuration

For increased security and flexibility, configuration of the ES processor is stored in a file named `dkron-processor-elasticsearch.yml` in the same locations as `dkron.yml`, and should include a list of configurations for Elasticsearch, it can include any number of configurations.

This is an example including all available parameters:
```yaml
es1:
  index: dkron-logs
  index_date_format: '2006-01-02'
  username: elastic
  password: XXXXXXXXXXXXX
  url: https://localhost:9200
```
And for each job you only need to configure the `config` parameter in the processors configuration:

```json
{
  "processors": {
    "elasticsearch": {
      "config": "es1", // configuration to use from the config file
      "forward": "false" // forward logs to the next processor (default: false)
    }
  }
}
```

---

### Pro Processors Email (`website/docs/pro/processors/email.md`)

# Email processor

The Email processor provides flexibility to job email notifications.

Configuration of the email processor is stored in a file named `dkron-processor-email.yml` in the same locations as `dkron.yml`, and should include a list of providers, it can include any number of providers.

Example:
```yaml
provider1:
  host: smtp.myprovider.com
  port: 25
  username: myusername
  password: mypassword
  from: cron@mycompany.com
  subjectPrefix: '[Staging] '
```

Then configure each job with the following options:

Example:

```json
{
  "processors": {
    "email": {
      "provider": "provider1",
      "emails": "team@mycompany.com, owner@mycompany.com",
      "onSuccess": "true"
    }
  }
}
```

By default the email procesor doesn't send emails on job success, the `onSuccess` parameter, enables it, like in the previous example.

---

### Pro Processors Slack (`website/docs/pro/processors/slack.md`)

# Slack processor

The Slack processor provides slack notifications with multiple configurations and rich format.

Configuration of the slack processor is stored in a file named `dkron-processor-slack.yml` in the same locations as `dkron.yml`, and should include a list of teams, it can include any number of teams.



Example:
```yaml
team1:
  webhook_url: https://hooks.slack.com/services/XXXXXXXXXXXXXXXXXXX
  bot_name: Dkron Production
```

Then configure each job with the following options:

Example:

```json
{
  "processors": {
    "slack": {
      "team": "team1",
      "channel": "#cron-production",
      "onSuccess": "true"
    }
  }
}
```

By default the slack procesor doesn't send notifications on job success, the `onSuccess` parameter, enables it, like in the previous example.

---

### Usage Executors Gcp Pubsub (`website/docs/usage/executors/gcp-pubsub.md`)

# Google Cloud PubSub Executor

A basic GC PubSub executor that produces a message on a Google Cloud PubSub topic.

## Configuration

Params

```
project:                     Google Cloud project ID. Required
topic:                       The topic name for the message. Required
data:                        The actual message body in base64 format. Required
attributes:                  The attributes of the message in JSON format. Optional
```

Example:

```json
{
  "executor": "gcppubsub",
  "executor_config": {
    "project": "project-id",
    "topic": "topic-name",
    "data": "aGVsbG8gd29ybGQ=",
    "attributes": "{\"hello\":\"world\",\"waka\":\"paka\"}"
  }
}
```

---

### Usage Executors Grpc (`website/docs/usage/executors/grpc.md`)

# GRPC Executor

GRPC executor can send a request to a GRPC Server

## Requirements

In order to serialize protobufs, the server needs to have the reflection service active.
Without that we cannot get proto descriptors required for serialization.

## Configuration

Params:

```
"url": Required, Request url
"body": Optional, POST body
"timeout": Optional, Request timeout, unit seconds
"expectCode": Optional, One of https://grpc.github.io/grpc/core/md_doc_statuscodes.html
```

Example:

```json
{
  "executor": "http",
  "executor_config": {
    "url": "127.0.0.1:9000/test.TestService/Test",
    "body": "{\"key\": \"value\"}",
    "timeout": "30",
    "expectCode": "0"
  }
}
```

---

### Usage Executors Http (`website/docs/usage/executors/http.md`)

# HTTP Executor

HTTP executor can send a request to an HTTP endpoint

## Configuration

Params:

```
method:     Request method in uppercase
url:        Request url
headers:    Json string, such as "[\"Content-Type: application/json\"]"
body:       POST body
timeout:    Request timeout, unit seconds
expectCode: Expect response code, such as 200,206
expectBody: Expect response body, support regexp, such as /success/
debug:      Debug option, will log everything when this option is not empty
tlsNoVerifyPeer:       false (default) or true. If true, disables verification of the remote SSL certificate's validity.
tlsCertificateFile:    Path to the PEM file containing the client certificate. Optional.
tlsCertificateKeyFile: Path to the PEM file containing the client certificate private key. Optional.
tlsRootCAsFile:        Path to the PEM file containing certificates to use as root CAs. Optional.
```

Example:

```json
{
  "executor": "http",
  "executor_config": {
    "method": "GET",
    "url": "http://example.com",
    "headers": "[]",
    "body": "",
    "timeout": "30",
    "expectCode": "200",
    "expectBody": "",
    "debug": "true"
  }
}
```

---

### Usage Executors Kafka (`website/docs/usage/executors/kafka.md`)

# Kafka Executor

A basic Kafka executor that produces a message on a Kafka broker.

## Configuration

Params:

```
brokerAddress:          Comma separated string containing "IP:port" of the brokers
key:                    The key of the message to produce
message:                The body of the message to produce
topic:                  The Kafka topic for this message
tlsEnable:              Enables TLS if set to true. Optional
tlsInsecureSkipVerify:  Disables verification of the remote SSL certificate's validity if set to true. Optional
saslUsername:           The SASL username for authentication. If set, saslPassword and saslMechanism must also be provided.
saslPassword:           The SASL password for authentication. If set, saslUsername and saslMechanism must also be provided.
saslMechanism:          The SASL SCRAM mechanism to use, either "sha256" or "sha512". This is required if both saslUsername and saslPassword are provided.
debug:                  Turns on debugging output if not empty
```

Example:

```json
{
  "executor": "kafka",
  "executor_config": {
    "brokerAddress": "localhost:9092,another.host:9092",
    "key": "My key",
    "message": "My message",
    "topic": "my_topic"
  }
}
```

---

### Usage Executors Nats (`website/docs/usage/executors/nats.md`)

# NATS Executor

The NATS executor sends a message to a NATS server/cluster.

Currently, only username/password authentication is supported.

## Configuration

Params:

```
url:      Comma separated list of NATS server URLs
message:  The message to send
subject:  The subject to send the message to
userName: username for authentication
password: password for authentication
debug:    If not empty, turns on debugging. Will log the NATS specific job config and the request sent.
```

Example:

```json
{
  "executor": "nats",
  "executor_config": {
    "url": "tls://nats.demo.io:4443",
    "message": "the message",
    "subject": "myfavoritesubject",
    "userName": "someusername",
    "password": "somepassword"
  }
}
```

---

### Usage Executors Rabbitmq (`website/docs/usage/executors/rabbitmq.md`)

# RabbitMQ Executor

A basic RabbitMQ executor that produces a message on a RabbitMQ server/cluster.

## Configuration

Params

```
url:                         URL of the RabbitMQ server/cluster with VHost. Example: amqp://guest:guest@localhost:5672/
queue.name:                  The name of the queue to publish the message to. Required
queue.durable:               Whether the queue is durable or not. Optional, defaults to false
queue.auto_delete:           Whether the queue is auto-deleted after publishing the message. Optional, defaults to false
queue.exclusive:             Whether the queue is exclusive or not. Optional, defaults to false
message.content_type:        The content type of the message
message.delivery_mode:       Message delivery mode. 1 for non-persistent, 2 for persistent. Optional, defaults to 0
message.messageId:           The message ID
message.body:                The actual message body in desired format that will be sent to the queue
message.base64:              Encoded message body in base64 format. Optional, but should not be set if message.body is set.
exchange:                    The RabbitMQ exchange. Optional, default to ""
```

Example:

```json
{
  "executor": "rabbitmq",
  "executor_config": {
    "url": "amqp://guest:guest@localhost:5672/",
    "exchange": "amq.default",
    "queue.name": "test",
    "queue.create": "true",
    "queue.durable": "true",
    "queue.auto_delete": "false",
    "queue.exclusive": "false",
    "message.content_type": "application/json",
    "message.delivery_mode": "2",
    "message.messageId": "4373732772",
    "message.body": "{\"key\":\"value\"}"
  }
}
```

---

### Usage Executors Shell (`website/docs/usage/executors/shell.md`)

# Shell Executor

The Shell executor is one of the most versatile executors in Dkron, allowing you to run any system command or script on target nodes.

## Overview

The shell executor runs commands on the target node's operating system. It can:

- Execute system commands
- Run scripts in various languages (bash, python, etc.)
- Perform file system operations
- Interact with local services and processes

## Configuration Parameters

| Parameter | Required | Description |
|-----------|:--------:|-------------|
| `shell` | No | When "true", runs the command in a shell environment (bash/cmd). Default: "false" |
| `command` | Yes | The command or script to execute |
| `env` | No | Environment variables in the format "KEY1=value1,KEY2=value2" |
| `cwd` | No | The working directory to run the command from |
| `su` | No | Unix only. Run the command as a different user or `user:group` |
| `timeout` | No | Maximum execution time after which the job is forcefully terminated |
| `mem_limit` | No | Maximum memory usage after which the job is forcefully terminated. Supports units: B, KB, MB, GB, TB |

## Basic Usage Examples

### Simple Command Execution

```json
{
  "executor": "shell",
  "executor_config": {
    "command": "echo Hello Dkron"
  }
}
```

### Using Shell Features

```json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "ps aux | grep nginx | wc -l"
  }
}
```

### Setting Environment Variables

```json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "python /scripts/process_data.py",
    "env": "DATA_PATH=/data/incoming,LOG_LEVEL=info,API_KEY=secret123"
  }
}
```

### Changing Working Directory

```json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "./run_backup.sh",
    "cwd": "/opt/backups"
  }
}
```

### Setting a Timeout

```json
{
  "executor": "shell",
  "executor_config": {
    "command": "/opt/scripts/long_running_task.sh",
    "timeout": "1h30m"
  }
}
```

### Setting a Memory Limit

```json
{
  "executor": "shell",
  "executor_config": {
    "command": "/opt/scripts/memory_intensive_task.sh",
    "mem_limit": "512MB"
  }
}
```

### Combining Timeout and Memory Limit

```json
{
  "executor": "shell",
  "executor_config": {
    "command": "/opt/scripts/resource_intensive_task.sh",
    "timeout": "30m",
    "mem_limit": "1GB"
  }
}
```

## Advanced Examples

### Running a Multi-line Script

```json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "#!/bin/bash\necho 'Starting job'\ncd /tmp\ndate > last_run.txt\necho 'Job completed'"
  }
}
```

### Database Backup Example

```json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "pg_dump -U postgres -d mydb | gzip > /backups/mydb_$(date +%Y%m%d_%H%M%S).sql.gz",
    "env": "PGPASSWORD=securepassword",
    "timeout": "30m"
  }
}
```

### System Health Check with Exit Code

```json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "if [ $(df -h | grep '/dev/sda1' | awk '{print $5}' | tr -d '%') -gt 90 ]; then echo 'Disk space critical'; exit 1; else echo 'Disk space OK'; fi"
  }
}
```

## Monitoring and Metrics

The shell executor exposes Prometheus metrics on port 9422 (configurable with `SHELL_EXECUTOR_PROMETHEUS_PORT` environment variable):

| Metric | Type | Description | Labels |
|--------|------|-------------|--------|
| `dkron_job_cpu_usage` | gauge | Current CPU usage by job | job_name |
| `dkron_job_mem_usage_kb` | gauge | Current memory consumed by job (KB) | job_name |

## Exit Codes and Success/Failure

By default, an exit code of 0 indicates success, while any non-zero exit code indicates failure. You can customize which exit codes are considered successful in the job definition:

```json
{
  "success_count": "0,1,2",  // Exit codes 0, 1, and 2 will be considered successful
  "executor": "shell",
  "executor_config": {
    "command": "/opt/scripts/check.sh"
  }
}
```

## Security Considerations

The shell executor runs commands with the same permissions as the Dkron process. Consider the following security best practices:

1. **Principle of Least Privilege**: Run Dkron with a dedicated user that has only the permissions required for its jobs
2. **Avoid Sensitive Data in Commands**: Use environment variables for sensitive values instead of embedding them in commands
3. **Input Validation**: Validate any dynamic parts of commands, especially if they come from external sources
4. **Output Sanitization**: Be cautious when using command output, particularly if it's displayed in the UI or logs

## Troubleshooting

Common issues and solutions when working with the shell executor:

### Command Not Found

If you see "command not found" errors:
- Verify the command is installed on the target node
- Use absolute paths to executables (e.g., `/usr/bin/python` instead of just `python`)
- Check if the command is in the PATH of the user running Dkron

### Permission Issues

If you encounter permission errors:
- Check if the Dkron user has the necessary permissions
- Adjust file/directory permissions as needed
- Consider using `sudo` if appropriate (requires sudo configuration)

### Timeouts

If jobs are timing out:
- Adjust the `timeout` parameter to match the expected runtime
- Optimize the command to run more efficiently
- Consider breaking large jobs into smaller, chained jobs

### Memory Limits

If jobs are being killed due to memory limits:
- Adjust the `mem_limit` parameter to accommodate the job's memory requirements
- Optimize the command to use less memory (e.g., process data in chunks)
- Monitor memory usage patterns to set appropriate limits
- Consider if the job should run on a node with more available memory

### Environment Variables

If environment variables aren't working:
- Ensure the format is correct (`KEY=value,KEY2=value2`)
- Remember that values are passed as strings
- For complex values with spaces or special characters, consider using files instead

## Additional Resources

- [Process Output with Processors](/docs/usage/processors)
- [Job Chaining for Complex Workflows](/docs/usage/chaining)
- [Security Best Practices](/docs/usage/concepts#security-considerations)

---

### Usage Plugins Develop (`website/docs/usage/plugins/develop.md`)

---
title: Developing plugins
weight: 99
---

## Developing a Plugin

:::warning Advanced topic!
Plugin development is a highly advanced topic, and is not required knowledge for day-to-day usage. If you don't plan on writing any plugins, we recommend not reading the following section of the documentation.
:::

Developing a plugin is simple. The only knowledge necessary to write a plugin is basic command-line skills and basic knowledge of the Go programming language.

Note: A common pitfall is not properly setting up a `$GOPATH`. This can lead to strange errors. You can read more about this here to familiarize yourself.

Create a new Go project somewhere in your `$GOPATH`. If you're a GitHub user, we recommend creating the project in the directory `$GOPATH/src/github.com/USERNAME/dkron-NAME-TYPE`, where `USERNAME` is your GitHub username and `NAME` is the name of the plugin you're developing. This structure is what Go expects and simplifies things down the road.

With the directory made, create a main.go file. This project will be a binary so the package is "main":

```go
package main

import (
	"github.com/distribworks/dkron/v3/plugin"
)

func main() {
	plugin.Serve(&plugin.ServeOpts{
		Processor: new(MyPlugin),
	})
}
```

And that's basically it! You'll have to change the argument given to plugin.Serve to be your actual plugin, but that is the only change you'll have to make. The argument should be a structure implementing one of the plugin interfaces (depending on what sort of plugin you're creating).

Dkron plugins must follow a very specific naming convention of `dkron-TYPE-NAME`. For example, `dkron-processor-files`, which tells Dkron that the plugin is a processor that can be referenced as "files".

---

### Usage Processors File (`website/docs/usage/processors/file.md`)

---
title: File Processor
---

File processor saves the execution output to a single log file in the specified directory

## Configuration

Parameters

```
log_dir: Path to the location where the log files will be saved
forward: Forward log output to the next processor
```

Example

```json
{
    "name": "job_name",
    "command": "echo 'Hello files'",
    "schedule": "@every 2m",
    "tags": {
        "role": "web"
    },
    "processors": {
        "files": {
            "log_dir": "/var/log/mydir",
            "forward": "true"
        }
    }
}
```

---

### Usage Processors Log (`website/docs/usage/processors/log.md`)

---
title: Log Processor
---

Log processor writes the execution output to stdout/stderr

## Configuration

Parameters

`forward: Forward the output to the next processor`

Example

```json
{
    "name": "job_name",
    "command": "echo 'Hello log'",
    "schedule": "@every 2m",
    "tags": {
        "role": "web"
    },
    "processors": {
        "log": {
            "forward": "true"
        }
    }
}
```

---

### Usage Processors Syslog (`website/docs/usage/processors/syslog.md`)

---
title: Syslog Processor
---

Syslog processor writes the execution output to the system syslog daemon.

Note: Only works on linux systems

## Configuration

Parameters

`forward: Forward the output to the next processor`

Example

```json
{
  "name": "job_name",
  "command": "echo 'Hello syslog'",
  "schedule": "@every 2m",
  "tags": {
    "role": "web"
  },
  "processors": {
    "syslog": {
      "forward": "true"
    }
  }
}
```

---

--- METRICS ---
- Files Extracted: 98
- Estimated Token Budget: ~61605 tokens
- Canonical Reference: https://codewiki.google/github.com/dkron-io/dkron