dkron Documentation

Dkron - Distributed, fault tolerant job scheduling system

4,740 stars Go Go 1 file · ~61,683 tokens #cron#distributed-systems#fault-tolerance#scheduled-jobs
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy dkron on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name dkron -p 8080:8080 --restart always dkron-io/dkron:latest
Configured Services & Ports
⚙️ dkron-leader
image: build locally
🔌 Ports: 8080:8080, 8946:8946, 6868:6868
docker-compose.yml
yaml (docker-compose.yml) 
# Docker Compose file for E2E testing
# This creates a minimal Dkron cluster for testing

services:
  # Primary leader node
  dkron-leader:
    build:
      context: ..
      dockerfile: Dockerfile
    command: dkron agent --server --log-level=debug --bootstrap-expect=1 --node-name=dkron-leader
    ports:
      - "8080:8080"
      - "8946:8946"
      - "6868:6868"
    environment:
      - GODEBUG=netdns=go
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 10s
    networks:
      - dkron-e2e

networks:
  dkron-e2e:
    driver: bridge

Technical Documentation: dkron-io/dkron

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

Dkron - Distributed, fault tolerant job scheduling system for cloud native environments [](https://godoc.org/github.com/distribworks/dkron) [](https://github.com/distribworks/dkron/actions) [](https://gitter.im/distribworks/dkron) [](https://gurubase.io/g/dkron)

Website: http://dkron.io/

Dkron is a distributed cron service, easy to setup and fault tolerant with focus in:

  • Easy: Easy to use with a great UI
  • Reliable: Completely fault tolerant
  • Highly scalable: Able to handle high volumes of scheduled jobs and thousands of nodes

Dkron is written in Go and leverage the power of the Raft protocol and Serf for providing fault tolerance, reliability
and scalability while keeping simple and easily installable.

Dkron is inspired by the google
whitepaper Reliable Cron across the Planet and by Airbnb Chronos
borrowing the same features from it.

Dkron runs on Linux, OSX and Windows. It can be used to run scheduled commands on a server cluster using any combination
of servers for each job. It has no single points of failure due to the use of the Gossip protocol and fault tolerant
distributed databases.

You can use Dkron to run the most important part of your company, scheduled jobs.

Installation

Installation instructions

Full, comprehensive documentation is accessible on the Dkron website

Quickstart

Deploying Dkron using Docker

The best way to test and develop dkron is using docker, you will need Docker with Docker
compose installed before proceeding.

bash
docker compose up -d

The UI should be available on http://localhost:8080/ui.

Using Dkron

To add jobs to the system read the API docs.

Scaling the cluster

To add more Dkron instances to the cluster:

bash
docker compose up -d --scale dkron-server=4
docker compose up -d --scale dkron-agent=10

Development

To develop Dkron, you can deploy the cluster with local changes applied with the following steps:

  1. Clone the repository.

  2. Run the docker compose:

    bash
    docker compose -f docker-compose.dev.yml up

Email Testing

For testing email notifications during development, Mailpit is included in the development docker-compose setup. Mailpit provides a local SMTP server that captures outgoing emails without sending them to real recipients.

Start Mailpit with the development environment:

bash
docker compose -f docker-compose.dev.yml up mailpit

Or run it standalone:

bash
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit

View captured emails at: http://localhost:8025

For more information, see docs/EMAIL_TESTING.md.

Testing CI Locally

To validate that your changes will pass in GitHub Actions before pushing:

bash
./scripts/test-ci-locally.sh

This script:

  • Starts Mailpit (simulating the CI service container)
  • Runs tests with the same configuration as GitHub Actions
  • Provides clear pass/fail results
  • Allows you to inspect emails in the Mailpit UI

See .github/TESTING.md for more information about CI testing.

Frontend development

Dkron dashboard is built using React Admin as a single page application.

To start developing the dashboard enter the ui directory and run npm install to get the frontend dependencies and
then start the local server with npm start it should start a new local web server and open a new browser window
serving de web ui.

Make your changes to the code, then run make ui to generate assets files. This is a method of embedding resources in
Go applications.

Resources

Chef cookbook
https://supermarket.chef.io/cookbooks/dkron

Python Client Library
https://github.com/oldmantaiter/pydkron

Ruby client
https://github.com/jobandtalent/dkron-rb

PHP client
https://github.com/gromo/dkron-php-adapter

Terraform provider
https://github.com/bozerkins/terraform-provider-dkron

Manage and run jobs in Dkron from your django project
https://github.com/surface-security/django-dkron

Contributors

Made with contrib.rocks.

Get in touch

Overview (`client/README.md`)

Dkron Go client

Installation

bash
go get github.com/distribworks/dkron/v4/client

Usage

go
package main

import (
	"context"

	"github.com/distribworks/dkron/v4/client"
	log "github.com/sirupsen/logrus"
)

func main() {
	newClient, err := client.NewClient("http://localhost:9090")
	if err != nil {
		log.Fatal(err)
	}

	// Create a new job
	params := &client.CreateOrUpdateJobParams{
		Runoncreate: nil,
	}
	body := client.CreateOrUpdateJobJSONRequestBody{
		Name:       "testjob1",
		Timezone:   "Europe/Paris",
		Schedule:   "@every 1m",
		Owner:      "test",
		OwnerEmail: "",
		Tags: map[string]string{
			"tag1": "value1",
			"tag2": "value2",
		},
		Retries:  3,
		Executor: "Shell",
		ExecutorConfig: map[string]string{
			"command": "echo hello world!",
		},
		Displayname: "Job",
		Ephemeral:   false,
	}

	_, err = newClient.CreateOrUpdateJob(context.Background(), params, body)
	if err != nil {
		log.WithError(err).Error("Failed to create job")
	}
}

Overview (`ui/README.md`)

ui

Installation

Install the application dependencies by running:

sh
yarn

Development

Start the application in development mode by running:

sh
yarn dev

Production

Build the application in production mode by running:

sh
yarn build

DataProvider

The included data provider use ra-data-simple-rest. It fits REST APIs using simple GET parameters for filters and sorting. This is the dialect used for instance in FakeRest.

You'll find an .env file at the project root that includes a VITE_JSON_SERVER_URL variable. Set it to the URL of your backend.

Overview (`website/README.md`)

Website

This website is built using Docusaurus 2, a modern static website generator.

Installation

text
$ yarn

Local Development

text
$ yarn start

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

Build

text
$ yarn build

This command generates static content into the build directory and can be served using any static contents hosting service.

Deployment

Using SSH:

text
$ USE_SSH=true yarn deploy

Not using SSH:

text
$ GIT_USER=<Your GitHub username> yarn deploy

If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the gh-pages branch.

Overview (`builtin/bins/dkron-executor-rabbitmq/README.md`)

RabbitMQ Executor

Executor Configuration

The options names are inherited from the RabbitMQ Publishers

Option Required Description Default
url yes RabbitMQ connection string -
exchange no RabbitMQ Exchange ""
queue.name yes Queue name to send message to -
queue.create no Create queue if not exists false
queue.durable no Durable queue false
queue.auto_delete no Auto delete queue false
queue.exclusive no Exclusive queue false
message.content_type no Message content type text/plain
message.delivery_mode no Message delivery mode 0
message.messageId no Message id ""
message.body yes Message body -
message.base64Body yes Base64 encoded message body -

Example

shell
curl localhost:8080/v1/jobs -XPOST -d '{
  "name": "job1",
  "schedule": "@every 10s",
  "timezone": "Europe/Berlin",
  "owner": "Platform Team",
  "owner_email": "[email protected]",
  "disabled": false,
  "tags": {
    "server": "true:1"
  },
  "metadata": {
    "user": "12345"
  },
  "concurrency": "allow",
  "executor": "rabbitmq",
  "executor_config": {
  		"url": "amqp://guest:guest@localhost:5672/",
      "exchange": "amq.default",
  		"queue.name": "test",
  		"queue.create": "true",
  		"queue.durable": "true",
  		"queue.auto_delete": "false",
  		"queue.exclusive": "false",
  		"message.content_type": "application/json",
  		"message.delivery_mode": "2",
  		"message.messageId": "4373732772",
  		"message.body": "{\"key\":\"value\"}"
  }
}'

Intro (`website/docs/intro.md`)

Intro

Dkron - Distributed, fault tolerant job scheduling system

Welcome to the Dkron documentation! This is the reference guide on how to use Dkron. If you want a getting started guide refer to the getting started guide.

What is Dkron

Dkron is a distributed system to run scheduled jobs against a server or a group of servers of any size. One of the machines is the leader and the others will be followers. If the leader fails or becomes unreachable, any other one will take over and reschedule all jobs to keep the system healthy.

In case the old leader becomes alive again, it'll become a follower.

Dkron is a distributed cron drop-in replacement, easy to setup and fault tolerant with focus in:

  • Easy: Easy to use with a great UI
  • Reliable: Completely fault tolerant
  • Highly scalable: Able to handle high volumes of scheduled jobs and thousands of nodes

Dkron is written in Go and leverages the power of distributed key value stores and Serf for providing fault tolerance, reliability and scalability while remaining simple and easily installable.

Dkron is inspired by the google whitepaper Reliable Cron across the Planet

Dkron runs on Linux, OSX and Windows. It can be used to run scheduled commands on a server cluster using any combination of servers for each job. It has no single points of failure due to the use of the fault tolerant distributed databases and can work at large scale thanks to the efficient and lightweight gossip protocol.

Dkron uses the efficient and lightweight gossip protocol underneath to communicate with nodes. Failure notification and task handling are run efficiently across an entire cluster of any size.

For a detailed overview of Dkron's internal architecture, components, and how they work together, see the Architecture documentation.

Key Concepts

Jobs

Jobs are the core entity in Dkron. A job consists of:

  • Name: Unique identifier for the job
  • Schedule: When to run the job (cron expression)
  • Command: What to run
  • Executor: How to run the command (shell, HTTP, etc.)
  • Processors: How to process the output
  • Tags: Key-value pairs for node selection
  • Concurrency: Options to control concurrent execution
  • Dependent Jobs: Jobs that should run after this job completes

Tags and Node Selection

Dkron uses tags to control which nodes execute specific jobs:

  • Node Tags: Assigned to nodes during startup (--tag key=value)
  • Job Tags: Specified in job definitions ("tags": {"role": "web"})
  • Tag Matching: Jobs run on nodes where all job tags match node tags

Concurrency Options

Dkron provides several options to control job concurrency:

  • Concurrency: Allow (or disallow) concurrent executions of the same job

Status Codes and Retries

Jobs can be configured with:

  • Retries: Number of times to retry a failed execution

Job Dependencies

Dkron supports job dependencies for complex workflows:

  • Parent-Child Relationships: Jobs can depend on other jobs
  • Status Checking: Child jobs run only if parent jobs succeed
  • Chained Execution: Create multi-step job pipelines

Web UI

The Dkron web UI provides an easy-to-use interface for:

  1. Creating and editing jobs
  2. Viewing execution history and logs
  3. Monitoring cluster status
  4. Running jobs manually
  5. Managing job dependencies

Design Philosophy

Dkron is designed to solve one problem well: executing commands at given intervals. Following the Unix philosophy of doing one thing and doing it well (like the battle-tested cron), but with the addition of being designed for the cloud era, removing single points of failure in environments where scheduled jobs need to be run across multiple servers.

Basics Getting Started (`website/docs/basics/getting-started.md`)

Getting started

Introduction

Dkron is a distributed job scheduling system that runs across multiple nodes. Understanding its architecture will help you deploy it effectively.

Node Types

Dkron nodes can work in two modes:

  • Agents: Cluster members that handle job executions, run scripts, and return the output to servers.
  • Servers: Cluster members that send job execution queries and also handle scheduling, data storage, and leader election.

Leadership and Job Execution

  • A single server node is elected as the leader in each Dkron cluster
  • The leader is responsible for initiating job execution queries across the cluster
  • Any Dkron agent or server can run scheduled jobs
  • By default, all nodes execute each job, but this is configurable

Job Targeting

You can control which nodes run a job by:

  • Specifying tags on nodes
  • Setting a count of target nodes with specific tags in your job definition

This provides flexibility in running jobs across clusters of any size with precise node targeting.

Execution Flow

  1. The leader schedules a job based on its timing configuration
  2. Target nodes are selected based on tags and count
  3. Selected nodes execute the job
  4. Execution responses are collected by the scheduler
  5. Results are stored in the database

State Storage

Dkron deployment is just a single binary. It stores state in an internal BoltDB instance and replicates all changes between server nodes using the Raft protocol. No external storage system is required.

Installation

See the installation.

Configuration

See the configuration.

Usage

By default Dkron uses the following ports:

  • 8946 for serf layer between agents
  • 8080 for HTTP for the API and Dashboard
  • 6868 for gRPC and raft layer comunication between agents.

:::info
Be sure you have opened this ports (or the ones that you configured) in your firewall or AWS security groups.
:::

Starting a single node

Works out of the box, good for non HA installations.

  • System service: If no changes are done to the default config files, dkron will start as a service in single mode.
  • Command line: Running a single node with default config can be done by running:
text
dkron agent --server --bootstrap-expect=1

Check your server is working: curl localhost:8080/v1

:::info
For a full list of configuration parameters and its description, see the CLI docs
:::

Create a Job

:::info
This job will only run in just one server node due to the node count in the tag. Refer to the target node spec for details.
:::

bash
curl localhost:8080/v1/jobs -XPOST -d '{
  "name": "job1",
  "schedule": "@every 10s",
  "timezone": "Europe/Berlin",
  "owner": "Platform Team",
  "owner_email": "[email protected]",
  "disabled": false,
  "tags": {
    "server": "true:1"
  },
  "metadata": {
    "user": "12345"
  },
  "concurrency": "allow",
  "executor": "shell",
  "executor_config": {
    "command": "date"
  }
}'

For full Job params description refer to the Job model in the API guide

That's it!

To start configuring an HA installation of Dkron follow the [clustering guide](/docs/usage/clustering)

Pro Index (`website/docs/pro/_index.md`)

title: Dkron Pro weight: 40

{{% children style="li" depth="999" %}}

Usage Executors Index (`website/docs/usage/executors/index.md`)

Executors

Executors are the core plugins that perform actual job tasks in Dkron. They implement different execution strategies and allow Dkron to interact with various systems and services.

Overview

Executors define how jobs are executed on target nodes. When a job is scheduled to run, Dkron uses the specified executor to perform the actual work. Each executor has its own configuration options and capabilities.

:::info
If you only plan to use the built-in executors, http and shell you can use the Dkron Light edition that only includes a single binary as the plugins are built-in.
:::

Built-in Executors

Dkron includes the following built-in executors:

Executor Description Use Case
Shell Executes shell commands on target nodes Running scripts, system commands
HTTP Makes HTTP requests to specified endpoints API integrations, web hooks

Extended Executors (Open Source)

These executors are available as separate plugins in the open source version:

Executor Description Use Case
Kafka Produces messages to Kafka topics Event-driven architectures, data pipelines
NATS Publishes messages to NATS servers Microservices communication, event publishing
RabbitMQ Sends messages to RabbitMQ exchanges Message queuing, work distribution
gRPC Makes calls to gRPC services Microservices integration, efficient RPC
Google Pub/Sub Publishes messages to Google Pub/Sub topics Cloud-native event publishing

Pro Executors

Dkron Pro includes additional commercially supported executors:

Executor Description Use Case
Docker Runs containers and commands within containers Isolated execution environments
AWS ECS Runs tasks on Amazon ECS clusters Serverless job execution in AWS

Executor Architecture

Executors follow a plugin architecture in Dkron:

mermaid
flowchart TD
    DkronServer["Dkron Server"] -->|Job Assignment| ExecutorPlugin["Executor Plugin"]
    ExecutorPlugin -->|Performs Action| TargetSystem["Target System"]
  1. The Dkron server schedules a job and assigns it to a node
  2. The node loads the specified executor plugin
  3. The executor performs its action on the target system
  4. Results are returned to Dkron for processing

Common Executor Configuration Patterns

Most executors share some common configuration concepts:

  1. Timeouts: Limit how long a job can run before being terminated
  2. Environment Variables: Pass context to the execution environment
  3. Credentials: Securely provide authentication information
  4. Retry Logic: Define behavior when execution fails

Executor Selection Guide

When choosing an executor for your job, consider:

  1. Target System: What system or service needs to be interacted with?
  2. Isolation Needs: Does the job need to run in an isolated environment?
  3. Resource Requirements: How much CPU/memory is needed?
  4. Security Considerations: What permissions or credentials are required?

Custom Executors

Dkron supports custom executors through its plugin system. You can develop your own executors to meet specific requirements not covered by existing plugins.

To develop a custom executor:

  1. Implement the Executor interface from the Dkron plugin package
  2. Follow the Go plugin system requirements
  3. Build and deploy your executor to the Dkron plugins directory

See the plugin development guide for more details.

Best Practices

  1. Use the Simplest Executor: Choose the simplest executor that meets your requirements
  2. Set Appropriate Timeouts: Always configure timeouts to prevent runaway jobs
  3. Validate Output: Use processors to validate and act on executor output
  4. Monitor Resource Usage: Pay attention to the resource consumption of jobs
  5. Security First: Follow the principle of least privilege when configuring executor permissions

Usage Plugins Index (`website/docs/usage/plugins/index.md`)

Plugins

Intro

Plugins in Dkron allow you to add funcionality that integrates with the workflow of the job execution in Dkron. It's a powerful system that allows you to extend and adapt Dkron to your special needs.

This page documents the basics of how the plugin system in Dkron works, and how to setup a basic development environment for plugin development if you're writing a Dkron plugin.

How it Works

Dkron execution execution processors are provided via plugins. Each plugin exposes functionality for modifying the execution. Plugins are executed as a separate process and communicate with the main Dkron binary over an RPC interface.

The code within the binaries must adhere to certain interfaces. The network communication and RPC is handled automatically by higher-level libraries. The exact interface to implement is documented in its respective documentation section.

Installing a Plugin

Dkron searches for plugins at startup, to install a plugin just drop the binary in one of the following locations:

  1. /etc/dkron/plugins
  2. Dkron executable directory

Usage Processors Index (`website/docs/usage/processors/index.md`)

Processors

Execution Processors

Processor plugins are called when an execution response has been received. They are passed the resulting execution data and configuration parameters, this plugins can perform a variety of operations with the execution and it's very flexible and per Job, examples of operations this plugins can do:

  • Execution output storage, forwarding or redirection.
  • Notification
  • Monitoring

For example, Processor plugins can be used to redirect the output of a job execution to different targets.

Currently Dkron provides you with some built-in plugins but the list keeps growing. Some of the features previously implemented in the application will be progessively moved to plugins.

Built-in processors

Dkron provides the following built-in processors:

  1. not specified - Store the output in the key value store (Slow performance, good for testing, default method)
  2. log - Output the execution log to Dkron stdout (Good performance, needs parsing)
  3. syslog - Output to the syslog (Good performance, needs parsing)
  4. files - Output to multiple files (Good performance, needs parsing)

Dkron Pro provides you with several more processors.

All plugins accepts one configuration option: forward Indicated if the plugin must forward the original execution output. This allows for chaining plugins and sending output to different targets at the same time.

EMAIL TESTING (`docs/EMAIL_TESTING.md`)

Email Testing with Mailpit

This document describes how to test email notifications in Dkron using Mailpit.

What is Mailpit?

Mailpit is a modern email testing tool for developers. It runs a fake SMTP server to capture outgoing emails instead of sending them to real recipients. This allows you to:

  • Test email functionality without sending real emails
  • Inspect email content, headers, and formatting via a web UI
  • Verify emails are being sent correctly during development and testing
  • View emails in a modern, responsive interface with search capabilities

Why Mailpit?

Mailpit offers several advantages over older tools:

  • Actively Maintained: Regular updates and bug fixes
  • Modern UI: Clean, responsive interface with better UX
  • Fast: Written in Go with excellent performance
  • Feature-Rich: Search, filtering, tagging, and more
  • Small Footprint: Lightweight Docker image (~15MB compressed)
  • API Support: RESTful API for automated testing

Running Mailpit

The easiest way to run Mailpit is using Docker:

bash
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit

This exposes:

  • Port 1025: SMTP server (for your application to send emails)
  • Port 8025: Web UI (to view captured emails)

Using Docker Compose

Mailpit is included in the development docker-compose configuration. Start it with:

bash
docker compose -f docker-compose.dev.yml up mailpit

Or start the entire development environment including Mailpit:

bash
docker compose -f docker-compose.dev.yml up

Standalone Installation

You can also install Mailpit directly on your system. See the Mailpit documentation for installation instructions.

Configuring Dkron to Use Mailpit

To configure Dkron to send emails through Mailpit, use the following SMTP settings:

yaml
MailHost: localhost
MailPort: 1025
MailFrom: [email protected]

Note: Mailpit does not require authentication, so you don't need to set MailUsername or MailPassword.

Running Email Tests

The email notification tests in dkron/notifier_test.go are configured to use Mailpit by default. The tests automatically verify that emails are correctly sent and received using Mailpit's REST API.

To run the email notification test:

bash
# Make sure Mailpit is running first
docker run -d -p 8025:8025 -p 1025:1025 axllent/mailpit

# Run the test
go test -v -run TestNotifier_sendExecutionEmail ./dkron

Or use the convenient Makefile target:

bash
make test-email

Viewing Captured Emails

  1. Open your browser and navigate to: http://localhost:8025
  2. The Mailpit web UI will show all emails sent during testing
  3. Click on any email to view its details, including:
    • Subject line
    • Recipients (To, Cc, Bcc)
    • Email body (HTML and plain text)
    • Attachments
    • Raw MIME content
    • Headers
    • Source code

Web UI Features

Mailpit's web interface includes:

  • Search: Full-text search across all emails
  • Filtering: Filter by sender, recipient, subject
  • Responsive Design: Works on desktop and mobile
  • Dark Mode: Toggle between light and dark themes
  • Real-time Updates: Emails appear instantly via WebSocket

Test Configuration

The test in dkron/notifier_test.go uses the following configuration:

go
c := &Config{
    MailHost:          "localhost",
    MailPort:          1025,
    MailFrom:          "[email protected]",
    MailSubjectPrefix: "[Test]",
}

This configuration:

  • Connects to Mailpit's SMTP server at localhost:1025
  • Sets the sender address to [email protected]
  • Adds a [Test] prefix to all email subjects
  • Uses Mailpit API at http://localhost:8025 to verify email delivery

API Verification

The test automatically verifies email delivery using Mailpit's REST API:

go
// The test performs the following verifications:
// 1. Checks that email was received
// 2. Verifies From address is correct
// 3. Verifies To address matches recipient
// 4. Confirms subject contains expected text
// 5. Validates email body contains execution output

This ensures that emails are not just sent, but actually captured correctly by Mailpit.

Troubleshooting

Connection Refused Errors

If you see connection errors when running tests:

  1. Verify Mailpit is running:

    bash
    docker ps | grep mailpit

  2. Check that port 1025 is accessible:

    bash
    nc -zv localhost 1025

  3. If using Docker, ensure the ports are properly mapped

Emails Not Appearing in Web UI

  1. Check the Mailpit web UI at http://localhost:8025
  2. Verify the test completed successfully without errors
  3. Check Mailpit logs for any issues:
    bash
    docker logs <mailpit-container-id>

Port Already in Use

If port 1025 or 8025 is already in use:

bash
# Find what's using the port
lsof -i :1025
lsof -i :8025

# Or use different ports
docker run -p 8026:8025 -p 1026:1025 axllent/mailpit
# Then update test configuration to use port 1026

GitHub Actions CI/CD Integration

Mailpit is automatically configured in the GitHub Actions test workflow. The workflow includes Mailpit as a service container, making it available during test execution.

How It Works

The .github/workflows/test.yml file includes Mailpit as a service:

yaml
services:
    mailpit:
        image: axllent/mailpit
        ports:
            - 1025:1025
            - 8025:8025

When tests run in GitHub Actions:

  1. Mailpit starts automatically as a service container
  2. The SMTP server is available at localhost:1025
  3. Tests can send emails without any additional configuration
  4. No secrets or credentials are needed

Viewing Emails in CI

Since GitHub Actions runners don't expose web UIs, you cannot view the Mailpit web interface during CI runs. However, you can:

  1. Verify emails are sent successfully by checking test results
  2. Tests automatically use Mailpit's API to verify email content
  3. Email assertions include subject, sender, recipient, and body content

Testing Locally Before CI

To ensure your tests will pass in GitHub Actions:

bash
# Start Mailpit locally with the same configuration as CI
docker run -p 1025:1025 -p 8025:8025 axllent/mailpit

# Run the full test suite
go test -v -timeout 200s ./...

Or use the validation script:

bash
./scripts/test-ci-locally.sh

Advantages Over External Services

Mailpit provides several advantages for local development and testing:

  • Free and Open Source: No account required, runs completely locally
  • No External Dependencies: Doesn't require internet connection
  • Simple Setup: Single Docker command to get started
  • Fast: No network latency from external services
  • Privacy: All emails stay on your local machine
  • CI/CD Friendly: Easy to integrate into automated testing pipelines
  • No Secrets Required: No API keys or credentials needed in CI

Production Email Configuration

Remember that Mailpit is for testing only. For production, configure Dkron with your actual SMTP server credentials:

yaml
MailHost: smtp.your-provider.com
MailPort: 587
MailUsername: your-username
MailPassword: your-password
MailFrom: [email protected]

Using Mailpit API in Tests

The email notification tests demonstrate how to use Mailpit's API for verification:

Get All Messages

go
resp, err := http.Get("http://localhost:8025/api/v1/messages")

Get Specific Message

go
resp, err := http.Get(fmt.Sprintf("http://localhost:8025/api/v1/message/%s", messageID))

Delete All Messages

go
req, err := http.NewRequest("DELETE", "http://localhost:8025/api/v1/messages", nil)
resp, err := http.DefaultClient.Do(req)

See dkron/notifier_test.go for a complete example of API usage in tests.

Additional Resources

GITHUB ACTIONS MAILPIT (`docs/GITHUB_ACTIONS_MAILPIT.md`)

Mailpit Integration with GitHub Actions

This document provides a comprehensive guide on how Mailpit is integrated with GitHub Actions for automated email testing in the Dkron project.

Overview

Mailpit is used in GitHub Actions as a service container to enable automated testing of email notifications without requiring external email services or credentials. This provides a reliable, fast, and secure way to test email functionality in CI/CD pipelines.

Why Mailpit for GitHub Actions?

Advantages

  1. No Secrets Required: Mailpit doesn't need authentication, eliminating the need to store SMTP credentials in GitHub Secrets
  2. Fast: Runs locally within the GitHub Actions runner with minimal overhead
  3. Reliable: No dependency on external services that might have downtime or rate limits
  4. Free: No costs associated with email testing in CI
  5. Consistent: Same behavior in local development and CI environments
  6. Secure: Emails never leave the CI runner, no risk of accidentally sending test emails
  7. Modern: Active maintenance and regular updates
  8. Lightweight: Small Docker image (~15MB) for faster CI runs

GitHub Actions Configuration

Service Container Setup

In .github/workflows/test.yml, Mailpit is configured as a service container:

yaml
jobs:
    test:
        name: Test
        runs-on: ubuntu-latest
        services:
            mailpit:
                image: axllent/mailpit
                ports:
                    - 1025:1025  # SMTP port
                    - 8025:8025  # Web UI port (not accessible in CI)
        steps:
            # ... test steps

How It Works

  1. Container Start: When the workflow runs, GitHub Actions automatically starts the Mailpit container
  2. Port Mapping: Ports 1025 (SMTP) and 8025 (Web UI) are mapped to the runner
  3. Network Access: Tests can access Mailpit at localhost:1025
  4. Automatic Cleanup: The container is automatically removed when the workflow completes

Test Configuration

Email Test Setup

The email notification tests in dkron/notifier_test.go are configured to use Mailpit:

go
func TestNotifier_sendExecutionEmail(t *testing.T) {
    mailHost := "localhost"
    mailPort := 1025  // Mailpit SMTP port
    
    // Check if Mailpit is available
    checkMailpitAvailable(t, mailHost, mailPort)
    
    c := &Config{
        MailHost:          mailHost,
        MailPort:          uint16(mailPort),
        MailFrom:          "[email protected]",
        MailSubjectPrefix: "[Test]",
    }
    
    // ... test implementation
}

Availability Check

The checkMailpitAvailable helper function:

  • Attempts to connect to Mailpit on the specified port
  • In CI: Mailpit is always available, tests run normally
  • Locally: If Mailpit isn't running, test is skipped (not failed)
go
func checkMailpitAvailable(t *testing.T, host string, port int) {
    t.Helper()
    address := fmt.Sprintf("%s:%d", host, port)
    conn, err := net.DialTimeout("tcp", address, 2*time.Second)
    if err != nil {
        t.Skipf("Mailpit is not available at %s. Start it with: docker run -p 8025:8025 -p 1025:1025 axllent/mailpit", address)
        return
    }
    conn.Close()
}

Workflow Execution Flow

1. Workflow Trigger

Tests run on:

  • Every push to any branch
  • Every pull request (excluding documentation-only changes)

2. Environment Setup

text
┌─────────────────────────────────────┐
│   GitHub Actions Runner (Ubuntu)    │
│                                      │
│  ┌────────────────────────────────┐ │
│  │   Mailpit Container            │ │
│  │   - SMTP: localhost:1025       │ │
│  │   - Web UI: localhost:8025     │ │
│  │   - Image: ~15MB               │ │
│  └────────────────────────────────┘ │
│                                      │
│  ┌────────────────────────────────┐ │
│  │   Go Test Process              │ │
│  │   - Connects to localhost:1025 │ │
│  │   - Sends test emails          │ │
│  └────────────────────────────────┘ │
└─────────────────────────────────────┘

3. Test Execution

  1. Mailpit container starts automatically
  2. Go sets up the test environment
  3. Tests run with 200-second timeout
  4. Email tests connect to Mailpit
  5. Emails are captured (but not viewable in CI)
  6. Tests verify email sending succeeded
  7. Coverage report generated
  8. Results uploaded to Codecov

4. Cleanup

  • Mailpit container automatically stopped and removed
  • No manual cleanup required

Running Tests Locally to Match CI

Option 1: Using the Validation Script

bash
./scripts/test-ci-locally.sh

This script:

  • ✅ Starts Mailpit with the same configuration as CI
  • ✅ Runs tests with the same timeout (200s)
  • ✅ Generates coverage report
  • ✅ Provides interactive access to Mailpit web UI
  • ✅ Automatically cleans up after completion

Option 2: Manual Setup

bash
# 1. Start Mailpit
docker run -d --rm --name mailpit -p 1025:1025 -p 8025:8025 axllent/mailpit

# 2. Run tests (matching CI configuration)
go test -v -timeout 200s -coverprofile=coverage.txt ./...

# 3. View emails (optional)
open http://localhost:8025

# 4. Cleanup
docker stop mailpit

Option 3: Using Make

bash
make test-email

Viewing Email in CI

Limitations

The Mailpit web UI (port 8025) is not accessible from outside the GitHub Actions runner. You cannot view the web interface during CI runs.

API-Based Verification

The email notification tests automatically use Mailpit's REST API to verify email delivery. This provides comprehensive validation without requiring web UI access:

Automatic Verification in Tests:

  1. ✅ Email was successfully received by Mailpit
  2. ✅ From address matches expected sender
  3. ✅ To address matches expected recipient
  4. ✅ Subject line contains expected text
  5. ✅ Email body contains execution output and details

Example from dkron/notifier_test.go:

go
// Get all messages from Mailpit API
messages := getMailpitMessages(t, "http://localhost:8025")

// Find and verify the test email
for _, msg := range messages {
    if strings.Contains(msg.Subject, "[Test]") {
        // Get full message details
        detail := getMailpitMessage(t, apiURL, msg.ID)
        
        // Verify content
        assert.Equal(t, "[email protected]", detail.From.Address)
        assert.Contains(t, detail.Text, "test-output")
    }
}

This approach ensures reliable email testing in CI without manual inspection.

Troubleshooting

Tests Fail in CI But Pass Locally

Possible causes:

  1. Different Go versions
  2. Environment-specific configurations
  3. Race conditions or timing issues

Solutions:

bash
# Run with the same Go version as CI
go version  # Check against .github/workflows/test.yml

# Run with the same timeout as CI
go test -v -timeout 200s ./...

# Run race detector
go test -race ./...

Mailpit Service Not Starting

Error: Tests skip with "Mailpit is not available"

Diagnosis:

  1. Check workflow file has services.mailpit section
  2. Verify port mappings are correct (1025:1025)
  3. Review GitHub Actions logs for container startup errors

Fix:
Ensure .github/workflows/test.yml contains:

yaml
services:
    mailpit:
        image: axllent/mailpit
        ports:
            - 1025:1025
            - 8025:8025

Port Conflicts in CI

Unlikely scenario: Port 1025 or 8025 already in use on runner

Solutions:

  • Use different ports in workflow configuration
  • Update test configuration to match

Coverage Upload Fails

Note: Coverage upload failures don't fail the build

Common causes:

  • CODECOV_TOKEN secret not set
  • Codecov service temporarily unavailable
  • Network issues

Fix:

  1. Verify secret exists in repository settings
  2. Check Codecov status page
  3. Update codecov-action to latest version

Best Practices

1. Keep Tests Fast

✅ Do:

go
func TestEmail(t *testing.T) {
    // Quick email send test
    err := sendEmail()
    assert.NoError(t, err)
}

❌ Don't:

go
func TestEmail(t *testing.T) {
    // Don't poll or wait unnecessarily
    time.Sleep(30 * time.Second)
    // ...
}

2. Use Helper Functions

go
// Reusable helper for email testing
func setupEmailTest(t *testing.T) *Config {
    t.Helper()
    checkMailpitAvailable(t, "localhost", 1025)
    return &Config{
        MailHost: "localhost",
        MailPort: 1025,
        // ...
    }
}

3. Clear Error Messages

go
if err != nil {
    t.Fatalf("Failed to send email: %v\nCheck that Mailpit is running", err)
}

4. Skip Gracefully

go
// Skip test if Mailpit unavailable (local dev)
// Test runs normally in CI (Mailpit always available)
checkMailpitAvailable(t, host, port)

Monitoring and Maintenance

Workflow Health

Monitor test runs in the GitHub Actions tab:

  • Check for consistent pass rates
  • Watch for timeout increases
  • Monitor for flaky tests

Mailpit Updates

The axllent/mailpit image is pulled automatically by GitHub Actions:

  • No manual updates needed
  • GitHub caches images for performance
  • Updates happen when new workflow runs pull newer images

Go Version Updates

When updating Go version in workflow:

  1. Update .github/workflows/test.yml:

    yaml
    - uses: actions/setup-go@v6
  with:
    go-version: '1.26.1'  # Update this

  2. Test locally with the same version:

    bash
    go version

  3. Update documentation if needed

Mailpit Features in CI

Performance Benefits

  • Fast Startup: Container starts in ~1-2 seconds
  • Low Memory: Minimal memory footprint during tests
  • Quick Processing: Emails processed instantly
  • Efficient Storage: In-memory storage for test duration

API Usage in Tests

The Dkron tests use Mailpit's API for comprehensive email verification. This is the recommended approach for CI testing:

Messages API:

go
// List all messages
resp, err := http.Get("http://localhost:8025/api/v1/messages")

// Response includes: total count, message list with ID, From, To, Subject
var result mailpitMessagesResponse
json.NewDecoder(resp.Body).Decode(&result)

Message Detail API:

go
// Get specific message with full content
resp, err := http.Get("http://localhost:8025/api/v1/message/MESSAGE_ID")

// Response includes: full Text/HTML content, headers, attachments
var detail mailpitMessageDetail
json.NewDecoder(resp.Body).Decode(&detail)

Cleanup API:

go
// Delete all messages (useful for test isolation)
req, err := http.NewRequest("DELETE", "http://localhost:8025/api/v1/messages", nil)
http.DefaultClient.Do(req)

See Mailpit API documentation for complete API reference.

Real Example: Check dkron/notifier_test.go for a complete implementation showing how to verify email delivery, content, and metadata using the API.

Additional Resources

Support

If you encounter issues with Mailpit in GitHub Actions:

  1. Check this documentation first
  2. Review the troubleshooting section
  3. Run tests locally with ./scripts/test-ci-locally.sh
  4. Check GitHub Actions logs for detailed error messages
  5. Review recent workflow changes
  6. Open an issue with reproduction steps

Summary

Mailpit integration with GitHub Actions provides:

  • ✅ Automated email testing without external dependencies
  • ✅ No secrets or credentials required in CI
  • ✅ Fast and reliable test execution with modern tooling
  • ✅ Consistent behavior between local and CI environments
  • ✅ Zero cost for email testing
  • ✅ Simple setup with minimal configuration
  • ✅ Active maintenance ensuring long-term reliability
  • ✅ Better performance with smaller Docker image and faster processing

The integration is production-ready and has been tested to work reliably in the Dkron CI/CD pipeline.

MAILPIT API EXAMPLES (`docs/MAILPIT_API_EXAMPLES.md`)

Mailpit API Usage Examples

This document provides practical examples of using Mailpit's REST API in tests to verify email delivery.

Overview

Mailpit provides a comprehensive REST API at http://localhost:8025/api/v1/ that allows programmatic access to captured emails. This is essential for automated testing in CI/CD environments where the web UI is not accessible.

Base URL

text
http://localhost:8025/api/v1/

API Response Types

Message Summary

go
type mailpitMessage struct {
    ID      string           `json:"ID"`
    From    mailpitAddress   `json:"From"`
    To      []mailpitAddress `json:"To"`
    Subject string           `json:"Subject"`
    Snippet string           `json:"Snippet"`
    Created time.Time        `json:"Created"`
}

type mailpitAddress struct {
    Name    string `json:"Name"`
    Address string `json:"Address"`
}

type mailpitMessagesResponse struct {
    Total    int              `json:"total"`
    Messages []mailpitMessage `json:"messages"`
}

Message Detail

go
type mailpitMessageDetail struct {
    ID      string           `json:"ID"`
    From    mailpitAddress   `json:"From"`
    To      []mailpitAddress `json:"To"`
    Cc      []mailpitAddress `json:"Cc"`
    Bcc     []mailpitAddress `json:"Bcc"`
    Subject string           `json:"Subject"`
    Text    string           `json:"Text"`
    HTML    string           `json:"HTML"`
    Created time.Time        `json:"Created"`
}

Common Operations

1. Get All Messages

Retrieve a list of all captured messages.

go
func getMailpitMessages(t *testing.T, apiURL string) []mailpitMessage {
    t.Helper()

    resp, err := http.Get(apiURL + "/api/v1/messages")
    require.NoError(t, err, "Failed to call Mailpit API")
    defer resp.Body.Close()

    require.Equal(t, http.StatusOK, resp.StatusCode)

    var result mailpitMessagesResponse
    err = json.NewDecoder(resp.Body).Decode(&result)
    require.NoError(t, err, "Failed to decode response")

    return result.Messages
}

Usage:

go
messages := getMailpitMessages(t, "http://localhost:8025")
assert.NotEmpty(t, messages, "Expected at least one message")

2. Get Message Details

Retrieve full details of a specific message including content.

go
func getMailpitMessage(t *testing.T, apiURL, messageID string) *mailpitMessageDetail {
    t.Helper()

    url := fmt.Sprintf("%s/api/v1/message/%s", apiURL, messageID)
    resp, err := http.Get(url)
    require.NoError(t, err, "Failed to get message detail")
    defer resp.Body.Close()

    require.Equal(t, http.StatusOK, resp.StatusCode)

    var result mailpitMessageDetail
    err = json.NewDecoder(resp.Body).Decode(&result)
    require.NoError(t, err, "Failed to decode message detail")

    return &result
}

Usage:

go
detail := getMailpitMessage(t, "http://localhost:8025", messageID)
assert.Contains(t, detail.Text, "expected content")

3. Search Messages

Search for messages by query string.

go
func searchMailpitMessages(t *testing.T, apiURL, query string) []mailpitMessage {
    t.Helper()

    url := fmt.Sprintf("%s/api/v1/search?query=%s", apiURL, url.QueryEscape(query))
    resp, err := http.Get(url)
    require.NoError(t, err, "Failed to search messages")
    defer resp.Body.Close()

    require.Equal(t, http.StatusOK, resp.StatusCode)

    var result mailpitMessagesResponse
    err = json.NewDecoder(resp.Body).Decode(&result)
    require.NoError(t, err, "Failed to decode search results")

    return result.Messages
}

Usage:

go
// Search by subject
messages := searchMailpitMessages(t, "http://localhost:8025", "subject:test")

// Search by recipient
messages := searchMailpitMessages(t, "http://localhost:8025", "to:[email protected]")

4. Find Specific Email

Find a specific email by subject or other criteria.

go
func findEmailBySubject(t *testing.T, apiURL, subjectContains string) *mailpitMessage {
    t.Helper()

    messages := getMailpitMessages(t, apiURL)
    
    for i := range messages {
        if strings.Contains(messages[i].Subject, subjectContains) {
            return &messages[i]
        }
    }
    
    return nil
}

Usage:

go
email := findEmailBySubject(t, "http://localhost:8025", "[Test]")
require.NotNil(t, email, "Email with subject containing '[Test]' not found")

5. Delete All Messages

Clean up all messages (useful for test isolation).

go
func deleteAllMailpitMessages(t *testing.T, apiURL string) {
    t.Helper()

    req, err := http.NewRequest("DELETE", apiURL+"/api/v1/messages", nil)
    require.NoError(t, err, "Failed to create delete request")

    resp, err := http.DefaultClient.Do(req)
    if err == nil {
        defer resp.Body.Close()
    }
    // Don't fail if delete fails - it's just cleanup
}

Usage:

go
// Clean up before test
deleteAllMailpitMessages(t, "http://localhost:8025")

// Run your test
sendTestEmail()

// Clean up after test (optional - can use defer)
defer deleteAllMailpitMessages(t, "http://localhost:8025")

6. Wait for Email

Wait for an email to arrive (with timeout).

go
func waitForEmail(t *testing.T, apiURL string, timeout time.Duration) *mailpitMessage {
    t.Helper()

    deadline := time.Now().Add(timeout)
    
    for time.Now().Before(deadline) {
        messages := getMailpitMessages(t, apiURL)
        if len(messages) > 0 {
            return &messages[0]
        }
        time.Sleep(100 * time.Millisecond)
    }
    
    t.Fatal("Timeout waiting for email")
    return nil
}

Usage:

go
// Send email
sendTestEmail()

// Wait up to 5 seconds for email to arrive
email := waitForEmail(t, "http://localhost:8025", 5*time.Second)
assert.NotNil(t, email)

Complete Test Example

Here's a complete example showing best practices:

text
/* Detailed source-code truncated for AI context efficiency. */

Advanced Examples

Verify Multiple Recipients

go
func TestMultipleRecipients(t *testing.T) {
    mailpitAPIURL := "http://localhost:8025"
    deleteAllMailpitMessages(t, mailpitAPIURL)
    
    // Send email to multiple recipients
    err := sendEmail([]string{"[email protected]", "[email protected]"}, "Subject", "Body")
    require.NoError(t, err)
    
    time.Sleep(100 * time.Millisecond)
    
    messages := getMailpitMessages(t, mailpitAPIURL)
    require.Len(t, messages, 1, "Expected exactly one email")
    
    detail := getMailpitMessage(t, mailpitAPIURL, messages[0].ID)
    
    // Verify all recipients
    assert.Len(t, detail.To, 2)
    recipients := []string{detail.To[0].Address, detail.To[1].Address}
    assert.Contains(t, recipients, "[email protected]")
    assert.Contains(t, recipients, "[email protected]")
}

Verify HTML Email

go
func TestHTMLEmail(t *testing.T) {
    mailpitAPIURL := "http://localhost:8025"
    deleteAllMailpitMessages(t, mailpitAPIURL)
    
    // Send HTML email
    err := sendHTMLEmail("[email protected]", "Subject", "<h1>Hello</h1><p>World</p>")
    require.NoError(t, err)
    
    time.Sleep(100 * time.Millisecond)
    
    messages := getMailpitMessages(t, mailpitAPIURL)
    require.NotEmpty(t, messages)
    
    detail := getMailpitMessage(t, mailpitAPIURL, messages[0].ID)
    
    // Verify HTML content
    assert.Contains(t, detail.HTML, "<h1>Hello</h1>")
    assert.Contains(t, detail.HTML, "<p>World</p>")
    
    // Verify text fallback exists
    assert.NotEmpty(t, detail.Text)
}

Verify Email Ordering

go
func TestEmailOrdering(t *testing.T) {
    mailpitAPIURL := "http://localhost:8025"
    deleteAllMailpitMessages(t, mailpitAPIURL)
    
    // Send multiple emails
    sendEmail("[email protected]", "First Email", "Body 1")
    time.Sleep(10 * time.Millisecond)
    sendEmail("[email protected]", "Second Email", "Body 2")
    time.Sleep(100 * time.Millisecond)
    
    messages := getMailpitMessages(t, mailpitAPIURL)
    require.Len(t, messages, 2)
    
    // Messages are typically returned newest first
    assert.Contains(t, messages[0].Subject, "Second Email")
    assert.Contains(t, messages[1].Subject, "First Email")
}

Test Email Not Sent

go
func TestEmailNotSent(t *testing.T) {
    mailpitAPIURL := "http://localhost:8025"
    deleteAllMailpitMessages(t, mailpitAPIURL)
    
    // Perform action that should NOT send email
    err := performActionWithoutEmail()
    require.NoError(t, err)
    
    time.Sleep(100 * time.Millisecond)
    
    // Verify no emails were sent
    messages := getMailpitMessages(t, mailpitAPIURL)
    assert.Empty(t, messages, "Expected no emails to be sent")
}

Helper Utilities

Check Mailpit Availability

go
func isMailpitAvailable(t *testing.T, apiURL string) bool {
    t.Helper()
    
    client := &http.Client{Timeout: 2 * time.Second}
    resp, err := client.Get(apiURL + "/api/v1/messages")
    if err != nil {
        return false
    }
    defer resp.Body.Close()
    
    return resp.StatusCode == http.StatusOK
}

Count Messages

go
func countMessages(t *testing.T, apiURL string) int {
    t.Helper()
    
    messages := getMailpitMessages(t, apiURL)
    return len(messages)
}

Assert Email Count

go
func assertEmailCount(t *testing.T, apiURL string, expected int) {
    t.Helper()
    
    messages := getMailpitMessages(t, apiURL)
    assert.Len(t, messages, expected, "Unexpected number of emails")
}

Best Practices

1. Always Clean Up Before Tests

go
// Clean up at the start of each test
deleteAllMailpitMessages(t, mailpitAPIURL)

This ensures test isolation and prevents false positives from previous test runs.

2. Add Small Delays

go
// Send email
sendEmail()

// Give Mailpit time to process
time.Sleep(100 * time.Millisecond)

// Now verify
messages := getMailpitMessages(t, mailpitAPIURL)

Mailpit is fast, but a small delay ensures the email is fully processed.

3. Use Helper Functions

Create reusable helper functions for common operations:

go
func requireEmailWithSubject(t *testing.T, apiURL, subject string) *mailpitMessageDetail {
    t.Helper()
    
    email := findEmailBySubject(t, apiURL, subject)
    require.NotNil(t, email, "Email with subject '%s' not found", subject)
    
    return getMailpitMessage(t, apiURL, email.ID)
}

4. Log Useful Information

go
t.Logf("Email verified:")
t.Logf("  ID: %s", detail.ID)
t.Logf("  From: %s", detail.From.Address)
t.Logf("  To: %s", detail.To[0].Address)
t.Logf("  Subject: %s", detail.Subject)

This helps with debugging when tests fail.

5. Check API Availability

go
if !isMailpitAvailable(t, mailpitAPIURL) {
    t.Skip("Mailpit not available. Start with: docker run -p 8025:8025 -p 1025:1025 axllent/mailpit")
}

Skip tests gracefully when Mailpit is not running locally.

Troubleshooting

API Returns 404

  • Verify Mailpit is running: docker ps | grep mailpit
  • Check the correct port (8025 for API, not 1025 for SMTP)
  • Ensure URL includes /api/v1/ path

No Messages Found

  • Add a delay after sending email: time.Sleep(100 * time.Millisecond)
  • Verify email was actually sent (check for send errors)
  • Check Mailpit logs: docker logs <container-id>
  • Ensure you're not checking before the email arrives

Message ID Invalid

  • Don't reuse message IDs between tests
  • Clean up messages at the start of tests
  • Message IDs are only valid until messages are deleted

Resources

See Also

MAILPIT QUICK START (`docs/MAILPIT_QUICK_START.md`)

Mailpit Quick Start Guide

Quick reference for getting started with Mailpit email testing in Dkron.

TL;DR

bash
# Start Mailpit
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit

# Run tests
make test-email

# View emails
open http://localhost:8025

What is Mailpit?

Mailpit is a modern email testing tool that captures outgoing emails without sending them.

Key Features:

  • 🚀 Fast and lightweight (~15MB Docker image)
  • 🎨 Modern UI with dark mode
  • 🔍 Full-text search
  • 📱 Responsive design
  • ⚡ Real-time updates via WebSocket
  • 🔓 No authentication required

Installation

bash
docker run -p 8025:8025 -p 1025:1025 axllent/mailpit

Option 2: Docker Compose

bash
docker compose -f docker-compose.dev.yml up mailpit

Option 3: Using Make

bash
make test-email

Ports

  • 1025 - SMTP server (send emails here)
  • 8025 - Web UI (view emails here)

Running Tests

Run Email Tests

bash
# Ensure Mailpit is running, then:
go test -v -run TestNotifier_sendExecutionEmail ./dkron

Note: Tests automatically verify email delivery using Mailpit's REST API, checking sender, recipient, subject, and content.

Run All Tests (CI Mode)

bash
./scripts/test-ci-locally.sh

This simulates GitHub Actions CI environment.

Web Interface

Open http://localhost:8025 in your browser to:

  • View all captured emails
  • Search emails by subject, sender, recipient
  • Toggle dark mode
  • Inspect headers and raw MIME
  • Delete or mark emails

API Access: Tests use the REST API at http://localhost:8025/api/v1/ for automated verification.

Configuration

Dkron is pre-configured to use Mailpit:

go
MailHost: "localhost"
MailPort: 1025
// No authentication required

GitHub Actions

Mailpit runs automatically in CI. No configuration needed!

The workflow includes Mailpit as a service container:

yaml
services:
    mailpit:
        image: axllent/mailpit
        ports:
            - 1025:1025
            - 8025:8025

Common Commands

bash
# Start Mailpit
docker run -d --name mailpit -p 8025:8025 -p 1025:1025 axllent/mailpit

# Stop Mailpit
docker stop mailpit

# Remove Mailpit
docker rm mailpit

# View logs
docker logs mailpit

# Restart Mailpit
docker restart mailpit

Troubleshooting

Port already in use

bash
# Find what's using the port
lsof -i :1025
lsof -i :8025

# Use different ports
docker run -p 8026:8025 -p 1026:1025 axllent/mailpit

Mailpit not starting

bash
# Check if Docker is running
docker ps

# Pull latest image
docker pull axllent/mailpit

# Check container logs
docker logs mailpit

Tests skip with "Mailpit not available"

bash
# Verify Mailpit is running
docker ps | grep mailpit

# Check connection
curl -I http://localhost:8025
nc -zv localhost 1025

API Usage

Mailpit provides a REST API for automation and verification. The Dkron email tests use this API to verify email delivery:

bash
# List all messages
curl http://localhost:8025/api/v1/messages

# Search messages
curl http://localhost:8025/api/v1/search?query=test

# Get specific message
curl http://localhost:8025/api/v1/message/{id}

# Delete all messages (cleanup)
curl -X DELETE http://localhost:8025/api/v1/messages

Test Integration: See dkron/notifier_test.go for a complete example of using the API to verify:

  • Email was received
  • Sender/recipient addresses
  • Subject line content
  • Email body text

See API docs for complete reference.

Next Steps

Resources

Support

If you encounter issues:

  1. Check this guide
  2. Review troubleshooting section
  3. Run ./scripts/test-ci-locally.sh to verify setup
  4. Check full documentation

Quick Help:

bash
# Fresh start
docker stop mailpit && docker rm mailpit
docker run -d --name mailpit -p 8025:8025 -p 1025:1025 axllent/mailpit
make test-email
open http://localhost:8025

Architecture (`website/docs/architecture.md`)

Architecture

This document provides a comprehensive overview of Dkron's architecture, internal components, and how they work together to provide a distributed, fault-tolerant job scheduling system.

System Architecture

Dkron utilizes a server-agent architecture where multiple agents can form a cluster for high availability. Here's a high-level overview of how Dkron works:

mermaid
flowchart TD
    Node1["Dkron Server Node 1<br/>(Leader)"] <--> Node2["Dkron Server Node 2<br/>(Follower)"]
    Node1 <--> Node3["Dkron Server Node 3<br/>(Follower)"]
    Node1 <--> Node4["Dkron Server Node 4<br/>(Follower)"]
    Node2 <--> Node3
    Node2 <--> Node4
    Node3 <--> Node4

Key Components

Dkron's architecture consists of several key components:

  1. Server Nodes: Nodes running in server mode (with --server flag) that participate in leader election and can schedule jobs.

  2. Leader Node: One server node is elected as leader and is responsible for:

    • Scheduling jobs
    • Assigning job executions to target nodes
    • Maintaining the cluster state

  3. Follower Nodes: Server nodes that are not the leader. They:

    • Can execute jobs when selected as targets
    • Are ready to become the leader if the current leader fails
    • Maintain a replicated log of all operations

  4. Embedded Data Store: Dkron uses dual embedded databases:

    • BuntDB for application data (job definitions, execution history)
    • BoltDB for Raft consensus log (operation replication)

  5. Serf Layer: Handles cluster membership, failure detection, and messaging between nodes using the gossip protocol.

  6. HTTP API and Web UI: Provides a RESTful API and web interface for job management.

  7. Executors: Plugins that handle the actual execution of job commands (shell, HTTP, etc.).

  8. Processors: Plugins that process the output of job executions (log, file, email, etc.).

Dkron Scheduling Flow

Here's how job scheduling works in Dkron:

mermaid
flowchart LR
    A[Job Definition] --> B[Leader Scheduling]
    B --> C[Target Selection]
    C --> D[Execution]
    D --> E[Processing Output]
    E --> F[Results Storage]
  1. Job Definition: Users define jobs with scheduling parameters (cron expression), execution options, and target node tags.
  2. Leader Scheduling: The leader node tracks job schedules and triggers executions at the appropriate times.
  3. Target Selection: The leader selects target nodes for job execution based on tags and execution options.
  4. Execution: Target nodes run the job using the specified executor.
  5. Processing Output: Job output is processed by configured processors.
  6. Results Storage: Execution results are stored in the distributed data store.

Job Execution Flow

This diagram illustrates the detailed function call flow when a job is executed in Dkron. It shows the interaction between three main components: the Agent Server (which receives requests), the Leader Server (which orchestrates scheduling), and the Worker Agent (which executes jobs).

text
/* Detailed source-code truncated for AI context efficiency. */

Component Details

Agent Server

  • Entry point for HTTP/gRPC requests
  • Routes job execution requests to the leader
  • Can forward requests if not the leader

Leader Server (Raft Leader)

  • Elected via Raft consensus protocol
  • Orchestrates all job scheduling via the Scheduler
  • Selects target nodes based on job tags
  • Coordinates execution across worker nodes
  • Applies state changes to Raft log for consistency
  • Stores execution results in BuntDB

Worker Agent

  • Executes jobs assigned by the leader
  • Streams execution progress back to leader in real-time
  • Can be any node in the cluster (including the leader)
  • Uses executor plugins for actual job execution

Storage Layer

  • BuntDB: Embedded key-value store for application data (jobs, executions)
  • BoltDB: Raft log storage for consensus and replication
  • Application data is replicated across all server nodes via Raft
  • Fast, embedded storage with no external dependencies

Executor Plugins

  • Pluggable execution backends (shell, HTTP, Kafka, NATS, etc.)
  • Implement the actual job execution logic
  • Stream output back to the worker agent during execution

Communication Protocols

  • HTTP/REST API for external clients
  • gRPC with bidirectional streaming for inter-node communication
  • Serf for cluster membership and failure detection
  • Raft for consensus and state replication

Leadership and Fault Tolerance

Dkron uses a leader-follower model for high availability:

Leader Responsibilities

  • Monitors leadership status through monitorLeadership()
  • When elected, calls establishLeadership() to initialize duties
  • Starts the Scheduler to begin scheduling jobs
  • Reconciles cluster members via Serf

Fault Tolerance

  • If the leader fails, Raft automatically elects a new leader from follower nodes
  • The new leader takes over scheduling without job loss
  • Running jobs continue to completion on their worker nodes
  • Serf detects the failure and updates cluster membership
  • State is preserved through Raft's replicated log

Consensus and Replication

Dkron uses the Raft consensus algorithm (via HashiCorp Raft library) to ensure data consistency across the cluster:

How Raft Works in Dkron

  1. Leader Election: Nodes vote to elect a leader. The leader handles all writes.
  2. Log Replication: The leader replicates all state changes to follower nodes.
  3. Commitment: Changes are committed only when a majority of nodes have replicated them.
  4. Safety: Once committed, changes are durable and will survive failures.

What Gets Replicated

  • Job definitions (create, update, delete)
  • Job execution results
  • Configuration changes
  • System state

Quorum Requirements

  • Minimum 3 server nodes recommended for production (allows 1 failure)
  • 5 server nodes allow 2 failures
  • Quorum = (N/2) + 1 where N is the number of servers

Cluster Membership with Serf

Serf provides cluster membership and failure detection through a gossip protocol:

Serf Responsibilities

  • Membership: Track which nodes are part of the cluster
  • Failure Detection: Quickly detect when nodes fail or become unreachable
  • Event Propagation: Broadcast custom events across the cluster
  • Node Metadata: Share node information (tags, roles, etc.)

Gossip Protocol Benefits

  • Scales to thousands of nodes efficiently
  • Eventually consistent membership information
  • Low network overhead
  • Fast failure detection (typically within seconds)

Data Storage

BuntDB (Application Data)

Dkron uses BuntDB as the primary application data store:

  • Key-Value Store: Simple, fast embedded database
  • ACID Transactions: Ensures data consistency
  • Indexes: Fast lookups for jobs and executions
  • Storage Limits: Keeps last 100 executions per job
  • File-Based: Data stored in a single file (dkron.db)

BoltDB (Raft Log)

Raft operations are stored in BoltDB:

  • Append-Only Log: Records all state changes
  • Snapshots: Periodic snapshots to reduce log size
  • Replication: Log entries are replicated to all servers
  • Recovery: Allows nodes to catch up after downtime

Plugin System

Dkron uses HashiCorp's go-plugin framework for a flexible plugin architecture:

Plugin Types

  1. Executors: Define how jobs are executed

    • Built-in: shell, http, grpc, kafka, nats, rabbitmq, gcppubsub
    • Custom: Users can write their own executors

  2. Processors: Define how job output is processed

    • Built-in: log, files, syslog, fluent
    • Custom: Users can write their own processors

Plugin Communication

  • Plugins run as separate processes
  • Communication via gRPC
  • Language-agnostic (can be written in any language with gRPC support)
  • Isolated from main process (plugin crashes don't affect Dkron)

Network Ports

Default ports used by Dkron:

Port Protocol Purpose
8080 HTTP REST API and Web UI
8946 TCP/UDP Serf gossip protocol
8190 TCP Raft consensus
6868 gRPC Inter-node communication

Scalability Considerations

Horizontal Scaling

  • Server Nodes: 3-5 servers recommended for most deployments
  • Agent Nodes: Can scale to hundreds or thousands of nodes
  • Job Distribution: Use tags to distribute load across agents

Vertical Scaling

  • Leader Node: May need more resources as job count increases
  • Memory: Increases with number of jobs and execution history
  • CPU: Leader uses CPU for scheduling; agents use CPU for execution

Performance Limits

  • Jobs: Can handle thousands of job definitions
  • Execution Frequency: Avoid sub-second schedules for many jobs
  • Cluster Size: Tested with hundreds of nodes (agent nodes)
  • Concurrent Executions: Limited by agent resources, not Dkron itself

Security Architecture

Network Security

  • TLS support for HTTP API
  • Encryption for Serf gossip traffic
  • Raft communication can be secured with TLS

Access Control

  • Basic authentication for HTTP API
  • ACL system available in Pro version
  • API token support

Data Security

  • Encryption at rest (optional, filesystem-level)
  • Encryption in transit (TLS)
  • Secret management through environment variables or external systems

High Availability Setup

Recommended HA configuration:

text
┌─────────────────────────────────────────┐
│         Load Balancer (Optional)        │
└─────────────────────────────────────────┘
           │              │              │
    ┌──────┴───┐   ┌─────┴────┐   ┌─────┴────┐
    │ Server 1 │   │ Server 2 │   │ Server 3 │
    │ (Leader) │   │(Follower)│   │(Follower)│
    └──────────┘   └──────────┘   └──────────┘
         │              │              │
    ┌────┴──────────────┴──────────────┴────┐
    │         Serf Gossip Network            │
    └─────────────────────────────────────────┘
         │              │              │
    ┌────┴───┐   ┌─────┴────┐   ┌─────┴────┐
    │Agent 1 │   │ Agent 2  │   │ Agent 3  │
    └────────┘   └──────────┘   └──────────┘

Best Practices

  1. Deploy servers across multiple availability zones
  2. Use odd number of servers (3 or 5)
  3. Monitor leader election events
  4. Set up proper backup procedures
  5. Use health checks and alerting
  6. Plan for disaster recovery

Design Philosophy

Dkron is designed to solve one problem well: executing commands at given intervals. Following the Unix philosophy of doing one thing and doing it well (like the battle-tested cron), but with the addition of being designed for the cloud era, removing single points of failure in environments where scheduled jobs need to be run across multiple servers.

Design Principles

  • Simplicity: Easy to deploy and operate
  • Reliability: No single points of failure
  • Scalability: Handle thousands of nodes and jobs
  • Observability: Built-in metrics and logging
  • Extensibility: Plugin system for custom functionality

Faq (`website/docs/faq.md`)

Frequently Asked Questions (FAQ)

This FAQ addresses the most common issues and questions reported by Dkron users based on GitHub issues and community discussions.

Performance and Scalability

Why is my Dkron cluster experiencing performance degradation with high-frequency jobs?

Issue: Leader node crashes or becomes unresponsive when running many jobs with frequent schedules (e.g., every second or minute).

Solutions:

  • Avoid scheduling jobs too frequently (overlapping on the second). Consider batching operations or using a queue system for very high-frequency tasks.
  • Offload logs to stdout/file using processors.
  • Monitor memory usage and ensure adequate resources are allocated to the leader node.
  • Distribute jobs across multiple agent nodes using tags to reduce load on the leader.
  • Review the number of concurrent jobs and adjust concurrency policies accordingly.
  • Consider upgrading to a more powerful server for the leader node in large-scale deployments.
  • Consider upgrading to Dkron Pro using Raft fastlog

Related: Issue #1828, Issue #1842

Does Dkron provide performance benchmark data?

Dkron does not currently provide official benchmark data. Performance varies significantly based on:

  • Job frequency and complexity
  • Cluster size and configuration
  • Hardware resources
  • Network latency
  • Type of executor used

For high-frequency scheduling scenarios, it's recommended to conduct your own benchmarking in an environment similar to your production setup.

Related: Issue #1839

Memory Issues

Why is Dkron consuming excessive memory or experiencing OOM kills?

Common causes:

  1. Memory leaks: Older versions had memory leak issues that have been addressed in recent releases.
  2. High execution count: Storing too many execution records (Dkron keeps up to 100 executions per job).
  3. Large job output: Jobs that produce large amounts of output can consume significant memory.
  4. Too many jobs: Running thousands of jobs simultaneously increases memory usage.

Solutions:

  • Update to the latest version of Dkron (4.x series has several memory improvements).
  • Use the files processor to write job output to disk instead of storing it in memory.
  • Limit job output size in executor configurations.
  • Monitor memory usage with Prometheus metrics.
  • Set appropriate memory limits in Docker/Kubernetes deployments.
  • Reduce the number of concurrent jobs or distribute them across more agents.

Related: Issue #1753, Issue #1566, Issue #1553

Clustering and High Availability

Why is my agent unable to connect to the cluster?

Common causes:

  • Network connectivity issues
  • Firewall blocking required ports (default: 8946 for Serf, 8190 for Raft)
  • Incorrect retry-join configuration
  • DNS resolution problems
  • Encryption key mismatch

Solutions:

  • Verify network connectivity between nodes using telnet or nc.
  • Check firewall rules allow traffic on ports 8946 (Serf) and 8190 (Raft).
  • Use IP addresses instead of hostnames in retry-join if DNS is unreliable.
  • Ensure all nodes use the same encryption key if encryption is enabled.
  • Check logs for specific error messages: level=error ts=<timestamp> caller=agent.go:xxx

Related: Issue #1727, Issue #1702

Why does retry-join fail after a server restart?

If using DNS names in retry-join, Dkron may cache the old IP addresses. Solutions:

  • Use static IP addresses in retry-join configuration.
  • Ensure DNS updates propagate before restarting nodes.
  • Set appropriate DNS TTL values for dynamic environments.

Related: Issue #1253, Issue #1213

How do I recover from "no leader" state after restart?

In single-node mode:

bash
# Check Raft peers
dkron raft list-peers

# If cluster state is corrupted, backup and remove the data directory
cp -r /var/lib/dkron /var/lib/dkron.backup
rm -rf /var/lib/dkron/*
# Restart dkron

In multi-node mode:

  • Ensure at least bootstrap-expect number of servers are running.
  • Check that all servers can communicate on the Raft port (8190).
  • Verify there's no network partition.
  • Review logs for Raft election errors.

Related: Issue #1529, Issue #1212

Why do servers have inconsistent job data?

Causes:

  • Network partitions causing split-brain scenarios
  • Raft replication issues
  • Corrupted data on one or more nodes

Solutions:

  • Ensure stable network connectivity between all server nodes.
  • Verify Raft replication is working: dkron raft list-peers
  • Check logs for Raft errors: grep -i raft /var/log/dkron.log
  • In extreme cases, restore from backup or rebuild the cluster.

Related: Issue #1371, Issue #1337

Kubernetes and Container Deployments

Why does one Dkron pod fail with "panic: log not found" in Kubernetes?

This typically occurs during Raft initialization in StatefulSets when persistent volumes are not properly configured.

Solutions:

  • Ensure persistent volumes are correctly configured and bound.
  • Verify bootstrap-expect matches the number of server replicas.
  • Check that all pods can resolve each other's DNS names.
  • Use StatefulSet with headless service for stable network identities.
  • Ensure proper initialization order with podManagementPolicy: Parallel or readiness probes.

Related: Issue #1557

How do I configure log rotation in Docker?

Dkron writes logs to stdout/stderr by default in Docker. To manage logs:

Docker Compose:

yaml
services:
    dkron:
        logging:
            driver: "json-file"
            options:
                max-size: "10m"
                max-file: "3"

Docker CLI:

bash
docker run --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  distribworks/dkron:latest

To mount logs to the host:

yaml
volumes:
    - ./logs:/var/log/dkron

Then configure Dkron to write to files via log-level and redirect output.

Related: Issue #1556

Job Execution

Why does my shell job fail to execute?

Common causes:

  1. Shell not available in the container/environment
  2. Incorrect command syntax
  3. Missing environment variables
  4. Permission issues

Solutions:

  • Verify shell is available: which sh or which bash
  • Use absolute paths for commands: /bin/bash -c "command"
  • Set the shell option explicitly in executor config:
    json
    {
    "executor": "shell",
    "executor_config": {
        "command": "your-command",
        "shell": "true"
    }
}
  • Check file permissions and ownership
  • Review execution logs for specific error messages

Related: Issue #1775

Why does "forbid" concurrency policy not prevent concurrent executions?

In distributed environments, there can be a race condition where multiple executions start before the concurrency check completes. This is a known limitation of distributed scheduling.

Workarounds:

  • Implement application-level locking (e.g., using Redis, etcd, or database locks).
  • Increase job duration to reduce the likelihood of overlapping schedules.
  • Use idempotent job designs that can handle concurrent executions safely.
  • Add a small delay at job start to check for other running instances.

Related: Issue #1569

Why does `cmd.Process.Kill()` not clean up child processes?

On Unix systems, killing the parent process doesn't automatically kill child processes. This can leave orphaned processes running.

Solution:
Use process groups to ensure all child processes are terminated:

bash
# In your script, create a process group
set -m  # Enable job control
trap 'kill -- -$$' EXIT INT TERM

# Your commands here

Or use the timeout command:

bash
timeout 300 your-long-running-command

Related: Issue #1385

Job Configuration

Why does `runoncreate` block the API call?

When runoncreate is set to true, the job creation API waits for the first execution to complete before returning. This is intentional behavior for validation purposes.

Workaround:
If you need immediate API response:

  1. Create the job with runoncreate: false
  2. Make a separate API call to trigger execution: POST /v1/jobs/{job_name}/run

Related: Issue #1268

Web UI

Why does the UI show "Empty output" for running jobs?

The UI displays execution output after the job completes. For long-running jobs, output is not streamed in real-time.

Solutions:

  • Wait for job completion to see output
  • Use the log or files processor to write output to a log aggregation system
  • Use the API to fetch execution details: GET /v1/executions/{execution_id}
  • Check agent logs directly on the executing node

Related: Issue #1742

Why does a running job show "failed" status with strange finished_at date?

This is a UI display bug that has been addressed in recent versions. The job is still running, but the UI incorrectly shows it as failed.

Solution:

  • Update to the latest version (4.x)
  • Check actual job status via API: GET /v1/jobs/{job_name}
  • Monitor execution status: GET /v1/executions/{execution_id}

Related: Issue #1817

Why is the job ID missing on small screens?

This was a responsive design issue in the Web UI that has been fixed in version 4.x.

Solution:
Update to the latest version of Dkron.

Related: Issue #1583

Plugins and Executors

Why doesn't the GRPC executor close connections?

Older versions had connection leak issues. Solutions:

  • Update to the latest version
  • Implement connection pooling in your gRPC service
  • Set appropriate timeout values in executor config

Related: Issue #1236

API

Why does the restore API not work?

The restore endpoint requires proper backup format and permissions.

Usage:

bash
# Create backup
curl http://localhost:8080/v1/backup > backup.json

# Restore backup
curl -X POST http://localhost:8080/v1/restore \
  -H "Content-Type: application/json" \
  -d @backup.json

Note: Restore should be performed on a clean cluster or during maintenance windows.

Related: Issue #1376

Configuration and Setup

How do I set up Dkron in a dual data center configuration?

Dkron uses Raft which requires a majority of nodes to be available. For multi-datacenter setups:

Option 1: Single Raft cluster across DCs (not recommended)

  • High latency can cause leader election issues
  • Network partitions can cause outages

Option 2: Separate clusters per DC (recommended)

  • Run independent Dkron clusters in each DC
  • Use external synchronization for job definitions
  • Configure failover at the application level

Option 3: Use Pro version features

  • Enhanced replication capabilities
  • Better multi-datacenter support

Related: Issue #1363

What are the resource requirements for Dkron?

Requirements vary based on:

  • Number of jobs
  • Job execution frequency
  • Cluster size
  • Execution history retention

Minimum recommendations:

  • Server nodes: 1-2 CPU cores, 512MB-1GB RAM
  • Agent nodes: 1 CPU core, 256MB-512MB RAM

Production recommendations:

  • Server nodes: 2-4 CPU cores, 2-4GB RAM
  • Agent nodes: 2 CPU cores, 1-2GB RAM
  • Storage: 10GB+ for execution history

Scale vertically for the leader node and horizontally for agents.

Related: Issue #1560

Troubleshooting

How do I debug "invalid memory address or nil pointer dereference" errors?

This panic typically indicates a bug in Dkron or a plugin. Steps to debug:

  1. Update to the latest version - Many nil pointer issues have been fixed
  2. Check logs for the full stack trace
  3. Identify the context - What operation was being performed?
  4. Reproduce - Can you consistently trigger the issue?
  5. Report - If it persists, file a GitHub issue with:
    • Dkron version
    • Configuration
    • Steps to reproduce
    • Full logs and stack trace

Related: Issue #1388, Issue #1265

Where can I find more help?

Best Practices

General recommendations for production deployments:

  1. High Availability: Run at least 3 server nodes (odd number for Raft quorum)
  2. Monitoring: Enable Prometheus metrics and set up alerts
  3. Backups: Regularly backup job configurations using the backup API
  4. Updates: Keep Dkron updated to get bug fixes and performance improvements
  5. Resource Limits: Set appropriate memory/CPU limits in container environments
  6. Job Design:
    • Use idempotent operations
    • Implement proper error handling
    • Keep job execution time reasonable (< 1 hour)
    • Avoid very high-frequency schedules (< 1 minute)
  7. Security:
    • Enable TLS for API communication
    • Use firewall rules to restrict access
    • Consider using ACLs (Pro version)
  8. Logging: Use processors (log, files, syslog) to persist job output
  9. Testing: Test job configurations in a staging environment first
  10. Documentation: Document your job configurations and dependencies

Basics Configuration (`website/docs/basics/configuration.md`)

Configuration

Configuration sources

Settings can be specified in three ways (in order of precedence):

  1. Command line arguments.
  2. Environment variables starting with DKRON_
  3. dkron.yml config file

:::caution
Dkron sends anonymous usage data to a server with the purpose of elaborating usage statistics, if you want to disable statistics collection, you can disable it in the dkron config file or in the command line using --disable-usage-stats parameter
:::

Config file location

Config file will be loaded from the following paths:

  • /etc/dkron
  • $HOME/.dkron
  • ./config

Config file example

yaml
# Dkron example configuration file
# server: false
# bootstrap-expect: 3
# data-dir: dkron.data
# log-level: debug
# tags:
#   dc: east
# encrypt: a-valid-key-generated-with-dkron-keygen
# retry-join:
#   - 10.0.0.1
#   - 10.0.0.2
#   - 10.0.0.3
# raft-multiplier: 1
# webhook-url: https://hooks.slack.com/services/XXXXXX/XXXXXXX/XXXXXXXXXXXXXXXXXXXX
# webhook-payload: "payload={\"text\": \"{{.Report}}\", \"channel\": \"#foo\"}"
# webhook-headers: Content-Type:application/x-www-form-urlencoded
# mail-host: email-smtp.eu-west-1.amazonaws.com
# mail-port: 25
# mail-username": mailuser
# mail-password": mailpassword
# mail-from": [email protected]
# mail-subject_prefix: [Dkron]
# mail-payload: "Job {{.JobName}} finished with status {{.Success}} on node {{.NodeName}}\n\nOutput:\n{{.Output}}"

Important Configuration Parameters

Here's a detailed explanation of the key configuration parameters:

Core Configuration

Parameter Description
server When set to true, the agent runs in server mode and participates in leader election. Default: false
bootstrap-expect Number of expected servers in the cluster. When this many servers are available, Dkron will bootstrap the cluster and start a leader election. This helps automate cluster formation.
data-dir Directory for storing server-specific data including the replicated log. Default: dkron.data
log-level Log verbosity level: debug, info, warn, error, fatal, panic. Default: info

Networking and Cluster Formation

Parameter Description
bind-addr Address to bind for network services. Uses go-sockaddr template format. Default: {{ GetPrivateIP }}:8946
http-addr Address to bind the UI web server to. Format: :8080
advertise-addr Address advertised to other nodes. Uses the bind address by default.
advertise-rpc-port RPC port advertised to clients. Uses rpc-port by default.
rpc-port Port used for RPC communication with clients. Default: 6868
join Initial agent to join with. Can be specified multiple times.
retry-join Address of agent to join with retries. Can be specified multiple times.
retry-interval Time to wait between join attempts. Default: 30s
retry-max Maximum number of join attempts. 0 means retry indefinitely.
encrypt Key for encrypting network traffic. Must be a base64-encoded 16-byte key generated by dkron keygen

Node Configuration

Parameter Description
node-name Name of this node. Must be unique in the cluster. Default: system hostname
datacenter Data center of the local agent. All members of a datacenter should share a LAN connection. Default: dc1
region Region the agent belongs to. A region typically maps to a geographic region. Default: global
tag Attach key/value tag pairs to the node. Format: key=value. Can be specified multiple times.

Performance and Scaling

Parameter Description
raft-multiplier Integer multiplier for Raft timing parameters. Lower values increase sensitivity to failures but require more resources. 1 is high-performance, 10 is max. Default: 1
profile Controls timing profiles. Options: lan, wan, etc. Default: lan
serf-reconnect-timeout Time to attempt reconnection to failed nodes before removing them. Default: 24h

Notification Configuration

Parameter Description
webhook-endpoint Endpoint URL to call for webhook notifications.
webhook-payload Body of the POST request to send when calling the webhook.
webhook-headers Headers to use when calling the webhook URL. Can be specified multiple times.
mail-host Mail server hostname for sending notifications.
mail-port Mail server port.
mail-username Username for mail server authentication.
mail-password Password for mail server authentication.
mail-from From email address to use in notifications.
mail-subject-prefix Prefix for the email subject line. Default: [Dkron]
mail-payload Custom Go template for the email body. If not set, defaults to the execution output. Available template variables: {{.Report}}, {{.JobName}}, {{.ReportingNode}}, {{.StartTime}}, {{.FinishedAt}}, {{.Success}}, {{.NodeName}}, {{.Output}}

Metrics Configuration

Parameter Description
statsd-addr Address of a statsd server for metrics collection.
dog-statsd-addr Address of a DataDog statsd agent.
dog-statsd-tags Tags to include with DataDog metrics. Format: key:value
enable-prometheus When true, enables serving Prometheus metrics at the /metrics endpoint.

Environment Variables

All configuration options can also be specified as environment variables by:

  1. Converting to uppercase
  2. Replacing dashes with underscores
  3. Prefixing with DKRON_

Example: server: true becomes DKRON_SERVER=true

SEE ALSO

Basics Installation (`website/docs/basics/installation.md`)

Installation

This guide covers different methods of installing Dkron for development, testing, and production environments.

Quick Start Methods

Running the binary

Download the packaged archive for your platform from the downloads page and extract the package to a shared location in your drive, like /opt/local/bin.

Run Dkron with default setting: dkron agent --server --bootstrap-expect=1

Navigate to http://localhost:8080/ui

Installing from package repositories

Debian/Ubuntu

APT repository:

text
deb [trusted=yes] https://repo.dkron.io/apt/ /

Setup and install:

bash
# Add the repository
echo "deb [trusted=yes] https://repo.dkron.io/apt/ /" | sudo tee /etc/apt/sources.list.d/dkron.list

# Update package list
sudo apt-get update

# Install Dkron
sudo apt-get install dkron

RHEL/CentOS/Fedora

YUM repository:

text
[dkron]
name=Dkron Repository
baseurl=https://repo.dkron.io/yum/
enabled=1
gpgcheck=0

Setup and install:

bash
# Create repo file
sudo tee /etc/yum.repos.d/dkron.repo << EOF
[dkron]
name=Dkron Repository
baseurl=https://repo.dkron.io/yum/
enabled=1
gpgcheck=0
EOF

# Install Dkron
sudo yum install dkron

After package installation, Dkron will be started as a system service and an example configuration file will be placed under /etc/dkron/dkron.yml.

Running in Docker

Dkron provides official Docker images via Docker Hub that can be used for deployment on any system running Docker.

:::info
If you only plan to use the built-in executors, http and shell you can use the Dkron Light edition that only includes a single binary as the plugins are built-in.
:::

Launching Dkron as a new container

Here's a quick one-liner to get you off the ground (please note, we recommend further configuration for production deployments below):

text
docker run -d -p 8080:8080 --name dkron dkron/dkron agent --server --bootstrap-expect=1 --node-name=node1

This will launch a Dkron server on port 8080 by default. You can use docker logs -f dkron to follow the rest of the initialization progress. Once the Dkron startup completes you can access the app at localhost:8080

Since Docker containers have their own ports and we just map them to the system ports as needed it's easy to move Dkron onto a different system port if you wish. For example running Dkron on port 12345:

text
docker run -d -p 12345:8080 --name dkron dkron/dkron agent --server --bootstrap-expect=1 --node-name=node1

Docker Image Variants

Dkron offers two Docker image variants:

  1. Standard Image (dkron/dkron): Includes all executors and processors
  2. Light Image (dkron/dkron:light): Contains only the main binary with built-in shell and HTTP executors

Mounting a mapped file storage volume

Dkron uses the local filesystem for storing the embedded database to store its own application data and the Raft protocol log. The end result is that your Dkron data will be on disk inside your container and lost if you ever remove the container.

To persist your data outside of the container and make it available for use between container launches we can mount a local path inside our container.

text
docker run -d -p 8080:8080 -v ~/dkron.data:/dkron.data --name dkron dkron/dkron agent --server --bootstrap-expect=1 --data-dir=/dkron.data

Now when you launch your container we are mounting that folder from our local filesystem into the container.

Running a Dkron cluster in Docker Compose

For development or simple multi-node setups, Docker Compose offers an easy way to run a Dkron cluster:

yaml
version: "3"
services:
  dkron-leader:
    image: dkron/dkron
    ports:
      - "8080:8080"
    command: agent --server --bootstrap-expect=3 --node-name=leader --retry-join=dkron-follower-1 --retry-join=dkron-follower-2 --data-dir=/dkron.data
    volumes:
      - leader-data:/dkron.data

  dkron-follower-1:
    image: dkron/dkron
    command: agent --server --node-name=follower1 --retry-join=dkron-leader --retry-join=dkron-follower-2 --data-dir=/dkron.data
    volumes:
      - follower1-data:/dkron.data
    depends_on:
      - dkron-leader

  dkron-follower-2:
    image: dkron/dkron
    command: agent --server --node-name=follower2 --retry-join=dkron-leader --retry-join=dkron-follower-1 --data-dir=/dkron.data
    volumes:
      - follower2-data:/dkron.data
    depends_on:
      - dkron-leader

volumes:
  leader-data:
  follower1-data:
  follower2-data:

Production Deployment Considerations

When deploying Dkron in production, consider these best practices:

System Requirements

  • CPU: Minimum 2 cores recommended, more for heavy workloads
  • Memory: Minimum 1GB RAM, 2GB+ recommended for production
  • Disk: SSD recommended for better performance, especially for the data directory
  • Network: Low-latency network between cluster nodes

Cluster Size

  • Minimum Nodes: 3 server nodes for high availability
  • Recommended: Use an odd number of server nodes (3, 5, 7) to avoid split-brain scenarios
  • Region Distribution: For multi-region deployments, use at least 3 servers per region

Security Recommendations

  1. Network Security:

    • Restrict access to Dkron API/UI using firewalls
    • Run behind a reverse proxy with TLS termination
    • Use VPN or private network for inter-node communication

  2. Authentication:

    • Use Dkron Pro for built-in authentication features
    • Consider implementing a reverse proxy with authentication

  3. Encryption:

    • Enable network encryption with --encrypt parameter using a key generated with dkron keygen

Data Directory Considerations

  • Place the data directory on persistent, high-performance storage
  • Implement regular backups of this directory
  • Monitor disk space usage

Configuration Example for Production

Example production configuration file (/etc/dkron/dkron.yml):

yaml
# Server settings
server: true
bootstrap-expect: 3
data-dir: /var/lib/dkron
log-level: info

# Network settings
bind-addr: "{{ GetPrivateIP }}:8946"
http-addr: ":8080"
advertise-addr: "{{ GetPublicIP }}"
encrypt: "YOUR_ENCRYPTION_KEY_GENERATED_WITH_DKRON_KEYGEN"

# Node identification
node-name: "dkron-prod-1"
datacenter: "dc1"
region: "us-west"

# Clustering
retry-join:
  - "10.0.1.10"
  - "10.0.1.11"
  - "10.0.1.12"
retry-interval: "30s"
raft-multiplier: 1

# Optional metrics for monitoring
statsd-addr: "localhost:8125"
enable-prometheus: true

# Tags for node selection
tags:
  role: "backend"
  env: "production"

Monitoring and Operations

  1. Health Monitoring:

    • Set up health checks against the API endpoint
    • Monitor node status and leader elections
    • Configure metrics collection with Prometheus or StatsD

  2. Log Management:

    • Configure log collection and aggregation
    • Set appropriate log levels (info for production, debug for troubleshooting)

  3. Backup Strategy:

    • Regular backups of the data directory
    • Document and test restoration procedures

Next Steps

After installation, follow the Getting Started guide to learn how to create and manage jobs with Dkron.

Cli Dkron (`website/docs/cli/dkron.md`)

date: 2022-06-05 title: "dkron" slug: dkron url: /cli/dkron/

dkron

Open source distributed job scheduling system

Synopsis

Dkron is a system service that runs scheduled jobs at given intervals or times,
just like the cron unix service but distributed in several machines in a cluster.
If a machine fails (the leader), a follower will take over and keep running the scheduled jobs without human intervention.

Options

text
--config string   config file path
  -h, --help            help for dkron

SEE ALSO

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Agent (`website/docs/cli/dkron_agent.md`)

date: 2022-06-05 title: "dkron agent" slug: dkron_agent url: /cli/dkron_agent/

dkron agent

Start a dkron agent

Synopsis

Start a dkron agent that schedules jobs, listens for executions and runs executors.
It also runs a web UI.

text
dkron agent [flags]

Options

text
/* Detailed source-code truncated for AI context efficiency. */

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

  • dkron - Open source distributed job scheduling system

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Completion (`website/docs/cli/dkron_completion.md`)

date: 2022-06-05 title: "dkron completion" slug: dkron_completion url: /cli/dkron_completion/

dkron completion

Generate the autocompletion script for the specified shell

Synopsis

Generate the autocompletion script for dkron for the specified shell.
See each sub-command's help for details on how to use the generated script.

Options

text
-h, --help   help for completion

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Completion Bash (`website/docs/cli/dkron_completion_bash.md`)

date: 2022-06-05 title: "dkron completion bash" slug: dkron_completion_bash url: /cli/dkron_completion_bash/

dkron completion bash

Generate the autocompletion script for bash

Synopsis

Generate the autocompletion script for the bash shell.

This script depends on the 'bash-completion' package.
If it is not installed already, you can install it via your OS's package manager.

To load completions in your current shell session:

text
source <(dkron completion bash)

To load completions for every new session, execute once:

Linux:

text
dkron completion bash > /etc/bash_completion.d/dkron

macOS:

text
dkron completion bash > /usr/local/etc/bash_completion.d/dkron

You will need to start a new shell for this setup to take effect.

text
dkron completion bash

Options

text
-h, --help              help for bash
      --no-descriptions   disable completion descriptions

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Completion Fish (`website/docs/cli/dkron_completion_fish.md`)

date: 2022-06-05 title: "dkron completion fish" slug: dkron_completion_fish url: /cli/dkron_completion_fish/

dkron completion fish

Generate the autocompletion script for fish

Synopsis

Generate the autocompletion script for the fish shell.

To load completions in your current shell session:

text
dkron completion fish | source

To load completions for every new session, execute once:

text
dkron completion fish > ~/.config/fish/completions/dkron.fish

You will need to start a new shell for this setup to take effect.

text
dkron completion fish [flags]

Options

text
-h, --help              help for fish
      --no-descriptions   disable completion descriptions

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Completion Powershell (`website/docs/cli/dkron_completion_powershell.md`)

date: 2022-06-05 title: "dkron completion powershell" slug: dkron_completion_powershell url: /cli/dkron_completion_powershell/

dkron completion powershell

Generate the autocompletion script for powershell

Synopsis

Generate the autocompletion script for powershell.

To load completions in your current shell session:

text
dkron completion powershell | Out-String | Invoke-Expression

To load completions for every new session, add the output of the above command
to your powershell profile.

text
dkron completion powershell [flags]

Options

text
-h, --help              help for powershell
      --no-descriptions   disable completion descriptions

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Completion Zsh (`website/docs/cli/dkron_completion_zsh.md`)

date: 2022-06-05 title: "dkron completion zsh" slug: dkron_completion_zsh url: /cli/dkron_completion_zsh/

dkron completion zsh

Generate the autocompletion script for zsh

Synopsis

Generate the autocompletion script for the zsh shell.

If shell completion is not already enabled in your environment you will need
to enable it. You can execute the following once:

text
echo "autoload -U compinit; compinit" >> ~/.zshrc

To load completions for every new session, execute once:

Linux:

text
dkron completion zsh > "~/.zsh/completion/_dkron"

macOS:

text
dkron completion zsh > /usr/local/share/zsh/site-functions/_dkron

You will need to start a new shell for this setup to take effect.

text
dkron completion zsh [flags]

Options

text
-h, --help              help for zsh
      --no-descriptions   disable completion descriptions

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Doc (`website/docs/cli/dkron_doc.md`)

date: 2022-06-05 title: "dkron doc" slug: dkron_doc url: /cli/dkron_doc/

dkron doc

Generate Markdown documentation for the Dkron CLI.

Synopsis

Generate Markdown documentation for the Dkron CLI.
This command is, mostly, used to create up-to-date documentation
of Dkron's command-line interface for http://dkron.io/.
It creates one Markdown file per command with front matter suitable
for rendering in Hugo.

text
dkron doc [flags]

Options

text
--dir string   the directory to write the doc. (default "/tmp/dkrondoc/")
  -h, --help         help for doc

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

  • dkron - Open source distributed job scheduling system

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Keygen (`website/docs/cli/dkron_keygen.md`)

date: 2022-06-05 title: "dkron keygen" slug: dkron_keygen url: /cli/dkron_keygen/

dkron keygen

Generates a new encryption key

Synopsis

Generates a new encryption key that can be used to configure the
agent to encrypt traffic. The output of this command is already
in the proper format that the agent expects.

text
dkron keygen [flags]

Options

text
-h, --help   help for keygen

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

  • dkron - Open source distributed job scheduling system

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Leave (`website/docs/cli/dkron_leave.md`)

date: 2022-06-05 title: "dkron leave" slug: dkron_leave url: /cli/dkron_leave/

dkron leave

Force an agent to leave the cluster

Synopsis

Stop stops an agent, if the agent is a server and is running for election
stop running for election, if this server was the leader
this will force the cluster to elect a new leader and start a new scheduler.

text
dkron leave [flags]

Options

text
-h, --help              help for leave
      --rpc-addr string   gRPC address of the agent (default "{{ GetPrivateIP }}:6868")

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

  • dkron - Open source distributed job scheduling system

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Raft (`website/docs/cli/dkron_raft.md`)

date: 2022-06-05 title: "dkron raft" slug: dkron_raft url: /cli/dkron_raft/

dkron raft

Command to perform some raft operations

Options

text
-h, --help              help for raft
      --rpc-addr string   gRPC address of the agent. (default "{{ GetPrivateIP }}:6868")

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Raft List Peers (`website/docs/cli/dkron_raft_list-peers.md`)

date: 2022-06-05 title: "dkron raft list-peers" slug: dkron_raft_list-peers url: /cli/dkron_raft_list-peers/

dkron raft list-peers

Command to list raft peers

text
dkron raft list-peers [flags]

Options

text
-h, --help   help for list-peers

Options inherited from parent commands

text
--config string     config file path
      --rpc-addr string   gRPC address of the agent. (default "{{ GetPrivateIP }}:6868")

SEE ALSO

  • dkron raft - Command to perform some raft operations

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Raft Remove Peer (`website/docs/cli/dkron_raft_remove-peer.md`)

date: 2022-06-05 title: "dkron raft remove-peer" slug: dkron_raft_remove-peer url: /cli/dkron_raft_remove-peer/

dkron raft remove-peer

Command to list raft peers

text
dkron raft remove-peer [flags]

Options

text
-h, --help             help for remove-peer
      --peer-id string   Remove a Dkron server with the given ID from the Raft configuration.

Options inherited from parent commands

text
--config string     config file path
      --rpc-addr string   gRPC address of the agent. (default "{{ GetPrivateIP }}:6868")

SEE ALSO

  • dkron raft - Command to perform some raft operations

Auto generated by spf13/cobra on 5-Jun-2022

Cli Dkron Version (`website/docs/cli/dkron_version.md`)

date: 2022-06-05 title: "dkron version" slug: dkron_version url: /cli/dkron_version/

dkron version

Show version

Synopsis

Show the version

text
dkron version [flags]

Options

text
-h, --help   help for version

Options inherited from parent commands

text
--config string   config file path

SEE ALSO

  • dkron - Open source distributed job scheduling system

Auto generated by spf13/cobra on 5-Jun-2022

Pro Acls (`website/docs/pro/acls.md`)

Access Control Lists

Dkron provides an optional Access Control List (ACL) system which can be used to control access to data and APIs. The ACL is capability-based, relying on policies to determine which fine-grained rules can be applied. Dkron's capability-based ACL system is very similar to common ACL systems you are used to.

ACL System Overview

Dkron's ACL system is implemented with the CNCF Open Policy Agent bringing a powerful system to suit your needs.

The ACL system is designed to be easy to use and fast to enforce while providing administrative insight. At the highest level, there are two major components to the ACL system:

  • OPA policy engine. OPA provides policy decision-making decoupling. Dkron integrates OPA as a library and provides default policy rules written in the OPA Policy language. These rules enforce permissions on request parameters to the /v1 API paths and are ready to use for most cases. You do not need to learn the OPA Policy language to start using Dkron's ACL system, but you can modify the default policy rules to adapt to your use case if needed. Read more in the OPA docs.

  • ACL Policies. Dkron's ACL policies are simple JSON documents that define patterns to allow access to resources. You can find below an example ACL policy that works with the default OPA policy. The ACL JSON structure is not rigid; you can adapt it to add new features in combination with the OPA Policy rules.

Tutorial

Configuring ACLs

ACLs are not enabled by default and must be enabled. To enable ACLs, set up the acl section in your config file:

yaml
acl:
  enabled: true

Below you can find the most basic example of an ACL policy:

Basic example policy:

json
{
    "path": {
        "/v1": {
            "capabilities": [
                "read"
            ]
        },
        "/v1/**": {
            "capabilities": [
                "create",
                "read",
                "update",
                "delete",
                "list"
            ]
        }
    }
}

This policy allows requests to the /v1 API paths. Paths use glob patterns, and capabilities allow operations on resources.

This is a much more granular policy file used as default policy:

json
{
    "path": {
        "/v1/members": {
            "capabilities": ["read"]
        },
        "/v1/jobs": {
            "capabilities": [
                "list",
                "read",
                "create",
                "update",
                "delete"
            ]
        },
        "/v1/jobs/*": {
            "capabilities": [
                "create",
                "read",
                "update",
                "delete"
            ]
        },            
        "/v1/jobs/*/run": {
            "capabilities": ["create"]
        },
        "/v1/jobs/*/toggle": {
            "capabilities": ["create"]
        },
        "/v1/jobs/*/executions*": {
            "capabilities": ["read"]
        },
        "/v1/jobs/*/executions/*": {
            "capabilities": ["read"]
        },
        "/v1/leader": {
            "capabilities": ["read"]
        },
        "/v1/isleader": {
            "capabilities": ["read"]
        },
        "/v1/leave": {
            "capabilities": ["create"]
        },
        "/v1/restore": {
            "capabilities": ["create"]
        },
        "/v1/busy": {
            "capabilities": ["read"]
        }
    }
}

Setup

The first step after enabling the ACL system and restarting the service is to bootstrap the management token.

:::note
The dkron acl bootstrap command currently expects the policies/default.json file to be present in the working directory. Ensure you run this command from the Dkron project root or a directory where the policies/ folder is available.
:::

Run this command to create the initial management token:

bash
dkron acl bootstrap

:::info
Adjust your --rpc-addr param if necessary.
:::

After this step you will obtain the details of the management token, something similar to:

text
Accessor ID: fc4a83e5-4657-4c18-92b0-723d7c5f6c1f
Secret: ca40c646-4a86-425d-ae55-b27fdd99d8c4
Name: bootstrap
Type: management
CreateTime: 2024-10-06 11:03:36.605368 +0000 UTC
Policies: default

From this point, you must use the "Secret" to communicate with the Dkron API. Management tokens can administer ACL policies, while client tokens are constrained by their assigned policies.

When making API requests, include the secret in the Authorization header:

bash
curl -H "Authorization: Bearer <Secret>" http://localhost:8080/v1/jobs

If you navigate to your Dkron installation, it should show the login page:

Enter the secret and click on "Sign in", after that you should be able to use the UI without limitations.

Check the full documentation on all the available ACL management commands.

Use the readonly policy

json
{
    "path": {
        "/v1/members": {
            "capabilities": ["read"]
        },
        "/v1/jobs": {
            "capabilities": [
                "list",
                "read"
            ]
        },
        "/v1/jobs/*": {
            "capabilities": [
                "read"
            ]
        },            
        "/v1/jobs/*/executions*": {
            "capabilities": ["read"]
        },
        "/v1/jobs/*/executions/*": {
            "capabilities": ["read"]
        },
        "/v1/leader": {
            "capabilities": ["read"]
        },
        "/v1/isleader": {
            "capabilities": ["read"]
        },
        "/v1/busy": {
            "capabilities": ["read"]
        }
    }
}

Save the JSON above to a local file named readonly.json. The policy document must have path at its top level; do not wrap it in a rules object.

Create or update the readonly policy from that file. The --rules-file flag belongs to acl policy apply:

bash
dkron acl policy apply --name readonly --rules-file ./readonly.json

Then create a client token and associate the existing readonly policy with it. acl token create accepts --policy, but it does not accept --rules-file:

Clients using this token will be able to perform GET requests, but any write attempts will return a 403 Forbidden error:

bash
dkron acl token create --name alice --type client --policy readonly

Provide the token details to the user securely.

Inspecting job contents in OPA policies

Policy input

Dkron supplies the following fields to the OPA policy:

Field Contents
input.method HTTP request method, such as POST.
input.path Request URL path, without query parameters.
input.token Authenticated bearer secret. Do not log this value.
input.rules The JSON rules from the token's first associated ACL policy.
input.body The decoded job on the job-write routes below; an empty object on other routes.

Job contents are available on these routes:

  • POST /v1/jobs
  • PATCH /v1/jobs
  • PUT /v1/jobs/:job

There is no PUT /v1/jobs route. The default OPA policy does not authorize PATCH. If you extend its method-to-capability mapping to permit PATCH, apply the same content restrictions as for POST and PUT.

input.body is the decoded job shared with the API handler, rather than the raw JSON document. It uses canonical job field names and includes defaults and zero values. The owner is the authenticated token's accessor, the ID is the job name, and concurrency defaults to allow. Unknown job fields are omitted. Sharing the decoded job ensures that authorization and job handling interpret repeated or differently capitalized JSON fields consistently. Normal job validation still runs after authorization.

With ACL enabled, job submissions must contain a single JSON object and are limited to 10 MiB. Invalid bodies return 400 Bad Request, oversized bodies return 413 Request Entity Too Large, and policy denials return 403 Forbidden.

Other routes retain input.body == {}. Their request streams are not parsed or consumed by the ACL middleware, including multipart restore uploads. Restrict access to /v1/restore separately if a token must not import arbitrary jobs.

Adding content restrictions

For example, a policy can inspect both the executor and its command:

rego
input.body.executor == "shell"
input.body.executor_config.command == "cronexec alice task"

These are conditions to use in a rule, not a complete policy. Apply them to job-write requests while retaining your existing method, path, capability, and token checks. Adding them unconditionally to a general rule would also deny reads and other routes whose body is empty.

:::warning Keep content checks in the authorization path
A separate allow rule provides an alternative way to authorize a request. Do not add a content-checking allow rule alongside an existing rule that already permits unrestricted job writes. Every rule that can authorize a job write must enforce the required restrictions.
:::

Keep user allow-lists in administrator-controlled ACL policy data, rather than in fields supplied with the job. For example, a custom ACL JSON policy can include an allowed_users array alongside path; your Rego policy can check a parsed wrapper user against input.rules.allowed_users. Associate different ACL policies with tokens that need different allow-lists.

When restricting commands to a wrapper such as cronexec, validate its argument grammar and execution settings as well as the command string. A prefix check alone is insufficient: enabling shell interpretation can allow additional commands after the wrapper invocation. Account for quoting, executor_config.shell, and environment settings that could change execution behavior.

Debugging OPA policies

Policy print() statements are disabled by default. Enable them in the agent configuration:

yaml
acl:
  enabled: true
  debug: true

Restart Dkron after changing the configuration. Policy print messages are emitted through Dkron's logger at info level with component=acl-policy, so the configured log level must permit info messages. Raising the log level alone does not enable print statements.

Print selected fields while evaluating a job-write rule, for example:

rego
print("job executor", input.body.executor)

Dkron does not automatically dump policy input. Avoid print(input): it includes the bearer secret, and job configuration may contain credentials. Disable acl.debug after troubleshooting.

You can also test Rego policies outside Dkron using the OPA CLI with synthetic input. Such tests validate policy logic but do not change which fields an older Dkron binary supplies.

Disable ACLs

To disable the ACL system, set the configuration value in your Dkron configuration file to false:

yaml
acl:
  enabled: false

Restart your system for the change to take effect. After the restart, you should be able to access the system without any restriction.

Pro Commercial Faq (`website/docs/pro/commercial-faq.md`)

title: Commercial FAQ

What is Dkron Pro?

Dkron Pro is a flavor of Dkron which add more functionality and provide additional support options for customers.

Is there a trial version?

There's no free trial but we do offer a 14 day period with full refund if it does not work for you.

Can I get a discount?

I'm sure you're very nice but no. Everyone pays the same price.

What is the license?

See COMM-LICENSE.

How does Pro licensing work?

Every organization running Dkron Pro on its own servers must have a license. There's no limit to the number of servers or environments used by that organization.

What happens if my subscription lapses?

You must have an active subscription to run Dkron Pro. After a one week grace period, you'll lose access to the package repository and priority support. You won't get any more updates or bug fixes and apt-get install dkron-pro won't work anymore.

Can I distribute Dkron Pro to my customers?

This is a common requirement for "on-site installs" or "appliances" sold to large corporations.

The standard license is only appropriate for SaaS usage as it does not allow distribution. Dkron Pro have an Appliance license option which does allow you to distribute them. The Appliance license is $7,500/yr for Pro. It allows you to distribute the Pro binaries as part of your application and each of your customers to run Dkron Pro as part of your application only. Email support@dkron.io to purchase.

Can you transfer a license?

Licenses are not transferrable to another company. We will transfer the license from a user-specific email to a group email address (e.g. [email protected] -> [email protected]) but only for the same domain. It is strongly recommended that you buy the license using a group email address so the license is not attached to any one employee's email address.

What does the license require me to do?

Your purchase gets you unique access credentials for downloading the Pro packages. The license agreement requires you to keep these access credentials private. If we find your access credentials are ever publicized:

  1. We'll send you a warning email with details. You need to remove the content and send a new email address so we can generate new credentials for you. The old credentials will stop working immediately so you'll need to update your apps.
  2. If your credentials are publicized a second time, we reserve the right to permanently remove access (but won't unless it's really egregious - sloppy contractors happen).

Can I get a refund?

Yes, up to two weeks after purchase. Let us know the reason and maybe we can help but either way it's not a problem. Email support@dkron.io.

How do I update my credit card info?

If you purchased Dkron Pro (settings) with a credit card, log into Gumroad with your email address, click the Settings, enter your card info and hit Save. Follow instructions in Gumroad docs https://help.gumroad.com/how-do-i-update-my-credit-card-information I can't provide support for the Gumroad website and don't have the ability to edit customer info - if you can't log in or change your credit card, you can always let your current subscription expire and purchase a new subscription.

Can I request a change to the license terms?

Dkron Pro customers purchasing the Appliance license can ask for changes to the terms and conditions.

If you don't need the Appliance license but wants to ask for changes to the terms and conditions you will need to purchase a Dkron Pro standard license per production server.

Email your concerns and we can negotiate something.

Can I pay via invoice and purchase order?

Dkron Pro is credit card only, no exceptions.

Contact Info

Distributed Works

All billing/support inquiries: [email protected]

Phone: not available

Pro Commercial Support (`website/docs/pro/commercial-support.md`)

title: Commercial Support

Dkron offers only community support. Dkro Pro offers priority support via email.

Priority Support

Covers 1 incident per quarter, with a max response time of 2 working days. Scope is limited to Dkron and Dkron Pro features and APIs, not application integration or infrastructure. For support, email support AT dkron.io. Please email using the same domain as the original license email or explain your connection to the licensed company.

Pro Configuration (`website/docs/pro/configuration.md`)

Configuration

Dkron Pro uses the same parameters as Dkron OSS and add some extra parameters.

Command line options

  • --username - Authentication username
  • --password - Authentication password
  • --cert-file - Path to the client server TLS cert file
  • --key-file - Path to the client server TLS key file
  • --client-crl-file - Path to the client certificate revocation list file
  • --trusted-ca-file - Path to the client server TLS trusted CA cert file
  • --client-cert-auth - Enable client cert authentication
  • --auto-tls - Client TLS using generated certificates
  • --fast - Enable Raft Fastlog storage engine

Pro Encryption (`website/docs/pro/encryption.md`)

Encryption

SSL encryption is used for communicating dkron pro nodes. Also if client auth is enabled. This means that no other software running on your local network will be able to talk to dkron servers.

This ensures that no unexpected usage of the Dkron's store will happen, unless it is another Dkron pro instance.

SSL encryption is enabled by default in Dkron Pro and can not be disabled, you don't need to do nothing to use it.

By default Dkron Pro runs with automatically generated SSL certificates, this is enough for using it in a trusted environment but to have a better grade of confidence, it is recommended to run Dkron Pro with custom SSL certificates.

Follow this tutorial to generate autosigned SSL certificates for your instances.

:::info
You don't need a client certificate for Dkron server, just add "client auth" usage to your server cert.
:::

yaml
# dkron.yaml
auto-tls: false # Set to false to use custom certs
key-file: server-key.pem
cert-file: server.pem
trusted-ca-file: ca.pem
client-cert-auth: true # Enable it to only allow certs signed by the same CA

Pro Failover (`website/docs/pro/failover.md`)

Cross region failover

:::warning
This feature is experimental and should be handled with care.
:::

Dkron Pro can run federated in failover mode, this allows to have two clusters running in different regions and configure one of the clusters in an active-passive fashion, doing a failover in case the active cluster dies.

Pro Raft Fastlog (`website/docs/pro/raft-fastlog.md`)

Raft FastLog

Dkron Pro supports the Raft FastLog storage engine for Raft state. FastLog keeps the Raft log in memory for fast access while persisting it to disk, which can reduce Raft write latency and improve throughput compared to the default BoltDB-backed Raft log.

Overview

FastLog is optimized for Raft's append-heavy write pattern. It is a good fit when the default Raft storage becomes a bottleneck for cluster coordination, scheduling activity, or leader-side state changes.

Typical reasons to evaluate FastLog include:

  • Higher Raft throughput for write-heavy workloads
  • Lower commit latency for Raft log operations
  • Faster in-memory access to recent Raft log entries
  • Durability tuning through the --raft-duration setting

FastLog is most useful when you are running busy production clusters, scheduling a high volume of jobs, or trying to reduce the overhead of Raft log persistence.

Configuration

Enable FastLog with the --fast flag when starting Dkron Pro.

CLI example

bash
dkron agent --server --bootstrap-expect=3 --fast --raft-duration=0

Environment variables

bash
DKRON_FAST=true
DKRON_RAFT_DURATION=0

Config file

yaml
fast: true
raft-duration: 0

Durability and performance tuning

The --raft-duration flag controls how aggressively FastLog flushes Raft log data to disk. Lower values favor performance. Higher values favor durability.

Value Mode Behavior Trade-off
-1 Low No explicit fsync by FastLog Highest performance, highest risk of losing recent writes after a crash or power loss
0 Mid Fsync approximately once per second Default setting and the best starting point for most deployments
1 High Fsync on every change Strongest durability, lowest write throughput

For most clusters, start with --raft-duration=0. Move to 1 if you need the strongest durability guarantees, or test -1 only if you fully understand the crash-recovery trade-off.

When to use FastLog

FastLog is a good option when:

  • You are running a busy production cluster with frequent scheduling activity.
  • Raft log persistence is contributing noticeable latency.
  • You want more control over the durability/performance trade-off for Raft writes.
  • You are scaling the server side of the cluster and want to reduce Raft storage overhead.

If your cluster is small and Raft storage is not a bottleneck, BoltDB may remain the simpler default choice.

Operational considerations

Storage format compatibility

FastLog uses a different on-disk format than BoltDB. The Raft storage files are not interchangeable.

:::warning
Do not point a FastLog-enabled node at an existing BoltDB Raft data directory and expect it to reuse the old log files. Treat the switch as a storage migration.
:::

Migration from BoltDB

Switching an existing cluster from BoltDB to FastLog should be planned as a maintenance operation.

  1. Stop all Dkron server nodes in the cluster.
  2. Back up the existing data-dir on every node.
  3. Remove or archive the existing Raft log data.
  4. Restart the cluster with --fast enabled on every server node.
  5. Verify cluster health, leadership, and job scheduling after the cluster comes back.

Because the storage formats differ, you should not mix old BoltDB Raft state with a FastLog deployment.

Rollout guidance

  • Test in staging first using a workload that resembles production.
  • Monitor memory usage because FastLog keeps the Raft log in memory for fast access.
  • Watch disk latency and Raft health after rollout, especially if you change --raft-duration.
  • Keep backups of your cluster data before changing Raft storage settings.

Summary

FastLog gives Dkron Pro operators a faster Raft storage backend with configurable durability. Enable it with --fast, tune it with --raft-duration, and plan migrations carefully because the underlying storage format is different from BoltDB.

Upgrading From V1 To V2 (`website/docs/upgrading/from_v1_to_v2.md`)

title: Upgrade from v1 to v2

Dkron v2 brings lots of changes to the previous version. To successfully upgrade from v1 to v2 you need to take care of certain steps.

Storage

Dkron v2 brings native storage, no external storage engine is needed. This change can be shocking for existing users and should be carefully planned to consider Dkron as a stateful service in contrast of a stateless service as in v1.

You must plan a backup strategy for the data directories, you can configure it using the --data-dir parameter, by default ./dkron.data.

This will ensure that you can recover cluster data in case of an unexpected failure.

Migrating Jobs

To migrate jobs from v1 to v2, export jobs from the v1 cluster and import them into v2.

A basic script to do that can be found here

You can take the opportunity to update your job definitions as explained in the following section:

Change job name

In v2 dkron doesn't accept jobs with invalid naming, adapt your scripts or api calls if necessary to set the job name with valid characters.

Changing tags and metadata

Tags behave different in dkron 2. There are several dkron tags that changed names and have become reserved. The reserved tags are:

  • dc
  • region
  • role
  • version
  • server
  • bootstrap
  • expect
  • rpc_addr
  • port

You can not set these tags with the tags param. Tags region and dc can be set using the region and datacenter params, respectively.

In Dkron 2, the job filtering API now filters on the metadata instead of the tags field. Jobs have a new param metadata that can be used to set any data to classify jobs. These can then be used to filter results returned by the API.

Upgrading From V2 0 To V2 2 (`website/docs/upgrading/from_v2_0_to_v2_2.md`)

title: Upgrade from v2.0.x to v2.2.x

Migrating Jobs

To migrate jobs from v2.0.x to v2.2.x, export jobs from the v2.0.x cluster and import them into v2.2.x

Refer to the backup&restore upgrade guide

Usage Chaining (`website/docs/usage/chaining.md`)

Job chaining

You can set some jobs to run after other job is executed. To setup a job that will be executed after any other given job, just set the parent_job property when saving the new job.

The dependent job will be executed after the main job finished a successful execution.

Child jobs schedule property will be ignored if it's present.

Take into account that parent jobs must be created before any child job.

Example:

json
{
  "name": "job1",
  "schedule": "@every 10s",
  "executor": "shell",
  "executor_config": {
    "command": "echo \"Hello from parent\""
  }
}

{
  "name": "child_job",
  "parent_job": "job1",
  "executor": "shell",
  "executor_config": {
    "command": "echo \"Hello from child\""
  }
}

Usage Cloud Auto Join (`website/docs/usage/cloud-auto-join.md`)

Dkron supports automatic cluster joining using cloud metadata on various providers.

Cloud Auto-join

As of Dkron 2.0.0, retry-join accepts a unified interface using the
go-discover library for doing
automatic cluster joining using cloud metadata. To use retry-join with a
supported cloud provider, specify the configuration on the command line or
configuration file as a key=value key=value ... string.

If the values contain spaces, equals, backslashes or double quotes then
they need to be double quoted and the usual escaping rules apply.

sh
$ dkron agent --retry-join 'provider=my-cloud config=val config2="some other val" ...'

or via a configuration file:

yml
retry-join": ["provider=my-cloud config=val config2=\"some other val\" ..."]

The cloud provider-specific configurations are detailed below. This can be
combined with static IP or DNS addresses or even multiple configurations
for different providers.

In order to use discovery behind a proxy, you will need to set
HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables per
Golang net/http library.

The following sections give the options specific to each supported cloud
provider.

Amazon EC2

This returns the first private IP address of all servers in the given
region which have the given tag_key and tag_value.

sh
$ dkron agent --retry-join "provider=aws tag_key=... tag_value=..."
yml
retry-join: ["provider=aws tag_key=... tag_value=..."]
  • provider (required) - the name of the provider ("aws" in this case).
  • tag_key (required) - the key of the tag to auto-join on.
  • tag_value (required) - the value of the tag to auto-join on.
  • region (optional) - the AWS region to authenticate in.
  • addr_type (optional) - the type of address to discover: private_v4, public_v4, public_v6. Default is private_v4. (>= 1.0)
  • access_key_id (optional) - the AWS access key for authentication (see below for more information about authenticating).
  • secret_access_key (optional) - the AWS secret access key for authentication (see below for more information about authenticating).

Authentication & Precedence

  • Static credentials access_key_id=... secret_access_key=...
  • Environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
  • Shared credentials file (~/.aws/credentials or the path specified by AWS_SHARED_CREDENTIALS_FILE)
  • ECS task role metadata (container-specific).
  • EC2 instance role metadata.

The only required IAM permission is ec2:DescribeInstances, and it is
recommended that you make a dedicated key used only for auto-joining. If the
region is omitted it will be discovered through the local instance's EC2
metadata
endpoint.

Microsoft Azure

This returns the first private IP address of all servers in the given region
which have the given tag_key and tag_value in the tenant and subscription, or in
the given resource_group of a vm_scale_set for Virtual Machine Scale Sets.

sh
$ dkron agent --retry-join "provider=azure tag_name=... tag_value=... tenant_id=... client_id=... subscription_id=... secret_access_key=..."
yml
retry-join: ["provider=azure tag_name=... tag_value=... tenant_id=... client_id=... subscription_id=... secret_access_key=..."]
  • provider (required) - the name of the provider ("azure" in this case).
  • tenant_id (required) - the tenant to join machines in.
  • client_id (required) - the client to authenticate with.
  • secret_access_key (required) - the secret client key. NOTE This value often may have an equals sign in it's value, especially if generated from the Azure Portal, so is important to wrap in single quotes eg. secret_acccess_key='fpOfcHQJAQBczjAxiVpeyLmX1M0M0KPBST+GU2GvEN4='

Variables can also be provided by environmental variables:

  • ARM_SUBSCRIPTION_ID for subscription
  • ARM_TENANT_ID for tenant
  • ARM_CLIENT_ID for client
  • ARM_CLIENT_SECRET for secret access key

Use these configuration parameters when using tags:

  • tag_name - the name of the tag to auto-join on.
  • tag_value - the value of the tag to auto-join on.

Use these configuration parameters (instead of tag_name and tag_value) when using Virtual Machine Scale Sets (Dkron 1.0.3 and later):

  • resource_group - the name of the resource group to filter on.
  • vm_scale_set - the name of the virtual machine scale set to filter on.

When using tags the only permission needed is Microsoft.Network/networkInterfaces.

When using Virtual Machine Scale Sets the only role action needed is Microsoft.Compute/virtualMachineScaleSets/*/read.

Google Compute Engine

This returns the first private IP address of all servers in the given
project which have the given tag_value.

sh
$ dkron agent --retry-join "provider=gce project_name=... tag_value=..."
yml
retry_join: ["provider=gce project_name=... tag_value=..."]
  • provider (required) - the name of the provider ("gce" in this case).
  • tag_value (required) - the value of the tag to auto-join on.
  • project_name (optional) - the name of the project to auto-join on. Discovered if not set.
  • zone_pattern (optional) - the list of zones can be restricted through an RE2 compatible regular expression. If omitted, servers in all zones are returned.
  • credentials_file (optional) - the credentials file for authentication. Note, if you set -config-dir do not store the credentials.json file in the configuration directory as it will be parsed as a config file and Dkron will fail to start. See below for more information.

Authentication & Precedence

  • Use credentials from credentials_file, if provided.
  • Use JSON file from GOOGLE_APPLICATION_CREDENTIALS environment variable.
  • Use JSON file in a location known to the gcloud command-line tool.
    • On Windows, this is %APPDATA%/gcloud/application_default_credentials.json.
    • On other systems, $HOME/.config/gcloud/application_default_credentials.json.
  • On Google Compute Engine, use credentials from the metadata
    server. In this final case any provided scopes are ignored.

Discovery requires a GCE Service
Account.
Credentials are searched using the following paths, in order of precedence.

IBM SoftLayer

This returns the first private IP address of all servers for the given
datacenter with the given tag_value.

sh
$ dkron agent --retry-join "provider=softlayer datacenter=... tag_value=... username=... api_key=..."
yml
retry-join: ["provider=softlayer datacenter=... tag_value=... username=... api_key=..."]
  • provider (required) - the name of the provider ("softlayer" in this case).
  • datacenter (required) - the name of the datacenter to auto-join in.
  • tag_value (required) - the value of the tag to auto-join on.
  • username (required) - the username to use for auth.
  • api_key (required) - the api key to use for auth.

Aliyun (Alibaba Cloud)

This returns the first private IP address of all servers for the given
region with the given tag_key and tag_value.

sh
$ dkron agent --retry-join "provider=aliyun region=... tag_key=dkron tag_value=... access_key_id=... access_key_secret=..."
yml
retry-join: ["provider=aliyun region=... tag_key=dkron tag_value=... access_key_id=... access_key_secret=..."]
  • provider (required) - the name of the provider ("aliyun" in this case).
  • region (required) - the name of the region.
  • tag_key (required) - the key of the tag to auto-join on.
  • tag_value (required) - the value of the tag to auto-join on.
  • access_key_id (required) -the access key to use for auth.
  • access_key_secret (required) - the secret key to use for auth.

The required RAM permission is ecs:DescribeInstances.
It is recommended you make a dedicated key used only for auto-joining.

Digital Ocean

This returns the first private IP address of all servers for the given
region with the given tag_name.

sh
$ dkron agent --retry-join "provider=digitalocean region=... tag_name=... api_token=..."
yml
retry-join: ["provider=digitalocean region=... tag_name=... api_token=..."]
  • provider (required) - the name of the provider ("digitalocean" in this case).
  • region (required) - the name of the region.
  • tag_name (required) - the value of the tag to auto-join on.
  • api_token (required) -the token to use for auth.

Openstack

This returns the first private IP address of all servers for the given
region with the given tag_key and tag_value.

sh
$ dkron agent --retry-join "provider=os tag_key=dkron tag_value=server username=... password=... auth_url=..."
yml
retry-join: ["provider=os tag_key=dkron tag_value=server username=... password=... auth_url=..."]
  • provider (required) - the name of the provider ("os" in this case).
  • tag_key (required) - the key of the tag to auto-join on.
  • tag_value (required) - the value of the tag to auto-join on.
  • project_id (optional) - the id of the project (tenant id).
  • username (optional) - the username to use for auth.
  • password (optional) - the password to use for auth.
  • token (optional) - the token to use for auth.
  • auth_url (optional) - the identity endpoint to use for auth.
  • insecure (optional) - indicates whether the API certificate should not be checked. Any value means true.

The configuration can also be provided by environment variables.

Scaleway

This returns the first private IP address of all servers for the given
region with the given tag_name.

sh
$ dkron agent --retry-join "provider=scaleway organization=my-org tag_name=dkron-server token=... region=..."
yml
retry-join: ["provider=scaleway organization=my-org tag_name=dkron-server token=... region=..."]
  • provider (required) - the name of the provider ("scaleway" in this case).
  • region (required) - the name of the region.
  • tag_name (required) - the name of the tag to auto-join on.
  • organization (required) - the organization access key to use for auth (equal to access key).
  • token (required) - the token to use for auth.

Joyent Triton

This returns the first PrimaryIP addresses for all servers with the given tag_key and tag_value.

sh
$ dkron agent --retry-join "provider=triton account=testaccount url=https://us-sw-1.api.joyentcloud.com key_id=... tag_key=dkron-role tag_value=server"
yml
retry-join: ["provider=triton account=testaccount url=https://us-sw-1.api.joyentcloud.com key_id=... tag_key=dkron-role tag_value=server"]
  • provider (required) - the name of the provider ("triton" in this case).
  • account (required) - the name of the account.
  • url (required) - the URL of the Triton api endpoint to use.
  • key_id (required) - the key id to use.
  • tag_key (optional) - the instance tag key to use.
  • tag_value (optional) - the tag value to use.

vSphere

This returns the first private IP address of all servers for the given region with the given tag_name and category_name.

sh
$ dkron agent --retry-join "provider=vsphere category_name=dkron-role tag_name=dkron-server host=... user=... password=... insecure_ssl=[true|false]"
yml
retry-join: ["provider=vsphere category_name=dkron-role tag_name=dkron-server host=... user=... password=... insecure_ssl=[true|false]"]
  • provider (required) - the name of the provider ("vsphere" is the provider here)
  • tag_name (required) - The name of the tag to look up.
  • category_name (required) - The category of the tag to look up.
  • host (required) - The host of the vSphere server to connect to.
  • user (required) - The username to connect as.
  • password (required) - The password of the user to connect to vSphere as.
  • insecure_ssl (optional) - Whether or not to skip SSL certificate validation.
  • timeout (optional) - Discovery context timeout (default: 10m)

Packet

This returns the first private IP address (or the IP address of address type) of all servers with the given project and auth_token.

sh
$ dkron agent --retry-join "provider=packet auth_token=token project=uuid url=... address_type=..."
yml
retry-join: ["provider=packet auth_token=token project=uuid url=... address_type=..."]
  • provider (required) - the name of the provider ("packet" is the provider here)
  • project (required) - the UUID of packet project
  • auth_token (required) - the authentication token for packet
  • url (optional) - a REST URL for packet
  • address_type (optional) - the type of address to check for in this provider ("private_v4", "public_v4" or "public_v6". Defaults to "private_v4")

Kubernetes (k8s)

The Kubernetes provider finds the IP addresses of pods with the matching
label or field selector.
This is useful for non-Kubernetes agents that are joining a server cluster
running within Kubernetes.

The pod IP is used by default, which requires that the agent connecting can
network to the pod IP. The host_network boolean can be set to true to use
the host IP instead, but this requires the agent ports (Gossip, RPC, etc.)
to be exported to the host as well.

By default, no port is specified. This causes Dkron to use the default
gossip port (default behavior with all join requests). The pod may specify
the dkron.hashicorp.com/auto-join-port annotation to set the port. The value
may be an integer or a named port.

sh
$ dkron agent --retry-join "provider=k8s label_selector=\"app=dkron,component=server\""
yml
retry-join: ["provider=k8s label_selector=..."]
  • provider (required) - the name of the provider ("k8s" is the provider here)
  • kubeconfig (optional) - path to the kubeconfig file. If this isn't
    set, then in-cluster auth will be attempted. If that fails, the default
    kubeconfig paths are tried ($HOME/.kube/config).
  • namespace (optional) - the namespace to search for pods. If this isn't
    set, it defaults to all namespaces.
  • label_selector (optional) - the label selector for matching pods.
  • field_selector (optional) - the field selector for matching pods.

Usage Clustering (`website/docs/usage/clustering.md`)

title: Clustering

Clustering

High Availability Clustering

Dkron is designed to operate as a cluster of multiple nodes for fault tolerance and high availability. This guide explains how to set up and manage a Dkron cluster.

mermaid
flowchart TD
    subgraph DC1[Datacenter 1]
        Server1[Server Node 1\nLeader] <--> Server2[Server Node 2\nFollower]
        Server1 <--> Server3[Server Node 3\nFollower]
        Server1 --- Agent1[Agent 1]
        Server2 --- Agent1
        Server3 --- Agent1
    end
    
    subgraph DC2[Datacenter 2]
        Server4[Server Node 4\nFollower] <--> Server5[Server Node 5\nFollower]
        Server4 --- Agent2[Agent 2]
        Server5 --- Agent2
    end
    
    DC1 <--> DC2

Cluster Formation

Bootstrap Methods

Dkron supports several methods for nodes to discover and join each other:

  1. Static List: Explicitly specify other cluster members
  2. Auto-join: Automatically discover other nodes
  3. Cloud Auto-join: Discover nodes in cloud environments

Basic Cluster Setup

To start a basic 3-node server cluster:

  1. Start the first node:

    text
    dkron agent --server --bootstrap-expect=3 --node-name=node1 --bind=10.0.0.1 --advertise=10.0.0.1

  2. Start additional server nodes:

    text
    dkron agent --server --bootstrap-expect=3 --node-name=node2 --bind=10.0.0.2 --advertise=10.0.0.2 --join=10.0.0.1
dkron agent --server --bootstrap-expect=3 --node-name=node3 --bind=10.0.0.3 --advertise=10.0.0.3 --join=10.0.0.1

  3. Add agent nodes:

    text
    dkron agent --node-name=agent1 --bind=10.0.0.4 --join=10.0.0.1

Key Configuration Parameters

Parameter Description Example
--server Run in server mode --server
--bootstrap-expect Expected number of servers --bootstrap-expect=3
--node-name Unique node identifier --node-name=node1
--bind Address to bind network services --bind=10.0.0.1
--advertise Address to advertise to cluster --advertise=10.0.0.1
--join Address of another node to join --join=10.0.0.1
--retry-join Auto-retry joining --retry-join=10.0.0.1
--tag Node tag for job targeting --tag role=web

Multi-Datacenter Configuration

Dkron can run in HA mode, avoiding SPOFs, this mode provides better scalability and better reliability for users that wants a high level of confidence in the cron jobs they need to run.

Manually bootstrapping a Dkron cluster does not rely on additional tooling, but does require operator participation in the cluster formation process. When bootstrapping, Dkron servers and clients must be started and informed with the address of at least one Dkron server.

As you can tell, this creates a chicken-and-egg problem where one server must first be fully bootstrapped and configured before the remaining servers and clients can join the cluster. This requirement can add additional provisioning time as well as ordered dependencies during provisioning.

First, we bootstrap a single Dkron server and capture its IP address. After we have that nodes IP address, we place this address in the configuration.

  1. First bootstrap a node with a configuration like this:
yaml
# dkron.yml
server: true
bootstrap-expect: 1

  1. Then stop the bootstrapped server and capture the server IP address.

  2. To form a cluster, configure server nodes (including the bootstrapped server) with the address of its peers as in the following example:

yaml
# dkron.yml
server: true
bootstrap-expect: 3
retry-join:
- 10.19.3.9
- 10.19.4.64
- 10.19.7.215

Deployment Table

Below is a table that shows quorum size and failure tolerance for various
cluster sizes. The recommended deployment is either 3 or 5 servers. A single
server deployment is highly discouraged as data loss is inevitable in a
failure scenario.

Servers Quorum Size Failure Tolerance
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2
6 4 2
7 4 3

Fault Tolerance and Recovery

Dkron's distributed architecture is designed to maintain operation even when nodes fail.

Leader Failure Recovery

When a leader node fails:

  1. Remaining server nodes detect the failure through gossip protocol
  2. A new leader election is automatically triggered
  3. A new leader is elected from the available server nodes
  4. The new leader takes over scheduling responsibilities
  5. All running jobs continue to operate without interruption

Quorum Requirements

  • A cluster must maintain a quorum (majority) of server nodes to operate
  • For a cluster of size N, at least (N/2)+1 nodes must be available
  • If quorum is lost, the cluster stops scheduling new jobs until quorum is restored
  • Refer to the Deployment Table above for specific quorum sizes and failure tolerance

Node Rejoining

When a previously failed node rejoins the cluster:

  1. The node establishes connection with existing cluster members
  2. It synchronizes its state with the current leader
  3. If it was previously a leader, it joins as a follower
  4. The node becomes available for job execution

Data Recovery

In case of catastrophic failure:

  1. Restore from backup if available
  2. Bootstrap a new cluster with at least one server node
  3. Import job definitions via API
  4. New executions will begin according to schedule

Cluster Maintenance

Proper maintenance ensures your Dkron cluster remains healthy and performant.

Adding Nodes

To add a new server node to an existing cluster:

  1. Install Dkron on the new server
  2. Configure with appropriate server settings
  3. Set --join parameter to point to existing cluster members
  4. Start the Dkron process
  5. Verify the node joins successfully via the web UI or API

Adding agent nodes follows the same process but without the --server flag.

Removing Nodes

To gracefully remove a node:

  1. For agents: simply stop the Dkron service
  2. For server nodes:
    • If possible, demote the node to an agent first
    • Ensure you maintain sufficient servers for quorum
    • Stop the Dkron service

Upgrading Dkron

For minimal disruption during upgrades:

  1. Rolling Upgrade (recommended):

    • Upgrade one node at a time
    • Start with agent nodes
    • Then upgrade server nodes, leaving the leader for last
    • Allow state synchronization between each node upgrade

  2. Version Compatibility:

    • Check release notes for compatibility information
    • Ensure all nodes in a cluster run compatible versions
    • Some upgrades may require special procedures

Monitoring Cluster Health

Regular health checks should include:

  1. Node Status: Check that all expected nodes are active
  2. Leadership: Verify leader election is stable
  3. Job Execution: Monitor successful job execution rates
  4. Resource Usage: Track CPU, memory, and disk usage
  5. Log Analysis: Review logs for errors or warnings

Backup and Recovery

Implement a regular backup strategy:

  1. Data Store Backup:

    • Stop Dkron service or use backup-friendly commands
    • Copy the data directory to a secure location
    • Restart the service if stopped

  2. Configuration Backup:

    • Maintain copies of configuration files
    • Document any custom settings

  3. Job Definition Export:

    • Use the API to export job definitions
    • Store in version control for tracking changes

Usage Concepts (`website/docs/usage/concepts.md`)

Concepts

Core Concepts

Dkron's architecture is based on a few key concepts that are essential to understand:

mermaid
flowchart TD
    Job["Job Definition"] --> Schedule["Schedule"]
    Job --> Executor["Executor"]
    Job --> Processors["Processors"]
    Job --> TargetNodes["Target Nodes"]
    
    Schedule --> Execution["Job Execution"]
    Executor --> Execution
    TargetNodes --> Execution
    
    Execution --> Status["Execution Status"]
    Execution --> Output["Execution Output"]
    
    Status --> Retry["Retry Logic"]
    Status --> Dependent["Dependent Jobs"]
    
    Output --> Processors

Jobs

Jobs are the central entity in Dkron. A job represents a scheduled task with the following properties:

Property Description
Name Unique identifier for the job
Schedule When to run (cron expression)
Timezone Reference timezone for the schedule
Owner Owner name for accountability
Disabled Flag to disable the job temporarily
Tags Key-value pairs for node targeting
Metadata Custom user data for the job
Concurrency How to handle overlapping executions
Executor Plugin to execute the job
Retry Configuration for retrying failed jobs
Dependent Jobs Jobs that should run after this one

Schedules

Jobs are scheduled using cron expressions. Dkron supports:

  • Standard cron expressions (* * * * *)
  • Predefined schedules (@hourly, @daily, @weekly)
  • Interval notation (@every 1h30m)
  • Custom timezone support

Executors

Executors are plugins that perform the actual job execution:

  • Shell: Executes commands on the target node
  • HTTP: Makes HTTP requests to specified endpoints
  • Kafka: Produces messages to Kafka topics
  • And more: Custom executors can be implemented

Processors

Processors handle the output from job executions:

  • Log: Records execution output to logs
  • Files: Saves output to files
  • Custom: User-implemented processors

Target Node Selection

Jobs can be targeted to specific nodes using:

  • Tags: Key-value pairs assigned to nodes
  • Count: Number of nodes to run the job on
  • Selector: Logic for choosing nodes

Example tag specification:

json
"tags": {
  "role": "web",
  "datacenter": "us-east:2"
}

Concurrency Control

Dkron provides several concurrency options:

  • Allow: Multiple executions can run simultaneously
  • Forbid: New executions are skipped if one is running
  • Replace: New executions replace running ones

Job Dependencies

Jobs can depend on other jobs:

  • Parent jobs must complete successfully before dependent jobs run
  • Dependencies can form complex workflows
  • Status codes determine success/failure

Advanced Concepts

Distributed Leadership

mermaid
flowchart LR
    subgraph Before
        Leader1[Node 1\nLeader] --> Follower1[Node 2\nFollower]
        Leader1 --> Follower2[Node 3\nFollower]
    end
    
    subgraph After[After Node 1 Failure]
        Follower3[Node 2\nNew Leader] --> X[Node 1\nDown]
        Follower3 --> Follower4[Node 3\nFollower]
    end
    
    Before --> After

Dkron uses a leader-follower model to ensure high availability:

  • One server node is elected as leader
  • The leader is responsible for scheduling
  • If the leader fails, a new leader is automatically elected
  • The cluster continues operating without interruption
  • When the old leader rejoins, it becomes a follower

Status Codes

Jobs return status codes that indicate success or failure:

  • By default, a zero exit code means success
  • You can customize which codes are considered successful
  • Status codes determine whether dependent jobs run
  • Processors can take different actions based on status codes

Storage Backend

Dkron uses an embedded BoltDB database for:

  • Storing job definitions
  • Recording execution history
  • Maintaining cluster state
  • Storing processor output

For larger deployments, consider these best practices:

  • Regular database backups
  • Monitoring storage usage
  • Setting appropriate retention policies

Security Considerations

mermaid
flowchart TD
    User[User] -->|Authentication| API[API Layer]
    API -->|Authorization| Jobs[Job Execution]
    API -->|TLS| Agent[Agent Communication]
    
    subgraph Security Controls
        Authentication[Authentication]
        Authorization[Authorization]
        TLS[Transport Layer Security]
        Encryption[Data Encryption]
    end

Securing your Dkron deployment:

  • Authentication (Pro feature): Protect the API with authentication
  • Authorization (Pro feature): Control who can manage jobs
  • TLS (Pro feature): Encrypt communication between nodes
  • Network Security: Use firewalls to restrict access
  • Secrets Management: Carefully handle credentials in jobs

Failure Recovery

Dkron provides multiple mechanisms for handling failures:

  • Automatic retries: Configure jobs to retry on failure
  • Dependent jobs with fallbacks: Create contingency plans
  • Notifications: Set up alerting for critical failures
  • Health checks: Monitor node and cluster status

Performance Optimization

For optimal performance:

  • Distribute jobs across nodes based on resource requirements
  • Use appropriate concurrency settings
  • Monitor execution times and adjust schedules
  • Consider time zone impacts on scheduling
  • Optimize command execution with appropriate timeouts

Usage Concurrency (`website/docs/usage/concurrency.md`)

title: Concurrency toc: true

Concurrency

Jobs can be configured to allow overlapping executions or forbid them.

Concurrency property accepts two option:

  • allow (default): Allow concurrent job executions.
  • forbid: If the job is already running don't send the execution, it will skip the executions until the next schedule.

Example:

json
{
  "name": "job1",
  "schedule": "@every 10s",
  "executor": "shell",
  "executor_config": {
    "command": "echo \"Hello from parent\""
  },
  "concurrency": "forbid"
}

Usage Cron Spec (`website/docs/usage/cron-spec.md`)

title: Cron Specification weight: 20

Cron Expression Format

Dkron uses a powerful cron expression format to define job schedules. This guide explains the syntax in detail with practical examples.

Basic Format

A cron expression consists of 6 space-separated fields:

Field name Mandatory? Allowed values Allowed special characters
Seconds Yes 0-59 * / , - ~
Minutes Yes 0-59 * / , - ~
Hours Yes 0-23 * / , - ~
Day of month Yes 1-31 * / , - ? ~
Month Yes 1-12 or JAN-DEC * / , - ~
Day of week Yes 0-6 or SUN-SAT * / , - ? ~

Note: Month and Day-of-week field values are case insensitive. "SUN", "Sun", and "sun" are equally accepted.

Special Characters

Asterisk ( * )

The asterisk indicates that the cron expression will match for all values of the field.

Example: * * * * * * runs every second of every minute of every hour, every day.

Slash ( / )

Slashes define increments within ranges.

Example: 0 */15 * * * * runs every 15 minutes (at 0:00, 0:15, 0:30, 0:45).

Comma ( , )

Commas are used to separate items in a list.

Example: 0 0 0 * * MON,WED,FRI runs at midnight on Mondays, Wednesdays, and Fridays.

Hyphen ( - )

Hyphens define ranges.

Example: 0 0 9-17 * * * runs every hour from 9 AM to 5 PM.

Question mark ( ? )

Question mark may be used instead of '*' for leaving either day-of-month or day-of-week blank to avoid conflicts.

Example: 0 0 0 1 * ? runs at midnight on the first day of each month, regardless of the day of the week.

Tilde ( ~ )

Tilde is replaced by a deterministic numeric value based on the job name, providing an even distribution of load.

Example: 0 ~ * * * * distributes jobs evenly across different minutes within the hour.

Predefined Schedules

For convenience, Dkron supports several predefined schedules:

Entry Description Equivalent To
@yearly (or @annually) Run once a year at midnight on January 1 0 0 0 1 1 *
@monthly Run once a month at midnight on first day 0 0 0 1 * *
@weekly Run once a week at midnight on Sunday 0 0 0 * * 0
@daily (or @midnight) Run once a day at midnight 0 0 0 * * *
@hourly Run once an hour at the beginning of hour 0 0 * * * *
@minutely Run once a minute at the beginning 0 * * * * *
@manually Never runs automatically (manual triggers) N/A
@at Run once at specified time N/A
@after Run once with grace period for late jobs N/A

Example: @daily is equivalent to 0 0 0 * * *

Fixed Intervals

You can schedule jobs to execute at fixed intervals using:

text
@every <duration>

Where "duration" is a string accepted by time.ParseDuration.

Example: @every 1h30m10s runs every 1 hour, 30 minutes, and 10 seconds.

Note: The interval does not account for job runtime. For example, if a job takes 3 minutes to run, and it is scheduled to run every 5 minutes, it will have only 2 minutes of idle time between each run.

One-time Execution

To schedule a job to be executed just once at a specific time:

text
@at <datetime>

Where "datetime" is a string in RFC3339 format.

Example: @at 2023-12-31T23:59:00Z will run the job once on December 31, 2023 at 11:59 PM UTC.

Important: If a job is created after the specified time, it will never run.

One-time Execution with Grace Period

To schedule a job that can still run if created slightly after its intended time (useful for handling network latency or delayed job creation):

text
@after <datetime> <duration>

Where:

Behavior:

  • If the job is created before the scheduled time → it runs at the scheduled time
  • If the job is created within the grace period after the scheduled time → it runs immediately
  • If the job is created after the grace period → it never runs

Duration Format:
The grace period uses ISO8601 duration format: P[n]Y[n]M[n]DT[n]H[n]M[n]S

  • P - Period designator (required)
  • T - Time designator (required for hours, minutes, seconds)
  • Years: P1Y = 1 year (365 days approximation)
  • Months: P1M = 1 month (30 days approximation)
  • Days: P1D = 1 day
  • Hours: PT1H = 1 hour
  • Minutes: PT1M = 1 minute
  • Seconds: PT1S = 1 second

Examples:

text
@after 2023-12-31T23:59:00Z <PT2H

Runs at 11:59 PM UTC on December 31, 2023, or immediately if created up to 2 hours late.

text
@after 2024-01-01T09:00:00Z <PT30M

Runs at 9:00 AM on January 1, 2024, or immediately if created up to 30 minutes late.

text
@after 2024-01-15T00:00:00Z <P1D

Runs at midnight on January 15, 2024, or immediately if created up to 1 day late.

text
@after 2024-02-01T02:00:00Z <P1DT2H30M

Runs at 2:00 AM on February 1, 2024, or immediately if created up to 1 day, 2 hours, and 30 minutes late.

Use Cases:

  • Network Latency Tolerance: When creating jobs via API, network delays might cause the job to be created after its intended execution time. The grace period allows it to still run.
  • Flexible Maintenance Windows: Schedule maintenance tasks that can run anytime within a specific window.
  • Event-Based Execution: Jobs that should run once after a specific event time but need flexibility for delayed creation.

Note: The grace period boundary is inclusive - a job created exactly at the end of the grace period will still run immediately.

Time Zones

Dkron supports scheduling jobs in specific time zones by specifying the timezone parameter in a job definition.

If no time zone is specified:

  • All scheduling is done in the server's local time zone
  • Jobs scheduled during daylight-savings leap-ahead transitions will not be run

When a timezone is specified, the job will be scheduled in the specified time zone, correctly handling daylight saving time transitions.

Example timezone values:

  • "America/New_York"
  • "Europe/London"
  • "Asia/Tokyo"
  • "UTC"

Practical Examples

Here are some common scheduling patterns with explanations:

Every weekday at 8 AM

text
0 0 8 * * 1-5

This runs at 8:00 AM Monday through Friday.

Every 10 minutes during business hours

text
0 */10 9-17 * * 1-5

This runs every 10 minutes from 9 AM to 5 PM, Monday through Friday.

First day of every month at 3 AM

text
0 0 3 1 * *

This runs at 3:00 AM on the first day of each month.

Every 15 minutes with even load distribution

text
0 ~/15 * * * *

This distributes jobs evenly across the hour in 15-minute intervals, with the exact minute determined by the job name.

Every quarter (Jan, Apr, Jul, Oct) on the first day at midnight

text
0 0 0 1 1,4,7,10 *

This runs at midnight on the first day of January, April, July, and October.

Every weekend at 10 PM

text
0 0 22 * * 0,6

This runs at 10:00 PM on Saturdays and Sundays.

Every day at 8 AM New York time

text
0 0 8 * * *

With timezone parameter set to "America/New_York" in the job configuration.

Best Practices

  1. Avoid Running Too Frequently: Consider resource usage when scheduling frequent jobs. Running jobs every few seconds can put unnecessary load on your system.

  2. Use Even Load Distribution: For jobs that don't need to run at an exact time, use the tilde (~) character to distribute load evenly.

  3. Consider Time Zones Carefully: Be explicit about time zones in distributed systems to avoid confusion, especially if your servers are in different locations.

  4. Avoid Overlapping Executions: For long-running jobs, ensure the schedule interval is longer than the expected execution time, or enable concurrency controls in the job configuration.

  5. Use Descriptive Job Names: With the tilde (~) feature, job names influence scheduling, so use consistent naming conventions.

  6. Test Complex Expressions: Use tools like crontab.guru to validate your cron expressions (note: these tools typically use 5-field format, while Dkron uses 6 fields with seconds).

  7. Document Job Schedules: Maintain documentation about why jobs are scheduled at specific times to help with maintenance and troubleshooting.

Usage Cronitor (`website/docs/usage/cronitor.md`)

Cronitor Integration

:::info
This feature is available since v3.2.0
:::

Dkron includes tight integration with Cronitor for advanced execution monitoring, check their product page for more information.

To setup the integration add this parameter to your dkron.yaml config file, including your private cronitor API endpoint key:

:::caution
Remember: do not check your config file into your source code repository, it contains sensitive information.
:::

text
cronitor-endpoint: https://cronitor.link/p/xxxxxxxxxxxxxxxxxxxxxx

You can also use the --cronitor-endpoint CLI parameter or the DKRON_CRONITOR_ENDPOINT env variable.

Dkron will call Cronitor before and after running a job, sending all necessary information, no further configuration is necessary as Cronitor automagically create a monitor for each job.

You can further configure your job using the Cronitor UI or API after it has run at least once.

Cronitor can be used to notify over different channels using integrations, go to the settings page to integrate it with several popular services:

Usage Ecs (`website/docs/usage/ecs.md`)

title: Use with AWS ECS

:::tip
Dkron Pro comes with a native ECS executor out of the box.
:::

Use with Amazon ECS

To use Dkron to schedule jobs that run in containers, a wrapper ECS script is needed.

Install the following snippet in the node that will run the call to ECS

Prerequisites

The node that will run the call to ECS will need to have installed

  • AWS cli
  • jq

Example

ecs-run --cluster cron --task-definition cron-taskdef --container-name cron --region us-east-1 --command "rake foo"

Usage Metatags (`website/docs/usage/metatags.md`)

title: Job metadata

Job metadata

Jobs can have an optional extra property field called metadata that allows to set arbitrary tags to jobs and query the jobs using the API:

json
{
    "name": "job_name",
    "command": "/bin/true",
    "schedule": "@every 2m",
    "metadata": {
        "user_id": "12345"
    }
}

And then query the API to get only the results needed:

text
$ curl http://localhost:8080/v1/jobs --data-urlencode "metadata[user_id]=12345"`

Usage Metrics (`website/docs/usage/metrics.md`)

title: Metrics

Dkron has the ability to send metrics to Statsd for dashboards and historical reporting or provide prometheus format metrics via the api. It sends job processing metrics, golang, and serf metrics.

Configuration

Statsd

Add this in your yaml config file to enable statsd metrics.

yaml
statsd-addr: "localhost:8125"
# Or for datadog statsd
dog-statsd-addr: "localhost:8125"

Prometheus

Add this to your yaml config file to enable serving prometheus metrics at the endpoint /metrics

yaml
enable-prometheus: true

Additionally, in your Prometheus config file (prometheus.yml), add the following to link dkron metric endpoint

yaml
scrape_configs:
  ... #initial configuration
  
  - job_name: "dkron_metrics"
    # metrics_path defaults to '/metrics'
    static_configs:
      - targets: ["localhost:8080"]

Monitoring Dashboard Setup

Setting up a monitoring dashboard with metrics from Dkron is highly recommended for production deployments. You have several options:

  1. Grafana + Prometheus: Create dashboards with job execution success rates, execution timing trends, and system health metrics
  2. DataDog: Use built-in dashboard templates with the DogStatsD integration
  3. Custom StatsD Dashboards: Configure with any StatsD-compatible visualization tool

Example Grafana dashboard panels for monitoring Dkron might include:

  • Job execution success rate over time
  • Average execution duration by job
  • System resource utilization
  • Cluster health status

Metrics Reference

Dkron emits several categories of metrics to help you monitor both the application and your job executions:

Agent Event Metrics

These metrics track internal events within Dkron agents:

Metric Description
dkron.agent.event_received.query_execution_done Count of completed job execution events received by the agent
dkron.agent.event_received.query_run_job Count of job run requests received by the agent

Network Communication Metrics

These metrics help monitor the health of inter-node communication:

Metric Description
dkron.memberlist.gossip Count of gossip protocol messages exchanged
dkron.memberlist.probeNode Count of node health probe checks
dkron.memberlist.pushPullNode Count of anti-entropy sync operations
dkron.memberlist.tcp.accept Count of accepted TCP connections
dkron.memberlist.tcp.connect Count of initiated TCP connections
dkron.memberlist.tcp.sent Count and bytes of TCP packets sent
dkron.memberlist.udp.received Count and bytes of UDP packets received
dkron.memberlist.udp.sent Count and bytes of UDP packets sent

gRPC Service Metrics

These metrics track the internal RPC communication:

Metric Description
dkron.grpc.call_execution_done Count and timing of execution completion RPC calls
dkron.grpc.call_get_job Count and timing of job retrieval RPC calls
dkron.grpc.execution_done Count of completed job executions
dkron.grpc.get_job Count of job information retrievals

Runtime Metrics

These metrics provide insights into the Go runtime health:

Metric Description
dkron.runtime.alloc_bytes Current bytes allocated by the application
dkron.runtime.free_count Count of memory free operations
dkron.runtime.gc_pause_ns Duration of the last garbage collection pause in nanoseconds
dkron.runtime.heap_objects Number of objects in the heap
dkron.runtime.malloc_count Count of memory allocation operations
dkron.runtime.num_goroutines Number of goroutines currently running
dkron.runtime.sys_bytes Total bytes of memory obtained from the OS
dkron.runtime.total_gc_pause_ns Total time spent in GC pauses
dkron.runtime.total_gc_runs Total number of completed GC cycles

Serf Metrics

These metrics track the cluster membership and failure detection system:

Metric Description
dkron.serf.coordinate.adjustment_ms Time adjustment for coordinate system in milliseconds
dkron.serf.msgs.received Count of messages received
dkron.serf.msgs.sent Count of messages sent
dkron.serf.queries Count of queries processed
dkron.serf.queries.execution_done Count of execution completion queries
dkron.serf.queries.run_job Count of job run queries
dkron.serf.query_acks Count of query acknowledgments
dkron.serf.query_responses Count of responses to queries
dkron.serf.queue.Event Count of events in processing queue
dkron.serf.queue.Intent Count of intent messages in queue
dkron.serf.queue.Query Count of queries in processing queue

Alerting on Metrics

To set up effective alerting based on Dkron metrics, consider these recommendations:

  1. Job Failure Alerts: Monitor dkron.serf.queries.execution_done with error status
  2. Cluster Health: Set alerts on node count changes or consistent gossip failures
  3. Performance Degradation: Watch for increases in job execution time trends
  4. Resource Constraints: Monitor dkron.runtime.gc_pause_ns and other runtime metrics for signs of resource pressure

For Prometheus users, example alerting rules:

yaml
groups:
- name: dkron_alerts
  rules:
  - alert: DkronJobFailures
    expr: increase(dkron_job_executions_failed_total[1h]) > 3
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Multiple job failures detected"
      description: "There have been more than 3 job failures in the last hour"

Usage Recovery (`website/docs/usage/recovery.md`)

title: Outage recovery

Outage Recovery

Don't panic! This is a critical first step.

Depending on your deployment configuration, it may take only a single server failure for cluster unavailability. Recovery requires an operator to intervene, but the process is straightforward.

:::tip
This guide is for recovery from a Dkron outage due to a majority of server nodes in a datacenter being lost. If you are looking to add or remove servers, see the clustering guide.
:::

Failure of a Single Server Cluster

If you had only a single server and it has failed, simply restart it. A single server configuration requires the -bootstrap-expect=1 flag. If the server cannot be recovered, you need to bring up a new server. See the clustering guide for more detail.

In the case of an unrecoverable server failure in a single server cluster, data loss is inevitable since data was not replicated to any other servers. This is why a single server deploy is never recommended.

Failure of a Server in a Multi-Server Cluster

If you think the failed server is recoverable, the easiest option is to bring it back online and have it rejoin the cluster with the same IP address, returning the cluster to a fully healthy state. Similarly, even if you need to rebuild a new Dkron server to replace the failed node, you may wish to do that immediately. Keep in mind that the rebuilt server needs to have the same IP address as the failed server. Again, once this server is online and has rejoined, the cluster will return to a fully healthy state.

Both of these strategies involve a potentially lengthy time to reboot or rebuild a failed server. If this is impractical or if building a new server with the same IP isn't an option, you need to remove the failed server. Usually, you can issue a dkron leave command to remove the failed server if it's still a member of the cluster.

If dkron leave isn't able to remove the server, you can use the dkron raft remove-peer command to remove the stale peer server on the fly with no downtime.

You can use the dkron raft list-peers command to inspect the Raft configuration:

text
$ dkron raft list-peers
Node                   ID               Address          State     Voter
dkron-server01.global  10.10.11.5:4647  10.10.11.5:4647  follower  true
dkron-server02.global  10.10.11.6:4647  10.10.11.6:4647  leader    true
dkron-server03.global  10.10.11.7:4647  10.10.11.7:4647  follower  true

Failure of Multiple Servers in a Multi-Server Cluster

In the event that multiple servers are lost, causing a loss of quorum and a complete outage, partial recovery is possible using data on the remaining servers in the cluster. There may be data loss in this situation because multiple servers were lost, so information about what's committed could be incomplete. The recovery process implicitly commits all outstanding Raft log entries, so it's also possible to commit data that was uncommitted before the failure.

See the section below for details of the recovery procedure. You simply include just the remaining servers in the raft/peers.json recovery file. The cluster should be able to elect a leader once the remaining servers are all restarted with an identical raft/peers.json configuration.

Any new servers you introduce later can be fresh with totally clean data directories.

In extreme cases, it should be possible to recover with just a single remaining server by starting that single server with itself as the only peer in the raft/peers.json recovery file.

The raft/peers.json recovery file is final, and a snapshot is taken after it is ingested, so you are guaranteed to start with your recovered configuration. This does implicitly commit all Raft log entries, so should only be used to recover from an outage, but it should allow recovery from any situation where there's some cluster data available.

Manual Recovery Using peers.json

To begin, stop all remaining servers. You can attempt a graceful leave, but it will not work in most cases. Do not worry if the leave exits with an error. The cluster is in an unhealthy state, so this is expected.

The peers.json file will be deleted after Dkron starts and ingests this file.

Using raft/peers.json for recovery can cause uncommitted Raft log entries to be implicitly committed, so this should only be used after an outage where no other option is available to recover a lost server. Make sure you don't have any automated processes that will put the peers file in place on a periodic basis.

The next step is to go to the data-dir of each Dkron server. Inside that directory, there will be a raft/ sub-directory. We need to create a raft/peers.json file. It should look something like:

json
[
  {
    "id": "node1",
    "address": "10.1.0.1:4647"
  },
  {
    "id": "node2",
    "address": "10.1.0.2:4647"
  },
  {
    "id": "node3",
    "address": "10.1.0.3:4647"
  }
]

Simply create entries for all remaining servers. You must confirm that servers you do not include here have indeed failed and will not later rejoin the cluster. Ensure that this file is the same across all remaining server nodes.

At this point, you can restart all the remaining servers. In Dkron 0.5.5 and later you will see them ingest recovery file:

text
Recovery log placeholder

It should be noted that any existing member can be used to rejoin the cluster as the gossip protocol will take care of discovering the server nodes.

At this point, the cluster should be in an operable state again. One of the nodes should claim leadership and emit a log like:

[INFO] Dkron: cluster leadership acquired

You can use the dkron raft list-peers command to inspect the Raft configuration:

text
$ dkron raft list-peers
Node   ID     Address          State     Voter
node1  node1  10.10.11.5:4647  follower  true
node2  node2  10.10.11.6:4647  leader    true
node3  node3  10.10.11.7:4647  follower  true

  • id (string: ) - Specifies the node ID of the server. This is the name of the node.

  • address (string: ) - Specifies the IP and port of the server in ip:port format. The port is the server's gRPC port used for cluster communications, typically 6868.

Usage Retries (`website/docs/usage/retries.md`)

title: Job retries

Jobs can be configured to retry in case of failure.

Configuration

json
{
  "name": "job1",
  "schedule": "@every 10s",
  "executor": "shell",
  "executor_config": {
    "command": "echo \"Hello from parent\""
  },
  "retries": 5
}

In case of failure to run the job in one node, it will try to run the job again in that node until the retries count reaches the limit.

Usage Storage (`website/docs/usage/storage.md`)

title: Embedded storage

Dkron has an embedded distributed KV store engine based on BuntDB. This works out of the box on each dkron server.

This ensures a dead easy install and setup, basically run dkron and you will have a full working node.

Usage Target Nodes Spec (`website/docs/usage/target-nodes-spec.md`)

title: Target nodes spec weight: 10

Target nodes spec

Default is for all nodes to execute each job. Dkron has the ability to run jobs in specific nodes by leveraging the use of tags. You can choose whether a job is run on a node or group of nodes by specifying tags and a count of target nodes having this tag do you want a job to run.

The target node syntax is:

text
"tag": "value[:count]"

To achieve this Nodes and Jobs have tags, for example, having a node with the following tags:

json
{
    "tags": {
        "dc": "dc1",
        "expect": "3",
        "port": "6868",
        "region": "global",
        "role": "dkron",
        "rpc_addr": "10.88.94.129:6868",
        "server": "true",
        "version": "devel",
        "my_role": "web"
    }
}

:::tip
You can specify tags for nodes in the dkron config file or in the command line using --tags parameter
:::

In case there is no matching nodes with the specified tags, the job will not run.

Following some examples using different tag combinations:

Target all nodes with a tag

json
{
    "name": "job_name",
    "command": "/bin/true",
    "schedule": "@every 2m",
    "tags": {
        "my_role": "web"
    }
}
mermaid
graph LR;
    J("Job tags: #quot;my_role#quot;: #quot;web#quot;") -->|Run Job|N1["Node1 tags: #quot;my_role#quot;: #quot;web#quot;"]
    J -->|Run Job|N2["Node2 tags: #quot;my_role#quot;: #quot;web#quot;"]
    J -->|Run Job|N3["Node2 tags: #quot;my_role#quot;: #quot;web#quot;"]

Target only one nodes of a group of nodes with a tag

json
{
    "name": "job_name",
    "command": "/bin/true",
    "schedule": "@every 2m",
    "tags": {
        "my_role": "web:1"
    }
}
mermaid
graph LR;
    J("Job tags: #quot;my_role#quot;: #quot;web:1#quot;") -->|Run Job|N1["Node1 tags: #quot;my_role#quot;: #quot;web#quot;"]
    J -.- N2["Node2 tags: #quot;my_role#quot;: #quot;web#quot;"]
    J -.- N3["Node2 tags: #quot;my_role#quot;: #quot;web#quot;"]

Dkron will try to run the job in the amount of nodes indicated by that count having that tag.

Details and limitations

Reserved tags

The region tag is reserved for multi-region setup purposes and should not be used for any other purpose.

Dkron will always run jobs in nodes with the same region tag.

You should use an alternative name.

Combinations

Tags specified in a Job are combined using AND, therefore a job that specifies several tags like:

json
{
    "tags": {
        "my_role": "web",
        "role": "dkron"
    }
}

Will try to run the job in nodes that have all speciefied tags.

Tags Limit

There is no limit in the tags that a job can have but having a Job with several tags with count like:

json
{
    "tags": {
        "my_role": "web:1",
        "role": "dkron:2"
    }
}

Will try to run the job in nodes that have all specified tags and using the lowest count. In the last example, it will run in one node having "my_role": "web" and "role": "dkron" tag, even if there is more than one node with these tags.

Usage Upgrade (`website/docs/usage/upgrade.md`)

title: Upgrade methods

Upgrade methods

Use the upgrade method that matches the kind of change you are making. In most cases, a rolling upgrade is the safest option because it lets you replace nodes gradually while keeping the cluster available. Use backup and restore when you are building a fresh cluster, migrating to new infrastructure, or when release notes require a full rebuild instead of mixed-version operation.

Before you start

Before upgrading any node:

  1. Read the release notes for the target version and check whether mixed-version clusters are supported during the transition.
  2. Make sure the current cluster is healthy and has quorum.
  3. Export the current jobs so you have a recovery point:
bash
curl -fsS http://localhost:8080/v1/jobs > backup.json
  1. Inspect the current Raft peers so you know which server is the leader and which peer IDs are registered:
bash
dkron raft list-peers

:::tip
When upgrading server nodes, it is usually best to leave the current leader for last. That reduces unnecessary leader elections while you rotate the rest of the cluster.
:::

Rolling upgrade

Use a rolling upgrade when you want to keep the cluster online and the target version supports a gradual transition.

  1. Upgrade agent-only nodes first.
  2. Upgrade follower server nodes one at a time.
  3. Upgrade the leader last.

Server rotation procedure

Use the following procedure to replace server nodes one at a time:

  1. Add a new server running the target version and configure it to join the existing cluster.
  2. Wait until the new server has joined successfully and the cluster is healthy.
  3. Stop Dkron on one old server.
  4. If that server was the leader, wait until a new leader is elected before continuing.
  5. List the current peers and identify the old server's peer ID:
bash
dkron raft list-peers
  1. Remove the old server from the Raft configuration:
bash
dkron raft remove-peer --peer-id <peer-id>
  1. Confirm the cluster is healthy again.
  2. Repeat the process until every old server has been replaced.

:::warning
Do not remove multiple server nodes at once. Dkron needs a healthy Raft quorum to continue scheduling jobs.
:::

Backup and restore

Use backup and restore when you need to recreate the cluster on new infrastructure or when a rolling upgrade is not appropriate.

Export jobs from the existing cluster

bash
curl -fsS http://localhost:8080/v1/jobs > backup.json

Restore jobs into the new cluster

After the new cluster is running and has elected a leader, restore the exported jobs file:

bash
curl -fsS -X POST http://localhost:8080/v1/restore \
  --form '[email protected]'

The restore endpoint expects a multipart form field named file. If a job in the file already exists in the target cluster, it is overwritten with the definition from the backup.

:::warning
This export and restore flow restores job definitions from the /v1/jobs payload. It should not be treated as a full cluster snapshot, and it does not recreate Raft state or execution history.
:::

After the upgrade

After either method completes:

  1. Run dkron raft list-peers and confirm the expected server set is present.
  2. Verify that one node is leader and the cluster remains stable.
  3. Check the UI or API and confirm the expected jobs are present.
  4. Watch the next scheduled executions to ensure jobs are still running as expected.
  5. Keep the exported backup.json until you are confident the upgrade is complete.

Pro Cli Dkron (`website/docs/pro/cli/dkron.md`)

date: 2024-10-06 title: "dkron" slug: dkron url: /docs/pro/cli/dkron/

dkron

Professional distributed job scheduling system

Synopsis

Dkron is a system service that runs scheduled jobs at given intervals or times,
just like the cron unix service but distributed in several machines in a cluster.
If a machine fails (the leader), a follower will take over and keep running the scheduled jobs without human intervention.

Options

text
--config string   config file (default is /etc/dkron/dkron.yml)
  -h, --help            help for dkron

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl (`website/docs/pro/cli/dkron_acl.md`)

date: 2024-10-06 title: "dkron acl" slug: dkron_acl url: /docs/pro/cli/dkron_acl/

dkron acl

Control ACL policies

Synopsis

Control ACL policies. This command allows to manage ACL policies.

Options

text
--cert-file string         Path to the client server TLS cert file
  -h, --help                     help for acl
      --key-file string          Path to the client server TLS key file
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Bootstrap (`website/docs/pro/cli/dkron_acl_bootstrap.md`)

date: 2024-10-06 title: "dkron acl bootstrap" slug: dkron_acl_bootstrap url: /docs/pro/cli/dkron_acl_bootstrap/

dkron acl bootstrap

Bootstrap ACL policies

Synopsis

Bootstrap ACL policies. This command allows to bootstrap ACL policies.

text
dkron acl bootstrap [flags]

Options

text
-h, --help   help for bootstrap

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Policy (`website/docs/pro/cli/dkron_acl_policy.md`)

date: 2024-10-06 title: "dkron acl policy" slug: dkron_acl_policy url: /docs/pro/cli/dkron_acl_policy/

dkron acl policy

Operate ACL policies

Synopsis

Operate ACL policies. This command allows to manage ACL policies.

Options

text
-h, --help   help for policy

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Policy Apply (`website/docs/pro/cli/dkron_acl_policy_apply.md`)

date: 2024-10-06 title: "dkron acl policy apply" slug: dkron_acl_policy_apply url: /docs/pro/cli/dkron_acl_policy_apply/

dkron acl policy apply

Apply ACL policies

Synopsis

Apply ACL policies. This command allows to apply ACL policies.

text
dkron acl policy apply [flags]

Options

text
--cert-file string         Path to the client server TLS cert file
  -h, --help                     help for apply
      --key-file string          Path to the client server TLS key file
  -n, --name string              Policy name
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
  -r, --rules-file string        Path to the file containing the policy rules
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Policy Delete (`website/docs/pro/cli/dkron_acl_policy_delete.md`)

date: 2024-10-06 title: "dkron acl policy delete" slug: dkron_acl_policy_delete url: /docs/pro/cli/dkron_acl_policy_delete/

dkron acl policy delete

Delete ACL policies

Synopsis

Delete ACL policies. This command allows to apply ACL policies.

text
dkron acl policy delete [flags]

Options

text
--cert-file string         Path to the client server TLS cert file
  -h, --help                     help for delete
      --key-file string          Path to the client server TLS key file
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Policy Info (`website/docs/pro/cli/dkron_acl_policy_info.md`)

date: 2024-10-06 title: "dkron acl policy info" slug: dkron_acl_policy_info url: /docs/pro/cli/dkron_acl_policy_info/

dkron acl policy info

Info ACL policy

Synopsis

Info ACL policy. Print policy details.

text
dkron acl policy info [flags]

Options

text
-h, --help   help for info

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Policy List (`website/docs/pro/cli/dkron_acl_policy_list.md`)

date: 2024-10-06 title: "dkron acl policy list" slug: dkron_acl_policy_list url: /docs/pro/cli/dkron_acl_policy_list/

dkron acl policy list

List ACL policies

Synopsis

List ACL policies. This command allows to list ACL policies.

text
dkron acl policy list [flags]

Options

text
-h, --help   help for list

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Token (`website/docs/pro/cli/dkron_acl_token.md`)

date: 2024-10-06 title: "dkron acl token" slug: dkron_acl_token url: /docs/pro/cli/dkron_acl_token/

dkron acl token

Operate ACL tokens

Synopsis

Operate ACL tokens. This command allows to manage ACL tokens.

Options

text
-h, --help   help for token

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Token Create (`website/docs/pro/cli/dkron_acl_token_create.md`)

date: 2024-10-06 title: "dkron acl token create" slug: dkron_acl_token_create url: /docs/pro/cli/dkron_acl_token_create/

dkron acl token create

Create a new ACL token

Synopsis

Create a new ACL token. This command allows to create a new ACL token.

text
dkron acl token create [flags]

Options

text
--cert-file string         Path to the client server TLS cert file
  -h, --help                     help for create
      --key-file string          Path to the client server TLS key file
  -n, --name string              Sets the human readable name for the ACL token.
  -p, --policy strings           Specifies a policy to associate with the token. Can be specified multiple times, but only with client type tokens. (default [default])
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file
  -t, --type string              Sets the type of token. Must be one of "client" (default), or "management". (default "client")

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Token Info (`website/docs/pro/cli/dkron_acl_token_info.md`)

date: 2024-10-06 title: "dkron acl token info" slug: dkron_acl_token_info url: /docs/pro/cli/dkron_acl_token_info/

dkron acl token info

Print ACL token details

Synopsis

Print ACL token details. This command allows to print ACL token details.

text
dkron acl token info [flags]

Options

text
-h, --help   help for info

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Acl Token List (`website/docs/pro/cli/dkron_acl_token_list.md`)

date: 2024-10-06 title: "dkron acl token list" slug: dkron_acl_token_list url: /docs/pro/cli/dkron_acl_token_list/

dkron acl token list

List ACL tokens

Synopsis

List ACL tokens. This command allows to list ACL tokens.

text
dkron acl token list [flags]

Options

text
-h, --help   help for list

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Agent (`website/docs/pro/cli/dkron_agent.md`)

date: 2024-10-06 title: "dkron agent" slug: dkron_agent url: /docs/pro/cli/dkron_agent/

dkron agent

Start a dkron agent

Synopsis

Start a dkron agent that schedule jobs, listen for executions and run executors.
It also runs a web UI.

text
dkron agent [flags]

Options

text
/* Detailed source-code truncated for AI context efficiency. */

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

  • dkron - Professional distributed job scheduling system

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Completion (`website/docs/pro/cli/dkron_completion.md`)

date: 2024-10-06 title: "dkron completion" slug: dkron_completion url: /docs/pro/cli/dkron_completion/

dkron completion

Generate the autocompletion script for the specified shell

Synopsis

Generate the autocompletion script for dkron for the specified shell.
See each sub-command's help for details on how to use the generated script.

Options

text
-h, --help   help for completion

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Completion Bash (`website/docs/pro/cli/dkron_completion_bash.md`)

date: 2024-10-06 title: "dkron completion bash" slug: dkron_completion_bash url: /docs/pro/cli/dkron_completion_bash/

dkron completion bash

Generate the autocompletion script for bash

Synopsis

Generate the autocompletion script for the bash shell.

This script depends on the 'bash-completion' package.
If it is not installed already, you can install it via your OS's package manager.

To load completions in your current shell session:

text
source <(dkron completion bash)

To load completions for every new session, execute once:

Linux:

text
dkron completion bash > /etc/bash_completion.d/dkron

macOS:

text
dkron completion bash > $(brew --prefix)/etc/bash_completion.d/dkron

You will need to start a new shell for this setup to take effect.

text
dkron completion bash

Options

text
-h, --help              help for bash
      --no-descriptions   disable completion descriptions

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Completion Fish (`website/docs/pro/cli/dkron_completion_fish.md`)

date: 2024-10-06 title: "dkron completion fish" slug: dkron_completion_fish url: /docs/pro/cli/dkron_completion_fish/

dkron completion fish

Generate the autocompletion script for fish

Synopsis

Generate the autocompletion script for the fish shell.

To load completions in your current shell session:

text
dkron completion fish | source

To load completions for every new session, execute once:

text
dkron completion fish > ~/.config/fish/completions/dkron.fish

You will need to start a new shell for this setup to take effect.

text
dkron completion fish [flags]

Options

text
-h, --help              help for fish
      --no-descriptions   disable completion descriptions

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Completion Powershell (`website/docs/pro/cli/dkron_completion_powershell.md`)

date: 2024-10-06 title: "dkron completion powershell" slug: dkron_completion_powershell url: /docs/pro/cli/dkron_completion_powershell/

dkron completion powershell

Generate the autocompletion script for powershell

Synopsis

Generate the autocompletion script for powershell.

To load completions in your current shell session:

text
dkron completion powershell | Out-String | Invoke-Expression

To load completions for every new session, add the output of the above command
to your powershell profile.

text
dkron completion powershell [flags]

Options

text
-h, --help              help for powershell
      --no-descriptions   disable completion descriptions

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Completion Zsh (`website/docs/pro/cli/dkron_completion_zsh.md`)

date: 2024-10-06 title: "dkron completion zsh" slug: dkron_completion_zsh url: /docs/pro/cli/dkron_completion_zsh/

dkron completion zsh

Generate the autocompletion script for zsh

Synopsis

Generate the autocompletion script for the zsh shell.

If shell completion is not already enabled in your environment you will need
to enable it. You can execute the following once:

text
echo "autoload -U compinit; compinit" >> ~/.zshrc

To load completions in your current shell session:

text
source <(dkron completion zsh)

To load completions for every new session, execute once:

Linux:

text
dkron completion zsh > "~/.zsh/completion/_dkron"

macOS:

text
dkron completion zsh > $(brew --prefix)/share/zsh/site-functions/_dkron

You will need to start a new shell for this setup to take effect.

text
dkron completion zsh [flags]

Options

text
-h, --help              help for zsh
      --no-descriptions   disable completion descriptions

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Doc (`website/docs/pro/cli/dkron_doc.md`)

date: 2024-10-06 title: "dkron doc" slug: dkron_doc url: /docs/pro/cli/dkron_doc/

dkron doc

Generate Markdown documentation for the Dkron CLI.

Synopsis

Generate Markdown documentation for the Dkron CLI.
This command is, mostly, used to create up-to-date documentation
of Dkron's command-line interface for http://dkron.io/.
It creates one Markdown file per command with front matter suitable
for rendering in Hugo.

text
dkron doc [flags]

Options

text
--dir string   the directory to write the doc. (default "/tmp/dkrondoc/")
  -h, --help         help for doc

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

  • dkron - Professional distributed job scheduling system

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Keygen (`website/docs/pro/cli/dkron_keygen.md`)

date: 2024-10-06 title: "dkron keygen" slug: dkron_keygen url: /docs/pro/cli/dkron_keygen/

dkron keygen

Generates a new encryption key

Synopsis

Generates a new encryption key that can be used to configure the
agent to encrypt traffic. The output of this command is already
in the proper format that the agent expects.

text
dkron keygen [flags]

Options

text
-h, --help   help for keygen

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

  • dkron - Professional distributed job scheduling system

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Leave (`website/docs/pro/cli/dkron_leave.md`)

date: 2024-10-06 title: "dkron leave" slug: dkron_leave url: /docs/pro/cli/dkron_leave/

dkron leave

Force an agent to leave the cluster

Synopsis

Stop stops an agent, if the agent is a server and is running for election
stop running for election, if this server was the leader
this will force the cluster to elect a new leader and start a new scheduler.
If this is a server and has the scheduler started stop it, ignoring if this server
was participating in leader election or not (local storage).
Then actually leave the cluster.

text
dkron leave [flags]

Options

text
--cert-file string         Path to the client server TLS cert file
  -h, --help                     help for leave
      --key-file string          Path to the client server TLS key file
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

  • dkron - Professional distributed job scheduling system

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Raft (`website/docs/pro/cli/dkron_raft.md`)

date: 2024-10-06 title: "dkron raft" slug: dkron_raft url: /docs/pro/cli/dkron_raft/

dkron raft

Command to perform some raft operations

Options

text
--cert-file string         Path to the client server TLS cert file
  -h, --help                     help for raft
      --key-file string          Path to the client server TLS key file
      --rpc-addr string          gRPC address of the agent (default "127.0.0.1:6868")
      --trusted-ca-file string   Path to the client server TLS trusted CA cert file

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Raft List Peers (`website/docs/pro/cli/dkron_raft_list-peers.md`)

date: 2024-10-06 title: "dkron raft list-peers" slug: dkron_raft_list-peers url: /docs/pro/cli/dkron_raft_list-peers/

dkron raft list-peers

Command to list raft peers

text
dkron raft list-peers [flags]

Options

text
-h, --help   help for list-peers

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

  • dkron raft - Command to perform some raft operations

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Raft Remove Peer (`website/docs/pro/cli/dkron_raft_remove-peer.md`)

date: 2024-10-06 title: "dkron raft remove-peer" slug: dkron_raft_remove-peer url: /docs/pro/cli/dkron_raft_remove-peer/

dkron raft remove-peer

Command to list raft peers

text
dkron raft remove-peer [flags]

Options

text
-h, --help             help for remove-peer
      --peer-id string   Remove a Dkron server with the given ID from the Raft configuration.

Options inherited from parent commands

text
--config string     config file (default is /etc/dkron/dkron.yml)
      --rpc-addr string   gRPC address of the agent (default "127.0.0.1:6868")

SEE ALSO

  • dkron raft - Command to perform some raft operations

Auto generated by spf13/cobra on 6-Oct-2024

Pro Cli Dkron Version (`website/docs/pro/cli/dkron_version.md`)

date: 2024-10-06 title: "dkron version" slug: dkron_version url: /docs/pro/cli/dkron_version/

dkron version

Show version

Synopsis

Show the version

text
dkron version [flags]

Options

text
-h, --help   help for version

Options inherited from parent commands

text
--config string   config file (default is /etc/dkron/dkron.yml)

SEE ALSO

  • dkron - Professional distributed job scheduling system

Auto generated by spf13/cobra on 6-Oct-2024

Pro Executors Docker (`website/docs/pro/executors/docker.md`)

Docker executor

Docker executor can launch docker based cron jobs using the docker command of the target node.

This executor needs a recent version of docker to be available and configured in the target node.

Configuration

To run a docker job create a job config with the docker executor as in this example:

json
{
  "executor": "docker",
  "executor_config": {
    "image": "alpine", //docker image to use
    "volumes": "/logs:/var/log/", //comma separated list of volume mappings
    "command": "echo \"Hello from dkron\"", //command to pass to run on container
    "env": "ENVIRONMENT=variable" //environment variables to pass to the container
  }
}

Pro Executors Ecs (`website/docs/pro/executors/ecs.md`)

AWS ECS Executor

The ECS exeutor is capable of launching tasks in ECS clusters, then listen to a stream of CloudWatch Logs and return the output.

To configure a job to be run in ECS, the executor needs a JSON Task definition template or an already defined task in ECS.

To allow the ECS Task runner to run tasks, the machine running Dkron needs to have the appropriate permissions configured in AWS IAM:

json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1460720941000",
            "Effect": "Allow",
            "Action": [
                "ecs:RunTask",
                "ecs:DescribeTasks",
                "ecs:DescribeTaskDefinition",
                "logs:FilterLogEvents",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

To configure a job to be run with the ECS executor:

Example using an existing taskdef

json
{
  "executor": "ecs",
  "executor_config": {
    "taskdefName": "mytaskdef-family",
    "region": "eu-west-1",
    "cluster": "default",
    "env": "ENVIRONMENT=variable",
    "service": "mycontainer",
    "overrides": "echo,\"Hello from dkron\"",
    "propagateTags": "TASK_DEFINITION"
  }
}

Example using a provided taskdef

json
{
  "executor": "ecs",
  "executor_config": {
    "taskdefBody": "{\"containerDefinitions\": [{\"essential\": true,\"image\": \"hello-world\",\"memory\": 100,\"name\": \"hello-world\"}],\"family\": \"helloworld\"}",
    "region": "eu-west-1",
    "cluster": "default",
    "fargate": "yes",
    "env": "ENVIRONMENT=variable",
    "maxAttempts": "5000"
  }
}

This is the complete list of configuration parameters of the plugin:

text
taskdefBody
taskdefName
region
cluster
logGroup
fargate
securityGroup
subnet
env
service
overrides
maxAttempts // Defaults to 2000, will perform a check every 6s * 2000 times waiting a total of 12000s or 3.3h
enableExecuteCommand

Pro Processors Elasticsearch (`website/docs/pro/processors/elasticsearch.md`)

Elasticsearch processor

The Elasticsearch processor can fordward execution logs to an ES cluster. It need an already available Elasticsearch installation that is visible in the same network of the target node.

The output logs of the job execution will be stored in the indicated ES instace.

:::info For Dkron Pro < v3.2.3
:::

Configuration

json
{
  "processors": {
    "elasticsearch": {
      "url": "http://localhost:9200", //comma separated list of Elasticsearch hosts urls (default: http://localhost:9200)
      "index": "dkron_logs", //desired index name (default: dkron_logs)
      "forward": "false" //forward logs to the next processor (default: false)
    }
  }
}

:::info For Dkron Pro > v3.2.3
:::

Configuration

For increased security and flexibility, configuration of the ES processor is stored in a file named dkron-processor-elasticsearch.yml in the same locations as dkron.yml, and should include a list of configurations for Elasticsearch, it can include any number of configurations.

This is an example including all available parameters:

yaml
es1:
  index: dkron-logs
  index_date_format: '2006-01-02'
  username: elastic
  password: XXXXXXXXXXXXX
  url: https://localhost:9200

And for each job you only need to configure the config parameter in the processors configuration:

json
{
  "processors": {
    "elasticsearch": {
      "config": "es1", // configuration to use from the config file
      "forward": "false" // forward logs to the next processor (default: false)
    }
  }
}

Pro Processors Email (`website/docs/pro/processors/email.md`)

Email processor

The Email processor provides flexibility to job email notifications.

Configuration of the email processor is stored in a file named dkron-processor-email.yml in the same locations as dkron.yml, and should include a list of providers, it can include any number of providers.

Example:

yaml
provider1:
  host: smtp.myprovider.com
  port: 25
  username: myusername
  password: mypassword
  from: [email protected]
  subjectPrefix: '[Staging] '

Then configure each job with the following options:

Example:

json
{
  "processors": {
    "email": {
      "provider": "provider1",
      "emails": "[email protected], [email protected]",
      "onSuccess": "true"
    }
  }
}

By default the email procesor doesn't send emails on job success, the onSuccess parameter, enables it, like in the previous example.

Pro Processors Slack (`website/docs/pro/processors/slack.md`)

Slack processor

The Slack processor provides slack notifications with multiple configurations and rich format.

Configuration of the slack processor is stored in a file named dkron-processor-slack.yml in the same locations as dkron.yml, and should include a list of teams, it can include any number of teams.

Example:

yaml
team1:
  webhook_url: https://hooks.slack.com/services/XXXXXXXXXXXXXXXXXXX
  bot_name: Dkron Production

Then configure each job with the following options:

Example:

json
{
  "processors": {
    "slack": {
      "team": "team1",
      "channel": "#cron-production",
      "onSuccess": "true"
    }
  }
}

By default the slack procesor doesn't send notifications on job success, the onSuccess parameter, enables it, like in the previous example.

Usage Executors Gcp Pubsub (`website/docs/usage/executors/gcp-pubsub.md`)

Google Cloud PubSub Executor

A basic GC PubSub executor that produces a message on a Google Cloud PubSub topic.

Configuration

Params

text
project:                     Google Cloud project ID. Required
topic:                       The topic name for the message. Required
data:                        The actual message body in base64 format. Required
attributes:                  The attributes of the message in JSON format. Optional

Example:

json
{
  "executor": "gcppubsub",
  "executor_config": {
    "project": "project-id",
    "topic": "topic-name",
    "data": "aGVsbG8gd29ybGQ=",
    "attributes": "{\"hello\":\"world\",\"waka\":\"paka\"}"
  }
}

Usage Executors Grpc (`website/docs/usage/executors/grpc.md`)

GRPC Executor

GRPC executor can send a request to a GRPC Server

Requirements

In order to serialize protobufs, the server needs to have the reflection service active.
Without that we cannot get proto descriptors required for serialization.

Configuration

Params:

text
"url": Required, Request url
"body": Optional, POST body
"timeout": Optional, Request timeout, unit seconds
"expectCode": Optional, One of https://grpc.github.io/grpc/core/md_doc_statuscodes.html

Example:

json
{
  "executor": "http",
  "executor_config": {
    "url": "127.0.0.1:9000/test.TestService/Test",
    "body": "{\"key\": \"value\"}",
    "timeout": "30",
    "expectCode": "0"
  }
}

Usage Executors Http (`website/docs/usage/executors/http.md`)

HTTP Executor

HTTP executor can send a request to an HTTP endpoint

Configuration

Params:

text
method:     Request method in uppercase
url:        Request url
headers:    Json string, such as "[\"Content-Type: application/json\"]"
body:       POST body
timeout:    Request timeout, unit seconds
expectCode: Expect response code, such as 200,206
expectBody: Expect response body, support regexp, such as /success/
debug:      Debug option, will log everything when this option is not empty
tlsNoVerifyPeer:       false (default) or true. If true, disables verification of the remote SSL certificate's validity.
tlsCertificateFile:    Path to the PEM file containing the client certificate. Optional.
tlsCertificateKeyFile: Path to the PEM file containing the client certificate private key. Optional.
tlsRootCAsFile:        Path to the PEM file containing certificates to use as root CAs. Optional.

Example:

json
{
  "executor": "http",
  "executor_config": {
    "method": "GET",
    "url": "http://example.com",
    "headers": "[]",
    "body": "",
    "timeout": "30",
    "expectCode": "200",
    "expectBody": "",
    "debug": "true"
  }
}

Usage Executors Kafka (`website/docs/usage/executors/kafka.md`)

Kafka Executor

A basic Kafka executor that produces a message on a Kafka broker.

Configuration

Params:

text
brokerAddress:          Comma separated string containing "IP:port" of the brokers
key:                    The key of the message to produce
message:                The body of the message to produce
topic:                  The Kafka topic for this message
tlsEnable:              Enables TLS if set to true. Optional
tlsInsecureSkipVerify:  Disables verification of the remote SSL certificate's validity if set to true. Optional
saslUsername:           The SASL username for authentication. If set, saslPassword and saslMechanism must also be provided.
saslPassword:           The SASL password for authentication. If set, saslUsername and saslMechanism must also be provided.
saslMechanism:          The SASL SCRAM mechanism to use, either "sha256" or "sha512". This is required if both saslUsername and saslPassword are provided.
debug:                  Turns on debugging output if not empty

Example:

json
{
  "executor": "kafka",
  "executor_config": {
    "brokerAddress": "localhost:9092,another.host:9092",
    "key": "My key",
    "message": "My message",
    "topic": "my_topic"
  }
}

Usage Executors Nats (`website/docs/usage/executors/nats.md`)

NATS Executor

The NATS executor sends a message to a NATS server/cluster.

Currently, only username/password authentication is supported.

Configuration

Params:

text
url:      Comma separated list of NATS server URLs
message:  The message to send
subject:  The subject to send the message to
userName: username for authentication
password: password for authentication
debug:    If not empty, turns on debugging. Will log the NATS specific job config and the request sent.

Example:

json
{
  "executor": "nats",
  "executor_config": {
    "url": "tls://nats.demo.io:4443",
    "message": "the message",
    "subject": "myfavoritesubject",
    "userName": "someusername",
    "password": "somepassword"
  }
}

Usage Executors Rabbitmq (`website/docs/usage/executors/rabbitmq.md`)

RabbitMQ Executor

A basic RabbitMQ executor that produces a message on a RabbitMQ server/cluster.

Configuration

Params

text
url:                         URL of the RabbitMQ server/cluster with VHost. Example: amqp://guest:guest@localhost:5672/
queue.name:                  The name of the queue to publish the message to. Required
queue.durable:               Whether the queue is durable or not. Optional, defaults to false
queue.auto_delete:           Whether the queue is auto-deleted after publishing the message. Optional, defaults to false
queue.exclusive:             Whether the queue is exclusive or not. Optional, defaults to false
message.content_type:        The content type of the message
message.delivery_mode:       Message delivery mode. 1 for non-persistent, 2 for persistent. Optional, defaults to 0
message.messageId:           The message ID
message.body:                The actual message body in desired format that will be sent to the queue
message.base64:              Encoded message body in base64 format. Optional, but should not be set if message.body is set.
exchange:                    The RabbitMQ exchange. Optional, default to ""

Example:

json
{
  "executor": "rabbitmq",
  "executor_config": {
    "url": "amqp://guest:guest@localhost:5672/",
    "exchange": "amq.default",
    "queue.name": "test",
    "queue.create": "true",
    "queue.durable": "true",
    "queue.auto_delete": "false",
    "queue.exclusive": "false",
    "message.content_type": "application/json",
    "message.delivery_mode": "2",
    "message.messageId": "4373732772",
    "message.body": "{\"key\":\"value\"}"
  }
}

Usage Executors Shell (`website/docs/usage/executors/shell.md`)

Shell Executor

The Shell executor is one of the most versatile executors in Dkron, allowing you to run any system command or script on target nodes.

Overview

The shell executor runs commands on the target node's operating system. It can:

  • Execute system commands
  • Run scripts in various languages (bash, python, etc.)
  • Perform file system operations
  • Interact with local services and processes

Configuration Parameters

Parameter Required Description
shell No When "true", runs the command in a shell environment (bash/cmd). Default: "false"
command Yes The command or script to execute
env No Environment variables in the format "KEY1=value1,KEY2=value2"
cwd No The working directory to run the command from
su No Unix only. Run the command as a different user or user:group
timeout No Maximum execution time after which the job is forcefully terminated
mem_limit No Maximum memory usage after which the job is forcefully terminated. Supports units: B, KB, MB, GB, TB

Basic Usage Examples

Simple Command Execution

json
{
  "executor": "shell",
  "executor_config": {
    "command": "echo Hello Dkron"
  }
}

Using Shell Features

json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "ps aux | grep nginx | wc -l"
  }
}

Setting Environment Variables

json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "python /scripts/process_data.py",
    "env": "DATA_PATH=/data/incoming,LOG_LEVEL=info,API_KEY=secret123"
  }
}

Changing Working Directory

json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "./run_backup.sh",
    "cwd": "/opt/backups"
  }
}

Setting a Timeout

json
{
  "executor": "shell",
  "executor_config": {
    "command": "/opt/scripts/long_running_task.sh",
    "timeout": "1h30m"
  }
}

Setting a Memory Limit

json
{
  "executor": "shell",
  "executor_config": {
    "command": "/opt/scripts/memory_intensive_task.sh",
    "mem_limit": "512MB"
  }
}

Combining Timeout and Memory Limit

json
{
  "executor": "shell",
  "executor_config": {
    "command": "/opt/scripts/resource_intensive_task.sh",
    "timeout": "30m",
    "mem_limit": "1GB"
  }
}

Advanced Examples

Running a Multi-line Script

json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "#!/bin/bash\necho 'Starting job'\ncd /tmp\ndate > last_run.txt\necho 'Job completed'"
  }
}

Database Backup Example

json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "pg_dump -U postgres -d mydb | gzip > /backups/mydb_$(date +%Y%m%d_%H%M%S).sql.gz",
    "env": "PGPASSWORD=securepassword",
    "timeout": "30m"
  }
}

System Health Check with Exit Code

json
{
  "executor": "shell",
  "executor_config": {
    "shell": "true",
    "command": "if [ $(df -h | grep '/dev/sda1' | awk '{print $5}' | tr -d '%') -gt 90 ]; then echo 'Disk space critical'; exit 1; else echo 'Disk space OK'; fi"
  }
}

Monitoring and Metrics

The shell executor exposes Prometheus metrics on port 9422 (configurable with SHELL_EXECUTOR_PROMETHEUS_PORT environment variable):

Metric Type Description Labels
dkron_job_cpu_usage gauge Current CPU usage by job job_name
dkron_job_mem_usage_kb gauge Current memory consumed by job (KB) job_name

Exit Codes and Success/Failure

By default, an exit code of 0 indicates success, while any non-zero exit code indicates failure. You can customize which exit codes are considered successful in the job definition:

json
{
  "success_count": "0,1,2",  // Exit codes 0, 1, and 2 will be considered successful
  "executor": "shell",
  "executor_config": {
    "command": "/opt/scripts/check.sh"
  }
}

Security Considerations

The shell executor runs commands with the same permissions as the Dkron process. Consider the following security best practices:

  1. Principle of Least Privilege: Run Dkron with a dedicated user that has only the permissions required for its jobs
  2. Avoid Sensitive Data in Commands: Use environment variables for sensitive values instead of embedding them in commands
  3. Input Validation: Validate any dynamic parts of commands, especially if they come from external sources
  4. Output Sanitization: Be cautious when using command output, particularly if it's displayed in the UI or logs

Troubleshooting

Common issues and solutions when working with the shell executor:

Command Not Found

If you see "command not found" errors:

  • Verify the command is installed on the target node
  • Use absolute paths to executables (e.g., /usr/bin/python instead of just python)
  • Check if the command is in the PATH of the user running Dkron

Permission Issues

If you encounter permission errors:

  • Check if the Dkron user has the necessary permissions
  • Adjust file/directory permissions as needed
  • Consider using sudo if appropriate (requires sudo configuration)

Timeouts

If jobs are timing out:

  • Adjust the timeout parameter to match the expected runtime
  • Optimize the command to run more efficiently
  • Consider breaking large jobs into smaller, chained jobs

Memory Limits

If jobs are being killed due to memory limits:

  • Adjust the mem_limit parameter to accommodate the job's memory requirements
  • Optimize the command to use less memory (e.g., process data in chunks)
  • Monitor memory usage patterns to set appropriate limits
  • Consider if the job should run on a node with more available memory

Environment Variables

If environment variables aren't working:

  • Ensure the format is correct (KEY=value,KEY2=value2)
  • Remember that values are passed as strings
  • For complex values with spaces or special characters, consider using files instead

Additional Resources

Usage Plugins Develop (`website/docs/usage/plugins/develop.md`)

title: Developing plugins weight: 99

Developing a Plugin

:::warning Advanced topic!
Plugin development is a highly advanced topic, and is not required knowledge for day-to-day usage. If you don't plan on writing any plugins, we recommend not reading the following section of the documentation.
:::

Developing a plugin is simple. The only knowledge necessary to write a plugin is basic command-line skills and basic knowledge of the Go programming language.

Note: A common pitfall is not properly setting up a $GOPATH. This can lead to strange errors. You can read more about this here to familiarize yourself.

Create a new Go project somewhere in your $GOPATH. If you're a GitHub user, we recommend creating the project in the directory $GOPATH/src/github.com/USERNAME/dkron-NAME-TYPE, where USERNAME is your GitHub username and NAME is the name of the plugin you're developing. This structure is what Go expects and simplifies things down the road.

With the directory made, create a main.go file. This project will be a binary so the package is "main":

go
package main

import (
	"github.com/distribworks/dkron/v3/plugin"
)

func main() {
	plugin.Serve(&plugin.ServeOpts{
		Processor: new(MyPlugin),
	})
}

And that's basically it! You'll have to change the argument given to plugin.Serve to be your actual plugin, but that is the only change you'll have to make. The argument should be a structure implementing one of the plugin interfaces (depending on what sort of plugin you're creating).

Dkron plugins must follow a very specific naming convention of dkron-TYPE-NAME. For example, dkron-processor-files, which tells Dkron that the plugin is a processor that can be referenced as "files".

Usage Processors File (`website/docs/usage/processors/file.md`)

title: File Processor

File processor saves the execution output to a single log file in the specified directory

Configuration

Parameters

text
log_dir: Path to the location where the log files will be saved
forward: Forward log output to the next processor

Example

json
{
    "name": "job_name",
    "command": "echo 'Hello files'",
    "schedule": "@every 2m",
    "tags": {
        "role": "web"
    },
    "processors": {
        "files": {
            "log_dir": "/var/log/mydir",
            "forward": "true"
        }
    }
}

Usage Processors Log (`website/docs/usage/processors/log.md`)

title: Log Processor

Log processor writes the execution output to stdout/stderr

Configuration

Parameters

forward: Forward the output to the next processor

Example

json
{
    "name": "job_name",
    "command": "echo 'Hello log'",
    "schedule": "@every 2m",
    "tags": {
        "role": "web"
    },
    "processors": {
        "log": {
            "forward": "true"
        }
    }
}

Usage Processors Syslog (`website/docs/usage/processors/syslog.md`)

title: Syslog Processor

Syslog processor writes the execution output to the system syslog daemon.

Note: Only works on linux systems

Configuration

Parameters

forward: Forward the output to the next processor

Example

json
{
  "name": "job_name",
  "command": "echo 'Hello syslog'",
  "schedule": "@every 2m",
  "tags": {
    "role": "web"
  },
  "processors": {
    "syslog": {
      "forward": "true"
    }
  }
}

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#cron Browse cron catalog → #distributed-systems Browse distributed-systems catalog → #fault-tolerance Browse fault-tolerance catalog → #scheduled-jobs Browse scheduled-jobs catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →