st2 Documentation

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and...

6,550 stars Python Python 13 files · ~4,574 tokens #auto-remediation#automation#chatops#cicd#deployment#devops#ifttt#python
RAW Doc

Technical Documentation: StackStorm/st2

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

StackStorm is a platform for integration and automation across services and tools, taking actions in response to events. Learn more at www.stackstorm.com.









TL;DR

StackStorm Overview

About

StackStorm is a platform for integration and automation across services and tools. It ties together your existing infrastructure and application environment so you can more easily automate that environment -- with a particular focus on taking actions in response to events.

StackStorm helps automate common operational patterns. Some examples are:

  • Facilitated Troubleshooting - triggering on system failures captured by Nagios, Sensu, New Relic and other monitoring, running a series of diagnostic checks on physical nodes, OpenStack or Amazon instances, and application components, and posting results to a shared communication context, like Slack or JIRA.
  • Automated remediation - identifying and verifying hardware failure on OpenStack compute node, properly evacuating instances and emailing VM about potential downtime, but if anything goes wrong - freezing the workflow and calling PagerDuty to wake up a human.
  • Continuous deployment - build and test with Jenkins, provision a new AWS cluster, turn on some traffic with the load balancer, and roll-forth or roll-back based on NewRelic app performance data.

StackStorm helps you compose these and other operational patterns as rules and workflows or actions; and these rules and workflows - the content within the StackStorm platform - are stored as code which means they support the same approach to collaboration that you use today for code development and can be shared with the broader open source community via StackStorm Exchange.

Who is using StackStorm?

See the list of known StackStorm ADOPTERS.md and Thought Leaders.

How it works

StackStorm architecture

StackStorm plugs into the environment via an extensible set of adapters: sensors and actions.

  • Sensors are Python plugins for inbound integration that watch for events from external systems and fire a StackStorm trigger when an event happens.

  • Triggers are StackStorm representations of external events. There are generic triggers (e.g., timers, webhooks) and integration triggers (e.g., Sensu alert, JIRA issue updated). A new trigger type can be defined by writing a sensor plugin.

  • Actions are StackStorm outbound integrations. There are generic actions (SSH, HTTP request), integrations (OpenStack, Docker, Puppet), or custom actions. Actions are either Python plugins, or any scripts, consumed into StackStorm by adding a few lines of metadata. Actions can be invoked directly by user via CLI, API, or the web UI, or used and called as part of automations - rules and workflows.

  • Rules map triggers to actions (or to workflows), applying matching criterias and map trigger payload data to action inputs.

  • Workflows stitch actions together into "uber-actions", defining the order, transition conditions, and passing context data from one action to the next. Most automations are multi-step (eg: more than one action). Workflows, just like "atomic" actions, are available in the action library, and can be invoked manually or triggered by rules.

  • Packs are the units of content deployment. They simplify the management and sharing of StackStorm pluggable content by grouping integrations (triggers and actions) and automations (rules and workflows). A growing number of packs is available on the StackStorm Exchange. Users can create their own packs, share them on GitHub, or submit them to the StackStorm Exchange organization.

  • Audit trail is the historical list of action executions, manual or automated, and is recorded and stored with full details of triggering context and execution results. It is also captured in audit logs for integrating with external logging and analytical tools: LogStash, Splunk, statsd, or syslog.

StackStorm is a service with modular architecture. It is comprised of loosely coupled microservice components that communicate over a message bus, and scales horizontally to deliver automation at scale. StackStorm has a full REST API, CLI client, and web UI for admins and users to operate it locally or remotely, as well as Python client bindings for developer convenience.

StackStorm is an established project and remains actively developed by a broad community.

Documentation

Additional documentation, including installation procedures, action/rule/workflow authoring, and how to setup and use triggers/sensors can be found at https://docs.stackstorm.com.

Hacking / Contributing

To set up a development environment and run StackStorm from sources, follow these instructions.

For information on how to contribute, our style guide, coding conventions and more,
please visit the Development section
in our documentation.

Security

If you believe you found a security issue or a vulnerability, please send a description of it to
our private mailing list at info [at] stackstorm [dot] com.

Once you've submitted an issue, you should receive an acknowledgment from one our of team members
in 48 hours or less. If further action is necessary, you may receive additional follow-up emails.

For more information, please refer to https://docs.stackstorm.com/latest/security.html

Copyright 2020 The StackStorm Authors.
Copyright 2019 Extreme Networks, Inc.
Copyright 2014-2018 StackStorm, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

http://www.apache.org/licenses/LICENSE-2.0

By contributing you agree that these contributions are your own (or approved by your employer) and you grant a full, complete, irrevocable copyright license to all users and developers of the project, present and future, pursuant to the license of the project.

Overview (`contrib/README.md`)

Contrib

  1. St2 contents that will be shipped with the product. These contents will be copied over to /opt/stackstorm on install and then auto-registered. Users are expected to be able to edit these contents post install:

    • st2/contrib/core
    • st2/contrib/packs
    • st2/contrib/linux
    • st2/contrib/chatops

  2. St2 samples that may be of interest to the users. These samples will be shipped with the product but will not be auto-registered.

    • st2/contrib/examples
    • st2/contrib/tests
    • st2/contrib/hello_st2

See st2/st2common/packaging to figure where and how it is all installed.

Overview (`lint-configs/README.md`)

StackStorm Lint Configs

This repository contains lint configs for different programming languages and
tools (flake8, pylint, etc.) used by different StackStorm repositories.

Configs are grouped in sub-directories by programming language.

Usage

To use those configs, add this repository as a git subtree to the repository
where you want to utilize those configs. After that is done, update make
targets (or similar) to correctly pass path to the configs to the tools
in question.

bash
git subtree add --prefix lint-configs https://github.com/StackStorm/lint-configs.git master --squash

To use it (example with pylint)

bash
pylint -E --rcfile=./lint-configs/python/.pylintrc
...

And once you want to pull changes / updates from the lint-configs repository:

bash
git subtree pull --prefix lint-configs https://github.com/StackStorm/lint-configs.git master --squash

Overview (`pants-plugins/README.md`)

pants plugins

This directory contains StackStorm-specific plugins for pantsbuild.

pants should be the primary entry point for development related tasks.
This replaces the Makefile and related scripts such that they are more discoverable.
The plugins here add custom goals or other logic into pants.

To see available goals, do "pants help goals" and "pants help $goal".

These plugins might be useful outside of the StackStorm project:

  • uses_services

These StackStorm-specific plugins might be useful in other StackStorm-related repos.

  • pack_metadata

These StackStorm-specific plugins are probably only useful for the st2 repo.

  • api_spec
  • macros.py (not a plugin - see pants.toml [GLOBAL].build_file_prelude_globs)
  • release
  • sample_conf
  • schemas

`api_spec` plugin

This plugin wires up pants to make sure st2common/st2common/openapi.yaml
gets regenerated if needed. Now, whenever someone runs the fmt goal
(eg pants fmt st2common/st2common/openapi.yaml), the api spec will
be regenerated if any of the files used to generate it has changed.
Also, running the lint goal will fail if the schemas need to be
regenerated.

This plugin also wires up pants so that the lint goal runs additional
api spec validation on st2common/st2common/openapi.yaml with something
like pants lint st2common/st2common/openapi.yaml.

`macros.py` macros

Macros are a pants feature
that can reduce "boilerplate"/duplication in BUILD files. The functions
defined in macros.py are available in all the BUILD files, and using
them looks just like using the normal BUILD targets.

For documentation about our macros, please refer to the function docstrings
in the macros.py file.

`pack_metadata` plugin

This plugin adds two new targets to pants:

  • pack_metadata
  • pack_metadata_in_git_submodule

These targets include all StackStorm pack metadata files in a pack.
Pack metadata includes top-level files (pack.yaml, .yaml.example,
config.schema.yaml, and icon.png) and metadata (*.yaml, *.yml)
for actions, action-aliases, policies, rules, and sensors.

This plugin also wires up the tailor goal, so that it will add a
pack_metadata(name="metadata") target wherever it finds a pack.yaml file.

One of the packs in this repo is in a git submodule to test our handling
of git submodules (st2tests/st2tests/fixtures/packs/test_content_version).
If it is not checked out, then some of the tests will fail.
If it is not checked out, pack_metadata_in_git_submodule handles providing
a helpful, instructive error message as early as possible.

`release` plugin

This plugin implements the SetupKwargs
plugin hook that pants uses when it auto-generates a setup.py file whenever
it builds a python_distribution() (ie a wheel or an sdist). This makes it
easy to centralize all of the common bits of metadata that need to go in all
the wheels (like author="StackStorm" or our project_urls).

`sample_conf` plugin

This plugin wires up pants to make sure conf/st2.conf.sample gets
regenerated whenever the source files change. Now, whenever someone runs
the fmt goal (eg pants fmt conf/st2.conf.sample), the sample will
be regenerated if any of the files used to generate it have changed.
Also, running the lint goal will fail if the sample needs to be
regenerated.

`schemas` plugin

This plugin wires up pants to make sure contrib/schemas/*.json gets
regenerated whenever the source files change. Now, whenever someone runs
the fmt goal (eg pants fmt contrib/schemas::), the schemas will
be regenerated if any of the files used to generate them have changed.
Also, running the lint goal will fail if the schemas need to be
regenerated.

`uses_services` plugin

This plugin validates that services are running if required. For example, some tests
need mongo, so this plugin can ensure mongo is running. If it is not running, then
an error with instructions on how to run it are given to the user.

uses_services has some StackStorm-specific assumptions in it, but it might be
generalizable. There are several other StackStorm-specific plugins, but some of
them are only useful in the st2 repo.

Overview (`contrib/chatops/README.md`)

Chatops Integration Pack

Pack that provides management/integration with Chatops. It is intended to replace Hubot pack eventually.

Actions

  • post_message - Post raw data to stream for chatops consumers to pick up
  • post_result - Send JSON formatted action output via hubot-stackstorm adapter
  • format_execution_result - Transforming trigger into a message for chatops client to show to user

Obligatory kudos to https://icons8.com/ for the icon.

Overview (`contrib/linux/README.md`)

Linux Integration Pack

This pack contains actions for commonly used Linux commands and tools.

Sensors

FileWatchSensor

This sensor monitors files that are declared by a rule (one file per rule). Once a new line is
detected, a trigger is emitted.

Example Rule:

This example rule will register with the FileWatchSensor to watch the file /tmp/st2_test. When a new line is
detected, the trigger will be emitted, and the action core.local will echo the trigger data.

text
---
name: sample_rule_file_watch
pack: "examples"
description: Sample rule custom trigger type - add a file to be watched by file_watch_sensor in linux pack.
enabled: true

trigger:
  parameters:
    file_path: /tmp/st2_test
  type: linux.file_watch.line

criteria: {}

action:
  parameters:
    cmd: echo "{{trigger}}"
  ref: core.local

linux.file_watch.line trigger

Example trigger payload:

json
{
    "file_path": "/var/log/auth.log",
    "file_name": "auth.log",
    "line": "Jan 18 13:38:15 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/3 ; PWD=/data/stanley ; USER=root ; COMMAND=/bin/ls"
}

Actions

  • vmstat - Wrapper around the vmstat command.
  • rsync - Wrapper around the rsync command.
  • netstat - Wrapper around the netstat command.
  • lsof - Wrapper around the lsof command.
  • service - Action which allows you to perform an action (start, stop,
    restart, etc.) on a system service. Currently it supports the following
    distributions: Ubuntu / Debian (upstart, sys init), RedHat / Fedora
    (systemd).
  • touch - Action which touches a file.
  • check_loadavg - Action which retrieves load average from a remote host.
  • check_processes - Action which retrieves useful information about
    matching process on a remote host.
  • dig - Python wrapper for dig command. Requires bind-utils rpm or dnsutils deb installed.

Troubleshooting

  • On CentOS7/RHEL7, dig is not installed by default. Run sudo yum install bind-utils to install.
  • On CentOS8/RHEL8, lsof is not installed by default. Run sudo yum install lsof to install.

Overview (`contrib/schemas/README.md`)

JSON schema for pack contents

The JSON files under this directory are auto-generated by st2-generate-schemas. To update the
schema files, run make schemasgen. Please do not edit these files manually.

The purpose of these schema files are used for validation of pack contents during pack development.
Development tools (i.e. VSCode extension) will need to be developed to take advantage of these
schema files.

Overview (`scripts/ci/README.md`)

CI Scripts

This directory contains various scripts used in CI/CD system.

Scripts in this directory are CI system agnostic.

Overview (`scripts/github/README.md`)

Github Actions Scripts

This directory contains various scripts which are used to run tests and other checks on Github
Actions CI.

As such, files in this directory and Github Actions specified.

Overview (`contrib/default/actions/README.md`)

actions

The actions folder contains action scripts and metadata files. See Actions
and Workflows for specifics on writing actions. Note that the lib sub-folder
is always available for access for an action script.

Overview (`contrib/default/rules/README.md`)

rules

The rules folder contains rules. See Rules for specifics on writing rules.

Overview (`contrib/default/sensors/README.md`)

sensors

The sensors folder contains sensors. See Sensors for specifics on writing
sensors and registering TriggerTypes.

Overview (`contrib/default/actions/workflows/README.md`)

actions/workflows

The workflows folder contains workflow files. See Workflows for specifics on writing workflows.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#auto-remediation Browse auto-remediation catalog → #automation Browse automation catalog → #chatops Browse chatops catalog → #cicd Browse cicd catalog → #deployment Browse deployment catalog → #devops Browse devops catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →