invoke Documentation

Pythonic task management & command execution.

4,780 stars Python Python 25 files Β· ~24,577 tokens
RAW Doc

Technical Documentation: pyinvoke/invoke

ℹ️ Provenance: Multi-Tier Ingestion Β· CodeWiki Reference

Sites Docs .Readthedocs.Yaml (`sites/docs/.readthedocs.yaml`)

version: 2

build:
os: "ubuntu-24.04"
tools:
python: "3.9"
jobs:
install:
- pip install pip~=25.1
- pip install . --group dev

sphinx:
configuration: sites/docs/conf.py
fail_on_warning: true


Sites Docs Getting Started (`sites/docs/getting-started.rst`)

=============== Getting started

This document presents a whirlwind tour of Invoke's feature set. Please see the
links throughout for detailed conceptual & API docs. For installation help, see
the project's installation page <https://www.pyinvoke.org/installing.html>_.

.. _defining-and-running-task-functions:

Defining and running task functions

The core use case for Invoke is setting up a collection of task functions and
executing them. This is pretty easy -- all you need is to make a file called
tasks.py importing the .task decorator and decorating one or more
functions. You will also need to add an arbitrarily-named context argument
(convention is to use c, ctx or context) as the first positional
arg. Don't worry about using this context parameter yet.

Let's start with a dummy Sphinx docs building task::

text
from invoke import task

@task
def build(c):
    print("Building!")

You can then execute that new task by telling Invoke's command line runner,
invoke, that you want it to run::

text
$ invoke build
Building!

The function body can be any Python you want -- anything at all.

Task parameters

Functions can have arguments, and thus so can tasks. By default, your task
functions' args/kwargs are mapped automatically to both long and short CLI
flags, as per :ref:the CLI docs <task-arguments>. For example, if we add a
clean argument and give it a boolean default, it will show up as a set of
toggle flags, --clean and -c::

text
@task
def build(c, clean=False):
    if clean:
        print("Cleaning!")
    print("Building!")

Invocations::

text
$ invoke build -c
$ invoke build --clean

Naturally, other default argument values will allow giving string or integer
values. Arguments with no default values are assumed to take strings, and can
also be given as positional arguments. Take this incredibly contrived snippet
for example::

text
@task
def hi(c, name):
    print("Hi {}!".format(name))

It can be invoked in the following ways, all resulting in "Hi Name!"::

text
$ invoke hi Name
$ invoke hi --name Name
$ invoke hi --name=Name
$ invoke hi -n Name
$ invoke hi -nName

Adding metadata via `@task <.task>`

@task <.task> can be used without any arguments, as above, but it's also a
convenient vector for additional metadata about the task function it decorates.
One common example is describing the task's arguments, via the help
parameter (in addition to optionally giving task-level help via the
docstring)::

text
@task(help={'name': "Name of the person to say hi to."})
def hi(c, name):
    """
    Say hi to someone.
    """
    print("Hi {}!".format(name))

This description will show up when invoking --help::

text
$ invoke --help hi
Usage: inv[oke] [--core-opts] hi [--options] [other tasks here ...]

Docstring:
  Say hi to someone.

Options:
  -n STRING, --name=STRING   Name of the person to say hi to.

More details on task parameterization and metadata can be found in
:doc:/concepts/invoking-tasks (for the command-line & parsing side of things)
and in the .task API documentation (for the declaration side).

Listing tasks

You'll sometimes want to see what tasks are available in a given
tasks.py -- invoke can be told to list them instead of executing
something::

text
$ invoke --list
Available tasks:

    build

This will also print the first line of each task’s docstring, if it has one. To
see what else is available besides --list, say invoke --help.

Running shell commands

Many use cases for Invoke involve running local shell commands, similar to
programs like Make or Rake. This is done via the ~.Context.run function::

text
from invoke import task

@task
def build(c):
    c.run("sphinx-build docs docs/_build")

You'll see the command's output in your terminal as it runs::

text
$ invoke build
Running Sphinx v1.1.3
loading pickled environment... done
...
build succeeded, 2 warnings.

~.Context.run has a number of arguments controlling its behavior, such as
activation of pseudo-terminals for complex programs requiring them, suppression
of exit-on-error behavior, hiding of subprocess' output (while still capturing
it for later review), and more. See its API docs <.Context.run> for details.

~.Context.run always returns a useful .Result object providing access to
the captured output, exit code, and other information.

.. _why-context:

Aside: what exactly is this 'context' arg anyway?

A common problem task runners face is transmission of "global" data - values
loaded from :doc:configuration files </concepts/configuration> or :ref:other configuration vectors <collection-configuration>, given via CLI flags,
generated in 'setup' tasks, etc.

Some libraries (such as Fabric <https://fabfile.org>_ 1.x) implement this via
module-level attributes, which makes testing difficult and error prone, limits
concurrency, and increases implementation complexity.

Invoke encapsulates state in explicit ~.Context objects, handed to tasks when
they execute . The context is the primary API endpoint, offering methods which
honor the current state (such as .Context.run) as well as access to that
state itself.

Declaring pre-tasks

Tasks may be configured in a number of ways via the .task decorator. One of
these is to select one or more other tasks you wish to always run prior to
execution of your task, indicated by name.

Let's expand our docs builder with a new cleanup task that runs before every
build (but which, of course, can still be executed on its own)::

text
from invoke import task

@task
def clean(c):
    c.run("rm -rf docs/_build")

@task(clean)
def build(c):
    c.run("sphinx-build docs docs/_build")

Now when you invoke build, it will automatically run clean first.

.. note::
If you're not a fan of the implicit "positional arguments are pre-run task
names" API, you can instead explicitly give the pre kwarg:
@task(pre=[clean]).

Details can be found in :ref:how-tasks-run.

Creating namespaces

Right now, our tasks.py is implicitly for documentation only, but maybe our
project needs other non-doc things, like packaging/deploying, testing, etc. At
that point, a single flat namespace isn't enough, so Invoke lets you easily
build a :doc:nested namespace <concepts/namespaces>. Here's a quick example.

Let's first rename our tasks.py to be docs.py; no other changes are
needed there. Then we create a new tasks.py, and for the sake of brevity
populate it with a new, truly top level task called deploy.

Finally, we can use a new API member, the .Collection class, to bind this
task and the docs module into a single explicit namespace. When Invoke
loads your task module, if a .Collection object bound as ns or
namespace exists it will get used for the root namespace::

text
from invoke import Collection, task
import docs

@task
def deploy(c):
    c.run("python setup.py sdist")
    c.run("twine upload dist/*")

namespace = Collection(docs, deploy)

The result::

text
$ invoke --list
Available tasks:

    deploy
    docs.build
    docs.clean

For a more detailed breakdown of how namespacing works, please see :doc:the docs <concepts/namespaces>.


Sites Docs Index (`sites/docs/index.rst`)

================================== Welcome to Invoke's documentation!

This site covers Invoke's conceptual & API documentation. For basic info on
what Invoke is, including its public changelog & how the project is maintained,
please see the main project website <https://pyinvoke.org>_.

Getting started

Many core ideas & API calls are explained in the tutorial/getting-started
document:

.. toctree::
:maxdepth: 2

text
getting-started

The ``invoke`` CLI tool

Details on the CLI interface to Invoke, available core flags, and tab
completion options.

.. toctree::
invoke

Concepts

Dig deeper into specific topics:

.. toctree::
:maxdepth: 2
:glob:

text
concepts/*

.. _api:

API

Know what you're looking for & just need API details? View our auto-generated
API documentation:

.. toctree::
:maxdepth: 1
:glob:

text
api/*

Sites Docs Invoke (`sites/docs/invoke.rst`)

.. _inv:

======================== ``inv[oke]`` core usage

.. seealso::
This page documents invoke's core arguments, options and behavior
(which includes options present in :ref:custom Invoke-based binaries <reusing-as-a-binary>). For details on invoking user-specified tasks and
other parser-related details, see :doc:/concepts/invoking-tasks.

Core options and flags

invoke's usage looks like::

text
$ inv[oke] [--core-opts] task1 [--task1-opts] ... taskN [--taskN-opts] [--
remainder-opts-or-values]

All core options & flags are below; almost all of them must be given before
any task names, with a few (such as :option:--help) being specially looked
for anywhere in the command line. (For parsing details, see
:ref:basic-cli-layout.)

.. option:: --complete

text
Print (line-separated) valid tab-completion options for an Invoke command
line given as the 'remainder' (i.e. after a ``--``). Used for building
:ref:`shell completion scripts <tab-completion>`.

For example, when the local tasks tree contains tasks named ``foo`` and
``bar``, and when ``foo`` takes flags ``--foo-arg`` and ``--foo-arg-2``,
you might use it like this::

    # Empty input: just task names
    $ inv --complete --
    foo
    bar

    # Input not ending with a dash: task names still
    $ inv --complete -- foo --foo-arg
    foo
    bar

    # Input ending with a dash: current context's flag names
    $ inv --complete -- foo -
    --foo-arg
    --foo-arg-2

For more details on how to make best use of this option, see
:option:`--print-completion-script`.

.. option:: --hide=STRING

text
Set default value of run()'s 'hide' kwarg.

.. option:: --no-dedupe

text
Disable task deduplication.

.. _print-completion-script:

.. option:: --print-completion-script=SHELL

text
Print a completion script for desired ``SHELL`` (e.g. ``bash``, ``zsh``,
etc). This can be sourced into the current session in order to enjoy
:ref:`tab-completion for tasks and options <tab-completion>`.

These scripts are bundled with Invoke's distributed codebase, and
internally make use of :option:`--complete`.

.. _prompt-for-sudo-password:

.. option:: --prompt-for-sudo-password

text
Prompt at the start of the session (before executing any tasks) for the
``sudo.password`` configuration value. This allows users who don't want to
keep sensitive material in the config system or their shell environment to
rely on user input, without otherwise interrupting the flow of the program.

.. option:: --write-pyc

text
By default, Invoke disables bytecode caching as it can cause hard-to-debug
problems with task files and (for the kinds of things Invoke is typically
used for) offers no noticeable speed benefit. If you really want your
``.pyc`` files back, give this option.

.. option:: -c STRING, --collection=STRING

text
Specify collection name to load.

.. option:: -d, --debug

text
Enable debug output.

.. option:: --dry

text
Echo commands instead of actually running them; specifically, causes any
``run`` calls to:

- Act as if the ``echo`` option has been turned on, printing the
  command-to-be-run to stdout;
- Skip actual subprocess invocation (returning before any of that machinery
  starts running);
- Return a dummy `~invoke.runners.Result` object with 'blank' values (empty
  stdout/err strings, ``0`` exit code, etc).

.. option:: -D, --list-depth=INT

text
Limit :option:`--list` display to the specified number of levels, e.g.
``--list-depth 1`` to show only top-level tasks and namespaces.

If an argument is given to ``--list``, then this depth is relative; so
``--list build --list-depth 1`` shows everything at the top level of the
``build`` subtree.

Default behavior if this is not given is to show all levels of the entire
task tree.

.. option:: -e, --echo

text
Echo executed commands before running.

.. option:: -f, --config

text
Specify a :ref:`runtime configuration file <config-hierarchy>` to load.

Note that you may instead use the ``INVOKE_RUNTIME_CONFIG`` environment
variable in place of this option. If both are given, the CLI option will
win out.

.. option:: -F, --list-format=STRING

text
Change the format used to display the output of :option:`--list`; may be
one of:

- ``flat`` (the default): single, flat vertical list with dotted task
  names.
- ``nested``: a nested (4-space indented) vertical list, where each level
  implicitly includes its parent (with leading dots as a strong visual clue
  that these are still subcollection tasks.)
- ``json``: intended for consumption by scripts or other programs, this
  format emits JSON representing the task tree, with each 'node' in the
  tree (the outermost document being the root node, and thus a JSON object)
  consisting of the following keys:

  - ``name``: String name of collection; for the root collection this is
    typically the module name, so unless you're supplying alternate
    collection name to the load process, it's usually ``"tasks"`` (from
    ``tasks.py``.)
  - ``help``: First line of collection's docstring, if it came from a
    module; null otherwise (or if module lacked a docstring.)
  - ``tasks``: Immediate children of this collection; an array of objects
    of the following form:

    - ``name``: Task's local name within its collection (i.e. not the full
      dotted path you might see with the ``flat`` format; reconstructing
      that path is left up to the consumer.)
    - ``help``: First line of task's docstring, or null if it had none.
    - ``aliases``: An array of string aliases for this task.

  - ``default``: String naming which task within this collection, if any,
    is the default task. Is null if no task is the default.
  - ``collections``: An array of any sub-collections within this
    collection, members of which which will have the same structure as this
    outermost document, recursively.

  The JSON emitted is not pretty-printed, but does end with a trailing
  newline.

.. option:: -h STRING, --help=STRING

text
When given without any task names, displays core help; when given with a
task name (may come before *or* after the task name) displays help for that
particular task.

.. option:: -l, --list=STRING

text
List available tasks. Shows all tasks by default; may give an explicit
namespace to 'root' the displayed task tree to only that namespace. (This
argument may contain periods, as with task names, so it's possible to show
only a small, deep portion of the overall tree if desired.)

.. option:: -p, --pty

text
Use a pty when executing shell commands.

.. option:: -r STRING, --search-root=STRING

text
Change root directory used for finding task modules.

.. option:: -T INT, --command-timeout=INT

text
Set a default command execution timeout of INT seconds. Maps to the
``timeouts.command`` config setting.

.. option:: -V, --version

text
Show version and exit.

.. option:: -w, --warn-only

text
Warn, instead of failing, when shell commands fail.

.. _remainder:

Unparsed input: the 'remainder'

Invoke takes a page from the GNU command-line tradition and many shell
builtins, and implements the concept of the remainder (aka the "end of
options" signal): a 'bare' -- not attached to any option name, followed by
arbitrary text.

Such text is not subject to Invoke's CLI parser, and is used to power features
like :ref:tab-completion or :ref:wrapper-tasks

.. _tab-completion:

Shell tab completion

Generating a completion script

Invoke's philosophy is to implement generic APIs and then "bake in" a few
common use cases built on top of those APIs; tab completion is no different.
Generic tab completion functionality (outputting a shell-compatible list of
completion tokens for a given command line context) is provided by the
:option:--complete core CLI option described above.

However, you probably won't need to use that flag yourself: we distribute a
handful of ready-made wrapper scripts aimed at the most common shells like
bash and zsh (plus others). These scripts can be automatically
generated from Invoke or :ref:any Invoke-driven command-line tool <reusing-as-a-binary>, using :option:--print-completion-script; the printed
scripts will contain the correct binary name(s) for the program generating
them.

For example, the following command prints (to stdout) a script which works for
zsh, instructs zsh to use it for the inv and invoke programs,
and calls invoke --complete at runtime to get dynamic completion
information::

text
$ invoke --print-completion-script zsh

.. note::
You'll probably want to source this command or store its output somewhere
permanently; more on that in the next section.

Similarly, the Fabric <https://fabfile.org>_ tool inherits from Invoke, and
only has a single binary name (fab); if you wanted to get Fabric completion
in bash, you would say::

text
$ fab --print-completion-script bash

In the rest of this section, we'll use inv in examples, but please remember
to replace it with the program you're actually using, if it's not Invoke
itself!

Sourcing the script

There are a few ways to utilize the output of the above commands, depending on
your needs, where the program is installed, and your shell:

  • The simplest and least disruptive method is to source the printed
    completion script inline, which doesn't place anything on disk, and will only
    affect the current shell session::

    $ source <(inv --print-completion-script zsh)

  • If you've got the program available in your system's global Python
    interpreter (and you're okay with running the program at the startup of each
    shell session - Python's speed is admittedly not its strong point) you could
    add that snippet to your shell's startup file, such as ~/.zshrc or
    ~/.bashrc.

  • If the program's available globally but you'd prefer to avoid running an
    extra Python program at shell startup, you can cache the output of the
    command in its own file; where this file lives is entirely up to you and how
    your shell is configured. For example, you might just drop it into your home
    directory as a hidden file::

    $ inv --print-completion-script zsh > ~/.invoke-completion.sh

    and then perhaps add the following to the end of ~/.zshrc::

    source ~/.invoke-completion.sh

    But again, this is entirely up to you and your shell.

    .. note::
    If you're using fish, you must use this tactic, as our fish
    completion script is not suitable for direct sourcing. Fish shell users
    should direct the output of the command to a file in the
    ~/.config/fish/completions/ directory.

  • Finally, if your copy of the needing-completion program is only installed in
    a specific environment like a virtualenv, you can use either of the above
    techniques:

    • Caching the output and referencing it in a global shell startup file will
      still work in this case, as it does not require the program to be
      available when the shell loads -- only when you actually attempt to tab
      complete.
    • Using the source <(inv --print-completion-script yourshell) approach
      will work as long as you place it in some appropriate per-environment
      startup file, which will vary depending on how you manage Python
      environments. For example, if you use virtualenvwrapper, you could
      append the source line in /path/to/virtualenv/bin/postactivate.

Utilizing tab completion itself

You've ensured that the completion script is active in your environment - what
have you gained?

  • By default, tabbing after typing inv or invoke will display task
    names from your current directory/project's tasks file.

  • Tabbing after typing a dash (-) or double dash (--) will display
    valid options/flags for the current context: core Invoke options if no task
    names have been typed yet; options for the most recently typed task
    otherwise.

    • Tabbing while typing a partial long option will complete matching long
      options, using your shell's native substring completion. E.g. if no task
      names have been typed yet, --e<tab> will offer --echo as a
      completion option.
  • Hitting tab when the most recent typed/completed token is a flag which takes
    a value, will 'fall through' to your shell's native filename completion.

    • For example, prior to typing a task name, --config <tab> will
      complete local file paths to assist in filling in a config file.

Sites Docs Api Init (`sites/docs/api/__init__.rst`)

============ ``__init__``

.. automodule:: invoke


Sites Docs Api Collection (`sites/docs/api/collection.rst`)

============== ``collection``

.. autoclass:: invoke.collection.Collection
:special-members:
:exclude-members: weakref, init


Sites Docs Api Config (`sites/docs/api/config.rst`)

========== ``config``

.. automodule:: invoke.config


Sites Docs Api Context (`sites/docs/api/context.rst`)

=========== ``context``

.. automodule:: invoke.context


Sites Docs Api Exceptions (`sites/docs/api/exceptions.rst`)

============== ``exceptions``

.. automodule:: invoke.exceptions


Sites Docs Api Executor (`sites/docs/api/executor.rst`)

============ ``executor``

.. autoclass:: invoke.executor.Executor


Sites Docs Api Loader (`sites/docs/api/loader.rst`)

========== ``loader``

.. autoclass:: invoke.loader.Loader
:special-members:
:exclude-members: weakref

.. autoclass:: invoke.loader.FilesystemLoader
:special-members:
:exclude-members: weakref


Sites Docs Api Parser (`sites/docs/api/parser.rst`)

========== ``parser``

The command-line parsing framework is split up into a handful of sub-modules:

  • parser.argument
  • parser.context (not to be confused with the top level context!)
  • parser.parser

API docs for all are below.

.. automodule:: invoke.parser.parser
:member-order: bysource
.. automodule:: invoke.parser.context
.. automodule:: invoke.parser.argument


Sites Docs Api Program (`sites/docs/api/program.rst`)

=========== ``program``

.. automodule:: invoke.program


Sites Docs Api Runners (`sites/docs/api/runners.rst`)

=========== ``runners``

.. automodule:: invoke.runners
:member-order: bysource


Sites Docs Api Tasks (`sites/docs/api/tasks.rst`)

========= ``tasks``

.. automodule:: invoke.tasks


Sites Docs Api Terminals (`sites/docs/api/terminals.rst`)

============= ``terminals``

.. automodule:: invoke.terminals


Sites Docs Api Util (`sites/docs/api/util.rst`)

======== ``util``

.. automodule:: invoke.util
:exclude-members: ExceptionWrapper

.. autoclass:: invoke.util.ExceptionWrapper
:no-members:


Sites Docs Api Watchers (`sites/docs/api/watchers.rst`)

============ ``watchers``

.. automodule:: invoke.watchers


Sites Docs Concepts Configuration (`sites/docs/concepts/configuration.rst`)

.. _configuration:

============= Configuration

Introduction

Invoke offers a multifaceted configuration mechanism allowing you to configure
both core behavior and that of your tasks, via a hierarchy of configuration
files, environment variables, :doc:task namespaces </concepts/namespaces> and
CLI flags.

The end result of configuration seeking, loading, parsing & merging is a
.Config object, which behaves like a (nested) Python dictionary. Invoke
references this object when it runs (determining the default behavior of
methods like .Context.run) and exposes it to users' tasks as
.Context.config or as shorthand attribute access on the .Context itself.

.. _config-hierarchy:

The configuration hierarchy

In brief, the order in which configuration values override one another is as
follows:

#. Internal default values for behaviors which are controllable via
configuration. See :ref:default-values for details.
#. Collection-driven configurations defined in tasks modules via
.Collection.configure. (See :ref:collection-configuration below for
details.)

text
- Sub-collections' configurations get merged into the top level collection
   and the final result forms the basis of the overall configuration setup.

#. System-level configuration file stored in /etc/, such as
/etc/invoke.yaml. (See :ref:config-files for details on this and the
other config-file entries.)
#. User-level configuration file found in the running user's home
directory, e.g. ~/.invoke.yaml.
#. Project-level configuration file living next to your top level
tasks.py. For example, if your run of Invoke loads
/home/user/myproject/tasks.py (see our docs on :doc:the load process </concepts/loading>), this might be /home/user/myproject/invoke.yaml.
#. Environment variables found in the invoking shell environment.

text
- These aren't as strongly hierarchical as the rest, nor is the shell
  environment namespace owned wholly by Invoke, so we must rely on slightly
  verbose prefixing instead - see :ref:`env-vars` for details.

#. Runtime configuration file whose path is given to :option:-f, e.g.
inv -f /random/path/to/config_file.yaml. This path may also be set via the
INVOKE_RUNTIME_CONFIG env var.
#. Command-line flags for certain core settings, such as :option:-e.
#. Modifications made by user code at runtime.

.. _default-values:

Default configuration values

Below is a list of all the configuration values and/or section Invoke itself
uses to control behaviors such as .Context.run's echo and pty
flags, task deduplication, and so forth.

.. note::
The storage location for these values is inside the .Config class,
specifically as the return value of .Config.global_defaults; see its API
docs for more details.

For convenience, we refer to nested setting names with a dotted syntax, so e.g.
foo.bar refers to what would be (in a Python config context) {'foo': {'bar': <value here>}}. Typically, these can be read or set on .Config and
.Context objects using attribute syntax, which looks nearly identical:
c.foo.bar.

  • The tasks config tree holds settings relating to task execution.

    • tasks.dedupe controls :ref:deduping and defaults to True. It
      can also be overridden at runtime via :option:--no-dedupe.

    • tasks.auto_dash_names controls whether task and collection names have
      underscores turned to dashes on the CLI. Default: True. See also
      :ref:dashes-vs-underscores.

    • tasks.collection_name controls the Python import name sought out by
      :ref:collection discovery <collection-discovery>, and defaults to
      "tasks".

    • tasks.executor_class allows users to override the class instantiated
      and used for task execution.

      Must be a fully-qualified dotted path of the form
      module(.submodule...).class, where all but .class will be handed
      to importlib.import_module, and class is expected to be an
      attribute on that resulting module object.

      Defaults to None, meaning to use the running .Program object's
      executor_class attribute.

      .. warning::
      Take care if using this setting in tandem with :ref:custom program binaries <reusing-as-a-binary>, since custom programs may specify
      their own default executor class (which your use of this setting will
      override!) and assume certain behaviors stemming from that.

    • tasks.ignore_unknown_help (default: False) lets users disable
      "help keys were supplied for nonexistent arguments" errors. Normally,
      Invoke assumes such a situation implies a typo in the help argument
      to @task, but sometimes users have good reasons for this.

    • tasks.search_root allows overriding the default :ref:collection discovery <collection-discovery> root search location. It defaults to
      None, which indicates to use the executing process' current working
      directory.

  • The run tree controls the behavior of .Runner.run. Each member of this
    tree (such as run.echo or run.pty) maps directly to a .Runner.run
    keyword argument of the same name; see that method's docstring for details on
    what these settings do & what their default values are.

  • The runners tree controls which runner classes map to which execution
    contexts; if you're using Invoke by itself, this will only tend to have a
    single member, runners.local. Client libraries may extend it with
    additional key/value pairs, such as runners.remote.

  • The sudo tree controls the behavior of .Context.sudo:

    • sudo.password controls the autoresponse password submitted to sudo's
      password prompt. Default: None.

      .. warning::
      While it's possible to store this setting, like any other, in
      :doc:configuration files </concepts/configuration> -- doing so is
      inherently insecure. We highly recommend filling this config value in
      at runtime from a secrets management system of some kind.

    • sudo.prompt holds the sudo password prompt text, which is both
      supplied to sudo -p, and searched for when performing
      :doc:auto-response </concepts/watchers>. Default: [sudo] password:.

  • A top level config setting, debug, controls whether debug-level output is
    logged; it defaults to False.

    debug can be toggled via the :option:-d CLI flag, which enables
    debugging after CLI parsing runs. It can also be toggled via the
    INVOKE_DEBUG environment variable which - unlike regular env vars - is
    honored from the start of execution and is thus useful for troubleshooting
    parsing and/or config loading.

  • A small config tree, timeouts, holds various kinds of timeout controls.
    At present, for Invoke, this only holds a command subkey, which controls
    subprocess execution timeouts.

    • Client code often adds more to this tree, and Invoke itself may add more
      in the future as well.

.. _config-files:

Configuration files

Loading

For each configuration file location mentioned in the previous section, we
search for files ending in .yaml, .yml, .json or .py (in that
order!
), load the first one we find, and ignore any others that might exist.

For example, if Invoke is run on a system containing both /etc/invoke.yml
and /etc/invoke.json, only the YAML file will be loaded. This helps
keep things simple, both conceptually and in the implementation.

Format

Invoke's configuration allows arbitrary nesting, and thus so do our config file
formats. All three of the below examples result in a configuration equivalent
to {'debug': True, 'run': {'echo': True}}:

  • YAML

    .. code-block:: yaml

    text
    debug: true
    run:
        echo: true
  • JSON

    .. code-block:: javascript

    text
    {
        "debug": true,
        "run": {
            "echo": true
        }
    }
  • Python::

    debug = True
    run = {
    "echo": True

    }

For further details, see these languages' own documentation.

.. _env-vars:

Environment variables

Environment variables are a bit different from other configuration-setting
methods, since they don't provide a clean way to nest configuration keys, and
are also implicitly shared amongst the entire system's installed application
base.

In addition, due to implementation concerns, env vars must be pre-determined by
the levels below them in the config hierarchy (in other words - env vars may
only be used to override existing config values). If you need Invoke to
understand a FOOBAR environment variable, you must first declare a
foobar setting in a configuration file or in your task collections.

Basic rules

To mitigate the shell namespace problem, we simply prefix all our env vars with
INVOKE_.

Nesting is performed via underscore separation, so a setting that looks like
e.g. {'run': {'echo': True}} at the Python level becomes
INVOKE_RUN_ECHO=1 in a typical shell. See :ref:env-var-nesting below for
more on this.

Type casting

Since env vars can only be used to override existing settings, the previous
value of a given setting is used as a guide in casting the strings we get back
from the shell:

  • If the current value is a Unicode string, it is replaced with the
    value from the environment, with no casting whatsoever;

  • If the current value is None, it too is replaced with the string from the
    environment;

  • Booleans are set as follows: 0 and the empty value/string (e.g.
    SETTING=, or unset SETTING, or etc) evaluate to False, and any
    other value evaluates to True.

  • Lists and tuples are currently unsupported and will raise an exception;

    • In the future we may implement convenience transformations, such as
      splitting on commas to form a list; however since users can always
      perform such operations themselves, it may not be a high priority.
  • All other types - integers, longs, floats, etc - are simply used as
    constructors for the incoming value.

    • For example, a foobar setting whose default value is the integer
      1 will run all env var inputs through int, and thus FOOBAR=5
      will result in the Python value 5, not "5".

.. _env-var-nesting:

Nesting vs underscored names

Since environment variable keys are single strings, we must use some form of
string parsing to allow access to nested configuration settings. As mentioned
above, in basic use cases this just means using an underscore character:
{'run': {'echo': True}} becomes INVOKE_RUN_ECHO=1.

However, ambiguity is introduced when the settings names themselves contain
underscores: is INVOKE_FOO_BAR=baz equivalent to {'foo': {'bar': 'baz'}}, or to {'foo_bar': 'baz'}? Thankfully, because env vars can only
be used to modify settings declared at the Python level or in config files, we
look at the current state of the config to determine the answer.

There is still a corner case where both possible interpretations exist as
valid config paths (e.g. {'foo': {'bar': 'default'}, 'foo_bar': 'otherdefault'}). In this situation, we honor the Zen of Python <http://zen-of-python.info/in-the-face-of-ambiguity-refuse-the-temptation-to-guess.html#12>_
and refuse to guess; an error is raised instead, counseling users to modify
their configuration layout or avoid using env vars for the setting in question.

.. _collection-configuration:

`.Collection`-based configuration

.Collection objects may contain a config mapping, set via
.Collection.configure, and (as per :ref:the hierarchy <config-hierarchy>)
this typically forms the lowest level of configuration in the system.

When collections are :doc:nested </concepts/namespaces>, configuration is
merged 'downwards' by default: when conflicts arise, outer namespaces closer to
the root will win, versus inner ones closer to the task being invoked.

.. note::
'Inner' tasks here are specifically those on the path from the root to the
one housing the invoked task. 'Sibling' subcollections are ignored.

A quick example of what this means::

text
from invoke import Collection, task

# This task & collection could just as easily come from
# another module somewhere.
@task
def mytask(c):
    print(c['conflicted'])
inner = Collection('inner', mytask)
inner.configure({'conflicted': 'default value'})

# Our project's root namespace.
ns = Collection(inner)
ns.configure({'conflicted': 'override value'})

The result of calling inner.mytask::

text
$ inv inner.mytask
override value

Example of real-world config use

The previous sections had small examples within them; this section provides a
more realistic-looking set of examples showing how the config system works.

Setup

We'll start out with semi-realistic tasks that hardcode their values, and build
up to using the various configuration mechanisms. A small module for building
Sphinx <http://sphinx-doc.org>_ docs might begin like this::

text
from invoke import task

@task
def clean(c):
    c.run("rm -rf docs/_build")

@task
def build(c):
    c.run("sphinx-build docs docs/_build")

Then maybe you refactor the build target::

text
target = "docs/_build"

@task
def clean(c):
    c.run("rm -rf {}".format(target))

@task
def build(c):
    c.run("sphinx-build docs {}".format(target))

We can also allow runtime parameterization::

text
default_target = "docs/_build"

@task
def clean(c, target=default_target):
    c.run("rm -rf {}".format(target))

@task
def build(c, target=default_target):
    c.run("sphinx-build docs {}".format(target))

This task module works for a single set of users, but what if we want to allow
reuse? Somebody may want to use this module with a different default target.
Using the configuration data (made available via the context arg) to configure
these settings is usually the better solution [1]_.

Configuring via task collection

The configuration setting <.Collection.configure> and getting <.Context.config> APIs enable moving otherwise 'hardcoded' default values into
a config structure which downstream users are free to redefine. Let's apply
this to our example. First we add an explicit namespace object::

text
from invoke import Collection, task

default_target = "docs/_build"

@task
def clean(c, target=default_target):
    c.run("rm -rf {}".format(target))

@task
def build(c, target=default_target):
    c.run("sphinx-build docs {}".format(target))

ns = Collection(clean, build)

Then we can move the default build target value into the collection's default
configuration, and refer to it via the context. At this point we also change
our kwarg default value to be None so we can determine whether or not a
runtime value was given. The result::

text
@task
def clean(c, target=None):
    if target is None:
        target = c.sphinx.target
    c.run("rm -rf {}".format(target))

@task
def build(c, target=None):
    if target is None:
        target = c.sphinx.target
    c.run("sphinx-build docs {}".format(target))

ns = Collection(clean, build)
ns.configure({'sphinx': {'target': "docs/_build"}})

The result isn't significantly more complex than what we began with, and as
we'll see next, it's now trivial for users to override your defaults in various
ways.

Configuration overriding

The lowest-level override is, of course, just modifying the local .Collection
tree into which a distributed module has been imported. E.g. if the above
module is distributed as myproject.docs, someone can define a tasks.py
that does this::

text
from invoke import Collection, task
from myproject import docs

@task
def mylocaltask(c):
    # Some local stuff goes here
    pass

# Add 'docs' to our local root namespace, plus our own task
ns = Collection(mylocaltask, docs)

And then they can add this to the bottom::

text
# Our docs live in 'built_docs', not 'docs/_build'
ns.configure({'sphinx': {'target': "built_docs"}})

Now we have a docs sub-namespace whose build target defaults to
built_docs instead of docs/_build. Runtime users can still override
this via flags (e.g. inv docs.build --target='some/other/dir') just as
before.

If you prefer configuration files over in-Python tweaking of your namespace
tree, that works just as well; instead of adding the line above to the previous
snippet, instead drop this into a file next to tasks.py named
invoke.yaml::

text
sphinx:
    target: built_docs

For this example, that sort of local-to-project conf file makes the most sense,
but don't forget that the :ref:config hierarchy <config-hierarchy> offers
additional configuration methods which may be suitable depending on your needs.

.. rubric:: Footnotes

.. [1]
Copying and modifying the file breaks code reuse; overriding the
module-level default_path variable won't play well with concurrency;
wrapping the tasks with different default arguments works but is fragile
and adds boilerplate.


Sites Docs Concepts Invoking Tasks (`sites/docs/concepts/invoking-tasks.rst`)

.. _invoking-tasks:

============== Invoking tasks

This page explains how to invoke your tasks on the CLI, both in terms of parser
mechanics (how your tasks' arguments are exposed as command-line options) and
execution strategies (which tasks actually get run, and in what order).

(For details on Invoke's core flags and options, see :doc:/invoke.)

.. contents::
:local:

.. _basic-cli-layout:

Basic command line layout

Invoke may be executed as invoke (or inv for short) and its command
line layout looks like this::

text
$ inv [--core-opts] task1 [--task1-opts] ... taskN [--taskN-opts]

Put plainly, Invoke's CLI parser <.Parser> splits your command line up into
multiple "parser contexts <.ParserContext>" which allows it to reason about
the args and options it will accept:

  • Before any task names are given, the parser is in the "core" parse context,
    and looks for core options and flags such as :option:--echo,
    :option:--list or :option:--help.
  • Any non-argument-like token (such as mytask) causes a switch into a
    per-task context (or an error, if no task matching that name seems to exist
    in the :doc:loaded collection </concepts/loading>).
  • At this point, argument-like tokens are expected to correspond to the
    arguments for the previously named task (see :ref:task-arguments).
  • Then this cycle repeats infinitely, allowing chained execution of arbitrary
    numbers of tasks. (In practice, most users only execute one or two at a
    time.)

For the core arguments and flags, see :doc:/invoke; for details on how your
tasks affect the CLI, read onwards.

.. note::
There is a minor convenience-minded exception to how parse contexts behave:
core options may also be given inside per-task contexts, if and only if
there is no conflict with similarly-named/prefixed arguments of the
being-parsed task.

text
For example, ``invoke mytask --echo`` will behave identically to ``invoke
--echo mytask``, *unless* ``mytask`` has its own ``echo`` flag (in which
case that flag is handed to the task context, as normal).

Similarly, ``invoke mytask -e`` will turn on command echoing too, unless
``mytask`` has its own argument whose shortflag ends up set to ``-e`` (e.g.
``def mytask(env)``).

.. _task-arguments:

Task command-line arguments

The simplest task invocation, for a task requiring no parameterization::

text
$ inv mytask

Tasks may take parameters in the form of flag arguments::

text
$ inv build --format=html
$ inv build --format html
$ inv build -f pdf
$ inv build -f=pdf

Note that both long and short style flags are supported, and that equals signs
are optional in both cases.

Boolean options are simple flags with no arguments::

text
$ inv build --progress-bar

Naturally, more than one flag may be given at a time::

text
$ inv build --progress-bar -f pdf

Type casting

Natively, a command-line string is just that -- a string -- requiring some
leaps of logic to arrive at any non-string values on the Python end. Invoke has
a number of these tricks already at hand, and more will be implemented in the
future. Currently:

  • Arguments with default values use those default values as a type hint, so
    def mytask(c, count=1) will see inv mytask --count=5 and result in
    the Python integer value 5 instead of the string "5".

    • Default values of None are effectively the same as having no default
      value at all - no type casting occurs and you're left with a string.
  • The primary exception to the previous rule is booleans: default values of
    True or False cause those arguments to show up as actual
    non-value-taking flags (--argname to set the value to True if the
    default was False, or --no-argment in the opposite case). See
    :ref:boolean-flags for more examples.

  • List values (which you wouldn't want to set as an argument's default value
    anyways -- it's a common Python misstep) are served by a special @task
    flag - see :ref:iterable-flag-values below.

  • There's currently no way to set other compound values (such as dicts) on
    the command-line; solving this more complex problem is left as an exercise to
    the reader (though we may add helpers for such things in the future).

Per-task help / printing available flags

To get help for a specific task, you can give the task name as an argument to
the core --help/-h option, or give --help/-h after the task
(which will trigger custom-to-help behavior where the task name itself is
given to --help as its argument value).

When help is requested, you'll see the task's docstring (if any) and
per-argument/flag help output::

text
$ inv --help build  # or invoke build --help

Docstring:
  none

Options for 'build':
  -f STRING, --format=STRING  Which build format type to use
  -p, --progress-bar          Display progress bar

Globbed short flags

Boolean short flags may be combined into one flag expression, so that e.g.::

text
$ inv build -qv

is equivalent to (and expanded into, during parsing)::

text
$ inv build -q -v

If the first flag in a globbed short flag token is not a boolean but takes a
value, the rest of the glob is taken to be the value instead. E.g.::

text
$ inv build -fpdf

is expanded into::

text
$ inv build -f pdf

and not::

text
$ inv build -f -p -d -f

.. _optional-values:

Optional flag values

You saw a hint of this with --help specifically, but non-core options may
also take optional values, if declared as optional. For example, say your
task has a --log flag that activates logging::

text
$ inv compile --log

but you also want it to be configurable regarding where to log::

text
$ inv compile --log=foo.log

You could implement this with an additional argument (e.g. --log and
--log-location) but sometimes the concise API is the more useful one.

To enable this, specify which arguments are of this 'hybrid' optional-value
type inside @task::

text
@task(optional=['log'])
def compile(c, log=None):
    if log:
        log_file = '/var/log/my.log'
        # Value was given, vs just-True
        if isinstance(log, str):
            log_file = log
        # Replace w/ your actual log setup...
        set_log_destination(log_file)
    # Do things that might log here...

When optional flag values are used, the values seen post-parse follow these
rules:

  • If the flag is not given at all (invoke compile) the default value
    is filled in as normal.
  • If it is given with a value (invoke compile --log=foo.log) then the value
    is stored normally.
  • If the flag is given with no value (invoke compile --log), it is treated
    as if it were a bool and set to True.

Resolving ambiguity

text
There are a number of situations where ambiguity could arise for a flag that
takes an optional value:

* When a task takes positional arguments and they haven't all been filled in by
  the time the parser arrives at the optional-value flag;
* When the token following one of these flags looks like it is itself a flag;
  or
* When that token has the same name as another task.

In most of these situations, Invoke's parser will `refuse the temptation to
guess
<http://zen-of-python.info/in-the-face-of-ambiguity-refuse-the-temptation-to-guess.html#12>`_
and raise an error.

However, in the case where the ambiguous token is flag-like, the current parse
context is checked to resolve the ambiguity:

- If the token is an otherwise legitimate argument, it is assumed that the user
  meant to give that argument immediately after the current one, and no
  optional value is set.

    - E.g. in ``invoke compile --log --verbose`` (assuming ``--verbose`` is
      another legit argument for ``compile``) the parser decides the user meant
      to give ``--log`` without a value, and followed it up with the
      ``--verbose`` flag.

- Otherwise, the token is interpreted literally and stored as the value for
  the current flag.

    - E.g. if ``--verbose`` is *not* a legitimate argument for ``compile``,
      then ``invoke compile --log --verbose`` causes the parser to assign
      ``"--verbose"`` as the value given to ``--log``. (This will probably
      cause other problems in our contrived use case, but it illustrates our
      point.)

.. _iterable-flag-values:

Iterable flag values
--------------------

A not-uncommon use case for CLI programs is the desire to build a list of
values for a given option, instead of a single value. While this *can* be done
via sub-string parsing -- e.g. having users invoke a command with ``--mylist
item1,item2,item3`` and splitting on the comma -- it's often preferable to
specify the option multiple times and store the values in a list (instead of
overwriting or erroring.)

In Invoke, this is enabled by hinting to the parser that one or more task
arguments are ``iterable`` in nature (similar to how one specifies ``optional``
or ``positional``)::

    @task(iterable=['my_list'])
    def mytask(c, my_list):
        print(my_list)

When not given at all, the default value for ``my_list`` will be an empty list;
otherwise, the result is a list, appending each value seen, in order, without
any other manipulation (so no deduplication, etc)::

    $ inv mytask
    []
    $ inv mytask --my-list foo
    ['foo']
    $ inv mytask --my-list foo --my-list bar
    ['foo', 'bar']
    $ inv mytask --my-list foo --my-list bar --my-list foo
    ['foo', 'bar', 'foo']

.. _incrementable-flag-values:

Incrementable flag values
-------------------------

This is arguably a sub-case of :ref:`iterable flag values
` (seen above) - it has the same core interface of "give
a CLI argument multiple times, and have that do something other than error or
overwrite a single value." However, 'incrementables' (as you may have guessed)
increment an integer instead of building a list of strings. This is commonly
found in verbosity flags and similar functionality.

An example of exactly that::

    @task(incrementable=['verbose'])
    def mytask(c, verbose=0):
        print(verbose)

And its use::

    $ inv mytask
    0
    $ inv mytask --verbose
    1
    $ inv mytask -v
    1
    $inv mytask -vvv
    3

Happily, because in Python ``0`` is 'falsey' and ``1`` (or any other number) is
'truthy', this functions a lot like a boolean flag as well, at least if one
defaults it to ``0``.

.. note::
    You may supply any integer default value for such arguments (it simply
    serves as the starting value), but take care that consumers of the argument
    are written understanding that it is always going to appear 'truthy' unless
    it's ``0``!

Dashes vs underscores in flag names
-----------------------------------

In Python, it's common to use ``underscored_names`` for keyword arguments,
e.g.::

    @task
    def mytask(c, my_option=False):
        pass

However, the typical convention for command-line flags is dashes, which aren't
valid in Python identifiers::

    $ inv mytask --my-option

Invoke works around this by automatically generating dashed versions of
underscored names, when it turns your task function signatures into
command-line parser flags.

Therefore, the two examples above actually work fine together -- ``my_option``
ends up mapping to ``--my-option``.

In addition, leading (``_myopt``) and trailing (``myopt_``) underscores are
ignored, since ``invoke ---myopt`` and ``invoke --myopt-`` don't make much
sense.

.. _boolean-flags:

Automatic Boolean inverse flags
-------------------------------

Boolean flags tend to work best when setting something that is normally
``False``, to ``True``::

    $ inv mytask --yes-please-do-x

However, in some cases, you want the opposite - a default of ``True``, which
can be easily disabled. For example, colored output::

    @task
    def run_tests(c, color=True):
        # ...

Here, what we really want on the command line is a ``--no-color`` flag that
sets ``color=False``. Invoke handles this for you: when setting up CLI flags,
booleans which default to ``True`` generate a ``--no-<name>`` flag instead.


.. _wrapper-tasks:

'Wrapper' tasks
---------------

An uncommon, but not rare, use case is one where your Invoke task is
essentially "wrapping" an existing command-line tool - the task performs setup
actions like "baking in" default arguments, setting shell env vars, or similar,
and then finally calls the underlying command with the result.

Prior to the addition of this feature, you had to use a clunky workaround:
stuffing all "passthrough" arguments into a single argument string of its own,
eg::

    @task
    def mywrapper(c, opts=""):
        default_args = "--baked-in-arg baked-in-value"
        c.run(f"some-command {default_args} {opts}")

Such tasks might be invoked like so::

    $ inv mywrapper
    # runs 'somecommand --baked-in-arg baked-in-value'
    $ inv mywrapper --opts "--foo --bar"
    # runs 'somecommand --baked-in-arg baked-in-value --foo --bar'

While this approach isn't *difficult*, it still carries friction, both in
having to cart this pattern around, and in the care required to invoke such
tasks properly - because it's easy to accidentally try the following::

    $ inv mywrapper --opts --foo --bar
    # will trigger a parsing exception! 'mywrapper' has no 'bar' argument!

To support this use case (among others), Invoke's parser offers the
:ref:`remainder <remainder>` feature, letting you pass in a pile of unparsed
text after a bare ``--`` signifier.

The remainder string is available in the task context argument,
`~invoke.context.Context`, as a ``.remainder`` attribute - giving us a somewhat
cleaner wrapper task::

    @task
    def mywrapper(c):
        c.run(f"somecommand --baked-in-arg baked-in-value {c.remainder}")

Used like so::

    $ inv mywrapper -- --foo --bar
    # runs 'somecommand --baked-in-arg baked-in-value --foo --bar'

The same result as before, but with a simpler task definition, and no need to
remember quoting or similar - just the extra double-dash.


.. _how-tasks-run:

How tasks run
=============

Base case
---------

In the simplest case, a task with no pre- or post-tasks runs one time.
Example::

    @task
    def hello(c):
        print("Hello, world!")

Execution::

    $ inv hello
    Hello, world!

.. _pre-post-tasks:

Pre- and post-tasks
-------------------

Tasks that should always have another task executed before or after them, may
use the ``@task`` deocator's ``pre`` and/or ``post`` kwargs, like so::

    @task
    def clean(c):
        print("Cleaning")

    @task
    def publish(c):
        print("Publishing")

    @task(pre=[clean], post=[publish])
    def build(c):
        print("Building")

Execution::

    $ inv build
    Cleaning
    Building
    Publishing

These keyword arguments always take iterables. As a convenience, pre-tasks (and
pre-tasks only) may be given as positional arguments, in a manner similar to
build systems like ``make``. E.g. we could present part of the above example
as::

    @task
    def clean(c):
        print("Cleaning")

    @task(clean)
    def build(c):
        print("Building")

As before, ``invoke build`` would cause ``clean`` to run, then ``build``.

Recursive/chained pre/post-tasks

Pre-tasks of pre-tasks will also be invoked (as will post-tasks of pre-tasks,
pre-tasks of post-tasks, etc) in a depth-first manner, recursively. Here's a
more complex (if slightly contrived) tasks file::

text
@task
def clean_html(c):
    print("Cleaning HTML")

@task
def clean_tgz(c):
    print("Cleaning .tar.gz files")

@task(clean_html, clean_tgz)
def clean(c):
    print("Cleaned everything")

@task
def makedirs(c):
    print("Making directories")

@task(clean, makedirs)
def build(c):
    print("Building")

@task(build)
def deploy(c):
    print("Deploying")

With a depth-first behavior, the below is hopefully intuitive to most users::

text
$ inv deploy
Cleaning HTML
Cleaning .tar.gz files
Cleaned everything
Making directories
Building
Deploying

.. _parameterizing-pre-post-tasks:

Parameterizing pre/post-tasks

text
By default, pre- and post-tasks are executed with no arguments, even if the
task triggering their execution was given some. When this is not suitable, you
can wrap the task objects with `~.tasks.call` objects which allow you to
specify a call signature::

    @task
    def clean(c, which=None):
        which = which or 'pyc'
        print("Cleaning {}".format(which))

    @task(pre=[call(clean, which='all')]) # or call(clean, 'all')
    def build(c):
        print("Building")

Example output::

    $ inv build
    Cleaning all
    Building


.. _deduping:

Task deduplication
------------------

By default, any task that would run more than once during a session (due e.g.
to inclusion in pre/post tasks), will only be run once. Example task file::

    @task
    def clean(c):
        print("Cleaning")

    @task(clean)
    def build(c):
        print("Building")

    @task(build)
    def package(c):
        print("Packaging")

With deduplication turned off (see below), the above would execute ``clean`` ->
``build`` -> ``build`` again -> ``package``. With deduplication, the double
``build`` does not occur::

    $ inv build package
    Cleaning
    Building
    Packaging

.. note::
    Parameterized pre-tasks (using `~.tasks.call`) are deduped based on their
    argument lists. For example, if ``clean`` was parameterized and hooked up
    as a pre-task in two different ways - e.g. ``call(clean, 'html')`` and
    ``call(clean, 'all')`` - they would not get deduped should both end up
    running in the same session.

    However, two separate references to ``call(clean, 'html')`` *would* become
    deduplicated.

Disabling deduplication
~~~~~~~~~~~~~~~~~~~~~~~

If you prefer your tasks to run every time no matter what, you can give the
``--no-dedupe`` core CLI option at runtime, or set the ``tasks.dedupe``
:doc:`config setting </concepts/configuration>` to ``False``. While it
doesn't make a ton of real-world sense, let's imagine we wanted to apply
``--no-dedupe`` to the above example; we'd see the following output::

    $ inv --no-dedupe build package
    Cleaning
    Building
    Building
    Packaging

The build step is now running twice.

---

### Sites Docs Concepts Library (`sites/docs/concepts/library.rst`)

=========================
Using Invoke as a library
=========================

While most of our documentation involves the user/CLI facing use cases of task
management and command execution, Invoke was designed for its constituent parts
to be usable independently by advanced users - either out of the box or with a
minimum of extra work. CLI parsing, subprocess command execution, task
organization, etc, are all written as broadly separated concerns.

This document outlines use cases already known to work (because downstream
tools like `Fabric <https://fabfile.org>`_ are already utilizing them).


.. _reusing-as-a-binary:

Reusing Invoke's CLI module as a distinct binary
================================================

A major use case is distribution of your own program using Invoke under the
hood, bound to a different binary name, and usually setting a specific task
:doc:`namespace </concepts/namespaces>` as the default. (This maps somewhat
closely to things like ``argparse`` from the standard library.) In some cases,
removing, replacing and/or adding core CLI flags is also desired.

Getting set up
--------------

Say you want to distribute a test runner called ``tester`` offering two
subcommands, ``unit`` and ``integration``, such that users could ``pip install
tester`` and have access to commands like ``tester unit``, ``tester
integration``, or ``tester integration --fail-fast``.

First, as with any distinct Python package providing CLI
'binaries', you'd inform your ``setup.py`` of your entrypoint::

    setup(
        name='tester',
        version='0.1.0',
        packages=['tester'],
        install_requires=['invoke'],
        entry_points={
            'console_scripts': ['tester = tester.main:program.run']
        }
    )

.. note::
    This is just an example snippet and is not a fully valid ``setup.py``; if
    you don't know how Python packaging works, a good starting place is `the
    Python Packaging User's Guide
    <https://python-packaging-user-guide.readthedocs.io>`_.

Nothing here is specific to Invoke - it's a standard way of telling Python to
install a ``tester`` script that executes the ``run`` method of a ``program``
object defined inside the module ``tester.main``.

Creating a ``Program``
----------------------

In our ``tester/main.py``, we start out importing Invoke's public CLI
functionality::

    from invoke import Program

Then we define the ``program`` object we referenced in ``setup.py``, which is a
simple `.Program` to do the heavy lifting, giving it our version number for
starters::

    program = Program(version='0.1.0')

At this point, installing ``tester`` would give you the same functionality as
Invoke's :doc:`built-in CLI tool `, except named ``tester`` and
exposing its own version number::

    $ tester --version
    Tester 0.1.0
    $ tester --help
    Usage: tester [--core-opts] task1 [--task1-opts] ... taskN [--taskN-opts]

    Core options:
        ... core Invoke options here ...

    $ tester --list
    Can't find any collection named 'tasks'!

This doesn't do us much good yet - there aren't any subcommands (and our users
don't care about arbitrary 'tasks', so Invoke's own default ``--help`` and
``--list`` output isn't a good fit).

Specifying subcommands
----------------------

For ``tester`` to expose ``unit`` and ``integration`` subcommands, we need to
define them, in a regular Invoke tasks module or :doc:`namespace
</concepts/namespaces>`. For our example, we'll just create ``tester/tasks.py``
(but as you'll see in a moment, this too is arbitrary and can be whatever you
like)::

    from invoke import task

    @task
    def unit(c):
        print("Running unit tests!")

    @task
    def integration(c):
        print("Running integration tests!")

As described in :doc:`/concepts/namespaces`, you can arrange this module
however you want - the above snippet uses an implicit namespace for brevity's
sake.

.. note::
    It's important to realize that there's nothing special about these
    "subcommands" - you could run them just as easily with vanilla Invoke,
    e.g. via ``invoke --collection=tester.tasks --list``.

Now the useful part: telling our custom `.Program` that this namespace of tasks
should be used as the subcommands for ``tester``, via the ``namespace`` kwarg::

    from invoke import Collection, Program
    from tester import tasks

    program = Program(namespace=Collection.from_module(tasks), version='0.1.0')

The result?

::

    $ tester --version
    Tester 0.1.0
    $ tester --help
    Usage: tester [--core-opts] <subcommand> [--subcommand-opts] ...

    Core options:
      ... core options here, minus task-related ones ...

    Subcommands:
      unit
      integration

    $ tester --list
    No idea what '--list' is!
    $ tester unit
    Running unit tests!

Notice how the 'usage' line changed (to specify 'subcommands' instead of
'tasks'); the list of specific subcommands is now printed as part of
``--help``; and ``--list`` has been removed from the options.

You can enable :ref:`tab-completion<tab-completion>` for your distinct
binary and subcommands.


Modifying core parser arguments
-------------------------------

A common need for this use case is tweaking the core parser arguments.
`.Program` makes it easy: default core `Arguments <.Argument>` are returned by
`.Program.core_args`. Extend this method's return value with ``super`` and
you're done::

    # Presumably, this is your setup.py-designated CLI module...

    from invoke import Program, Argument

    class MyProgram(Program):
        def core_args(self):
            core_args = super().core_args()
            extra_args = [
                Argument(names=('foo', 'f'), help="Foo the bars"),
                # ...
            ]
            return core_args + extra_args

    program = MyProgram()

.. warning::
    We don't recommend *omitting* any of the existing core arguments; a lot of
    basic functionality relies on their existence, even when left to default
    values.


.. _customizing-config-defaults:

Customizing the configuration system's defaults
===============================================

Besides the CLI-oriented content of the previous section, another area of
functionality that frequently needs updating when redistributing an Invoke
codebase (CLI or no CLI) is configuration. There are typically two concerns
here:

- Configuration filenames and the env var prefix - crucial if you ever expect
  your users to use the configuration system;
- Default configuration values - less critical (most defaults aren't labeled
  with anything Invoke-specific) but still sometimes desirable.

.. note::
    Both of these involve subclassing `.Config` (and, if using the CLI
    machinery, informing your `.Program` to use that subclass instead of the
    default one.)


Changing filenames and/or env var prefix
----------------------------------------

By default, Invoke's config system looks for files like ``/etc/invoke.yaml``,
``~/.invoke.json``, etc. If you're distributing client code named something
else, like the ``Tester`` example earlier, you might instead want the config
system to load ``/etc/tester.json`` or ``$CWD/tester.py``.

Similarly, the environment variable config level looks for env vars like
``INVOKE_RUN_ECHO``; you might prefer ``TESTER_RUN_ECHO``.

There are a few `.Config` attributes controlling these values:

- ``prefix``: A generic, catchall prefix used directly as the file prefix, and
  used via all-caps as the env var prefix;
- ``file_prefix``: For overriding just the filename prefix - otherwise, it
  defaults to the value of ``prefix``;
- ``env_prefix``: For overriding just the env var prefix - as you might have
  guessed, it too defaults to the value of ``prefix``.

Continuing our 'Tester' example, you'd do something like this::

    from invoke import Config

    class TesterConfig(Config):
        prefix = 'tester'

Or, to seek ``tester.yaml`` as before, but ``TEST_RUN_ECHO`` instead of
``TESTER_RUN_ECHO``::

    class TesterConfig(Config):
        prefix = 'tester'
        env_prefix = 'TEST'

Modifying default config values
-------------------------------

Default config values are simple - they're just the return value of the
staticmethod `.Config.global_defaults`, so override that and return whatever
you like - ideally something based on the superclass' values, as many defaults
are assumed to exist by the rest of the system. (The helper function
`invoke.config.merge_dicts` can be useful here.)

For example, say you want Tester to always echo shell commands by default when
your codebase calls `.Context.run`::

    from invoke import Program
    from invoke.config import Config, merge_dicts

    class TesterConfig(Config):
        @staticmethod
        def global_defaults():
            their_defaults = Config.global_defaults()
            my_defaults = {
                'run': {
                    'echo': True,
                },
            }
            return merge_dicts(their_defaults, my_defaults)

    program = Program(config_class=TesterConfig, version='0.1.0')

For reference, Invoke's own base defaults (the...default defaults, you could
say) are documented at :ref:`default-values`.

---

### Sites Docs Concepts Loading (`sites/docs/concepts/loading.rst`)

.. _loading-collections:

===================
Loading collections
===================

The core of Invoke's execution model involves one or more Collection objects.
While these may be created programmatically, in typical use Invoke will create
them for you from Python modules it finds or is told to use.


.. _collection-discovery:

Task module discovery
=====================

With no other configuration, simply calling ``invoke`` will look for a single
Python module or package named ``tasks``, and will treat it as the root
namespace. ``tasks`` (or any other name given via :ref:`loading configuration
options <configuring-loading>`) is searched for in the following ways:

* First, if a valid tasks module by that name already exists on Python's
  `sys.path <https://docs.python.org/release/2.7/library/sys.html#sys.path>`_,
  no more searching is done -- that module is selected.
* Failing that, search towards the root of the local filesystem, starting with
  the user's current working directory (`os.getcwd
  <https://docs.python.org/release/2.7/library/os.html#os.getcwd>`_) and try
  importing again with each directory temporarily added to ``sys.path``.

    * Due to how Python's import machinery works, this approach will always
      favor a package directory (``tasks/`` containing an ``__init__.py``) over
      a module file (``tasks.py``) in the same location.
    * If a candidate is found and successfully imported, its parent directory
      will **stay** on ``sys.path`` during the rest of the Python session --
      this allows task code to make convenient assumptions concerning sibling
      modules' importability.

Candidate modules/packages are introspected to make sure they can actually be
used as valid task collections. Any that fail are discarded, the ``sys.path``
munging done to import them is reverted, and the search continues.


.. _configuring-loading:

Configuring the loading process
===============================

You can configure the above behavior, requesting that Invoke alter the
collection name searched for and/or the path where filesystem-level loading
starts looking.

For example, you may already have a project-level ``tasks.py`` that you can't
easily rename; or you may want to host a number of tasks collections stored
outside the project root and make it easy to switch between them; or any number
of reasons.

Both the sought collection name and the search root can be specified via
:ref:`configuration file options <config-files>` or as :doc:`runtime CLI flags
`:

- **Change the collection name**: Set the ``tasks.collection_name``
  configuration option, or use :option:`--collection`. It should be a Python
  module name and not a file name (so ``mytasks``, not ``mytasks.py`` or
  ``mytasks/``.)
- **Change the root search path**: Configure ``tasks.search_root`` or use
  :option:`--search-root`. This value may be any valid directory path.

---

### Sites Docs Concepts Namespaces (`sites/docs/concepts/namespaces.rst`)

.. _task-namespaces:

=======================
Constructing namespaces
=======================

The :doc:`base case </getting-started>` of loading a single module of tasks
works fine initially, but advanced users typically need more organization, such
as separating tasks into a tree of nested namespaces.

The `.Collection` class provides an API for organizing tasks (and :ref:`their
configuration <collection-configuration>`) into a tree-like structure. When
referenced by strings (e.g. on the CLI or in pre/post hooks) tasks in nested
namespaces use a dot-separated syntax, e.g. ``docs.build``.

In this section, we show how building namespaces with this API is flexible but
also allows following Python package layouts with minimal boilerplate.


Starting out
============

One unnamed ``Collection`` is always the namespace root; in the implicit base
case, Invoke creates one for you from the tasks in your tasks module.  Create
your own, named ``namespace`` or ``ns``, to set up an explicit namespace (i.e.
to skip the default "pull in all Task objects" behavior)::

    from invoke import Collection

    ns = Collection()
    # or: namespace = Collection()

Add tasks with `.Collection.add_task`. `~.Collection.add_task` can take an
`.Task` object, such as those generated by the `.task` decorator::

    from invoke import Collection, task

    @task
    def release(c):
        c.run("python setup.py sdist register upload")

    ns = Collection()
    ns.add_task(release)

Our available tasks list now looks like this::

    $ invoke --list
    Available tasks:

        release


Naming your tasks
=================

By default, a task's function name is used as its namespace identifier, but you
may override this by giving a ``name`` argument to either `@task <.task>` (i.e.
at definition time) or `.Collection.add_task` (i.e. at binding/attachment
time).

For example, say you have a variable name collision in your tasks module --
perhaps you want to expose a ``dir`` task, which shadows a Python builtin.
Naming your function itself ``dir`` is a bad idea, but you can name the
function something like ``dir_`` and then tell ``@task`` the "real" name::

    @task(name='dir')
    def dir_(c):
        # ...

On the other side, you might have obtained a task object that doesn't fit with
the names you want in your namespace, and can rename it at attachment time.
Maybe we want to rename our ``release`` task to be called ``deploy`` instead::

    ns = Collection()
    ns.add_task(release, name='deploy')

The result::

    $ invoke --list
    Available tasks:

        deploy

.. note::
    The ``name`` kwarg is the 2nd argument to `~.Collection.add_task`, so those
    in a hurry can phrase it as::

        ns.add_task(release, 'deploy')

Aliases
-------

Tasks may have additional names or aliases, given as the ``aliases`` keyword
argument; these are appended to, instead of replacing, any implicit or explicit
``name`` value::

    ns.add_task(release, aliases=('deploy', 'pypi'))

Result, with three names for the same task::

    $ invoke --list
    Available tasks:

        release
        deploy
        pypi

.. note::
    The convenience decorator `@task <.task>` is another method of
    setting aliases (e.g. ``@task(aliases=('foo', 'bar'))``, and is useful for
    ensuring a given task always has some aliases set no matter how it's added
    to a namespace.

.. _dashes-vs-underscores:

Dashes vs underscores
---------------------

In the common case of functions-as-tasks, you'll often find yourself writing
task names that contain underscores::

    @task
    def my_awesome_task(c):
        print("Awesome!")

Similar to how task arguments are processed to turn their underscores into
dashes (since that's a common command-line convention) all underscores in task
or collection names are interpreted to be dashes instead, by default::

    $ inv --list
    Available tasks:

      my-awesome-task

    $ inv my-awesome-task
    Awesome!

If you'd prefer the underscores to remain instead, you can update your
configuration to set ``tasks.auto_dash_names`` to ``False`` in one of the
non-runtime :ref:`config files <config-files>` (system, user, or project.) For
example, in ``~/.invoke.yml``::

    tasks:
        auto_dash_names: false

.. note::
    In the interests of avoiding confusion, this setting is "exclusive" in
    nature - underscored version of task names *are not valid* on the CLI
    unless ``auto_dash_names`` is disabled. (However, at the pure function
    level within Python, they must continue to be referenced with underscores,
    as dashed names are not valid Python syntax!)


Nesting collections
===================

The point of namespacing is to have sub-namespaces; to do this in Invoke,
create additional `.Collection` instances and add them to their parent
collection via `.Collection.add_collection`. For example, let's say we have a
couple of documentation tasks::

    @task
    def build_docs(c):
        c.run("sphinx-build docs docs/_build")

    @task
    def clean_docs(c):
        c.run("rm -rf docs/_build")

We can bundle them up into a new, named collection like so::

    docs = Collection('docs')
    docs.add_task(build_docs, 'build')
    docs.add_task(clean_docs, 'clean')

And then add this new collection under the root namespace with
``add_collection``::

    ns.add_collection(docs)

The result (assuming for now that ``ns`` currently just contains the original
``release`` task)::

    $ invoke --list
    Available tasks:

        release
        docs.build
        docs.clean

As with tasks, collections may be explicitly bound to their parents with a
different name than they were originally given (if any) via a ``name`` kwarg
(also, as with ``add_task``, the 2nd regular arg)::

    ns.add_collection(docs, 'sphinx')

Result::

    $ invoke --list
    Available tasks:

        release
        sphinx.build
        sphinx.clean


Importing modules as collections
================================

A simple tactic which Invoke itself uses in the trivial, single-module
case is to use `.Collection.from_module` -- a classmethod
serving as an alternate ``Collection`` constructor which takes a Python module
object as its first argument.

Modules given to this method are scanned for ``Task`` instances, which are
added to a new ``Collection``. By default, this collection's name is taken from
the module name (the ``__name__`` attribute), though it can also be supplied
explicitly.

.. note::
    As with the default task module, you can override this default loading
    behavior by declaring a ``ns`` or ``namespace`` `.Collection` object at top
    level in the loaded module.

For example, let's reorganize our earlier single-file example into a Python
package with several submodules. First, ``tasks/release.py``::

    from invoke import task

    @task
    def release(c):
        c.run("python setup.py sdist register upload")

And ``tasks/docs.py``::

    from invoke import task

    @task
    def build(c):
        c.run("sphinx-build docs docs/_build")

    @task
    def clean(c):
        c.run("rm -rf docs/_build")

Tying them together is ``tasks/__init__.py``::

    from invoke import Collection

    import release, docs

    ns = Collection()
    ns.add_collection(Collection.from_module(release))
    ns.add_collection(Collection.from_module(docs))

This form of the API is a little unwieldy in practice. Thankfully there's a
shortcut: ``add_collection`` will notice when handed a module object as its
first argument and call ``Collection.from_module`` for you internally::

    ns = Collection()
    ns.add_collection(release)
    ns.add_collection(docs)

Either way, the result::

    $ invoke --list
    Available tasks:

        release.release
        docs.build
        docs.clean


Default tasks
=============

Tasks may be declared as the default task to invoke for the collection they
belong to, e.g. by giving ``default=True`` to `@task <.task>` (or to
`.Collection.add_task`.) This is useful when you have a bunch of related tasks
in a namespace but one of them is the most commonly used, and maps well to the
namespace as a whole.

For example, in the documentation submodule we've been experimenting with so
far, the ``build`` task makes sense as a default, so we can say things like
``invoke docs`` as a shortcut to ``invoke docs.build``. This is easy to do::

    @task(default=True)
    def build(c):
        # ...

When imported into the root namespace (as shown above) this alters the output
of ``--list``, highlighting the fact that ``docs.build`` can be invoked as
``docs`` if desired::

    $ invoke --list
    Available tasks:

        release.release
        docs.build (docs)
        docs.clean

Default subcollections
----------------------

As of version 1.5, this functionality is also extended to subcollections: a
subcollection can be specified as the default when being added to its parent
collection, and that subcollection's own default task (or sub-subcollection!)
will be invoked as the default for the parent.

An example probably makes that clearer. Here's a tiny inline task tree with two
subcollections, each with their own default task::

    from invoke import Collection, task

    @task(default=True)
    def build_all(c):
        print("build ALL THE THINGS!")

    @task
    def build_wheel(c):
        print("Just the wheel")

    build = Collection(all=build_all, wheel=build_wheel)

    @task(default=True)
    def build_docs(c):
        print("Code without docs is no code at all")

    docs = Collection(build_docs)

Then we tie those into one top level collection, setting the ``build``
subcollection as the overall default::

    ns = Collection()
    ns.add_collection(build, default=True)
    ns.add_collection(docs)

The result is that ``build.all`` becomes the absolute default task::

    $ invoke
    build ALL THE THINGS!

Mix and match
=============

You're not limited to the specific tactics shown above -- now that you know
the basic tools of ``add_task`` and ``add_collection``, use whatever approach
best fits your needs.

For example, let's say you wanted to keep things organized into submodules, but
wanted to "promote" ``release.release`` back to the top level for convenience's
sake. Just because it's stored in a module doesn't mean we must use
``add_collection`` -- we could instead import the task itself and use
``add_task`` directly::

    from invoke import Collection

    import docs
    from release import release

    ns = Collection()
    ns.add_collection(docs)
    ns.add_task(release)

Result::

    $ invoke --list
    Available tasks:

        release
        docs.build
        docs.clean


More shortcuts
==============

Finally, you can even skip ``add_collection`` and ``add_task`` if your needs
are simple enough -- `.Collection`'s constructor will take
unknown arguments and build the namespace from their values as
appropriate::

    from invoke import Collection

    import docs, release

    ns = Collection(release.release, docs)

Notice how we gave both a task object (``release.release``) and a module
containing tasks (``docs``). The result is identical to the above::

    $ invoke --list
    Available tasks:

        release
        docs.build
        docs.clean

If given as keyword arguments, the keywords act like the ``name`` arguments do
in the ``add_*`` methods. Naturally, both can be mixed together as well::

    ns = Collection(docs, deploy=release.release)

Result::

    $ invoke --list
    Available tasks:

        deploy
        docs.build
        docs.clean

.. note::
    You can still name these ``Collection`` objects with a leading string
    argument if desired, which can be handy when building sub-collections.

---

### Sites Docs Concepts Testing (`sites/docs/concepts/testing.rst`)

.. _testing-user-code:

==============================
Testing Invoke-using codebases
==============================

Strategies for testing codebases that use Invoke; some applicable to code
focused on CLI tasks, and others applicable to more generic/refactored setups.


Subclass & modify Invoke 'internals'
====================================

A quick foreword: most users will find the subsequent approaches suitable, but
advanced users should note that Invoke has been designed so it is itself easily
testable. This means that in many cases, even Invoke's "internals" are exposed
as low/no-shared-responsibility, publicly documented classes which can be
subclassed and modified to inject test-friendly values or mocks. Be sure to
look over the :ref:`API documentation <api>`!


Use `.MockContext`
==================

An instance of subclassing Invoke's public API for test purposes is our own
`.MockContext`. Codebases which revolve heavily around `.Context` objects and
their methods (most task-oriented code) will find it easy to test by injecting
`.MockContext` objects which have been instantiated to yield partial `.Result`
objects.

For example, take this task::

    from invoke import task

    @task
    def get_platform(c):
        uname = c.run("uname -s").stdout.strip()
        if uname == 'Darwin':
            return "You paid the Apple tax!"
        elif uname == 'Linux':
            return "Year of Linux on the desktop!"

An example of testing it with `.MockContext` could be the following::

    from invoke import MockContext, Result
    from mytasks import get_platform

    def test_get_platform_on_mac():
        c = MockContext(run=Result("Darwin\n"))
        assert "Apple" in get_platform(c)

    def test_get_platform_on_linux():
        c = MockContext(run=Result("Linux\n"))
        assert "desktop" in get_platform(c)

Putting the ``Mock`` in `.MockContext`
--------------------------------------

Starting in Invoke 1.5, `.MockContext` will attempt to import the ``mock``
library at instantiation time and wrap its methods within ``Mock`` objects.
This lets you not only present realistic return values to your code, but make
test assertions about what commands your code is running.

Here's another "platform sensitive" task::

    from invoke import task

    @task
    def replace(c, path, search, replacement):
        # Assume systems have sed, and that some (eg macOS w/ Homebrew) may
        # have gsed, implying regular sed is BSD style.
        has_gsed = c.run("which gsed", warn=True, hide=True)
        # Set command to run accordingly
        binary = "gsed" if has_gsed else "sed"
        c.run(f"{binary} -e 's/{search}/{replacement}/g' {path}")

The test code (again, which presumes that eg ``MockContext.run`` is now a
``Mock`` wrapper) relies primarily on 'last call' assertions
(``Mock.assert_called_with``) but you can of course use any ``Mock`` methods
you need. It also shows how you can set the mock context to respond to multiple
possible commands, using a dict value::

    from invoke import MockContext, Result
    from mytasks import replace

    def test_regular_sed():
        expected_sed = "sed -e s/foo/bar/g file.txt"
        c = MockContext(run={
            "which gsed": Result(exited=1),
            expected_sed: Result(),
        })
        replace(c, 'file.txt', 'foo', 'bar')
        c.run.assert_called_with(expected_sed)

    def test_homebrew_gsed():
        expected_sed = "gsed -e s/foo/bar/g file.txt"
        c = MockContext(run={
            "which gsed": Result(exited=0),
            expected_sed: Result(),
        })
        replace(c, 'file.txt', 'foo', 'bar')
        c.run.assert_called_with(expected_sed)

Boolean mock results
--------------------

You may have noticed the above example uses a handful of 'empty' `.Result`
objects; these stand in for "succeeded, but otherwise had no useful attributes"
command executions (as `.Result` defaults to an exit code of ``0`` and empty
strings for stdout/stderr).

This is relatively common - think "interrogative" commands where the caller
only cares for a boolean result, or times when a command is called purely for
its side effects. To support this, there's a shorthand in `.MockContext`:
passing ``True`` or ``False`` to stand in for otherwise blank Results with exit
codes of ``0`` or ``1`` respectively.

The example tests then look like this::

    from invoke import MockContext, Result
    from mytasks import replace

    def test_regular_sed():
        expected_sed = "sed -e s/foo/bar/g file.txt"
        c = MockContext(run={
            "which gsed": False,
            expected_sed: True,
        })
        replace(c, 'file.txt', 'foo', 'bar')
        c.run.assert_called_with(expected_sed)

    def test_homebrew_gsed():
        expected_sed = "gsed -e s/foo/bar/g file.txt"
        c = MockContext(run={
            "which gsed": True,
            expected_sed: True,
        })
        replace(c, 'file.txt', 'foo', 'bar')
        c.run.assert_called_with(expected_sed)

String mock results
-------------------

Another convenient shorthand is using string values, which are interpreted to
be the stdout of the resulting `.Result`. This only really saves you from
writing out the class itself (since ``stdout`` is the first positional arg of
`.Result`!) but "command X results in stdout Y" is a common enough use case
that we implemented it anyway.

By example, let's modify an earlier example where we cared about stdout::

    from invoke import MockContext
    from mytasks import get_platform

    def test_get_platform_on_mac():
        c = MockContext(run="Darwin\n")
        assert "Apple" in get_platform(c)

    def test_get_platform_on_linux():
        c = MockContext(run="Linux\n")
        assert "desktop" in get_platform(c)

As with everything else in this document, this tactic can be applied to
iterators or mappings as well as individual values.

Regular expression command matching
-----------------------------------

The dict form of `.MockContext` kwarg can accept regular expression objects as
keys, in addition to strings; ideal for situations where you either don't know
the exact command being invoked, or simply don't need or want to write out the
entire thing.

Imagine you're writing a function to run package management commands on a few
different Linux distros and you're trying to test its error handling. You might
want to set up a context that pretends any arbitrary ``apt`` or ``yum`` command
fails, and ensure the function returns stderr when it encounters a problem::

    import re
    from invoke import MockContext
    from mypackage.tasks import install

    package_manager = re.compile(r"^(apt(-get)?|yum) .*")

    def test_package_success_returns_True():
        c = MockContext(run={package_manager: True})
        assert install(c, package="somepackage") is True

    def test_package_explosions_return_stderr():
        c = MockContext(run={
            package_manager: Result(stderr="oh no!", exited=1),
        })
        assert install(c, package="otherpackage") == "oh no!"

A bit contrived - there are a bunch of other ways to organize this exact test
code so you don't truly need the regex - but hopefully it's clear that when you
*do* need this flexibility, this is how you could go about it.

Repeated results
----------------

By default, the values in these mock structures are consumed, causing
`.MockContext` to raise ``NotImplementedError`` afterwards (as it does for any
unexpected command executions). This was designed with the assumption that
most code under test will run a given command once.

If your situation doesn't match this, give ``repeat=True`` to the constructor,
and you'll see values repeat indefinitely instead (or in cycles, for
iterables).


Expect `Results <.Result>`
==========================

The core Invoke subprocess methods like `~.Context.run` all return `.Result`
objects - which (as seen above) can be readily instantiated by themselves with
only partial data (e.g. standard output, but no exit code or standard error).

This means that well-organized code can be even easier to test and doesn't
require as much use of `.MockContext`.

An iteration on the initial `.MockContext`-using example above::

    from invoke import task

    @task
    def get_platform(c):
        print(platform_response(c.run("uname -s")))

    def platform_response(result):
        uname = result.stdout.strip()
        if uname == 'Darwin':
            return "You paid the Apple tax!"
        elif uname == 'Linux':
            return "Year of Linux on the desktop!"

With the logic encapsulated in a subroutine, you can just unit test that
function by itself, deferring testing of the task or its context::

    from invoke import Result
    from mytasks import platform_response

    def test_platform_response_on_mac():
        assert "Apple" in platform_response(Result("Darwin\n"))

    def test_platform_response_on_linux():
        assert "desktop" in platform_response(Result("Linux\n"))


Avoid mocking dependency code paths altogether
==============================================

This is more of a general software engineering tactic, but the natural endpoint
of the above code examples would be where your primary logic doesn't care about
Invoke at all -- only about basic Python (or locally defined) data types. This
allows you to test logic in isolation and either ignore testing the Invoke side
of things, or write targeted tests solely for where your code interfaces with
Invoke.

Another minor tweak to the task code::

    from invoke import task

    @task
    def show_platform(c):
        uname = c.run("uname -s").stdout.strip()
        print(platform_response(uname))

    def platform_response(uname):
        if uname == 'Darwin':
            return "You paid the Apple tax!"
        elif uname == 'Linux':
            return "Year of Linux on the desktop!"

And the tests::

    from mytasks import platform_response

    def test_platform_response_on_mac():
        assert "Apple" in platform_response("Darwin\n")

    def test_platform_response_on_linux():
        assert "desktop" in platform_response("Linux\n")

---

### Sites Docs Concepts Watchers (`sites/docs/concepts/watchers.rst`)

.. _autoresponding:

==========================================
Automatically responding to program output
==========================================

Background
==========

Command-line programs tend to be designed for interactive shells, which
frequently manifests as waiting around for user input, or "prompts".
Well-designed programs offer options for pre-empting such prompts, resulting in
an easily automated workflow -- but with the rest, interactivity is
unavoidable.

Thankfully, Invoke's `.Runner` class not only forwards your standard input to
the running program (allowing you to manually respond to prompts) but it can
also be configured to respond automatically on your behalf.

Basic use
=========

The mechanism for this automation is the ``watchers`` kwarg to the
`.Runner.run` method (and its wrappers elsewhere, such as `.Context.run` and
`invoke.run`), which is a list of `.StreamWatcher`-subclass instances
configured to watch for patterns & respond accordingly. The simplest of these
is `.Responder`, which just replies with its configured response every time its
pattern is seen; others can be found in the :doc:`watchers module
</api/watchers>`.

.. note::
    As with all other arguments to ``run``, you can also set the default set of
    watchers globally via :doc:`configuration files <configuration>`.

Take for example this program which expects a manual response to a yes/no
prompt::

    $ excitable-program
    When you give the OK, I'm going to do the things. All of them!!
    Are you ready? [Y/n] y
    OK! I just did all sorts of neat stuff. You're welcome! Bye!

You *could* call ``run("excitable-program")``, manually watch for the
prompt, and mash Y by hand. But if you instead supply a `.Responder` like so::

    @task
    def always_ready(c):
        responder = Responder(
            pattern=r"Are you ready? \[Y/n\] ",
            response="y\n",
        )
        c.run("excitable-program", watchers=[responder])

Then `.Runner` passes the program's ``stdout`` and ``stderr`` through
``responder``, which watches for ``"Are you ready? [Y/n] "`` and automatically
writes ``y`` (plus ``\n`` to simulate hitting Enter/Return) to the program's
``stdin``.

.. note::
    The pattern argument to `.Responder` is treated as a `regular expression
    <re>`, requiring more care (note how we had to escape our square-brackets
    in the above example) but providing more power as well.

---

--- METRICS ---
- Files Extracted: 25
- Estimated Token Budget: ~24498 tokens
- Canonical Reference: https://codewiki.google/github.com/pyinvoke/invoke