Technical Documentation: pyinvoke/invoke
βΉοΈ Provenance: Multi-Tier Ingestion Β· CodeWiki Reference
Sites Docs .Readthedocs.Yaml (`sites/docs/.readthedocs.yaml`)
version: 2
build:
os: "ubuntu-24.04"
tools:
python: "3.9"
jobs:
install:
- pip install pip~=25.1
- pip install . --group dev
sphinx:
configuration: sites/docs/conf.py
fail_on_warning: true
Sites Docs Getting Started (`sites/docs/getting-started.rst`)
=============== Getting started
This document presents a whirlwind tour of Invoke's feature set. Please see the
links throughout for detailed conceptual & API docs. For installation help, see
the project's installation page <https://www.pyinvoke.org/installing.html>_.
.. _defining-and-running-task-functions:
Defining and running task functions
The core use case for Invoke is setting up a collection of task functions and
executing them. This is pretty easy -- all you need is to make a file calledtasks.py importing the .task decorator and decorating one or more
functions. You will also need to add an arbitrarily-named context argument
(convention is to use c, ctx or context) as the first positional
arg. Don't worry about using this context parameter yet.
Let's start with a dummy Sphinx docs building task::
from invoke import task
@task
def build(c):
print("Building!")You can then execute that new task by telling Invoke's command line runner,invoke, that you want it to run::
$ invoke build
Building!The function body can be any Python you want -- anything at all.
Task parameters
Functions can have arguments, and thus so can tasks. By default, your task
functions' args/kwargs are mapped automatically to both long and short CLI
flags, as per :ref:the CLI docs <task-arguments>. For example, if we add aclean argument and give it a boolean default, it will show up as a set of
toggle flags, --clean and -c::
@task
def build(c, clean=False):
if clean:
print("Cleaning!")
print("Building!")Invocations::
$ invoke build -c
$ invoke build --cleanNaturally, other default argument values will allow giving string or integer
values. Arguments with no default values are assumed to take strings, and can
also be given as positional arguments. Take this incredibly contrived snippet
for example::
@task
def hi(c, name):
print("Hi {}!".format(name))It can be invoked in the following ways, all resulting in "Hi Name!"::
$ invoke hi Name
$ invoke hi --name Name
$ invoke hi --name=Name
$ invoke hi -n Name
$ invoke hi -nNameAdding metadata via `@task <.task>`
@task <.task> can be used without any arguments, as above, but it's also a
convenient vector for additional metadata about the task function it decorates.
One common example is describing the task's arguments, via the help
parameter (in addition to optionally giving task-level help via the
docstring)::
@task(help={'name': "Name of the person to say hi to."})
def hi(c, name):
"""
Say hi to someone.
"""
print("Hi {}!".format(name))This description will show up when invoking --help::
$ invoke --help hi
Usage: inv[oke] [--core-opts] hi [--options] [other tasks here ...]
Docstring:
Say hi to someone.
Options:
-n STRING, --name=STRING Name of the person to say hi to.More details on task parameterization and metadata can be found in
:doc:/concepts/invoking-tasks (for the command-line & parsing side of things)
and in the .task API documentation (for the declaration side).
Listing tasks
You'll sometimes want to see what tasks are available in a giventasks.py -- invoke can be told to list them instead of executing
something::
$ invoke --list
Available tasks:
buildThis will also print the first line of each taskβs docstring, if it has one. To
see what else is available besides --list, say invoke --help.
Running shell commands
Many use cases for Invoke involve running local shell commands, similar to
programs like Make or Rake. This is done via the ~.Context.run function::
from invoke import task
@task
def build(c):
c.run("sphinx-build docs docs/_build")You'll see the command's output in your terminal as it runs::
$ invoke build
Running Sphinx v1.1.3
loading pickled environment... done
...
build succeeded, 2 warnings.~.Context.run has a number of arguments controlling its behavior, such as
activation of pseudo-terminals for complex programs requiring them, suppression
of exit-on-error behavior, hiding of subprocess' output (while still capturing
it for later review), and more. See its API docs <.Context.run> for details.
~.Context.run always returns a useful .Result object providing access to
the captured output, exit code, and other information.
.. _why-context:
Aside: what exactly is this 'context' arg anyway?
A common problem task runners face is transmission of "global" data - values
loaded from :doc:configuration files </concepts/configuration> or :ref:other configuration vectors <collection-configuration>, given via CLI flags,
generated in 'setup' tasks, etc.
Some libraries (such as Fabric <https://fabfile.org>_ 1.x) implement this via
module-level attributes, which makes testing difficult and error prone, limits
concurrency, and increases implementation complexity.
Invoke encapsulates state in explicit ~.Context objects, handed to tasks when
they execute . The context is the primary API endpoint, offering methods which
honor the current state (such as .Context.run) as well as access to that
state itself.
Declaring pre-tasks
Tasks may be configured in a number of ways via the .task decorator. One of
these is to select one or more other tasks you wish to always run prior to
execution of your task, indicated by name.
Let's expand our docs builder with a new cleanup task that runs before every
build (but which, of course, can still be executed on its own)::
from invoke import task
@task
def clean(c):
c.run("rm -rf docs/_build")
@task(clean)
def build(c):
c.run("sphinx-build docs docs/_build")Now when you invoke build, it will automatically run clean first.
.. note::
If you're not a fan of the implicit "positional arguments are pre-run task
names" API, you can instead explicitly give the pre kwarg:
@task(pre=[clean]).
Details can be found in :ref:how-tasks-run.
Creating namespaces
Right now, our tasks.py is implicitly for documentation only, but maybe our
project needs other non-doc things, like packaging/deploying, testing, etc. At
that point, a single flat namespace isn't enough, so Invoke lets you easily
build a :doc:nested namespace <concepts/namespaces>. Here's a quick example.
Let's first rename our tasks.py to be docs.py; no other changes are
needed there. Then we create a new tasks.py, and for the sake of brevity
populate it with a new, truly top level task called deploy.
Finally, we can use a new API member, the .Collection class, to bind this
task and the docs module into a single explicit namespace. When Invoke
loads your task module, if a .Collection object bound as ns ornamespace exists it will get used for the root namespace::
from invoke import Collection, task
import docs
@task
def deploy(c):
c.run("python setup.py sdist")
c.run("twine upload dist/*")
namespace = Collection(docs, deploy)The result::
$ invoke --list
Available tasks:
deploy
docs.build
docs.cleanFor a more detailed breakdown of how namespacing works, please see :doc:the docs <concepts/namespaces>.
Sites Docs Index (`sites/docs/index.rst`)
================================== Welcome to Invoke's documentation!
This site covers Invoke's conceptual & API documentation. For basic info on
what Invoke is, including its public changelog & how the project is maintained,
please see the main project website <https://pyinvoke.org>_.
Getting started
Many core ideas & API calls are explained in the tutorial/getting-started
document:
.. toctree::
:maxdepth: 2
getting-startedThe ``invoke`` CLI tool
Details on the CLI interface to Invoke, available core flags, and tab
completion options.
.. toctree::
invoke
Concepts
Dig deeper into specific topics:
.. toctree::
:maxdepth: 2
:glob:
concepts/*.. _api:
API
Know what you're looking for & just need API details? View our auto-generated
API documentation:
.. toctree::
:maxdepth: 1
:glob:
api/*Sites Docs Invoke (`sites/docs/invoke.rst`)
.. _inv:
======================== ``inv[oke]`` core usage
.. seealso::
This page documents invoke's core arguments, options and behavior
(which includes options present in :ref:custom Invoke-based binaries <reusing-as-a-binary>). For details on invoking user-specified tasks and
other parser-related details, see :doc:/concepts/invoking-tasks.
Core options and flags
invoke's usage looks like::
$ inv[oke] [--core-opts] task1 [--task1-opts] ... taskN [--taskN-opts] [--
remainder-opts-or-values]All core options & flags are below; almost all of them must be given before
any task names, with a few (such as :option:--help) being specially looked
for anywhere in the command line. (For parsing details, see
:ref:basic-cli-layout.)
.. option:: --complete
Print (line-separated) valid tab-completion options for an Invoke command
line given as the 'remainder' (i.e. after a ``--``). Used for building
:ref:`shell completion scripts <tab-completion>`.
For example, when the local tasks tree contains tasks named ``foo`` and
``bar``, and when ``foo`` takes flags ``--foo-arg`` and ``--foo-arg-2``,
you might use it like this::
# Empty input: just task names
$ inv --complete --
foo
bar
# Input not ending with a dash: task names still
$ inv --complete -- foo --foo-arg
foo
bar
# Input ending with a dash: current context's flag names
$ inv --complete -- foo -
--foo-arg
--foo-arg-2
For more details on how to make best use of this option, see
:option:`--print-completion-script`... option:: --hide=STRING
Set default value of run()'s 'hide' kwarg... option:: --no-dedupe
Disable task deduplication... _print-completion-script:
.. option:: --print-completion-script=SHELL
Print a completion script for desired ``SHELL`` (e.g. ``bash``, ``zsh``,
etc). This can be sourced into the current session in order to enjoy
:ref:`tab-completion for tasks and options <tab-completion>`.
These scripts are bundled with Invoke's distributed codebase, and
internally make use of :option:`--complete`... _prompt-for-sudo-password:
.. option:: --prompt-for-sudo-password
Prompt at the start of the session (before executing any tasks) for the
``sudo.password`` configuration value. This allows users who don't want to
keep sensitive material in the config system or their shell environment to
rely on user input, without otherwise interrupting the flow of the program... option:: --write-pyc
By default, Invoke disables bytecode caching as it can cause hard-to-debug
problems with task files and (for the kinds of things Invoke is typically
used for) offers no noticeable speed benefit. If you really want your
``.pyc`` files back, give this option... option:: -c STRING, --collection=STRING
Specify collection name to load... option:: -d, --debug
Enable debug output... option:: --dry
Echo commands instead of actually running them; specifically, causes any
``run`` calls to:
- Act as if the ``echo`` option has been turned on, printing the
command-to-be-run to stdout;
- Skip actual subprocess invocation (returning before any of that machinery
starts running);
- Return a dummy `~invoke.runners.Result` object with 'blank' values (empty
stdout/err strings, ``0`` exit code, etc)... option:: -D, --list-depth=INT
Limit :option:`--list` display to the specified number of levels, e.g.
``--list-depth 1`` to show only top-level tasks and namespaces.
If an argument is given to ``--list``, then this depth is relative; so
``--list build --list-depth 1`` shows everything at the top level of the
``build`` subtree.
Default behavior if this is not given is to show all levels of the entire
task tree... option:: -e, --echo
Echo executed commands before running... option:: -f, --config
Specify a :ref:`runtime configuration file <config-hierarchy>` to load.
Note that you may instead use the ``INVOKE_RUNTIME_CONFIG`` environment
variable in place of this option. If both are given, the CLI option will
win out... option:: -F, --list-format=STRING
Change the format used to display the output of :option:`--list`; may be
one of:
- ``flat`` (the default): single, flat vertical list with dotted task
names.
- ``nested``: a nested (4-space indented) vertical list, where each level
implicitly includes its parent (with leading dots as a strong visual clue
that these are still subcollection tasks.)
- ``json``: intended for consumption by scripts or other programs, this
format emits JSON representing the task tree, with each 'node' in the
tree (the outermost document being the root node, and thus a JSON object)
consisting of the following keys:
- ``name``: String name of collection; for the root collection this is
typically the module name, so unless you're supplying alternate
collection name to the load process, it's usually ``"tasks"`` (from
``tasks.py``.)
- ``help``: First line of collection's docstring, if it came from a
module; null otherwise (or if module lacked a docstring.)
- ``tasks``: Immediate children of this collection; an array of objects
of the following form:
- ``name``: Task's local name within its collection (i.e. not the full
dotted path you might see with the ``flat`` format; reconstructing
that path is left up to the consumer.)
- ``help``: First line of task's docstring, or null if it had none.
- ``aliases``: An array of string aliases for this task.
- ``default``: String naming which task within this collection, if any,
is the default task. Is null if no task is the default.
- ``collections``: An array of any sub-collections within this
collection, members of which which will have the same structure as this
outermost document, recursively.
The JSON emitted is not pretty-printed, but does end with a trailing
newline... option:: -h STRING, --help=STRING
When given without any task names, displays core help; when given with a
task name (may come before *or* after the task name) displays help for that
particular task... option:: -l, --list=STRING
List available tasks. Shows all tasks by default; may give an explicit
namespace to 'root' the displayed task tree to only that namespace. (This
argument may contain periods, as with task names, so it's possible to show
only a small, deep portion of the overall tree if desired.).. option:: -p, --pty
Use a pty when executing shell commands... option:: -r STRING, --search-root=STRING
Change root directory used for finding task modules... option:: -T INT, --command-timeout=INT
Set a default command execution timeout of INT seconds. Maps to the
``timeouts.command`` config setting... option:: -V, --version
Show version and exit... option:: -w, --warn-only
Warn, instead of failing, when shell commands fail... _remainder:
Unparsed input: the 'remainder'
Invoke takes a page from the GNU command-line tradition and many shell
builtins, and implements the concept of the remainder (aka the "end of
options" signal): a 'bare' -- not attached to any option name, followed by
arbitrary text.
Such text is not subject to Invoke's CLI parser, and is used to power features
like :ref:tab-completion or :ref:wrapper-tasks
.. _tab-completion:
Shell tab completion
Generating a completion script
Invoke's philosophy is to implement generic APIs and then "bake in" a few
common use cases built on top of those APIs; tab completion is no different.
Generic tab completion functionality (outputting a shell-compatible list of
completion tokens for a given command line context) is provided by the
:option:--complete core CLI option described above.
However, you probably won't need to use that flag yourself: we distribute a
handful of ready-made wrapper scripts aimed at the most common shells likebash and zsh (plus others). These scripts can be automatically
generated from Invoke or :ref:any Invoke-driven command-line tool <reusing-as-a-binary>, using :option:--print-completion-script; the printed
scripts will contain the correct binary name(s) for the program generating
them.
For example, the following command prints (to stdout) a script which works forzsh, instructs zsh to use it for the inv and invoke programs,
and calls invoke --complete at runtime to get dynamic completion
information::
$ invoke --print-completion-script zsh.. note::
You'll probably want to source this command or store its output somewhere
permanently; more on that in the next section.
Similarly, the Fabric <https://fabfile.org>_ tool inherits from Invoke, and
only has a single binary name (fab); if you wanted to get Fabric completion
in bash, you would say::
$ fab --print-completion-script bashIn the rest of this section, we'll use inv in examples, but please remember
to replace it with the program you're actually using, if it's not Invoke
itself!
Sourcing the script
There are a few ways to utilize the output of the above commands, depending on
your needs, where the program is installed, and your shell:
The simplest and least disruptive method is to
sourcethe printed
completion script inline, which doesn't place anything on disk, and will only
affect the current shell session::$ source <(inv --print-completion-script zsh)
If you've got the program available in your system's global Python
interpreter (and you're okay with running the program at the startup of each
shell session - Python's speed is admittedly not its strong point) you could
add that snippet to your shell's startup file, such as~/.zshrcor~/.bashrc.If the program's available globally but you'd prefer to avoid running an
extra Python program at shell startup, you can cache the output of the
command in its own file; where this file lives is entirely up to you and how
your shell is configured. For example, you might just drop it into your home
directory as a hidden file::$ inv --print-completion-script zsh > ~/.invoke-completion.sh
and then perhaps add the following to the end of
~/.zshrc::source ~/.invoke-completion.sh
But again, this is entirely up to you and your shell.
.. note::
If you're usingfish, you must use this tactic, as our fish
completion script is not suitable for direct sourcing. Fish shell users
should direct the output of the command to a file in the
~/.config/fish/completions/directory.Finally, if your copy of the needing-completion program is only installed in
a specific environment like a virtualenv, you can use either of the above
techniques:- Caching the output and referencing it in a global shell startup file will
still work in this case, as it does not require the program to be
available when the shell loads -- only when you actually attempt to tab
complete. - Using the
source <(inv --print-completion-script yourshell)approach
will work as long as you place it in some appropriate per-environment
startup file, which will vary depending on how you manage Python
environments. For example, if you usevirtualenvwrapper, you could
append thesourceline in/path/to/virtualenv/bin/postactivate.
- Caching the output and referencing it in a global shell startup file will
Utilizing tab completion itself
You've ensured that the completion script is active in your environment - what
have you gained?
By default, tabbing after typing
invorinvokewill display task
names from your current directory/project's tasks file.Tabbing after typing a dash (
-) or double dash (--) will display
valid options/flags for the current context: core Invoke options if no task
names have been typed yet; options for the most recently typed task
otherwise.- Tabbing while typing a partial long option will complete matching long
options, using your shell's native substring completion. E.g. if no task
names have been typed yet,--e<tab>will offer--echoas a
completion option.
- Tabbing while typing a partial long option will complete matching long
Hitting tab when the most recent typed/completed token is a flag which takes
a value, will 'fall through' to your shell's native filename completion.- For example, prior to typing a task name,
--config <tab>will
complete local file paths to assist in filling in a config file.
- For example, prior to typing a task name,
Sites Docs Api Init (`sites/docs/api/__init__.rst`)
============ ``__init__``
.. automodule:: invoke
Sites Docs Api Collection (`sites/docs/api/collection.rst`)
============== ``collection``
.. autoclass:: invoke.collection.Collection
:special-members:
:exclude-members: weakref, init
Sites Docs Api Config (`sites/docs/api/config.rst`)
========== ``config``
.. automodule:: invoke.config
Sites Docs Api Context (`sites/docs/api/context.rst`)
=========== ``context``
.. automodule:: invoke.context
Sites Docs Api Exceptions (`sites/docs/api/exceptions.rst`)
============== ``exceptions``
.. automodule:: invoke.exceptions
Sites Docs Api Executor (`sites/docs/api/executor.rst`)
============ ``executor``
.. autoclass:: invoke.executor.Executor
Sites Docs Api Loader (`sites/docs/api/loader.rst`)
========== ``loader``
.. autoclass:: invoke.loader.Loader
:special-members:
:exclude-members: weakref
.. autoclass:: invoke.loader.FilesystemLoader
:special-members:
:exclude-members: weakref
Sites Docs Api Parser (`sites/docs/api/parser.rst`)
========== ``parser``
The command-line parsing framework is split up into a handful of sub-modules:
parser.argumentparser.context(not to be confused with the top levelcontext!)parser.parser
API docs for all are below.
.. automodule:: invoke.parser.parser
:member-order: bysource
.. automodule:: invoke.parser.context
.. automodule:: invoke.parser.argument
Sites Docs Api Program (`sites/docs/api/program.rst`)
=========== ``program``
.. automodule:: invoke.program
Sites Docs Api Runners (`sites/docs/api/runners.rst`)
=========== ``runners``
.. automodule:: invoke.runners
:member-order: bysource
Sites Docs Api Tasks (`sites/docs/api/tasks.rst`)
========= ``tasks``
.. automodule:: invoke.tasks
Sites Docs Api Terminals (`sites/docs/api/terminals.rst`)
============= ``terminals``
.. automodule:: invoke.terminals
Sites Docs Api Util (`sites/docs/api/util.rst`)
======== ``util``
.. automodule:: invoke.util
:exclude-members: ExceptionWrapper
.. autoclass:: invoke.util.ExceptionWrapper
:no-members:
Sites Docs Api Watchers (`sites/docs/api/watchers.rst`)
============ ``watchers``
.. automodule:: invoke.watchers
Sites Docs Concepts Configuration (`sites/docs/concepts/configuration.rst`)
.. _configuration:
============= Configuration
Introduction
Invoke offers a multifaceted configuration mechanism allowing you to configure
both core behavior and that of your tasks, via a hierarchy of configuration
files, environment variables, :doc:task namespaces </concepts/namespaces> and
CLI flags.
The end result of configuration seeking, loading, parsing & merging is a.Config object, which behaves like a (nested) Python dictionary. Invoke
references this object when it runs (determining the default behavior of
methods like .Context.run) and exposes it to users' tasks as.Context.config or as shorthand attribute access on the .Context itself.
.. _config-hierarchy:
The configuration hierarchy
In brief, the order in which configuration values override one another is as
follows:
#. Internal default values for behaviors which are controllable via
configuration. See :ref:default-values for details.
#. Collection-driven configurations defined in tasks modules via
.Collection.configure. (See :ref:collection-configuration below for
details.)
- Sub-collections' configurations get merged into the top level collection
and the final result forms the basis of the overall configuration setup.#. System-level configuration file stored in /etc/, such as
/etc/invoke.yaml. (See :ref:config-files for details on this and the
other config-file entries.)
#. User-level configuration file found in the running user's home
directory, e.g. ~/.invoke.yaml.
#. Project-level configuration file living next to your top level
tasks.py. For example, if your run of Invoke loads
/home/user/myproject/tasks.py (see our docs on :doc:the load process </concepts/loading>), this might be /home/user/myproject/invoke.yaml.
#. Environment variables found in the invoking shell environment.
- These aren't as strongly hierarchical as the rest, nor is the shell
environment namespace owned wholly by Invoke, so we must rely on slightly
verbose prefixing instead - see :ref:`env-vars` for details.#. Runtime configuration file whose path is given to :option:-f, e.g.
inv -f /random/path/to/config_file.yaml. This path may also be set via the
INVOKE_RUNTIME_CONFIG env var.
#. Command-line flags for certain core settings, such as :option:-e.
#. Modifications made by user code at runtime.
.. _default-values:
Default configuration values
Below is a list of all the configuration values and/or section Invoke itself
uses to control behaviors such as .Context.run's echo and pty
flags, task deduplication, and so forth.
.. note::
The storage location for these values is inside the .Config class,
specifically as the return value of .Config.global_defaults; see its API
docs for more details.
For convenience, we refer to nested setting names with a dotted syntax, so e.g.foo.bar refers to what would be (in a Python config context) {'foo': {'bar': <value here>}}. Typically, these can be read or set on .Config and.Context objects using attribute syntax, which looks nearly identical:c.foo.bar.
The
tasksconfig tree holds settings relating to task execution.tasks.dedupecontrols :ref:dedupingand defaults toTrue. It
can also be overridden at runtime via :option:--no-dedupe.tasks.auto_dash_namescontrols whether task and collection names have
underscores turned to dashes on the CLI. Default:True. See also
:ref:dashes-vs-underscores.tasks.collection_namecontrols the Python import name sought out by
:ref:collection discovery <collection-discovery>, and defaults to"tasks".tasks.executor_classallows users to override the class instantiated
and used for task execution.Must be a fully-qualified dotted path of the form
module(.submodule...).class, where all but.classwill be handed
toimportlib.import_module, andclassis expected to be an
attribute on that resulting module object.Defaults to
None, meaning to use the running.Programobject'sexecutor_classattribute... warning::
Take care if using this setting in tandem with :ref:custom program binaries <reusing-as-a-binary>, since custom programs may specify
their own default executor class (which your use of this setting will
override!) and assume certain behaviors stemming from that.tasks.ignore_unknown_help(default:False) lets users disable
"help keys were supplied for nonexistent arguments" errors. Normally,
Invoke assumes such a situation implies a typo in thehelpargument
to@task, but sometimes users have good reasons for this.tasks.search_rootallows overriding the default :ref:collection discovery <collection-discovery>root search location. It defaults toNone, which indicates to use the executing process' current working
directory.
The
runtree controls the behavior of.Runner.run. Each member of this
tree (such asrun.echoorrun.pty) maps directly to a.Runner.run
keyword argument of the same name; see that method's docstring for details on
what these settings do & what their default values are.The
runnerstree controls which runner classes map to which execution
contexts; if you're using Invoke by itself, this will only tend to have a
single member,runners.local. Client libraries may extend it with
additional key/value pairs, such asrunners.remote.The
sudotree controls the behavior of.Context.sudo:sudo.passwordcontrols the autoresponse password submitted to sudo's
password prompt. Default:None... warning::
While it's possible to store this setting, like any other, in
:doc:configuration files </concepts/configuration>-- doing so is
inherently insecure. We highly recommend filling this config value in
at runtime from a secrets management system of some kind.sudo.promptholds the sudo password prompt text, which is both
supplied tosudo -p, and searched for when performing
:doc:auto-response </concepts/watchers>. Default:[sudo] password:.
A top level config setting,
debug, controls whether debug-level output is
logged; it defaults toFalse.debugcan be toggled via the :option:-dCLI flag, which enables
debugging after CLI parsing runs. It can also be toggled via theINVOKE_DEBUGenvironment variable which - unlike regular env vars - is
honored from the start of execution and is thus useful for troubleshooting
parsing and/or config loading.A small config tree,
timeouts, holds various kinds of timeout controls.
At present, for Invoke, this only holds acommandsubkey, which controls
subprocess execution timeouts.- Client code often adds more to this tree, and Invoke itself may add more
in the future as well.
- Client code often adds more to this tree, and Invoke itself may add more
.. _config-files:
Configuration files
Loading
For each configuration file location mentioned in the previous section, we
search for files ending in .yaml, .yml, .json or .py (in that
order!), load the first one we find, and ignore any others that might exist.
For example, if Invoke is run on a system containing both /etc/invoke.yml
and /etc/invoke.json, only the YAML file will be loaded. This helps
keep things simple, both conceptually and in the implementation.
Format
Invoke's configuration allows arbitrary nesting, and thus so do our config file
formats. All three of the below examples result in a configuration equivalent
to {'debug': True, 'run': {'echo': True}}:
YAML
.. code-block:: yaml
textdebug: true run: echo: trueJSON
.. code-block:: javascript
text{ "debug": true, "run": { "echo": true } }Python::
debug = True
run = {
"echo": True
}
For further details, see these languages' own documentation.
.. _env-vars:
Environment variables
Environment variables are a bit different from other configuration-setting
methods, since they don't provide a clean way to nest configuration keys, and
are also implicitly shared amongst the entire system's installed application
base.
In addition, due to implementation concerns, env vars must be pre-determined by
the levels below them in the config hierarchy (in other words - env vars may
only be used to override existing config values). If you need Invoke to
understand a FOOBAR environment variable, you must first declare afoobar setting in a configuration file or in your task collections.
Basic rules
To mitigate the shell namespace problem, we simply prefix all our env vars withINVOKE_.
Nesting is performed via underscore separation, so a setting that looks like
e.g. {'run': {'echo': True}} at the Python level becomesINVOKE_RUN_ECHO=1 in a typical shell. See :ref:env-var-nesting below for
more on this.
Type casting
Since env vars can only be used to override existing settings, the previous
value of a given setting is used as a guide in casting the strings we get back
from the shell:
If the current value is a Unicode string, it is replaced with the
value from the environment, with no casting whatsoever;If the current value is
None, it too is replaced with the string from the
environment;Booleans are set as follows:
0and the empty value/string (e.g.SETTING=, orunset SETTING, or etc) evaluate toFalse, and any
other value evaluates toTrue.Lists and tuples are currently unsupported and will raise an exception;
- In the future we may implement convenience transformations, such as
splitting on commas to form a list; however since users can always
perform such operations themselves, it may not be a high priority.
- In the future we may implement convenience transformations, such as
All other types - integers, longs, floats, etc - are simply used as
constructors for the incoming value.- For example, a
foobarsetting whose default value is the integer1will run all env var inputs throughint, and thusFOOBAR=5
will result in the Python value5, not"5".
- For example, a
.. _env-var-nesting:
Nesting vs underscored names
Since environment variable keys are single strings, we must use some form of
string parsing to allow access to nested configuration settings. As mentioned
above, in basic use cases this just means using an underscore character:{'run': {'echo': True}} becomes INVOKE_RUN_ECHO=1.
However, ambiguity is introduced when the settings names themselves contain
underscores: is INVOKE_FOO_BAR=baz equivalent to {'foo': {'bar': 'baz'}}, or to {'foo_bar': 'baz'}? Thankfully, because env vars can only
be used to modify settings declared at the Python level or in config files, we
look at the current state of the config to determine the answer.
There is still a corner case where both possible interpretations exist as
valid config paths (e.g. {'foo': {'bar': 'default'}, 'foo_bar': 'otherdefault'}). In this situation, we honor the Zen of Python <http://zen-of-python.info/in-the-face-of-ambiguity-refuse-the-temptation-to-guess.html#12>_
and refuse to guess; an error is raised instead, counseling users to modify
their configuration layout or avoid using env vars for the setting in question.
.. _collection-configuration:
`.Collection`-based configuration
.Collection objects may contain a config mapping, set via.Collection.configure, and (as per :ref:the hierarchy <config-hierarchy>)
this typically forms the lowest level of configuration in the system.
When collections are :doc:nested </concepts/namespaces>, configuration is
merged 'downwards' by default: when conflicts arise, outer namespaces closer to
the root will win, versus inner ones closer to the task being invoked.
.. note::
'Inner' tasks here are specifically those on the path from the root to the
one housing the invoked task. 'Sibling' subcollections are ignored.
A quick example of what this means::
from invoke import Collection, task
# This task & collection could just as easily come from
# another module somewhere.
@task
def mytask(c):
print(c['conflicted'])
inner = Collection('inner', mytask)
inner.configure({'conflicted': 'default value'})
# Our project's root namespace.
ns = Collection(inner)
ns.configure({'conflicted': 'override value'})The result of calling inner.mytask::
$ inv inner.mytask
override valueExample of real-world config use
The previous sections had small examples within them; this section provides a
more realistic-looking set of examples showing how the config system works.
Setup
We'll start out with semi-realistic tasks that hardcode their values, and build
up to using the various configuration mechanisms. A small module for buildingSphinx <http://sphinx-doc.org>_ docs might begin like this::
from invoke import task
@task
def clean(c):
c.run("rm -rf docs/_build")
@task
def build(c):
c.run("sphinx-build docs docs/_build")Then maybe you refactor the build target::
target = "docs/_build"
@task
def clean(c):
c.run("rm -rf {}".format(target))
@task
def build(c):
c.run("sphinx-build docs {}".format(target))We can also allow runtime parameterization::
default_target = "docs/_build"
@task
def clean(c, target=default_target):
c.run("rm -rf {}".format(target))
@task
def build(c, target=default_target):
c.run("sphinx-build docs {}".format(target))This task module works for a single set of users, but what if we want to allow
reuse? Somebody may want to use this module with a different default target.
Using the configuration data (made available via the context arg) to configure
these settings is usually the better solution [1]_.
Configuring via task collection
The configuration setting <.Collection.configure> and getting <.Context.config> APIs enable moving otherwise 'hardcoded' default values into
a config structure which downstream users are free to redefine. Let's apply
this to our example. First we add an explicit namespace object::
from invoke import Collection, task
default_target = "docs/_build"
@task
def clean(c, target=default_target):
c.run("rm -rf {}".format(target))
@task
def build(c, target=default_target):
c.run("sphinx-build docs {}".format(target))
ns = Collection(clean, build)Then we can move the default build target value into the collection's default
configuration, and refer to it via the context. At this point we also change
our kwarg default value to be None so we can determine whether or not a
runtime value was given. The result::
@task
def clean(c, target=None):
if target is None:
target = c.sphinx.target
c.run("rm -rf {}".format(target))
@task
def build(c, target=None):
if target is None:
target = c.sphinx.target
c.run("sphinx-build docs {}".format(target))
ns = Collection(clean, build)
ns.configure({'sphinx': {'target': "docs/_build"}})The result isn't significantly more complex than what we began with, and as
we'll see next, it's now trivial for users to override your defaults in various
ways.
Configuration overriding
The lowest-level override is, of course, just modifying the local .Collection
tree into which a distributed module has been imported. E.g. if the above
module is distributed as myproject.docs, someone can define a tasks.py
that does this::
from invoke import Collection, task
from myproject import docs
@task
def mylocaltask(c):
# Some local stuff goes here
pass
# Add 'docs' to our local root namespace, plus our own task
ns = Collection(mylocaltask, docs)And then they can add this to the bottom::
# Our docs live in 'built_docs', not 'docs/_build'
ns.configure({'sphinx': {'target': "built_docs"}})Now we have a docs sub-namespace whose build target defaults tobuilt_docs instead of docs/_build. Runtime users can still override
this via flags (e.g. inv docs.build --target='some/other/dir') just as
before.
If you prefer configuration files over in-Python tweaking of your namespace
tree, that works just as well; instead of adding the line above to the previous
snippet, instead drop this into a file next to tasks.py namedinvoke.yaml::
sphinx:
target: built_docsFor this example, that sort of local-to-project conf file makes the most sense,
but don't forget that the :ref:config hierarchy <config-hierarchy> offers
additional configuration methods which may be suitable depending on your needs.
.. rubric:: Footnotes
.. [1]
Copying and modifying the file breaks code reuse; overriding the
module-level default_path variable won't play well with concurrency;
wrapping the tasks with different default arguments works but is fragile
and adds boilerplate.
Sites Docs Concepts Invoking Tasks (`sites/docs/concepts/invoking-tasks.rst`)
.. _invoking-tasks:
============== Invoking tasks
This page explains how to invoke your tasks on the CLI, both in terms of parser
mechanics (how your tasks' arguments are exposed as command-line options) and
execution strategies (which tasks actually get run, and in what order).
(For details on Invoke's core flags and options, see :doc:/invoke.)
.. contents::
:local:
.. _basic-cli-layout:
Basic command line layout
Invoke may be executed as invoke (or inv for short) and its command
line layout looks like this::
$ inv [--core-opts] task1 [--task1-opts] ... taskN [--taskN-opts]Put plainly, Invoke's CLI parser <.Parser> splits your command line up into
multiple "parser contexts <.ParserContext>" which allows it to reason about
the args and options it will accept:
- Before any task names are given, the parser is in the "core" parse context,
and looks for core options and flags such as :option:--echo,
:option:--listor :option:--help. - Any non-argument-like token (such as
mytask) causes a switch into a
per-task context (or an error, if no task matching that name seems to exist
in the :doc:loaded collection </concepts/loading>). - At this point, argument-like tokens are expected to correspond to the
arguments for the previously named task (see :ref:task-arguments). - Then this cycle repeats infinitely, allowing chained execution of arbitrary
numbers of tasks. (In practice, most users only execute one or two at a
time.)
For the core arguments and flags, see :doc:/invoke; for details on how your
tasks affect the CLI, read onwards.
.. note::
There is a minor convenience-minded exception to how parse contexts behave:
core options may also be given inside per-task contexts, if and only if
there is no conflict with similarly-named/prefixed arguments of the
being-parsed task.
For example, ``invoke mytask --echo`` will behave identically to ``invoke
--echo mytask``, *unless* ``mytask`` has its own ``echo`` flag (in which
case that flag is handed to the task context, as normal).
Similarly, ``invoke mytask -e`` will turn on command echoing too, unless
``mytask`` has its own argument whose shortflag ends up set to ``-e`` (e.g.
``def mytask(env)``)... _task-arguments:
Task command-line arguments
The simplest task invocation, for a task requiring no parameterization::
$ inv mytaskTasks may take parameters in the form of flag arguments::
$ inv build --format=html
$ inv build --format html
$ inv build -f pdf
$ inv build -f=pdfNote that both long and short style flags are supported, and that equals signs
are optional in both cases.
Boolean options are simple flags with no arguments::
$ inv build --progress-barNaturally, more than one flag may be given at a time::
$ inv build --progress-bar -f pdfType casting
Natively, a command-line string is just that -- a string -- requiring some
leaps of logic to arrive at any non-string values on the Python end. Invoke has
a number of these tricks already at hand, and more will be implemented in the
future. Currently:
Arguments with default values use those default values as a type hint, so
def mytask(c, count=1)will seeinv mytask --count=5and result in
the Python integer value5instead of the string"5".- Default values of
Noneare effectively the same as having no default
value at all - no type casting occurs and you're left with a string.
- Default values of
The primary exception to the previous rule is booleans: default values of
TrueorFalsecause those arguments to show up as actual
non-value-taking flags (--argnameto set the value toTrueif the
default wasFalse, or--no-argmentin the opposite case). See
:ref:boolean-flagsfor more examples.List values (which you wouldn't want to set as an argument's default value
anyways -- it's a common Python misstep) are served by a special@task
flag - see :ref:iterable-flag-valuesbelow.There's currently no way to set other compound values (such as dicts) on
the command-line; solving this more complex problem is left as an exercise to
the reader (though we may add helpers for such things in the future).
Per-task help / printing available flags
To get help for a specific task, you can give the task name as an argument to
the core --help/-h option, or give --help/-h after the task
(which will trigger custom-to-help behavior where the task name itself is
given to --help as its argument value).
When help is requested, you'll see the task's docstring (if any) and
per-argument/flag help output::
$ inv --help build # or invoke build --help
Docstring:
none
Options for 'build':
-f STRING, --format=STRING Which build format type to use
-p, --progress-bar Display progress barGlobbed short flags
Boolean short flags may be combined into one flag expression, so that e.g.::
$ inv build -qvis equivalent to (and expanded into, during parsing)::
$ inv build -q -vIf the first flag in a globbed short flag token is not a boolean but takes a
value, the rest of the glob is taken to be the value instead. E.g.::
$ inv build -fpdfis expanded into::
$ inv build -f pdfand not::
$ inv build -f -p -d -f.. _optional-values:
Optional flag values
You saw a hint of this with --help specifically, but non-core options may
also take optional values, if declared as optional. For example, say your
task has a --log flag that activates logging::
$ inv compile --logbut you also want it to be configurable regarding where to log::
$ inv compile --log=foo.logYou could implement this with an additional argument (e.g. --log and--log-location) but sometimes the concise API is the more useful one.
To enable this, specify which arguments are of this 'hybrid' optional-value
type inside @task::
@task(optional=['log'])
def compile(c, log=None):
if log:
log_file = '/var/log/my.log'
# Value was given, vs just-True
if isinstance(log, str):
log_file = log
# Replace w/ your actual log setup...
set_log_destination(log_file)
# Do things that might log here...When optional flag values are used, the values seen post-parse follow these
rules:
- If the flag is not given at all (
invoke compile) the default value
is filled in as normal. - If it is given with a value (
invoke compile --log=foo.log) then the value
is stored normally. - If the flag is given with no value (
invoke compile --log), it is treated
as if it were abooland set toTrue.
Resolving ambiguity
There are a number of situations where ambiguity could arise for a flag that
takes an optional value:
* When a task takes positional arguments and they haven't all been filled in by
the time the parser arrives at the optional-value flag;
* When the token following one of these flags looks like it is itself a flag;
or
* When that token has the same name as another task.
In most of these situations, Invoke's parser will `refuse the temptation to
guess
<http://zen-of-python.info/in-the-face-of-ambiguity-refuse-the-temptation-to-guess.html#12>`_
and raise an error.
However, in the case where the ambiguous token is flag-like, the current parse
context is checked to resolve the ambiguity:
- If the token is an otherwise legitimate argument, it is assumed that the user
meant to give that argument immediately after the current one, and no
optional value is set.
- E.g. in ``invoke compile --log --verbose`` (assuming ``--verbose`` is
another legit argument for ``compile``) the parser decides the user meant
to give ``--log`` without a value, and followed it up with the
``--verbose`` flag.
- Otherwise, the token is interpreted literally and stored as the value for
the current flag.
- E.g. if ``--verbose`` is *not* a legitimate argument for ``compile``,
then ``invoke compile --log --verbose`` causes the parser to assign
``"--verbose"`` as the value given to ``--log``. (This will probably
cause other problems in our contrived use case, but it illustrates our
point.)
.. _iterable-flag-values:
Iterable flag values
--------------------
A not-uncommon use case for CLI programs is the desire to build a list of
values for a given option, instead of a single value. While this *can* be done
via sub-string parsing -- e.g. having users invoke a command with ``--mylist
item1,item2,item3`` and splitting on the comma -- it's often preferable to
specify the option multiple times and store the values in a list (instead of
overwriting or erroring.)
In Invoke, this is enabled by hinting to the parser that one or more task
arguments are ``iterable`` in nature (similar to how one specifies ``optional``
or ``positional``)::
@task(iterable=['my_list'])
def mytask(c, my_list):
print(my_list)
When not given at all, the default value for ``my_list`` will be an empty list;
otherwise, the result is a list, appending each value seen, in order, without
any other manipulation (so no deduplication, etc)::
$ inv mytask
[]
$ inv mytask --my-list foo
['foo']
$ inv mytask --my-list foo --my-list bar
['foo', 'bar']
$ inv mytask --my-list foo --my-list bar --my-list foo
['foo', 'bar', 'foo']
.. _incrementable-flag-values:
Incrementable flag values
-------------------------
This is arguably a sub-case of :ref:`iterable flag values
` (seen above) - it has the same core interface of "give
a CLI argument multiple times, and have that do something other than error or
overwrite a single value." However, 'incrementables' (as you may have guessed)
increment an integer instead of building a list of strings. This is commonly
found in verbosity flags and similar functionality.
An example of exactly that::
@task(incrementable=['verbose'])
def mytask(c, verbose=0):
print(verbose)
And its use::
$ inv mytask
0
$ inv mytask --verbose
1
$ inv mytask -v
1
$inv mytask -vvv
3
Happily, because in Python ``0`` is 'falsey' and ``1`` (or any other number) is
'truthy', this functions a lot like a boolean flag as well, at least if one
defaults it to ``0``.
.. note::
You may supply any integer default value for such arguments (it simply
serves as the starting value), but take care that consumers of the argument
are written understanding that it is always going to appear 'truthy' unless
it's ``0``!
Dashes vs underscores in flag names
-----------------------------------
In Python, it's common to use ``underscored_names`` for keyword arguments,
e.g.::
@task
def mytask(c, my_option=False):
pass
However, the typical convention for command-line flags is dashes, which aren't
valid in Python identifiers::
$ inv mytask --my-option
Invoke works around this by automatically generating dashed versions of
underscored names, when it turns your task function signatures into
command-line parser flags.
Therefore, the two examples above actually work fine together -- ``my_option``
ends up mapping to ``--my-option``.
In addition, leading (``_myopt``) and trailing (``myopt_``) underscores are
ignored, since ``invoke ---myopt`` and ``invoke --myopt-`` don't make much
sense.
.. _boolean-flags:
Automatic Boolean inverse flags
-------------------------------
Boolean flags tend to work best when setting something that is normally
``False``, to ``True``::
$ inv mytask --yes-please-do-x
However, in some cases, you want the opposite - a default of ``True``, which
can be easily disabled. For example, colored output::
@task
def run_tests(c, color=True):
# ...
Here, what we really want on the command line is a ``--no-color`` flag that
sets ``color=False``. Invoke handles this for you: when setting up CLI flags,
booleans which default to ``True`` generate a ``--no-<name>`` flag instead.
.. _wrapper-tasks:
'Wrapper' tasks
---------------
An uncommon, but not rare, use case is one where your Invoke task is
essentially "wrapping" an existing command-line tool - the task performs setup
actions like "baking in" default arguments, setting shell env vars, or similar,
and then finally calls the underlying command with the result.
Prior to the addition of this feature, you had to use a clunky workaround:
stuffing all "passthrough" arguments into a single argument string of its own,
eg::
@task
def mywrapper(c, opts=""):
default_args = "--baked-in-arg baked-in-value"
c.run(f"some-command {default_args} {opts}")
Such tasks might be invoked like so::
$ inv mywrapper
# runs 'somecommand --baked-in-arg baked-in-value'
$ inv mywrapper --opts "--foo --bar"
# runs 'somecommand --baked-in-arg baked-in-value --foo --bar'
While this approach isn't *difficult*, it still carries friction, both in
having to cart this pattern around, and in the care required to invoke such
tasks properly - because it's easy to accidentally try the following::
$ inv mywrapper --opts --foo --bar
# will trigger a parsing exception! 'mywrapper' has no 'bar' argument!
To support this use case (among others), Invoke's parser offers the
:ref:`remainder <remainder>` feature, letting you pass in a pile of unparsed
text after a bare ``--`` signifier.
The remainder string is available in the task context argument,
`~invoke.context.Context`, as a ``.remainder`` attribute - giving us a somewhat
cleaner wrapper task::
@task
def mywrapper(c):
c.run(f"somecommand --baked-in-arg baked-in-value {c.remainder}")
Used like so::
$ inv mywrapper -- --foo --bar
# runs 'somecommand --baked-in-arg baked-in-value --foo --bar'
The same result as before, but with a simpler task definition, and no need to
remember quoting or similar - just the extra double-dash.
.. _how-tasks-run:
How tasks run
=============
Base case
---------
In the simplest case, a task with no pre- or post-tasks runs one time.
Example::
@task
def hello(c):
print("Hello, world!")
Execution::
$ inv hello
Hello, world!
.. _pre-post-tasks:
Pre- and post-tasks
-------------------
Tasks that should always have another task executed before or after them, may
use the ``@task`` deocator's ``pre`` and/or ``post`` kwargs, like so::
@task
def clean(c):
print("Cleaning")
@task
def publish(c):
print("Publishing")
@task(pre=[clean], post=[publish])
def build(c):
print("Building")
Execution::
$ inv build
Cleaning
Building
Publishing
These keyword arguments always take iterables. As a convenience, pre-tasks (and
pre-tasks only) may be given as positional arguments, in a manner similar to
build systems like ``make``. E.g. we could present part of the above example
as::
@task
def clean(c):
print("Cleaning")
@task(clean)
def build(c):
print("Building")
As before, ``invoke build`` would cause ``clean`` to run, then ``build``.
Recursive/chained pre/post-tasksPre-tasks of pre-tasks will also be invoked (as will post-tasks of pre-tasks,
pre-tasks of post-tasks, etc) in a depth-first manner, recursively. Here's a
more complex (if slightly contrived) tasks file::
@task
def clean_html(c):
print("Cleaning HTML")
@task
def clean_tgz(c):
print("Cleaning .tar.gz files")
@task(clean_html, clean_tgz)
def clean(c):
print("Cleaned everything")
@task
def makedirs(c):
print("Making directories")
@task(clean, makedirs)
def build(c):
print("Building")
@task(build)
def deploy(c):
print("Deploying")With a depth-first behavior, the below is hopefully intuitive to most users::
$ inv deploy
Cleaning HTML
Cleaning .tar.gz files
Cleaned everything
Making directories
Building
Deploying.. _parameterizing-pre-post-tasks:
Parameterizing pre/post-tasks
By default, pre- and post-tasks are executed with no arguments, even if the
task triggering their execution was given some. When this is not suitable, you
can wrap the task objects with `~.tasks.call` objects which allow you to
specify a call signature::
@task
def clean(c, which=None):
which = which or 'pyc'
print("Cleaning {}".format(which))
@task(pre=[call(clean, which='all')]) # or call(clean, 'all')
def build(c):
print("Building")
Example output::
$ inv build
Cleaning all
Building
.. _deduping:
Task deduplication
------------------
By default, any task that would run more than once during a session (due e.g.
to inclusion in pre/post tasks), will only be run once. Example task file::
@task
def clean(c):
print("Cleaning")
@task(clean)
def build(c):
print("Building")
@task(build)
def package(c):
print("Packaging")
With deduplication turned off (see below), the above would execute ``clean`` ->
``build`` -> ``build`` again -> ``package``. With deduplication, the double
``build`` does not occur::
$ inv build package
Cleaning
Building
Packaging
.. note::
Parameterized pre-tasks (using `~.tasks.call`) are deduped based on their
argument lists. For example, if ``clean`` was parameterized and hooked up
as a pre-task in two different ways - e.g. ``call(clean, 'html')`` and
``call(clean, 'all')`` - they would not get deduped should both end up
running in the same session.
However, two separate references to ``call(clean, 'html')`` *would* become
deduplicated.
Disabling deduplication
~~~~~~~~~~~~~~~~~~~~~~~
If you prefer your tasks to run every time no matter what, you can give the
``--no-dedupe`` core CLI option at runtime, or set the ``tasks.dedupe``
:doc:`config setting </concepts/configuration>` to ``False``. While it
doesn't make a ton of real-world sense, let's imagine we wanted to apply
``--no-dedupe`` to the above example; we'd see the following output::
$ inv --no-dedupe build package
Cleaning
Building
Building
Packaging
The build step is now running twice.
---
### Sites Docs Concepts Library (`sites/docs/concepts/library.rst`)
=========================
Using Invoke as a library
=========================
While most of our documentation involves the user/CLI facing use cases of task
management and command execution, Invoke was designed for its constituent parts
to be usable independently by advanced users - either out of the box or with a
minimum of extra work. CLI parsing, subprocess command execution, task
organization, etc, are all written as broadly separated concerns.
This document outlines use cases already known to work (because downstream
tools like `Fabric <https://fabfile.org>`_ are already utilizing them).
.. _reusing-as-a-binary:
Reusing Invoke's CLI module as a distinct binary
================================================
A major use case is distribution of your own program using Invoke under the
hood, bound to a different binary name, and usually setting a specific task
:doc:`namespace </concepts/namespaces>` as the default. (This maps somewhat
closely to things like ``argparse`` from the standard library.) In some cases,
removing, replacing and/or adding core CLI flags is also desired.
Getting set up
--------------
Say you want to distribute a test runner called ``tester`` offering two
subcommands, ``unit`` and ``integration``, such that users could ``pip install
tester`` and have access to commands like ``tester unit``, ``tester
integration``, or ``tester integration --fail-fast``.
First, as with any distinct Python package providing CLI
'binaries', you'd inform your ``setup.py`` of your entrypoint::
setup(
name='tester',
version='0.1.0',
packages=['tester'],
install_requires=['invoke'],
entry_points={
'console_scripts': ['tester = tester.main:program.run']
}
)
.. note::
This is just an example snippet and is not a fully valid ``setup.py``; if
you don't know how Python packaging works, a good starting place is `the
Python Packaging User's Guide
<https://python-packaging-user-guide.readthedocs.io>`_.
Nothing here is specific to Invoke - it's a standard way of telling Python to
install a ``tester`` script that executes the ``run`` method of a ``program``
object defined inside the module ``tester.main``.
Creating a ``Program``
----------------------
In our ``tester/main.py``, we start out importing Invoke's public CLI
functionality::
from invoke import Program
Then we define the ``program`` object we referenced in ``setup.py``, which is a
simple `.Program` to do the heavy lifting, giving it our version number for
starters::
program = Program(version='0.1.0')
At this point, installing ``tester`` would give you the same functionality as
Invoke's :doc:`built-in CLI tool `, except named ``tester`` and
exposing its own version number::
$ tester --version
Tester 0.1.0
$ tester --help
Usage: tester [--core-opts] task1 [--task1-opts] ... taskN [--taskN-opts]
Core options:
... core Invoke options here ...
$ tester --list
Can't find any collection named 'tasks'!
This doesn't do us much good yet - there aren't any subcommands (and our users
don't care about arbitrary 'tasks', so Invoke's own default ``--help`` and
``--list`` output isn't a good fit).
Specifying subcommands
----------------------
For ``tester`` to expose ``unit`` and ``integration`` subcommands, we need to
define them, in a regular Invoke tasks module or :doc:`namespace
</concepts/namespaces>`. For our example, we'll just create ``tester/tasks.py``
(but as you'll see in a moment, this too is arbitrary and can be whatever you
like)::
from invoke import task
@task
def unit(c):
print("Running unit tests!")
@task
def integration(c):
print("Running integration tests!")
As described in :doc:`/concepts/namespaces`, you can arrange this module
however you want - the above snippet uses an implicit namespace for brevity's
sake.
.. note::
It's important to realize that there's nothing special about these
"subcommands" - you could run them just as easily with vanilla Invoke,
e.g. via ``invoke --collection=tester.tasks --list``.
Now the useful part: telling our custom `.Program` that this namespace of tasks
should be used as the subcommands for ``tester``, via the ``namespace`` kwarg::
from invoke import Collection, Program
from tester import tasks
program = Program(namespace=Collection.from_module(tasks), version='0.1.0')
The result?
::
$ tester --version
Tester 0.1.0
$ tester --help
Usage: tester [--core-opts] <subcommand> [--subcommand-opts] ...
Core options:
... core options here, minus task-related ones ...
Subcommands:
unit
integration
$ tester --list
No idea what '--list' is!
$ tester unit
Running unit tests!
Notice how the 'usage' line changed (to specify 'subcommands' instead of
'tasks'); the list of specific subcommands is now printed as part of
``--help``; and ``--list`` has been removed from the options.
You can enable :ref:`tab-completion<tab-completion>` for your distinct
binary and subcommands.
Modifying core parser arguments
-------------------------------
A common need for this use case is tweaking the core parser arguments.
`.Program` makes it easy: default core `Arguments <.Argument>` are returned by
`.Program.core_args`. Extend this method's return value with ``super`` and
you're done::
# Presumably, this is your setup.py-designated CLI module...
from invoke import Program, Argument
class MyProgram(Program):
def core_args(self):
core_args = super().core_args()
extra_args = [
Argument(names=('foo', 'f'), help="Foo the bars"),
# ...
]
return core_args + extra_args
program = MyProgram()
.. warning::
We don't recommend *omitting* any of the existing core arguments; a lot of
basic functionality relies on their existence, even when left to default
values.
.. _customizing-config-defaults:
Customizing the configuration system's defaults
===============================================
Besides the CLI-oriented content of the previous section, another area of
functionality that frequently needs updating when redistributing an Invoke
codebase (CLI or no CLI) is configuration. There are typically two concerns
here:
- Configuration filenames and the env var prefix - crucial if you ever expect
your users to use the configuration system;
- Default configuration values - less critical (most defaults aren't labeled
with anything Invoke-specific) but still sometimes desirable.
.. note::
Both of these involve subclassing `.Config` (and, if using the CLI
machinery, informing your `.Program` to use that subclass instead of the
default one.)
Changing filenames and/or env var prefix
----------------------------------------
By default, Invoke's config system looks for files like ``/etc/invoke.yaml``,
``~/.invoke.json``, etc. If you're distributing client code named something
else, like the ``Tester`` example earlier, you might instead want the config
system to load ``/etc/tester.json`` or ``$CWD/tester.py``.
Similarly, the environment variable config level looks for env vars like
``INVOKE_RUN_ECHO``; you might prefer ``TESTER_RUN_ECHO``.
There are a few `.Config` attributes controlling these values:
- ``prefix``: A generic, catchall prefix used directly as the file prefix, and
used via all-caps as the env var prefix;
- ``file_prefix``: For overriding just the filename prefix - otherwise, it
defaults to the value of ``prefix``;
- ``env_prefix``: For overriding just the env var prefix - as you might have
guessed, it too defaults to the value of ``prefix``.
Continuing our 'Tester' example, you'd do something like this::
from invoke import Config
class TesterConfig(Config):
prefix = 'tester'
Or, to seek ``tester.yaml`` as before, but ``TEST_RUN_ECHO`` instead of
``TESTER_RUN_ECHO``::
class TesterConfig(Config):
prefix = 'tester'
env_prefix = 'TEST'
Modifying default config values
-------------------------------
Default config values are simple - they're just the return value of the
staticmethod `.Config.global_defaults`, so override that and return whatever
you like - ideally something based on the superclass' values, as many defaults
are assumed to exist by the rest of the system. (The helper function
`invoke.config.merge_dicts` can be useful here.)
For example, say you want Tester to always echo shell commands by default when
your codebase calls `.Context.run`::
from invoke import Program
from invoke.config import Config, merge_dicts
class TesterConfig(Config):
@staticmethod
def global_defaults():
their_defaults = Config.global_defaults()
my_defaults = {
'run': {
'echo': True,
},
}
return merge_dicts(their_defaults, my_defaults)
program = Program(config_class=TesterConfig, version='0.1.0')
For reference, Invoke's own base defaults (the...default defaults, you could
say) are documented at :ref:`default-values`.
---
### Sites Docs Concepts Loading (`sites/docs/concepts/loading.rst`)
.. _loading-collections:
===================
Loading collections
===================
The core of Invoke's execution model involves one or more Collection objects.
While these may be created programmatically, in typical use Invoke will create
them for you from Python modules it finds or is told to use.
.. _collection-discovery:
Task module discovery
=====================
With no other configuration, simply calling ``invoke`` will look for a single
Python module or package named ``tasks``, and will treat it as the root
namespace. ``tasks`` (or any other name given via :ref:`loading configuration
options <configuring-loading>`) is searched for in the following ways:
* First, if a valid tasks module by that name already exists on Python's
`sys.path <https://docs.python.org/release/2.7/library/sys.html#sys.path>`_,
no more searching is done -- that module is selected.
* Failing that, search towards the root of the local filesystem, starting with
the user's current working directory (`os.getcwd
<https://docs.python.org/release/2.7/library/os.html#os.getcwd>`_) and try
importing again with each directory temporarily added to ``sys.path``.
* Due to how Python's import machinery works, this approach will always
favor a package directory (``tasks/`` containing an ``__init__.py``) over
a module file (``tasks.py``) in the same location.
* If a candidate is found and successfully imported, its parent directory
will **stay** on ``sys.path`` during the rest of the Python session --
this allows task code to make convenient assumptions concerning sibling
modules' importability.
Candidate modules/packages are introspected to make sure they can actually be
used as valid task collections. Any that fail are discarded, the ``sys.path``
munging done to import them is reverted, and the search continues.
.. _configuring-loading:
Configuring the loading process
===============================
You can configure the above behavior, requesting that Invoke alter the
collection name searched for and/or the path where filesystem-level loading
starts looking.
For example, you may already have a project-level ``tasks.py`` that you can't
easily rename; or you may want to host a number of tasks collections stored
outside the project root and make it easy to switch between them; or any number
of reasons.
Both the sought collection name and the search root can be specified via
:ref:`configuration file options <config-files>` or as :doc:`runtime CLI flags
`:
- **Change the collection name**: Set the ``tasks.collection_name``
configuration option, or use :option:`--collection`. It should be a Python
module name and not a file name (so ``mytasks``, not ``mytasks.py`` or
``mytasks/``.)
- **Change the root search path**: Configure ``tasks.search_root`` or use
:option:`--search-root`. This value may be any valid directory path.
---
### Sites Docs Concepts Namespaces (`sites/docs/concepts/namespaces.rst`)
.. _task-namespaces:
=======================
Constructing namespaces
=======================
The :doc:`base case </getting-started>` of loading a single module of tasks
works fine initially, but advanced users typically need more organization, such
as separating tasks into a tree of nested namespaces.
The `.Collection` class provides an API for organizing tasks (and :ref:`their
configuration <collection-configuration>`) into a tree-like structure. When
referenced by strings (e.g. on the CLI or in pre/post hooks) tasks in nested
namespaces use a dot-separated syntax, e.g. ``docs.build``.
In this section, we show how building namespaces with this API is flexible but
also allows following Python package layouts with minimal boilerplate.
Starting out
============
One unnamed ``Collection`` is always the namespace root; in the implicit base
case, Invoke creates one for you from the tasks in your tasks module. Create
your own, named ``namespace`` or ``ns``, to set up an explicit namespace (i.e.
to skip the default "pull in all Task objects" behavior)::
from invoke import Collection
ns = Collection()
# or: namespace = Collection()
Add tasks with `.Collection.add_task`. `~.Collection.add_task` can take an
`.Task` object, such as those generated by the `.task` decorator::
from invoke import Collection, task
@task
def release(c):
c.run("python setup.py sdist register upload")
ns = Collection()
ns.add_task(release)
Our available tasks list now looks like this::
$ invoke --list
Available tasks:
release
Naming your tasks
=================
By default, a task's function name is used as its namespace identifier, but you
may override this by giving a ``name`` argument to either `@task <.task>` (i.e.
at definition time) or `.Collection.add_task` (i.e. at binding/attachment
time).
For example, say you have a variable name collision in your tasks module --
perhaps you want to expose a ``dir`` task, which shadows a Python builtin.
Naming your function itself ``dir`` is a bad idea, but you can name the
function something like ``dir_`` and then tell ``@task`` the "real" name::
@task(name='dir')
def dir_(c):
# ...
On the other side, you might have obtained a task object that doesn't fit with
the names you want in your namespace, and can rename it at attachment time.
Maybe we want to rename our ``release`` task to be called ``deploy`` instead::
ns = Collection()
ns.add_task(release, name='deploy')
The result::
$ invoke --list
Available tasks:
deploy
.. note::
The ``name`` kwarg is the 2nd argument to `~.Collection.add_task`, so those
in a hurry can phrase it as::
ns.add_task(release, 'deploy')
Aliases
-------
Tasks may have additional names or aliases, given as the ``aliases`` keyword
argument; these are appended to, instead of replacing, any implicit or explicit
``name`` value::
ns.add_task(release, aliases=('deploy', 'pypi'))
Result, with three names for the same task::
$ invoke --list
Available tasks:
release
deploy
pypi
.. note::
The convenience decorator `@task <.task>` is another method of
setting aliases (e.g. ``@task(aliases=('foo', 'bar'))``, and is useful for
ensuring a given task always has some aliases set no matter how it's added
to a namespace.
.. _dashes-vs-underscores:
Dashes vs underscores
---------------------
In the common case of functions-as-tasks, you'll often find yourself writing
task names that contain underscores::
@task
def my_awesome_task(c):
print("Awesome!")
Similar to how task arguments are processed to turn their underscores into
dashes (since that's a common command-line convention) all underscores in task
or collection names are interpreted to be dashes instead, by default::
$ inv --list
Available tasks:
my-awesome-task
$ inv my-awesome-task
Awesome!
If you'd prefer the underscores to remain instead, you can update your
configuration to set ``tasks.auto_dash_names`` to ``False`` in one of the
non-runtime :ref:`config files <config-files>` (system, user, or project.) For
example, in ``~/.invoke.yml``::
tasks:
auto_dash_names: false
.. note::
In the interests of avoiding confusion, this setting is "exclusive" in
nature - underscored version of task names *are not valid* on the CLI
unless ``auto_dash_names`` is disabled. (However, at the pure function
level within Python, they must continue to be referenced with underscores,
as dashed names are not valid Python syntax!)
Nesting collections
===================
The point of namespacing is to have sub-namespaces; to do this in Invoke,
create additional `.Collection` instances and add them to their parent
collection via `.Collection.add_collection`. For example, let's say we have a
couple of documentation tasks::
@task
def build_docs(c):
c.run("sphinx-build docs docs/_build")
@task
def clean_docs(c):
c.run("rm -rf docs/_build")
We can bundle them up into a new, named collection like so::
docs = Collection('docs')
docs.add_task(build_docs, 'build')
docs.add_task(clean_docs, 'clean')
And then add this new collection under the root namespace with
``add_collection``::
ns.add_collection(docs)
The result (assuming for now that ``ns`` currently just contains the original
``release`` task)::
$ invoke --list
Available tasks:
release
docs.build
docs.clean
As with tasks, collections may be explicitly bound to their parents with a
different name than they were originally given (if any) via a ``name`` kwarg
(also, as with ``add_task``, the 2nd regular arg)::
ns.add_collection(docs, 'sphinx')
Result::
$ invoke --list
Available tasks:
release
sphinx.build
sphinx.clean
Importing modules as collections
================================
A simple tactic which Invoke itself uses in the trivial, single-module
case is to use `.Collection.from_module` -- a classmethod
serving as an alternate ``Collection`` constructor which takes a Python module
object as its first argument.
Modules given to this method are scanned for ``Task`` instances, which are
added to a new ``Collection``. By default, this collection's name is taken from
the module name (the ``__name__`` attribute), though it can also be supplied
explicitly.
.. note::
As with the default task module, you can override this default loading
behavior by declaring a ``ns`` or ``namespace`` `.Collection` object at top
level in the loaded module.
For example, let's reorganize our earlier single-file example into a Python
package with several submodules. First, ``tasks/release.py``::
from invoke import task
@task
def release(c):
c.run("python setup.py sdist register upload")
And ``tasks/docs.py``::
from invoke import task
@task
def build(c):
c.run("sphinx-build docs docs/_build")
@task
def clean(c):
c.run("rm -rf docs/_build")
Tying them together is ``tasks/__init__.py``::
from invoke import Collection
import release, docs
ns = Collection()
ns.add_collection(Collection.from_module(release))
ns.add_collection(Collection.from_module(docs))
This form of the API is a little unwieldy in practice. Thankfully there's a
shortcut: ``add_collection`` will notice when handed a module object as its
first argument and call ``Collection.from_module`` for you internally::
ns = Collection()
ns.add_collection(release)
ns.add_collection(docs)
Either way, the result::
$ invoke --list
Available tasks:
release.release
docs.build
docs.clean
Default tasks
=============
Tasks may be declared as the default task to invoke for the collection they
belong to, e.g. by giving ``default=True`` to `@task <.task>` (or to
`.Collection.add_task`.) This is useful when you have a bunch of related tasks
in a namespace but one of them is the most commonly used, and maps well to the
namespace as a whole.
For example, in the documentation submodule we've been experimenting with so
far, the ``build`` task makes sense as a default, so we can say things like
``invoke docs`` as a shortcut to ``invoke docs.build``. This is easy to do::
@task(default=True)
def build(c):
# ...
When imported into the root namespace (as shown above) this alters the output
of ``--list``, highlighting the fact that ``docs.build`` can be invoked as
``docs`` if desired::
$ invoke --list
Available tasks:
release.release
docs.build (docs)
docs.clean
Default subcollections
----------------------
As of version 1.5, this functionality is also extended to subcollections: a
subcollection can be specified as the default when being added to its parent
collection, and that subcollection's own default task (or sub-subcollection!)
will be invoked as the default for the parent.
An example probably makes that clearer. Here's a tiny inline task tree with two
subcollections, each with their own default task::
from invoke import Collection, task
@task(default=True)
def build_all(c):
print("build ALL THE THINGS!")
@task
def build_wheel(c):
print("Just the wheel")
build = Collection(all=build_all, wheel=build_wheel)
@task(default=True)
def build_docs(c):
print("Code without docs is no code at all")
docs = Collection(build_docs)
Then we tie those into one top level collection, setting the ``build``
subcollection as the overall default::
ns = Collection()
ns.add_collection(build, default=True)
ns.add_collection(docs)
The result is that ``build.all`` becomes the absolute default task::
$ invoke
build ALL THE THINGS!
Mix and match
=============
You're not limited to the specific tactics shown above -- now that you know
the basic tools of ``add_task`` and ``add_collection``, use whatever approach
best fits your needs.
For example, let's say you wanted to keep things organized into submodules, but
wanted to "promote" ``release.release`` back to the top level for convenience's
sake. Just because it's stored in a module doesn't mean we must use
``add_collection`` -- we could instead import the task itself and use
``add_task`` directly::
from invoke import Collection
import docs
from release import release
ns = Collection()
ns.add_collection(docs)
ns.add_task(release)
Result::
$ invoke --list
Available tasks:
release
docs.build
docs.clean
More shortcuts
==============
Finally, you can even skip ``add_collection`` and ``add_task`` if your needs
are simple enough -- `.Collection`'s constructor will take
unknown arguments and build the namespace from their values as
appropriate::
from invoke import Collection
import docs, release
ns = Collection(release.release, docs)
Notice how we gave both a task object (``release.release``) and a module
containing tasks (``docs``). The result is identical to the above::
$ invoke --list
Available tasks:
release
docs.build
docs.clean
If given as keyword arguments, the keywords act like the ``name`` arguments do
in the ``add_*`` methods. Naturally, both can be mixed together as well::
ns = Collection(docs, deploy=release.release)
Result::
$ invoke --list
Available tasks:
deploy
docs.build
docs.clean
.. note::
You can still name these ``Collection`` objects with a leading string
argument if desired, which can be handy when building sub-collections.
---
### Sites Docs Concepts Testing (`sites/docs/concepts/testing.rst`)
.. _testing-user-code:
==============================
Testing Invoke-using codebases
==============================
Strategies for testing codebases that use Invoke; some applicable to code
focused on CLI tasks, and others applicable to more generic/refactored setups.
Subclass & modify Invoke 'internals'
====================================
A quick foreword: most users will find the subsequent approaches suitable, but
advanced users should note that Invoke has been designed so it is itself easily
testable. This means that in many cases, even Invoke's "internals" are exposed
as low/no-shared-responsibility, publicly documented classes which can be
subclassed and modified to inject test-friendly values or mocks. Be sure to
look over the :ref:`API documentation <api>`!
Use `.MockContext`
==================
An instance of subclassing Invoke's public API for test purposes is our own
`.MockContext`. Codebases which revolve heavily around `.Context` objects and
their methods (most task-oriented code) will find it easy to test by injecting
`.MockContext` objects which have been instantiated to yield partial `.Result`
objects.
For example, take this task::
from invoke import task
@task
def get_platform(c):
uname = c.run("uname -s").stdout.strip()
if uname == 'Darwin':
return "You paid the Apple tax!"
elif uname == 'Linux':
return "Year of Linux on the desktop!"
An example of testing it with `.MockContext` could be the following::
from invoke import MockContext, Result
from mytasks import get_platform
def test_get_platform_on_mac():
c = MockContext(run=Result("Darwin\n"))
assert "Apple" in get_platform(c)
def test_get_platform_on_linux():
c = MockContext(run=Result("Linux\n"))
assert "desktop" in get_platform(c)
Putting the ``Mock`` in `.MockContext`
--------------------------------------
Starting in Invoke 1.5, `.MockContext` will attempt to import the ``mock``
library at instantiation time and wrap its methods within ``Mock`` objects.
This lets you not only present realistic return values to your code, but make
test assertions about what commands your code is running.
Here's another "platform sensitive" task::
from invoke import task
@task
def replace(c, path, search, replacement):
# Assume systems have sed, and that some (eg macOS w/ Homebrew) may
# have gsed, implying regular sed is BSD style.
has_gsed = c.run("which gsed", warn=True, hide=True)
# Set command to run accordingly
binary = "gsed" if has_gsed else "sed"
c.run(f"{binary} -e 's/{search}/{replacement}/g' {path}")
The test code (again, which presumes that eg ``MockContext.run`` is now a
``Mock`` wrapper) relies primarily on 'last call' assertions
(``Mock.assert_called_with``) but you can of course use any ``Mock`` methods
you need. It also shows how you can set the mock context to respond to multiple
possible commands, using a dict value::
from invoke import MockContext, Result
from mytasks import replace
def test_regular_sed():
expected_sed = "sed -e s/foo/bar/g file.txt"
c = MockContext(run={
"which gsed": Result(exited=1),
expected_sed: Result(),
})
replace(c, 'file.txt', 'foo', 'bar')
c.run.assert_called_with(expected_sed)
def test_homebrew_gsed():
expected_sed = "gsed -e s/foo/bar/g file.txt"
c = MockContext(run={
"which gsed": Result(exited=0),
expected_sed: Result(),
})
replace(c, 'file.txt', 'foo', 'bar')
c.run.assert_called_with(expected_sed)
Boolean mock results
--------------------
You may have noticed the above example uses a handful of 'empty' `.Result`
objects; these stand in for "succeeded, but otherwise had no useful attributes"
command executions (as `.Result` defaults to an exit code of ``0`` and empty
strings for stdout/stderr).
This is relatively common - think "interrogative" commands where the caller
only cares for a boolean result, or times when a command is called purely for
its side effects. To support this, there's a shorthand in `.MockContext`:
passing ``True`` or ``False`` to stand in for otherwise blank Results with exit
codes of ``0`` or ``1`` respectively.
The example tests then look like this::
from invoke import MockContext, Result
from mytasks import replace
def test_regular_sed():
expected_sed = "sed -e s/foo/bar/g file.txt"
c = MockContext(run={
"which gsed": False,
expected_sed: True,
})
replace(c, 'file.txt', 'foo', 'bar')
c.run.assert_called_with(expected_sed)
def test_homebrew_gsed():
expected_sed = "gsed -e s/foo/bar/g file.txt"
c = MockContext(run={
"which gsed": True,
expected_sed: True,
})
replace(c, 'file.txt', 'foo', 'bar')
c.run.assert_called_with(expected_sed)
String mock results
-------------------
Another convenient shorthand is using string values, which are interpreted to
be the stdout of the resulting `.Result`. This only really saves you from
writing out the class itself (since ``stdout`` is the first positional arg of
`.Result`!) but "command X results in stdout Y" is a common enough use case
that we implemented it anyway.
By example, let's modify an earlier example where we cared about stdout::
from invoke import MockContext
from mytasks import get_platform
def test_get_platform_on_mac():
c = MockContext(run="Darwin\n")
assert "Apple" in get_platform(c)
def test_get_platform_on_linux():
c = MockContext(run="Linux\n")
assert "desktop" in get_platform(c)
As with everything else in this document, this tactic can be applied to
iterators or mappings as well as individual values.
Regular expression command matching
-----------------------------------
The dict form of `.MockContext` kwarg can accept regular expression objects as
keys, in addition to strings; ideal for situations where you either don't know
the exact command being invoked, or simply don't need or want to write out the
entire thing.
Imagine you're writing a function to run package management commands on a few
different Linux distros and you're trying to test its error handling. You might
want to set up a context that pretends any arbitrary ``apt`` or ``yum`` command
fails, and ensure the function returns stderr when it encounters a problem::
import re
from invoke import MockContext
from mypackage.tasks import install
package_manager = re.compile(r"^(apt(-get)?|yum) .*")
def test_package_success_returns_True():
c = MockContext(run={package_manager: True})
assert install(c, package="somepackage") is True
def test_package_explosions_return_stderr():
c = MockContext(run={
package_manager: Result(stderr="oh no!", exited=1),
})
assert install(c, package="otherpackage") == "oh no!"
A bit contrived - there are a bunch of other ways to organize this exact test
code so you don't truly need the regex - but hopefully it's clear that when you
*do* need this flexibility, this is how you could go about it.
Repeated results
----------------
By default, the values in these mock structures are consumed, causing
`.MockContext` to raise ``NotImplementedError`` afterwards (as it does for any
unexpected command executions). This was designed with the assumption that
most code under test will run a given command once.
If your situation doesn't match this, give ``repeat=True`` to the constructor,
and you'll see values repeat indefinitely instead (or in cycles, for
iterables).
Expect `Results <.Result>`
==========================
The core Invoke subprocess methods like `~.Context.run` all return `.Result`
objects - which (as seen above) can be readily instantiated by themselves with
only partial data (e.g. standard output, but no exit code or standard error).
This means that well-organized code can be even easier to test and doesn't
require as much use of `.MockContext`.
An iteration on the initial `.MockContext`-using example above::
from invoke import task
@task
def get_platform(c):
print(platform_response(c.run("uname -s")))
def platform_response(result):
uname = result.stdout.strip()
if uname == 'Darwin':
return "You paid the Apple tax!"
elif uname == 'Linux':
return "Year of Linux on the desktop!"
With the logic encapsulated in a subroutine, you can just unit test that
function by itself, deferring testing of the task or its context::
from invoke import Result
from mytasks import platform_response
def test_platform_response_on_mac():
assert "Apple" in platform_response(Result("Darwin\n"))
def test_platform_response_on_linux():
assert "desktop" in platform_response(Result("Linux\n"))
Avoid mocking dependency code paths altogether
==============================================
This is more of a general software engineering tactic, but the natural endpoint
of the above code examples would be where your primary logic doesn't care about
Invoke at all -- only about basic Python (or locally defined) data types. This
allows you to test logic in isolation and either ignore testing the Invoke side
of things, or write targeted tests solely for where your code interfaces with
Invoke.
Another minor tweak to the task code::
from invoke import task
@task
def show_platform(c):
uname = c.run("uname -s").stdout.strip()
print(platform_response(uname))
def platform_response(uname):
if uname == 'Darwin':
return "You paid the Apple tax!"
elif uname == 'Linux':
return "Year of Linux on the desktop!"
And the tests::
from mytasks import platform_response
def test_platform_response_on_mac():
assert "Apple" in platform_response("Darwin\n")
def test_platform_response_on_linux():
assert "desktop" in platform_response("Linux\n")
---
### Sites Docs Concepts Watchers (`sites/docs/concepts/watchers.rst`)
.. _autoresponding:
==========================================
Automatically responding to program output
==========================================
Background
==========
Command-line programs tend to be designed for interactive shells, which
frequently manifests as waiting around for user input, or "prompts".
Well-designed programs offer options for pre-empting such prompts, resulting in
an easily automated workflow -- but with the rest, interactivity is
unavoidable.
Thankfully, Invoke's `.Runner` class not only forwards your standard input to
the running program (allowing you to manually respond to prompts) but it can
also be configured to respond automatically on your behalf.
Basic use
=========
The mechanism for this automation is the ``watchers`` kwarg to the
`.Runner.run` method (and its wrappers elsewhere, such as `.Context.run` and
`invoke.run`), which is a list of `.StreamWatcher`-subclass instances
configured to watch for patterns & respond accordingly. The simplest of these
is `.Responder`, which just replies with its configured response every time its
pattern is seen; others can be found in the :doc:`watchers module
</api/watchers>`.
.. note::
As with all other arguments to ``run``, you can also set the default set of
watchers globally via :doc:`configuration files <configuration>`.
Take for example this program which expects a manual response to a yes/no
prompt::
$ excitable-program
When you give the OK, I'm going to do the things. All of them!!
Are you ready? [Y/n] y
OK! I just did all sorts of neat stuff. You're welcome! Bye!
You *could* call ``run("excitable-program")``, manually watch for the
prompt, and mash Y by hand. But if you instead supply a `.Responder` like so::
@task
def always_ready(c):
responder = Responder(
pattern=r"Are you ready? \[Y/n\] ",
response="y\n",
)
c.run("excitable-program", watchers=[responder])
Then `.Runner` passes the program's ``stdout`` and ``stderr`` through
``responder``, which watches for ``"Are you ready? [Y/n] "`` and automatically
writes ``y`` (plus ``\n`` to simulate hitting Enter/Return) to the program's
``stdin``.
.. note::
The pattern argument to `.Responder` is treated as a `regular expression
<re>`, requiring more care (note how we had to escape our square-brackets
in the above example) but providing more power as well.
---
--- METRICS ---
- Files Extracted: 25
- Estimated Token Budget: ~24498 tokens
- Canonical Reference: https://codewiki.google/github.com/pyinvoke/invoke