Source/Examples/Done
Here's an example of using :ref:done to create a multiprocess pool, wheresh.your_parallel_command is executed concurrently at no more than 10 at a
time:
.. code-block:: python
import sh
from threading import Semaphore
pool = Semaphore(10)
def done(cmd, success, exit_code):
pool.release()
def do_thing(arg):
pool.acquire()
return sh.your_parallel_command(arg, _bg=True, _done=done)
procs = []
for arg in range(100):
procs.append(do_thing(arg))
# essentially a join
[p.wait() for p in procs]Source/Sections/Architecture
.. _architecture:
Architecture Overview
#####################
Launch
When it comes time to launch a process
#. Open pipes and/or TTYs STDIN/OUT/ERR.
#. Open a pipe for communicating pre-exec exceptions from the child to the
parent.
#. Open a pipe for child/parent launch synchronization.
#. :func:os.fork a child process.
From here, we have two concurrent processes running:
Child
#. If :ref:_bg=True is set, we ignore :py:data:signal.SIGHUP.
#. If :ref:_new_session=True <new_session>, become a session leader with
:func:os.setsid, else become a process group leader with
:func:os.setpgrp.
#. Write our session id to the a pipe connected to the parent. This is mainly
to synchronize with our parent that our session/group logic has finished.
#. :func:os.dup2 the file descriptors of our previously-setup TTYs/pipes to
our STDIN/OUT/ERR file descriptors.
#. If we're a session leader and our STDIN is a TTY, via :ref:_tty_in=True <tty_in>, acquire a controlling
terminal, thereby becoming the controlling process of the session.
#. Set our GID/UID if we've set a custom one via :ref:_uid .
#. Close all file descriptors greater than STDERR.
#. Call :func:os.execv.
Parent
#. Check for any exceptions via the exception pipe connected to the child.
#. Block and read our child's session id from a pipe connected to the child.
This synchronizes to us that the child has finished moving between
sessions/groups and we can now accurately determine its current session id
and process group.
#. If we're using a TTY for STDIN, via :ref:_tty_in=True <tty_in>, disable
echoing on the TTY, so that data sent to STDIN is not echoed to STDOUT.
Running
An instance of :ref:oproc_class contains two internal threads, one for STDIN,
and one for STDOUT and STDERR. The purpose of these threads is to handle
reading/writing to the read/write ends of the process's standard descriptors.
For example, the STDOUT/ERR thread continually runs :func:select.select on the
master ends of the TTYs/pipes connected to STDOUT/ERR, and if they're ready to
read, reads the available data and aggregates it into the appropriate place.
.. _arch_buffers:
Buffers
A couple of different buffers must be considered when thinking about how data
flows through an sh process.
The first buffer is the buffer associated with the underlying pipe or TTY
attached to STDOUT/ERR. In the case of a TTY (the default for output), the
buffer size is 0, so output is immediate -- a byte written by the process is a
byte received by sh. For a pipe, however, the buffer size of the pipe is
typically 4-64kb. :manpage:pipe(2).
.. seealso:: FAQ: :ref:faq_tty_out
The second buffer is sh's internal buffers, one for STDOUT and one for STDERR.
These buffers aggregate data that has been read from the master end of the TTY
or pipe attached to the output fd, but before that data is sent along to the
appropriate output handler (queue, file object, function, etc). Data sits in
these buffers until we reach the size specified with :ref:internal_bufsize, at
which point the buffer flushes to the output handler.
Exit
STDIN Thread Shutdown
On process completion, our internal threads must complete, as the read end of
STDIN, for example, which is connected to the process, is no longer open, so
writing to the slave end will no longer work.
STDOUT/ERR Thread Shutdown
The STDOUT/ERR thread is a little more complicated, because although the process
is not alive, output data may still exist in the pipe/TTY buffer that must be
collected. So we essentially just :func:select.select on the read ends until
they return nothing, indicating that they are complete, then we break out of our
read loop.
.. _arch_exit_code:
Exit Code Processing
The exit code is obtained from the reaped process. If the process ended from a
signal, the exit code is the negative value of that signal. For example,
SIGKILL would result in an exit code -9.
Done Callback
If specified, the :ref:done callback is executed with the :ref:RunningCommand <running_command> instance, a boolean indicating success, and the adjusted exit
code. After the callback returns, error processing continues. In other words,
the done callback is called regardless of success or failure, and there's
nothing it can do to prevent the :ref:ErrorReturnCode <error_return_code>
exceptions from being raised after it completes.
Source/Sections/Asynchronous Execution
.. _async:
Asynchronous Execution
######################
sh provides a few methods for running commands and obtaining output in a
non-blocking fashion.
AsyncIO
.. versionadded:: 2.0.0
Sh supports asyncio on commands with the :ref:_async=True <async_kw> special
kwarg. This let's you incrementally await output produced from your command.
.. code-block:: python
import asyncio
import sh
async def main():
await sh.sleep(3, _async=True)
asyncio.run(main()).. _iterable:
Incremental Iteration
You may also create asynchronous commands by iterating over them with the
:ref:iter special kwarg. This creates an iterable (specifically, a generator)
that you can loop over:
.. code-block:: python
from sh import tail
# runs forever
for line in tail("-f", "/var/log/some_log_file.log", _iter=True):
print(line)By default, :ref:iter iterates over STDOUT, but you can change set this
specifically by passing either "err" or "out" to :ref:iter (instead ofTrue). Also by default, output is line-buffered, so the body of the loop
will only run when your process produces a newline. You can change this by
changing the buffer size of the command's output with :ref:out_bufsize.
.. note::
If you need a *fully* non-blocking iterator, use :ref:`iter_noblock`. If
the current iteration would block, :py:data:`errno.EWOULDBLOCK` will be
returned, otherwise you'll receive a chunk of output, as normal... _background:
Background Processes
By default, each running command blocks until completion. If you have a
long-running command, you can put it in the background with the :ref:_bg=True special kwarg:
.. code-block:: python
# blocks
sleep(3)
print("...3 seconds later")
# doesn't block
p = sleep(3, _bg=True)
print("prints immediately!")
p.wait()
print("...and 3 seconds later")You'll notice that you need to call :meth:RunningCommand.wait in order to exit
after your command exits.
Commands launched in the background ignore SIGHUP, meaning that when their
controlling process (the session leader, if there is a controlling terminal)
exits, they will not be signalled by the kernel. But because sh commands launch
their processes in their own sessions by default, meaning they are their own
session leaders, ignoring SIGHUP will normally have no impact. So the only
time ignoring SIGHUP will do anything is if you use :ref:_new_session=False <new_session>, in which case the controlling process will probably be the shell
from which you launched python, and exiting that shell would normally send aSIGHUP to all child processes.
.. seealso::
For more information on the exact launch process, see :ref:`architecture`... _callbacks:
Output Callbacks
In combination with :ref:_bg=True, sh can use callbacks to process output
incrementally by passing a callable function to :ref:out and/or :ref:err.
This callable will be called for each line (or chunk) of data that your command
outputs:
.. code-block:: python
from sh import tail
def process_output(line):
print(line)
p = tail("-f", "/var/log/some_log_file.log", _out=process_output, _bg=True)
p.wait()To control whether the callback receives a line or a chunk, use
:ref:out_bufsize. To "quit" your callback, simply return True. This
tells the command not to call your callback anymore.
The line or chunk received by the callback can either be of type str orbytes. If the output could be decoded using the provided encoding, astr will be passed to the callback, otherwise it would be raw bytes.
.. note::
Returning ``True`` does not kill the process, it only keeps the callback
from being called again. See :ref:`interactive_callbacks` for how to kill a
process from a callback... seealso:: :ref:red_func
.. _interactive_callbacks:
Interactive callbacks
Commands may communicate with the underlying process interactively through a
specific callback signature
Each command launched through sh has an internal STDIN :class:queue.Queue
that can be used from callbacks:
.. code-block:: python
def interact(line, stdin):
if line == "What... is the air-speed velocity of an unladen swallow?":
stdin.put("What do you mean? An African or European swallow?")
elif line == "Huh? I... I don't know that....AAAAGHHHHHH":
cross_bridge()
return True
else:
stdin.put("I don't know....AAGGHHHHH")
return True
p = sh.bridgekeeper(_out=interact, _bg=True)
p.wait().. note::
If you use a queue, you can signal the end of the input (EOF) with ``None``You can also kill or terminate your process (or send any signal, really) from
your callback by adding a third argument to receive the process object:
.. code-block:: python
def process_output(line, stdin, process):
print(line)
if "ERROR" in line:
process.kill()
return True
p = tail("-f", "/var/log/some_log_file.log", _out=process_output, _bg=True)The above code will run, printing lines from some_log_file.log until the
word "ERROR" appears in a line, at which point the tail process will be
killed and the script will end.
.. note::
You may also use :meth:`RunningCommand.terminate` to send a SIGTERM, or
:meth:`RunningCommand.signal` to send a general signal.Done Callbacks
A done callback called when the process exits, either normally (through
a success or error exit code) or through a signal. It is always called.
.. include:: /examples/done.rst
Source/Sections/Baking
.. _baking:
Baking
sh is capable of "baking" arguments into commands. This is essentiallypartial application <https://en.wikipedia.org/wiki/Partial_application>_,
like you might do with :func:functools.partial.
.. code-block:: python
from sh import ls
ls = ls.bake("-la")
print(ls) # "/usr/bin/ls -la"
# resolves to "ls -la /"
print(ls("/"))The idea here is that now every call to ls will have the "-la" arguments
already specified. Baking can become very useful when you combine it with
:ref:subcommands:
.. code-block:: python
from sh import ssh
# calling whoami on a server. this is a lot to type out, especially if
# you wanted to call many commands (not just whoami) back to back on
# the same server
iam1 = ssh("myserver.com", "-p 1393", "whoami")
# wouldn't it be nice to bake the common parameters into the ssh command?
myserver = ssh.bake("myserver.com", p=1393)
print(myserver) # "/usr/bin/ssh myserver.com -p 1393"
# resolves to "/usr/bin/ssh myserver.com -p 1393 whoami"
iam2 = myserver.whoami()
assert(iam1 == iam2) # True!Now that the "myserver" callable represents a baked ssh command, you
can call anything on the server easily:
.. code-block:: python
# executes "/usr/bin/ssh myserver.com -p 1393 tail /var/log/dumb_daemon.log -n 100"
print(myserver.tail("/var/log/dumb_daemon.log", n=100))Source/Sections/Command Class
API
.. _command_class:
Command Class
The Command class represents a program that exists on the system and can be
run at some point in time. An instance of Command is never running; an
instance of :ref:RunningCommand <running_command> is spawned for that.
An instance of Command can take the form of a manually instantiated object,
or as an object instantiated by dynamic lookup:
.. code-block:: python
import sh
ls1 = sh.Command("ls")
ls2 = sh.ls
assert ls1 == ls2.. py:class:: Command(name, search_paths=None)
Instantiates a Command instance, where *name* is the name of a program that
exists on the user's ``$PATH``, or is a full path itself. If *search_paths*
is specified, it must be a list of all the paths to look for the program
name.
.. code-block:: python
from sh import Command
ifconfig = Command("ifconfig")
ifconfig = Command("/sbin/ifconfig").. py:method:: Command.bake(*args, **kwargs)
Returns a new Command with ``*args`` and ``**kwargs`` baked in as
positional and keyword arguments, respectively. Any future calls to the
returned Command will include ``*args`` and ``**kwargs`` automatically:
.. code-block:: python
from sh import ls
long_ls = ls.bake("-l")
print(ls("/var"))
print(ls("/tmp"))
.. seealso::
:ref:`baking`Similar to the above, arguments to the sh.Command must be separate.
e.g. the following does not work::
lscmd = sh.Command("/bin/ls -l")
tarcmd = sh.Command("/bin/tar cvf /tmp/test.tar /my/home/directory/")You will run into CommandNotFound(path) exception even when correct full path is specified.
The correct way to do this is to :
#. build Command object using only the binary
#. pass the arguments to the object when invoking
as follows::
lscmd = sh.Command("/bin/ls")
lscmd("-l")
tarcmd = sh.Command("/bin/tar")
tarcmd("cvf", "/tmp/test.tar", "/my/home/directory/").. _running_command:
RunningCommand Class
This represents a :ref:Command <command_class> instance that has been
or is being executed. It exists as a wrapper around the low-level :ref:OProc <oproc_class>. Most of your interaction with sh objects are with instances of
this class. It is only returned if _return_cmd=True when you execute a command.
.. warning::
Objects of this class behave very much like strings. This was an
intentional design decision to make the "output" of an executing Command
behave more intuitively.
Be aware that functions that accept real strings only, for example
``json.dumps``, will not work on instances of RunningCommand, even though it
look like a string... _wait_method:
.. py:method:: RunningCommand.wait(timeout=None)
:param timeout: An optional non-negative number to wait for the command to complete. If it doesn't complete by the
timeout, we raise :ref:`timeout_exc`.
Block and wait for the command to finish execution and obtain an exit code.
If the exit code represents a failure, we raise the appropriate exception.
See :ref:`exceptions <exceptions>`.
.. note::
Calling this method multiple times only yields an exception on the first
call.
This is called automatically by sh unless your command is being executed
:ref:`asynchronously <async>`, in which case, you may want to call this
manually to ensure completion.
If an instance of :ref:`Command <command_class>` is being used as the stdin
argument (see :ref:`piping `), :meth:`wait` is also called on that
instance, and any exceptions resulting from that process are propagated up... py:attribute:: RunningCommand.process
The underlying :ref:`OProc <oproc_class>` instance... py:attribute:: RunningCommand.stdout
A ``@property`` that calls :meth:`wait` and then returns the contents of
what the process wrote to stdout... py:attribute:: RunningCommand.stderr
A ``@property`` that calls :meth:`wait` and then returns the contents of
what the process wrote to stderr... py:attribute:: RunningCommand.exit_code
A ``@property`` that calls :meth:`wait` and then returns the process's exit
code... py:attribute:: RunningCommand.pid
The process id of the process... py:attribute:: RunningCommand.sid
The session id of the process. This will typically be a different session
than the current python process, unless :ref:`_new_session=False
<new_session>` was specified... py:attribute:: RunningCommand.pgid
The process group id of the process... py:attribute:: RunningCommand.ctty
The controlling terminal device, if there is one... py:method:: RunningCommand.signal(sig_num)
Sends *sig_num* to the process. Typically used with a value from the
:mod:`signal` module, like :py:data:`signal.SIGHUP` (see :manpage:`signal(7)`)... py:method:: RunningCommand.signal_group(sig_num)
Sends *sig_num* to every process in the process group. Typically used with
a value from the :mod:`signal` module, like :py:data:`signal.SIGHUP` (see
:manpage:`signal(7)`)... py:method:: RunningCommand.terminate()
Shortcut for :meth:`RunningCommand.signal(signal.SIGTERM)
<RunningCommand.signal>`... py:method:: RunningCommand.kill()
Shortcut for :meth:`RunningCommand.signal(signal.SIGKILL)
<RunningCommand.signal>`... py:method:: RunningCommand.kill_group()
Shortcut for :meth:`RunningCommand.signal_group(signal.SIGKILL)
<RunningCommand.signal_group>`... py:method:: RunningCommand.is_alive()
Returns whether or not the process is still alive.
:rtype: bool.. _oproc_class:
OProc Class
.. warning::
Don't use instances of this class directly. It is being documented here for
posterity, not for direct use... py:method:: OProc.wait()
Block until the process completes, aggregate the output, and populate
:attr:`OProc.exit_code`... py:attribute:: OProc.stdout
A :class:`collections.deque`, sized to :ref:`_internal_bufsize
` items, that contains the process's STDOUT... py:attribute:: OProc.stderr
A :class:`collections.deque`, sized to :ref:`_internal_bufsize
` items, that contains the process's STDERR... py:attribute:: OProc.exit_code
Contains the process's exit code, or ``None`` if the process has not yet
exited... py:attribute:: OProc.pid
The process id of the process... py:attribute:: OProc.sid
The session id of the process. This will typically be a different session
than the current python process, unless :ref:`_new_session=False
<new_session>` was specified... py:attribute:: OProc.pgid
The process group id of the process... py:attribute:: OProc.ctty
The controlling terminal device, if there is one... py:method:: OProc.signal(sig_num)
Sends *sig_num* to the process. Typically used with a value from the
:mod:`signal` module, like :py:data:`signal.SIGHUP` (see :manpage:`signal(7)`)... py:method:: OProc.signal_group(sig_num)
Sends *sig_num* to every process in the process group. Typically used with
a value from the :mod:`signal` module, like :py:data:`signal.SIGHUP` (see
:manpage:`signal(7)`)... py:method:: OProc.terminate()
Shortcut for :meth:`OProc.signal(signal.SIGTERM) <OProc.signal>`... py:method:: OProc.kill()
Shortcut for :meth:`OProc.signal(signal.SIGKILL) <OProc.signal>`... py:method:: OProc.kill_group()
Shortcut for :meth:`OProc.signal_group(signal.SIGKILL)
<OProc.signal_group>`.Exceptions
.. _error_return_code:
ErrorReturnCode
.. py:class:: ErrorReturnCode
This is the base class for, as the name suggests, error return codes. It
subclasses :py:class:`Exception`... py:attribute:: ErrorReturnCode.full_cmd
The full command that was executed, as a string, so that you can try it on
the commandline if you wish... py:attribute:: ErrorReturnCode.stdout
The total aggregated STDOUT for the process... py:attribute:: ErrorReturnCode.stderr
The total aggregated STDERR for the process... py:attribute:: ErrorReturnCode.exit_code
The process's adjusted exit code.
.. seealso:: :ref:`arch_exit_code`.. _signal_exc:
SignalException
Subclasses :ref:ErrorReturnCode <error_return_code>. Raised when a command
receives a signal that causes it to exit.
.. _timeout_exc:
TimeoutException
Raised when a command specifies a non-null :ref:timeout and the command times out:
.. code-block:: python
import sh
try:
sh.sleep(10, _timeout=1)
except sh.TimeoutException:
print("we timed out, as expected")Also raised when you specify a timeout to :ref:RunningCommand.wait(timeout=None)<wait_method>:
.. code-block:: python
import sh
p = sh.sleep(10, _bg=True)
try:
p.wait(timeout=1)
except sh.TimeoutException:
print("we timed out waiting")
p.kill().. _not_found_exc:
CommandNotFound
This exception is raised in one of the following conditions:
- The program cannot be found on your path.
- You do not have permissions to execute the program.
- The program is not marked executable.
The last two bullets may seem strange, but they fall in line with how a shell like Bash behaves when looking up a
program to execute.
.. note::
``CommandNotFound`` subclasses ``AttributeError``. As such, the `repr` of it is simply the name of the missing
attribute.Helper Functions
.. py:function:: which(name, search_paths=None)
Resolves *name* to program's absolute path, or ``None`` if it cannot be
found. If *search_paths* is list of paths, use that list to look for the
program, otherwise use the environment variable ``$PATH``... py:function:: pushd(directory)
This function provides a ``with`` context that behaves similar to Bash's
`pushd
<https://www.gnu.org/software/bash/manual/html_node/Directory-Stack-Builtins.html>`_
by pushing to the provided directory, and popping out of it at the end of
the context.
.. code-block:: python
import sh
with sh.pushd("/tmp"):
sh.touch("a_file")
.. note::
It should be noted that we use a reentrant lock, so that different threads
using this function will have the correct behavior inside of their ``with``
contexts.Source/Sections/Contrib
.. _contrib:
Contrib Commands
################
Contrib is an sh sub-module that provides friendly wrappers to useful commands.
Typically, the commands being wrapped are unintuitive, and the contrib version
makes them intuitive.
.. note::
Contrib commands should be considered generally unstable. They will grow and change as the community figures out the
best interface for them.Commands
Sudo
Allows you to enter your password from the terminal at runtime, or as a string
in your script.
.. py:function:: sudo(password=None, *args, **kwargs)
Call sudo with ``password``, if specified, else ask the executing user for a
password at runtime via :func:`getpass.getpass`... seealso:: :ref:contrib_sudo
.. _contrib_git:
Git
Many git commands use a pager for output, which can cause an unexpected behavior
when run through sh. To account for this, the contrib version sets_tty_out=False for all git commands.
.. py:function:: git(*args, **kwargs)
Call git with STDOUT connected to a pipe, instead of a TTY... code-block:: python
from sh.contrib import git
repo_log = git.log().. seealso:: :ref:faq_tty_out and :ref:faq_color_output
.. _contrib_ssh:
SSH
.. versionadded:: 1.13.0
SSH password-based logins :ref:can be a pain <tutorial2>. This contrib command performs all of the ugly setup and
provides a clean interface to using SSH.
.. py:function:: ssh(interact=None, password=None, prompt_match=None, login_success=None, *args, **kwargs)
:param interact: A callback to handle SSH session interaction *after* login is successful. Required.
:param password: A password string or a function that returns a password string. Optional. If not provided, :func:`getpass.getpass` is used.
:param prompt_match: The string to match in order to determine when to provide SSH with the password. Or a function
that matches on the output. Optional.
:param login_success: A function to determine if SSH login is successful. Optional.The interact parameter takes a callback with a signature that is slightly different to the function callbacks for
:ref:redirection <red_func>:
.. py:function:: fn(content, stdin_queue)
:param content: An instance of an ephemeral :ref:`SessionContent <session_content>` class whose job is to hold the
characters that the SSH session has written to STDOUT.
:param stdin_queue: A :class:`queue.Queue` object to communicate with STDIN programmatically.password can be simply a string that will be used to type the password. If it's not provided, it will be read from STDIN
at runtime via :func:getpass.getpass. It can also be a callable that returns the password string.
prompt_match is a string to match before the contrib command will provide the SSH process with the password. It is
optional, and if left unspecified, will default to "password: ". It can also be a callable that is called on a
:ref:SessionContent <session_content> instance and returns True or False for a match.
login_success is a function that takes a :ref:SessionContent <session_content> object and returns a boolean for
whether or not a successful login occurred. It is optional, and if unspecified, simply evaluates to True, meaning
any password submission results in a successful login (obviously not always correct). It is recommended that you specify
this.
.. _session_content:
.. py:class:: SessionContent()
This class contains a record lines and characters written to the SSH processes's STDOUT. It should be all you need
from the callbacks to determine how to interact with the SSH process... py:attribute:: SessionContent.chars
:type: :class:`collections.deque`
The previous 50,000 characters... py:attribute:: SessionContent.lines
:type: :class:`collections.deque`
The previous 5,000 lines... py:attribute:: SessionContent.line_chars
:type: list
The characters in the line currently being aggregated... py:attribute:: SessionContent.cur_line
:type: str
A string of the line currently being aggregated... py:attribute:: SessionContent.last_line
:type: str
The previous line... py:attribute:: SessionContent.cur_char
:type: str
The currently written character... _contrib_bash:
Bash
Often users may find themselves having to run bash commands directly, whether due
to commands having special characters (e.g. dash, or dot) or other reasons.
This can lead into recurrently having to bake the bash command to call it directly. To
account for this, the contrib version provides a bash command baked in:
.. py:function:: bash(*args, **kwargs)
Call bash with the prefix of "bash -c [...]"... code-block:: python
from sh.contrib import bash
# Calling commands directly
bash.ls() # equivalent to "bash -c ls"
# Or adding the full commands
bash("command-with-dashes args")Extending
For developers.
To extend contrib, simply decorate a function in sh with the @contrib
decorator, and pass in the name of the command you wish to shadow to the
decorator. This method must return an instance of :ref:Command <command_class>:
.. code-block:: python
@contrib("ls")
def my_ls(original):
ls = original.bake("-l")
return lsNow you can run your custom contrib command from your scripts, and you'll be
using the command returned from your decorated function:
.. code-block:: python
from sh.contrib import ls
# executing: ls -l
print(ls("/"))For even more flexibility, you can design your contrib command to rewrite its
options based on executed arguments. For example, say you only wish to set a
command's argument if another argument is set. You can accomplish it like this:
.. code-block:: python
@contrib("ls")
def my_ls(original):
def process(args, kwargs):
if "-a" in args:
args.append("-L")
return args, kwargs
ls = original.bake("-l")
return ls, processReturning a process function along with the command will tell sh to use that
function to preprocess the arguments at execution time using the
:ref:_arg_preprocess special kwarg.
Source/Sections/Default Arguments
.. _default_arguments:
Default Arguments
Many times, you want to override the default arguments of all commands launched
through sh. For example, suppose you want the output of all commands to be
aggregated into a :class:io.StringIO buffer. The naive way would be this:
.. code-block:: python
import sh
from io import StringIO
buf = StringIO()
sh.ls("/", _out=buf)
sh.whoami(_out=buf)
sh.ps("auxwf", _out=buf)Clearly, this gets tedious quickly. Fortunately, we can create execution
contexts that allow us to set default arguments on all commands spawned from
that context:
.. code-block:: python
import sh
from io import StringIO
buf = StringIO()
sh2 = sh.bake(_out=buf)
sh2.ls("/")
sh2.whoami()
sh2.ps("auxwf")Now, anything launched from sh2 will send its output to the StringIO
instance buf.
Source/Sections/Envs
.. _environments:
Environments
The :ref:_env <env> special kwarg allows you to pass a dictionary of
environment variables and their corresponding values:
.. code-block:: python
import sh
sh.google_chrome(_env={"SOCKS_SERVER": "localhost:1234"}):ref:_env <env> replaces your process's environment completely. Only the
key-value pairs in :ref:_env <env> will be used for its environment. If you
want to add new environment variables for a process in addition to your
existing environment, try something like this:
.. code-block:: python
import os
import sh
new_env = os.environ.copy()
new_env["SOCKS_SERVER"] = "localhost:1234"
sh.google_chrome(_env=new_env).. seealso::
To make an environment apply to all sh commands look into
:ref:`default_arguments`.Source/Sections/Exit Codes
.. _exit_codes:
Exit Codes & Exceptions
Normal processes exit with exit code 0. This can be seen from
:attr:RunningCommand.exit_code:
.. code-block:: python
output = ls("/", _return_cmd=True)
print(output.exit_code) # should be 0If a process terminates, and the exit code is not 0, an exception is generated
dynamically. This lets you catch a specific return code, or catch all error
return codes through the base class :class:ErrorReturnCode:
.. code-block:: python
try:
print(ls("/some/non-existent/folder"))
except ErrorReturnCode_2:
print("folder doesn't exist!")
create_the_folder()
except ErrorReturnCode:
print("unknown error")You can also customize which exit codes indicate an error with :ref:ok_code. For example:
.. code-block:: python
for i in range(10):
sh.grep("string to check", f"file_{i}.txt", _ok_code=(0, 1))
where the :ref:ok_code makes a failure to find a match a no-op.
Signals
Signals are raised whenever your process terminates from a signal. The
exception raised in this situation is :ref:signal_exc, which subclasses
:ref:error_return_code.
.. code-block:: python
try:
p = sh.sleep(3, _bg=True)
p.kill()
except sh.SignalException_SIGKILL:
print("killed")This behavior could be blocked by appending the negative value of the signal to
:ref:ok_code. All signals that raises :ref:signal_exc are [SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGINT, SIGKILL, SIGPIPE, SIGQUIT, SIGSEGV, SIGTERM, SIGTERM].
.. note::
You can catch :ref:`signal_exc` by using either a number or a signal name.
For example, the following two exception classes are equivalent:
.. code-block:: python
assert sh.SignalException_SIGKILL == sh.SignalException_9Source/Sections/Faq
.. _faq:
FAQ
How do I execute a bash builtin?
.. code-block:: python
import sh
sh.bash("-c", "your_builtin")Or
.. code-block:: python
import sh
builtins = sh.bash.bake("-c")
builtins("your_builtin")Will Windows be supported?
There are no plans to support Windows.
.. _faq_append:
How do I append output to a file?
Use a file object opened in the mode you desire:
.. code-block:: python
import sh
h = open("/tmp/output", "a")
sh.ls("/dir1", _out=h)
sh.ls("/dir2", _out=h).. _faq_color_output:
Why does my command's output have color?
Typically the reason for this is that your program detected that its STDOUT was
connected to a TTY, and therefore decided to print color escape sequences in its
output. The typical solution is to use :ref:_tty_out=False <tty_out>, which
will force a pipe to be connected to STDOUT, and probably change the behavior of
the program.
.. seealso::
Git is one of the programs that makes extensive use of terminal colors (as
well as pagers) in its output, so we added :ref:`a contrib version
<contrib_git>` for convenience... _faq_tty_out:
Why is _tty_out=True the default?
This was a design decision made for two reasons:
- To make programs behave in the same way as seen on the commandline.
- To provide better buffering control than pipes allow.
For #1, we want sh to produce output that is identical to what the user sees
from the commandline, because that's typically the only output they ever see
from their command. This makes the output easy to understand.
For #2, using a TTY for STDOUT allows us to precisely control the buffering of a
command's output to sh's internal code.
.. seealso:: :ref:arch_buffers
Of course, there are some gotchas with TTY STDOUT. One of them is commands that
use a pager, for example:
.. code-block:: python
import sh
print(sh.git.log())This will sometimes raise a SignalException_SIGPIPE. The reason is becausegit log detects a TTY STDOUT and forks the system’s pager (typicallyless) to handle the output. The pager checks for a controlling terminal,
and, finding none, exits with exit code 1. The exit of the pager means no more
readers on git log’s output, and thus a SIGPIPE is received.
One solution to the git log problem above is simply to use_tty_out=False. Another option, specifically for git, is to use thegit --no-pager option:
.. code-block:: python
import sh
print(sh.git('--no-pager', 'log'))Why doesn't "*" work as a command argument?
Glob expansion is a feature of a shell, like Bash, and is performed by the shell
before passing the results to the program to be exec'd. Because sh is not a
shell, but rather tool to execute programs directly, we do not handle glob
expansion like a shell would.
So in order to use "*" like you would on the commandline, pass it into
:func:glob.glob first:
.. code-block:: python
import sh
import glob
sh.ls(glob.glob("*.py")).. _faq_path:
How do I call a program that isn't in ``$PATH``?
Use the :meth:Command constructor to instantiate an instance of Command
directly, then execute that:
.. code-block:: python
import sh
cmd = sh.Command("/path/to/command")
cmd("-v", "arg1")How do I execute a program with a dash in its name?
If it's in your $PATH, substitute the dash for an underscore:
.. code-block:: python
import sh
sh.google_chrome("http://google.com")The above will run google-chrome http://google.com
.. note::
If a program named ``google_chrome`` exists on your system, that will be
called instead. In that case, in order to execute the program with a dash
in the name, you'll have to use the method described :ref:`here.
<faq_special>`.. _faq_special:
How do I execute a program with a special character in its name?
Programs with non-alphanumeric, non-dash characters in their names cannot be
executed directly as an attribute on the sh module. For example, this will not
work:
.. code-block:: python
import sh
sh.mkfs.ext4()The reason should be fairly obvious. In Python, characters like . have
special meaning, in this case, attribute access. What sh is trying to do in the
above example is find the program "mkfs" (which may or may not exist) and then
perform a :ref:subcommand lookup <subcommands> with the name "ext4". In other
words, it will try to call mkfs with the argument ext4, which is
probably not what you want.
The workaround is instantiating the :ref:Command Class <command_class> with
the string of the program you're looking for:
.. code-block:: python
import sh
mkfsext4 = sh.Command("mkfs.ext4")
mkfsext4() # run it.. _faq_pipe_syntax:
Why not use ``|`` to pipe commands?
I prefer the syntax of sh to resemble function composition instead of a
pipeline. One of the goals of sh is to make executing processes more like
calling functions, not making function calls more like Bash.
Why isn't piping asynchronous by default?
There is a non-obvious reason why async piping is not possible by default.
Consider the following example:
.. code-block:: python
import sh
sh.cat(sh.echo("test\n1\n2\n3\n"))When this is run, sh.echo executes and finishes, then the entire output
string is fed into sh.cat. What we would really like is each
newline-delimited chunk to flow to sh.cat incrementally.
But for this example to flow data asynchronously from echo to cat, the echo
command would need to not block. But how can the inner command know the
context of its execution, to know to block sometimes but not other times? It
can't know that without something explicit.
This is why the :ref:piped special kwarg was introduced. By default, commands
executed block until they are finished, so in order for an inner command to not
block, _piped=True signals to the inner command that it should not block.
This way, the inner command starts running, then very shortly after, the outer
command starts running, and both are running simultaneously. Data can then flow
from the inner command to the outer command asynchronously:
.. code-block:: python
import sh
sh.cat(sh.echo("test\n1\n2\n3\n", _piped=True))Again, this example is contrived -- a better example would be a long-running
command that produces a lot of output that you wish to pipe through another
program incrementally.
How do I run a command and connect it to sys.stdout and sys.stdin?
There are two ways to do this
.. seealso:: :ref:fg
You can use :data:sys.stdin, :data:sys.stdout, and :data:sys.stderr as
arguments to :ref:in, :ref:out, :ref:err, respectively, and it should
mostly work as expected:
.. code-block:: python
import sh
import sys
sh.your_command(_in=sys.stdin, _out=sys.stdout)There are a few reasons why this probably won't work. The first reason is that
:data:sys.stdin is probably a controlling TTY (attached to the shell that
launched the python process), and probably not set in raw mode
:manpage:termios(3), which means that, among other things, input is buffered
by newlines.
The real solution is to use :ref:_fg=True <fg>:
.. code-block:: python
import sh
sh.top(_fg=True).. _faq_separate_args:
Why do my arguments need to be separate strings?
This confuses many new sh users. They want to do something like this and expect
it to just work:
.. code-block:: python
from sh import tar
tar("cvf /tmp/test.tar /my/home/directory")But instead they'll get a confusing error message:
.. code-block:: none
RAN: '/bin/tar cvf /tmp/test.tar /my/home/directory'
STDOUT:
STDERR:
/bin/tar: Old option 'f' requires an argument.
Try '/bin/tar --help' or '/bin/tar --usage' for more information.The reason why they expect it to work is because shells, like Bash, automatically
parse your commandline and break up arguments for you, before sending them to
the binary. They have a complex set of rules (some of which are represented by
:mod:shlex) to take a single string of a command and arguments and separate
them.
Even if we wanted to implement this in sh (which we don't), it would hurt the
ability for users to parameterize parts of their arguments. They would have to
use string interpolation, which would be ugly and error prone:
.. code-block:: python
from sh import tar
tar("cvf %s %s" % ("/tmp/tar1.tar", "/home/oh no a space")In the above example, "/home/oh", "no", "a", and "space" would
all be separate arguments to tar, causing the program to behave unexpectedly.
Basically every command with parameterized arguments would need to expect
characters that could break the parser.
.. _faq_arg_ordering:
How do I order keyword arguments?
Typically this question gets asked when a user is trying to execute something
like the following commandline:
.. code-block:: none
my-command --arg1=val1 arg2 --arg3=val3This is usually the first attempt that they make:
.. code-block:: python
sh.my_command(arg1="val1", "arg2", arg3="val3")This doesn't work because, in Python, position arguments, like arg2 cannot
come after keyword arguments.
Furthermore, it is entirely possible that --arg3=val3 comes before--arg1=val1. The reason for this is that a function's **kwargs is an
unordered mapping, and so key-value pairs are not guaranteed to resolve to a
specific order.
So the solution here is to forego the usage of the keyword argument
convenience, and just use raw ordered arguments:
.. code-block:: python
sh.my_command("--arg1=val1", "arg2", "--arg3=val3").. _faq_pylint:
How to disable pylint E1101 no-member errors?
Pylint complains with E1101 no-member to almost all sh.command invocations,
because it doesn't know, that these members are generated dynamically.
Starting with Pylint 1.6 these messages can be suppressed using generated-members <https://docs.pylint.org/en/1.6.0/features.html#id28>_ option.
Just add following lines to pylintrc::
[TYPECHECK]
generated-members=shHow do I patch sh in my tests?
sh can be patched in your tests the typical way, with
:func:unittest.mock.patch:
.. code-block:: python
from unittest.mock import patch
import sh
def get_something():
return sh.pwd()
@patch("sh.pwd", create=True)
def test_something(pwd):
pwd.return_value = "/"
assert get_something() == "/"The important thing to note here is that create=True is set. This is
required because sh is a bit magical and patch will fail to find the pwd
command as an attribute on the sh module.
You may also patch the :class:Command class:
.. code-block:: python
from unittest.mock import patch
import sh
def get_something():
pwd = sh.Command("pwd")
return pwd()
@patch("sh.Command")
def test_something(Command):
Command().return_value = "/"
assert get_something() == "/"Notice here we do not need create=True, because :class:Command is not an
automatically generated object on the sh module (it actually exists).
Why is sh just a single file?
When sh was first written, the design decision was made to make it a single-file
module. This has pros and cons:
Cons:
- Auditing the code is more challenging
- Without file-enforced structure, adding more features and abstractions makes
the code harder to follow - Cognitively, it feels cluttered
Pros:
- Can be used easily on systems without Python package managers
- Can be embedded/bundled together with other software more easily
- Cognitively, it feels more self-contained
In my mind, because the primary target audience of sh users is generally more
scrappy devops, systems people, or people just trying to stitch together some
clunky system programs, the listed pros weigh a little more heavily than the
cons. Sacrificing some development advantages to give those users a more
flexible tool is a win to me.
Down the road, the development disadvantages of a single file can be solved with
additional development tools, for example, with a tool that compiles multiple
modules into the single sh.py file. Realistically, though, sh is pretty mature,
so I don't see it growing much more in complexity or code size.
How do I see the commands sh is running?
Use logging:
.. code-block:: python
import logging
import sh
logging.basicConfig(level=logging.INFO)
sh.ls().. code-block:: none
INFO:sh.command:<Command '/bin/ls'>: starting process
INFO:sh.command:<Command '/bin/ls', pid 32394>: process started
INFO:sh.command:<Command '/bin/ls', pid 32394>: process completed
...Source/Sections/Migration
.. _migration:
Migrating from 1.x to 2.x
##########################
This document provides an upgrade path from 1.* to 2.*.
``sh.cd`` builtin removed
There is no sh.cd command anymore. It was always a command implemented in
sh, as some systems provide it as a shell builtin, while others have an actual
binary. But neither of them persisted the directory change between other sh
calls, which is why it was implemented in sh.
Workaround
If you were using sh.cd(dir), use the context manager with sh.pushd(dir)
instead. All of the commands in the managed context will have the correct
directory.
Removed execution contexts / default arguments
In 1.* you could spawn a new module from the sh module, one which had
customized defaults for the special keyword arguments. This module could then be
accessed just like sh, and you could even import commands from it.
Unfortunately the magic required to make that work was brittle. Also it was not
aligned syntactically with the similar baking concept. We have therefore changed
the syntax to align with baking, and also removed the ability to import directly
from this new baked execution context.
Workaround
.. code-block:: python
sh2 = sh(_tty_out=False)
sh2.ls()Becomes:
.. code-block:: python
sh2 = sh.bake(_tty_out=False)
sh2.ls()And:
.. code-block:: python
sh2 = sh.bake(_tty_out=False)
from sh2 import ls
ls()Becomes:
.. code-block:: python
sh2 = sh.bake(_tty_out=False)
ls = sh2.ls
ls()Return value now a true string
In 2.*, the return value of an executed sh command has changed (in most
cases) from a RunningCommand object to a unicode string. This makes using
the output of a command more natural.
Workaround
To continue returning a RunningCommand object, use the _return_cmd=True
special keyword argument. You can achieve this on each file with the following
code at the top of files that use sh:
.. code-block:: python
import sh
sh = sh.bake(_return_cmd=True)Piping to STDIN
Previously, if the first argument of a sh command was an instance ofRunningCommand, it was automatically fed into the process's STDIN. This is
no longer the case and you must explicitly use _in=.
.. code-block:: python
from sh import wc, ls
print(wc(ls("/home/", "-l"), "-l"))Becomes:
.. code-block:: python
from sh import wc, ls
print(wc("-l", _in=ls("/home/", "-l")))Or:
.. code-block:: python
from sh import wc, ls
print(wc("-l", _in=ls("/home/", "-l", _return_cmd=True)))Workaround
None.
New processes don't launch in new session
In 1.*, _new_session defaulted to True. It now defaults toFalse. The reason for this is that it makes more sense for launched
processes to default to being in the process group of the Python script, so
that they receive SIGINTs correctly.
Workaround
To preserve the old behavior:
.. code-block:: python
import sh
sh = sh.bake(_new_session=True)Source/Sections/Passing Arguments
.. _passing_arguments:
Passing Arguments
When passing multiple arguments to a command, each argument must be a separate
string:
.. code-block:: python
from sh import tar
tar("cvf", "/tmp/test.tar", "/my/home/directory/")This will not work:
.. code-block:: python
from sh import tar
tar("cvf /tmp/test.tar /my/home/directory").. seealso:: :ref:faq_separate_args
Keyword Arguments
sh supports short-form -a and long-form --arg arguments as
keyword arguments:
.. code-block:: python
# resolves to "curl http://duckduckgo.com/ -o page.html --silent"
curl("http://duckduckgo.com/", o="page.html", silent=True)
# or if you prefer not to use keyword arguments, this does the same thing:
curl("http://duckduckgo.com/", "-o", "page.html", "--silent")
# resolves to "adduser amoffat --system --shell=/bin/bash --no-create-home"
adduser("amoffat", system=True, shell="/bin/bash", no_create_home=True)
# or
adduser("amoffat", "--system", "--shell", "/bin/bash", "--no-create-home").. seealso:: :ref:faq_arg_ordering
Source/Sections/Piping
.. _piping:
Piping
Basic
Bash style piping is performed using function composition. Just pass one
command as the input to another's _in argument, and sh will send the output of
the inner command to the input of the outer command:
.. code-block:: python
# sort this directory by biggest file
print(sort("-rn", _in=du(glob("*"), "-sb")))
# print(the number of folders and files in /etc
print(wc("-l", _in=ls("/etc", "-1"))).. note::
This basic piping does not flow data through asynchronously; the inner
command blocks until it finishes, before sending its data to the outer
command.By default, any command that is piping another command in waits for it to
complete. This behavior can be changed with the :ref:_piped special
kwarg on the command being piped, which tells it not to complete before sending
its data, but to send its data incrementally. Read ahead for examples of this.
.. _advanced_piping:
Advanced
By default, all piped commands execute sequentially. What this means is that the
inner command executes first, then sends its data to the outer command:
.. code-block:: python
print(wc("-l", _in=ls("/etc", "-1")))In the above example, ls executes, gathers its output, then sends that output
to wc. This is fine for simple commands, but for commands where you need
parallelism, this isn't good enough. Take the following example:
.. code-block:: python
for line in tr(_in=tail("-f", "test.log"), "[:upper:]", "[:lower:]", _iter=True):
print(line)This won't work because the tail -f command never finishes. What you
need is for tail to send its output to tr as it receives it. This is where
the :ref:_piped special kwarg comes in handy:
.. code-block:: python
for line in tr(_in=tail("-f", "test.log", _piped=True), "[:upper:]", "[:lower:]", _iter=True):
print(line)This works by telling tail -f that it is being used in a pipeline, and that
it should send its output line-by-line to tr. By default, :ref:piped sends
STDOUT, but you can easily make it send STDERR instead by using _piped="err"
Source/Sections/Redirection
.. _redirection:
Redirection
sh can redirect the STDOUT and STDERR of a process to many different types of
targets, using the :ref:_out <out> and :ref:_err <err> special kwargs.
Filename
If a string is used, it is assumed to be a filename. The filename is opened as
"wb", meaning truncate-write and binary mode.
.. code-block:: python
import sh
sh.ifconfig(_out="/tmp/interfaces").. seealso:: :ref:faq_append
File-like Object
You may also use any object that supports .write(data), like
:class:io.StringIO:
.. code-block:: python
import sh
from io import StringIO
buf = StringIO()
sh.ifconfig(_out=buf)
print(buf.getvalue()).. _red_func:
Function Callback
A callback function may also be used as a target. The function must conform to
one of three signatures:
.. py:function:: fn(data)
:noindex:
The function takes just the chunk of data from the process... py:function:: fn(data, stdin_queue)
:noindex:
In addition to the previous signature, the function also takes a
:class:`queue.Queue`, which may be used to communicate programmatically with
the process... py:function:: fn(data, stdin_queue, process)
:noindex:
In addition to the previous signature, the function takes a
:class:`weakref.ref` to the :ref:`OProc <oproc_class>` object... seealso:: :ref:callbacks
.. seealso:: :ref:tutorial2
Source/Sections/Special Arguments
.. _special_arguments:
.. |def| replace:: Default value:
Special Kwargs
##############
These arguments alter a command's behavior. They are not passed to the program.
You can use them on any command that you run, but some may not be used together.
sh will tell you if there are conflicts.
To set default special keyword arguments on every command run, you may use
:ref:default_arguments.
Controlling Output
.. _out:
_out
|def| None
What to redirect STDOUT to. If this is a string, it will be treated as a file
name. You may also pass a file object (or file-like object), an int
(representing a file descriptor, like the result of :func:os.pipe), a
:class:io.StringIO object, or a callable.
.. code-block:: python
import sh
sh.ls(_out="/tmp/output").. seealso::
:ref:redirection
.. _err:
_err
|def| None
What to redirect STDERR to. See :ref:_out<out>.
_err_to_out
|def| False
If True, duplicate the file descriptor bound to the process's STDOUT also to
STDERR, effectively causing STDERR and STDOUT to go to the same place.
_encoding
|def| sh.DEFAULT_ENCODING
The character encoding of the process's STDOUT. By default, this is the
locale's default encoding.
_decode_errors
.. versionadded:: 1.07.0
|def| "strict"
This is how Python should handle decoding errors of the process's output.
By default, this is "strict", but you can use any value that's valid
to :meth:bytes.decode, such as "ignore".
_tee
.. versionadded:: 1.07.0
|def| None
As of 1.07.0, any time redirection is used, either for STDOUT or STDERR, the
respective internal buffers are not filled. For example, if you're downloading
a file and using a callback on STDOUT, the internal STDOUT buffer, nor the pipe
buffer be filled with data from STDOUT. This option forces one of stderr
(_tee='err') or stdout (_tee='out' or _tee=True) to be filled
anyways, in effect "tee-ing" the output into two places (the callback/redirect
handler, and the internal buffers).
_truncate_exc
.. versionadded:: 1.12.0
|def| True
Whether or not exception output should be truncated.
Execution
.. _fg:
_fg
.. versionadded:: 1.12.0
|def| False
Runs a command in the foreground, meaning it is spawned using :func:os.spawnle(). The current process's STDIN/OUT/ERR
is :func:os.dup2'd to the new process and so the new process becomes the foreground of the shell executing the
script. This is only really useful when you want to launch a lean, interactive process that sh is having trouble
running, for example, ssh.
.. warning::
``_fg=True`` side-steps a lot of sh's functionality. You will not be returned a process object and most (likely
all) other special kwargs will not work.If you are looking for similar functionality, but still retaining sh's features, use the following:
.. code-block:: python
import sh
import sys
sh.your_command(_in=sys.stdin, _out=sys.stdout, _err=sys.stderr).. _bg:
_bg
|def| False
Runs a command in the background. The command will return immediately, and you
will have to run :meth:RunningCommand.wait on it to ensure it terminates.
.. seealso:: :ref:background.
.. _bg_exc:
_bg_exc
.. versionadded:: 1.12.9
|def| True
Automatically report exceptions for the background command. If you set this toFalse you should make sure to call :meth:RunningCommand.wait or you may
swallow exceptions that happen in the background command.
.. _async_kw:
_async
.. versionadded:: 2.0.0
|def| False
Allows your command to become awaitable. Use in combination with :ref:_iter
and async for to incrementally await output as it is produced.
.. _env:
_env
|def| None
A dictionary defining the only environment variables that will be made
accessible to the process. If not specified, the calling process's environment
variables are used.
.. note::
This dictionary is the authoritative environment for the process. If you
wish to change a single variable in your current environment, you must pass
a copy of your current environment with the overridden variable to sh... seealso:: :ref:environments
.. _timeout:
_timeout
|def| None
How much time, in seconds, we should give the process to complete. If the
process does not finish within the timeout, it will be sent the signal defined
by :ref:timeout_signal.
.. _timeout_signal:
_timeout_signal
|def| signal.SIGKILL
The signal to be sent to the process if :ref:timeout is not None.
_cwd
|def| None
A string that sets the current working directory of the process.
.. _ok_code:
_ok_code
|def| 0
Either an integer, a list, or a tuple containing the exit code(s) that are
considered "ok", or in other words: do not raise an exception. Some misbehaved
programs use exit codes other than 0 to indicate success.
.. code-block:: python
import sh
sh.weird_program(_ok_code=[0,3,5])If the process is killed by a signal, a :ref:signal_exc is raised by
default. This behavior could be blocked by appending a negative number to
:ref:ok_code that represents the signal.
.. code-block:: python
import sh
# the process won't raise SignalException if SIGINT, SIGKILL, or SIGTERM
# are sent to kill the process
p = sh.sleep(3, _bg=True, _ok_code=[0, -2, -9, -15])
# No exception will be raised here
p.kill().. seealso:: :ref:exit_codes
.. _new_session:
_new_session
|def| False
Determines if our forked process will be executed in its own session via
:func:os.setsid.
.. versionchanged:: 2.0.0
The default value of _new_session was changed from True to False
because it makes more sense for a launched process to default to being in
the process group of python script, so that it receives SIGINTs correctly.
.. seealso:: :ref:architecture
_new_group
|def| False
Determines if our forked process will be executed in its own group via :func:os.setpgid
.. _uid:
_uid
.. versionadded:: 1.12.0
|def| None
The user id to assume before the child process calls :func:os.execv.
_preexec_fn
.. versionadded:: 1.12.0
|def| None
A function to be run directly before the child process calls :func:os.execv.
Typically not used by normal users.
.. _pass_fds:
_pass_fds
.. versionadded:: 1.13.0
|def| {} (empty set)
A whitelist iterable of integer file descriptors to be inherited by the child. Passing anything in this argument causes :ref:_close_fds <close_fds> to be True.
.. _close_fds:
_close_fds
.. versionadded:: 1.13.0
|def| True
Causes all inherited file descriptors besides stdin, stdout, and stderr to be automatically closed. This option is
automatically enabled when :ref:_pass_fds is given a value.
Communication
.. _in:
_in
|def| None
Specifies an argument for the process to use as its standard input. This may be
a string, a :class:queue.Queue, a file-like object, or any iterable.
.. seealso:: :ref:stdin
.. _piped:
_piped
|def| None
May be True, "out", or "err". Signals a command that it is being
used as the input to another command, so it should return its output
incrementally as it receives it, instead of aggregating it all at once.
.. seealso:: :ref:Advanced Piping <advanced_piping>
.. _iter:
_iter
|def| None
May be True, "out", or "err". Puts a command in iterable mode. In
this mode, you can use a for or while loop to iterate over a command's
output in real-time.
.. code-block:: python
import sh
for line in sh.cat("/tmp/file", _iter=True):
print(line).. seealso:: :ref:iterable.
.. _iter_noblock:
_iter_noblock
|def| None
Same as :ref:_iter , except the loop will not block if there is no
output to iterate over. Instead, the output from the command will be
:py:data:errno.EWOULDBLOCK.
.. code-block:: python
import sh
import errno
import time
for line in sh.tail("-f", "stuff.log", _iter_noblock=True):
if line == errno.EWOULDBLOCK:
print("doing something else...")
time.sleep(0.5)
else:
print("processing line!").. seealso:: :ref:iterable.
.. _with:
_with
|def| False
Explicitly tells us that we're running a command in a with context. This is
only necessary if you're using a command in a with context and passing
parameters to it.
.. code-block:: python
import sh
with sh.contrib.sudo(password="abc123", _with=True):
print(sh.ls("/root")).. seealso:: :ref:with_contexts
.. _done:
_done
.. versionadded:: 1.11.0
|def| None
A callback that is always called when the command completes, even if it
completes with an exit code that would raise an exception. After the callback
is run, any exception that would be raised is raised.
The callback is passed the :ref:RunningCommand <running_command> instance, a
boolean indicating success, and the exit code.
.. include:: /examples/done.rst
TTYs
.. _tty_in:
_tty_in
|def| False, meaning a :func:os.pipe will be used.
If True, sh creates a TTY for STDIN, essentially emulating a terminal, as if
your command was entered from the commandline. This is necessary for commands
that require STDIN to be a TTY.
.. _tty_out:
_tty_out
|def| True
If True, sh creates a TTY for STDOUT, otherwise use a :func:os.pipe. This
is necessary for commands that require STDOUT to be a TTY.
.. seealso:: :ref:faq_tty_out
.. _unify_ttys:
_unify_ttys
.. versionadded:: 1.13.0
|def| False
If True, sh will combine the STDOUT and STDIN TTY into a single
pseudo-terminal. This is sometimes required by picky programs which expect to be
dealing with a single pseudo-terminal, like SSH.
.. seealso:: :ref:tutorial2
_tty_size
|def| (20, 80)
The (rows, columns) of stdout's TTY. Changing this may affect how much your
program prints per line, for example.
Performance & Optimization
_in_bufsize
|def| 0
The STDIN buffer size. 0 for unbuffered, 1 for line buffered, anything else for
a buffer of that amount.
.. _out_bufsize:
_out_bufsize
|def| 1
The STDOUT buffer size. 0 for unbuffered, 1 for line buffered, anything
else for a buffer of that amount.
.. _err_bufsize:
_err_bufsize
|def| 1
Same as :ref:out_bufsize, but with STDERR.
.. _internal_bufsize:
_internal_bufsize
|def| 3 * 1024**2 chunks
How much of STDOUT/ERR your command will store internally. This value
represents the number of bufsize chunks not the total number of bytes. For
example, if this value is 100, and STDOUT is line buffered, you will be able to
retrieve 100 lines from STDOUT. If STDOUT is unbuffered, you will be able to
retrieve only 100 characters.
_no_out
.. versionadded:: 1.07.0
|def| False
Disables STDOUT being internally stored. This is useful for commands
that produce huge amounts of output that you don't need, that would
otherwise be hogging memory if stored internally by sh.
_no_err
.. versionadded:: 1.07.0
|def| False
Disables STDERR being internally stored. This is useful for commands that
produce huge amounts of output that you don't need, that would otherwise be
hogging memory if stored internally by sh.
_no_pipe
.. versionadded:: 1.07.0
|def| False
Similar to _no_out, this explicitly tells the sh command that it will never
be used for piping its output into another command, so it should not fill its
internal pipe buffer with the process's output. This is also useful for
conserving memory.
Program Arguments
These are options that affect how command options are fed into the program.
_long_sep
.. versionadded:: 1.12.0
|def| "="
This is the character(s) that separate a program's long argument's key from the
value, when using kwargs to specify your program's long arguments. For example,
if your program expects a long argument in the form --name value, the way to
achieve this would be to set _long_sep=" ".
.. code-block:: python
import sh
sh.your_program(key=value, _long_sep=" ")Would send the following list of arguments to your program:
.. code-block:: python
["--key value"]If your program expects the long argument name to be separate from its value,
pass None into _long_sep instead:
.. code-block:: python
import sh
sh.your_program(key=value, _long_sep=None)Would send the following list of arguments to your program:
.. code-block:: python
["--key", "value"]_long_prefix
.. versionadded:: 1.12.0
|def| "--"
This is the character(s) that prefix a long argument for the program being run.
Some programs use single dashes, for example, and do not understand double
dashes.
.. _preprocess:
_arg_preprocess
.. versionadded:: 1.12.0
|def| None
This is an advanced option that allows you to rewrite a command's arguments on
the fly, based on other command arguments, or some other variable. It is really
only useful in conjunction with :ref:baking , and only currently used when
constructing :ref:contrib <contrib> wrappers.
Example:
.. code-block:: python
import sh
def processor(args, kwargs):
return args, kwargs
my_ls = sh.bake.ls(_arg_preprocess=processor).. warning::
The interface to the ``_arg_preprocess`` function may change without
warning. It is generally only for internal sh use, so don't use it unless
you absolutely have to.Misc
_log_msg
|def| None
.. versionadded:: 1.12.0
This allows for a custom logging header for :ref:command_class instances. For example, the default logging looks like this:
.. code-block:: python
import logging
import sh
logging.basicConfig(level=logging.INFO)
sh.ls("-l").. code-block:: none
INFO:sh.command:<Command '/bin/ls -l'>: starting process
INFO:sh.command:<Command '/bin/ls -l', pid 28952>: process started
INFO:sh.command:<Command '/bin/ls -l', pid 28952>: process completedPeople can find this <Command .. section long and not relevant. _log_msg allows you to customize this:
.. code-block:: python
import logging
import sh
logging.basicConfig(level=logging.INFO)
def custom_log(ran, call_args, pid=None):
return ran
sh.ls("-l", _log_msg=custom_log).. code-block:: none
INFO:sh.command:/bin/ls -l: starting process
INFO:sh.command:/bin/ls -l: process started
INFO:sh.command:/bin/ls -l: process completedThe first argument, ran, is the program's execution string and arguments, as close as we can get it to be how you'd
type in the shell. call_args is a dictionary of all of the special kwargs that were passed to the command. And pid
is the process id of the forked process. It defaults to None because the _log_msg callback is actually called
twice: first to construct the logger for the :ref:running_command instance, before the process itself is spawned, then
a second time after the process is spawned via :ref:oproc_class, when we have a pid.
Source/Sections/Stdin
.. _stdin:
Input via STDIN
STDIN is sent to a process directly by using a command's :ref:in special
kwarg:
.. code-block:: python
print(cat(_in="test"))Any command that takes input from STDIN can be used this way:
.. code-block:: python
print(tr("[:lower:]", "[:upper:]", _in="sh is awesome"))You're also not limited to using just strings. You may use a file object, a
:class:queue.Queue, or any iterable (list, set, dictionary, etc):
.. code-block:: python
stdin = ["sh", "is", "awesome"]
out = tr("[:lower:]", "[:upper:]", _in=stdin).. note::
If you use a queue, you can signal the end of the queue (EOF) with ``None``Source/Sections/Subcommands
.. _subcommands:
Sub-commands
Many programs have their own command subsets, like git (branch, checkout),
svn (update, status), and sudo (where any command following sudo is considered
a sub-command). sh handles subcommands through attribute access:
.. code-block:: python
from sh import git, sudo
# resolves to "git branch -v"
print(git.branch("-v"))
print(git("branch", "-v")) # the same command
# resolves to "sudo /bin/ls /root"
print(sudo.ls("/root"))
print(sudo("/bin/ls", "/root")) # the same commandSub-commands are mainly syntax sugar that makes calling some programs look conceptually nicer.
.. seealso::
If you're using sudo as a subcommand, please be sure to see :ref:sudo.
Source/Sections/Sudo
.. _sudo:
Using Sudo
There are 3 ways of using sudo to execute commands in your script. These
are listed in order of usefulness and security. In most cases, you should just
use a variation of :ref:contrib_sudo.
.. _contrib_sudo:
sh.contrib.sudo
Because sudo is so frequently used, we have added a contrib version of the
command to make sudo usage more intuitive. This contrib version is simply a
wrapper around the :ref:sudo_raw raw command, but we bake in some
:ref:special keyword argument <special_arguments> to make it well-behaved. In
particular, the contrib version allows you to specify your password at execution
time via terminal input, or as a string in your script.
Terminal Input
^^^^^^^^^^^^^^
Via a :ref:with context <with_contexts>:
.. code-block:: python
import sh
with sh.contrib.sudo:
print(ls("/root"))Or alternatively via :ref:subcommands <subcommands>:
.. code-block:: python
import sh
print(sh.contrib.sudo.ls("/root"))Output:
.. code-block:: none
[sudo] password for youruser: *************
your_root_files.txtIn the above example, sh.contrib.sudo automatically asks you for a password
using :func:getpass.getpass under the hood.
This method is the most secure, because it lowers the chances of doing something
insecure, like including your password in your python script, or by saying that
a particular user can execute anything inside of a particular script (the
NOPASSWD method).
.. note::
``sh.contrib.sudo`` does not do password caching like the sudo binary does.
Thie means that each time a sudo command is run in your script, you will be
asked to type in a password.String Input
^^^^^^^^^^^^
You may also specify your password to sh.contrib.sudo as a string:
.. code-block:: python
import sh
password = get_your_password()
with sh.contrib.sudo(password=password, _with=True):
print(ls("/root")).. warning::
This method is less secure because it becomes tempting to hard-code your
password into the python script, and that's a bad idea. However, it is more
flexible, because it allows you to obtain your password from another source,
so long as the end result is a string./etc/sudoers NOPASSWD
With this method, you can use the raw sh.sudo command directly, because
you're being guaranteed that the system will not ask you for a password. It
first requires you set up your user to have root execution privileges
Edit your sudoers file:
.. code-block:: none
$> sudo visudoAdd or edit the line describing your user's permissions:
.. code-block:: none
yourusername ALL = (root) NOPASSWD: /path/to/your/programThis says yourusername on ALL hosts will be able to run as root, but
only root (root) (no other users), and that no password NOPASSWD will be
asked of /path/to/your/program.
.. warning::
This method can be insecure if an unprivileged user can edit your script,
because the entire script will be exited as a privileged user. A malicious
user could put something bad in this script... _sudo_raw:
sh.sudo
Using the raw command sh.sudo (which resolves directly to the system'ssudo binary) without NOPASSWD is possible, provided you wire up the special
keyword arguments on your own to make it behave correctly. This method is
discussed generally for educational purposes; if you take the time to wire upsh.sudo on your own, then you have in essence just recreated
:ref:contrib_sudo.
.. code-block:: python
import sh
# password must end in a newline
my_password = "password\n"
# -S says "get the password from stdin"
my_sudo = sh.sudo.bake("-S", _in=my_password)
print(my_sudo.ls("root"))_fg=True
Another less-obvious way of using sudo is by executing the raw sh.sudo
command but also putting it in the foreground. This way, sudo will work
correctly automatically, by hooking up stdin/out/err automatically, and by
asking you for a password if it requires one. The downsides of using
:ref:_fg=True <fg>, however, are that you cannot capture its output -- everything is
just printed to your terminal as if you ran it from a shell.
.. code-block:: python
import sh
sh.sudo.ls("/root", _fg=True)Source/Sections/With
.. _with_contexts:
'With' Contexts
Commands can be run within a Python with context. Popular commands using
this might be sudo or fakeroot:
.. code-block:: python
with sh.contrib.sudo:
print(ls("/root")).. seealso::
:ref:`contrib_sudo`If you need to run a command in a with context and pass in arguments, for
example, specifying a -p prompt with sudo, you need to use the :ref:_with=True <with> This let's the command know that it's being run from a with context so
it can behave correctly:
.. code-block:: python
with sh.contrib.sudo(k=True, _with=True):
print(ls("/root"))Source/Tutorials/Interacting With Processes
.. _tutorial2:
Entering an SSH password
Here we will attempt to SSH into a server and enter a password programmatically.
.. note::
It is recommended that you just ``ssh-copy-id`` to copy your public key to
the server so you don't need to enter your password, but for the purposes of
this demonstration, we try to enter a password.To interact with a process, we need to assign a callback to STDOUT. The
callback signature we'll use will take a :class:queue.Queue object for the
second argument, and we'll use that to send STDIN back to the process.
.. seealso:: :ref:red_func
Here's our first attempt:
.. code-block:: python
from sh import ssh
def ssh_interact(line, stdin):
line = line.strip()
print(line)
if line.endswith("password:"):
stdin.put("correcthorsebatterystaple")
ssh("10.10.10.100", _out=ssh_interact)If you run this (substituting an IP that you can SSH to), you'll notice that
nothing is printed from within the callback. The problem has to do with STDOUT
buffering. By default, sh line-buffers STDOUT, which means thatssh_interact will only receive output when sh encounters a newline in the
output. This is a problem because the password prompt has no newline:
.. code-block:: none
[email protected]'s password:Because a newline is never encountered, nothing is sent to the ssh_interact
callback. So we need to change the STDOUT buffering. We do this with the
:ref:_out_bufsize <out_bufsize> special kwarg. We'll set
it to 0 for unbuffered output:
.. code-block:: python
from sh import ssh
def ssh_interact(line, stdin):
line = line.strip()
print(line)
if line.endswith("password:"):
stdin.put("correcthorsebatterystaple")
ssh("10.10.10.100", _out=ssh_interact, _out_bufsize=0)If you run this updated version, you'll notice a new problem. The output looks
like this:
.. code-block:: none
a
m
o
f
f
a
t
@
1
0
.
1
0
.
1
0
.
1
0
0
'
s
p
a
s
s
w
o
r
d
:This is because the chunks of STDOUT our callback is receiving are unbuffered,
and are therefore individual characters, instead of entire lines. What we need
to do now is aggregate this character-by-character data into something more
meaningful for us to test if the pattern password: has been sent, signifying
that SSH is ready for input.
It would make sense to encapsulate the variable we'll use for aggregating into
some kind of closure or class, but to keep it simple, we'll just use a global:
.. code-block:: python
from sh import ssh
import sys
aggregated = ""
def ssh_interact(char, stdin):
global aggregated
sys.stdout.write(char.encode())
sys.stdout.flush()
aggregated += char
if aggregated.endswith("password: "):
stdin.put("correcthorsebatterystaple")
ssh("10.10.10.100", _out=ssh_interact, _out_bufsize=0)You'll also notice that the example still doesn't work. There are two problems:
The first is that your password must end with a newline, as if you had typed it
and hit the return key. This is because SSH has no idea how long your password
is, and is line-buffering STDIN.
The second problem lies deeper in SSH. SSH needs a TTY attached to its STDIN in
order to work properly. This tricks SSH into believing that it is interacting
with a real user in a real terminal session. To enable TTY, we can add the
:ref:_tty_in <tty_in> special kwarg. We also need to use :ref:_unify_ttys special kwarg.
This tells sh to make STDOUT and STDIN come from a single pseudo-terminal, which is a requirement of SSH:
.. code-block:: python
from sh import ssh
import sys
aggregated = ""
def ssh_interact(char, stdin):
global aggregated
sys.stdout.write(char.encode())
sys.stdout.flush()
aggregated += char
if aggregated.endswith("password: "):
stdin.put("correcthorsebatterystaple\n")
ssh("10.10.10.100", _out=ssh_interact, _out_bufsize=0, _tty_in=True, _unify_ttys=True)And now our remote login script works!
.. code-block:: none
[email protected]'s password:
Linux 10.10.10.100 testhost #1 SMP Tue Jun 21 10:29:24 EDT 2011 i686 GNU/Linux
Ubuntu 10.04.2 LTS
Welcome to Ubuntu!
* Documentation: https://help.ubuntu.com/
66 packages can be updated.
53 updates are security updates.
Ubuntu 10.04.2 LTS
Welcome to Ubuntu!
* Documentation: https://help.ubuntu.com/
You have new mail.
Last login: Thu Sep 13 03:53:00 2012 from some.ip.address
[email protected]:~$SSH Contrib command
The above process can be simplified by using a :ref:contrib. The :ref:SSH contrib command <contrib_ssh> does
all the ugly kwarg argument setup for you, and provides a simple but powerful interface for doing SSH password logins.
Please see the :ref:SSH contrib command <contrib_ssh> for more details about the exact api:
.. code-block:: python
from sh.contrib import ssh
def ssh_interact(content, stdin):
sys.stdout.write(content.cur_char)
sys.stdout.flush()
# automatically logs in with password and then presents subsequent content to
# the ssh_interact callback
ssh("10.10.10.100", password="correcthorsebatterystaple", interact=ssh_interact)How you should REALLY be using SSH
Many people want to learn how to enter an SSH password by script because they
want to execute remote commands on a server. Instead of trying to log in
through SSH and then sending terminal input of the command to run, let's see how
we can do it another way.
First, open a terminal and run ssh-copy-id yourservername. You'll be asked
to enter your password for the server. After entering your password, you'll be
able to SSH into the server without needing a password again. This simplifies
things greatly for sh.
The second thing we want to do is use SSH's ability to pass a command to run
to the server you're SSHing to. Here's how you can run ifconfig on a server
without having to use that server's shell directly:
.. code-block:: none
ssh [email protected] ifconfigTranslating this to sh, it becomes:
.. code-block:: python
import sh
print(sh.ssh("[email protected]", "ifconfig"))We can make this even nicer by taking advantage of sh's :ref:baking to bind
our server username/ip to a command object:
.. code-block:: python
import sh
my_server = sh.ssh.bake("[email protected]")
print(my_server("ifconfig"))
print(my_server("whoami"))Now we have a reusable command object that we can use to call remote commands.
But there is room for one more improvement. We can also use sh's
:ref:subcommands feature which expands attribute access into command
arguments:
.. code-block:: python
import sh
my_server = sh.ssh.bake("[email protected]")
print(my_server.ifconfig())
print(my_server.whoami())Source/Tutorials/Real Time Output
.. _tutorial1:
Tailing a real-time log file
sh has the ability to respond to subprocesses in an event-driven fashion.
A typical example of where this would be useful is tailing a log file for
a specific pattern, then responding to that value immediately::
from sh import tail
for line in tail("-f", "info.log", _iter=True):
if "ERROR" in line:
send_an_email_to_support(line)The :ref:_iter special kwarg takes a command that would normally block
until completion, and turns its output into a real-time iterable.
.. seealso:: :ref:iterable
Of course, you can do more than just tail log files. Any program that
produces output can be iterated over. Say you wanted to send an email to a
coworker if their C code emits a warning:
.. code-block:: python
from sh import gcc, git
for line in gcc("-o", "awesome_binary", "awesome_source.c", _iter=True):
if "warning" in line:
# parse out the relevant info
filename, line, char, message = line.split(":", 3)
# find the commit using git
commit = git("blame", "-e", filename, L="%d,%d" % (line,line))
# send them an email
email_address = parse_email_from_commit_line(commit)
send_email(email_address, message)Using :ref:_iter is a great way to respond to events from another
program, but your blocks while you're looping, making you unable to do anything
else. To be truly event-driven, sh provides callbacks:
.. code-block:: python
from sh import tail
def process_log_line(line):
if "ERROR" in line:
send_an_email_to_support(line)
process = tail("-f", "info.log", _out=process_log_line, _bg=True)
# ... do other stuff here ...
process.wait()The :ref:_out <out> special kwarg lets you to assign a callback to STDOUT.
This callback will receive each line of output from tail -f and allow you to
do the same processing that we did earlier.
.. seealso:: :ref:callbacks
.. seealso:: :ref:redirection
Source/Fulldoc
Full Documentation
This single page repeats the full documentation for sh <https://github.com/amoffat/sh/>, making it easier to put into an LLM's context window. There is nothing on this page that is not mentioned already elsewhere on this site, it's just reorganized as a single page.
.. include:: index.rst
.. include:: tutorials/interacting_with_processes.rst
.. include:: tutorials/real_time_output.rst
.. content linked in index.rst
.. include:: sections/faq.rst
.. include:: sections/contrib.rst
.. include:: sections/sudo.rst
.. include:: sections/migration.rst
.. content of usage.rst
.. include:: sections/passing_arguments.rst
.. include:: sections/exit_codes.rst
.. include:: sections/redirection.rst
.. include:: sections/asynchronous_execution.rst
.. also contains reference to example/done.rst so no need to mention it explicitly
.. .. include:: examples/done.rst
.. include:: sections/baking.rst
.. include:: sections/piping.rst
.. include:: sections/subcommands.rst
.. include:: sections/default_arguments.rst
.. include:: sections/envs.rst
.. include:: sections/stdin.rst
.. include:: sections/with.rst
.. content of reference.rst
.. include:: sections/special_arguments.rst
.. include:: sections/architecture.rst
.. include:: sections/command_class.rst
Source/Index
.. toctree::
:hidden:
usage
reference
sections/contrib
sections/sudo
sections/migration
tutorials
sections/faq
ref_to_fulldoc.. image:: images/logo-230.png
:alt: Logo
sh
.. image:: https://img.shields.io/pypi/v/sh.svg?style=flat-square
:target: https://pypi.python.org/pypi/sh
:alt: Version
.. image:: https://img.shields.io/pypi/dm/sh.svg?style=flat-square
:target: https://pypi.python.org/pypi/sh
:alt: Downloads Status
.. image:: https://img.shields.io/pypi/pyversions/sh.svg?style=flat-square
:target: https://pypi.python.org/pypi/sh
:alt: Python Versions
.. image:: https://img.shields.io/coveralls/amoffat/sh.svg?style=flat-square
:target: https://coveralls.io/r/amoffat/sh?branch=master
:alt: Coverage Status
.. image:: https://img.shields.io/github/stars/amoffat/sh.svg?style=social&label=Star
:target: https://github.com/amoffat/sh
:alt: Github
sh is a full-fledged subprocess replacement for Python 3.10+ and PyPy that
allows you to call any program as if it were a function:
.. code-block:: python
from sh import git
print(git("status", "--short"))Note that these aren't Python functions, these are running the binary commands
on your system by dynamically resolving your $PATH, much like Bash does, and
then wrapping the binary in a function. In this way, all the programs on your
system are easily available to you from within Python.
sh relies on various Unix system calls and only works on Unix-like operating
systems - Linux, macOS, BSDs etc. Specifically, Windows is not supported.
Installation
.. code-block:: none
pip install shQuick Reference
Passing Arguments
.. code-block:: python
sh.ls("-l", "/tmp", color="never"):ref:Read More
Exit Codes
.. code-block:: python
try:
sh.ls("/doesnt/exist")
except sh.ErrorReturnCode_2:
print("directory doesn't exist"):ref:Read More <exit_codes>
Redirection
.. code-block:: python
sh.ls(_out="/tmp/dir_contents")
with open("/tmp/dir_contents", "w") as h:
sh.ls(_out=h)
from io import StringIO
buf = StringIO()
sh.ls(_out=buf):ref:Read More <redirection>
Baking
.. code-block:: python
my_ls = sh.ls.bake("-l")
# equivalent
my_ls("/tmp")
sh.ls("-l", "/tmp"):ref:Read More
Piping
.. code-block:: python
sh.wc("-l", _in=sh.ls("-1")):ref:Read More
Subcommands
.. code-block:: python
# equivalent
sh.git("show", "HEAD")
sh.git.show("HEAD"):ref:Read More <subcommands>
Background Processes
.. code-block:: python
p = sh.find("-name", "sh.py", _bg=True)
# ... do other things ...
p.wait():ref:Read More
.. include:: ref_to_fulldoc.rst
Source/Ref To Fulldoc
Single Page
The page below repeats the full documentation for sh <https://github.com/amoffat/sh/> as a single page, making it easier to put into an LLM's context window.
:doc:./fulldoc
Source/Reference
Reference
.. toctree::
sections/special_arguments
sections/architecture
sections/command_class
Source/Tutorials
Tutorials
.. toctree::
tutorials/real_time_output
tutorials/interacting_with_processes
Source/Usage
Usage
.. toctree::
sections/passing_arguments
sections/exit_codes
sections/redirection
sections/asynchronous_execution
sections/baking
sections/piping
sections/subcommands
sections/default_arguments
sections/envs
sections/stdin
sections/withCHANGELOG
Changelog
2.4.0 - 7/25/26
- Dropped support for Python < 3.10
- Significantly improved
.pyityping stub with modern type aliases and annotations Commandnow supports generic subscript syntax (Command[str]) viaGenericAlias- Bugfix where signal names containing numbers were not matched correctly
stubtestandpyrefly coverage checknow run in CI
2.3.0 - 6/8/25
- Fix sdist builds from littering files
- Add all special keyword args to .pyi stub file
2.2.6 - 6/7/25
- Include missing sdist files #778
2.2.5 - 6/6/25
- Single page doc for LLM context window #757
- Formal support for Python 3.13 and 3.14
- Allow boolean arguments to override baked arguments #770
- Added .pyi typing stub
- Improvements to str representation of
Command#756 - Bugfix where rare
ProcessLookupErrorcan be thrown #769 - Various spelling corrections #774
2.2.4 - 6/06/25
- Bugfix regression where correct
gidwas not set on the launched process
2.2.3 - 6/05/25
- Bugfix where supplemental groups were not dropped when using
_uid
2.2.2 - 2/23/25
- Bugfix where it was impossible to use a signal as an
ok_code#699
2.2.1 - 1/9/25
- Bugfix where
asyncandreturn_cmddoes not raise exceptions #746
2.2.0 - 1/9/25
return_cmdwithawaitnow works correctly #743- Formal support for Python 3.12
2.1.0 - 10/8/24
- Add contrib command
sh.contrib.bash#736
2.0.7 - 5/31/24
- Fix
sh.globarguments #708 - Misc modernizations
2.0.6 - 8/9/23
- Add back appropriate sdist files comment
2.0.5 - 8/7/23
2.0.4 - 5/13/22
2.0.2 / 2.0.3 (misversioned) - 2/13/22
- Performance regression when using a generator with
_in#650 - Adding test support for python 3.11
2.0.0 - 2/9/22
- Executed commands now return a unicode string by default
- Removed magical module-like execution contexts #636
- Added basic asyncio support via
_async - Dropped support for Python < 3.8
- Bumped default tty size to more standard (24, 80)
- First argument being a RunningCommand no longer automatically passes it as stdin
RunningCommand.__eq__no longer has the side effect of executing the command #518_teenow supports both "err" and "out" #215- Removed the builtin override
cdlink - Altered process launching model to behave more expectedly #495
- Bugfix where
_no_outisn't allowed with_iter="err"#638 - Allow keyword arguments to have a list of values #529
1.14.3 - 7/17/22
- Bugfix where
Commandwas not aware of default call args when wrapping the module #559
1.14.1 - 10/24/20
- bugfix where setting
_ok_codeto not include 0, but 0 was the exit code #545
1.14.0 - 8/28/20
_envnow more lenient in accepting dictionary-like objects #527NoneandFalsearguments now do not pass through to underlying command #525- Implemented
find_specon the fancy importer, which fixes some Python3.4+ issues #536
1.13.1 - 4/28/20
- regression fix if
_fg=False#520
1.13.0 - 4/27/20
- minor Travis CI fixes #492
- bugfix for boolean long options not respecting
_long_prefix#488 - fix deprecation warning on Python 3.6 regexes #482
_pass_fdsand_close_fdsspecial kwargs for controlling file descriptor inheritance in child.- more efficiently closing inherited fds #406
- bugfix where passing invalid dictionary to
_envwill cause a mysterious child 255 exit code. #497 - bugfix where
_inusing 0 orsys.stdinwasn't behaving like a TTY, if it was in fact a TTY. #514 - bugfix where
help(sh)raised an exception #455 - bugfix fixing broken interactive ssh tutorial from docs
- change to automatic tty merging into a single pty if
_tty_in=Trueand_tty_out=True - introducing
_unify_ttys, default False, which allows explicit tty merging into single pty - contrib command for
sshconnections requiring passwords - performance fix for polling output too fast when using
_iter#462 - execution contexts can now be used in python shell #466
- bugfix
ErrorReturnCodeinstances can now be pickled - bugfix passing empty string or
Nonefor_inhanged #427 - bugfix where passing a filename or file-like object to
_outwasn't using os.dup2 #449 - regression make
_fgwork with_cwdagain #330 - an invalid
_cwdnow raises aForkExceptionnot anOSError. - AIX support #477
- added a
timeout=Noneparam toRunningCommand.wait()#515
1.12.14 - 6/6/17
- bugfix for poor sleep performance #378
- allow passing raw integer file descriptors for
_outand_errhandlers - bugfix for when
_teeand_outare used, and the_outis a tty or pipe #384 - bugfix where python 3.3+ detected different arg counts for bound method output callbacks #380
1.12.12, 1.12.13 - 3/30/17
- pypi readme doc bugfix PR#377
1.12.11 - 3/13/17
- bugfix for relative paths to
sh.Commandnot expanding to absolute paths #372 - updated for python 3.6
- bugfix for SIGPIPE not being handled correctly on pipelined processes #373
1.12.10 - 3/02/17
- bugfix for file descriptors over 1024 #356
- bugfix when
_err_to_outis True and_outis pipe or tty #365
1.12.9 - 1/04/17
- added
_bg_excfor silencing exceptions in background threads #350
1.12.8 - 12/16/16
- bugfix for patched glob.glob on python3.5 #341
1.12.7 - 12/07/16
- added
_outand_out_bufsizevalidator #346 - bugfix for internal stdout thread running when it shouldn't #346
1.12.6 - 12/02/16
- regression bugfix on timeout #344
- regression bugfix on
_ok_code=None
1.12.5 - 12/01/16
- further improvements on cpu usage
1.12.4 - 11/30/16
- regression in cpu usage #339
1.12.3 - 11/29/16
- fd leak regression and fix for flawed fd leak detection test #337
1.12.2 - 11/28/16
- support for
io.StringIOin python2
1.12.1 - 11/28/16
- added support for using raw file descriptors for
_in,_out, and_err - removed
.close()ing_outhandler if FIFO detected
1.12.0 - 11/21/16
- composed commands no longer propagate
_bg - better support for using
sys.stdinandsys.stdoutfor_inand_out - bugfix where
which()would not stop searching at the first valid executable found in PATH - added
_long_prefixfor programs whose long arguments start with something other than--#278 - added
_log_msgfor advanced configuration of log message #311 - added
sh.contrib.sudo - added
_arg_preprocessfor advanced command wrapping - alter callable
_inarguments to signify completion with falsy chunk - bugfix where pipes passed into
_outor_errwere not flushed on process end #252 - deprecated
with sh.args(**kwargs)in favor ofsh2 = sh(**kwargs) - made
sh.pushdthread safe - added
.kill_group()and.signal_group()methods for better process control #237 - added
new_sessionspecial keyword argument for controlling spawned process session #266 - bugfix better handling for EINTR on system calls #292
- bugfix where with-contexts were not threadsafe #247
_uidnew special keyword param for specifying the user id of the process #133- bugfix where exceptions were swallowed by processes that weren't waited on #309
- bugfix where processes that dupd their stdout/stderr to a long running child process would cause sh to hang #310
- improved logging output #323
- bugfix for python3+ where binary data was passed into a process's stdin #325
- Introduced execution contexts which allow baking of common special keyword arguments into all commands #269
Commandandwhichnow can take an optionalpathsparameter which specifies the search paths #226_preexec_fnoption for executing a function after the child process forks but before it execs #260_fgreintroduced, with limited functionality. hurrah! #92- bugfix where a command would block if passed a fd for stdin that wasn't yet ready to read #253
_long_sepcan now takeNonewhich splits the long form arguments into individual arguments #258- making
_pipedperform "direct" piping by default (linking fds together). this fixes memory problems #270 - bugfix where calling
next()on an iterable process that has raisedStopIteration, hangs #273 sh.cdcalled with no arguments no changes into the user's home directory, like nativecd#275sh.globremoved entirely. the rationale is correctness over hand-holding. #279- added
_truncate_exc, defaulting toTrue, which tells our exceptions to truncate output. - bugfix for exceptions whose messages contained unicode
_donecallback no longer assumes you want your command put in the background._donecallback is now called asynchronously in a separate thread._donecallback is called regardless of exception, which is necessary in order to release held resources, for example a process pool
1.10 - 12/30/14
- partially applied functions with
functools.partialhave been fixed for_outand_errcallbacks #160 _outor_errbeing callables no longer puts the running command in the background. to achieve the previous behavior, pass_bg=Trueto your command.- deprecated
_withcontexts #195 _timeout_signalallows you to specify your own signal to kill a timed-out process with. use a constant from thesignalstdlib module. #171- signal exceptions can now be caught by number or name.
SignalException_9 == SignalException_SIGKILL - child processes that timeout via
_timeoutraisesh.TimeoutExceptioninstead ofsh.SignalExeception_9#172 - fixed
help(sh)from the python shell andpydoc shfrom the command line. #173 - program names can no longer be shadowed by names that sh.py defines internally. removed the requirement of trailing underscores for programs that could have their names shadowed, like
id. - memory optimization when a child process's stdin is a newline-delimted string and our bufsize is newlines
- feature,
_donespecial keyword argument that accepts a callback to be called when the command completes successfully #185 - bugfix for being unable to print a baked command in python3+ #176
- bugfix for cwd not existing and causing the child process to continue running parent process code #202
- child process is now guaranteed to exit on exception between fork and exec.
- fix python2 deprecation warning when running with -3 PR #165
- bugfix where sh.py was attempting to execute directories #196, PR #189
- only backgrounded processes will ignore SIGHUP
- allowed
ok_codeto take arangeobject. #PR 210 - added
sh.argswith context which allows overriding of all command defaults for the duration of that context. - added
sh.pushdwith context which takes a directory name and changes to that directory for the duration of that with context. PR #206 - tests now include python 3.4 if available. tests also stop on the first
python that suite that fails. - SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGPIPE, SIGSYS have been added to the list of signals that throw an exception PR #201
- "callable" builtin has been faked for python3.1, which lacks it.
- "direct" option added to
_pipedspecial keyword argument, which allows sh to hand off a process's stdout fd directly to another process, instead of buffering its stdout internally, then handing it off. #119
1.09 - 9/08/13
- Fixed encoding errors related to a system encoding "ascii". #123
- Added exit_code attribute to SignalException and ErrorReturnCode exception classes. #127
- Making the default behavior of spawned processes to not be explicitly killed when the parent python process ends. Also making the spawned process ignore SIGHUP. #139
- Made OSX sleep hack to apply to PY2 as well as PY3.
1.08 - 1/29/12
- Added SignalException class and made all commands that end terminate by a signal defined in SIGNALS_THAT_SHOULD_THROW_EXCEPTION raise it. #91
- Bugfix where CommandNotFound was not being raised if Command was created by instantiation. #113
- Bugfix for Commands that are wrapped with functools.wraps() [#121](https://github.com/amoffat/sh/issues/121]
- Bugfix where input arguments were being assumed as ascii or unicode, but never as a string in a different encoding.
- _long_sep keyword argument added joining together a dictionary of arguments passed in to a command
- Commands can now be passed a dictionary of args, and the keys will be interpretted "raw", with no underscore-to-hyphen conversion
- Reserved Python keywords can now be used as subcommands by appending an underscore
_to them
1.07 - 11/21/12
- Bugfix for PyDev when
locale.getpreferredencoding()is empty. - Fixes for IPython3 that involve
sh.<tab>andsh? - Added
_teespecial keyword argument to force stdout/stderr to store internally and make available for piping data that is being redirected. - Added
_decode_errorsto be passed to all stdout/stderr decoding of a process. - Added
_no_out,_no_err, and_no_pipespecial keyword arguments. These are used for long-running processes with lots of output. - Changed custom loggers that were created for each process to fixed loggers, so there are no longer logger references laying around in the logging module after the process ends and it garbage collected.
1.06 - 11/10/12
- Removed old undocumented cruft of ARG1..ARGN and ARGV.
- Bugfix where
logging_enabledcould not be set from the importing module. - Disabled garbage collection before fork to prevent garbage collection in child process.
- Major bugfix where cyclical references were preventing process objects (and their associated stdout/stderr buffers) from being garbage collected.
- Bugfix in RunningCommand and OProc loggers, which could get really huge if a command was called that had a large number of arguments.
1.05 - 10/20/12
- Changing status from alpha to beta.
- Python 3.3 officially supported.
- Documentation fix. The section on exceptions now references the fact that signals do not raise an exception, even for signals that might seem like they should, e.g. segfault.
- Bugfix with Python 3.3 where importing commands from the sh namespace resulted in an error related to
__path__ - Long-form and short-form options to commands may now be given False to disable the option from being passed into the command. This is useful to pass in a boolean flag that you flip to either True or False to enable or disable some functionality at runtime.
1.04 - 10/07/12
- Making
Commandclass resolve thepathparameter withwhichby default instead of expecting it to be resolved before it is passed in. This change shouldn't affect backwards compatibility. - Fixing a bug when an exception is raised from a program, and the error output has non-ascii text. This didn't work in Python < 3.0, because .decode()'s default encoding is typically ascii.
SECURITY
Security Policy
Supported Versions
| Version | Supported |
|---|---|
| >= 2.2.5 | :white_check_mark: |
Reporting a Vulnerability
Please disclose any vulnerabilities to [email protected]