### Source/Examples/Done

Here's an example of using :ref:`done` to create a multiprocess pool, where
``sh.your_parallel_command`` is executed concurrently at no more than 10 at a
time:

.. code-block:: python

    import sh
    from threading import Semaphore

    pool = Semaphore(10)

    def done(cmd, success, exit_code):
        pool.release()

    def do_thing(arg):
        pool.acquire()
        return sh.your_parallel_command(arg, _bg=True, _done=done)

    procs = []
    for arg in range(100):
        procs.append(do_thing(arg))

    # essentially a join
    [p.wait() for p in procs]

---

### Source/Sections/Architecture

.. _architecture:

Architecture Overview
#####################

Launch
======

When it comes time to launch a process

#. Open pipes and/or TTYs STDIN/OUT/ERR.
#. Open a pipe for communicating pre-exec exceptions from the child to the
   parent.
#. Open a pipe for child/parent launch synchronization.
#. :func:`os.fork` a child process.

From here, we have two concurrent processes running:

Child
-----

#. If :ref:`_bg=True <bg>` is set, we ignore :py:data:`signal.SIGHUP`.
#. If :ref:`_new_session=True <new_session>`, become a session leader with
   :func:`os.setsid`, else become a process group leader with
   :func:`os.setpgrp`.
#. Write our session id to the a pipe connected to the parent.  This is mainly
   to synchronize with our parent that our session/group logic has finished.
#. :func:`os.dup2` the file descriptors of our previously-setup TTYs/pipes to
   our STDIN/OUT/ERR file descriptors.
#. If we're a session leader and our STDIN is a TTY, via :ref:`_tty_in=True
   <tty_in>`, acquire a controlling
   terminal, thereby becoming the controlling process of the session.
#. Set our GID/UID if we've set a custom one via :ref:`_uid <uid>`.
#. Close all file descriptors greater than STDERR.
#. Call :func:`os.execv`.

Parent
------

#. Check for any exceptions via the exception pipe connected to the child.
#. Block and read our child's session id from a pipe connected to the child.
   This synchronizes to us that the child has finished moving between
   sessions/groups and we can now accurately determine its current session id
   and process group.
#. If we're using a TTY for STDIN, via :ref:`_tty_in=True <tty_in>`, disable
   echoing on the TTY, so that data sent to STDIN is not echoed to STDOUT.

Running
=======

An instance of :ref:`oproc_class` contains two internal threads, one for STDIN,
and one for STDOUT and STDERR.  The purpose of these threads is to handle
reading/writing to the read/write ends of the process's standard descriptors.

For example, the STDOUT/ERR thread continually runs :func:`select.select` on the
master ends of the TTYs/pipes connected to STDOUT/ERR, and if they're ready to
read, reads the available data and aggregates it into the appropriate place.

.. _arch_buffers:

Buffers
-------

A couple of different buffers must be considered when thinking about how data
flows through an sh process.

The first buffer is the buffer associated with the underlying pipe or TTY
attached to STDOUT/ERR.  In the case of a TTY (the default for output), the
buffer size is 0, so output is immediate -- a byte written by the process is a
byte received by sh.  For a pipe, however, the buffer size of the pipe is
typically 4-64kb.  :manpage:`pipe(2)`.

.. seealso:: FAQ: :ref:`faq_tty_out`

The second buffer is sh's internal buffers, one for STDOUT and one for STDERR.
These buffers aggregate data that has been read from the master end of the TTY
or pipe attached to the output fd, but before that data is sent along to the
appropriate output handler (queue, file object, function, etc).  Data sits in
these buffers until we reach the size specified with :ref:`internal_bufsize`, at
which point the buffer flushes to the output handler.


Exit
====

STDIN Thread Shutdown
---------------------

On process completion, our internal threads must complete, as the read end of
STDIN, for example, which is connected to the process, is no longer open, so
writing to the slave end will no longer work.

STDOUT/ERR Thread Shutdown
--------------------------

The STDOUT/ERR thread is a little more complicated, because although the process
is not alive, output data may still exist in the pipe/TTY buffer that must be
collected.  So we essentially just :func:`select.select` on the read ends until
they return nothing, indicating that they are complete, then we break out of our
read loop.

.. _arch_exit_code:

Exit Code Processing
--------------------

The exit code is obtained from the reaped process.  If the process ended from a
signal, the exit code is the negative value of that signal.  For example,
SIGKILL would result in an exit code -9.

Done Callback
-------------

If specified, the :ref:`done` callback is executed with the :ref:`RunningCommand
<running_command>` instance, a boolean indicating success, and the adjusted exit
code.  After the callback returns, error processing continues.  In other words,
the done callback is called regardless of success or failure, and there's
nothing it can do to prevent the :ref:`ErrorReturnCode <error_return_code>`
exceptions from being raised after it completes.

---

### Source/Sections/Asynchronous Execution

.. _async:

Asynchronous Execution
######################

sh provides a few methods for running commands and obtaining output in a
non-blocking fashion.

AsyncIO
=======

.. versionadded:: 2.0.0

Sh supports asyncio on commands with the :ref:`_async=True <async_kw>` special
kwarg. This let's you incrementally ``await`` output produced from your command.

.. code-block:: python

	import asyncio
	import sh

	async def main():
	    await sh.sleep(3, _async=True)

	asyncio.run(main())

.. _iterable:
	    
Incremental Iteration
=====================

You may also create asynchronous commands by iterating over them with the
:ref:`iter` special kwarg.  This creates an iterable (specifically, a generator)
that you can loop over:

.. code-block:: python

	from sh import tail

	# runs forever
	for line in tail("-f", "/var/log/some_log_file.log", _iter=True):
	    print(line)
	    
By default, :ref:`iter` iterates over STDOUT, but you can change set this
specifically by passing either ``"err"`` or ``"out"`` to :ref:`iter` (instead of
``True``).  Also by default, output is line-buffered, so the body of the loop
will only run when your process produces a newline.  You can change this by
changing the buffer size of the command's output with :ref:`out_bufsize`.

.. note::

    If you need a *fully* non-blocking iterator, use :ref:`iter_noblock`.  If
    the current iteration would block, :py:data:`errno.EWOULDBLOCK` will be
    returned, otherwise you'll receive a chunk of output, as normal.

.. _background:
	
Background Processes
====================

By default, each running command blocks until completion.  If you have a
long-running command, you can put it in the background with the :ref:`_bg=True
<bg>` special kwarg:

.. code-block:: python

	# blocks
	sleep(3)
	print("...3 seconds later")
	
	# doesn't block
	p = sleep(3, _bg=True)
	print("prints immediately!")
	p.wait()
	print("...and 3 seconds later")

You'll notice that you need to call :meth:`RunningCommand.wait` in order to exit
after your command exits.

Commands launched in the background ignore ``SIGHUP``, meaning that when their
controlling process (the session leader, if there is a controlling terminal)
exits, they will not be signalled by the kernel.  But because sh commands launch
their processes in their own sessions by default, meaning they are their own
session leaders, ignoring ``SIGHUP`` will normally have no impact.  So the only
time ignoring ``SIGHUP`` will do anything is if you use :ref:`_new_session=False
<new_session>`, in which case the controlling process will probably be the shell
from which you launched python, and exiting that shell would normally send a
``SIGHUP`` to all child processes.

.. seealso::

    For more information on the exact launch process, see :ref:`architecture`.

.. _callbacks:

Output Callbacks
----------------
	    
In combination with :ref:`_bg=True<bg>`, sh can use callbacks to process output
incrementally by passing a callable function to :ref:`out` and/or :ref:`err`.
This callable will be called for each line (or chunk) of data that your command
outputs:

.. code-block:: python

	from sh import tail
	
	def process_output(line):
	    print(line)
	
	p = tail("-f", "/var/log/some_log_file.log", _out=process_output, _bg=True)
    p.wait()

To control whether the callback receives a line or a chunk, use
:ref:`out_bufsize`.  To "quit" your callback, simply return ``True``.  This
tells the command not to call your callback anymore.

The line or chunk received by the callback can either be of type ``str`` or
``bytes``. If the output could be decoded using the provided encoding, a
``str`` will be passed to the callback, otherwise it would be raw ``bytes``.

.. note::

    Returning ``True`` does not kill the process, it only keeps the callback
    from being called again.  See :ref:`interactive_callbacks` for how to kill a
    process from a callback.
	
.. seealso:: :ref:`red_func`

.. _interactive_callbacks:
	    
Interactive callbacks
---------------------

Commands may communicate with the underlying process interactively through a
specific callback signature
Each command launched through sh has an internal STDIN :class:`queue.Queue`
that can be used from callbacks:

.. code-block:: python

	def interact(line, stdin):
	    if line == "What... is the air-speed velocity of an unladen swallow?":
	        stdin.put("What do you mean? An African or European swallow?")
			
	    elif line == "Huh? I... I don't know that....AAAAGHHHHHH":
	        cross_bridge()
	        return True
			
	    else:
	        stdin.put("I don't know....AAGGHHHHH")
	        return True
			
	p = sh.bridgekeeper(_out=interact, _bg=True)
    p.wait()

.. note::

    If you use a queue, you can signal the end of the input (EOF) with ``None``

You can also kill or terminate your process (or send any signal, really) from
your callback by adding a third argument to receive the process object:

.. code-block:: python

	def process_output(line, stdin, process):
	    print(line)
	    if "ERROR" in line:
	        process.kill()
	        return True
	
	p = tail("-f", "/var/log/some_log_file.log", _out=process_output, _bg=True)
	
The above code will run, printing lines from ``some_log_file.log`` until the
word ``"ERROR"`` appears in a line, at which point the tail process will be
killed and the script will end.

.. note::

    You may also use :meth:`RunningCommand.terminate` to send a SIGTERM, or
    :meth:`RunningCommand.signal` to send a general signal.


Done Callbacks
--------------

A done callback called when the process exits, either normally (through
a success or error exit code) or through a signal.  It is *always* called.

.. include:: /examples/done.rst

---

### Source/Sections/Baking

.. _baking:

Baking
======

sh is capable of "baking" arguments into commands.  This is essentially
`partial application <https://en.wikipedia.org/wiki/Partial_application>`_,
like you might do with :func:`functools.partial`.

.. code-block:: python

	from sh import ls
	
	ls = ls.bake("-la")
	print(ls) # "/usr/bin/ls -la"
	
	# resolves to "ls -la /"
	print(ls("/"))

The idea here is that now every call to ``ls`` will have the "-la" arguments
already specified.  Baking can become very useful when you combine it with
:ref:`subcommands`:

.. code-block:: python

	from sh import ssh
	
	# calling whoami on a server.  this is a lot to type out, especially if
	# you wanted to call many commands (not just whoami) back to back on
	# the same server
	iam1 = ssh("myserver.com", "-p 1393", "whoami")
	
	# wouldn't it be nice to bake the common parameters into the ssh command?
	myserver = ssh.bake("myserver.com", p=1393)
	
	print(myserver) # "/usr/bin/ssh myserver.com -p 1393"
	
	# resolves to "/usr/bin/ssh myserver.com -p 1393 whoami"
	iam2 = myserver.whoami()
	
	assert(iam1 == iam2) # True!
	
Now that the "myserver" callable represents a baked ssh command, you
can call anything on the server easily:

.. code-block:: python
	
	# executes "/usr/bin/ssh myserver.com -p 1393 tail /var/log/dumb_daemon.log -n 100"
	print(myserver.tail("/var/log/dumb_daemon.log", n=100))

---

### Source/Sections/Command Class

API
###


.. _command_class:

Command Class
==============

The ``Command`` class represents a program that exists on the system and can be
run at some point in time.  An instance of ``Command`` is never running; an
instance of :ref:`RunningCommand <running_command>` is spawned for that.

An instance of ``Command`` can take the form of a manually instantiated object,
or as an object instantiated by dynamic lookup:

.. code-block:: python

    import sh

    ls1 = sh.Command("ls")
    ls2 = sh.ls
    
    assert ls1 == ls2


.. py:class:: Command(name, search_paths=None)

    Instantiates a Command instance, where *name* is the name of a program that
    exists on the user's ``$PATH``, or is a full path itself.  If *search_paths*
    is specified, it must be a list of all the paths to look for the program
    name.

    .. code-block:: python

        from sh import Command

        ifconfig = Command("ifconfig")
        ifconfig = Command("/sbin/ifconfig")


.. py:method:: Command.bake(*args, **kwargs)

    Returns a new Command with ``*args`` and ``**kwargs`` baked in as
    positional and keyword arguments, respectively.  Any future calls to the
    returned Command will include ``*args`` and ``**kwargs`` automatically:

    .. code-block:: python

        from sh import ls

        long_ls = ls.bake("-l")
        print(ls("/var"))
        print(ls("/tmp"))
        
    
    .. seealso::

        :ref:`baking`


Similar to the above, arguments to the ``sh.Command`` must be separate.
e.g. the following does not work::

		lscmd = sh.Command("/bin/ls -l")
		tarcmd = sh.Command("/bin/tar cvf /tmp/test.tar /my/home/directory/")

You will run into ``CommandNotFound(path)`` exception even when correct full path is specified.
The correct way to do this is to :

#. build ``Command`` object using *only* the binary
#. pass the arguments to the object *when invoking*

as follows::

		lscmd = sh.Command("/bin/ls")
		lscmd("-l")
		tarcmd = sh.Command("/bin/tar")
		tarcmd("cvf", "/tmp/test.tar", "/my/home/directory/")

.. _running_command:

RunningCommand Class
====================

This represents a :ref:`Command <command_class>` instance that has been
or is being executed.  It exists as a wrapper around the low-level :ref:`OProc
<oproc_class>`.  Most of your interaction with sh objects are with instances of
this class. It is only returned if ``_return_cmd=True`` when you execute a command.

.. warning::

    Objects of this class behave very much like strings.  This was an
    intentional design decision to make the "output" of an executing Command
    behave more intuitively.

    Be aware that functions that accept real strings only, for example
    ``json.dumps``, will not work on instances of RunningCommand, even though it
    look like a string.

.. _wait_method:

.. py:method:: RunningCommand.wait(timeout=None)

    :param timeout: An optional non-negative number to wait for the command to complete. If it doesn't complete by the
        timeout, we raise :ref:`timeout_exc`.

    Block and wait for the command to finish execution and obtain an exit code.
    If the exit code represents a failure, we raise the appropriate exception.
    See :ref:`exceptions <exceptions>`.

    .. note::
        
        Calling this method multiple times only yields an exception on the first
        call.

    This is called automatically by sh unless your command is being executed
    :ref:`asynchronously <async>`, in which case, you may want to call this
    manually to ensure completion.

    If an instance of :ref:`Command <command_class>` is being used as the stdin
    argument (see :ref:`piping <piping>`), :meth:`wait` is also called on that
    instance, and any exceptions resulting from that process are propagated up.

.. py:attribute:: RunningCommand.process

    The underlying :ref:`OProc <oproc_class>` instance.

.. py:attribute:: RunningCommand.stdout

    A ``@property`` that calls :meth:`wait` and then returns the contents of
    what the process wrote to stdout.

.. py:attribute:: RunningCommand.stderr

    A ``@property`` that calls :meth:`wait` and then returns the contents of
    what the process wrote to stderr.

.. py:attribute:: RunningCommand.exit_code

    A ``@property`` that calls :meth:`wait` and then returns the process's exit
    code.

.. py:attribute:: RunningCommand.pid

    The process id of the process.

.. py:attribute:: RunningCommand.sid

    The session id of the process.  This will typically be a different session
    than the current python process, unless :ref:`_new_session=False
    <new_session>` was specified.

.. py:attribute:: RunningCommand.pgid

    The process group id of the process.

.. py:attribute:: RunningCommand.ctty

    The controlling terminal device, if there is one.

.. py:method:: RunningCommand.signal(sig_num)

    Sends *sig_num* to the process.  Typically used with a value from the
    :mod:`signal` module, like :py:data:`signal.SIGHUP` (see :manpage:`signal(7)`).

.. py:method:: RunningCommand.signal_group(sig_num)

    Sends *sig_num* to every process in the process group.  Typically used with
    a value from the :mod:`signal` module, like :py:data:`signal.SIGHUP` (see
    :manpage:`signal(7)`).

.. py:method:: RunningCommand.terminate()

    Shortcut for :meth:`RunningCommand.signal(signal.SIGTERM)
    <RunningCommand.signal>`.

.. py:method:: RunningCommand.kill()

    Shortcut for :meth:`RunningCommand.signal(signal.SIGKILL)
    <RunningCommand.signal>`.

.. py:method:: RunningCommand.kill_group()

    Shortcut for :meth:`RunningCommand.signal_group(signal.SIGKILL)
    <RunningCommand.signal_group>`.

.. py:method:: RunningCommand.is_alive()

    Returns whether or not the process is still alive.

    :rtype: bool

.. _oproc_class:

OProc Class
===========

.. warning::

    Don't use instances of this class directly.  It is being documented here for
    posterity, not for direct use.

.. py:method:: OProc.wait()

    Block until the process completes, aggregate the output, and populate
    :attr:`OProc.exit_code`.

.. py:attribute:: OProc.stdout

    A :class:`collections.deque`, sized to :ref:`_internal_bufsize
    <internal_bufsize>` items, that contains the process's STDOUT.

.. py:attribute:: OProc.stderr

    A :class:`collections.deque`, sized to :ref:`_internal_bufsize
    <internal_bufsize>` items, that contains the process's STDERR.

.. py:attribute:: OProc.exit_code

    Contains the process's exit code, or ``None`` if the process has not yet
    exited.

.. py:attribute:: OProc.pid

    The process id of the process.

.. py:attribute:: OProc.sid

    The session id of the process.  This will typically be a different session
    than the current python process, unless :ref:`_new_session=False
    <new_session>` was specified.

.. py:attribute:: OProc.pgid

    The process group id of the process.

.. py:attribute:: OProc.ctty

    The controlling terminal device, if there is one.

.. py:method:: OProc.signal(sig_num)

    Sends *sig_num* to the process.  Typically used with a value from the
    :mod:`signal` module, like :py:data:`signal.SIGHUP` (see :manpage:`signal(7)`).

.. py:method:: OProc.signal_group(sig_num)

    Sends *sig_num* to every process in the process group.  Typically used with
    a value from the :mod:`signal` module, like :py:data:`signal.SIGHUP` (see
    :manpage:`signal(7)`).

.. py:method:: OProc.terminate()

    Shortcut for :meth:`OProc.signal(signal.SIGTERM) <OProc.signal>`.

.. py:method:: OProc.kill()

    Shortcut for :meth:`OProc.signal(signal.SIGKILL) <OProc.signal>`.

.. py:method:: OProc.kill_group()

    Shortcut for :meth:`OProc.signal_group(signal.SIGKILL)
    <OProc.signal_group>`.

Exceptions
==========

.. _error_return_code:

ErrorReturnCode
---------------

.. py:class:: ErrorReturnCode

    This is the base class for, as the name suggests, error return codes.  It
    subclasses :py:class:`Exception`.

.. py:attribute:: ErrorReturnCode.full_cmd

    The full command that was executed, as a string, so that you can try it on
    the commandline if you wish.

.. py:attribute:: ErrorReturnCode.stdout

    The total aggregated STDOUT for the process.

.. py:attribute:: ErrorReturnCode.stderr

    The total aggregated STDERR for the process.

.. py:attribute:: ErrorReturnCode.exit_code

    The process's adjusted exit code.

    .. seealso:: :ref:`arch_exit_code`


.. _signal_exc:

SignalException
---------------

Subclasses :ref:`ErrorReturnCode <error_return_code>`.  Raised when a command
receives a signal that causes it to exit.

.. _timeout_exc:

TimeoutException
----------------

Raised when a command specifies a non-null :ref:`timeout` and the command times out:

.. code-block:: python

    import sh

    try:
        sh.sleep(10, _timeout=1)
    except sh.TimeoutException:
        print("we timed out, as expected")

Also raised when you specify a timeout to :ref:`RunningCommand.wait(timeout=None)<wait_method>`:

.. code-block:: python

    import sh

    p = sh.sleep(10, _bg=True)
    try:
        p.wait(timeout=1)
    except sh.TimeoutException:
        print("we timed out waiting")
        p.kill()

.. _not_found_exc:

CommandNotFound
---------------

This exception is raised in one of the following conditions:

* The program cannot be found on your path.
* You do not have permissions to execute the program.
* The program is not marked executable.

The last two bullets may seem strange, but they fall in line with how a shell like Bash behaves when looking up a
program to execute.

.. note::

    ``CommandNotFound`` subclasses ``AttributeError``. As such, the `repr` of it is simply the name of the missing
    attribute.


Helper Functions
================

.. py:function:: which(name, search_paths=None)

    Resolves *name* to program's absolute path, or ``None`` if it cannot be
    found.  If *search_paths* is list of paths, use that list to look for the
    program, otherwise use the environment variable ``$PATH``.

.. py:function:: pushd(directory)

    This function provides a ``with`` context that behaves similar to Bash's
    `pushd
    <https://www.gnu.org/software/bash/manual/html_node/Directory-Stack-Builtins.html>`_
    by pushing to the provided directory, and popping out of it at the end of
    the context.

    .. code-block:: python
        
        import sh

        with sh.pushd("/tmp"):
            sh.touch("a_file")

    .. note::

        It should be noted that we use a reentrant lock, so that different threads
        using this function will have the correct behavior inside of their ``with``
        contexts.

---

### Source/Sections/Contrib

.. _contrib:

Contrib Commands
################

Contrib is an sh sub-module that provides friendly wrappers to useful commands.
Typically, the commands being wrapped are unintuitive, and the contrib version
makes them intuitive.

.. note::

    Contrib commands should be considered generally unstable. They will grow and change as the community figures out the
    best interface for them.

Commands
========

Sudo
----

Allows you to enter your password from the terminal at runtime, or as a string
in your script.

.. py:function:: sudo(password=None, *args, **kwargs)

    Call sudo with ``password``, if specified, else ask the executing user for a
    password at runtime via :func:`getpass.getpass`.

.. seealso:: :ref:`contrib_sudo`

.. _contrib_git:

Git
---

Many git commands use a pager for output, which can cause an unexpected behavior
when run through sh.  To account for this, the contrib version sets
``_tty_out=False`` for all git commands.

.. py:function:: git(*args, **kwargs)

    Call git with STDOUT connected to a pipe, instead of a TTY.

.. code-block:: python

    from sh.contrib import git
    repo_log = git.log()

.. seealso:: :ref:`faq_tty_out` and :ref:`faq_color_output`

.. _contrib_ssh:

SSH
---

.. versionadded:: 1.13.0

SSH password-based logins :ref:`can be a pain <tutorial2>`. This contrib command performs all of the ugly setup and
provides a clean interface to using SSH.

.. py:function:: ssh(interact=None, password=None, prompt_match=None, login_success=None, *args, **kwargs)

    :param interact: A callback to handle SSH session interaction *after* login is successful. Required.
    :param password: A password string or a function that returns a password string. Optional. If not provided, :func:`getpass.getpass` is used.
    :param prompt_match: The string to match in order to determine when to provide SSH with the password. Or a function
            that matches on the output. Optional.
    :param login_success: A function to determine if SSH login is successful. Optional.

The ``interact`` parameter takes a callback with a signature that is slightly different to the function callbacks for
:ref:`redirection <red_func>`:

.. py:function:: fn(content, stdin_queue)
    
    :param content: An instance of an ephemeral :ref:`SessionContent <session_content>` class whose job is to hold the
            characters that the SSH session has written to STDOUT.
    :param stdin_queue: A :class:`queue.Queue` object to communicate with STDIN programmatically.

``password`` can be simply a string that will be used to type the password. If it's not provided, it will be read from STDIN
at runtime via :func:`getpass.getpass`. It can also be a callable that returns the password string.

``prompt_match`` is a string to match before the contrib command will provide the SSH process with the password. It is
optional, and if left unspecified, will default to "password: ". It can also be a callable that is called on a
:ref:`SessionContent <session_content>` instance and returns ``True`` or ``False`` for a match.

``login_success`` is a function that takes a :ref:`SessionContent <session_content>` object and returns a boolean for
whether or not a successful login occurred. It is optional, and if unspecified, simply evaluates to ``True``, meaning
any password submission results in a successful login (obviously not always correct). It is recommended that you specify
this.

.. _session_content:

.. py:class:: SessionContent()

    This class contains a record lines and characters written to the SSH processes's STDOUT. It should be all you need
    from the callbacks to determine how to interact with the SSH process.

.. py:attribute:: SessionContent.chars
    
    :type: :class:`collections.deque`
    
    The previous 50,000 characters.

.. py:attribute:: SessionContent.lines
    
    :type: :class:`collections.deque`
    
    The previous 5,000 lines.

.. py:attribute:: SessionContent.line_chars
    
    :type: list

    The characters in the line currently being aggregated.

.. py:attribute:: SessionContent.cur_line
    
    :type: str

    A string of the line currently being aggregated.

.. py:attribute:: SessionContent.last_line
    
    :type: str

    The previous line.

.. py:attribute:: SessionContent.cur_char
    
    :type: str

    The currently written character.


.. _contrib_bash:

Bash
---

Often users may find themselves having to run bash commands directly, whether due
to commands having special characters (e.g. dash, or dot) or other reasons. 
This can lead into recurrently having to bake the ``bash`` command to call it directly. To 
account for this, the contrib version provides a ``bash`` command baked in:

.. py:function:: bash(*args, **kwargs)

    Call bash with the prefix of "bash -c [...]".

.. code-block:: python

    from sh.contrib import bash

    # Calling commands directly
    bash.ls() # equivalent to "bash -c ls"

    # Or adding the full commands
    bash("command-with-dashes args")


Extending
=========

For developers.

To extend contrib, simply decorate a function in sh with the ``@contrib``
decorator, and pass in the name of the command you wish to shadow to the
decorator.  This method must return an instance of :ref:`Command
<command_class>`:

.. code-block:: python

    @contrib("ls")
    def my_ls(original):
        ls = original.bake("-l")
        return ls

Now you can run your custom contrib command from your scripts, and you'll be
using the command returned from your decorated function:


.. code-block:: python

    from sh.contrib import ls

    # executing: ls -l
    print(ls("/"))

For even more flexibility, you can design your contrib command to rewrite its
options based on *executed* arguments.  For example, say you only wish to set a
command's argument if another argument is set.  You can accomplish it like this:

.. code-block:: python

    @contrib("ls")
    def my_ls(original):
        def process(args, kwargs):
            if "-a" in args:
                args.append("-L")
            return args, kwargs

        ls = original.bake("-l")
        return ls, process

Returning a process function along with the command will tell sh to use that
function to preprocess the arguments at execution time using the
:ref:`_arg_preprocess <preprocess>` special kwarg.

---

### Source/Sections/Default Arguments

.. _default_arguments:

Default Arguments
=================

Many times, you want to override the default arguments of all commands launched
through sh.  For example, suppose you want the output of all commands to be
aggregated into a :class:`io.StringIO` buffer.  The naive way would be this:

.. code-block:: python

    import sh
    from io import StringIO

    buf = StringIO()

    sh.ls("/", _out=buf)
    sh.whoami(_out=buf)
    sh.ps("auxwf", _out=buf)

Clearly, this gets tedious quickly.  Fortunately, we can create execution
contexts that allow us to set default arguments on all commands spawned from
that context:

.. code-block:: python

    import sh
    from io import StringIO

    buf = StringIO()
    sh2 = sh.bake(_out=buf)

    sh2.ls("/")
    sh2.whoami()
    sh2.ps("auxwf")

Now, anything launched from ``sh2`` will send its output to the ``StringIO``
instance ``buf``.

---

### Source/Sections/Envs

.. _environments:

Environments
============

The :ref:`_env <env>` special kwarg allows you to pass a dictionary of
environment variables and their corresponding values:

.. code-block:: python

	import sh
	sh.google_chrome(_env={"SOCKS_SERVER": "localhost:1234"})
	

:ref:`_env <env>` replaces your process's environment completely.  Only the
key-value pairs in :ref:`_env <env>` will be used for its environment.  If you
want to add new environment variables for a process *in addition to* your
existing environment, try something like this:

.. code-block:: python

    import os
    import sh
    
    new_env = os.environ.copy()
    new_env["SOCKS_SERVER"] = "localhost:1234"
    
    sh.google_chrome(_env=new_env)

.. seealso::

    To make an environment apply to all sh commands look into
    :ref:`default_arguments`.

---

### Source/Sections/Exit Codes

.. _exit_codes:

Exit Codes & Exceptions
=======================

Normal processes exit with exit code 0.  This can be seen from
:attr:`RunningCommand.exit_code`:

.. code-block:: python

	output = ls("/", _return_cmd=True)
	print(output.exit_code) # should be 0
	
If a process terminates, and the exit code is not 0, an exception is generated
dynamically.  This lets you catch a specific return code, or catch all error
return codes through the base class :class:`ErrorReturnCode`:

.. code-block:: python

    try:
        print(ls("/some/non-existent/folder"))
    except ErrorReturnCode_2:
        print("folder doesn't exist!")
        create_the_folder()
    except ErrorReturnCode:
        print("unknown error")

You can also customize which exit codes indicate an error with :ref:`ok_code`. For example:

.. code-block:: python

   for i in range(10):
    	sh.grep("string to check", f"file_{i}.txt", _ok_code=(0, 1))

where the :ref:`ok_code` makes a failure to find a match a no-op.

Signals
-------

Signals are raised whenever your process terminates from a signal.  The
exception raised in this situation is :ref:`signal_exc`, which subclasses
:ref:`error_return_code`.

.. code-block:: python

    try:
        p = sh.sleep(3, _bg=True)
        p.kill()
    except sh.SignalException_SIGKILL:
        print("killed")

This behavior could be blocked by appending the negative value of the signal to
:ref:`ok_code`. All signals that raises :ref:`signal_exc` are ``[SIGABRT, 
SIGBUS, SIGFPE, SIGILL, SIGINT, SIGKILL, SIGPIPE, SIGQUIT, SIGSEGV, SIGTERM, 
SIGTERM]``.

.. note::

    You can catch :ref:`signal_exc` by using either a number or a signal name.
    For example, the following two exception classes are equivalent:

    .. code-block:: python

        assert sh.SignalException_SIGKILL == sh.SignalException_9

---

### Source/Sections/Faq

.. _faq:

FAQ
===

How do I execute a bash builtin?
--------------------------------

.. code-block:: python

    import sh

    sh.bash("-c", "your_builtin")

Or

.. code-block:: python

    import sh

    builtins = sh.bash.bake("-c")
    builtins("your_builtin")


Will Windows be supported?
--------------------------

There are no plans to support Windows.

.. _faq_append:

How do I append output to a file?
---------------------------------

Use a file object opened in the mode you desire:

.. code-block:: python

    import sh

    h = open("/tmp/output", "a")

    sh.ls("/dir1", _out=h)
    sh.ls("/dir2", _out=h)

.. _faq_color_output:

Why does my command's output have color?
----------------------------------------

Typically the reason for this is that your program detected that its STDOUT was
connected to a TTY, and therefore decided to print color escape sequences in its
output.  The typical solution is to use :ref:`_tty_out=False <tty_out>`, which
will force a pipe to be connected to STDOUT, and probably change the behavior of
the program.

.. seealso::

    Git is one of the programs that makes extensive use of terminal colors (as
    well as pagers) in its output, so we added :ref:`a contrib version
    <contrib_git>` for convenience.

.. _faq_tty_out:

Why is _tty_out=True the default?
---------------------------------

This was a design decision made for two reasons:

1. To make programs behave in the same way as seen on the commandline.
2. To provide better buffering control than pipes allow.

For #1, we want sh to produce output that is identical to what the user sees
from the commandline, because that's typically the only output they ever see
from their command.  This makes the output easy to understand.

For #2, using a TTY for STDOUT allows us to precisely control the buffering of a
command's output to sh's internal code.

.. seealso:: :ref:`arch_buffers`

Of course, there are some gotchas with TTY STDOUT.  One of them is commands that
use a pager, for example:

.. code-block:: python

    import sh
    print(sh.git.log())


This will sometimes raise a ``SignalException_SIGPIPE``. The reason is because
``git log`` detects a TTY STDOUT and forks the system’s pager (typically
``less``) to handle the output. The pager checks for a controlling terminal,
and, finding none, exits with exit code 1. The exit of the pager means no more
readers on ``git log``’s output, and thus a ``SIGPIPE`` is received.

One solution to the ``git log`` problem above is simply to use
``_tty_out=False``. Another option, specifically for git, is to use the
``git --no-pager`` option:

.. code-block:: python

    import sh
    print(sh.git('--no-pager', 'log'))


Why doesn't "*" work as a command argument?
-------------------------------------------

Glob expansion is a feature of a shell, like Bash, and is performed by the shell
before passing the results to the program to be exec'd.  Because sh is not a
shell, but rather tool to execute programs directly, we do not handle glob
expansion like a shell would.

So in order to use ``"*"`` like you would on the commandline, pass it into
:func:`glob.glob` first:

.. code-block:: python

    import sh
    import glob
    sh.ls(glob.glob("*.py"))


.. _faq_path:

How do I call a program that isn't in ``$PATH``?
------------------------------------------------

Use the :meth:`Command` constructor to instantiate an instance of Command
directly, then execute that:

.. code-block:: python

    import sh
    cmd = sh.Command("/path/to/command")
    cmd("-v", "arg1")

How do I execute a program with a dash in its name?
---------------------------------------------------

If it's in your ``$PATH``, substitute the dash for an underscore:

.. code-block:: python

    import sh
    sh.google_chrome("http://google.com")

The above will run ``google-chrome http://google.com``

.. note::

    If a program named ``google_chrome`` exists on your system, that will be
    called instead.  In that case, in order to execute the program with a dash
    in the name, you'll have to use the method described :ref:`here.
    <faq_special>`

.. _faq_special:

How do I execute a program with a special character in its name?
----------------------------------------------------------------

Programs with non-alphanumeric, non-dash characters in their names cannot be
executed directly as an attribute on the sh module.  For example, **this will not
work:**

.. code-block:: python

    import sh
    sh.mkfs.ext4()

The reason should be fairly obvious.  In Python, characters like ``.`` have
special meaning, in this case, attribute access.  What sh is trying to do in the
above example is find the program "mkfs" (which may or may not exist) and then
perform a :ref:`subcommand lookup <subcommands>` with the name "ext4".  In other
words, it will try to call ``mkfs`` with the argument ``ext4``, which is
probably not what you want.

The workaround is instantiating the :ref:`Command Class <command_class>` with
the string of the program you're looking for:

.. code-block:: python

    import sh
    mkfsext4 = sh.Command("mkfs.ext4")
    mkfsext4() # run it

.. _faq_pipe_syntax:


Why not use ``|`` to pipe commands?
-----------------------------------

I prefer the syntax of sh to resemble function composition instead of a
pipeline.  One of the goals of sh is to make executing processes more like
calling functions, not making function calls more like Bash.

Why isn't piping asynchronous by default?
-----------------------------------------

There is a non-obvious reason why async piping is not possible by default.
Consider the following example:

.. code-block:: python

    import sh

    sh.cat(sh.echo("test\n1\n2\n3\n"))

When this is run, ``sh.echo`` executes and finishes, then the entire output
string is fed into ``sh.cat``.  What we would really like is each
newline-delimited chunk to flow to ``sh.cat`` incrementally.

But for this example to flow data asynchronously from echo to cat, the echo
command would need to *not block.*  But how can the inner command know the
context of its execution, to know to block sometimes but not other times?  It
can't know that without something explicit.

This is why the :ref:`piped` special kwarg was introduced.  By default, commands
executed block until they are finished, so in order for an inner command to not
block, ``_piped=True`` signals to the inner command that it should not block.
This way, the inner command starts running, then very shortly after, the outer
command starts running, and both are running simultaneously.  Data can then flow
from the inner command to the outer command asynchronously:

.. code-block:: python

    import sh

    sh.cat(sh.echo("test\n1\n2\n3\n", _piped=True))

Again, this example is contrived -- a better example would be a long-running
command that produces a lot of output that you wish to pipe through another
program incrementally.

How do I run a command and connect it to sys.stdout and sys.stdin?
------------------------------------------------------------------

There are two ways to do this

.. seealso:: :ref:`fg`

You can use :data:`sys.stdin`, :data:`sys.stdout`, and :data:`sys.stderr` as
arguments to :ref:`in`, :ref:`out`, :ref:`err`, respectively, and it *should*
mostly work as expected:

.. code-block:: python

    import sh
    import sys
    sh.your_command(_in=sys.stdin, _out=sys.stdout)

There are a few reasons why this probably won't work.  The first reason is that
:data:`sys.stdin` is probably a controlling TTY (attached to the shell that
launched the python process), and probably not set in raw mode
:manpage:`termios(3)`, which means that, among other things, input is buffered
by newlines.

The real solution is to use :ref:`_fg=True <fg>`:

.. code-block:: python

    import sh
    sh.top(_fg=True)


.. _faq_separate_args:

Why do my arguments need to be separate strings?
------------------------------------------------

This confuses many new sh users.  They want to do something like this and expect
it to just work:

.. code-block:: python

    from sh import tar
    tar("cvf /tmp/test.tar /my/home/directory")

But instead they'll get a confusing error message:

.. code-block:: none

    RAN: '/bin/tar cvf /tmp/test.tar /my/home/directory'

    STDOUT:

    STDERR:
    /bin/tar: Old option 'f' requires an argument.
    Try '/bin/tar --help' or '/bin/tar --usage' for more information.

The reason why they expect it to work is because shells, like Bash, automatically
parse your commandline and break up arguments for you, before sending them to
the binary.  They have a complex set of rules (some of which are represented by
:mod:`shlex`) to take a single string of a command and arguments and separate
them.

Even if we wanted to implement this in sh (which we don't), it would hurt the
ability for users to parameterize parts of their arguments.  They would have to
use string interpolation, which would be ugly and error prone:

.. code-block:: python

    from sh import tar
    tar("cvf %s %s" % ("/tmp/tar1.tar", "/home/oh no a space")

In the above example, ``"/home/oh"``, ``"no"``, ``"a"``, and ``"space"`` would
all be separate arguments to tar, causing the program to behave unexpectedly.
Basically every command with parameterized arguments would need to expect
characters that could break the parser.

.. _faq_arg_ordering:

How do I order keyword arguments?
---------------------------------

Typically this question gets asked when a user is trying to execute something
like the following commandline:

.. code-block:: none

    my-command --arg1=val1 arg2 --arg3=val3

This is usually the first attempt that they make:

.. code-block:: python

    sh.my_command(arg1="val1", "arg2", arg3="val3")

This doesn't work because, in Python, position arguments, like ``arg2`` cannot
come after keyword arguments.

Furthermore, it is entirely possible that ``--arg3=val3`` comes before
``--arg1=val1``.  The reason for this is that a function's ``**kwargs`` is an
unordered mapping, and so key-value pairs are not guaranteed to resolve to a
specific order.

So the solution here is to forego the usage of the keyword argument
*convenience*, and just use raw ordered arguments:

.. code-block:: python

    sh.my_command("--arg1=val1", "arg2", "--arg3=val3")

.. _faq_pylint:

How to disable pylint E1101 no-member errors?
---------------------------------------------

Pylint complains with E1101 no-member to almost all ``sh.command`` invocations,
because it doesn't know, that these members are generated dynamically.
Starting with Pylint 1.6 these messages can be suppressed using `generated-members <https://docs.pylint.org/en/1.6.0/features.html#id28>`_ option.

Just add following lines to ``pylintrc``::

    [TYPECHECK]
    generated-members=sh


How do I patch sh in my tests?
------------------------------

sh can be patched in your tests the typical way, with
:func:`unittest.mock.patch`:

.. code-block:: python

    from unittest.mock import patch
    import sh

    def get_something():
        return sh.pwd()

    @patch("sh.pwd", create=True)
    def test_something(pwd):
        pwd.return_value = "/"
        assert get_something() == "/"

The important thing to note here is that ``create=True`` is set.  This is
required because sh is a bit magical and ``patch`` will fail to find the ``pwd``
command as an attribute on the sh module.

You may also patch the :class:`Command` class:

.. code-block:: python

    from unittest.mock import patch
    import sh

    def get_something():
        pwd = sh.Command("pwd")
        return pwd()

    @patch("sh.Command")
    def test_something(Command):
        Command().return_value = "/"
        assert get_something() == "/"

Notice here we do not need ``create=True``, because :class:`Command` is not an
automatically generated object on the sh module (it actually exists).


Why is sh just a single file?
-----------------------------

When sh was first written, the design decision was made to make it a single-file
module.  This has pros and cons:

Cons:

- Auditing the code is more challenging
- Without file-enforced structure, adding more features and abstractions makes
  the code harder to follow
- Cognitively, it feels cluttered

Pros:

- Can be used easily on systems without Python package managers
- Can be embedded/bundled together with other software more easily
- Cognitively, it feels more self-contained

In my mind, because the primary target audience of sh users is generally more
scrappy devops, systems people, or people just trying to stitch together some
clunky system programs, the listed pros weigh a little more heavily than the
cons.  Sacrificing some development advantages to give those users a more
flexible tool is a win to me.

Down the road, the development disadvantages of a single file can be solved with
additional development tools, for example, with a tool that compiles multiple
modules into the single sh.py file.  Realistically, though, sh is pretty mature,
so I don't see it growing much more in complexity or code size.

How do I see the commands sh is running?
----------------------------------------

Use logging:

.. code-block:: python

    import logging
    import sh

    logging.basicConfig(level=logging.INFO)
    sh.ls()

.. code-block:: none

    INFO:sh.command:<Command '/bin/ls'>: starting process
    INFO:sh.command:<Command '/bin/ls', pid 32394>: process started
    INFO:sh.command:<Command '/bin/ls', pid 32394>: process completed
    ...

---

### Source/Sections/Migration

.. _migration:

Migrating from 1.x to 2.x
##########################

This document provides an upgrade path from ``1.*`` to ``2.*``.


``sh.cd`` builtin removed
=========================

There is no ``sh.cd`` command anymore. It was always a command implemented in
sh, as some systems provide it as a shell builtin, while others have an actual
binary. But neither of them persisted the directory change between other ``sh``
calls, which is why it was implemented in sh.

Workaround
----------

If you were using ``sh.cd(dir)``, use the context manager ``with sh.pushd(dir)``
instead. All of the commands in the managed context will have the correct
directory.


Removed execution contexts / default arguments
===============================================

In ``1.*`` you could spawn a new module from the ``sh`` module, one which had
customized defaults for the special keyword arguments. This module could then be
accessed just like ``sh``, and you could even import commands from it.

Unfortunately the magic required to make that work was brittle. Also it was not
aligned syntactically with the similar baking concept. We have therefore changed
the syntax to align with baking, and also removed the ability to import directly
from this new baked execution context.

Workaround
----------

.. code-block:: python

    sh2 = sh(_tty_out=False)
    sh2.ls()

Becomes:

.. code-block:: python

    sh2 = sh.bake(_tty_out=False)
    sh2.ls()

And:

.. code-block:: python

    sh2 = sh.bake(_tty_out=False)
    from sh2 import ls
    ls()

Becomes:

.. code-block:: python

    sh2 = sh.bake(_tty_out=False)
    ls = sh2.ls
    ls()


Return value now a true string
===============================

In ``2.*``, the return value of an executed ``sh`` command has changed (in most
cases) from a ``RunningCommand`` object to a unicode string. This makes using
the output of a command more natural.

Workaround
----------

To continue returning a ``RunningCommand`` object, use the ``_return_cmd=True``
special keyword argument. You can achieve this on each file with the following
code at the top of files that use ``sh``:

.. code-block:: python

    import sh

    sh = sh.bake(_return_cmd=True)


Piping to STDIN
===============

Previously, if the first argument of a sh command was an instance of
``RunningCommand``, it was automatically fed into the process's STDIN. This is
no longer the case and you must explicitly use ``_in=``.

.. code-block:: python

    from sh import wc, ls

    print(wc(ls("/home/<user>", "-l"), "-l"))

Becomes:

.. code-block:: python

    from sh import wc, ls

    print(wc("-l", _in=ls("/home/<user>", "-l")))

Or:

.. code-block:: python

    from sh import wc, ls

    print(wc("-l", _in=ls("/home/<user>", "-l", _return_cmd=True)))

Workaround
----------

None.


New processes don't launch in new session
==========================================

In ``1.*``, ``_new_session`` defaulted to ``True``. It now defaults to
``False``. The reason for this is that it makes more sense for launched
processes to default to being in the process group of the Python script, so
that they receive SIGINTs correctly.

Workaround
----------

To preserve the old behavior:

.. code-block:: python

    import sh

    sh = sh.bake(_new_session=True)

---

### Source/Sections/Passing Arguments

.. _passing_arguments:

Passing Arguments
=================

When passing multiple arguments to a command, each argument *must* be a separate
string:

.. code-block:: python

    from sh import tar
    tar("cvf", "/tmp/test.tar", "/my/home/directory/")

This *will not work*:

.. code-block:: python

    from sh import tar
    tar("cvf /tmp/test.tar /my/home/directory")
	
.. seealso:: :ref:`faq_separate_args`


Keyword Arguments
-----------------

sh supports short-form ``-a`` and long-form ``--arg`` arguments as
keyword arguments:

.. code-block:: python

	# resolves to "curl http://duckduckgo.com/ -o page.html --silent"
	curl("http://duckduckgo.com/", o="page.html", silent=True)
	
	# or if you prefer not to use keyword arguments, this does the same thing:
	curl("http://duckduckgo.com/", "-o", "page.html", "--silent")
	
	# resolves to "adduser amoffat --system --shell=/bin/bash --no-create-home"
	adduser("amoffat", system=True, shell="/bin/bash", no_create_home=True)
	
	# or
	adduser("amoffat", "--system", "--shell", "/bin/bash", "--no-create-home")

.. seealso:: :ref:`faq_arg_ordering`

---

### Source/Sections/Piping

.. _piping:

Piping
======

Basic
-----

Bash style piping is performed using function composition.  Just pass one
command as the input to another's ``_in`` argument, and sh will send the output of
the inner command to the input of the outer command:

.. code-block:: python

	# sort this directory by biggest file
	print(sort("-rn", _in=du(glob("*"), "-sb")))
	
	# print(the number of folders and files in /etc
	print(wc("-l", _in=ls("/etc", "-1")))

.. note::

    This basic piping does not flow data through asynchronously; the inner
    command blocks until it finishes, before sending its data to the outer
    command.
	
By default, any command that is piping another command in waits for it to
complete.  This behavior can be changed with the :ref:`_piped <piped>` special
kwarg on the command being piped, which tells it not to complete before sending
its data, but to send its data incrementally.  Read ahead for examples of this.

.. _advanced_piping:

Advanced
--------

By default, all piped commands execute sequentially.  What this means is that the
inner command executes first, then sends its data to the outer command:

.. code-block:: python

	print(wc("-l", _in=ls("/etc", "-1")))
	
In the above example, ``ls`` executes, gathers its output, then sends that output
to ``wc``.  This is fine for simple commands, but for commands where you need
parallelism, this isn't good enough.  Take the following example:

.. code-block:: python

	for line in tr(_in=tail("-f", "test.log"), "[:upper:]", "[:lower:]", _iter=True):
	    print(line)
	
**This won't work** because the ``tail -f`` command never finishes.  What you
need is for ``tail`` to send its output to ``tr`` as it receives it.  This is where
the :ref:`_piped <piped>` special kwarg comes in handy:

.. code-block:: python

	for line in tr(_in=tail("-f", "test.log", _piped=True), "[:upper:]", "[:lower:]", _iter=True):
	    print(line)
	    
This works by telling ``tail -f`` that it is being used in a pipeline, and that
it should send its output line-by-line to ``tr``.  By default, :ref:`piped` sends
STDOUT, but you can easily make it send STDERR instead by using ``_piped="err"``

---

### Source/Sections/Redirection

.. _redirection:

Redirection
===========

sh can redirect the STDOUT and STDERR of a process to many different types of
targets, using the :ref:`_out <out>` and :ref:`_err <err>` special kwargs.

Filename
--------

If a string is used, it is assumed to be a filename.  The filename is opened as
"wb", meaning truncate-write and binary mode.

.. code-block:: python

    import sh
    sh.ifconfig(_out="/tmp/interfaces")

.. seealso:: :ref:`faq_append`

File-like Object
----------------

You may also use any object that supports ``.write(data)``, like
:class:`io.StringIO`:

.. code-block:: python

    import sh
    from io import StringIO

    buf = StringIO()
    sh.ifconfig(_out=buf)
    print(buf.getvalue())

.. _red_func:

Function Callback
-----------------

A callback function may also be used as a target.  The function must conform to
one of three signatures:

.. py:function:: fn(data)
    :noindex:

    The function takes just the chunk of data from the process.

.. py:function:: fn(data, stdin_queue)
    :noindex:

    In addition to the previous signature, the function also takes a
    :class:`queue.Queue`, which may be used to communicate programmatically with
    the process.

.. py:function:: fn(data, stdin_queue, process)
    :noindex:

    In addition to the previous signature, the function takes a
    :class:`weakref.ref` to the :ref:`OProc <oproc_class>` object.

.. seealso:: :ref:`callbacks`

.. seealso:: :ref:`tutorial2`

---

### Source/Sections/Special Arguments

.. _special_arguments:

.. |def| replace:: Default value:

Special Kwargs
##############

These arguments alter a command's behavior.  They are not passed to the program.
You can use them on any command that you run, but some may not be used together.
sh will tell you if there are conflicts.

To set default special keyword arguments on *every* command run, you may use
:ref:`default_arguments`.

Controlling Output
==================

.. _out:

_out
----
|def| ``None``

What to redirect STDOUT to.  If this is a string, it will be treated as a file
name.  You may also pass a file object (or file-like object), an int
(representing a file descriptor, like the result of :func:`os.pipe`), a
:class:`io.StringIO` object, or a callable.

.. code-block:: python

    import sh
    sh.ls(_out="/tmp/output")

.. seealso::
    :ref:`redirection`
		
.. _err:

_err
----
|def| ``None``

What to redirect STDERR to.  See :ref:`_out<out>`.
    
_err_to_out
-----------
|def| ``False``

If ``True``, duplicate the file descriptor bound to the process's STDOUT also to
STDERR, effectively causing STDERR and STDOUT to go to the same place.

_encoding
---------
|def| ``sh.DEFAULT_ENCODING``

The character encoding of the process's STDOUT.  By default, this is the
locale's default encoding.
			
_decode_errors
--------------
.. versionadded:: 1.07.0

|def| ``"strict"``

This is how Python should handle decoding errors of the process's output.
By default, this is ``"strict"``, but you can use any value that's valid
to :meth:`bytes.decode`, such as ``"ignore"``.
		
_tee
----
.. versionadded:: 1.07.0

|def| ``None``

As of 1.07.0, any time redirection is used, either for STDOUT or STDERR, the
respective internal buffers are not filled.  For example, if you're downloading
a file and using a callback on STDOUT, the internal STDOUT buffer, nor the pipe
buffer be filled with data from STDOUT.  This option forces one of stderr
(``_tee='err'``) or stdout (``_tee='out'`` or ``_tee=True``) to be filled
anyways, in effect "tee-ing" the output into two places (the callback/redirect
handler, and the internal buffers).


_truncate_exc
-------------
.. versionadded:: 1.12.0

|def| ``True``

Whether or not exception output should be truncated.

Execution
=========

.. _fg:

_fg
---
.. versionadded:: 1.12.0

|def| ``False``

Runs a command in the foreground, meaning it is spawned using :func:`os.spawnle()`.  The current process's STDIN/OUT/ERR
is :func:`os.dup2`'d to the new process and so the new process becomes the *foreground* of the shell executing the
script.  This is only really useful when you want to launch a lean, interactive process that sh is having trouble
running, for example, ssh.

.. warning::

    ``_fg=True`` side-steps a lot of sh's functionality.  You will not be returned a process object and most (likely
    all) other special kwargs will not work.

If you are looking for similar functionality, but still retaining sh's features, use the following:

.. code-block:: python
        
    import sh
    import sys
    sh.your_command(_in=sys.stdin, _out=sys.stdout, _err=sys.stderr)


.. _bg:

_bg
---
|def| ``False``

Runs a command in the background.  The command will return immediately, and you
will have to run :meth:`RunningCommand.wait` on it to ensure it terminates.

.. seealso:: :ref:`background`.

.. _bg_exc:

_bg_exc
-------
.. versionadded:: 1.12.9

|def| ``True``

Automatically report exceptions for the background command. If you set this to
``False`` you should make sure to call :meth:`RunningCommand.wait` or you may
swallow exceptions that happen in the background command.

.. _async_kw:

_async
------
.. versionadded:: 2.0.0

|def| ``False``

Allows your command to become awaitable. Use in combination with :ref:`_iter <iter>`
and ``async for`` to incrementally await output as it is produced.

.. _env:

_env
----
|def| ``None``

A dictionary defining the only environment variables that will be made
accessible to the process.  If not specified, the calling process's environment
variables are used.

.. note::

    This dictionary is the authoritative environment for the process.  If you
    wish to change a single variable in your current environment, you must pass
    a copy of your current environment with the overridden variable to sh.

.. seealso:: :ref:`environments`

.. _timeout:

_timeout
--------
|def| ``None``

How much time, in seconds, we should give the process to complete.  If the
process does not finish within the timeout, it will be sent the signal defined
by :ref:`timeout_signal`.

.. _timeout_signal:

_timeout_signal
---------------
|def| ``signal.SIGKILL``

The signal to be sent to the process if :ref:`timeout` is not ``None``.

_cwd
----
|def| ``None``

A string that sets the current working directory of the process.

.. _ok_code:

_ok_code
--------
|def| ``0``

Either an integer, a list, or a tuple containing the exit code(s) that are
considered "ok", or in other words: do not raise an exception.  Some misbehaved
programs use exit codes other than 0 to indicate success.

.. code-block:: python

    import sh
    sh.weird_program(_ok_code=[0,3,5])

If the process is killed by a signal, a :ref:`signal_exc` is raised by
default. This behavior could be blocked by appending a negative number to
:ref:`ok_code` that represents the signal.

.. code-block:: python

    import sh
    # the process won't raise SignalException if SIGINT, SIGKILL, or SIGTERM
    # are sent to kill the process
    p = sh.sleep(3, _bg=True, _ok_code=[0, -2, -9, -15])

    # No exception will be raised here
    p.kill()

.. seealso:: :ref:`exit_codes`

.. _new_session:

_new_session
------------
|def| ``False``

Determines if our forked process will be executed in its own session via
:func:`os.setsid`.

.. versionchanged:: 2.0.0
    The default value of ``_new_session`` was changed from ``True`` to ``False``
    because it makes more sense for a launched process to default to being in
    the process group of python script, so that it receives SIGINTs correctly.

.. seealso:: :ref:`architecture`

_new_group
----------
|def| ``False``

Determines if our forked process will be executed in its own group via :func:`os.setpgid`

.. _uid:

_uid
----
.. versionadded:: 1.12.0

|def| ``None``

The user id to assume before the child process calls :func:`os.execv`.

_preexec_fn
-----------
.. versionadded:: 1.12.0

|def| ``None``

A function to be run directly before the child process calls :func:`os.execv`.
Typically not used by normal users.

.. _pass_fds:

_pass_fds
---------
.. versionadded:: 1.13.0

|def| ``{}`` (empty set)

A whitelist iterable of integer file descriptors to be inherited by the child. Passing anything in this argument causes :ref:`_close_fds <close_fds>` to be ``True``.
		
.. _close_fds:

_close_fds
----------
.. versionadded:: 1.13.0

|def| ``True``

Causes all inherited file descriptors besides stdin, stdout, and stderr to be automatically closed. This option is
automatically enabled when :ref:`_pass_fds <pass_fds>` is given a value.

Communication
=============

.. _in:

_in
---

|def| ``None``

Specifies an argument for the process to use as its standard input.  This may be
a string, a :class:`queue.Queue`, a file-like object, or any iterable.

.. seealso:: :ref:`stdin`
		
.. _piped:

_piped
------

|def| ``None``

May be ``True``, ``"out"``, or ``"err"``.  Signals a command that it is being
used as the input to another command, so it should return its output
incrementally as it receives it, instead of aggregating it all at once.

.. seealso:: :ref:`Advanced Piping <advanced_piping>`

.. _iter:
		
_iter
-----

|def| ``None``

May be ``True``, ``"out"``, or ``"err"``.  Puts a command in iterable mode.  In
this mode, you can use a ``for`` or ``while`` loop to iterate over a command's
output in real-time.

.. code-block:: python

    import sh 
    for line in sh.cat("/tmp/file", _iter=True):
        print(line)

.. seealso:: :ref:`iterable`.

.. _iter_noblock:
		
_iter_noblock
-------------
|def| ``None``

Same as :ref:`_iter <iter>`, except the loop will not block if there is no
output to iterate over.  Instead, the output from the command will be
:py:data:`errno.EWOULDBLOCK`.

.. code-block:: python

    import sh
    import errno
    import time

    for line in sh.tail("-f", "stuff.log", _iter_noblock=True):
        if line == errno.EWOULDBLOCK:
            print("doing something else...")
            time.sleep(0.5)
        else:
            print("processing line!")


.. seealso:: :ref:`iterable`.

.. _with:

_with
-----
|def| ``False``

Explicitly tells us that we're running a command in a ``with`` context.  This is
only necessary if you're using a command in a ``with`` context **and** passing
parameters to it.

.. code-block:: python

    import sh
    with sh.contrib.sudo(password="abc123", _with=True):
        print(sh.ls("/root"))

.. seealso:: :ref:`with_contexts`

.. _done:

_done
-----
.. versionadded:: 1.11.0

|def| ``None``

A callback that is *always* called when the command completes, even if it
completes with an exit code that would raise an exception.  After the callback
is run, any exception that would be raised is raised.

The callback is passed the :ref:`RunningCommand <running_command>` instance, a
boolean indicating success, and the exit code.

.. include:: /examples/done.rst
		
TTYs
====

.. _tty_in:

_tty_in
-------

|def| ``False``, meaning a :func:`os.pipe` will be used.

If ``True``, sh creates a TTY for STDIN, essentially emulating a terminal, as if
your command was entered from the commandline.  This is necessary for commands
that require STDIN to be a TTY.

.. _tty_out:
    
_tty_out
--------

|def| ``True``

If ``True``, sh creates a TTY for STDOUT, otherwise use a :func:`os.pipe`. This
is necessary for commands that require STDOUT to be a TTY.

.. seealso:: :ref:`faq_tty_out`

.. _unify_ttys:

_unify_ttys
-----------
.. versionadded:: 1.13.0

|def| ``False``

If ``True``, sh will combine the STDOUT and STDIN TTY into a single
pseudo-terminal. This is sometimes required by picky programs which expect to be
dealing with a single pseudo-terminal, like SSH.

.. seealso:: :ref:`tutorial2`

_tty_size
---------

|def| ``(20, 80)``

The (rows, columns) of stdout's TTY.  Changing this may affect how much your
program prints per line, for example.

Performance & Optimization
==========================

_in_bufsize
-----------
|def| ``0``

The STDIN buffer size.  0 for unbuffered, 1 for line buffered, anything else for
a buffer of that amount.

.. _out_bufsize:
		
_out_bufsize
------------
|def| ``1``

The STDOUT buffer size.  0 for unbuffered, 1 for line buffered, anything
else for a buffer of that amount.

.. _err_bufsize:

_err_bufsize
------------
|def| ``1``

Same as :ref:`out_bufsize`, but with STDERR.

.. _internal_bufsize:
		
_internal_bufsize
-----------------
|def| ``3 * 1024**2`` chunks

How much of STDOUT/ERR your command will store internally.  This value
represents the *number of bufsize chunks* not the total number of bytes.  For
example, if this value is 100, and STDOUT is line buffered, you will be able to
retrieve 100 lines from STDOUT.  If STDOUT is unbuffered, you will be able to
retrieve only 100 characters.
		
_no_out
-------
.. versionadded:: 1.07.0

|def| ``False``

Disables STDOUT being internally stored.  This is useful for commands
that produce huge amounts of output that you don't need, that would
otherwise be hogging memory if stored internally by sh.
		
_no_err
-------
.. versionadded:: 1.07.0

|def| ``False``

Disables STDERR being internally stored.  This is useful for commands that
produce huge amounts of output that you don't need, that would otherwise be
hogging memory if stored internally by sh.
		
_no_pipe
--------
.. versionadded:: 1.07.0

|def| ``False``

Similar to ``_no_out``, this explicitly tells the sh command that it will never
be used for piping its output into another command, so it should not fill its
internal pipe buffer with the process's output.  This is also useful for
conserving memory.
		

Program Arguments
=================

These are options that affect how command options are fed into the program.

_long_sep
---------
.. versionadded:: 1.12.0

|def| ``"="``

This is the character(s) that separate a program's long argument's key from the
value, when using kwargs to specify your program's long arguments.  For example,
if your program expects a long argument in the form ``--name value``, the way to
achieve this would be to set ``_long_sep=" "``.

.. code-block:: python

    import sh
    sh.your_program(key=value, _long_sep=" ")

Would send the following list of arguments to your program:

.. code-block:: python

    ["--key value"]

If your program expects the long argument name to be separate from its value,
pass ``None`` into ``_long_sep`` instead:

.. code-block:: python

    import sh
    sh.your_program(key=value, _long_sep=None)

Would send the following list of arguments to your program:

.. code-block:: python

    ["--key", "value"]

_long_prefix
------------
.. versionadded:: 1.12.0

|def| ``"--"``

This is the character(s) that prefix a long argument for the program being run.
Some programs use single dashes, for example, and do not understand double
dashes.

.. _preprocess:

_arg_preprocess
---------------
.. versionadded:: 1.12.0

|def| ``None``

This is an advanced option that allows you to rewrite a command's arguments on
the fly, based on other command arguments, or some other variable.  It is really
only useful in conjunction with :ref:`baking <baking>`, and only currently used when
constructing :ref:`contrib <contrib>` wrappers.

Example:

.. code-block:: python

    import sh

    def processor(args, kwargs):
        return args, kwargs

    my_ls = sh.bake.ls(_arg_preprocess=processor)

.. warning::

    The interface to the ``_arg_preprocess`` function may change without
    warning.  It is generally only for internal sh use, so don't use it unless
    you absolutely have to.

Misc
====

_log_msg
--------

|def| ``None``

.. versionadded:: 1.12.0

This allows for a custom logging header for :ref:`command_class` instances.  For example, the default logging looks like this:

.. code-block:: python

    import logging
    import sh

    logging.basicConfig(level=logging.INFO)

    sh.ls("-l")

.. code-block:: none

    INFO:sh.command:<Command '/bin/ls -l'>: starting process
    INFO:sh.command:<Command '/bin/ls -l', pid 28952>: process started
    INFO:sh.command:<Command '/bin/ls -l', pid 28952>: process completed

People can find this ``<Command ..`` section long and not relevant. ``_log_msg`` allows you to customize this:

.. code-block:: python

    import logging
    import sh

    logging.basicConfig(level=logging.INFO)

    def custom_log(ran, call_args, pid=None):
        return ran

    sh.ls("-l", _log_msg=custom_log)

.. code-block:: none

    INFO:sh.command:/bin/ls -l: starting process
    INFO:sh.command:/bin/ls -l: process started
    INFO:sh.command:/bin/ls -l: process completed

The first argument, ``ran``, is the program's execution string and arguments, as close as we can get it to be how you'd
type in the shell.  ``call_args`` is a dictionary of all of the special kwargs that were passed to the command.  And ``pid``
is the process id of the forked process.  It defaults to ``None`` because the ``_log_msg`` callback is actually called
twice: first to construct the logger for the :ref:`running_command` instance, before the process itself is spawned, then
a second time after the process is spawned via :ref:`oproc_class`, when we have a pid.

---

### Source/Sections/Stdin

.. _stdin:

Input via STDIN
===============

STDIN is sent to a process directly by using a command's :ref:`in` special
kwarg:

.. code-block:: python

	print(cat(_in="test"))
	
Any command that takes input from STDIN can be used this way:

.. code-block:: python

	print(tr("[:lower:]", "[:upper:]", _in="sh is awesome"))
	
You're also not limited to using just strings.  You may use a file object, a
:class:`queue.Queue`, or any iterable (list, set, dictionary, etc):

.. code-block:: python

	stdin = ["sh", "is", "awesome"]
	out = tr("[:lower:]", "[:upper:]", _in=stdin)

.. note::

    If you use a queue, you can signal the end of the queue (EOF) with ``None``

---

### Source/Sections/Subcommands

.. _subcommands:

Sub-commands
============

Many programs have their own command subsets, like git (branch, checkout),
svn (update, status), and sudo (where any command following sudo is considered
a sub-command).  sh handles subcommands through attribute access:

.. code-block:: python

	from sh import git, sudo
	
	# resolves to "git branch -v"
	print(git.branch("-v"))
	print(git("branch", "-v")) # the same command
	
	# resolves to "sudo /bin/ls /root"
	print(sudo.ls("/root"))
	print(sudo("/bin/ls", "/root")) # the same command
	
Sub-commands are mainly syntax sugar that makes calling some programs look conceptually nicer.

.. seealso::
    If you're using ``sudo`` as a subcommand, please be sure to see :ref:`sudo`.

---

### Source/Sections/Sudo

.. _sudo:

Using Sudo
==========

There are 3 ways of using ``sudo`` to execute commands in your script.  These
are listed in order of usefulness and security.  In most cases, you should just
use a variation of :ref:`contrib_sudo`.

.. _contrib_sudo:

sh.contrib.sudo
---------------

Because ``sudo`` is so frequently used, we have added a contrib version of the
command to make sudo usage more intuitive.  This contrib version is simply a
wrapper around the :ref:`sudo_raw` raw command, but we bake in some
:ref:`special keyword argument <special_arguments>` to make it well-behaved.  In
particular, the contrib version allows you to specify your password at execution
time via terminal input, or as a string in your script.

Terminal Input
^^^^^^^^^^^^^^

Via a :ref:`with context <with_contexts>`:

.. code-block:: python

    import sh

    with sh.contrib.sudo:
        print(ls("/root"))

Or alternatively via :ref:`subcommands <subcommands>`:

.. code-block:: python

    import sh
    print(sh.contrib.sudo.ls("/root"))

Output:

.. code-block:: none

    [sudo] password for youruser: *************
    your_root_files.txt

In the above example, ``sh.contrib.sudo`` automatically asks you for a password
using :func:`getpass.getpass` under the hood.

This method is the most secure, because it lowers the chances of doing something
insecure, like including your password in your python script, or by saying that
a particular user can execute anything inside of a particular script (the
NOPASSWD method).

.. note::

    ``sh.contrib.sudo`` does not do password caching like the sudo binary does.
    Thie means that each time a sudo command is run in your script, you will be
    asked to type in a password.

String Input
^^^^^^^^^^^^

You may also specify your password to ``sh.contrib.sudo`` as a string:

.. code-block:: python

    import sh

    password = get_your_password()

    with sh.contrib.sudo(password=password, _with=True):
        print(ls("/root"))

.. warning::

    This method is less secure because it becomes tempting to hard-code your
    password into the python script, and that's a bad idea.  However, it is more
    flexible, because it allows you to obtain your password from another source,
    so long as the end result is a string.

/etc/sudoers NOPASSWD
---------------------

With this method, you can use the raw ``sh.sudo`` command directly, because
you're being guaranteed that the system will not ask you for a password.  It
first requires you set up your user to have root execution privileges

Edit your sudoers file:

.. code-block:: none

    $> sudo visudo

Add or edit the line describing your user's permissions:

.. code-block:: none

    yourusername ALL = (root) NOPASSWD: /path/to/your/program

This says ``yourusername`` on ``ALL`` hosts will be able to run as root, but
only root ``(root)`` (no other users), and that no password ``NOPASSWD`` will be
asked of ``/path/to/your/program``.

.. warning::
    
    This method can be insecure if an unprivileged user can edit your script,
    because the entire script will be exited as a privileged user.  A malicious
    user could put something bad in this script.

.. _sudo_raw:

sh.sudo
-------

Using the raw command ``sh.sudo`` (which resolves directly to the system's
``sudo`` binary) without NOPASSWD is possible, provided you wire up the special
keyword arguments on your own to make it behave correctly.  This method is
discussed generally for educational purposes; if you take the time to wire up
``sh.sudo`` on your own, then you have in essence just recreated
:ref:`contrib_sudo`.

.. code-block:: python

    import sh

    # password must end in a newline
    my_password = "password\n"

    # -S says "get the password from stdin"
    my_sudo = sh.sudo.bake("-S", _in=my_password)

    print(my_sudo.ls("root"))

_fg=True
--------

Another less-obvious way of using sudo is by executing the raw ``sh.sudo``
command but also putting it in the foreground.  This way, sudo will work
correctly automatically, by hooking up stdin/out/err automatically, and by
asking you for a password if it requires one.  The downsides of using
:ref:`_fg=True <fg>`, however, are that you cannot capture its output -- everything is
just printed to your terminal as if you ran it from a shell.

.. code-block:: python

    import sh
    sh.sudo.ls("/root", _fg=True)

---

### Source/Sections/With

.. _with_contexts:

'With' Contexts
===============

Commands can be run within a Python ``with`` context.  Popular commands using
this might be ``sudo`` or ``fakeroot``:

.. code-block:: python

	with sh.contrib.sudo:
	    print(ls("/root"))

.. seealso::

    :ref:`contrib_sudo`
		
If you need to run a command in a with context and pass in arguments, for
example, specifying a -p prompt with sudo, you need to use the :ref:`_with=True
<with>` This let's the command know that it's being run from a with context so
it can behave correctly:

.. code-block:: python

	with sh.contrib.sudo(k=True, _with=True):
	    print(ls("/root"))

---

### Source/Tutorials/Interacting With Processes

.. _tutorial2:

Entering an SSH password
========================

Here we will attempt to SSH into a server and enter a password programmatically.

.. note::

    It is recommended that you just ``ssh-copy-id`` to copy your public key to
    the server so you don't need to enter your password, but for the purposes of
    this demonstration, we try to enter a password.

To interact with a process, we need to assign a callback to STDOUT.  The
callback signature we'll use will take a :class:`queue.Queue` object for the
second argument, and we'll use that to send STDIN back to the process.

.. seealso:: :ref:`red_func`

Here's our first attempt:

.. code-block:: python

    from sh import ssh

    def ssh_interact(line, stdin):
        line = line.strip()
        print(line)
        if line.endswith("password:"):
            stdin.put("correcthorsebatterystaple")

    ssh("10.10.10.100", _out=ssh_interact)

If you run this (substituting an IP that you can SSH to), you'll notice that
nothing is printed from within the callback.  The problem has to do with STDOUT
buffering.  By default, sh line-buffers STDOUT, which means that
``ssh_interact`` will only receive output when sh encounters a newline in the
output.  This is a problem because the password prompt has no newline:

.. code-block:: none

    amoffat@10.10.10.100's password:

Because a newline is never encountered, nothing is sent to the ``ssh_interact``
callback.  So we need to change the STDOUT buffering.  We do this with the
:ref:`_out_bufsize <out_bufsize>` special kwarg.  We'll set
it to 0 for unbuffered output:

.. code-block:: python

    from sh import ssh
    
    def ssh_interact(line, stdin):
        line = line.strip()
        print(line)
        if line.endswith("password:"):
            stdin.put("correcthorsebatterystaple")

    ssh("10.10.10.100", _out=ssh_interact, _out_bufsize=0)

If you run this updated version, you'll notice a new problem.  The output looks
like this:

.. code-block:: none

    a
    m
    o
    f
    f
    a
    t
    @
    1
    0
    .
    1
    0
    .
    1
    0
    .
    1
    0
    0
    '
    s
    
    p
    a
    s
    s
    w
    o
    r
    d
    :

This is because the chunks of STDOUT our callback is receiving are unbuffered,
and are therefore individual characters, instead of entire lines.  What we need
to do now is aggregate this character-by-character data into something more
meaningful for us to test if the pattern ``password:`` has been sent, signifying
that SSH is ready for input.

It would make sense to encapsulate the variable we'll use for aggregating into
some kind of closure or class, but to keep it simple, we'll just use a global:

.. code-block:: python

    from sh import ssh
    import sys

    aggregated = ""
    def ssh_interact(char, stdin):
        global aggregated
        sys.stdout.write(char.encode())
        sys.stdout.flush()
        aggregated += char
        if aggregated.endswith("password: "):
            stdin.put("correcthorsebatterystaple")

    ssh("10.10.10.100", _out=ssh_interact, _out_bufsize=0)

You'll also notice that the example still doesn't work.  There are two problems:
The first is that your password must end with a newline, as if you had typed it
and hit the return key.  This is because SSH has no idea how long your password
is, and is line-buffering STDIN.

The second problem lies deeper in SSH.  SSH needs a TTY attached to its STDIN in
order to work properly.  This tricks SSH into believing that it is interacting
with a real user in a real terminal session.  To enable TTY, we can add the
:ref:`_tty_in <tty_in>` special kwarg.  We also need to use :ref:`_unify_ttys <unify_ttys>` special kwarg.
This tells sh to make STDOUT and STDIN come from a single pseudo-terminal, which is a requirement of SSH:

.. code-block:: python

    from sh import ssh
    import sys

    aggregated = ""
    def ssh_interact(char, stdin):
        global aggregated
        sys.stdout.write(char.encode())
        sys.stdout.flush()
        aggregated += char
        if aggregated.endswith("password: "):
            stdin.put("correcthorsebatterystaple\n")

    ssh("10.10.10.100", _out=ssh_interact, _out_bufsize=0, _tty_in=True, _unify_ttys=True)
    
And now our remote login script works!

.. code-block:: none

    amoffat@10.10.10.100's password: 
    Linux 10.10.10.100 testhost #1 SMP Tue Jun 21 10:29:24 EDT 2011 i686 GNU/Linux
    Ubuntu 10.04.2 LTS
    
    Welcome to Ubuntu!
     * Documentation:  https://help.ubuntu.com/
    
    66 packages can be updated.
    53 updates are security updates.
    
    Ubuntu 10.04.2 LTS
    
    Welcome to Ubuntu!
     * Documentation:  https://help.ubuntu.com/
    You have new mail.
    Last login: Thu Sep 13 03:53:00 2012 from some.ip.address
    amoffat@10.10.10.100:~$ 

SSH Contrib command
-------------------

The above process can be simplified by using a :ref:`contrib`. The :ref:`SSH contrib command <contrib_ssh>` does
all the ugly kwarg argument setup for you, and provides a simple but powerful interface for doing SSH password logins.
Please see the :ref:`SSH contrib command <contrib_ssh>` for more details about the exact api:

.. code-block:: python

    from sh.contrib import ssh

    def ssh_interact(content, stdin):
        sys.stdout.write(content.cur_char)
        sys.stdout.flush()

    # automatically logs in with password and then presents subsequent content to
    # the ssh_interact callback
    ssh("10.10.10.100", password="correcthorsebatterystaple", interact=ssh_interact)

How you should REALLY be using SSH
----------------------------------

Many people want to learn how to enter an SSH password by script because they
want to execute remote commands on a server.  Instead of trying to log in
through SSH and then sending terminal input of the command to run, let's see how
we can do it another way.

First, open a terminal and run ``ssh-copy-id yourservername``.  You'll be asked
to enter your password for the server.  After entering your password, you'll be
able to SSH into the server without needing a password again.  This simplifies
things greatly for sh.

The second thing we want to do is use SSH's ability to pass a command to run
to the server you're SSHing to.  Here's how you can run ``ifconfig`` on a server
without having to use that server's shell directly:

.. code-block:: none

    ssh amoffat@10.10.10.100 ifconfig 

Translating this to sh, it becomes:

.. code-block:: python

    import sh

    print(sh.ssh("amoffat@10.10.10.100", "ifconfig"))

We can make this even nicer by taking advantage of sh's :ref:`baking` to bind
our server username/ip to a command object:

.. code-block:: python

    import sh

    my_server = sh.ssh.bake("amoffat@10.10.10.100")
    print(my_server("ifconfig"))
    print(my_server("whoami"))

Now we have a reusable command object that we can use to call remote commands.
But there is room for one more improvement.  We can also use sh's
:ref:`subcommands` feature which expands attribute access into command
arguments:

.. code-block:: python

    import sh

    my_server = sh.ssh.bake("amoffat@10.10.10.100")
    print(my_server.ifconfig())
    print(my_server.whoami())

---

### Source/Tutorials/Real Time Output

.. _tutorial1:

Tailing a real-time log file
============================

sh has the ability to respond to subprocesses in an event-driven fashion.
A typical example of where this would be useful is tailing a log file for
a specific pattern, then responding to that value immediately::

	from sh import tail
	
	for line in tail("-f", "info.log", _iter=True):
	    if "ERROR" in line:
	        send_an_email_to_support(line)
			
			
The :ref:`_iter <iter>` special kwarg takes a command that would normally block
until completion, and turns its output into a real-time iterable.

.. seealso:: :ref:`iterable`

Of course, you can do more than just tail log files.  Any program that
produces output can be iterated over.  Say you wanted to send an email to a
coworker if their C code emits a warning:

.. code-block:: python

	from sh import gcc, git
	
	for line in gcc("-o", "awesome_binary", "awesome_source.c", _iter=True):
	    if "warning" in line:
	        # parse out the relevant info
	        filename, line, char, message = line.split(":", 3)
	        
	        # find the commit using git
	        commit = git("blame", "-e", filename, L="%d,%d" % (line,line))
	        
	        # send them an email
	        email_address = parse_email_from_commit_line(commit)
	        send_email(email_address, message)

Using :ref:`_iter <iter>` is a great way to respond to events from another
program, but your blocks while you're looping, making you unable to do anything
else.  To be truly event-driven, sh provides callbacks:

.. code-block:: python

	from sh import tail
	
	def process_log_line(line):
	    if "ERROR" in line:
	        send_an_email_to_support(line)
	
	process = tail("-f", "info.log", _out=process_log_line, _bg=True)
	
	# ... do other stuff here ...
	
	process.wait()
	
The :ref:`_out <out>` special kwarg lets you to assign a callback to STDOUT.
This callback will receive each line of output from ``tail -f`` and allow you to
do the same processing that we did earlier.

.. seealso:: :ref:`callbacks`

.. seealso:: :ref:`redirection`

---

### Source/Fulldoc

Full Documentation
==================

This single page repeats the full documentation for `sh <https://github.com/amoffat/sh/>`, making it easier to put into an LLM's context window. There is nothing on this page that is not mentioned already elsewhere on this site, it's just reorganized as a single page.

.. include:: index.rst

.. include:: tutorials/interacting_with_processes.rst
.. include:: tutorials/real_time_output.rst


.. content linked in index.rst
.. include:: sections/faq.rst
.. include:: sections/contrib.rst
.. include:: sections/sudo.rst
.. include:: sections/migration.rst

.. content of usage.rst
.. include:: sections/passing_arguments.rst
.. include:: sections/exit_codes.rst
.. include:: sections/redirection.rst
.. include:: sections/asynchronous_execution.rst
.. also contains reference to example/done.rst so no need to mention it explicitly
.. .. include:: examples/done.rst
.. include:: sections/baking.rst
.. include:: sections/piping.rst
.. include:: sections/subcommands.rst
.. include:: sections/default_arguments.rst
.. include:: sections/envs.rst
.. include:: sections/stdin.rst
.. include:: sections/with.rst

.. content of reference.rst
.. include:: sections/special_arguments.rst
.. include:: sections/architecture.rst
.. include:: sections/command_class.rst

---

### Source/Index

.. toctree::
    :hidden:

    usage
    reference

    sections/contrib
    sections/sudo
    sections/migration

    tutorials
    sections/faq

    ref_to_fulldoc

.. image:: images/logo-230.png
    :alt: Logo

sh
##

.. image:: https://img.shields.io/pypi/v/sh.svg?style=flat-square
    :target: https://pypi.python.org/pypi/sh
    :alt: Version
.. image:: https://img.shields.io/pypi/dm/sh.svg?style=flat-square
    :target: https://pypi.python.org/pypi/sh
    :alt: Downloads Status
.. image:: https://img.shields.io/pypi/pyversions/sh.svg?style=flat-square
    :target: https://pypi.python.org/pypi/sh
    :alt: Python Versions
.. image:: https://img.shields.io/coveralls/amoffat/sh.svg?style=flat-square
    :target: https://coveralls.io/r/amoffat/sh?branch=master
    :alt: Coverage Status
.. image:: https://img.shields.io/github/stars/amoffat/sh.svg?style=social&label=Star
    :target: https://github.com/amoffat/sh
    :alt: Github

sh is a full-fledged subprocess replacement for Python 3.10+ and PyPy that
allows you to call any program as if it were a function:


.. code-block:: python

    from sh import git
    print(git("status", "--short"))
	
Note that these aren't Python functions, these are running the binary commands
on your system by dynamically resolving your ``$PATH``, much like Bash does, and
then wrapping the binary in a function.  In this way, all the programs on your
system are easily available to you from within Python.

sh relies on various Unix system calls and only works on Unix-like operating
systems - Linux, macOS, BSDs etc. Specifically, Windows is not supported.


Installation
============

.. code-block:: none

    pip install sh


Quick Reference
===============

Passing Arguments
-----------------

.. code-block:: python
    
    sh.ls("-l", "/tmp", color="never")

:ref:`Read More <passing_arguments>`

Exit Codes
----------

.. code-block:: python

    try:
        sh.ls("/doesnt/exist")
    except sh.ErrorReturnCode_2:
        print("directory doesn't exist")

:ref:`Read More <exit_codes>`

Redirection
-----------

.. code-block:: python

    sh.ls(_out="/tmp/dir_contents")

    with open("/tmp/dir_contents", "w") as h:
        sh.ls(_out=h)

    from io import StringIO
    buf = StringIO()
    sh.ls(_out=buf)

:ref:`Read More <redirection>`

Baking
------

.. code-block:: python

    my_ls = sh.ls.bake("-l")

    # equivalent
    my_ls("/tmp")
    sh.ls("-l", "/tmp")

:ref:`Read More <baking>`

Piping
------

.. code-block:: python

    sh.wc("-l", _in=sh.ls("-1"))

:ref:`Read More <piping>`

Subcommands
-----------

.. code-block:: python

    # equivalent
    sh.git("show", "HEAD")
    sh.git.show("HEAD")

:ref:`Read More <subcommands>`


Background Processes
--------------------

.. code-block:: python

    p = sh.find("-name", "sh.py", _bg=True)
    # ... do other things ...
    p.wait()

:ref:`Read More <background>`

.. include:: ref_to_fulldoc.rst

---

### Source/Ref To Fulldoc

Single Page
===========

The page below repeats the full documentation for `sh <https://github.com/amoffat/sh/>` as a single page, making it easier to put into an LLM's context window.

:doc:`./fulldoc`

---

### Source/Reference

Reference
=========

.. toctree::
    sections/special_arguments
    sections/architecture
    sections/command_class

---

### Source/Tutorials

Tutorials
=========

.. toctree::
   tutorials/real_time_output
   tutorials/interacting_with_processes

---

### Source/Usage

Usage
=====

.. toctree::
    
    sections/passing_arguments
    sections/exit_codes
    sections/redirection
    sections/asynchronous_execution
    sections/baking
    sections/piping
    sections/subcommands
    sections/default_arguments
    sections/envs
    sections/stdin
    sections/with

---

### CHANGELOG

# Changelog

## 2.4.0 - 7/25/26

- Dropped support for Python < 3.10
- Significantly improved `.pyi` typing stub with modern type aliases and annotations
- `Command` now supports generic subscript syntax (`Command[str]`) via `GenericAlias`
- Bugfix where signal names containing numbers were not matched correctly
- `stubtest` and `pyrefly coverage check` now run in CI

## 2.3.0 - 6/8/25

- Fix sdist builds from littering files
- Add all special keyword args to .pyi stub file

## 2.2.6 - 6/7/25

- Include missing sdist files [#778](https://github.com/amoffat/sh/issues/778)

## 2.2.5 - 6/6/25

- Single page doc for LLM context window [#757](https://github.com/amoffat/sh/pull/757)
- Formal support for Python 3.13 and 3.14
- Allow boolean arguments to override baked arguments [#770](https://github.com/amoffat/sh/pull/770)
- Added .pyi typing stub
- Improvements to str representation of `Command` [#756](https://github.com/amoffat/sh/pull/756)
- Bugfix where rare `ProcessLookupError` can be thrown [#769](https://github.com/amoffat/sh/pull/769)
- Various spelling corrections [#774](https://github.com/amoffat/sh/pull/774)

## 2.2.4 - 6/06/25

- Bugfix regression where correct `gid` was not set on the launched process

## 2.2.3 - 6/05/25

- Bugfix where supplemental groups were not dropped when using `_uid`

## 2.2.2 - 2/23/25

- Bugfix where it was impossible to use a signal as an `ok_code` [#699](https://github.com/amoffat/sh/issues/699)

## 2.2.1 - 1/9/25

- Bugfix where `async` and `return_cmd` does not raise exceptions [#746](https://github.com/amoffat/sh/pull/746)

## 2.2.0 - 1/9/25

- `return_cmd` with `await` now works correctly [#743](https://github.com/amoffat/sh/issues/743)
- Formal support for Python 3.12

## 2.1.0 - 10/8/24

- Add contrib command `sh.contrib.bash` [#736](https://github.com/amoffat/sh/pull/736)

## 2.0.7 - 5/31/24

- Fix `sh.glob` arguments [#708](https://github.com/amoffat/sh/issues/708)
- Misc modernizations

## 2.0.6 - 8/9/23

- Add back appropriate sdist files [comment](https://github.com/amoffat/sh/commit/89333ae48069a5b445b3535232195b2de6f4648f)

## 2.0.5 - 8/7/23

- Allow nested `with` contexts [#690](https://github.com/amoffat/sh/issues/690)
- Call correct asyncio function for getting event loop [#683](https://github.com/amoffat/sh/issues/683)

## 2.0.4 - 5/13/22

- Allow `ok_code` to be used with `fg` [#665](https://github.com/amoffat/sh/pull/665)
- Make sure `new_group` never creates a new session [#675](https://github.com/amoffat/sh/pull/675)

## 2.0.2 / 2.0.3 (misversioned) - 2/13/22

- Performance regression when using a generator with `_in` [#650](https://github.com/amoffat/sh/pull/650)
- Adding test support for python 3.11

## 2.0.0 - 2/9/22

- Executed commands now return a unicode string by default
- Removed magical module-like execution contexts [#636](https://github.com/amoffat/sh/issues/636)
- Added basic asyncio support via `_async`
- Dropped support for Python < 3.8
- Bumped default tty size to more standard (24, 80)
- First argument being a RunningCommand no longer automatically passes it as stdin
- `RunningCommand.__eq__` no longer has the side effect of executing the command [#518](https://github.com/amoffat/sh/pull/531)
- `_tee` now supports both "err" and "out" [#215](https://github.com/amoffat/sh/issues/215)
- Removed the builtin override `cd` [link](https://github.com/amoffat/sh/pull/584#discussion_r698055681)
- Altered process launching model to behave more expectedly [#495](https://github.com/amoffat/sh/issues/495)
- Bugfix where `_no_out` isn't allowed with `_iter="err"` [#638](https://github.com/amoffat/sh/issues/638)
- Allow keyword arguments to have a list of values [#529](https://github.com/amoffat/sh/issues/529)

## 1.14.3 - 7/17/22

- Bugfix where `Command` was not aware of default call args when wrapping the module [#559](https://github.com/amoffat/sh/pull/573)

## 1.14.1 - 10/24/20

- bugfix where setting `_ok_code` to not include 0, but 0 was the exit code [#545](https://github.com/amoffat/sh/pull/545)

## 1.14.0 - 8/28/20

- `_env` now more lenient in accepting dictionary-like objects [#527](https://github.com/amoffat/sh/issues/527)
- `None` and `False` arguments now do not pass through to underlying command [#525](https://github.com/amoffat/sh/pull/525)
- Implemented `find_spec` on the fancy importer, which fixes some Python3.4+ issues [#536](https://github.com/amoffat/sh/pull/536)

## 1.13.1 - 4/28/20

- regression fix if `_fg=False` [#520](https://github.com/amoffat/sh/issues/520)

## 1.13.0 - 4/27/20

- minor Travis CI fixes [#492](https://github.com/amoffat/sh/pull/492)
- bugfix for boolean long options not respecting `_long_prefix` [#488](https://github.com/amoffat/sh/pull/488)
- fix deprecation warning on Python 3.6 regexes [#482](https://github.com/amoffat/sh/pull/482)
- `_pass_fds` and `_close_fds` special kwargs for controlling file descriptor inheritance in child.
- more efficiently closing inherited fds [#406](https://github.com/amoffat/sh/issues/406)
- bugfix where passing invalid dictionary to `_env` will cause a mysterious child 255 exit code. [#497](https://github.com/amoffat/sh/pull/497)
- bugfix where `_in` using 0 or `sys.stdin` wasn't behaving like a TTY, if it was in fact a TTY. [#514](https://github.com/amoffat/sh/issues/514)
- bugfix where `help(sh)` raised an exception [#455](https://github.com/amoffat/sh/issues/455)
- bugfix fixing broken interactive ssh tutorial from docs
- change to automatic tty merging into a single pty if `_tty_in=True` and `_tty_out=True`
- introducing `_unify_ttys`, default False, which allows explicit tty merging into single pty
- contrib command for `ssh` connections requiring passwords
- performance fix for polling output too fast when using `_iter` [#462](https://github.com/amoffat/sh/issues/462)
- execution contexts can now be used in python shell [#466](https://github.com/amoffat/sh/pull/466)
- bugfix `ErrorReturnCode` instances can now be pickled
- bugfix passing empty string or `None` for `_in` hanged [#427](https://github.com/amoffat/sh/pull/427)
- bugfix where passing a filename or file-like object to `_out` wasn't using os.dup2 [#449](https://github.com/amoffat/sh/issues/449)
- regression make `_fg` work with `_cwd` again [#330](https://github.com/amoffat/sh/issues/330)
- an invalid `_cwd` now raises a `ForkException` not an `OSError`.
- AIX support [#477](https://github.com/amoffat/sh/issues/477)
- added a `timeout=None` param to `RunningCommand.wait()` [#515](https://github.com/amoffat/sh/issues/515)

## 1.12.14 - 6/6/17

- bugfix for poor sleep performance [#378](https://github.com/amoffat/sh/issues/378)
- allow passing raw integer file descriptors for `_out` and `_err` handlers
- bugfix for when `_tee` and `_out` are used, and the `_out` is a tty or pipe [#384](https://github.com/amoffat/sh/issues/384)
- bugfix where python 3.3+ detected different arg counts for bound method output callbacks [#380](https://github.com/amoffat/sh/issues/380)

## 1.12.12, 1.12.13 - 3/30/17

- pypi readme doc bugfix [PR#377](https://github.com/amoffat/sh/pull/377)

## 1.12.11 - 3/13/17

- bugfix for relative paths to `sh.Command` not expanding to absolute paths [#372](https://github.com/amoffat/sh/issues/372)
- updated for python 3.6
- bugfix for SIGPIPE not being handled correctly on pipelined processes [#373](https://github.com/amoffat/sh/issues/373)

## 1.12.10 - 3/02/17

- bugfix for file descriptors over 1024 [#356](https://github.com/amoffat/sh/issues/356)
- bugfix when `_err_to_out` is True and `_out` is pipe or tty [#365](https://github.com/amoffat/sh/issues/365)

## 1.12.9 - 1/04/17

- added `_bg_exc` for silencing exceptions in background threads [#350](https://github.com/amoffat/sh/pull/350)

## 1.12.8 - 12/16/16

- bugfix for patched glob.glob on python3.5 [#341](https://github.com/amoffat/sh/issues/341)

## 1.12.7 - 12/07/16

- added `_out` and `_out_bufsize` validator [#346](https://github.com/amoffat/sh/issues/346)
- bugfix for internal stdout thread running when it shouldn't [#346](https://github.com/amoffat/sh/issues/346)

## 1.12.6 - 12/02/16

- regression bugfix on timeout [#344](https://github.com/amoffat/sh/issues/344)
- regression bugfix on `_ok_code=None`

## 1.12.5 - 12/01/16

- further improvements on cpu usage

## 1.12.4 - 11/30/16

- regression in cpu usage [#339](https://github.com/amoffat/sh/issues/339)

## 1.12.3 - 11/29/16

- fd leak regression and fix for flawed fd leak detection test [#337](https://github.com/amoffat/sh/pull/337)

## 1.12.2 - 11/28/16

- support for `io.StringIO` in python2

## 1.12.1 - 11/28/16

- added support for using raw file descriptors for `_in`, `_out`, and `_err`
- removed `.close()`ing `_out` handler if FIFO detected

## 1.12.0 - 11/21/16

- composed commands no longer propagate `_bg`
- better support for using `sys.stdin` and `sys.stdout` for `_in` and `_out`
- bugfix where `which()` would not stop searching at the first valid executable found in PATH
- added `_long_prefix` for programs whose long arguments start with something other than `--` [#278](https://github.com/amoffat/sh/pull/278)
- added `_log_msg` for advanced configuration of log message [#311](https://github.com/amoffat/sh/pull/311)
- added `sh.contrib.sudo`
- added `_arg_preprocess` for advanced command wrapping
- alter callable `_in` arguments to signify completion with falsy chunk
- bugfix where pipes passed into `_out` or `_err` were not flushed on process end [#252](https://github.com/amoffat/sh/pull/252)
- deprecated `with sh.args(**kwargs)` in favor of `sh2 = sh(**kwargs)`
- made `sh.pushd` thread safe
- added `.kill_group()` and `.signal_group()` methods for better process control [#237](https://github.com/amoffat/sh/pull/237)
- added `new_session` special keyword argument for controlling spawned process session [#266](https://github.com/amoffat/sh/issues/266)
- bugfix better handling for EINTR on system calls [#292](https://github.com/amoffat/sh/pull/292)
- bugfix where with-contexts were not threadsafe [#247](https://github.com/amoffat/sh/issues/195)
- `_uid` new special keyword param for specifying the user id of the process [#133](https://github.com/amoffat/sh/issues/133)
- bugfix where exceptions were swallowed by processes that weren't waited on [#309](https://github.com/amoffat/sh/issues/309)
- bugfix where processes that dupd their stdout/stderr to a long running child process would cause sh to hang [#310](https://github.com/amoffat/sh/issues/310)
- improved logging output [#323](https://github.com/amoffat/sh/issues/323)
- bugfix for python3+ where binary data was passed into a process's stdin [#325](https://github.com/amoffat/sh/issues/325)
- Introduced execution contexts which allow baking of common special keyword arguments into all commands [#269](https://github.com/amoffat/sh/issues/269)
- `Command` and `which` now can take an optional `paths` parameter which specifies the search paths [#226](https://github.com/amoffat/sh/issues/226)
- `_preexec_fn` option for executing a function after the child process forks but before it execs [#260](https://github.com/amoffat/sh/issues/260)
- `_fg` reintroduced, with limited functionality. hurrah! [#92](https://github.com/amoffat/sh/issues/92)
- bugfix where a command would block if passed a fd for stdin that wasn't yet ready to read [#253](https://github.com/amoffat/sh/issues/253)
- `_long_sep` can now take `None` which splits the long form arguments into individual arguments [#258](https://github.com/amoffat/sh/issues/258)
- making `_piped` perform "direct" piping by default (linking fds together). this fixes memory problems [#270](https://github.com/amoffat/sh/issues/270)
- bugfix where calling `next()` on an iterable process that has raised `StopIteration`, hangs [#273](https://github.com/amoffat/sh/issues/273)
- `sh.cd` called with no arguments no changes into the user's home directory, like native `cd` [#275](https://github.com/amoffat/sh/issues/275)
- `sh.glob` removed entirely. the rationale is correctness over hand-holding. [#279](https://github.com/amoffat/sh/issues/279)
- added `_truncate_exc`, defaulting to `True`, which tells our exceptions to truncate output.
- bugfix for exceptions whose messages contained unicode
- `_done` callback no longer assumes you want your command put in the background.
- `_done` callback is now called asynchronously in a separate thread.
- `_done` callback is called regardless of exception, which is necessary in order to release held resources, for example a process pool

## 1.10 - 12/30/14

- partially applied functions with `functools.partial` have been fixed for `_out` and `_err` callbacks [#160](https://github.com/amoffat/sh/issues/160)
- `_out` or `_err` being callables no longer puts the running command in the background. to achieve the previous behavior, pass `_bg=True` to your command.
- deprecated `_with` contexts [#195](https://github.com/amoffat/sh/issues/195)
- `_timeout_signal` allows you to specify your own signal to kill a timed-out process with. use a constant from the `signal` stdlib module. [#171](https://github.com/amoffat/sh/issues/171)
- signal exceptions can now be caught by number or name. `SignalException_9 == SignalException_SIGKILL`
- child processes that timeout via `_timeout` raise `sh.TimeoutException` instead of `sh.SignalExeception_9` [#172](https://github.com/amoffat/sh/issues/172)
- fixed `help(sh)` from the python shell and `pydoc sh` from the command line. [#173](https://github.com/amoffat/sh/issues/173)
- program names can no longer be shadowed by names that sh.py defines internally. removed the requirement of trailing underscores for programs that could have their names shadowed, like `id`.
- memory optimization when a child process's stdin is a newline-delimted string and our bufsize is newlines
- feature, `_done` special keyword argument that accepts a callback to be called when the command completes successfully [#185](https://github.com/amoffat/sh/issues/185)
- bugfix for being unable to print a baked command in python3+ [#176](https://github.com/amoffat/sh/issues/176)
- bugfix for cwd not existing and causing the child process to continue running parent process code [#202](https://github.com/amoffat/sh/issues/202)
- child process is now guaranteed to exit on exception between fork and exec.
- fix python2 deprecation warning when running with -3 [PR #165](https://github.com/amoffat/sh/pull/165)
- bugfix where sh.py was attempting to execute directories [#196](https://github.com/amoffat/sh/issues/196), [PR #189](https://github.com/amoffat/sh/pull/189)
- only backgrounded processes will ignore SIGHUP
- allowed `ok_code` to take a `range` object. [#PR 210](https://github.com/amoffat/sh/pull/210/files)
- added `sh.args` with context which allows overriding of all command defaults for the duration of that context.
- added `sh.pushd` with context which takes a directory name and changes to that directory for the duration of that with context. [PR #206](https://github.com/amoffat/sh/pull/206)
- tests now include python 3.4 if available. tests also stop on the first
  python that suite that fails.
- SIGABRT, SIGBUS, SIGFPE, SIGILL, SIGPIPE, SIGSYS have been added to the list of signals that throw an exception [PR #201](https://github.com/amoffat/sh/pull/201)
- "callable" builtin has been faked for python3.1, which lacks it.
- "direct" option added to `_piped` special keyword argument, which allows sh to hand off a process's stdout fd directly to another process, instead of buffering its stdout internally, then handing it off. [#119](https://github.com/amoffat/sh/issues/119)

## 1.09 - 9/08/13

- Fixed encoding errors related to a system encoding "ascii". [#123](https://github.com/amoffat/sh/issues/123)
- Added exit_code attribute to SignalException and ErrorReturnCode exception classes. [#127](https://github.com/amoffat/sh/issues/127)
- Making the default behavior of spawned processes to not be explicitly killed when the parent python process ends. Also making the spawned process ignore SIGHUP. [#139](https://github.com/amoffat/sh/issues/139)
- Made OSX sleep hack to apply to PY2 as well as PY3.

## 1.08 - 1/29/12

- Added SignalException class and made all commands that end terminate by a signal defined in SIGNALS_THAT_SHOULD_THROW_EXCEPTION raise it. [#91](https://github.com/amoffat/sh/issues/91)
- Bugfix where CommandNotFound was not being raised if Command was created by instantiation. [#113](https://github.com/amoffat/sh/issues/113)
- Bugfix for Commands that are wrapped with functools.wraps() [#121](https://github.com/amoffat/sh/issues/121]
- Bugfix where input arguments were being assumed as ascii or unicode, but never as a string in a different encoding.
- \_long_sep keyword argument added joining together a dictionary of arguments passed in to a command
- Commands can now be passed a dictionary of args, and the keys will be interpretted "raw", with no underscore-to-hyphen conversion
- Reserved Python keywords can now be used as subcommands by appending an underscore `_` to them

## 1.07 - 11/21/12

- Bugfix for PyDev when `locale.getpreferredencoding()` is empty.
- Fixes for IPython3 that involve `sh.<tab>` and `sh?`
- Added `_tee` special keyword argument to force stdout/stderr to store internally and make available for piping data that is being redirected.
- Added `_decode_errors` to be passed to all stdout/stderr decoding of a process.
- Added `_no_out`, `_no_err`, and `_no_pipe` special keyword arguments. These are used for long-running processes with lots of output.
- Changed custom loggers that were created for each process to fixed loggers, so there are no longer logger references laying around in the logging module after the process ends and it garbage collected.

## 1.06 - 11/10/12

- Removed old undocumented cruft of ARG1..ARGN and ARGV.
- Bugfix where `logging_enabled` could not be set from the importing module.
- Disabled garbage collection before fork to prevent garbage collection in child process.
- Major bugfix where cyclical references were preventing process objects (and their associated stdout/stderr buffers) from being garbage collected.
- Bugfix in RunningCommand and OProc loggers, which could get really huge if a command was called that had a large number of arguments.

## 1.05 - 10/20/12

- Changing status from alpha to beta.
- Python 3.3 officially supported.
- Documentation fix. The section on exceptions now references the fact that signals do not raise an exception, even for signals that might seem like they should, e.g. segfault.
- Bugfix with Python 3.3 where importing commands from the sh namespace resulted in an error related to `__path__`
- Long-form and short-form options to commands may now be given False to disable the option from being passed into the command. This is useful to pass in a boolean flag that you flip to either True or False to enable or disable some functionality at runtime.

## 1.04 - 10/07/12

- Making `Command` class resolve the `path` parameter with `which` by default instead of expecting it to be resolved before it is passed in. This change shouldn't affect backwards compatibility.
- Fixing a bug when an exception is raised from a program, and the error output has non-ascii text. This didn't work in Python < 3.0, because .decode()'s default encoding is typically ascii.

---

### SECURITY

# Security Policy

## Supported Versions

| Version | Supported          |
| ------- | ------------------ |
| >= 2.2.5   | :white_check_mark: |

## Reporting a Vulnerability

Please disclose any vulnerabilities to arwmoffat@gmail.com

---