## File: README.md Pipenv: Python Development Workflow for Humans ============================================== [](https://python.org/pypi/pipenv) [](https://python.org/pypi/pipenv) [](https://github.com/pypa/pipenv/actions/workflows/ci.yaml) [](https://python.org/pypi/pipenv) ------------------------------------------------------------------------ **Pipenv** is a Python virtualenv management tool that supports a multitude of systems and nicely bridges the gaps between pip, python (using system python, pyenv or asdf) and virtualenv. *Linux, macOS, and Windows are all first-class citizens in pipenv.* Pipenv automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your `Pipfile` as you install/uninstall packages. It also generates a project `Pipfile.lock`, which is used to produce deterministic builds. Pipenv is primarily meant to provide users and developers of applications with an easy method to arrive at a consistent working project environment. The problems that Pipenv seeks to solve are multi-faceted: - You no longer need to use `pip` and `virtualenv` separately: they work together. - Managing a `requirements.txt` file with package hashes can be problematic. Pipenv uses `Pipfile` and `Pipfile.lock` to separate abstract dependency declarations from the last tested combination. - Hashes are documented in the lock file which are verified during install. Security considerations are put first. - Strongly encourage the use of the latest versions of dependencies to minimize security risks [arising from outdated components](https://www.owasp.org/index.php/Top_10-2017_A9-Using_Components_with_Known_Vulnerabilities). - Gives you insight into your dependency graph (e.g. `$ pipenv graph`). - Streamline development workflow by supporting local customizations with `.env` files. Table Of Contents ------------------ - [Pipenv](#pipenv-python-development-workflow-for-humans) - [Installation](#installation) - [Features](#features) - [Basic Concepts](#basic-concepts) - [Other Commands](#other-commands) - [Shell Completion](#shell-completion) - [Usage](#usage) - [Usage Examples](#usage-examples) - [Commands](#commands) - [Locate the Project](#locate-the-project) - [Locate the virtualenv](#locate-the-virtualenv) - [Locate the Python Interpreter](#locate-the-python-interpreter) - [Install Packages](#install-packages) - [Installing from git](#installing-from-git) - [Install a dev dependency](#install-a-dev-dependency) - [Show a dependency graph](#show-a-dependency-graph) - [Generate a lockfile](#generate-a-lockfile) - [Install all dev dependencies](#install-all-dev-dependencies) - [Uninstall everything](#uninstall-everything) - [Use the shell](#use-the-shell) - [Documentation](#documentation) Installation ------------ **Pipenv can be installed with Python 3.10 and above.** For most users, we recommend installing Pipenv using `pip`: pip install --user pipenv Or, if you\'re using FreeBSD: pkg install py39-pipenv Or, if you\'re using Gentoo: sudo emerge pipenv Or, if you\'re using Void Linux: sudo xbps-install -S python3-pipenv Alternatively, some users prefer to use [Pipx](https://pypi.org/p/pipx): pipx install pipenv Or, some users prefer to use Python pip module. python -m pip install pipenv Refer to the [documentation](https://pipenv.pypa.io/en/latest/#install-pipenv-today) for latest instructions. ✨🍰✨ Features ---------- - Enables truly *deterministic builds*, while easily specifying *only what you want*. - Generates and checks file hashes for locked dependencies. - Automatically install required Pythons, if `pyenv` or `asdf` is available. - Automatically finds your project home, recursively, by looking for a `Pipfile`. - Automatically generates a `Pipfile`, if one doesn\'t exist. - Automatically creates a virtualenv in a standard location. - Automatically adds/removes packages to a `Pipfile` when they are installed/uninstalled. - Automatically loads `.env` files, if they exist. For command reference, see [Commands](https://pipenv.pypa.io/en/latest/commands/). ### Basic Concepts - A virtualenv will automatically be created, when one doesn\'t exist. - When no parameters are passed to `install`, all packages `[packages]` specified will be installed. - Otherwise, whatever virtualenv defaults to will be the default. ### Shell Completion Shell completion is powered by [argcomplete](https://kislyuk.github.io/argcomplete/). Install the optional dependency first: pip install "pipenv[completion]" For **fish**, add this to `~/.config/fish/completions/pipenv.fish`: register-python-argcomplete --shell fish pipenv | source There is also a [fish plugin](https://github.com/fisherman/pipenv), which will automatically activate your subshells for you! For **zsh**, add this to your `~/.zshrc`: autoload -U bashcompinit bashcompinit eval "$(register-python-argcomplete pipenv)" For **bash** (including Git Bash on Windows), add this to `~/.bashrc` or `~/.bash_profile`: eval "$(register-python-argcomplete pipenv)" Then reload your shell, and tab completions are enabled! > **Note:** If you are upgrading from pipenv < 2026.5.0, the old > `_PIPENV_COMPLETE` environment variable method no longer works. > Replace it with the `register-python-argcomplete` commands above. > See the [Shell Completion docs](https://pipenv.pypa.io/en/latest/shell.html#shell-completion) > for details. Usage ------- $ pipenv --help Usage: pipenv [OPTIONS] COMMAND [ARGS]... Options: --where Output project home information. --venv Output virtualenv information. --py Output Python interpreter information. --envs Output Environment Variable options. --rm Remove the virtualenv. --bare Minimal output. --man Display manpage. --support Output diagnostic information for use in GitHub issues. --site-packages / --no-site-packages Enable site-packages for the virtualenv. [env var: PIPENV_SITE_PACKAGES] --python TEXT Specify which version of Python virtualenv should use. --clear Clears caches (pipenv, pip). [env var: PIPENV_CLEAR] -q, --quiet Quiet mode. -v, --verbose Verbose mode. --pypi-mirror TEXT Specify a PyPI mirror. --version Show the version and exit. -h, --help Show this message and exit. ### Usage Examples: Create a new project using Python 3.7, specifically: $ pipenv --python 3.7 Remove project virtualenv (inferred from current directory): $ pipenv --rm Install all dependencies for a project (including dev): $ pipenv install --dev Create a lockfile containing pre-releases: $ pipenv lock --pre Show a graph of your installed dependencies: $ pipenv graph Check your installed dependencies for security vulnerabilities: $ pipenv check Install a local setup.py into your virtual environment/Pipfile: $ pipenv install -e . Use a lower-level pip command: $ pipenv run pip freeze ### Commands: check Checks for PyUp Safety security vulnerabilities and against PEP 508 markers provided in Pipfile. clean Uninstalls all packages not specified in Pipfile.lock. graph Displays currently-installed dependency graph information. install Installs provided packages and adds them to Pipfile, or (if no packages are given), installs all packages from Pipfile. lock Generates Pipfile.lock. open View a given module in your editor. requirements Generate a requirements.txt from Pipfile.lock. run Spawns a command installed into the virtualenv. scripts Lists scripts in current environment config. shell Spawns a shell within the virtualenv. sync Installs all packages specified in Pipfile.lock. uninstall Uninstalls a provided package and removes it from Pipfile. update Runs lock, then sync. upgrade Update the lock of the specified dependency / sub-dependency, but does not actually install the packages. verify Verify the hash in Pipfile.lock is up-to-date. ### Locate the project: $ pipenv --where /Users/kennethreitz/Library/Mobile Documents/com~apple~CloudDocs/repos/kr/pipenv/test ### Locate the virtualenv: $ pipenv --venv /Users/kennethreitz/.local/share/virtualenvs/test-Skyy4vre ### Locate the Python interpreter: $ pipenv --py /Users/kennethreitz/.local/share/virtualenvs/test-Skyy4vre/bin/python ### Install packages: $ pipenv install Creating a virtualenv for this project... ... No package provided, installing all dependencies. Virtualenv location: /Users/kennethreitz/.local/share/virtualenvs/test-EJkjoYts Installing dependencies from Pipfile.lock... ... To activate this project's virtualenv, run the following: $ pipenv shell ### Installing from git: You can install packages with pipenv from git and other version control systems using URLs formatted according to the following rule: +:////@# The only optional section is the `@` section. When using git over SSH, you may use the shorthand vcs and scheme alias `git+git@:/@#`. Note that this is translated to `git+ssh://git@` when parsed. Valid values for `` include `git`, `bzr`, `svn`, and `hg`. Valid values for `` include `http,`, `https`, `ssh`, and `file`. In specific cases you also have access to other schemes: `svn` may be combined with `svn` as a scheme, and `bzr` can be combined with `sftp` and `lp`. Note that it is **strongly recommended** that you install any version-controlled dependencies in editable mode, using `pipenv install -e`, in order to ensure that dependency resolution can be performed with an up to date copy of the repository each time it is performed, and that it includes all known dependencies. Below is an example usage which installs the git repository located at `https://github.com/requests/requests.git` from tag `v2.19.1` as package name `requests`: $ pipenv install -e git+https://github.com/requests/requests.git@v2.19#egg=requests Creating a Pipfile for this project... Installing -e git+https://github.com/requests/requests.git@v2.19.1#egg=requests... [...snipped...] Adding -e git+https://github.com/requests/requests.git@v2.19.1#egg=requests to Pipfile's [packages]... [...] You can read more about [pip's implementation of vcs support here](https://pip.pypa.io/en/stable/topics/vcs-support/). ### Install a dev dependency: $ pipenv install pytest --dev Installing pytest... ... Adding pytest to Pipfile's [dev-packages]... ### Show a dependency graph: $ pipenv graph requests==2.18.4 - certifi [required: >=2017.4.17, installed: 2017.7.27.1] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.7, installed: 2.6] - urllib3 [required: <1.23,>=1.21.1, installed: 1.22] ### Generate a lockfile: $ pipenv lock Assuring all dependencies from Pipfile are installed... Locking [dev-packages] dependencies... Locking [packages] dependencies... Note: your project now has only default [packages] installed. To install [dev-packages], run: $ pipenv install --dev ### Install all dev dependencies: $ pipenv install --dev Pipfile found at /Users/kennethreitz/repos/kr/pip2/test/Pipfile. Considering this to be the project home. Pipfile.lock out of date, updating... Assuring all dependencies from Pipfile are installed... Locking [dev-packages] dependencies... Locking [packages] dependencies... ### Uninstall everything: $ pipenv uninstall --all No package provided, un-installing all dependencies. Found 25 installed package(s), purging... ... Environment now purged and fresh! ### Use the shell: $ pipenv shell Loading .env environment variables... Launching subshell in virtual environment. Type 'exit' or 'Ctrl+D' to return. $ β–― ### PURPOSE AND ADVANTAGES OF PIPENV To understand the problems that Pipenv solves, it's useful to show how Python package management has evolved. Take yourself back to the first Python iteration. We had Python, but there was no clean way to install packages. Then came Easy Install, a package that installs other Python packages with relative ease. But it came with a catch: it wasn't easy to uninstall packages that were no longer needed. Enter pip, which most Python users are familiar with. pip lets us install and uninstall packages. We could specify versions, run pip freeze > requirements.txt to output a list of installed packages to a text file, and use that same text file to install everything an app needed with pip install -r requirements.txt. But pip didn't include a way to isolate packages from each other. We might work on apps that use different versions of the same libraries, so we needed a way to enable that. Pipenv aims to solve several problems. First, the problem of needing the pip library for package installation, plus a library for creating a virtual environment, plus a library for managing virtual environments, plus all the commands associated with those libraries. That's a lot to manage. Pipenv ships with package management and virtual environment support, so you can use one tool to install, uninstall, track, and document your dependencies and to create, use, and organize your virtual environments. When you start a project with it, Pipenv will automatically create a virtual environment for that project if you aren't already using one. Pipenv accomplishes this dependency management by abandoning the requirements.txt norm and trading it for a new document called a Pipfile. When you install a library with Pipenv, a Pipfile for your project is automatically updated with the details of that installation, including version information and possibly the Git repository location, file path, and other information. Second, Pipenv wants to make it easier to manage complex interdependencies. Using Pipenv, which gives you Pipfile, lets you avoid these problems by managing dependencies for different environments for you. This command will install the main project dependencies: pipenv install Adding the --dev tag will install the dev/testing requirements: pipenv install --dev To generate a Pipfile.lock file, run: pipenv lock You can also run Python scripts with Pipenv. To run a top-level Python script called hello.py, run: pipenv run python hello.py And you will see your expected result in the console. To start a shell, run: pipenv shell If you would like to convert a project that currently uses a requirements.txt file to use Pipenv, install Pipenv and run: pipenv install -r requirements.txt This will create a Pipfile and install the specified requirements. Documentation --------------- Documentation resides over at [pipenv.pypa.io](https://pipenv.pypa.io/en/latest/). ## Star History --- ## File: docker/README.md # Public docker images Build all images with: ``` $ make build-all PIPENV=2023.07.3 ``` Build a single image with with: ``` $ make docker-build docker-push TAG=3.11-alpine-v2023-6-26 PYVERSION=3.11-alpine PIPENV=2023.6.26 ``` --- ## File: docs/dev/contributing.md # Contributing to Pipenv If you're reading this, you're probably interested in contributing to Pipenv. Thank you very much! Open source projects live-and-die based on the support they receive from others, and the fact that you're even considering contributing to the Pipenv project is _very_ generous of you. This document lays out guidelines and advice for contributing to this project. If you're thinking of contributing, please start by reading this document and getting a feel for how contributing to this project works. The guide is split into sections based on the type of contribution you're thinking of making, with a section that covers general guidelines for all contributors. ## General Guidelines ### Be Cordial > **Be cordial or be on your way**. _β€”Kenneth Reitz_ Pipenv has one very important rule governing all forms of contribution, including reporting bugs or requesting features. This golden rule is [be cordial or be on your way](https://kennethreitz.org/essays/2013/01/27/be-cordial-or-be-on-your-way) **All contributions are welcome**, as long as everyone involved is treated with respect. ### Get Early Feedback If you are contributing, do not feel the need to sit on your contribution until it is perfectly polished and complete. It helps everyone involved for you to seek feedback as early as you possibly can. Submitting an early, unfinished version of your contribution for feedback in no way prejudices your chances of getting that contribution accepted, and can save you from putting a lot of work into a contribution that is not suitable for the project. ### Contribution Suitability Our project maintainers have the last word on whether or not a contribution is suitable for Pipenv. All contributions will be considered carefully, but from time to time, contributions will be rejected because they do not suit the current goals or needs of the project. If your contribution is rejected, don't despair! As long as you followed these guidelines, you will have a much better chance of getting your next contribution accepted. ## Questions The GitHub issue tracker is for _bug reports_ and _feature requests_. Please do not use it to ask questions about how to use Pipenv. These questions should instead be directed to [Stack Overflow](https://stackoverflow.com/). Make sure that your question is tagged with the `pipenv` tag when asking it on Stack Overflow, to ensure that it is answered promptly and accurately. ## Code Contributions ### Steps for Submitting Code When contributing code, you'll want to follow this checklist: 1. Fork the repository on GitHub. 2. Set up your [development environment](#development-setup) 3. Run the tests from [here](#run-the-tests) to confirm they all pass on your system. If they don't, you'll need to investigate why they fail. If you're unable to diagnose this yourself, raise it as a bug report by following the guidelines in this [document](#bug-reports). 4. Write tests that demonstrate your bug or feature. Ensure that they fail. 5. Make your change. 6. Run the entire test suite again, confirming that all tests pass _including the ones you just added_. 7. Send a GitHub Pull Request to the main repository's `main` branch. GitHub Pull Requests are the expected method of code collaboration on this project. The following sub-sections go into more detail on some of the points above. ### Development Setup The repository version of Pipenv must be installed over other global versions to resolve conflicts with the `pipenv` folder being implicitly added to `sys.path`. See [pypa/pipenv#2557](https://github.com/pypa/pipenv/issues/2557) for more details. Pipenv now uses pre-commit hooks similar to Pip in order to apply linting and code formatting automatically! The build now also checks that these linting rules have been applied to the code before running the tests. The build will fail when linting changes are detected so be sure to sync dev requirements and install the pre-commit hooks locally: ```bash $ python -m pip install -e ".[tests,dev]" $ pipenv install --dev $ pipenv run python -m pip install -e ".[tests,dev]" # This will configure running the pre-commit checks at start of each commit $ pre-commit install # Should you want to check the pre-commit configuration against all configured project files $ pre-commit run --all-files --verbose ``` ### Code Review Contributions will not be merged until they have been code reviewed. You should implement any code review feedback unless you strongly object to it. In the event that you object to the code review feedback, you should make your case clearly and calmly. If, after doing so, the feedback is judged to still apply, you must either apply the feedback or withdraw your contribution. ### Package Index To speed up testing, tests that rely on a package index for locking and installing use a local server that contains vendored packages in the `tests/pypi` directory. Each vendored package should have it's own folder containing the necessary releases. When adding a release for a package, it is easiest to use either the `.tar.gz` or universal wheels (ex: `py2.py3-none`). If a `.tar.gz` or universal wheel is not available, add wheels for all available architectures and platforms. ### Owned code vs. vendored code Every Python file under `pipenv/` lives in exactly one of two states. Knowing which state a file is in tells you who is allowed to edit it and under what rules. - **Owned** β€” everything under `pipenv/` *except* `pipenv/patched/` and `pipenv/vendor/`. This is project-owned code. Refactor freely. Project conventions, linting (`ruff check pipenv/`), and test coverage apply. There is no upstream to sync against and no expectation of doing so. - **Vendored** β€” `pipenv/patched/` (patched copy of pip) and `pipenv/vendor/` (third-party packages). These trees are managed by the project's vendoring tooling. Do not hand-edit them; let the tooling re-sync from upstream. Updates land via `vendor:`-prefixed commits driven by that tooling, not by direct edits. **There is no third state.** Inlining upstream code into `pipenv/utils/` (or anywhere else outside `pipenv/vendor/`) without owning it is explicitly out of policy going forward. The reason is hard-won: the "worst-of-both-worlds" middle state β€” code that was copy-pasted from an upstream library years ago, never re-synced, and silently drifted from the upstream while still being treated as "someone else's code we shouldn't touch" β€” is exactly what the Initiative B sweep spent five work units cleaning up. The pipenv project does not have the maintainer capacity to chase upstream changes against inlined copies that have already drifted. If a piece of upstream code is worth keeping in-tree, either re-vendor it under `pipenv/vendor/` (so the tooling tracks it) or adopt it (so the project owns it). Pick one. **Exception: deliberate forks.** It is fine to keep a fork of an upstream function in the owned tree when the fork has *intentionally* divergent behaviour from the upstream version β€” but the fork must carry a provenance docstring that names the upstream symbol it forked from and spells out what divergence is intentional. Two examples shipped as part of Initiative B's Wave 1c make good templates: - `pipenv/utils/requirements.py` β€” `redact_netloc` and `redact_auth_from_url` are deliberate forks of `pip._internal.utils.misc.redact_netloc` / `redact_auth_from_url`. The divergence (env-var placeholders and standard SSH usernames are preserved instead of being rewritten to `****`) is documented inline. - `pipenv/utils/requirementslib.py` β€” `unpack_url` and `get_http_url` are verbatim adoptions of `pip._internal.operations.prepare.unpack_url` and `get_http_url`. Their docstrings name the upstream symbol and flag the follow-up question of whether the patched-pip copies can be called directly instead. When you add or substantially refactor a fork, follow the same convention: a "Provenance:" paragraph in the docstring naming the upstream symbol and the intentional divergence. A reader two years from now should never have to guess whether a function is a stale copy or a deliberate fork. For the per-symbol audit that drove this policy, see [`docs/dev/initiative-b-triage.md`](./initiative-b-triage.md). ## News Fragments (Changelog Entries) Pipenv uses [towncrier](https://towncrier.readthedocs.io/) to manage its changelog. Every pull request that changes user-visible behaviour should include a **news fragment** so the change is captured in the next release's `CHANGELOG.md`. ### Creating a News Fragment News fragments live in the `news/` directory. Create a file named: ``` news/..rst ``` Where `` is one of: | Type | Description | Appears in CHANGELOG? | |------|-------------|----------------------| | `feature` | New feature or improvement | Yes | | `behavior` | Change to existing behavior | Yes | | `bugfix` | Bug fix | Yes | | `vendor` | Update to a vendored library | Yes | | `doc` | Documentation-only improvement | Yes | | `removal` | Deprecation or removal | Yes | | `process` | Change to dev/CI process | Yes | | `trivial` | Typo fix, refactor, or other minor change | **No** (not shown in CHANGELOG) | For example, to document a bug fix for issue #1234: ```bash $ echo "Fixed the frobnication logic when the widget is None." > news/1234.bugfix.rst ``` ### Trivial Fragments Use the `trivial` type for changes that do **not** warrant a user-facing changelog entry (e.g. internal refactors, test-only changes, typo fixes in comments): ```bash $ echo "." > news/1234.trivial.rst ``` ```{note} Trivial fragments **are** required by CI to confirm you have considered whether a changelog entry is needed β€” but they are intentionally excluded from the generated `CHANGELOG.md`. This is expected: `towncrier` collects them, deletes the files, and silently omits them from the output. You do not need to add anything to the trivial fragment's content; a single `.` is conventional. ``` ### Generating the Changelog Locally To preview what the next changelog section will look like: ```bash $ towncrier build --draft --version 9999.0.0 ``` This prints the draft changelog to stdout without modifying any files. ## Documentation Contributions Documentation improvements are always welcome! The documentation files live in the `docs/` directory of the codebase. They're written in [MarkDown](https://www.markdownguide.org/), and use [Sphinx](http://sphinx-doc.org/index.html) to generate the full suite of documentation. When contributing documentation, please do your best to follow the style of the documentation files. This means a soft-limit of 79 characters wide in your text files and a semi-formal, yet friendly and approachable, prose style. When presenting Python code, use single-quoted strings (`'hello'` instead of `"hello"`). ## Bug Reports Bug reports are hugely important! They are recorded as [GitHub issues](https://github.com/pypa/pipenv/issues). Please be aware of the following things when filing bug reports: 1. Avoid raising duplicate issues. _Please_ use the GitHub issue search feature to check whether your bug report or feature request has been mentioned in the past. Duplicate bug reports and feature requests are a huge maintenance burden on the limited resources of the project. If it is clear from your report that you would have struggled to find the original, that's okay, but if searching for a selection of words in your issue title would have found the duplicate then the issue will likely be closed extremely abruptly. 2. When filing bug reports about exceptions or tracebacks, please include the _complete_ traceback. Partial tracebacks, or just the exception text, are not helpful. Issues that do not contain complete tracebacks may be closed without warning. 3. Make sure you provide a suitable amount of information to work with. This means you should provide: - Guidance on **how to reproduce the issue**. Ideally, this should be a _small_ code sample that can be run immediately by the maintainers. Failing that, let us know what you're doing, how often it happens, what environment you're using, etc. Be thorough: it prevents us needing to ask further questions. - Tell us **what you expected to happen**. When we run your example code, what are we expecting to happen? What does "success" look like for your code? - Tell us **what actually happens**. It's not helpful for you to say "it doesn't work" or "it fails". Tell us _how_ it fails: do you get an exception? A hang? The packages installed seem incorrect? How was the actual result different from your expected result? - Tell us **what version of Pipenv you're using**, and **how you installed it**. Different versions of Pipenv behave differently and have different bugs, and some distributors of Pipenv ship patches on top of the code we supply. If you do not provide all of these things, it will take us much longer to fix your problem. If we ask you to clarify these and you never respond, we will close your issue without fixing it. ## Run the tests Tests are written in `pytest` style and can be run very simply: ```bash pytest ``` However many tests depend on running a private pypi server on localhost:8080. This can be accomplished by using either the `run-tests.sh` or `run-tests.bat` scripts which will start the `pypiserver` process ahead of invoking pytest. You may also manually perform this step and then invoke pytest as you would normally. Example: # Linux or MacOS pipenv run pypi-server run -v --host=0.0.0.0 --port=8080 --hash-algo=sha256 --disable-fallback ./tests/pypi/ ./tests/fixtures & # Windows cmd /c start pipenv run pypi-server run -v --host=0.0.0.0 --port=8080 --hash-algo=sha256 --disable-fallback ./tests/pypi/ ./tests/fixtures This will run all Pipenv tests, which can take awhile. To run a subset of the tests, the standard pytest filters are available, such as: - provide a directory or file: `pytest tests/unit` or `pytest tests/unit/test_cmdparse.py` - provide a keyword expression: `pytest -k test_lock_editable_vcs_without_install` - provide a nodeid: `pytest tests/unit/test_cmdparse.py::test_parse` - provide a test marker: `pytest -m lock` There are a few other ways of running the tests: 1. test scripts The scripts for bash or windows: `run-tests.sh` and `run-tests.bat` Note that, you override the default Python Pipenv will use with PIPENV_PYTHON and the Python binary name with PYTHON in case it is not called `python` on your system or in case you have many. Here is an example how you can override both variables (you can override just one too): $ PYTHON=python3.8 PIPENV_PYTHON=python3.9 run-tests.sh You can also do: $ PYTHON=/opt/python/python3.10/python3 run-tests.sh If you need to change how pytest is invoked, see how to run the test suite manually. The `run-tests.sh` script does the same steps the Github CI workflow does, and as such it is recommended you run it before you open a PR. Taking this second approach, will allow you, for example, to run a single test case, or `fail fast` if you need it. 2. Manually This repeats the steps of the scripts above: ```console $ git clone https://github.com/pypa/pipenv.git $ cd pipenv $ git submodule sync && git submodule update --init --recursive $ python -m pip install -e ".[tests,dev]" $ pipenv install --dev $ pipenv run python -m pip install -e ".[tests,dev]" $ pipenv run pytest [--any optional arguments to pytest] ``` The second options assumes you already have `pipenv` on your system. And simply repeats all the steps in the script above. Preferably, you should be running your tests in a Linux container (or FreeBSD Jail or even VM). This will guarantee that you don't break stuff, and that the tests run in a pristine environment. Consider doing something like this: $ docker run --rm -v $(pwd):/usr/src -it python:3.7 bash # inside the container # adduser --disabled-password debian # su debian && cd /usr/src/ # bash run-tests.sh 3. Using the Makefile: The Makefile automates all the task as in the script. However, it allows one more fine grained control on every step. For example: $ make ramdisk # create a ram disk to preserve your SSDs life $ make ramdisk-virtualenv $ make test suite="-m not cli" # run all tests but cli or $ make tests parallel="" suite="tests/integration/test_cli.py::test_pipenv_check" It is important that your environment is setup correctly, and this may take some work, for example, on a specific Mac installation, the following steps may be needed: ```bash # Make sure the tests can access github if [ "$SSH_AGENT_PID" = "" ] then eval ``ssh-agent`` ssh-add fi # Use unix like utilities, installed with brew, # e.g. brew install coreutils for d in /usr/local/opt/*/libexec/gnubin /usr/local/opt/python/libexec/bin do [[ ":$PATH:" != *":$d:"* ]] && PATH="$d:${PATH}" done export PATH # PIP_FIND_LINKS currently breaks test_uninstall.py unset PIP_FIND_LINKS ``` --- ## File: docs/advanced.md # Advanced Pipenv Usage This guide covers advanced features and techniques for using Pipenv effectively in complex scenarios. These topics build on the basic functionality covered in other documentation sections. ## Passing Additional Arguments to pip When you need more control over the underlying pip commands that Pipenv executes, you can pass additional arguments directly to pip. ### Using --extra-pip-args The `--extra-pip-args` option allows you to supply additional arguments to pip during installation: ```bash $ pipenv install --extra-pip-args="--use-feature=truststore --proxy=127.0.0.1" ``` This is particularly useful for: - Using experimental pip features - Configuring proxies - Setting build options for packages with C extensions - Controlling cache behavior - Specifying platform-specific wheels ### Common Use Cases #### Using System Certificate Stores ```bash $ pipenv install --extra-pip-args="--use-feature=truststore" ``` #### Installing Platform-Specific Packages ```bash $ pipenv install --extra-pip-args="--platform=win_amd64 --only-binary=:all:" ``` #### Setting Build Options ```bash $ pipenv install pycurl --extra-pip-args="--global-option=--with-openssl-dir=/usr/local/opt/openssl" ``` ### Using Environment Variables During Installation Some packages require environment variables to be set during installation for proper compilation or configuration. You can set these variables before running `pipenv install`: #### Example: Installing CUDA-enabled Packages For packages that need CMake arguments, such as `llama-cpp-python` with CUDA support: ```bash # Unix/Linux/macOS $ CMAKE_ARGS="-DGGML_CUDA=on" pipenv install llama-cpp-python # Or export for multiple commands $ export CMAKE_ARGS="-DGGML_CUDA=on" $ pipenv install llama-cpp-python ``` #### Example: Other Build Environment Variables ```bash # For packages that need specific compiler flags $ CFLAGS="-O3" CXXFLAGS="-O3" pipenv install some-package # For packages using meson build system $ MESON_ARGS="-Dfeature=enabled" pipenv install some-package # For packages requiring specific library paths $ LDFLAGS="-L/usr/local/lib" CPPFLAGS="-I/usr/local/include" pipenv install some-package ``` ```{note} These environment variables are passed to pip and subsequently to the package's build system (setuptools, CMake, meson, etc.). The specific variables needed depend on the package being installed. ``` ## Deployment Strategies Pipenv provides several approaches for deploying applications in production environments. ### Using --deploy Flag The `--deploy` flag ensures that your `Pipfile.lock` is up-to-date with your `Pipfile` before installation: ```bash $ pipenv install --deploy ``` This will fail if: - The `Pipfile.lock` is out of date - The `Pipfile.lock` is missing - The hash in `Pipfile.lock` doesn't match the `Pipfile` This is crucial for production deployments to ensure you're installing exactly what you expect. ### System-Wide Installation For some deployment scenarios (like containerized applications), you may want to install packages directly to the system Python rather than in a virtual environment: ```bash $ pipenv install --system --deploy ``` This installs all packages specified in `Pipfile.lock` to the system Python. Use this approach with caution, as it can potentially conflict with system packages. ### Verifying Lock File Without Installing To verify that your `Pipfile.lock` is up-to-date without installing packages: ```bash $ pipenv verify ``` This is useful in CI/CD pipelines to ensure the lock file has been properly updated after changes to the `Pipfile`. ### Docker Deployment For Docker deployments, a multi-stage build approach is recommended: ```dockerfile FROM python:3.10-slim AS builder WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Install pipenv and dependencies RUN pip install pipenv && \ pipenv install --deploy --system FROM python:3.10-slim WORKDIR /app # Copy installed packages from builder stage COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages # Copy application code COPY . . # Run the application CMD ["python", "app.py"] ``` This approach: 1. Installs dependencies in a builder stage 2. Copies only the necessary files to the final image 3. Results in a smaller, more secure image ## Working with Different Python Distributions Pipenv can work with various Python distributions and installations. ### Using Specific Python Interpreters To use a specific Python interpreter: ```bash $ pipenv --python /path/to/python ``` This is useful when you have multiple Python versions installed or need to use a specific distribution. ### Anaconda/Conda Integration Pipenv can work with Anaconda/Conda Python installations, though there are some important considerations. #### Basic Usage To use Pipenv with Anaconda, specify the full path to the Conda Python interpreter: ```bash $ pipenv --python /path/to/anaconda/bin/python ``` **Important**: You must provide the complete path to the Python executable. A partial version number like `--python 3.10` may not work if Pipenv cannot find the Conda installation in its search paths. To find your Conda Python path: ```bash # With conda environment activated $ which python /home/user/anaconda3/envs/myenv/bin/python # Or check conda info $ conda info --envs ``` #### Reusing Conda-Installed Packages To allow the Pipenv virtualenv to access packages installed in Conda: ```bash $ pipenv --python /path/to/anaconda/bin/python --site-packages ``` The `--site-packages` flag allows the virtual environment to access packages installed in the parent Python installation. #### Troubleshooting Conda Issues **Virtualenv Creation Fails with Import Errors** If you encounter errors like `ImportError: cannot import name '_io'` or similar during virtualenv creation, this is typically due to outdated packages in your Conda installation: ```bash # Update conda's virtualenv and Python packages $ conda update python virtualenv ``` **Python Version Not Found** If Pipenv reports that a Python version was not found: 1. Verify the Python version is installed in Conda: ```bash $ conda search python ``` 2. Use the full path instead of just the version number: ```bash # Instead of this: $ pipenv --python 3.10 # Use the full path: $ pipenv --python ~/anaconda3/envs/py310/bin/python ``` **Mixing Conda and Pipenv** While Pipenv can work with Conda Python, be aware that: - Pipenv creates its own virtualenv separate from Conda environments - Package installations via Pipenv are isolated from Conda - For best results, choose either Conda or Pipenv for dependency management in a project, not both ### pyenv Integration Pipenv automatically detects and works with pyenv: ```bash # Set local Python version with pyenv $ pyenv local 3.10.4 # Pipenv will use this version automatically $ pipenv install ``` If the specified Python version isn't installed, Pipenv will prompt you to install it with pyenv: ```bash $ pipenv --python 3.11 Warning: Python 3.11 was not found on your system... Would you like us to install latest CPython 3.11 with pyenv? [Y/n]: y Installing CPython 3.11.0 with pyenv... ``` ### asdf Integration Similar to pyenv, Pipenv also works with asdf: ```bash # Install Python with asdf $ asdf install python 3.10.4 # Use this version with Pipenv $ pipenv --python 3.10.4 ``` ## Generating Requirements Files While Pipenv uses `Pipfile` and `Pipfile.lock`, you may need to generate traditional `requirements.txt` files for compatibility with other tools. ### Basic Requirements Generation ```bash $ pipenv requirements > requirements.txt ``` This generates a `requirements.txt` file from your `Pipfile.lock` with exact versions. ### Including Development Dependencies ```bash $ pipenv requirements --dev > requirements-dev.txt ``` ### Development Dependencies Only ```bash $ pipenv requirements --dev-only > dev-requirements.txt ``` ### Including Hashes ```bash $ pipenv requirements --hash > requirements.txt ``` This includes package hashes for additional security. ### Excluding PEP 508 Markers ```bash $ pipenv requirements --exclude-markers > requirements.txt ``` This removes environment markers (like `python_version >= '3.7'`). ### Specific Package Categories ```bash $ pipenv requirements --categories="docs,tests" > requirements-docs-tests.txt ``` This generates requirements for specific custom package categories. ## Security Features Pipenv includes several advanced security features to help protect your projects. ### Vulnerability Scanning Pipenv integrates with the [safety](https://github.com/pyupio/safety) package to scan for known vulnerabilities: ```bash $ pipenv scan ``` This checks your dependencies against the PyUp Safety database of known vulnerabilities. #### Using a Custom Vulnerability Database ```bash $ pipenv scan --db /path/to/custom/db ``` #### Ignoring Specific Vulnerabilities ```bash $ pipenv scan --ignore 12345 ``` #### Different Output Formats ```bash $ pipenv scan --output json > vulnerabilities.json ``` ### Automatic Python Installation Pipenv can automatically install the required Python version if you have pyenv or asdf installed: ```toml # Pipfile [requires] python_version = "3.11" ``` When you run `pipenv install`, it will check if Python 3.11 is available and prompt to install it if needed. ## Custom Scripts Pipenv allows you to define custom scripts in your `Pipfile` for common tasks. ### Defining Scripts ```toml [scripts] start = "python app.py" test = "pytest" lint = "flake8 ." format = "black ." ``` ### Running Scripts ```bash $ pipenv run start $ pipenv run test ``` ### Script with Arguments ```bash $ pipenv run test tests/test_api.py -v ``` ### Complex Script Definitions You can define more complex scripts using the extended syntax: ```toml [scripts] start = {cmd = "python app.py"} complex = {call = "package.module:function('arg1', 'arg2')"} ``` ## Environment Variables and Configuration ### Automatic Loading of .env Pipenv automatically loads environment variables from `.env` files in your project directory: ``` # .env DEBUG=True DATABASE_URL=postgresql://user:password@localhost/dbname ``` These variables are available when you run `pipenv shell` or `pipenv run`. ### Custom .env Location ```bash $ PIPENV_DOTENV_LOCATION=/path/to/.env pipenv shell ``` ### Variable Expansion in .env You can use variable expansion in your `.env` files: ``` # .env HOME_DIR=${HOME} CONFIG_PATH=${HOME_DIR}/.config/app ``` ## Working with Air-Gapped Environments In environments without internet access, you can still use Pipenv by preparing packages in advance. ### Downloading Packages On a connected system: ```bash # Generate requirements with hashes $ pipenv requirements --hash > requirements.txt # Download packages $ pip download -r requirements.txt -d ./packages ``` ### Installing in Air-Gapped Environment Transfer the packages directory to the air-gapped environment, then: ```bash $ pip install --no-index --find-links=./packages -r requirements.txt ``` ## Testing and CI/CD Integration ### Using Pipenv with tox Here's an example `tox.ini` for testing with multiple Python versions: ```ini [tox] envlist = py37, py38, py39, py310, py311 [testenv] deps = pipenv commands = pipenv install --dev pipenv run pytest {posargs:tests} ``` ### GitHub Actions Integration ```yaml name: Python Tests on: [push, pull_request] jobs: test: runs-on: ubuntu-latest strategy: matrix: python-version: ['3.8', '3.9', '3.10', '3.11'] steps: - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Install dependencies run: | python -m pip install --upgrade pip pip install pipenv pipenv install --dev - name: Verify lock file run: pipenv verify - name: Run tests run: pipenv run pytest - name: Security scan run: pipenv scan ``` ## Performance Optimization ### Caching Pipenv maintains a cache to speed up installations. You can control this cache: ```bash # Clear the cache $ pipenv lock --clear # Set a custom cache location $ export PIPENV_CACHE_DIR=/path/to/custom/cache ``` ### Speeding Up Installations For faster installations during development: ```bash # Skip lock file generation $ export PIPENV_SKIP_LOCK=1 $ pipenv install package-name ``` ```{warning} Only use PIPENV_SKIP_LOCK during development, not in production environments. ``` ### Handling Large Dependency Trees For projects with many dependencies: ```bash # Increase the maximum depth for dependency resolution $ export PIPENV_MAX_DEPTH=20 $ pipenv lock ``` ## Community Integrations Pipenv works well with various tools and services in the Python ecosystem: - **Heroku**: Automatically detects and uses Pipfile/Pipfile.lock - **Platform.sh**: Native support for Pipenv projects - **PyUp**: Security monitoring for Pipenv projects - **VS Code**: Built-in support for Pipenv environments - **PyCharm**: Supports Pipenv for project environments - **Emacs**: Integration via pipenv.el - **Fish Shell**: Automatic shell activation ## Opening Modules in Your Editor Pipenv allows you to quickly open installed packages in your editor: ```bash $ pipenv open requests ``` This opens the source code of the requests package in your default editor (defined by the `EDITOR` environment variable). You can specify a different editor for a one-time use: ```bash $ EDITOR=code pipenv open flask ``` This is useful for: - Exploring package source code - Debugging issues - Understanding how a package works - Adding temporary patches ## Conclusion These advanced features make Pipenv a powerful tool for Python dependency management in complex scenarios. By leveraging these capabilities, you can create more efficient, secure, and maintainable Python projects. Remember that while these advanced features provide additional flexibility and power, they should be used judiciously. Always follow best practices for security and reproducibility, especially in production environments. --- ## File: docs/cli.md # Pipenv Command Line Interface This comprehensive guide covers Pipenv's command line interface, including all available commands, options, and usage examples. ## Overview Pipenv provides a powerful command line interface (CLI) that simplifies Python dependency management and virtual environment workflows. The CLI follows a consistent pattern: ```bash pipenv [OPTIONS] COMMAND [ARGS]... ``` ## Global Options These options can be used with any Pipenv command: | Option | Description | |--------|-------------| | `--where` | Output project home information | | `--venv` | Output virtualenv information | | `--py` | Output Python interpreter information | | `--envs` | Output environment variable options | | `--rm` | Remove the virtualenv (deprecated: use `pipenv remove` instead) | | `--bare` | Minimal output | | `--man` | Display manpage | | `--support` | Output diagnostic information for GitHub issues | | `--site-packages` | Enable site-packages for the virtualenv | | `--python TEXT` | Specify which Python version to use | | `--clear` | Clear caches (pipenv, pip) | | `-q, --quiet` | Quiet mode | | `-v, --verbose` | Verbose mode | | `--pypi-mirror TEXT` | Specify a PyPI mirror | | `--version` | Show the version and exit | | `-h, --help` | Show help message and exit | ## Core Commands ### install Installs packages and adds them to Pipfile, or installs all packages from Pipfile.lock if no packages are specified. ```bash pipenv install [OPTIONS] [PACKAGES]... ``` #### Options | Option | Description | |--------|-------------| | `--dev` | Install both development and default packages | | `--categories TEXT` | Install packages to specified category groups | | `--system` | Install to system Python instead of virtualenv | | `--deploy` | Abort if Pipfile.lock is out-of-date | | `--ignore-pipfile` | Install from Pipfile.lock, ignoring Pipfile | | `--skip-lock` | Skip locking of dependencies | | `--requirements, -r TEXT` | Import a requirements.txt file | | `--extra-pip-args TEXT` | Pass additional arguments to pip | | `--python TEXT` | Specify which Python version to use | | `--dry-run` | Show what would happen without making changes | #### Examples Install a specific package: ```bash $ pipenv install requests ``` Install a package with version constraint: ```bash $ pipenv install "requests>=2.20.0" ``` Install a development dependency: ```bash $ pipenv install pytest --dev ``` Install from requirements.txt: ```bash $ pipenv install -r requirements.txt ``` Install with deployment check: ```bash $ pipenv install --deploy ``` ### uninstall Uninstalls a provided package and removes it from Pipfile. ```bash pipenv uninstall [OPTIONS] [PACKAGES]... ``` #### Options | Option | Description | |--------|-------------| | `--all` | Purge all files from the virtual environment | | `--all-dev` | Remove all development packages | | `--skip-lock` | Skip locking of dependencies | | `--categories TEXT` | Uninstall from specified category groups | #### Examples Uninstall a package: ```bash $ pipenv uninstall requests ``` Uninstall all packages: ```bash $ pipenv uninstall --all ``` Uninstall all development packages: ```bash $ pipenv uninstall --all-dev ``` ### lock Generates Pipfile.lock with all dependencies and their sub-dependencies. ```bash pipenv lock [OPTIONS] ``` #### Options | Option | Description | |--------|-------------| | `--pre` | Allow pre-releases | | `--clear` | Clear the dependency cache | | `--verbose` | Show verbose output | | `--categories TEXT` | Lock specified categories only | | `--keep-outdated` | Keep outdated dependencies | | `--python TEXT` | Specify which Python version to use for resolution | | `--extra-pip-args TEXT` | Pass additional arguments to pip | #### Examples Generate a lock file: ```bash $ pipenv lock ``` Generate a lock file with pre-releases: ```bash $ pipenv lock --pre ``` Lock specific categories: ```bash $ pipenv lock --categories="docs,tests" ``` ### sync Installs all packages specified in Pipfile.lock. ```bash pipenv sync [OPTIONS] ``` #### Options | Option | Description | |--------|-------------| | `--dev` | Install both development and default packages | | `--all` | Install packages from all categories defined in the Pipfile | | `--categories TEXT` | Install packages from specified category groups | | `--python TEXT` | Specify which Python version to use | | `--extra-pip-args TEXT` | Pass additional arguments to pip | #### Examples Install packages from lock file: ```bash $ pipenv sync ``` Install development packages too: ```bash $ pipenv sync --dev ``` Install all categories (default, dev, and any custom categories): ```bash $ pipenv sync --all ``` Install specific categories: ```bash $ pipenv sync --categories="docs,tests" ``` ### update Runs lock when no packages are specified, or upgrade for specific packages, and then sync. ```bash pipenv update [OPTIONS] [PACKAGES]... ``` #### Options | Option | Description | |--------|-------------| | `--dev` | Update development packages | | `--all` | Update packages from all categories defined in the Pipfile | | `--categories TEXT` | Update packages in specified categories | | `--outdated` | List out-of-date dependencies | | `--dry-run` | Show what would happen without making changes | | `--python TEXT` | Specify which Python version to use | | `--clear` | Clear the dependency cache | #### Examples Update all packages: ```bash $ pipenv update ``` Update specific packages: ```bash $ pipenv update requests pytest ``` Check for outdated packages: ```bash $ pipenv update --outdated ``` ### upgrade Updates the lock file for specified dependencies without installing them. ```bash pipenv upgrade [OPTIONS] [PACKAGES]... ``` #### Options | Option | Description | |--------|-------------| | `--dev` | Upgrade development packages | | `--all` | Upgrade packages from all categories defined in the Pipfile | | `--categories TEXT` | Upgrade packages in specified categories | | `--dry-run` | Show what would happen without making changes | | `--python TEXT` | Specify which Python version to use | | `--clear` | Clear the dependency cache | #### Examples Upgrade a specific package in the lock file: ```bash $ pipenv upgrade requests ``` Upgrade development packages: ```bash $ pipenv upgrade --dev ``` ### check Checks for security vulnerabilities and PEP 508 marker compliance. ```bash pipenv check [OPTIONS] ``` **Note**: The legacy check implementation is deprecated and will be removed in v2027.0.0. Use `pipenv audit` for vulnerability scanning, or use the `--scan` option to run Safety's newer scan command during the transition. #### Options | Option | Description | |--------|-------------| | `--db TEXT` | Path or URL to a PyUp Safety vulnerabilities database | | `--ignore, -i TEXT` | Ignore specified vulnerability | | `--output [screen\|text\|json\|bare]` | Specify output format | | `--key TEXT` | Safety API key from PyUp.io | | `--use-installed` | Use installed packages instead of lockfile | | `--categories TEXT` | Check packages in specified categories | | `--auto-install` | Automatically install safety if not already installed | | `--scan` | Use the new scan command instead | #### Examples Check for vulnerabilities: ```bash $ pipenv check ``` Check with a specific output format: ```bash $ pipenv check --output json ``` Use the new scan command: ```bash $ pipenv check --scan ``` ### scan Enhanced security scanning (replacement for check). ```bash pipenv scan [OPTIONS] ``` #### Options Similar to the `check` command, with enhanced functionality. #### Examples Scan for vulnerabilities: ```bash $ pipenv scan ``` Scan with a specific output format: ```bash $ pipenv scan --output json ``` ### run Spawns a command installed into the virtualenv. ```bash pipenv run [OPTIONS] COMMAND [ARGS]... ``` #### Examples Run a Python script: ```bash $ pipenv run python app.py ``` Run tests: ```bash $ pipenv run pytest ``` Run a custom script defined in Pipfile: ```bash $ pipenv run start ``` ### shell Spawns a shell within the virtualenv. ```bash pipenv shell [OPTIONS] [SHELL_ARGS]... ``` #### Options | Option | Description | |--------|-------------| | `--fancy` | Use a fancy shell activation method | #### Examples Activate the virtual environment: ```bash $ pipenv shell ``` Activate with fancy mode: ```bash $ pipenv shell --fancy ``` ### graph Displays currently-installed dependency graph information. ```bash pipenv graph [OPTIONS] ``` #### Options | Option | Description | |--------|-------------| | `--bare` | Output graph in bare format | | `--json` | Output graph in JSON format | | `--json-tree` | Output graph as JSON tree | | `--reverse` | Reversed dependency graph | #### Examples Show dependency graph: ```bash $ pipenv graph ``` Show reverse dependencies: ```bash $ pipenv graph --reverse ``` Show JSON output: ```bash $ pipenv graph --json ``` ### clean Uninstalls all packages not specified in Pipfile.lock. ```bash pipenv clean [OPTIONS] ``` #### Options | Option | Description | |--------|-------------| | `--dry-run` | Show what would be removed without removing | #### Examples Remove unused packages: ```bash $ pipenv clean ``` Preview what would be removed: ```bash $ pipenv clean --dry-run ``` ### requirements Generate a requirements.txt from Pipfile.lock. ```bash pipenv requirements [OPTIONS] ``` #### Options | Option | Description | |--------|-------------| | `--dev` | Include development packages | | `--dev-only` | Only include development packages | | `--hash` | Include package hashes | | `--exclude-markers` | Exclude PEP 508 markers | | `--categories TEXT` | Include packages from specified categories | #### Examples Generate requirements.txt: ```bash $ pipenv requirements > requirements.txt ``` Include development packages: ```bash $ pipenv requirements --dev > requirements-dev.txt ``` Include hashes: ```bash $ pipenv requirements --hash > requirements.txt ``` ### scripts Lists scripts defined in the current environment config. ```bash pipenv scripts [OPTIONS] ``` #### Examples List available scripts: ```bash $ pipenv scripts ``` ### open Opens a Python module in your editor. ```bash pipenv open [OPTIONS] MODULE ``` #### Examples Open the requests module: ```bash $ pipenv open requests ``` Open with a specific editor: ```bash $ EDITOR=code pipenv open requests ``` ### verify Verifies that the Pipfile.lock is up-to-date with the Pipfile. ```bash pipenv verify [OPTIONS] ``` #### Examples Verify lock file integrity: ```bash $ pipenv verify ``` ## Environment Variables Pipenv's behavior can be customized through environment variables. Here are some commonly used ones: | Variable | Description | |----------|-------------| | `PIPENV_VENV_IN_PROJECT` | Create virtualenv in project directory | | `PIPENV_IGNORE_VIRTUALENVS` | Ignore active virtualenvs | | `PIPENV_PIPFILE` | Custom Pipfile location | | `PIPENV_DOTENV_LOCATION` | Custom .env file location | | `PIPENV_CACHE_DIR` | Custom cache directory | | `PIPENV_TIMEOUT` | Timeout for pip operations | | `PIPENV_SKIP_LOCK` | Skip lock file generation | | `PIPENV_PYPI_MIRROR` | PyPI mirror URL | | `PIPENV_MAX_DEPTH` | Maximum depth for dependency resolution | | `PIPENV_DONT_LOAD_ENV` | Don't load .env files | For a complete list, see the [Configuration](configuration.md) page. ## Command Relationships Understanding how Pipenv commands relate to each other can help you use them more effectively: - `install`: Adds packages to Pipfile and updates Pipfile.lock - `lock`: Updates Pipfile.lock without installing packages - `sync`: Installs packages from Pipfile.lock without modifying it - `update`: Combines `lock` and `sync` (or `upgrade` and `sync` for specific packages) - `upgrade`: Updates Pipfile.lock for specific packages without installing them - `uninstall`: Removes packages from virtualenv and Pipfile - `clean`: Removes packages from virtualenv that aren't in Pipfile.lock - `remove`: Deletes the entire virtualenv for the current project ## Best Practices ### CI/CD Pipelines For continuous integration and deployment: ```bash # Verify the lock file is up-to-date $ pipenv verify # Install dependencies $ pipenv sync --dev # Run tests $ pipenv run pytest ``` ### Production Deployment For production environments: ```bash # Install only production dependencies $ pipenv install --deploy # Or for systems without virtualenv support $ pipenv install --system --deploy ``` ### Development Workflow For daily development: ```bash # Install dependencies including development packages $ pipenv install --dev # Activate environment $ pipenv shell # Run your application $ python app.py ``` ## Troubleshooting ### Common Issues - **Command not found**: Ensure Pipenv is installed and in your PATH - **Pipfile not found**: Run commands from your project directory - **Lock file out of date**: Run `pipenv lock` to update it - **Dependency conflicts**: Try `pipenv lock --clear` to clear the cache For more troubleshooting tips, see the [Troubleshooting](troubleshooting.md) guide. ## Conclusion Pipenv's CLI provides a comprehensive set of commands for managing Python dependencies and virtual environments. By understanding these commands and their options, you can streamline your Python development workflow and ensure consistent, reproducible environments. --- ## File: docs/commands.md # Pipenv Commands Reference This document provides a comprehensive reference for all Pipenv commands, including detailed explanations, options, and practical examples. ## Core Commands Overview | Command | Description | |---------|-------------| | `install` | Install packages or create/update virtual environment | | `uninstall` | Remove packages from virtual environment and Pipfile | | `lock` | Generate Pipfile.lock with dependencies and hashes | | `sync` | Install packages from Pipfile.lock without modifying the lockfile | | `update` | Update dependencies and install them (lock + sync) | | `upgrade` | Update the lock of specified dependencies without installing | | `check` | Check for security vulnerabilities and PEP 508 compliance (use `--scan` for enhanced scanning) | | `shell` | Spawn a shell within the virtual environment | | `run` | Run a command within the virtual environment | | `graph` | Display dependency graph information | | `remove` | Remove the virtualenv for the current project | | `clean` | Remove packages not specified in Pipfile.lock | | `verify` | Verify the Pipfile.lock hash is up-to-date | | `requirements` | Generate a requirements.txt from Pipfile.lock | | `scripts` | List scripts defined in the environment | | `open` | Open a Python module in your editor | ## install The `install` command is used for installing packages into the Pipenv virtual environment and updating your Pipfile and Pipfile.lock. ### Basic Usage ```bash $ pipenv install [package_name] ``` When run without arguments, `pipenv install` will create a virtual environment if one doesn't exist, and install all packages specified in the Pipfile. When a package name is provided, it will install that package, add it to the Pipfile, and update the Pipfile.lock. ### Examples Install a specific package: ```bash $ pipenv install requests ``` Install a package with version constraint: ```bash $ pipenv install "requests>=2.20.0" ``` Install a package from a Git repository: ```bash $ pipenv install -e git+https://github.com/requests/requests.git@master#egg=requests ``` Install a package as a development dependency: ```bash $ pipenv install pytest --dev ``` Install packages from a requirements.txt file: ```bash $ pipenv install -r requirements.txt ``` ### Options | Option | Description | |--------|-------------| | `--dev` | Install both development and default packages from Pipfile | | `--categories` | Install packages to the specified category groups | | `--deploy` | Abort if Pipfile.lock is out-of-date | | `--ignore-pipfile` | Install from Pipfile.lock, ignoring Pipfile | | `--system` | Install to system Python instead of virtual environment | | `--python` | Specify which Python version to use | | `--requirements, -r` | Import a requirements.txt file | | `--extra-pip-args` | Pass additional arguments to pip | ### Important Note Prior to Pipenv 2024, the `install` command would relock the lock file every time it was run. Based on user feedback, this behavior was changed so that `install` only updates the lock when adding or changing a package. To relock the entire set of Pipfile specifiers, use `pipenv lock`. ### --ignore-pipfile vs sync The `--ignore-pipfile` flag is similar to `pipenv sync`, but with an important difference: - **`pipenv install --ignore-pipfile`**: Installs packages from `Pipfile.lock`, ignoring the `Pipfile`. However, it may still attempt to re-lock your dependencies unless you also use the `--deploy` flag. - **`pipenv sync`**: Installs packages exactly as specified in `Pipfile.lock` without ever attempting to re-lock. This is considered an atomic operation. **For deployment scenarios, `pipenv sync` is the recommended command** because it guarantees that no modifications will be made to your lock file. ```bash # Development: may re-lock if needed $ pipenv install --ignore-pipfile # Production: never re-locks, fails if lock is out of date $ pipenv install --ignore-pipfile --deploy # Production (recommended): atomic install from lock file $ pipenv sync ``` ## sync The `sync` command installs dependencies from the Pipfile.lock without making any changes to the lockfile. This is useful for deployment scenarios where you want to ensure exact package versions are installed. ### Basic Usage ```bash $ pipenv sync ``` ### Examples Install only default packages: ```bash $ pipenv sync ``` Install both default and development packages: ```bash $ pipenv sync --dev ``` Install all categories (default, dev, and any custom categories): ```bash $ pipenv sync --all ``` Install specific package categories: ```bash $ pipenv sync --categories="tests,docs" ``` ### Options | Option | Description | |--------|-------------| | `--dev` | Install both development and default packages | | `--all` | Install packages from all categories defined in the Pipfile | | `--categories` | Install packages from specified category groups | ## uninstall The `uninstall` command removes packages from your virtual environment and Pipfile. ### Basic Usage ```bash $ pipenv uninstall [package_name] ``` ### Examples Uninstall a specific package: ```bash $ pipenv uninstall requests ``` Uninstall multiple packages: ```bash $ pipenv uninstall requests pytest ``` Uninstall all packages: ```bash $ pipenv uninstall --all ``` Uninstall all development packages: ```bash $ pipenv uninstall --all-dev ``` Uninstall a dev package using --dev flag: ```bash $ pipenv uninstall ruff --dev ``` ### Options | Option | Description | |--------|-------------| | `--all` | Remove all packages from virtual environment | | `--all-dev` | Remove all development packages | | `--dev` | Uninstall package from dev-packages section | | `--categories` | Specify which categories to uninstall from | | `--skip-lock` | Don't update Pipfile.lock after uninstalling | ## lock The `lock` command generates a Pipfile.lock file, which contains all dependencies (including sub-dependencies) with their exact versions and hashes. ### Basic Usage ```bash $ pipenv lock ``` ### Examples Generate a lockfile including pre-release versions: ```bash $ pipenv lock --pre ``` Generate a lockfile for a specific Python version: ```bash $ pipenv lock --python 3.9 ``` ### Options | Option | Description | |--------|-------------| | `--pre` | Allow pre-releases to be pinned | | `--clear` | Clear the dependency cache | | `--python` | Specify which Python version to use for resolution | | `--categories` | Lock specified categories only | ## update The `update` command runs `lock` when no packages are specified, or `upgrade` for specific packages, and then runs `sync` to install the updated packages. ### Basic Usage ```bash $ pipenv update [package_name] ``` ### Examples Update all packages: ```bash $ pipenv update ``` Update specific packages: ```bash $ pipenv update requests pytest ``` Check for outdated packages without updating: ```bash $ pipenv update --outdated ``` ### Options | Option | Description | |--------|-------------| | `--outdated` | List out-of-date dependencies | | `--dev` | Update development packages | | `--categories` | Update packages in specified categories | ## upgrade The `upgrade` command updates the lock file for specified dependencies and their sub-dependencies, but does not install the updated packages. ### Basic Usage ```bash $ pipenv upgrade [package_name] ``` ### Examples Upgrade a specific package in the lock file: ```bash $ pipenv upgrade requests ``` Upgrade multiple packages: ```bash $ pipenv upgrade requests pytest ``` ### Options | Option | Description | |--------|-------------| | `--dev` | Upgrade development packages | | `--categories` | Upgrade packages in specified categories | ## check The `check` command checks for security vulnerabilities in your dependencies and verifies that your environment meets PEP 508 requirements. ### Basic Usage ```bash $ pipenv check ``` ### Examples Check with a specific vulnerability database: ```bash $ pipenv check --db /path/to/db ``` Check with a specific output format: ```bash $ pipenv check --output json ``` ### Options | Option | Description | |--------|-------------| | `--db` | Path or URL to a PyUp Safety vulnerabilities database | | `--ignore, -i` | Ignore specified vulnerability | | `--output` | Specify output format (screen, text, json, bare) | | `--key` | Safety API key from PyUp.io | | `--use-installed` | Use installed packages instead of lockfile | | `--categories` | Check packages in specified categories | | `--auto-install` | Automatically install safety if not already installed | | `--scan` | Enable the newer version of the check command with improved functionality. | **Note**: The legacy check implementation is deprecated and will be removed in v2027.0.0. Use `pipenv audit` for vulnerability scanning, or use `pipenv check --scan` to run Safety's newer scan command during the transition. ## run The `run` command executes a command within the context of the virtual environment. ### Basic Usage ```bash $ pipenv run [command] ``` ### Examples Run a Python script: ```bash $ pipenv run python main.py ``` Run a test suite: ```bash $ pipenv run pytest ``` Run a custom script defined in Pipfile: ```bash $ pipenv run start ``` ## shell The `shell` command spawns a shell within the virtual environment, allowing you to interact with your installed packages. ### Basic Usage ```bash $ pipenv shell ``` ### Examples Activate the shell with a specific shell program: ```bash $ pipenv shell --fancy ``` ### Options | Option | Description | |--------|-------------| | `--fancy` | Use a fancy shell activation method | ## graph The `graph` command displays a dependency graph of your installed packages. ### Basic Usage ```bash $ pipenv graph ``` ### Examples Show a dependency graph with reverse dependencies: ```bash $ pipenv graph --reverse ``` ### Options | Option | Description | |--------|-------------| | `--bare` | Output graph in bare format | | `--json` | Output graph in JSON format | | `--json-tree` | Output graph as JSON tree | | `--reverse` | Reversed dependency graph | ## requirements The `requirements` command generates a requirements.txt file from your Pipfile.lock. ### Basic Usage ```bash $ pipenv requirements ``` ### Examples Generate requirements with hashes: ```bash $ pipenv requirements --hash ``` Generate requirements for development packages only: ```bash $ pipenv requirements --dev-only ``` Generate requirements for specific categories: ```bash $ pipenv requirements --categories="tests,docs" ``` ### Options | Option | Description | |--------|-------------| | `--dev` | Include development packages | | `--dev-only` | Only include development packages | | `--hash` | Include package hashes | | `--exclude-markers` | Exclude PEP 508 markers | | `--categories` | Include packages from specified categories | ## remove The `remove` command deletes the virtualenv associated with the current project. ### Basic Usage ```bash $ pipenv remove ``` ### Examples Remove the current project's virtualenv: ```bash $ pipenv remove Removing virtualenv (/home/user/.local/share/virtualenvs/myproject-abc123)... ``` You can then recreate it with `pipenv install`. > **Note**: The legacy `pipenv --rm` flag performs the same action but is deprecated and will be removed in a future release. Use `pipenv remove` instead. ## clean The `clean` command uninstalls all packages not specified in Pipfile.lock. ### Basic Usage ```bash $ pipenv clean ``` ### Examples Dry run to see what would be removed: ```bash $ pipenv clean --dry-run ``` ### Options | Option | Description | |--------|-------------| | `--dry-run` | Show what would be removed without removing | ## verify The `verify` command checks that the Pipfile.lock is up-to-date with the Pipfile. ### Basic Usage ```bash $ pipenv verify ``` This command is useful in CI/CD pipelines to ensure that the lock file is synchronized with the Pipfile before deployment. ## scripts The `scripts` command lists scripts defined in the current environment configuration. ### Basic Usage ```bash $ pipenv scripts ``` ## open The `open` command opens a Python module in your editor. ### Basic Usage ```bash $ pipenv open [module_name] ``` ### Examples Open the requests module in your editor: ```bash $ pipenv open requests ``` **Note**: This command uses the `EDITOR` environment variable to determine which editor to use. --- ## File: docs/configuration.md # Pipenv Configuration This document covers the various ways to configure Pipenv's behavior through environment variables, configuration files, and command-line options. ## Environment Variables Pipenv can be customized through environment variables, which is particularly useful for CI/CD pipelines, team-wide settings, or personal preferences. ### Setting Environment Variables #### Temporary (Session-Only) ```bash # Unix/Linux/macOS $ export PIPENV_VENV_IN_PROJECT=1 $ pipenv install # Windows (Command Prompt) > set PIPENV_VENV_IN_PROJECT=1 > pipenv install # Windows (PowerShell) > $env:PIPENV_VENV_IN_PROJECT=1 > pipenv install ``` #### Permanent For Unix/Linux/macOS, add to your shell profile (e.g., `~/.bashrc`, `~/.zshrc`): ```bash export PIPENV_VENV_IN_PROJECT=1 ``` For Windows, set through System Properties > Environment Variables. ### Boolean Options To enable boolean options, set the variable to a true value: `"1"`, `"true"`, `"yes"`, or `"on"`. To disable a boolean option, set it to a false value: `"0"`, `"false"`, `"no"`, or `"off"`. ### Available Environment Variables #### Virtual Environment | Variable | Description | Default | |----------|-------------|---------| | `PIPENV_VENV_IN_PROJECT` | Create virtualenv in project directory | `0` | | `PIPENV_IGNORE_VIRTUALENVS` | Ignore active virtualenvs | `0` | | `PIPENV_CUSTOM_VENV_NAME` | Use custom virtualenv name | None | | `PIPENV_VIRTUALENV` | Path to virtualenv executable | Detected from system | | `PIPENV_PYTHON` | Path to Python executable | Detected from system | | `PIPENV_DEFAULT_PYTHON_VERSION` | Default Python version to use | System default | | `VIRTUAL_ENV` | Current active virtualenv | None | #### Installation and Dependencies | Variable | Description | Default | |----------|-------------|---------| | `PIPENV_INSTALL_DEPENDENCIES` | Install package dependencies | `1` | | `PIPENV_RESOLVE_VCS` | Resolve VCS dependencies | `1` | | `PIPENV_SKIP_LOCK` | Skip lock when installing | `0` | | `PIPENV_PYPI_MIRROR` | PyPI mirror URL | None | | `PIPENV_MAX_DEPTH` | Maximum depth for dependency resolution | `10` | | `PIPENV_TIMEOUT` | Timeout for pip operations | `15` | | `PIPENV_INSTALL_TIMEOUT` | Timeout for package installation | `900` | #### File Locations | Variable | Description | Default | |----------|-------------|---------| | `PIPENV_PIPFILE` | Custom Pipfile location | `./Pipfile` | | `PIPENV_CACHE_DIR` | Custom cache directory | `~/.cache/pipenv` | | `PIPENV_DOTENV_LOCATION` | Custom .env file location | `./.env` | #### Behavior | Variable | Description | Default | |----------|-------------|---------| | `PIPENV_DONT_LOAD_ENV` | Don't load .env files | `0` | | `PIPENV_DONT_USE_PYENV` | Don't use pyenv | `0` | | `PIPENV_DONT_USE_ASDF` | Don't use asdf | `0` | | `PIPENV_SHELL_FANCY` | Use fancy shell | `0` | | `PIPENV_NOSPIN` | Disable spinner animation | `0` | | `PIPENV_QUIET` | Quiet mode | `0` | | `PIPENV_VERBOSE` | Verbose mode | `0` | | `PIPENV_YES` | Yes to all prompts | `0` | | `PIPENV_IGNORE_PIPFILE` | Ignore Pipfile, use only lock | `0` | | `PIPENV_REQUESTS_TIMEOUT` | Timeout for HTTP requests | `10` | | `PIPENV_CLEAR` | Clear caches on run | `0` | | `PIPENV_SITE_PACKAGES` | Enable site-packages for virtualenv | `0` | #### Security and Credentials | Variable | Description | Default | |----------|-------------|---------| | `PIPENV_PYUP_API_KEY` | PyUp.io API key for security checks | None | | `PIPENV_KEYRING_PROVIDER` | Keyring provider for credential lookup (`auto`, `disabled`, `import`, `subprocess`) | None (pip default: `auto`) | #### Runtime Environment Variables These variables are automatically set by Pipenv when running `pipenv run` or `pipenv shell`: | Variable | Description | |----------|-------------| | `PIPENV_ACTIVE` | Set to `1` when inside a Pipenv-managed environment | | `PIPENV_PROJECT_DIR` | Path to the project root directory (where Pipfile is located) | | `VIRTUAL_ENV` | Path to the virtualenv directory | The `PIPENV_PROJECT_DIR` variable is particularly useful for scripts that need to reference project-relative paths regardless of the current working directory: ```toml # Pipfile [scripts] test = "pytest $PIPENV_PROJECT_DIR/tests" mypy = "mypy --config-file=$PIPENV_PROJECT_DIR/mypy.ini $PIPENV_PROJECT_DIR/src" ``` ### Examples #### Store virtualenvs in the project directory ```bash export PIPENV_VENV_IN_PROJECT=1 ``` This creates a `.venv` directory in your project, making it easier to manage and find the virtualenv. #### Use a custom Python version by default ```bash export PIPENV_DEFAULT_PYTHON_VERSION=3.10 ``` This sets Python 3.10 as the default when creating new environments. #### Skip lock file generation during development ```bash export PIPENV_SKIP_LOCK=1 ``` This speeds up installation during development, but should not be used in production environments. #### Use a PyPI mirror ```bash export PIPENV_PYPI_MIRROR=https://mirrors.aliyun.com/pypi/simple/ ``` This is useful in regions where accessing the official PyPI might be slow. #### Increase timeout for large packages ```bash export PIPENV_INSTALL_TIMEOUT=1800 ``` This increases the timeout to 30 minutes for installing large packages. ## Configuration with pip Pipenv uses pip under the hood, so you can also use pip's configuration options. These can be set through: 1. Environment variables (e.g., `PIP_TIMEOUT`, `PIP_INDEX_URL`) 2. pip configuration files (`pip.conf` or `pip.ini`) ### Common pip Environment Variables | Variable | Description | |----------|-------------| | `PIP_INDEX_URL` | Base URL of the Python Package Index | | `PIP_EXTRA_INDEX_URL` | Additional index URLs | | `PIP_TRUSTED_HOST` | Mark a host as trusted | | `PIP_RETRIES` | Number of retries for network operations | | `PIP_TIMEOUT` | Timeout for HTTP requests | | `PIP_DEFAULT_TIMEOUT` | Default timeout for HTTP requests | | `PIP_FIND_LINKS` | Additional locations to find packages | | `PIP_NO_CACHE_DIR` | Disable the cache | | `PIP_CACHE_DIR` | Cache directory | ### Examples #### Using a private package index ```bash export PIP_INDEX_URL=https://private-repo.example.com/simple export PIP_TRUSTED_HOST=private-repo.example.com ``` #### Passing additional options to pip ```bash export PIP_INSTALL_OPTION="--no-deps" ``` #### Combining with Pipenv options ```bash export PIP_INDEX_URL=https://private-repo.example.com/simple export PIPENV_TIMEOUT=60 pipenv install requests ``` ## Project-Specific Configuration ### Using .env Files Pipenv automatically loads environment variables from `.env` files in your project directory. This is useful for project-specific settings: ``` # .env file PIPENV_VENV_IN_PROJECT=1 PIP_INDEX_URL=https://private-repo.example.com/simple ``` ### Custom .env Location You can specify a custom location for your `.env` file: ```bash export PIPENV_DOTENV_LOCATION=/path/to/custom/.env pipenv shell ``` ### Disabling .env Loading If you don't want Pipenv to load `.env` files: ```bash export PIPENV_DONT_LOAD_ENV=1 pipenv shell ``` ## Command-Line Options Many configuration options can also be set directly via command-line arguments, which take precedence over environment variables: ```bash pipenv install --python 3.9 --site-packages ``` ## Advanced Configuration ### Changing Cache Location The default cache location is `~/.cache/pipenv` on Unix/Linux/macOS and `%LOCALAPPDATA%\pipenv\Cache` on Windows. You can change this: ```bash export PIPENV_CACHE_DIR=/path/to/custom/cache ``` This is useful when: - You have limited space in your home directory - You want to share the cache across users - You're working in an environment with restricted permissions ### Network Configuration in Restricted Environments For environments with network restrictions: ```bash # Increase timeout for slow connections export PIPENV_TIMEOUT=60 export PIP_TIMEOUT=60 # Use a proxy export HTTP_PROXY=http://proxy.example.com:8080 export HTTPS_PROXY=http://proxy.example.com:8080 # Specify trusted hosts that don't need HTTPS export PIP_TRUSTED_HOST=internal-repo.example.com ``` ### CI/CD Pipeline Configuration For continuous integration environments: ```bash # Non-interactive mode export PIPENV_YES=1 export PIPENV_NOSPIN=1 export PIPENV_QUIET=1 # Fail if lock file is out of date pipenv install --deploy ``` ### Development vs. Production Settings #### Development ```bash # Development environment export PIPENV_VENV_IN_PROJECT=1 # Keep virtualenv with project export PIPENV_MAX_DEPTH=20 # Allow deeper dependency resolution ``` #### Production ```bash # Production environment export PIPENV_IGNORE_PIPFILE=1 # Use only the lock file export PIPENV_NOSPIN=1 # Disable spinner for cleaner logs pipenv install --deploy # Fail if lock file is out of date ``` ## Troubleshooting Configuration Issues ### Checking Current Configuration To see what environment variables Pipenv is using: ```bash pipenv --support ``` This shows all active Pipenv-related environment variables and their values. ### Common Configuration Problems #### Virtualenv Creation Fails If virtualenv creation fails, check: - `PIPENV_PYTHON` points to a valid Python executable - You have permissions to write to the virtualenv directory - `PIPENV_VENV_IN_PROJECT=1` if you're in a directory with restricted permissions #### Package Installation Timeouts If package installations time out: - Increase `PIPENV_TIMEOUT` and `PIPENV_INSTALL_TIMEOUT` - Check network connectivity to PyPI or your custom index - Consider using a PyPI mirror with `PIPENV_PYPI_MIRROR` #### Lock File Generation Issues If lock file generation fails: - Ensure there are no conflicting dependencies - Try `PIPENV_RESOLVE_VCS=0` if you have VCS dependencies causing issues - Increase `PIPENV_MAX_DEPTH` for complex dependency trees ## Best Practices 1. **Version Control**: Don't commit environment-specific settings to version control. Use `.env` files that are excluded via `.gitignore`. 2. **Documentation**: Document required environment variables in your project's README. 3. **Consistency**: Use the same configuration across development, testing, and production environments when possible. 4. **Minimal Configuration**: Only set the variables you need to change from defaults. 5. **Security**: Be careful with security-sensitive settings like API keys. Use environment variables rather than committing them to files. --- ## File: docs/credentials.md # Managing Credentials in Pipenv This guide covers best practices for securely managing credentials and authentication in Pipenv, including environment variables, private repositories, and security considerations. ## Credentials in Package Sources When working with private package repositories or authenticated services, you need to securely manage credentials in your Pipenv workflow. ### Environment Variable Expansion Pipenv automatically expands environment variables in your `Pipfile`, providing a secure way to inject credentials without storing them in version control. #### Basic Environment Variable Syntax You can use environment variables in your `Pipfile` using the following syntax: - `${VARIABLE_NAME}` - Standard syntax - `$VARIABLE_NAME` - Alternative syntax - `%VARIABLE_NAME%` - Windows-specific syntax (also supported on all platforms) #### Example: Private Repository Authentication ```toml [[source]] url = "https://${USERNAME}:${PASSWORD}@mypypi.example.com/simple" verify_ssl = true name = "private-pypi" ``` When Pipenv reads this `Pipfile`, it will replace `${USERNAME}` and `${PASSWORD}` with the values of the corresponding environment variables. #### Setting Environment Variables Before running Pipenv commands, set the required environment variables: **Linux/macOS:** ```bash $ export USERNAME="your-username" $ export PASSWORD="your-password" $ pipenv install ``` **Windows (Command Prompt):** ```cmd > set USERNAME=your-username > set PASSWORD=your-password > pipenv install ``` **Windows (PowerShell):** ```powershell > $env:USERNAME="your-username" > $env:PASSWORD="your-password" > pipenv install ``` ### Security Considerations Pipenv hashes your `Pipfile` *before* expanding environment variables and substitutes them again when installing from the lock file. This means: 1. Your credentials are never stored in the lock file 2. You don't need to commit any secrets to version control 3. Different developers can use different credentials with the same `Pipfile` ```{warning} While environment variables provide better security than hardcoded credentials, they are still accessible to any process running with the same user permissions. For highly sensitive credentials, consider using a dedicated secrets management solution. ``` ### URL Encoding for Special Characters If your credentials contain special characters, they must be URL-encoded according to [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986). For example, if your password is `p@ssw0rd!`, it should be encoded as `p%40ssw0rd%21`: ```toml [[source]] url = "https://${USERNAME}:${PASSWORD_ENCODED}@mypypi.example.com/simple" verify_ssl = true name = "private-pypi" ``` You can generate URL-encoded strings using Python: ```python import urllib.parse print(urllib.parse.quote("p@ssw0rd!", safe="")) # Output: p%40ssw0rd%21 ``` ## Credentials in Package Requirements Environment variables can also be used in package requirement specifiers, but with some limitations. ### VCS Repository Authentication For version control system (VCS) repositories that require authentication: ```toml [packages] requests = {git = "git://${USERNAME}:${PASSWORD}@private.git.com/psf/requests.git", ref = "2.22.0"} ``` ```{note} For VCS repositories, only the `${VAR_NAME}` syntax is supported. Neither `$VAR_NAME` nor `%VAR_NAME%` will work in this context. ``` ### Runtime vs. Install-time Expansion It's important to understand that environment variables are expanded at runtime, not when the `Pipfile` or `Pipfile.lock` is created. This means: 1. The entries in `Pipfile` and `Pipfile.lock` remain untouched 2. You need to have the environment variables set every time you run Pipenv commands 3. Different environments (development, CI/CD, production) can use different credentials ## Keyring Integration Pipenv supports keyring integration for authentication to private registries. With pip 23.1+, there are two approaches to configure keyring authentication. ### Modern Approach: System-Wide Keyring (pip >= 23.1) Starting with pip 23.1, you can install keyring system-wide (outside of your project virtualenv) and configure pip to use it via the subprocess provider. This is the recommended approach as it doesn't require installing keyring in every project. #### Setup Steps 1. **Install keyring globally** (using pip, pipx, or your system package manager): ```bash # Using pip (user install) $ pip install --user keyring keyrings.google-artifactregistry-auth # Or using pipx (recommended for CLI tools) $ pipx install keyring $ pipx inject keyring keyrings.google-artifactregistry-auth ``` 2. **Ensure keyring is on your PATH**: ```bash $ which keyring /home/user/.local/bin/keyring ``` 3. **Ensure virtualenv seeds pip >= 23.1** (one-time setup): ```bash $ virtualenv --upgrade-embed-wheels ``` 4. **Configure the keyring provider** using one of these methods: **Option A: Environment variable (recommended for Pipenv)**: ```bash $ export PIPENV_KEYRING_PROVIDER=subprocess ``` **Option B: pip configuration**: ```bash # Global configuration $ pip config set --global keyring-provider subprocess # Or user-level configuration $ pip config set --user keyring-provider subprocess ``` !!! note Setting `PIPENV_KEYRING_PROVIDER` ensures that keyring credentials are used during both dependency resolution and installation, even though Pipenv disables interactive pip input by default. This is particularly important for credential managers like Windows Credential Manager. 5. **Configure your index URL with the appropriate username**: - For Google Artifact Registry: use `oauth2accesstoken` as username - For Azure Artifacts: use `VssSessionToken` as username ```toml # Pipfile [[source]] url = "https://oauth2accesstoken@europe-python.pkg.dev/my-project/python/simple/" verify_ssl = true name = "private-gcp" ``` ### Legacy Approach: Keyring in Project Virtualenv Alternatively, you can install keyring directly in your project's virtual environment. This approach works with all pip versions but requires installing keyring in every project. ### Google Cloud Artifact Registry Google Cloud Artifact Registry supports authentication via keyring: 1. Install the required packages: ```bash $ pipenv run pip install keyring keyrings.google-artifactregistry-auth ``` 2. Configure your `Pipfile`: ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [[source]] url = "https://europe-python.pkg.dev/my-project/python/simple" verify_ssl = true name = "private-gcp" [packages] flask = "*" private-test-package = {version = "*", index = "private-gcp"} ``` 3. If the keyring might ask for user input, you may need to disable input enforcement: ```toml [pipenv] disable_pip_input = false ``` Alternatively, if your keyring provider does not require user interaction (e.g. Windows Credential Manager), you can set `PIPENV_KEYRING_PROVIDER=subprocess` instead. ### Azure Artifact Registry For Azure Artifact Registry: 1. Install the required packages: ```bash $ pipenv run pip install keyring artifacts-keyring ``` 2. Configure your `Pipfile` similar to the Google Cloud example above, using `VssSessionToken` as the username in your index URL. ### Windows Credential Manager On Windows, pip can use credentials stored in Windows Credential Manager via the keyring library. To use these credentials with Pipenv: 1. **Install keyring globally**: ```powershell > pip install --user keyring ``` 2. **Set the keyring provider**: ```powershell > $env:PIPENV_KEYRING_PROVIDER="subprocess" ``` Or set it permanently via system environment variables. 3. **Store your credentials** in Windows Credential Manager (via Control Panel or `cmdkey`). 4. **Configure your `Pipfile`** with the private index URL (with or without the username): ```toml [[source]] url = "https://username@private-repo.example.com/simple" verify_ssl = true name = "private" ``` ### AWS CodeArtifact For AWS CodeArtifact: 1. Install the required packages: ```bash $ pipenv run pip install keyring keyrings.codeartifact ``` 2. Configure your `Pipfile` with the appropriate AWS CodeArtifact URL. ## Best Practices for Credential Management ### Use Environment Variables Always use environment variables instead of hardcoding credentials: ```toml # Good url = "https://${USERNAME}:${PASSWORD}@private-repo.example.com/simple" # Bad url = "https://actual-username:actual-password@private-repo.example.com/simple" ``` ### Store Credentials in .env Files For local development, store credentials in `.env` files that are not committed to version control: ``` # .env USERNAME=your-username PASSWORD=your-password ``` Add `.env` to your `.gitignore`: ``` # .gitignore .env ``` Pipenv will automatically load variables from `.env` files when you run commands. ### Provide Templates Include a template `.env.example` file in your repository: ``` # .env.example USERNAME= PASSWORD= ``` This helps other developers understand which environment variables they need to set. ### Use Different Credentials for Different Environments Maintain separate credential sets for different environments: ``` # .env.development USERNAME=dev-username PASSWORD=dev-password # .env.production USERNAME=prod-username PASSWORD=prod-password ``` Use them with: ```bash $ PIPENV_DOTENV_LOCATION=.env.development pipenv install ``` ### Rotate Credentials Regularly Regularly rotate your credentials to minimize the impact of potential leaks: 1. Generate new credentials 2. Update your environment variables or `.env` files 3. Revoke the old credentials ### Consider Credential Managers For team environments, consider using dedicated credential management solutions: - [HashiCorp Vault](https://www.vaultproject.io/) - [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) - [Google Secret Manager](https://cloud.google.com/secret-manager) - [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/) These can be integrated into your workflow to provide secure, centralized credential management. ## CI/CD Integration ### GitHub Actions For GitHub Actions, use secrets to store credentials: ```yaml name: Python Package on: [push, pull_request] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: '3.10' - name: Install dependencies env: USERNAME: ${{ secrets.PYPI_USERNAME }} PASSWORD: ${{ secrets.PYPI_PASSWORD }} run: | pip install pipenv pipenv install --dev ``` ### GitLab CI For GitLab CI, use CI/CD variables: ```yaml stages: - test test: stage: test image: python:3.10 variables: USERNAME: ${PYPI_USERNAME} PASSWORD: ${PYPI_PASSWORD} script: - pip install pipenv - pipenv install --dev - pipenv run pytest ``` ### Jenkins For Jenkins, use credentials binding: ```groovy pipeline { agent { docker { image 'python:3.10' } } stages { stage('Build') { steps { withCredentials([ string(credentialsId: 'pypi-username', variable: 'USERNAME'), string(credentialsId: 'pypi-password', variable: 'PASSWORD') ]) { sh 'pip install pipenv' sh 'pipenv install --dev' sh 'pipenv run pytest' } } } } } ``` ## Troubleshooting ### Common Issues #### Environment Variables Not Being Expanded If environment variables aren't being expanded: 1. Verify the variables are set correctly: ```bash $ echo $USERNAME $ echo $PASSWORD ``` 2. Check the syntax in your `Pipfile`: - Use `${VARIABLE_NAME}` for the most reliable expansion - Ensure there are no typos in variable names 3. Try setting the variables directly in the command: ```bash $ USERNAME=user PASSWORD=pass pipenv install ``` #### Authentication Failures If you're experiencing authentication failures: 1. Verify your credentials work outside of Pipenv: ```bash $ curl -u "${USERNAME}:${PASSWORD}" https://private-repo.example.com/simple ``` 2. Check for special characters that might need URL encoding 3. Ensure your credentials have the necessary permissions #### Keychain Integration Issues If keychain integration isn't working: 1. Verify the keychain packages are installed: ```bash $ pipenv run pip list | grep keyring ``` 2. Check if the keychain is properly configured for your system 3. Try disabling pip input enforcement: ```toml [pipenv] disable_pip_input = false ``` ## Security Considerations ### Credential Leakage Risks Be aware of these common credential leakage risks: 1. **Command history**: Credentials passed directly in commands are stored in shell history 2. **Process listing**: Environment variables set on the command line may be visible in process listings 3. **Log files**: Debug logs might include expanded environment variables 4. **Core dumps**: Application crashes might include memory containing credentials ### Mitigating Risks To mitigate these risks: 1. Use `.env` files instead of setting variables on the command line 2. Limit access to environments where credentials are used 3. Use temporary credentials or tokens when possible 4. Implement the principle of least privilege for all credentials 5. Monitor for unauthorized access to your repositories ## Conclusion Properly managing credentials in Pipenv is essential for security and maintainability. By using environment variables, `.env` files, and following best practices, you can securely authenticate to private repositories and services without compromising sensitive information. Remember that credential management is an ongoing process that requires regular review and updates to maintain security. Always follow the principle of least privilege and rotate credentials regularly to minimize potential security risks. --- ## File: docs/diagnose.md # Diagnosing and Troubleshooting Pipenv Issues This guide provides comprehensive information on diagnosing and resolving common issues with Pipenv, including detailed troubleshooting steps, diagnostic commands, and best practices. ## Diagnostic Tools and Commands Pipenv includes several built-in tools to help diagnose issues with your environment and dependencies. ### The `--support` Flag The `--support` flag generates diagnostic information that can be helpful when reporting issues: ```bash $ pipenv --support ``` This command outputs: - Python version and path - Pipenv version - PIP version - System information - Environment variables - Pipenv configuration - Installed packages Include this output when filing issues on GitHub to help maintainers understand your environment. ### Verbose Mode For more detailed output during Pipenv operations, use the verbose flag: ```bash $ pipenv install --verbose ``` This shows detailed information about what Pipenv is doing, which can help identify where issues are occurring. ### Debug Mode For even more detailed debugging information, set the `PIPENV_DEBUG` environment variable: ```bash $ PIPENV_DEBUG=1 pipenv install ``` This enables debug-level logging, showing internal operations that can help diagnose complex issues. ## Common Issues and Solutions ### Dependency Resolution Problems #### Issue: Dependencies Could Not Be Resolved If Pipenv can't resolve your dependencies, try clearing the resolver cache: ```bash $ pipenv lock --clear ``` If that doesn't work, try manually deleting the cache directory: - **Linux/macOS**: `~/.cache/pipenv` - **Windows**: `%LOCALAPPDATA%\pipenv\pipenv\Cache` - **macOS (alternative)**: `~/Library/Caches/pipenv` #### Issue: Pre-release Versions Not Installing By default, Pipenv doesn't install pre-release versions. To enable them: ```bash # Command line flag $ pipenv install --pre package-name # Or in Pipfile # [pipenv] # allow_prereleases = true ``` #### Issue: Dependency Conflicts If you have conflicting dependencies: 1. Use `pipenv graph` to visualize your dependency tree 2. Look for packages with overlapping version requirements 3. Try relaxing version constraints in your Pipfile 4. Consider using custom package categories to separate conflicting dependencies ### Installation and Environment Issues #### Issue: Module Not Found Errors If you get "No module named X" errors: 1. Verify the package is installed: ```bash $ pipenv graph | grep package-name ``` 2. Check if you're running the command within the Pipenv environment: ```bash $ pipenv run python -c "import package_name" ``` 3. Ensure you're not mixing system packages with Pipenv: ```bash $ pipenv --rm # Remove the virtual environment $ pipenv install # Recreate it ``` #### Issue: Python Version Not Found If Pipenv can't find the specified Python version: 1. Check available Python versions: ```bash # If using pyenv $ pyenv versions # If using asdf $ asdf list python ``` 2. Install the required version: ```bash # With pyenv $ pyenv install 3.10.4 # With asdf $ asdf install python 3.10.4 ``` 3. Specify the Python version explicitly: ```bash $ pipenv --python 3.10 ``` #### Issue: Pipenv Doesn't Respect pyenv's Python Versions Pipenv by default uses the Python it was installed against. To use your current pyenv interpreter: ```bash $ pipenv --python $(pyenv which python) ``` ### Locale and Encoding Issues #### Issue: ValueError: unknown locale: UTF-8 This is a common issue on macOS due to a bug in locale detection: ```bash # Add to your shell configuration file (~/.bashrc, ~/.zshrc, etc.) export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 ``` You can change both the `en_US` and `UTF-8` parts to match your preferred language and encoding. #### Issue: /bin/pip: No such file or directory This may be related to locale settings. Apply the same fix as above. ### Lock File Issues #### Issue: Lock File Out of Date If your lock file is out of date: ```bash $ pipenv lock ``` #### Issue: Exception During Locking If an exception occurs during locking: ```bash $ pipenv lock --clear ``` This clears the cache and forces a fresh resolution of all dependencies. #### Issue: Hash Verification Failure If you encounter hash verification failures: 1. Update your lock file: ```bash $ pipenv lock ``` 2. If that doesn't work, try installing with: ```bash $ pipenv install --ignore-pipfile ``` 3. If you're still having issues, check for network problems or proxy settings that might be interfering with downloads. ### Virtual Environment Issues #### Issue: Virtual Environment Not Found If Pipenv can't find your virtual environment: 1. Check if it exists: ```bash $ pipenv --venv ``` 2. If it doesn't exist, create it: ```bash $ pipenv install ``` 3. If you've moved or renamed your project, remove the old environment and create a new one: ```bash $ pipenv --rm $ pipenv install ``` #### Issue: Shell Activation Problems If `pipenv shell` doesn't work correctly: 1. Try compatibility mode (the default): ```bash $ pipenv shell ``` 2. If that doesn't work, try fancy mode: ```bash $ pipenv shell --fancy ``` 3. If neither works, use `pipenv run` instead: ```bash $ pipenv run python ``` ### Package Index Issues #### Issue: Package Not Found If a package can't be found: 1. Verify the package name is correct 2. Check that the package exists on the specified index 3. Ensure you have proper authentication for private repositories 4. Try installing with verbose output: ```bash $ pipenv install package-name --verbose ``` #### Issue: Authentication Problems with Private Repositories If you're having authentication issues with private repositories: 1. Check your credentials 2. Ensure environment variables are properly set 3. Verify URL encoding for special characters in credentials 4. Test access to the repository outside of Pipenv ## Advanced Troubleshooting ### Debugging Dependency Resolution For complex dependency resolution issues: ```bash $ PIPENV_RESOLVER_DEBUG=1 pipenv install ``` This shows detailed information about the dependency resolution process, including which packages are being considered and why certain versions are being selected or rejected. ### Analyzing Lock File Changes To understand what changed in your lock file: ```bash $ git diff Pipfile.lock ``` Look for: - Version changes - New or removed dependencies - Changes in hashes - Changes in markers or requirements ### Inspecting the Virtual Environment To inspect the packages installed in your virtual environment: ```bash # List all installed packages $ pipenv run pip list # Show details for a specific package $ pipenv run pip show package-name # Check for outdated packages $ pipenv update --outdated ``` ### Checking for Conflicting Dependencies To identify conflicting dependencies: ```bash # Show the dependency graph $ pipenv graph # Show reverse dependencies (what depends on a package) $ pipenv graph --reverse ``` Look for packages that appear multiple times with different version constraints. ### Debugging Path and Environment Issues If you suspect path or environment issues: ```bash # Show the Python interpreter path $ pipenv --py # Show environment variables $ pipenv --envs # Run a command with verbose output $ PIPENV_VERBOSE=1 pipenv run python -c "import sys; print(sys.path)" ``` ## Preventive Measures ### Regular Maintenance Perform regular maintenance to prevent issues: 1. Keep Pipenv updated: ```bash $ pip install --user --upgrade pipenv ``` 2. Regularly update dependencies: ```bash $ pipenv update ``` 3. Scan for security vulnerabilities: ```bash $ pipenv scan ``` ### Best Practices for Avoiding Issues 1. **Use specific version constraints** for critical dependencies 2. **Commit both Pipfile and Pipfile.lock** to version control 3. **Use the `--deploy` flag** in production environments 4. **Regularly clean unused packages**: ```bash $ pipenv clean ``` 5. **Document your environment setup** in your project's README 6. **Use a consistent Python version** across development and production 7. **Set up CI/CD pipelines** to catch issues early ## Creating Reproducible Bug Reports When reporting issues to the Pipenv maintainers: 1. **Include the `--support` output**: ```bash $ pipenv --support ``` 2. **Provide a minimal reproducible example**: - Simplified Pipfile - Steps to reproduce - Expected vs. actual behavior 3. **Include relevant logs**: - Use `--verbose` or `PIPENV_DEBUG=1` - Include complete error messages 4. **Specify your environment**: - Operating system and version - Python version - Pipenv version ## Conclusion Diagnosing issues with Pipenv often involves understanding the underlying dependency resolution process, virtual environment management, and Python environment configuration. By using the diagnostic tools and following the troubleshooting steps in this guide, you can resolve most common Pipenv issues. Remember that Pipenv is a tool that combines pip, virtualenv, and Pipfile to simplify Python dependency management. Understanding how these components interact can help you diagnose and resolve issues more effectively. If you encounter persistent issues that aren't covered in this guide, consider reaching out to the Pipenv community through: - [GitHub Issues](https://github.com/pypa/pipenv/issues) - [PyPA Discussions](https://discuss.python.org/c/packaging/14) - [Stack Overflow](https://stackoverflow.com/questions/tagged/pipenv) --- ## File: docs/docker.md # Using Pipenv with Docker This guide provides comprehensive instructions for integrating Pipenv with Docker, including best practices, optimization techniques, and example configurations for different scenarios. ## Docker and Pipenv: Core Concepts Docker containers provide isolated, reproducible environments for applications, while Pipenv manages Python dependencies. When used together, they create a powerful workflow for Python application deployment. ### Why Use Pipenv with Docker? - **Dependency consistency**: Ensure the same packages are installed in development and production - **Reproducible builds**: Lock files guarantee identical environments across deployments - **Security**: Hash verification prevents supply chain attacks - **Simplified workflow**: Manage dependencies with a familiar tool inside containers ## Basic Docker Integration ### Simple Dockerfile Example Here's a basic Dockerfile that uses Pipenv: ```dockerfile FROM python:3.10-slim # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy Pipfile and Pipfile.lock COPY Pipfile Pipfile.lock ./ # Install dependencies RUN pipenv install --deploy --system # Copy application code COPY . . # Run the application CMD ["python", "app.py"] ``` This approach: 1. Starts with a Python base image 2. Installs Pipenv 3. Copies dependency files 4. Installs dependencies system-wide 5. Copies application code 6. Runs the application ### Key Flags for Docker Environments - `--system`: Installs packages to the system Python instead of creating a virtual environment - `--deploy`: Ensures the Pipfile.lock is up-to-date and fails if it isn't - `--ignore-pipfile`: Uses only the lock file for installation, ignoring the Pipfile ### System Installation vs. Virtual Environments in Docker Docker containers provide isolated environments, which raises the question: should you use `--system` to install packages globally, or create a virtual environment inside the container? **Using `--system` (Recommended for most Docker use cases)** - Simpler setup with no nested isolation - Smaller image size (no virtualenv overhead) - Packages are directly accessible without activation - Works well with `pipenv run` for executing scripts defined in Pipfile - Best for single-application containers following the "one process per container" principle **Using a Virtual Environment** - Provides an extra layer of isolation from system Python packages - Useful if your container runs multiple Python applications or system tools that depend on Python - Preferred when extending base images that have Python-based system utilities (e.g., `add-apt-repository` in Ubuntu) - Consider this approach if you use `apt-get install python3-*` packages alongside your application For most production containers that run a single Python application, `--system` is the simpler and more efficient choice. The concerns about breaking system Python tools (raised in older guidance) are less relevant when: - Your container runs only your application - You're using slim/minimal base images - You're not installing Python-based system packages via apt/yum ## Optimized Docker Builds ### Multi-Stage Builds Multi-stage builds create smaller, more secure images by separating the build environment from the runtime environment: ```dockerfile FROM python:3.10-slim AS builder # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Install dependencies RUN pipenv install --deploy --system FROM python:3.10-slim # Set working directory WORKDIR /app # Copy installed packages from builder stage COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages COPY --from=builder /usr/local/bin /usr/local/bin # Copy application code COPY . . # Run as non-root user RUN useradd -m appuser USER appuser # Run the application CMD ["python", "app.py"] ``` This approach: 1. Uses a builder stage to install dependencies 2. Copies only the necessary files to the final image 3. Runs the application as a non-root user 4. Results in a smaller, more secure image ### Layer Caching Optimization To take advantage of Docker's layer caching and speed up builds: ```dockerfile FROM python:3.10-slim # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files only COPY Pipfile Pipfile.lock ./ # Install dependencies RUN pipenv install --deploy --system # Copy application code (changes more frequently) COPY . . # Run the application CMD ["python", "app.py"] ``` This separates dependency installation from code copying, so dependencies are only reinstalled when Pipfile or Pipfile.lock change. ## Development vs. Production Configurations ### Development Dockerfile For development environments, you might want to include development dependencies: ```dockerfile FROM python:3.10-slim # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Install dependencies including development packages RUN pipenv install --dev --system # Copy application code COPY . . # Run the development server CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"] ``` ### Production Dockerfile For production, focus on security and minimal image size: ```dockerfile FROM python:3.10-slim AS builder # Install pipenv and dependencies RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Install only production dependencies RUN pipenv install --deploy --system FROM python:3.10-slim # Set working directory WORKDIR /app # Copy installed packages from builder stage COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages COPY --from=builder /usr/local/bin /usr/local/bin # Copy application code COPY . . # Create and use non-root user RUN useradd -m appuser USER appuser # Set production environment variables ENV PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 \ PYTHONPATH=/app # Run the application with gunicorn (for web applications) CMD ["gunicorn", "app:app", "--bind", "0.0.0.0:8000", "--workers", "4"] ``` ## Docker Compose Integration ### Basic Docker Compose Example ```yaml version: '3.8' services: web: build: . ports: - "8000:8000" volumes: - .:/app environment: - DATABASE_URL=postgresql://postgres:postgres@db:5432/app depends_on: - db db: image: postgres:14 volumes: - postgres_data:/var/lib/postgresql/data environment: - POSTGRES_PASSWORD=postgres - POSTGRES_USER=postgres - POSTGRES_DB=app volumes: postgres_data: ``` ### Development Environment with Hot Reload ```yaml version: '3.8' services: web: build: context: . dockerfile: Dockerfile.dev ports: - "8000:8000" volumes: - .:/app environment: - DEBUG=True - DATABASE_URL=postgresql://postgres:postgres@db:5432/app depends_on: - db command: python manage.py runserver 0.0.0.0:8000 db: image: postgres:14 volumes: - postgres_data:/var/lib/postgresql/data environment: - POSTGRES_PASSWORD=postgres - POSTGRES_USER=postgres - POSTGRES_DB=app volumes: postgres_data: ``` ## Advanced Docker Techniques ### Using Project-Local Virtual Environments For some workflows, you might want to use a project-local virtual environment: ```dockerfile FROM python:3.10-slim # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Create a project-local virtual environment ENV PIPENV_VENV_IN_PROJECT=1 RUN pipenv install --deploy # Copy application code COPY . . # Run the application using the virtual environment CMD ["./.venv/bin/python", "app.py"] ``` ### Handling Different Python Versions If your application requires a specific Python version: ```dockerfile FROM python:3.10-slim # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Install dependencies RUN pipenv install --deploy --system --python 3.10 # Copy application code COPY . . # Run the application CMD ["python", "app.py"] ``` ### Using Custom Package Indexes For private packages or custom indexes: ```dockerfile FROM python:3.10-slim # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Set environment variables for private repository authentication ARG PRIVATE_REPO_USERNAME ARG PRIVATE_REPO_PASSWORD ENV PIP_EXTRA_INDEX_URL=https://${PRIVATE_REPO_USERNAME}:${PRIVATE_REPO_PASSWORD}@private-repo.example.com/simple # Install dependencies RUN pipenv install --deploy --system # Copy application code COPY . . # Run the application CMD ["python", "app.py"] ``` ## Security Best Practices ### Running as Non-Root User Always run your application as a non-root user: ```dockerfile FROM python:3.10-slim # Install pipenv and dependencies RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Install dependencies RUN pipenv install --deploy --system # Copy application code COPY . . # Create and use non-root user RUN useradd -m appuser && \ chown -R appuser:appuser /app USER appuser # Run the application CMD ["python", "app.py"] ``` ### Handling Secrets Securely Use build arguments and environment variables for secrets: ```dockerfile FROM python:3.10-slim # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Use build arguments for secrets (only available during build) ARG API_KEY ENV API_KEY=${API_KEY} # Install dependencies RUN pipenv install --deploy --system # Copy application code COPY . . # Run the application CMD ["python", "app.py"] ``` Then build with: ```bash $ docker build --build-arg API_KEY=your-secret-key -t your-image . ``` For runtime secrets, use environment variables or Docker secrets. ### Scanning for Vulnerabilities Integrate security scanning into your Docker workflow: ```dockerfile FROM python:3.10-slim AS builder # Install pipenv RUN pip install pipenv # Set working directory WORKDIR /app # Copy dependency files COPY Pipfile Pipfile.lock ./ # Install dependencies RUN pipenv install --deploy --system # Scan for vulnerabilities RUN pipenv scan FROM python:3.10-slim # Set working directory WORKDIR /app # Copy installed packages from builder stage COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages COPY --from=builder /usr/local/bin /usr/local/bin # Copy application code COPY . . # Run the application CMD ["python", "app.py"] ``` ## CI/CD Integration ### GitHub Actions Example ```yaml name: Build and Push Docker Image on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v2 - name: Login to DockerHub uses: docker/login-action@v2 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - name: Build and push uses: docker/build-push-action@v4 with: context: . push: true tags: yourusername/yourapp:latest ``` ### GitLab CI Example ```yaml stages: - build - test - deploy build: stage: build image: docker:20.10.16 services: - docker:20.10.16-dind script: - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA . - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA test: stage: test image: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA script: - python -m pytest deploy: stage: deploy image: docker:20.10.16 services: - docker:20.10.16-dind script: - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest - docker push $CI_REGISTRY_IMAGE:latest ``` ## Troubleshooting ### Common Issues and Solutions #### Permission Denied Errors If you encounter permission issues: ```dockerfile # Add this to your Dockerfile RUN pip install --user pipenv ENV PATH="/root/.local/bin:${PATH}" ``` #### Package Installation Failures For packages with system dependencies: ```dockerfile FROM python:3.10-slim # Install system dependencies RUN apt-get update && \ apt-get install -y --no-install-recommends \ build-essential \ libpq-dev \ && rm -rf /var/lib/apt/lists/* # Install pipenv RUN pip install pipenv # Continue with your Dockerfile... ``` #### Slow Builds To speed up builds: ```dockerfile # Use a specific pip version RUN pip install --upgrade pip==22.2.2 pipenv==2022.8.5 # Use pip cache ENV PIP_CACHE_DIR=/var/cache/pip # Install with multiple workers RUN pipenv install --deploy --system --extra-pip-args="--use-feature=fast-deps" ``` ## Best Practices 1. **Use multi-stage builds** to create smaller, more secure images 2. **Separate dependency installation from code copying** to leverage Docker's layer caching 3. **Run applications as non-root users** to improve security 4. **Use `--deploy` flag** to ensure Pipfile.lock is up-to-date 5. **Use `--system` for single-application containers** to install dependencies system-wide, or use a virtual environment if your container runs multiple Python applications 6. **Include only necessary files** in your Docker image 7. **Set appropriate environment variables** like `PYTHONUNBUFFERED=1` for better logging 8. **Scan for vulnerabilities** as part of your build process 9. **Use build arguments** for build-time configuration 10. **Use environment variables or Docker secrets** for runtime configuration ## Conclusion Integrating Pipenv with Docker creates a powerful workflow for Python application deployment. By following the best practices and examples in this guide, you can create efficient, secure, and reproducible Docker images for your Python applications. Remember that Docker and Pipenv are both tools that help with reproducibility and dependency management. When used together correctly, they complement each other and provide a robust solution for Python application deployment.