Features

pip-compile-multi supports many options to customize compilation.

Requirements Directory

While it’s a common practice to put requirements files inside requirements directory, it’s not always the case. The directory can be overridden with this option:

-d, --directory TEXT   Directory path with requirements files

Requirements Files Extensions

By default pip-compile-multi compiles *.txt from *.in files. While this is a common naming pattern, each project can use it’s own:

-i, --in-ext TEXT      File extension of input files
-o, --out-ext TEXT     File extension of output files

Disable upgrades

When new dependencies are added it’s tempting to keep everything else the same. To recompile .txt keeping satisfying version use --no-upgrade:

--upgrade / --no-upgrade    Upgrade package version (default true)

The option has no effect if there are no existing .txt files.

Upgrade only selected packages

To upgrade only one package and keep everything else untouched, use following option:

-P, --upgrade-package TEXT  Only upgrade named package.
                            Can be supplied multiple times.

Under the hood it uses the same option of pip-compile and runs compilation only for files that have one of the passed packages.

This option implies --no-upgrade and takes precedence over --upgrade.

Thanks to Jonathan Rogers.

Use Cache

By default pip-compile-multi executes pip-compile with --rebuild flag. This flag clears any caches upfront and rebuilds from scratch. Such a strategy has proven to be more reliable in edge cases, but causes significant performance degradation.

Option --use-cache removes --rebuild flag from the call to pip-compile.

-u, --use-cache             Use pip-tools cache to speed up compilation.

Note

When using --use-cache, pip-compile-multi can run 10 times faster. But if you run into “magical” issues, try to rerun compilation without this flag first.

Compatible Releases

PEP-440 describes compatible release operator ~=. Sometimes it’s useful to have some of the dependencies pinned using this operator. For example, rapidly changing internal libraries. The format for this option is

-c, --compatible TEXT

where TEXT is a glob pattern for library name. This option can be supplied multiple times.

Generate hashes

Put package hash after pinned version for additional security. Format for this option is

-g, --generate-hashes TEXT  Input file name (base.in, requirements/test.in, etc)
                            that needs packages hashes.
                            Can be supplied multiple times.
                            For backwards compatibility can be short
                            environment name (base, test, etc.)

Example invocation:

$ pip-compile-multi -g base -g docs

Example output:

pip-tools==1.11.0         --hash=sha256:50288eb066ce66dbef5401a21530712a93c659fe480c7d8d34e2379300555fa1         --hash=sha256:ba427b68443466c389e3b0b0ef55f537ab39344190ea980dfebb333d0e6a50a3
first==2.0.1         --hash=sha256:3bb3de3582cb27071cfb514f00ed784dc444b7f96dc21e140de65fe00585c95e         --hash=sha256:41d5b64e70507d0c3ca742d68010a76060eea8a3d863e9b5130ab11a4a91aa0e         # via pip-tools

pip requires all packages to have hashes if at least one has it. pip-compile-multi will recursively propagate this option to all environments that are referencing or referenced by selected environments.

Allow Unsafe Packages

If your project depends on packages that include setuptools or other packages considered “unsafe” by pip-tools and you still wish to have them included in the resulting requirements files, you can pass this option to do so:

-s, --allow-unsafe          Whether or not to include 'unsafe' packages
                            in generated requirements files. Consult
                            pip-compile --help for more information

This is commonly used with –generate-hashes to avoid generating requirements files which cannot be installed.

Custom Header

pip-compile-multi adds a brief header into generated files. Override it with

-h, --header TEXT      File path with custom header text for generated files

Limit input files

By default pip-compile-multi compiles all .in files in requirements directory. To limit compilation to only a subset, use

-t, --only-path TEXT        Compile only for passed input file paths and
                            their references.
                            Can be supplied multiple times.

For example, to compile one file under Python2.7 and another under Python3.6, run:

$ virtual-env27/bin/pip-compile-multi -t requirements/deps27.in
Locking requirements/deps27.in to requirements/deps27.txt. References: []
$ virtual-env36/bin/pip-compile-multi -t requirements/deps36.in
Locking requirements/deps36.in to requirements/deps36.txt. References: []

Limit environments

Warning

This flag is deprecated and will be removed in future releases. Use Limit input files instead.

By default pip-compile-multi compiles all .in files in requirements directory. To limit compilation to only a subset, use

-n, --only-name TEXT        Compile only for passed environment names and
                            their references. Can be supplied multiple
                            times.

For example, to compile one file under Python2.7 and another under Python3.6, run:

$ virtual-env27/bin/pip-compile-multi -n deps27
Locking requirements/deps27.in to requirements/deps27.txt. References: []
$ virtual-env36/bin/pip-compile-multi -n deps36
Locking requirements/deps36.in to requirements/deps36.txt. References: []

Add index URL annotation

Control addition of --index-url options to the output files. Corresponds to pip-compile’s --emit-index-url / --no-emit-index-url flag.

The URL can be defined in the input file as in this example:

--index-url=https://pypi.tuna.tsinghua.edu.cn/simple/
six
click

This option is similar to extra_index_url. The difference is that URL is saved in the output files, which means that it doesn’t need to be configured in every environment.

--annotate-index / --no-annotate-index
                                Add index URL to generated files
                                (default false)

Note: the default behavior is not to add the index, i.e., --no-annotate-index.

Add additional index URL to search

Pip accepts URLs for additional package indexes through --extra-index-url. The same option can be passed to pip-compile-multi during compilation.

This option is similar to annotate_index. The difference is that the passed URL is not saved in any of the files, which is helpful if the URL contains private credentials.

--extra-index-url TEXT          Add additional package index URL to search
                                for package versions. Can be supplied
                                multiple times.

Warning

Using --extra-index-url option to search for packages that are not in the main repository (such as private packages) is unsafe, per a security vulnerability called dependency confusion: an attacker can claim the package on the public repository in a way that will ensure it gets chosen over the private package.

Use the --index-url option in pip’s configuration file or command line to specify the feed, overriding the default. Avoid the --extra-index-url option, which is additive and may lead to having multiple indexes.

See pip documentation for details.

Add trusted host annotation

Control addition of trusted hosts for index URLs to generated files. Trusted hosts can be defined in pip.conf file, input requirements file, or through an argument to pip command. Trusted hosts can have invalid HTTPS certificate, or use unencrypted HTTP protocol.

By default, trusted hosts are saved in the generated files. Pass --no-emit-trusted-host to remove it.

If no trusted hosts are defined, this flag doesn’t have any effect.

--emit-trusted-host / --no-emit-trusted-host
                                Add trusted host to generated files
                                (default true)

See also: annotate_index and extra_index_url options.

See pip-tools issue #382 for more details.

Autoresolve cross-file conflicts

Compile requirements file, that references all other files first, and than use it as a constraint.

--autoresolve/--no-autoresolve Automatically resolve
                               cross-file conflicts.

This strategy allows to resolve cross-file conflicts of two types:

1. Files FOO and BAR both have dependency PKG, but compile it to different versions.

2. FOO has PKG resolved to version 3. BAR references FOO, but also has another dependency, that constraints PKG to version 2.

By compiling all-including file (aka sink), that references both FOO and BAR first, pip-compile-multi generates conflict-free set of versions.

After that, this compiled file is passed as a constraint for compiling all requirements files.

As the last step, the sink file is compiled again preserving reference files and skipping duplicate packages.

Note

This feature works only if your project has a single requirements file, that references (directly or indirectly) all other files.

Backtracking resolver

Pip has an option to enable backtracking conflict resolution logic, which can automatically downgrade some dependencies to meet constraints from other packages. See also the a note on resolvers in pip-compile project.

--backtracking / --no-backtracking
                            Enable backtracking resolver. Translates to
                            pip-compile --resolver=backtracking option.

Skip constraints in comments of output files

When input files contain constraint references (e.g. ‘-c constraints.in’), pip-compile adds it to “via” comments. For example:

rsa==3.4.2
     # via
     #   -c constraints.txt
     #   google-auth

When this option is enabled that snippet will be converted to:

rsa==3.4.2
     # via google-auth

Saving two lines in .txt file.

--skip-constraints      Remove constraints from "via" comments.

This feature is especially useful in combination with Autoresolve cross-file conflicts.

Live output

Print debug output from pip-compile live. If the option is disabled (by default) the debug output is printed only in case of failure.

--live / --no-live              Print debug output from pip-compile live.

Check that pip-compile-multi was run after changes in .in file

pip-compile-multi adds a special line (before header) at the beginning of each generated file. This line contains a SHA1 hash of the .in file’s contents.

Command

$ pip-compile-multi verify
Verifying that requirements/base.txt was generated from requirements/base.in.
Success - comments match.
Verifying that requirements/test.txt was generated from requirements/test.in.
Success - comments match.
Verifying that requirements/local.txt was generated from requirements/local.in.
Success - comments match.

recalculates hashes for .in files and compares them with the stored values.

If verification fails, an error message is logged and exit code 1 is returned:

$ pip-compile-multi verify
Verifying that requirements/base.txt was generated from requirements/base.in.
Success - comments match.
Verifying that requirements/test.txt was generated from requirements/test.in.
FAILURE!
Expecting: # SHA1:c93d71964e14b04f3c8327d16dbc4d6b1bbc3b1d
Found:     # SHA1:6c2562322ca1bdc8309b08581a2aa4efbb5a4534
Verifying that requirements/local.txt was generated from requirements/local.in.
Success - comments match.

In big teams it might be a good idea to have this check in tox.ini:

[testenv:verify]
skipsdist = true
skip_install = true
deps = pip-compile-multi
commands = pip-compile-multi verify
whitelist_externals = pip-compile-multi