How to Use This Guide
This is a guide on how to produce a reproducible and upgradeable environment for Python applications. A great deal of inspiration came from a pythonspeed.com article about this very topic.1 While the article is a wonderful resource, I needed more so I developed this guide.
You can use the table of contents to get a quick look at topics. Or you can skip ahead to Your Platform Setup which is the beginning of the walkthrough.
Ultimately, this guide is just that, a guide. The goal was to make the concepts easy to follow so that you can apply it to your use case. You may use a different platform than this guide but hopefully still benefit from the content.
Gotchas
- This guide uses
pipx
for parts of this guide, you may choose not to. - This guide use
mamba
in place ofconda
. - My platform may not be your platform so keep an eye out for incompatible commands.
Goals for the Environment
Reproducible: The environment can be reproduced across platforms. Upgradeable: Packages in the environment can be easily changed (e.g., version, new package, etc.)
Q&A
Why do you suggest using mamba
AND pip
?
Sometimes a package installed via mamba
breaks my environment. I have found using pip
as the exception to the rule has resolved most of these issues.
Where should I install conda-lock
?
Using pipx
to install conda-lock
allows you to have peace of mind about running the package outside of your mamba
environment (even your mamba
"base" environment). Alternatively, you could install it in your mamba
"base," but I ran into issues that way and this walkthrough will not cover that angle.
How can I find the version of my conda dependency?
A command I like to use is,
(base) $ mamba list | grep PACKAGE_NAME
Your Platform Setup
"Your platform" is the platform being used by you from a global perspective. Your environment is going to be the place we initialize the process of standardizing the environment for the goals listed above. By the end of this guide, you should be able to share the environment to another platform.
Global Dependencies
- pipx — "...help you install and run end-user applications written in Python"
- mamba (via mambaforge) — "Drop-in, faster conda..."
- pip — "...the package installer for Python"
- conda-lock — "...a lightweight library that can be used to generate fully reproducible lock files for conda environments."
- conda-forge — "A community-led collection of recipes, build infrastructure and distributions for the conda package manager."
Global Install Instructions
Find more detailed guides to installing these tools on their websites below.
* Install pipx
* Install conda-lock
via pipx
or install 'conda-lock[pip_support]'
if you expect the need for pip
* Install mamba
Workflow 1: Locking a New Project
In an ideal world, you can do environment management workflow before you write one line of code.
If you have mamba installed properly, your terminal should open like this,
(base) $
env-management-demo
.
(base) $ cd env-management-demo
mamba
environment for your project
(base) $ mamba create -n env-management-demo -c conda-forge python
(base)
changes to (env-management-demo)
.
(base) $ mamba activate env-management-demo
(env-management-demo) $
mamba
environment.yml
file. This step will generate a file that will need some editing in a moment.
(env-management-demo) $ mamba env export --from-history > environment.yml
environment.yml
will populate with your current environment parameters and some configurations. It may look different from mine but your channels must match your actual mamba
channels. I have noted which lines to remove.
(Optional) Check your channels
(env-management-demo) $ mamba config --show channels
The following shows a before and after of edits. You will see a # REMOVE
next to changes.
Before...
# environment.yml
name: env-management-demo # REMOVE
channels:
- conda-forge
dependencies:
- python
prefix: /Users/uname/miniconda3/envs/env-management-demo # REMOVE
# environment.yml
channels:
- conda-forge
dependencies:
- python
conda-forge
non-standard format.
# environment.yml
channels:
- conda-forge
dependencies:
- python
platforms: # This is non-standard and recommended by conda-lock.
- linux-64
- osx-64
- win-64
- osx-arm64 # For Apple Silicon, e.g. M1/M2
dependencies
sequence key unless you have to, for some reason, download a package using pip. Here is an example of that process:
(env-management-demo) $ mamba install -c conda-forge pandas
...
environment.yml
(indicated by # NEW
.)
# environment.yml
channels:
- conda-forge
dependencies:
- python
- pandas # NEW
platforms:
- linux-64
- osx-64
- win-64
- osx-arm64
conda-lock.yml
using the conda-lock
command.
(env-management-demo) $ conda-lock -f environment.yml
Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']...
...
conda-lock.yml
file.
(env-management-demo) $ mamba deactivate
(base) $
(base) $ conda-lock install -n env-locked
INFO:root:Downloading and Extracting Packages
INFO:root:numpy-1.24.1
INFO:root:pandas-1.5.3
...
(base) $ mamba activate env-locked
(env-locked) $
conda-lock.yml
file with other developers and have them run the code with the same conda-lock
command.
Workflow 2: Locking an Existing Project (TBD)
While this part of the tutorial is unfinished, you could get a pretty good idea of how to do this without this section. For now, you can continue to Reproduce. Good luck!
Reproduce
Once you have the environment locked into a conda-lock.yml
multi-platform file, you will be able to reproduce the environment (i.e., create and activate it) with the following command as long as the file is in your current directory:
(base) $ conda-lock install -n NAME
(base) $ mamba activate NAME
(NAME) $
With Docker in Mind
To minimize the size of the lockfile, you may want to render a single-platform lockfile for linux.
(base) $ conda-lock render -p linux-64
...
conda-linux-64.lock
. This lockfile can then be copied into an image and used from within the image.
mamba create -n my-locked-env --file conda-linux-64.lock
Updating the Environment
Currently the --update
flag for the conda-lock
command does not work as expected. For now, the way to update a package (e.g., upgrade, downgrade, remove, or add a package) is a workflow described like so:
1. Update your environment, for example, add the package matplotlib
via mamba
:
(env-management-demo) $ mamba install matplotlib
environment.yml
file:
# environment.yml
channels:
- conda-forge
dependencies:
- python
- pandas
- matplotlib # NEW
platforms:
- linux-64
- osx-64
- win-64
- osx-arm64
conda-lock
which will resolve the entire environent:
(env-management-demo) $ conda-lock -f environment.yml
Note that it is not necessary to add the package to your environment as seen in step 1 above, but I would reccomend doing this and testing your code before persisitng this change to conda-lock.yml
.
Adding a New Package via pip
Sometimes conda
does not have everything you need so you lean on pip
for a dependency. conda-lock
comes with pip
support if you install it with the extra like so:
(base) $ pipx install 'conda-lock[pip_support]'
...
conda
env with pip
.
(env-management-demo) $ pip install fastapi
...
environment.yml
.
# environment.yml
channels:
- conda-forge
dependencies:
- python
- pandas
- matplotlib=3.5
- pip: # NEW
- fastapi # NEW
platforms:
- linux-64
- osx-64
- win-64
- osx-arm64
conda-lock
command.
(env-management-demo) $ mamba deactivate
(base) $ conda-lock --update fastapi
Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']...
...
Debugging
Last Update: February 6, 2023
This section will be updated as issues are discovered. If issues are discovered, they should be checked against the conda-lock
repo linked below so contributors can be made aware.
conda-lock
giving vague AssertionError
Sometimes an error from conda-lock
is vague and hard to debug. I am logging some fixes I have discovered during development.
Are your channels consistent?
Are the channels you have set in mamba
and your environment.yml
file consistent? If not, make sure they are by editing your environment.yml
file to match. Once I have found that deleting the existing conda-lock.yml
file and trying again will resolve this issue.
Terms
platform : the OS being used for a task (e.g., MacOS, Windows, etc.)
References and Resources
- pythonspeed.com — Reproducible and upgradable Conda environments with conda-lock
- pypa.github.io — Global Python package manager
pipx
- docs.conda.io — Environment manager
conda
via miniconda - github.com — Cross-platform environment locker
conda-lock
- github.com — Recommended direct dependency update path
- github.com — Optional groups of dependencies #7502
Versions
My local versions for reference.
System Version: macOS 13.0 (22A380)
Kernel Version: Darwin 22.1.0
conda 22.11.1
mamba 1.1.0
pipx 1.1.0
conda-lock 1.4.0, installed using Python 3.11.0
zsh 5.8.1 (x86_64-apple-darwin22.0)