Conda: Importing and Exporting Environments

Overview

This article is intended to provide detailed information on importing and exporting Conda environments. The basic process is fairly straightforward, but the export process does not always capture everything required to make the environment work, so some diligence is required.

Additional complications can arise if the environment being exported contains Python packages (usually installed by pip) installed in the user’s /home/.local directory, or in any other location that is on the sys.path of the export environment.

Preparing for Export

It is important to take an “inventory” of those packages in the export user’s environment that are relevant to the Conda environment being exported and transferred. In the example that follows, we wish to export the working Conda environment “jupylink2” so it can be imported by another user.

From the Conda BASE environment:

(base) [jbjohnston@hpc-login-p01 ~]$ conda activate juplink2
(juplink2) [jbjohnston@hpc-login-p01 ~]$ conda list | tee myenv.conda
(juplink2) [jbjohnston@hpc-login-p01 ~]$ pip list | tee juplink2.pip

The commands above store the Conda and Pip packages used by the environment into two files. We can use these to help determine whether the requisite packages have been exported to the environment YAML file. Note that in some cases, a key package will be listed in the YAML file that when installed, will pull-in other dependencies. Thus, the match of the YAML file to these dump files may not be exactly one-to-one.

Exporting the Environment

To export the environment, we simply need to do the following from within the activated environment:

(juplink2) [jbjohnston@hpc-login-p01 ~]$ conda env export > juplink2_export.yml

This produces a YAML file (juplink2_export.yml) that we can now inspect and compare to our package dumps from the previous section. Presented below is an abridged version of the file:

juplink2_export.yml
name: juplink2
channels:
  - defaults
  - conda-forge
  - bioconda
  - menpo
  - https://conda.anaconda.org/menpo
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - anyio=4.2.0=py310h06a4308_0
  - argon2-cffi=21.3.0=pyhd3eb1b0_0
  - argon2-cffi-bindings=21.2.0=py310h7f8727e_0
  - asttokens=2.0.5=pyhd3eb1b0_0
  - async-lru=2.0.4=py310h06a4308_0
  - attrs=23.1.0=py310h06a4308_0
  - babel=2.11.0=py310h06a4308_0
  - beautifulsoup4=4.12.3=py310h06a4308_0
  - bleach=4.1.0=pyhd3eb1b0_0
    .............
 - pip:
      - contourpy==1.3.0
      - cycler==0.12.1
      - filelock==3.16.1
      - fonttools==4.54.1
      - fsspec==2024.9.0
      - kiwisolver==1.4.7
      - matplotlib==3.9.2
      - mpmath==1.3.0
      - mrx-link==2.0.0
      - numpy==2.1.1
    .............
      -prefix: /home/j/jbjohnston/.conda/envs/juplink2

You can now perform comparisons to determine whether any packages are missing. From experience, missing dependencies are generally going to most like occur under the “pip” section.

If there are dependencies missing from the YAML file, simply manually edit the file and add them. Then transfer the file to the importing user environment for testing.

Importing the Environment

Once you have transferred the exported environment to the new user, the environment can be imported as follows (from the base Conda environment):

(base) [hpctester02@hpc-login-p01 ~]$ conda env create -f juplink2_export.yml
(base) [hpctester02@hpc-login-p01 ~]$ conda activate juplink2
(juplink2) [hpctester02@hpc-login-p01 ~]$ 

Now test the environment, or if necessary, resolve any package conflicts until the environment is properly installed and can be activated.

---

CategoryHPC