Conda and Python Troubleshooting

Contents

Overview

This document is intended to provide assistance for some commonly encountered problems when using Conda environments, and/or Python scripts inside or outside of Conda environments.

OpenSSL Errors

This problem generally manifests itself as an "UnsafeLegacyRenegotiation" failure when working in Conda environments, and can also affect "pip install" inside of Conda. The issue is caused when "openssl" packages > 1.1.1 are installed. This problem first appeared with openssl==3.x. So if you have a version of openssl installed > 1.1.1, the solution is to simply downgrade openssl in your Conda environment: 

conda install openssl=1.1.1

PLEASE NOTE: It is recommended that you install all of your Conda packages inside an active environment. Installing in the base environment can lead to version conflicts. If you choose to install inside the base environment, be aware that the package you install may not be compatible with something within the active environment.

Testing OpenSSL

Confirmatory tests can be performed to verify the problem/resolution using a few simple commands. From the node where you're attempting to run something inside your Conda environment, use one of the following: 

openssl s_client -connect huggingface.co:443

or, 

curl -v https://huggingface.co:443 > /dev/null

Examine output from one of the above commands to determine if the connection was verified, intact, and/or open.

SSL Verify Failed

Python Script Workaround

Another common problem is related to the verification of SSL certificates. This sometimes happens when the user's environment does not have the proper path established to the node's SSL certificates, and generally occurs when running "request" operations inside of a Python script. This error generally manifests itself as shown below: 

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)

The solution is generally to include a few statements inside of your Python script (whether run inside or outside of Conda environment). These are shown below: 

import os
os.environ["CURL_CA_BUNDLE"]="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"
os.environ["REQUESTS_CA_BUNDLE"]="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"

This imports the Python "os" package, and then specifies the paths needed for certificate verification.

Pip and Conda Export Workaround

This technique may be useful when working on an interactive session on a gpu or compute node. Upon logging into the node via an interactive session, activate your Conda environment and then export the following environmental variables: 

export CURL_CA_BUNDLE="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"
export REQUESTS_CA_BUNDLE="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"

Now proceed with your Conda or pip installs.

Openssl.cnf File Workaround

Although it is recommended to use the "miniconda3" modulefile provided by UTS on Matilda, some users will install their own version of Anaconda. In many of these cases, the user may experience problems with OpenSSL connectivity, including SSL verification failures. This is generally caused by newer versions of OpenSSL (i.e., > 1.1.1) being installed. Within the user's self-installed Anaconda directory, there is likely a file named "openssl.cnf" that is being used that may prevent even a downgrade to openssl=1.1.1. This problem can be remedied as follows:

  • Create your own "openssl.cnf" file with the contents shown below: 

openssl_conf = default_conf
[ default_conf ]
ssl_conf = ssl_sect
[ssl_sect]
system_default = ssl_default_sect
[ssl_default_sect]
Options = UnsafeLegacyRenegotiation
  • After noting the location of the file you just created, set the relevant environmental variable to that file's path: 

export OPENSSL_CONF=/path/to/my/file/openssl.cnf
  • Alternately, you can set OPENSSL_CONF temporarily by using it in-line with a conda install command: 

OPENSSL_CONF=`pwd`/openssl.cnf conda install openssl=1.1.1
  • If you used the "export" method, you can clear that value once openssl has been downgraded: 

unset OPENSSL_CONF
  • Now you should be able to use Conda normally.

Conda Create Solution

One simple way to avoid having to deal with the above SSL errors, is to make sure that when creating a new Conda environment it is setup with the correct version of OpenSSL. For example, to create a Conda environment with Python 3.11 we could use: 

conda create -n myenvironment python=3.11 openssl=1.1.1

This will ensure that our environment is created with a workable version of OpenSSL, and that all related packages are also compatible.

PLEASE NOTE: You will not be able to install a version of Python > 11.x. Versions of Python >= 12.x require OpenSSL version 3.x.

Conda Environment Problems

In many cases, users will find that they have installed previous Conda environments, but upon creating a new environment it fails to function as-expected. Conversely, some users find that they install a new Conda environment and it works properly, but environments installed previously are now broken. There can be many potential causes for these issues. A few of the more common ones are discussed in the sections below.

Conflicting Python Packages

The Importance of sys.path

When Python is invoked in a Conda environment, a "sys.path" is used to search for installed packages. An example of a typical sys.path is presented below for a Conda environment named "juplink2" (along with the command to check it): 

(base) [jbjohnston@hpc-login-p01 ~]$ conda activate juplink2
(juplink2) [jbjohnston@hpc-login-p01 ~]$ python -c "import sys; print('\n'.join(sys.path))"

/home/j/jbjohnston/.conda/envs/juplink2/lib/python310.zip
/home/j/jbjohnston/.conda/envs/juplink2/lib/python3.10
/home/j/jbjohnston/.conda/envs/juplink2/lib/python3.10/lib-dynload
/home/j/jbjohnston/.local/lib/python3.10/site-packages
/home/j/jbjohnston/.conda/envs/juplink2/lib/python3.10/site-packages

As noted above, there are a total of 5 directory locations in this sys.path - two (2) are located within the environment itself, and one is located in the "~/.local" directory. Now lets deactivate the environment and check the sys.path of the Conda base: 

(juplink2) [jbjohnston@hpc-login-p01 ~]$ conda deactivate
(base) [jbjohnston@hpc-login-p01 ~]$ python -c "import sys; print('\n'.join(sys.path))"

/cm/shared/apps/miniconda3/24.1.2-py311/lib/python311.zip
/cm/shared/apps/miniconda3/24.1.2-py311/lib/python3.11
/cm/shared/apps/miniconda3/24.1.2-py311/lib/python3.11/lib-dynload
/home/j/jbjohnston/.local/lib/python3.11/site-packages
/cm/shared/apps/miniconda3/24.1.2-py311/lib/python3.11/site-packages

In this case, we can see that the modulefile for miniconda3 maps to four (4) locations (these were added when you did your first "conda init"). These locations are not writable by the user. However, the "~/.local" directory is still on our sys.path.

Problems can occur when Python packages of different versions are installed in both the "~/.local" directory, AND in the activated environment. For example, if we had installed a version of "torch" in the base environment using "pip", and then later installed a different version of torch in the environment itself, our environment will now be referencing two different versions of torch inside the activated environment. This will almost certainly cause conflicts and generate errors.

In some cases users may want to use the same version of a package in multiple environments to save disk space, or so they only have to update the package once in the base environment so that it can be accessed in all activated environments. While that is an acceptable way to do things, keep in mind that if you were to install a package that required an updated version of the package you have already installed in the base environment (e.g. torch), Conda may install that updated version in the activated working environment, thus generating the conflict.

How Conflicts Occur

It is generally recommended you keep your base environment very basic, and install the packages you need for each working activated environment within that environment itself. The installation commands "conda install" and "pip install" will install those packages in the Conda environment itself if they are run in the activated environment. If "pip install" is run in the base environment, it will install them under "~/.local", however, "conda install" will fail as it will try to write to the "miniconda3" directory which is not writable by users. In addition, if you install other python packages using the "--user" flag (e.g. python3 -m pip install <package> --user) for your use in something outside conda, those packages will also be installed in "~/.local".

This is how many (if not most) environments "break" when using Conda. If you are having a problem, check to see that you do not have multiple versions of the same package installed in different locations in your sys.path.

Understanding Package Locations

If you inspect the contents of "~/.local/lib/" you may see several different python version subdirectories. For example: "~/.local/lib/python3.9", "~/.local/lib/python3.10", "~/.local/lib/python3.11", etc. Each of these subdirectories will contain packages installed using the version of Python/pip that was used during installation outside of a Conda working environment (i.e. base). They will not all be referenced inside your activated Conda working environment sys.path. For example, if you built your Conda environment with Python 3.11, the "~/.local/lib/python3.11" subdirectory will be on your sys.path, but not the ones for any other Python version (e.g. python3.9 or python3.10).

Additionally, the actual package locations will always be on a path like: "./lib/pythonX.X/site-packages" - whether under "/.local" or in the working activated Conda environment. Check your sys.path as shown previously, and examine those directories for potential conflicts.

Resolution

Conflicts can be resolved in a variety of ways, depending on user requirements. One obvious solution for a conflict - for example, different versions of numpy - would be to return to the base environment and run "pip uninstall numpy". If your current version of Python in your base environment is newer than a working environment version of Python you have used, you can simply navigate to the "~/.local/lib/pythonX.X/site-packages" directory, and delete all of the numpy package elements there (e.g. rm -Rf numpy*). The base environment version of Python that is used comes from miniconda3. Depending on what version of miniconda3 you loaded when you first setup Conda, that will be the dominant version used. If you used "module load Python/3.X" for a user-based install of a package (e.g. module load Python/3.9.0) a "lib/python3.9" subdirectory will be created under "~/.local". This is how you can wind-up with several different Python version subdirectories under "~/.local".

You will also want to check under "~/.local/bin" as executables related to the conflicting package may be installed there. If you are able to uninstall using "pip uninstall" those should be cleaned up in most cases. If you have to manually remove subdirectories of files under "site-packages" these executables may need to be manually removed as well.

Miniconda3 Versions

Occasionaly, users will want to install a newer Python package or package version that may not be compatible with their base Conda environment.

UTS periodically installs updates of miniconda3 which are accessible via the "module command": 

[jbjohnston@hpc-login-p01 ~]$ module av miniconda3

---------------------------------------------------------------- /cm/shared/modulefiles ----------------------------------------------------------------
   miniconda3/current    miniconda3/4.9.2-py385 (D)    miniconda3/4.10.3-py385    miniconda3/24.1.2-py311

  Where:
   D:  Default Module

When you first setup Conda you would have loaded one of these and then run "conda init". This tacks on some lines to the bottom of your "~/.bashrc" file that define the environment for that version of miniconda3. UTS has added a symbolic link to the latest version, which you can load using "module load miniconda3/current". If you desire to update your version of miniconda3 that you are using for your base, load the "miniconda3/current" modulefile, and rerun "conda init". This should update your ".bashrc" file. For the new environment to take effect, you can either log out and back in again, or run "source .bashrc". Conda indicates that existing environments running older versions of Python should not be adversely impacted by this upgrade. 

module reset
module load minicoda3/current
conda init

If the newest installed version of Conda required is not installed on Matilda, please send an upgrade request to "[email protected]".

If you have installed a Python package using "pip" in your base environment, additional "dot directories" may be created under ~/. For example, installing Jupyter will create a "~/.jupyter" subdirectory. If this is a package which is causing a conflict in your working environment, you will probably need to delete this as well in addition to doing a "pip uninstall" or manual removal (e.g. rm -Rf ~/.jupyter). In that case, you may need to activate the working environment again after removal, and then reinstall the package in that environment to recreate any additional "dot directories" that are compliant with the version you wish to use.

CategoryHPC