Conda and Python Troubleshooting

Contents

Overview

This document is intended to provide assistance for some commonly encountered problems when using Conda environments, and/or Python scripts inside or outside of Conda environments.

OpenSSL Errors

This problem generally manifests itself as an "UnsafeLegacyRenegotiation" failure when working in Conda environments, and can also affect "pip install" inside of Conda. The issue is caused when "openssl" packages > 1.1.1 are installed. This problem first appeared with openssl==3.x. So if you have a version of openssl installed > 1.1.1, the solution is to simply downgrade openssl in your Conda environment: 

conda install openssl=1.1.1

Testing OpenSSL

Confirmatory tests can be performed to verify the problem/resolution using a few simple commands. From the node where you're attempting to run something inside your Conda environment, use one of the following: 

openssl s_client -connect huggingface.co:443

or, 

curl -v https://huggingface.co:443 > /dev/null

Examine output from one of the above commands to determine if the connection was verified, intact, and/or open.

SSL Verify Failed

Python Script Workaround

Another common problem is related to the verification of SSL certificates. This sometimes happens when the user's environment does not have the proper path established to the node's SSL certificates, and generally occurs when running "request" operations inside of a Python script. This error generally manifests itself as shown below: 

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)

The solution is generally to include a few statements inside of your Python script (whether run inside or outside of Conda environment). These are shown below: 

import os
os.environ["CURL_CA_BUNDLE"]="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"
os.environ["REQUESTS_CA_BUNDLE"]="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"

This imports the Python "os" package, and then specifies the paths needed for certificate verification.

Pip and Conda Export Workaround

This technique may be useful when working on an interactive session on a gpu or compute node. Upon logging into the node via an interactive session, activate your Conda environment and then export the following environmental variables: 

export CURL_CA_BUNDLE="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"
export REQUESTS_CA_BUNDLE="/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"

Now proceed with your Conda or pip installs.

Openssl.cnf File Workaround

Although it is recommended to use the "miniconda3" modulefile provided by UTS on Matilda, some users will install their own version of Anaconda. In many of these cases, the user may experience problems with OpenSSL connectivity, including SSL verification failures. This is generally caused by newer versions of OpenSSL (i.e., > 1.1.1) being installed. Within the user's self-installed Anaconda directory, there is likely a file named "openssl.cnf" that is being used that may prevent even a downgrade to openssl=1.1.1. This problem can be remedied as follows:

  • Create your own "openssl.cnf" file with the contents shown below: 

openssl_conf = default_conf
[ default_conf ]
ssl_conf = ssl_sect
[ssl_sect]
system_default = ssl_default_sect
[ssl_default_sect]
Options = UnsafeLegacyRenegotiation
  • After noting the location of the file you just created, set the relevant environmental variable to that file's path: 

export OPENSSL_CONF=/path/to/my/file/openssl.cnf
  • Alternately, you can set OPENSSL_CONF temporarily by using it in-line with a conda install command: 

OPENSSL_CONF=`pwd`/openssl.cnf conda install openssl=1.1.1
  • If you used the "export" method, you can clear that value once openssl has been downgraded: 

unset OPENSSL_CONF
  • Now you should be able to use Conda normally.

Conda Create Solution

One simple way to avoid having to deal with the above SSL errors, is to make sure that when creating a new Conda environment it is setup with the correct version of OpenSSL. For example, to create a Conda environment with Python 3.11 we could use: 

conda create -n myenvironment python=3.11 openssl=1.1.1

This will ensure that our environment is created with a workable version of OpenSSL, and that all related packages are also compatible.

PLEASE NOTE: You will not be able to install a version of Python > 11.x. Versions of Python >= 12.x require OpenSSL version 3.x.

CategoryHPC