Python package installation in Ubuntu 24.04 AEE
In a generic Ubuntu 24.04 environment, installing packages from the PyPI repository using python3 -m pip install
results in an externally-managed-environment
error for both system-wide and user-specific installations (see PEP 668) and a recommendation to install Python packages in a Python virtual environment (venv
).
In Ubuntu 24.04 DNAnexus Application Execution Environment (AEE), the environment variable PIP_BREAK_SYSTEM_PACKAGES=1
is set. This environment variable disables pip
's externally managed environment check so that Python package installation works the same way as in Ubuntu 20.04 AEE, allowing python3 -m pip install
outside of a Python virtual environment and simplifying migration of apps from 20.04 to 24.04 AEE.
However, this may lead to Python package version conflicts between system-wide Python packages installed via apt-get install
and user-specific Python packages installed via python3 -m pip install
. For example, issuing python3 -m pip install jsonschema==4.23
in Ubuntu 24.04 AEE will fail due to a conflict with apt-installed version of jsonschema
installed as a part of the python3-jsonschema
apt package required by docker-ce
apt package:
root@job-xxxx:~# aptitude why python3-jsonschema
i docker-ce Depends iptables
i A iptables Suggests firewalld
p firewalld Depends python3-nftables (>= 0.9.3-2~)
p python3-nftables Depends python3-jsonschema
root@job-xxxx:~# python3 -m pip install jsonschema==4.23
...
Attempting uninstall: jsonschema...Found existing installation: jsonschema 4.10.3
ERROR: Cannot uninstall jsonschema 4.10.3, RECORD file not found. Hint: The package was installed by debian.
Such conflict can be avoided by using Python venv
:
root@job-xxxx:~# python3 -m venv myvenv
root@job-xxxx:~# source myvenv/bin/activate
(myvenv) root@job-xxxx:~# python3 -m pip install jsonschema==4.23
...
Successfully installed jsonschema-4.23.0 ...
Similar conflicts may lead to DXExecDependencyError
when running an applet with runSpec.execDepends
pip3
dependencies. For example, running an applet with a pip3
runSpec.execDepends
that includes ipython[all]
which depends on a Python jsonschema
package with a version newer than 4.10.3 will result in a DXExecDependencyError
while trying to install ipython[all]
during the job runtime:
$ grep -C2 ipython dxapp.json
{
"package_manager": "pip3",
"name": "ipython[all]"
}
$ dx build .
applet-xxxx
$ dx run applet-xxxx
Found existing installation: jsonschema 4.10.3
ERROR: Cannot uninstall jsonschema 4.10.3, RECORD file not found. Hint: The package was installed by debian.
…
dxpy.utils.exec_utils.DXExecDependencyError: Error while installing pip3 packages [{'name': 'ipython[all]', 'package_manager': 'pip3'}]
The following workarounds exist for such conflicts:
If an apt package with similar functionality is available, use
apt
instead ofpip3
inrunSpec.execDepends
. For example, you can installipython
package (without extras) by replacing{'name': 'ipython[all]', 'package_manager': 'pip3'}
with{'name': python3-ipython'}
in applet'srunSpec.execDepends
.If the Python dependency is installed to provide a command-line tool, you can remove the conflicting
runSpec.execDepends
pip3
dependency fromdxapp.json
and usepipx
inside the applet script to install the command line toolsExample of installing
ipython
CLI tool usingpipx
:sudo apt-get install pipx -y pipx install ipython[all] pipx ensurepath ipython
pipx inject
can be used to install another package inside a previously installed package, e.g.$ pipx inject ipython dxpy
pipx
doesn't see global packages which were installed by system
Otherwise, remove the conflicting
runSpec.execDepends
dependency fromdxapp.json
and usevenv
inside the applet scriptExample of installing
ipython[all]
package withvenv
:python3 -m venv --prompt ipython ipython_venv source ipython_venv/bin/activate python3 -m pip install ipython[all]
To allow
venv
to see packages installed at the system level, create yourvenv
with the--system-site-packages
flag
Last updated
Was this helpful?