Python 3 Support and Python 2 End of Life (EOL)

As stated on python.org, the Python core development team supports only Python 3.x, having sunsetted Python 2.x support on January 1, 2020. This FAQ explains Python 3.x and Python 2.x support on the DNAnexus platform.

DNAnexus provides Python 3 app execution environment ("Python 3 AEE") based on Ubuntu 16.04 that includes the following:

• The stock Ubuntu python2.7 interpreter, available at /usr/bin/python

• The stock Ubuntu python3.5.2 interpreter, available at /usr/bin/python3

• "dx" command-line client based on python3 interpreter

• "dxpy" python module installations that work with both python2.7 and python3.5.2 interpreters

An app with dxapp.json containing "runSpec": {"release": "16.04", "version": "1", "interpreter": "python3" ...} will be executed in "Python3 AEE" using python3.5.2 interpreter.

An app with dxapp.json containing "runSpec": {"release": "16.04", "version": "1", "interpreter": "bash" ...} will be executed in "Python3 AEE" using bash interpreter and the dx commands will invoke python3.5.2 installation of dx-toolkit.

For reference, Ubuntu 16.04 apps without"version":"1" in dxapp.json are executed in the legacy "Python 2 AEE" that includes:

• The dx-toolkit package (including the "dx" command-line client and the "dxpy" python module), configured in a way that requires Python 2.

• The stock Ubuntu python2.7 interpreter, available at /usr/bin/python.

• The stock Ubuntu python3.5.2 interpreter, available at /usr/bin/python3.

Does the "dxpy" python module work with Python 2 and Python 3?

The dxpy module works with both Python 2 and Python 3. For historical reasons, the default installation of dxpy in the existing "Python 2 AEE" is python2.7-specific. "Python 3 AEE" removes this constraint. You can continue using pip3 to install a python3-compatible version of the dxpy module outside of the DNAnexus execution environment (e.g. on your laptop).

Do apps that use Python 2 work after January 1, 2020?

Apps that do not rely on installing python2.7 packages using pip at runtime (either through pip dependencies in dxapp.json or throughpip install commands in the shell script) should continue running on the platform.

An app that installs python2.7 packages at runtime may stop working if it depends on the latest version of a package that has dropped python2.7 support, such as numpy or pandas. For such packages, you must specify in your app’s dependencies a specific version of the library that does support python2.7 to ensure your app will continue working with python2.7.

For example, if your dxapp.json specifies the numpy package without a version, i.e. via"execDepends": [ {"name": "numpy", "package_manager": "pip"], the execution will fail trying to install the latest version of numpy with the error message RuntimeError: Python version >= 3.5 required. You can address the problem by specifying an earlier version of numpy, i.e. via"execDepends": [ {"name": "numpy", "package_manager": "pip", "version":"1.16"}]

What are the potential security implications of using Python 2 after January 1, 2020?

Any new security vulnerabilities discovered in Python 2 after January 1, 2020, will remain unpatched. The DNAnexus execution environment isolates the execution of apps in a secure Linux container, and mitigates the impact of potential Python 2 security vulnerabilities. Given the lack of support after Python 2 goes End-of-Life (EOL), significant security vulnerabilities may cause the DNAnexus Platform to disable execution of Python 2 or have you assume full liability for execution of Python 2 customer code.

How long will DNAnexus support the "Python 2 AEE"?

On March 1, 2021, we will stop updating the "Python 2 AEE."

Starting March 1, 2021:

• We will deprecate and stop updating "Python 2 AEE". We will no longer accept new applets or apps with the "Python 2 AEE" option specified in dxapp.json, or with the "interpreter" option set to "python2.7."

• The "dx-toolkit" package (including the "dx" command-line client and the "dxpy" python module) will remove support for Python 2. Previously published versions of "dxpy" that had Python 2 support will still be available via pip.

• For reproducibility reasons, existing apps or applets that rely on the "Python 2 AEE" will continue to run, and will use the last version of dxpy (prior to the cutoff date) that was compatible with Python 2.

How can an app that uses Python 2 be migrated to Python 3?

After we introduce the new "Python 3 AEE" option in dxapp.json, rebuild your app with that option specified. This also applies to bash apps that don't call python code explicitly, but rely on the "dx" command-line client. In addition, if your dxapp.json has the "interpreter" option set to "python2.7," change it to "python3."

If your app includes any Python code, make sure it is compatible with Python 3. For more information about porting Python 2 code to Python 3, consult the Python porting guide at https://docs.python.org/3/howto/pyporting.html.

If your app depends on other resources, assets, or packages that use Python, ensure that these dependencies have also been updated.

Other alternatives for removing Python 2 dependencies from your app include the following:

• Re-implement your app in a higher-level WDL workflow description language.
• Encapsulate your app execution environment and app dependencies in a docker container for advanced use cases such as interpreting your app using python 3.8 or running on Ubuntu 18.04, and use docker command supported by Ubuntu 16.04 AEEs.

• Engage the DNAnexus Professional Services team for assistance with the transition.