Python 2 End of Life (EOL) and Python 3 Support

Learn how the sunsetting of Python 2.x by pthon.org affects your apps and use of the DNAnexus Platform.

As stated on python.org, the Python core development team will sunset Python 2.x and support only Python 3.x after January 1, 2020. This FAQ explains the impact of this change on your DNAnexus apps and explains Python 3 support on the DNAnexus Platform.

As of December 2019, we provide an Ubuntu 16.04 app execution environment, "Python 2 AEE," which includes the following:

  • The dx-toolkit package (including the "dx" command-line client and the "dxpy" python module), configured in a way that requires Python 2.

  • The stock Ubuntu python2.7 interpreter, available at /usr/bin/python.

  • The stock Ubuntu python3.5.2 interpreter, available at /usr/bin/python3.

To facilitate the migration to Python 3, we will provide a new Ubuntu 16.04 AEE in the first quarter of 2020. This new "Python 3 AEE" will include the dx-toolkit package configured in a way that makes "dxpy" compatible with both Python 2 and Python 3. The "dx" command-line client will require Python 3.

Furthermore, we will introduce a new configuration option to dxapp.json so that you can select between "Python 2 AEE" and the new "Python 3 AEE." In addition, we will introduce a new "python3" value for the "interpreter" dxapp.json configuration option.

Does the "dxpy" python module work with Python 2 and Python 3?

The dxpy module works with both Python 2 and Python 3. For historical reasons, the default installation of dxpy in the existing "Python 2 AEE" is python2.7-specific, but the "Python 3 AEE" will remove this constraint. You can continue using pip3 to install a python3-compatible version of the dxpy module outside of the DNAnexus execution environment (e.g. on your laptop).

Will apps that use Python 2 stop working on January 1, 2020?

Apps that do not rely on installing python2.7 packages using pip at runtime (either through pip dependencies in dxapp.json or through `pip install` commands in the shell script) will continue running.

An app that installs python2.7 packages at runtime may stop working before or after January 1, 2020 if it depends on the latest version of a package that has dropped python2.7 support, such as numpy and pandas. For such packages, you must specify in your app’s dependencies a specific version of the library that does support python2.7 to ensure your app will continue working with python2.7.

For example, if your dxapp.json specifies the numpy package without a version, i.e. via"execDepends": [ {"name": "numpy", "package_manager": "pip"], the execution will fail trying to install the latest version of numpy with the error message RuntimeError: Python version >= 3.5 required. You can address the problem by specifying an earlier version of numpy, i.e. via"execDepends": [ {"name": "numpy", "package_manager": "pip", "version":"1.16"}]

What are the potential security implications of using Python 2 after January 1, 2020?

Any new security vulnerabilities discovered in Python 2 after January 1, 2020, will remain unpatched. The DNAnexus execution environment isolates the execution of apps in a secure Linux container, and mitigates the impact of potential Python 2 security vulnerabilities. Given the lack of support after Python 2 goes End-of-Life (EOL), significant security vulnerabilities may cause the DNAnexus Platform to disable execution of Python 2 or have you assume full liability for execution of Python 2 customer code.

How long will DNAnexus support the "Python 2 AEE"?

On March 1, 2021, we will stop updating the "Python 2 AEE."

Starting March 1, 2021:

  • We will deprecate and stop updating "Python 2 AEE". We will no longer accept new applets or apps with the "Python 2 AEE" option specified in dxapp.json, or with the "interpreter" option set to "python2.7."

  • The "dx-toolkit" package (including the "dx" command-line client and the "dxpy" python module) will remove support for Python 2. Previously published versions of "dxpy" that had Python 2 support will still be available via pip.

  • For reproducibility reasons, existing apps or applets that rely on the "Python 2 AEE" will continue to run, and will use the last version of dxpy (prior to the cutoff date) that was compatible with Python 2.

How can an app that uses Python 2 be migrated to Python 3?

After we introduce the new "Python 3 AEE" option in dxapp.json, rebuild your app with that option specified. This also applies to bash apps that don't call python code explicitly, but rely on the "dx" command-line client. In addition, if your dxapp.json has the "interpreter" option set to "python2.7," change it to "python3."

If your app includes any Python code, make sure it is compatible with Python 3. For more information about porting Python 2 code to Python 3, consult the Python porting guide at https://docs.python.org/3/howto/pyporting.html.

If your app depends on other resources, assets, or packages that use Python, ensure that these dependencies have also been updated.

Other alternatives for removing Python 2 dependencies from your app include the following:

  • Re-implement your app in a higher-level WDL workflow description language.

  • Encapsulate your app execution environment and app dependencies in a docker container for advanced use cases such as interpreting your app using python 3.8 or running on Ubuntu 18.04, and use docker command supported by Ubuntu 16.04 AEEs.

  • Engage the DNAnexus xVantage team for assistance with the transition.