Managing Python versions with pyenv

Juggling Python versions using pyenv

Python is now the most popular language (ref TIOBE index and Stack Overflow Trends) as of 2023 and 2024. It is very popular for AI and machine learning solutions and web platforms such as Flask, Tornado, and Django.

Python applications

In Devops oriented roles, I have used Python for build automation and deployment with Fabric, and change configuration with Ansible and Salt Stack.

The one question is:

Which Python? Or which Python3 language?

The Problem and Solution

Image: python environment (source: https://xkcd.com/1987/)

Different projects may use different Python language versions. So there is a need to easily switch between versions. On top of this, historically, there were two major language platforms in use: Python 2 and Python 3. There may be a need to work with legacy code that requires Python 2.

Is there an easy way to manage all of this complexity? Wouldn’t it be nice to install all the Python versions you need and easily switch between the versions?

Enter pyenv to the rescue. With this tool, you can set the defaults for python, python2, python3 binaries, and easily switch between them. Some systems may have python binary set to run Python 2, while others may have it set to Python 3. In some cases, like Ubuntu 22.04, the default python is not even setup, but python3 is configured. This makes it hard to set the shebang line in your Python scripts to something that works consistently. So being able to set these defaults assures that the scripts will work correctly.

Additionally, you can have the default python version automatically set when you navigate to your project folder, such using cd and pushd commands to change into the directory. This is done by configuring a file .python-version with the desired version, so that once you enter that directory, the default python will set to the desired version.

This article will walk you through how to install and use pyenv on a Debian or Ubuntu based Linux distro.

📓 NOTE: This is article is an update to an earlier article that I wrote in 2018 and is updated for Ubuntu Jammy (v22.04).

The Requirements

• Ubuntu or Debian Linux for instructions covered in this guide. Other distros or systems will need to find similar packages on their system to meet the dependencies for compiling Python.
• POSIX Shell [sh] such as bash [bash] or zsh [zsh] are used to run the commands. These come standard on most Linux distros.
• GNU Grep [grep] supports extracting string patterns using extended Regex and PCRE. This comes default on Linux distros.

Testing with Vagrant (optional)

These scripts were tested in Ubuntu Jammy (22.04 LTS) using a virtual machine (Virtualbox) with automation from Vagrant (using jammy64 box).

If you would like to use this test environment and you have these both Virtualbox and Vagrant installed, you can type:

vagrant init ubuntu/jammy64
vagrant up
vagrant ssh

The Installation

For installation, you’ll need to install compilers and some system libraries as dependencies for pyenv. Then after, you can install the desired Python versions.

Install Compilers

Before installing and compiling Python, a compiler is required.

sudo apt-get update -qq \
 && sudo apt-get install -y build-essential

Install Libraries

These system libraries are needed to compile Python.

# Required
sudo apt-get install -y \
  libffi-dev \
  libreadline-dev \
  libssl-dev \
  zlib1g-dev

# Optional (recommended)
sudo apt-get install -y \
  libbz2-dev \
  liblzma-dev \
  libsqlite3-dev

Install pyenv

The pyenv tool will install python within a local user account, so it will not interfere with the system installed python. You can install pyenv using the pyenv-installer script.

PROJ="pyenv-installer"
SCRIPT_URL="https://github.com/pyenv/$PROJ/raw/master/bin/$PROJ"

curl --silent --location $SCRIPT_URL | bash

After you can initialize your shell environment with:

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Test if the pyenv command is available:

pyenv --version
# pyenv 2.3.35

Configure Shell Startup Environment

You will want pyenv configuration for your shell startup environment, so that it is always available. As an example, with bash, you can run the following below:

######
# Add pyenv config to $HOME/.bashrc
# WARNING: Dangerous command, make sure to back up
########################
cat <<'EOF' >> ~/.bashrc

# pyenv configuration
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
EOF

📓 IMPORTANT: Normally, any Unix or Linux system will use $HOME/.bashrc, but macOS is the exception, where either $HOME/.bash_profile or $HOME/.profile will be used instead.

These setup also works for zsh by configuring the equivalent startup files, such as $HOME/.zshrc.

The Usage

The pyenv tool can configure the default Python in three different contexts:

• global: default for when starting a shell for the current user
• shell: updates the current shell session
• local: use the local project version specified in a .python-version file. Python is set at the current directory level, useful for managing differnt python versions per project.

Install Python Versions

With pyenv installed and configured in the shell, you can start installing packages with pyenv install. For example:

pyenv install 2.7.18
pyenv install 3.7.1
pyenv install 3.12.1

Configure Default Python

You can configure a list of default python versions, and both python2 and python3 will be intelligently configured. The first item in the list will be the default python. For example:

pyenv global 3.12.1 2.7.18

You print the configurations with this:

PYTHONS=($(python2 --version 2>&1 && python3 --version  && python --version))
printf "python2: %s\npython3: %s\npython:  %s\n" \
  $(echo ${PYTHONS[*]} | grep -o '[^Python ]*')

This should should show the following:

python2: 2.7.18
python3: 3.12.1
python:  3.12.1

You can print out the list with pyenv global for the global context, which will show, with the default being the first in the list:

3.12.1
2.7.18

Update Pip and Setuptools

It is good practice to update both Pip and Setuptools. For those unfamiliar with this, these are package installers, where pip is the most popular.

Setuptools uses the command line easy_install to install packages. There is no uninstall option. These packages can include Python bytecode, called egg packages, which makes packages not portable between operating systems.

Pip can both install and uninstall packages, and only includes source code packages, which means only scripts, no byte-code, so they are usable across different Linux distributions and operating systems.

Here’s how you upgrade these tools within both Python 2 and Python 3 contexts.

pip2 install --upgrade pip setuptools
pip3 install --upgrade pip setuptools

Installing Packages

You can install packages within either Python 2 or Python 3 environments. Here’s an example of installing virtualenv, a popular tool for segregating Python environments, so that packages are installed and isolated within a virtualenv directory structure.

pip2 install virtualenv
pip3 install virtualenv

# make the new tools available
pyenv rehash

Segregating environments is a good practice that will increase determinism in testing where package versions are isolated to a single virtualenv.

I wrote an article covering virtualenv: Python Virtualenv.

Demo: Automatic Project Configuration

After the desired Python versions are installed, you can test this feature out in a few seconds.

First of course, is to install the desired versions. For example:

# install desired versions
pyenv install 3.7.1
pyenv install 3.12.1

# verify available versions
pyenv versions

After the desired versions are installed, you can go about creating some pretend project directories:

mkdir example/{projA,projB}
echo "3.7.1" > ./example/projA/.python-version
echo "3.10.5" > ./example/projB/.python-version

This will create the following file structure:

example
├── projA
│   └── .python-version
└── projB
    └── .python-version

Now test that the automatic language configuration works.

pushd ./example/projA/
python --version 
# Expecting `Python 3.7.1'
popd

Now test that the results of this test

cd ./example/projB/
python --version # Python 3.10.5
# Expecting `Python 3.10.5'

Should any of the versions not be previously installed, you’ll see an error message similar to this:

pyenv: version `3.7.1' is not installed (set by /path/to/example/projA/.python-version)

Related Articles

Previous Article

This was the original 2018 article that I wrote that has some notes for macOS and CentOS, as well as an alternative for Windows.

Installing Pythons with PyEnv

Integration with VirtualEnv

This is an article that shows how to to use virtualenv to keep package installs isolated, and also integration with pyenv.

Python VirtualEnv

Conclusion

I wanted to revisit using pyenv to setup Python to see if there were any changes required, especially as package names change with new releases of Debian or Ubuntu. Everything seems to work the same and only the dependencies needed to be refreshed.

I am curious to see how maybe asdf stacks up to pyenv. Apparently asdf is just an abstraction on top of pyenv. In general, the number of tools across different language platforms adds complexity and increases fiction. So my interests in asdf is to find ways to ease management these tools.