Python VirtualEnv

When managing a Python project, there comes a time to keep Python packages separated from your main environment. There may be incompatible versions that clobber each other, and you can get into potentially bizarre indeterministic behavior.

Getting Python

You need Python obviously. If you have python already, skip ahead.

  • Python 3

The pyenv Option

The best way in my experience, is to use a pyenv to install either or exact python2 and python3 python versions. This is an easy way to control the exact python version on *nix systems like macOS, FreeBSD, and Linux.

macOS Python

You can install a local python for your current user with Homebrew:

brew install python@2

Windows Python

You can use Chocolatey to install Python:

choco install python2

Linux: Ubuntu 16.04

Ubuntu already comes with Python. You can use this to both Python versions, Python 2 and Python 3 environments on Ubuntu.

# upgrade latest pythons
sudo apt-get update
sudo apt-get -y upgrade
# install pip package manager (required)
sudo apt-get install -y python-pip python3-pip
# install compilers/libraries (required for some packages)
sudo apt-get install -y \
build-essential \
libssl-dev \
libffi-dev \
python-dev

Getting VirtualEnv

The tool virtualenv runs on both Python2 and Python3.

pip install virtualenv

Using VirtualEnv

Now we can create a new python virtual environment:

mkdir -p ~/.virtualenvs
virtualenv
~/.virtualenvs/myproj
source ~/.virtualenvs/myproj/bin/activate
pip install flask
Package      Version
------------ -------
Click 7.0
Flask 1.0.2
itsdangerous 1.1.0
Jinja2 2.10
MarkupSafe 1.1.0
pip 18.1
setuptools 40.6.2
Werkzeug 0.14.1
wheel 0.32.3
deactivate
pip (8.1.1)
pycrypto (2.6.1)
setuptools (20.7.0)
virtualenv (16.1.0)
wheel (0.29.0)

VitualEnvWrapper

There are some convenience tools virtualenvwrapper that offers a more intuitive user experience. Install and setup with the following:

# install
pip
install virtualenvwrapper
# setup workon area
mkdir -p ~/.virtualenvs
export WORKON_HOME=~/.virtualenvs
# add wrapper functions
source ~/.local/bin/virtualenvwrapper.sh

Using VirtualEnvWrappers

mkvirtualenv mynewproj
workon mynewproj
Package    Version
---------- -------
pip 18.1
setuptools 40.6.2
wheel 0.32.3
pip install ansible
Package      Version
------------ -------
ansible 2.7.2
asn1crypto 0.24.0
bcrypt 3.1.4
cffi 1.11.5
cryptography 2.4.2
enum34 1.1.6
idna 2.7
ipaddress 1.0.22
Jinja2 2.10
MarkupSafe 1.1.0
paramiko 2.4.2
pip 18.1
pyasn1 0.4.4
pycparser 2.19
PyNaCl 1.3.0
PyYAML 3.13
setuptools 40.6.2
six 1.11.0
wheel 0.32.3
deactivate

Selecting Python Version

If you have multiple Python versions on your system, and would like to use a particular version of Python for your virtualenv, you can do this with the --python=/path/to/python option with either virtualenv or mkvirtualenv commands.

Example Usage

Here’s an example using Ubuntu 16.04 environment that has both python2 and python3 executables in the PATH:

# example of python2 project (virtualenv)
virtualenv --python=$(which python2) ~/.virtualenvs/site1
source ~/.virtualenvs/site1/bin/activate
pip install django==1.8
# example of python3 project (virtualenvwrappers)
mkvirtualenv --python=$(which python2) site2
workon site2
pip install django==2.1.3

Automatic VirtualEnv with DirEnv

We can use a tool like DirEnv to do automation of our virtualenv or virtualenvwrapper tools when we enter a project directory.

Install DirEnv

For installation, on macOS, we can use brew insdtall direnv and on Ubuntu 16.04, we can do sudo apt-get install direnv . Other operating systems should have some similar procedure. See https://direnv.net/.

Setup DirEnv

We can setup and configure DirEnv for a python environment with this (in bash):

cat <<-'DIRENV_CONFIG' > ~/.direnvrc
layout_virtualenv() {
local venv_path="$1"
source ${venv_path}/bin/activate
}
layout_virtualenvwrapper() {
local venv_path="${WORKON_HOME}/$1"
layout_virtualenv $venv_path
}
DIRENV_CONFIG
eval "$(direnv hook bash)"

Using DirEnv

Now that we have DirEnv setup, let’s create some environment, using virtualenvwrappers as we did before.

# setup 
export
WORKON_HOME=~/.virtualenvs
source ~/.local/bin/virtualenvwrapper.sh
# create virtualenv
mkvirtualenv deployprojs
mkdir -p myweb && cd myweb
echo 'layout virtualenv $HOME/.virtualenvs/deployprojs' > .envrc
direnv allow
pip install fabric
fabric       2.4.0

Saving Environments

So save the packages in a package manifests for future use on another machine, we can run:

pip freeze > requirements.txt
pip install -r requirements.txt

Wrapping Up

This may seem complex, because, well, it is. The takeaways from this are:

  • freezing (saving) python package versions per project
  • automating all of this using DirEnv

Linux NinjaPants Automation Engineering Mutant — exploring DevOps, o11y, k8s, progressive deployment (ci/cd), cloud native infra, infra as code