Unmessing Python packages with virtual environments

30 Apr 2017

When I teach programming to beginners, I often recommend Anaconda or Canopy as the default environment for coding in Python. Such programs offer a convenient solution for someone who is just starting to learn how to code, simplifying the process of installing and managing Python packages. Also, such programs alleviate cognitive overload of trying to understand what’s happening “under the hood”, which might be desirable for someone who just learned what a variable or a for loop is. However, once a user starts to manage their own packages and once the understanding of things “under the hood” becomes relevant, things can easily get tricky. This post explains a simple and consistent way to manage Python packages.

50 shades of headaches

Python package management can be a mess if some advanced functionality is needed. For example, if you are a grad student and you just joined a lab that speaks Python instead of Matlab/R/something else you’ve been used to. Here are a few examples of situations which you might encounter:

One way to deal with these kinds of issues is by using Python virtual environments. A virtual environment (VE) is a program that isolates a group of Python packages and keeps them separate from all other Python packages. Then, when a group of packages is needed, the virtual environment containing that packages can be activated and packages can be imported as usual with import package. The activation of a VE happens in a terminal, so there are no changes to the Python code. The nice thing about this system is that there can be many VEs, and one VE can be activated in one shell prompt while another one can be activated in a different shell prompt. On my computer, I have one general VE with Python 2.X installation that I use frequently, then I keep a separate one for Python 3.X installation, and sometimes I have other VEs for individual research projects. Also, packages installed within a VE are kept separate from system-wide, global Python packages that needed to run the operating system. And messing with global Python packages that are used to run the OS is exactly what it sounds like – not a great idea. This kind of mess with global packages can happen when sudo pip install is used to install Python packages (which, unfortunately, often shows up as a “solution” when googling for problems related to installing Python packages).

Setting up Python Virtual Environments

In the next few paragraphs I will explain how to get VEs up and running. I will assume a *nix system and some basic familiarity with shell commands (e.g., navigation, creating and removing directories). A tool used to create Python VEs is called virtualenv, and another tool called virtenvwrapper is a set of extensions to virtualenv that provide shortcuts that allow setting up a VE more quickly. Together, they make the management of Python packages convenient and fairly simple.

Both of these packages can be installed with pip:

$ pip install --user virtualenv virtualenvwrapper

The --user flag ensures installation to a local user folder, so no sudo privileges are required for that. virtualenv and virtualenvwrapper should be now installed somewhere in a local folder, in my case they can be found in:


but different versions of pip might put them in a different place. I use locate virtualenv.py or locate virtualenvwrapper to double-check whey there are. Before we can create a new VE, we need to do two things: (1) specify a directory where different virtual environments will be stored and (2) tell bash about new commands we want to use (commands for maintaining VEs). We can do this by adding a few lines in .bashrc. At this point, it is good to check where virtualenvwrapper.sh is located, as we will need to add that path in our .bashrc. For me, it is in /home/ivana/.local/bin/virtualenvwrapper.sh. Then, we can append following lines to the .bashrc:

# Python VE stuff
export WORKON_HOME=~/.virtualenvs
source /home/ivana/.local/bin/virtualenvwrapper.sh

virtualenv will create the directory specified in WORKON_HOME the first time we create a new VE (explained next) and it will also take care of managing the content in the directory.

In theory, this should be the end of the setup. You can test if it works by opening a new terminal and creating a new environment:

$ mkvirtualenv first_env

The output should look similar to this one:

ivana@cassiopeia:~$ mkvirtualenv first_env
New python executable in /home/ivana/.virtualenvs/first_env/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/ivana/.virtualenvs/first_env/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/ivana/.virtualenvs/first_env/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/ivana/.virtualenvs/first_env/bin/preactivate
virtualenvwrapper.user_scripts creating /home/ivana/.virtualenvs/first_env/bin/postactivate
virtualenvwrapper.user_scripts creating /home/ivana/.virtualenvs/first_env/bin/get_env_details
(first_env) ivana@cassiopeia:~$ 

The name of the virtual environment has been prepended in front of a user name and everything has been set up successfully. In practice, something might go wrong here. A little troubleshooting section below gives some guidelines (feel free to e-mail me if you get different errors).


When I first tried creating an environment, I had an issue with paths:

ivana@cassiopeia:~$ mkvirtualenv first_env
ERROR: virtualenvwrapper could not find virtualenv in your path

the sudo-less way to resolve this is to add /home/ivana/.local/bin to the PATH in .bashrc:

export PATH=$PATH:/home/ivana/.local/bin

Using Virtual Environments

Within a virtual environment packages can be installed as usual:

(first_env) ivana@cassiopeia:~$ pip install xkcd
Collecting xkcd
Installing collected packages: xkcd
Successfully installed xkcd-2.4.2

To get a list of installed packages use pip freeze:

(first_env) ivana@cassiopeia:~$ pip freeze

deactivate deactivates the current environment, and an environment can be activated again with workon first_env. Use rmvirtualenv to remove a virtual environment (this can be done only if that environment is not currently active). If there is an environment that is used frequently, it can be useful to add it to .bashrc so that every time a new terminal is opened, this environment is activated by default. To do so, just add workon first_env (or whatever name of that environment is) in .bashrc after source /home/ivana/.local/bin/virtualenvwrapper.sh.

This was a light-weight introduction into the world of virtual environments, and there are plenty of other features which I have not mentioned here. This nice tutorial provides more information on commands that exists and related useful tools.