How Virtual Environments Work (on Windows)

Brett Cannon made a short (and quite interesting post) on virtual environments and their context, though this focused on their application to a Unix-based OS rather than Windows. I’d like to summarize the content there and adapt it to Windows.

History

Why do we have virtual environments? This may be a perplexing question to someone just starting out with Python, or someone who has spent time mostly developing scripts for short-term or local use. In this scenario, the Python developer has downloaded and installed the latest version of Python and installs all the packages (via pip install <package>) to the global Python. The ‘global’ Python is where the executable/installer placed the program (on Windows this defaults to something like: C:\Users\[USERNAME]\AppData\Local\Programs\Python. If you use the pip in this directory to install packages, they get installed to that Python’s site-packages directory (on Windows, C:\Users\[USERNAME]\AppData\Local\Programs\Python\Python311\Lib\site-packages).

This situation is fine for starting out or if you’re developing only against the standard library (i.e., you’re not using pip)…but suppose you want to share your code with someone else? Or move it to a new computer? Or get it working with a new version Python on your machine? What external libraries do you need to install? Additionally, certain programs that you write (or depend on) may have conflicting requirements, etc., etc. At some point, it’s convenient to have different Python installs for your different projects…but it’s not convenient to install Python multiple times across your system.

Instead, virtual environments conveniently add a quick method to setup a ‘clean’ version of Python anywhere across your computer. Each ‘project’ you write can have its own virtual environment.

Creating a Virtual Environment

Each time I start a new project (i.e., decide to attempt a new task), my steps are:

1. Create a new directory (cd C:\code; mkdir new_project)
   • This is usually in the same folder (e.g., ‘workspace’ or ‘code’)
2. Go into that new directory (cd new_project)
3. Create a virtual environment (python -m venv .venv)
   • The command will probably require the full path rather than python (e.g., something like C:\Users\[USERNAME]\AppData\Local\Programs\Python\Python311\python.exe -m venv .venv)
   • This creates the virtual environment in a folder called .venv
   • NB: Brett suggests using python -m venv --without-pip .venv which will skip the pip update check and save some time…
4. Initialise this as a git repoistory (git init)
   • Remember that .venv should be included in your .gitignore

Before running any code, it’s important to either first activate the virtual environment (run .venv\Scripts\Activate.ps1 in a Powershell console) or include the full path to this Python (C:\code\new_project\.venv\Scripts\python.exe program.py).

I also tend to have a system-wide virtual environment which I have powershell load by default. This is useful for doing quick tests on data, etc., so I’ll have certain libraries installed there (e.g., pandas, seaborn, notebook and/or jupyterlab, sqlalchemy). The powershell function included in my $profile is:

function Activate-Venv {
  param (
    [string]$Path='.'
  )
   if ($Path -eq '.') {
    $Path = Resolve-Path $Path
   }
   if($Path -eq '') {
       if (Test-Path C:\.venv\scripts\activate.ps1) {
        C:\.venv\scripts\activate.ps1
       } elseif (Test-Path D:\.venv\scripts\activate.ps1) {
        D:\.venv\scripts\activate.ps1
       } else {
        echo "Unable to activate virtual enviornment."
       }
   } elseif (Test-Path "$Path\.venv") {
       &("$Path\.venv\scripts\activate.ps1")
   } else {
      Activate-Venv -Path (Split-Path $Path)
   }
}

What Happens?

Unlike with Unix-based OSs, Windows is more temperamental with its symlinks, so the venv command prefers to avoid those complications. Instead of symlinking, a redirector script (before 3.7, this was a copy of the Python executable…which took some time) is created along with several directories:

• Lib: this contains the site-packages folder where pip will install new packages when the virtual environment is activated.
  • Or, if not activating the virtual environment, install by calling the pip in the .venv\Scripts folder.
• Scripts: this contains executables for pip, python, and the activate shell scripts to start the virtual environment.

Virtual environments are not intended to be moved — they are quick enough to create (even on Windows). They will also update when you update the Python patch version (this is the ‘Z’ in X.Y.Z, so 3.11.1 is major version 3, minor version 11, and patch version 1) by downloading and running the installer on Windows.

To update to the next minor version (e.g., 3.12.0 from 3.11.6):

• Install the new version downloaded from Python website
• Delete the old virtual environment
• Create a new virtual environment

Before deleting the old virtual environment, it might be useful to record what packages have been installed and place them in a requirements.txt file using pip freeze.