Python Version and Dependency Management

TL;DR

Verbose explanations of each part can be found lower in the document.

        # install
        brew install pyenv pyenv-virtualenv openssl readline sqlite3 xz zlib
        
        # make sure this is in your .bashrc / .zshrc
        if command -v pyenv 1>/dev/null 2>&1; then
           eval "$(pyenv init -)"
        fi
        if which pyenv-virtualenv-init > /dev/null; then
           eval "$(pyenv virtualenv-init -)"
        fi
        
        # install python version
        # versions: https://www.python.org/doc/versions/
        pyenv install <version>
        
        # list all installed python versions and virtual environments
        pyenv versions
        
        # set global python version
        pyenv global <version>
        
        # create new virtual environment
        pyenv virtualenv <version> <virtual_environment_name>
        
        # tie virtual environment to repo
        cd path/to/repo
        pyenv local <virtual_environment_name>
        
        # delete virtual environment
        pyenv uninstall <virtual_environment_name>
        

FAQ

What problem is this solving?

• As a developer, constantly switching between Python versions and project dependencies can be frustrating.
• As a DevOps engineer, it is pivotal that the developer’s local environment is exactly the same as the environment that the project is deployed in.
• As part of Engineering Excellence, it is our job to cut down on the annoyances and time wasters that are slowing down our developers, while at the same time not forgoing security and all of our other best practices.
• Standardizing how we manage Python versions and project dependencies is important to our success as an organization. Hopefully this resource documents a worthy solution to our problems.

What’s the urgency?

The quicker we implement a solution, the quicker we get on top of this and cut down on developer strain.

pyenv

pyenv is a simple Python version management tool. You can install any number of different releases of Python all at once, and have your terminal automatically switch between them.

pyenv Setup

These setup instructions were summarized and streamlined from the pyenv repository README.md for FIRE purposes.

1. Install pyenv
   1. brew install pyenv
2. Append this to the bottom of your shell (.bashrc, .zshrc, etc) to enable shims and autocompletion.

   1. 
              if command -v pyenv 1>/dev/null 2>&1; then
                 eval "$(pyenv init -)"
              fi
              

   2. Make sure this is at the end of your shell configuration file since it manipulates PATH during initialization.
3. Restart your shell
4. Make sure you have necessary Python build dependencies before you install a new Python version
   1. brew install openssl readline sqlite3 xz zlib
5. You’re all set! Continue on to the setup instructions for pyenv-virtualenv!

Helpful resources:

• Repository
• Managing Multiple Python Versions With pyenv
• How to use pyenv to run multiple versions of Python on a Mac

pyenv-virtualenv

pyenv-virtualenv is a pyenv plugin to manage virtual environments.

pyenv-virtualenv Setup

1. Install pyenv-virtualenv
   1. brew install pyenv-virtualenv
2. Append this to the bottom of your shell (.bashrc, .zshrc, etc) to enable shims and autocompletion.

   1. 
              if which pyenv-virtualenv-init > /dev/null; then
                 eval "$(pyenv virtualenv-init -)"
              fi
              

   2. Make sure this is at the end of your shell configuration file since it manipulates PATH during initialization.
   3. This code block is different from step 2 from the pyenv setup section.
3. Restart your shell
4. Make sure you have necessary Python build dependencies before you install a new Python version
   1. brew install openssl readline sqlite3 xz zlib
   2. This is the same as step 4 from the pyenv setup section.
5. You’re all set! Continue on to the Usage section for instructions on automatically switching between Python versions and virtual environments!

Helpful resources:

• Repository
• How to properly use pyenv and venv

Install a new Python version

1. pyenv install <version>
   1. Replace <version> with a valid Python release version
      1. If you’re not sure what Python version a project requires check the README.md, setup.py, tox.ini, or reach out to one of the project’s maintainers.
   2. e.g. pyenv install 3.9.18

Set your default global Python version

1. pyenv global <version>
   1. Replace <version> with any of the versions you’ve installed
      1. You can list what versions you’ve installed by running: pyenv versions
2. Double check that your global Python version matches the one you’ve set
   1. python --version
   2. If the version is incorrect, make sure to run Step 2 from both the "pyenv Setup" and "pyenv-virtualenv Setup" sections. This will make sure that the pyenv init line will always run first thing when starting a fresh terminal.

Create a new virtual environment

1. pyenv virtualenv <version> <virtual_environment_name>
   1. Replace <version> with any of the versions you’ve installed
      1. You can list what versions you’ve installed by running: pyenv versions
   2. Replace <virtual_environment_name> with a unique string name for this project’s virtual environment
   3. e.g. pyenv virtualenv 3.9.18 venv_my_project

Automatically activate and switch between Python versions and virtual environments

“When you run a Python command, pyenv will look for a .python-version file in the current directory and each parent directory. If no such file is found in the tree, pyenv will use the global Python version specified with pyenv global.

1. cd into your project’s local repository directory
2. List your available Python versions
   1. pyenv versions
   2. This should show a list of all the Python versions you’ve installed, as well as all virtual environments you’ve created
3. Set your preferred virtual environment
   1. pyenv local <virtual_environment_name>
      1. Replace <virtual_environment_name> with the unique virtual environment name you’ve created for this project
      2. e.g. pyenv local venv_my_project
   2. This will generate a .python-version file in your repository where you called it. It’s best if it lives in project root.
      1. This file should probably be gitignored, unless you align every developer of your project to use the same exact <virtual_environment_name>. That could definitely work, as long as it is outlined in the project README.md or such.
4. You’re all set! Now, everytime you cd into your project, you will have the correct Python version and virtual environment automatically activated for you! Less time wondering why tox is bugging out, and more time completing the WI you actually came to do.

Double check PyCharm is pointing to the correct virtual environment

1. Inside PyCharm, go to “PyCharm” → “Preferences” → “Project” → “Python Interpreter”
2. At the top, select the dropdown titled “Python Interpreter” and choose the “Show all” option
3. If you don’t see the virtual environment you’ve created for this project, click the + symbol at the bottom-left
4. Click the radio button titled “Existing Environment”
5. Select the dropdown titled “Interpreter” and select the correct virtual environment
   1. Note: if the virtual environment you created doesn’t show up, close all of these dialogs windows and redo these steps. PyCharm caches these results, so reopening these dialogs should repopulate the list properly
6. Click “OK” and “Apply” until you’ve escaped all dialogs
7. That’s it! Now your terminal and PyCharm are all synced up!

How to delete a virtual environment

1. pyenv uninstall <virtual_environment_name>
   1. Replace <virtual_environment_name> with the unique virtual environment name you want to delete
2. If you’ve set this virtual environment via pyenv local <virtual_environment_name> in one of your projects, make sure to remove the .python-version and/or replace it with a new virtual environment