Yesterday, I was looking into a Python side-project after 2 years. It was using pipenv so I thought the process of getting it to build will be easy. But that wasn't the case.

The pipenv installation was all broken because the Python version had changed on my system. Fixing this let me down a rabbit hole of understanding the ecosystem of Python distribution and how to avoid this problem in the future.

This post is about that. It will show you how to use Python on a Mac correctly.

Do not use system Python

The first step in using Python correctly is stopping using system Python. That Python is present at the following paths.

/usr/bin/python3 --version
/usr/bin/python --version

Why? Because it doesn't get frequent updates and it may be used by the Apple system. So you can't depend on it and you can't modify it (because it is managed by Apple). It's just a pain to work with. So don't use it.

Use Homebrew Python as your daily driver

Install python using Homebrew.

brew install python

This distribution of Python is regular with updates, and if you are using homebrew for installing tools & software(as you should), it will get installed as a dependency anyway. So no point in not using it over System python.

You will also need to add it to your shell so it can be used over system Python. In your .zshrc or .bashrc, add the following lines.

alias python=/usr/local/bin/python3
alias pip=/usr/local/bin/pip3

Reload the shell and you should be good to go.

>> which python
python: aliased to /usr/local/bin/python3

Do not use Homebrew Python for development

This is what I realized after yesterday's shenanigans. The problem with using Homebrew python is that it's not in your control. That is, it can upgrade from 3.6 to 3.9 without you doing it. For example, this can happen when some Homebrew-based tool that uses Python as dependency is updated. And when that happens, you also lose your site-packages. So all the pip packages that you had installed suddenly becomes inaccessible.

# when python was 3.6
pip -V
pip 20.2.4 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
# when python was 3.9
pip -V
pip 20.2.4 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
# notice the different site-packages path

So it's hard to use Homebrew Python for development. When your Python interpreter and site-packages are changing without you knowing it, it can be hard to manage. That brings me to my next point.

Use brew to install global python tools

Global tools like httpie, youtube-dl can be installed via brew as well. Use that instead of pip.

Why? As we learned in the last section, a major python update changes the site-packages path. So the pip libraries you had installed before and were accessing from PATH suddenly aren't accessible anymore. Or they are accessible, but a pain to manage since pip points to a different pip now.

So that's why we install these tools via homebrew so they can be built again when homebrew python updates.

brew install httpie youtube-dl

This also includes dev tools like pipenv and pyenv that we will talk about in the next part.

brew install pipenv pyenv

Use pipenv and pyenv for development projects

For all development projects, you should use pipenv. That goes without saying, really.

Easy dependency management, deterministic builds, Kenneth Reitz, what's not to like.

But we will use pyenv with it too. Why? So that we can better control what Python interpreter we use in a project. Since both system and Homebrew Python interpreters are unreliable because of the reasons discussed above, we need to use an alternative Python interpreter which we can control. Pyenv solves that problem for us.

To use pyenv, specify the python version in pipenv install command. If that python version is not present, pipenv will install it in pyenv and start using it.

cd my_python_project
pipenv install --python 3.8

The benefit of this stack is that pyenv is managing the various python versions. So you can easily have any number of Python versions available at one time (inside pyenv) and it won't conflict with anything.

And with pipenv managing dependencies, you can have multiple dependencies of a single package on your system, say Flask==1.0.2 and Flask==1.1.2, and they will work peacefully as well.

Decoupling both, the interpreter and the dependencies makes it easy to work on Python projects as no piece strongly depends on the other. Your global python(homebrew) is free to change as needed. It won't affect your development setup at all.