How to manage different Python Versions on Mac OS 11 Big Sur
Before you go on reading this article ensure that Homebrew is installed on your Mac. There are installation instructions on my blog digitaldocblog.com or you can just go to the Homebrew website and follow the instructions there.
But first of all I would like to describe my initial situation. I think some Mac users have python installed on their mac by themselves in one way or another and are surprised when checking the current version with python -V or when checking which python binary is currently being executed with the command which python. I installed python 2 and 3 with Homebrew and after executing these commands I was surprised and found that on my system different python versions were running than I expected.
I found that there must be a problem when you install different Python versions on a Mac and I was looking for an easy way to switch between these versions. Switching between the versions is necessary because when you are a developer you might have the need to run your code on a certain version. So it has to be easy to install different python versions on your Mac and to switch between them as needed. There are of course several ways of dealing with this matter. I decided to use the version manager Pyenv. But more on that later. First of all, back to my initial situation.
Initial Situation
It all started with a Homebrew error message after my update to Mac OS Big Sur (Mac OS 11). I had python 2 and 3 installed on my system with Homebrew. I had carried out these installations under Catalina. The brew doctor command gave me the error message saying I had installed kegs without a formula and I should uninstall python@2 because it is an outdated and no longer supported python version. What ?
My first check was on the /usr/local/bin directory because I wanted to know whether the python binaries had been linked correctly from Homebrew.
patrick@PatrickMBNeu ~ % ls -l /usr/local/bin/python*
lrwxr-xr-x 1 patrick admin 32 6 Mär 10:23 python -> ../Cellar/python@2/2.7.17/bin/python
lrwxr-xr-x 1 patrick admin 32 6 Mär 10:23 python2 -> ../Cellar/python@2/2.7.17/bin/python2
lrwxr-xr-x 1 patrick admin 32 6 Mär 10:23 python2.7 -> ../Cellar/python@2/2.7.17/bin/python2.7
Here it seems to me that only python 2 has been installed by Homebrew and the binary links all refer to the python@2 directory in the Homebrew Cellar. My second check was on the Hombrew Cellar directory because I wanted to know whether there are Formulars of the two python versions available.
patrick@PatrickMBNeu ~ % ls -l /usr/local/Cellar/python*
drwxr-xr-x 3 patrick staff 96 26 Feb 18:46 python@2
drwxr-xr-x 3 patrick staff 96 26 Feb 18:46 python@3.8
This made it clear: Python 2 is correctly installed and linked in the Cellar (but out of date) but Python 3 was obviously not installed correctly (no binary links in /usr/local/bin). I understood that this must be the message from Homebrews error message. But then I was curious what the version check would tell me.
patrick@PatrickMBNeu ~ % which python
/usr/local/bin/python
patrick@PatrickMBNeu ~ % python --version
2.7.17
patrick@PatrickMBNeu ~ % which python3
/usr/bin/python3
patrick@PatrickMBNeu ~ % python3 --version
3.8.2
The command which python refers to the python 2 version installed with Homebrew in /usr/local/bin but what the hell is in /usr/bin ?
I googled around a bit and finally I found out that it must be the python version that comes with Mac OS. So the command which python3 refers to the python 3 version pre-installed on the Mac in /usr/bin. And I learned that it must be avoided in any case to remove these so-called System Python Versions from the system. Apple install in /usr/bin while Homebrew install in /usr/local/bin and /usr/local/Cellar. So stay away from everything that is installed in /usr/bin with regard to Python.
Then I wanted to understand what Apple installed in /usr/bin.
patrick@PatrickMBNeu ~ % ls -l /usr/bin/python*
lrwxr-xr-x 1 root wheel 75 1 Jan 2020 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x 1 root wheel 75 1 Jan 2020 /usr/bin/python2 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x 1 root wheel 75 1 Jan 2020 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
-rwxr-xr-x 1 root wheel 137552 1 Jan 2020 /usr/bin/python3
patrick@PatrickMBNeu ~ %
Apple apparently installs Python 2 and Python 3. But the command which python clearly tells me (see above) that the Python 2 version of Homebrew from /usr/local/bin is currently active in some way, while the command which python3 apparently refers to the System Python 3 version in /usr/bin.
How do I get the mess cleaned up ?
I decided to uninstall the python 2 version of Homebrew normally with the command brew uninstall python@2. And I also removed manually the /usr/local/Cellar/python@3.8 directory. Then I checked the python versions again.
patrick@PatrickMBNeu ~ % which python
/usr/bin/python
patrick@PatrickMBNeu ~ % python --version
2.7.16
patrick@PatrickMBNeu ~ % which python3
/usr/bin/python3
patrick@PatrickMBNeu ~ % python3 --version
3.8.2
I was satisfied with the result. Both System Pythons in /usr/bin are currently active on my system and this is exactly what I wanted to achieve with my clean up. I can now think about how I would like to manage other, different python versions on my system. And this is exactly where I come back to Pyenv.
Managing Python on Mac OS with Pyenv
First I install Pyenv with Homebrew.
patrick@PatrickMBNeu ~ % brew update
patrick@PatrickMBNeu ~ % brew install pyenv
Then I create a ~/.zshrc file in my home directory and add the command pyenv init at the end of the ~/.zshrc configuration file to enable pyenv shims in my PATH variable.
patrick@PatrickMBNeu ~ % echo -e 'eval "$(pyenv init -)" ' >> ~/.zshrc
patrick@PatrickMBNeu ~ % cat .zshrc
eval "$(pyenv init -)"
patrick@PatrickMBNeu ~ %
It is very important to make sure that the command eval "$(pyenv init -)" is placed at the end of the shell configuration because it manipulates the PATH environment variable during the shell initialization.
Then I close the terminal and restart it again.
With the command pyenv versions you can check the existing python versions on your Mac. After my cleanup the only version on my Mac is the System Python and marked with system.
patrick@PatrickMBNeu ~ % pyenv versions
* system (set by /Users/patrick/.pyenv/version)
patrick@PatrickMBNeu ~ % python --version
Python 2.7.16
patrick@PatrickMBNeu ~ % python3 --version
Python 3.8.2
patrick@PatrickMBNeu ~ %
When you check the version of the current System Python with the command python --``version and python3 --``version you see that I still run the System Python Versions 2.7.16 and 3.8.2. Thats what I expected.
With Pyenv you can also install Python using the pyenv install command. With the command pyenv install --list you can list all available Python version that can be installed. This would be a long list but you can pipe it into grep to restrict the displayed list a little. Here I restrict the list to all available Python 3.8.* and 3.9.* versions.
patrick@PatrickMBNeu ~ % pyenv install --list | grep " 3\.[89]"
3.8.0
3.8-dev
3.8.1
3.8.2
3.8.3
3.8.4
3.8.5
3.8.6
3.8.7
3.9.0
3.9-dev
3.9.1
patrick@PatrickMBNeu ~ %
Then I install Python 3.9.1 which is the latest Python available at the time of writing this article.
patrick@PatrickMBNeu ~ % pyenv install 3.9.1
Then I check the existing pythons again with pyenv versions and I can already see that the 3.9.1 version has been added. Now comes the real advantage of pyenv. With the command pyenv global I set the 3.9.1 version as the currently active one.
patrick@PatrickMBNeu ~ % pyenv versions
* system (set by /Users/patrick/.pyenv/version)
3.9.1
patrick@PatrickMBNeu ~ % pyenv global 3.9.1
patrick@PatrickMBNeu ~ % pyenv versions
system
* 3.9.1 (set by /Users/patrick/.pyenv/version)
patrick@PatrickMBNeu ~ % python --version
Python 3.9.1
patrick@PatrickMBNeu ~ % python3 --version
Python 3.9.1
patrick@PatrickMBNeu ~ % python
Python 3.9.1 (default, Mar 8 2021, 18:50:54)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
patrick@PatrickMBNeu ~ %
I check again if the default python is now set to 3.9.1 using the command python --``version. The output in the terminal shows the active version marked with a star. This is Python 3.9.1 and exactly what I expected. Then I login into the interactive python command interpreter and this also confirmed that it is currently the 3.9.1 version.
So I manage my Pythons on my Mac OS Big Sur from now on with Pyenv.
Make Pyenv working perfectly with Homebrew
When I finished installing Pyenv everything worked fine until I tried to use the brew doctor command to check my Homebrew installations.
patrick@PatrickMBNeu ~ % brew doctor
Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.
Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:
/Users/patrick/.pyenv/shims/python-config
/Users/patrick/.pyenv/shims/python3-config
/Users/patrick/.pyenv/shims/python3.9-config
patrick@PatrickMBNeu ~ %
I googled a little and I found out this. Very interesting Article about the PATH settings I enforced in my ~/.zshrc file when I installed Pyenv (see above).
Then I checked my environment variables with the command env.
patrick@PatrickMBNeu ~ % env
TMPDIR=/var/folders/zy/3s2n4mh502z3320837q80qdm0000gp/T/
__CFBundleIdentifier=com.apple.Terminal
XPC_FLAGS=0x0
TERM=xterm-256color
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.ZIRHZWPkYD/Listeners
XPC_SERVICE_NAME=0
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=440
TERM_SESSION_ID=53DE5B52-55E2-4D61-A4E4-7F47D5C401AC
SHELL=/bin/zsh
HOME=/Users/patrick
LOGNAME=patrick
USER=patrick
PATH=/Users/patrick/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin
SHLVL=1
PWD=/Users/patrick
OLDPWD=/Users/patrick
PYENV_SHELL=zsh
LANG=de_DE.UTF-8
_=/usr/bin/env
patrick@PatrickMBNeu ~ %
The problem seems to be the PATH which is set by the entry eval "$ (pyenv init -)" in my ~/.zshrc file.
patrick@PatrickMBNeu ~ % cat .zshrc
eval "$(pyenv init -)"
patrick@PatrickMBNeu ~ %
This entry leads to that /Users/patrick/.pyenv/shims is set at the beginning of the PATH variable when zsh is executed.
When you look at the complete PATH you see that the command brew doctor can be executed without specifying /usr/local/bin/brew because the required path specification /usr/local/bin is also available but further back. So brew reads first the path /Users/patrick/.pyenv/shims and only then the path /usr/local/bin and exactly there in /Users/patrick/.pyenv/shims are the *-config files used from Pyenv. So it seems to be that brew has problems with these config files.
Since these config files seems to confuse brew in some way, the best would be if brew could somehow ignore these files. This is possible if we define an alias environment variable. We can define brew as an alias and then instruct zsh that in the event that brew is called a specific PATH applies namely the PATH without the shims directory. For this we have to adapt the ~/.zshrc as follows.
patrick@PatrickMBNeu ~ % cat .zshrc
alias brew='PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin brew'
eval "$(pyenv init -)"
patrick@PatrickMBNeu ~ %
After restarting the terminal I was able to run the brew doctor command without this error message. The problem is solved.
patrick@PatrickMBNeu ~ % brew update
Already up-to-date.
patrick@PatrickMBNeu ~ % brew doctor
Your system is ready to brew.
patrick@PatrickMBNeu ~ %