How to manage different Python Versions on Mac OS 11 Big Sur
Before you go on reading this article ensure that Homebrew is installed on your Mac. There are installation instructions on my blog digitaldocblog.com or you can just go to the Homebrew website and follow the instructions there.
But first of all I would like to describe my initial situation. I think some Mac users have python installed on their mac by themselves in one way or another and are surprised when checking the current version with
python -V or when checking which python binary is currently being executed with the command
which python. I installed python 2 and 3 with Homebrew and after executing these commands I was surprised and found that on my system different python versions were running than I expected.
I found that there must be a problem when you install different Python versions on a Mac and I was looking for an easy way to switch between these versions. Switching between the versions is necessary because when you are a developer you might have the need to run your code on a certain version. So it has to be easy to install different python versions on your Mac and to switch between them as needed. There are of course several ways of dealing with this matter. I decided to use the version manager Pyenv. But more on that later. First of all, back to my initial situation.
It all started with a Homebrew error message after my update to Mac OS Big Sur (Mac OS 11). I had python 2 and 3 installed on my system with Homebrew. I had carried out these installations under Catalina. The
brew doctor command gave me the error message saying I had installed kegs without a formula and I should uninstall python@2 because it is an outdated and no longer supported python version. What ?
My first check was on the
/usr/local/bin directory because I wanted to know whether the python binaries had been linked correctly from Homebrew.
patrick@PatrickMBNeu ~ % ls -l /usr/local/bin/python* lrwxr-xr-x 1 patrick admin 32 6 Mär 10:23 python -> ../Cellar/python@2/2.7.17/bin/python lrwxr-xr-x 1 patrick admin 32 6 Mär 10:23 python2 -> ../Cellar/python@2/2.7.17/bin/python2 lrwxr-xr-x 1 patrick admin 32 6 Mär 10:23 python2.7 -> ../Cellar/python@2/2.7.17/bin/python2.7
Here it seems to me that only python 2 has been installed by Homebrew and the binary links all refer to the python@2 directory in the Homebrew Cellar. My second check was on the Hombrew Cellar directory because I wanted to know whether there are Formulars of the two python versions available.
patrick@PatrickMBNeu ~ % ls -l /usr/local/Cellar/python* drwxr-xr-x 3 patrick staff 96 26 Feb 18:46 python@2 drwxr-xr-x 3 patrick staff 96 26 Feb 18:46 firstname.lastname@example.org
This made it clear: Python 2 is correctly installed and linked in the Cellar (but out of date) but Python 3 was obviously not installed correctly (no binary links in
/usr/local/bin). I understood that this must be the message from Homebrews error message. But then I was curious what the version check would tell me.
patrick@PatrickMBNeu ~ % which python /usr/local/bin/python patrick@PatrickMBNeu ~ % python --version 2.7.17 patrick@PatrickMBNeu ~ % which python3 /usr/bin/python3 patrick@PatrickMBNeu ~ % python3 --version 3.8.2
which python refers to the python 2 version installed with Homebrew in
/usr/local/bin but what the hell is in
I googled around a bit and finally I found out that it must be the python version that comes with Mac OS. So the command
which python3 refers to the python 3 version pre-installed on the Mac in
/usr/bin. And I learned that it must be avoided in any case to remove these so-called System Python Versions from the system. Apple install in
/usr/bin while Homebrew install in
/usr/local/Cellar. So stay away from everything that is installed in
/usr/bin with regard to Python.
Then I wanted to understand what Apple installed in
patrick@PatrickMBNeu ~ % ls -l /usr/bin/python* lrwxr-xr-x 1 root wheel 75 1 Jan 2020 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 lrwxr-xr-x 1 root wheel 75 1 Jan 2020 /usr/bin/python2 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 lrwxr-xr-x 1 root wheel 75 1 Jan 2020 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 -rwxr-xr-x 1 root wheel 137552 1 Jan 2020 /usr/bin/python3 patrick@PatrickMBNeu ~ %
Apple apparently installs Python 2 and Python 3. But the command
which python clearly tells me (see above) that the Python 2 version of Homebrew from
/usr/local/bin is currently active in some way, while the command
which python3 apparently refers to the System Python 3 version in
How do I get the mess cleaned up ?
I decided to uninstall the python 2 version of Homebrew normally with the command
brew uninstall python@2. And I also removed manually the
/usr/local/Cellaremail@example.com directory. Then I checked the python versions again.
patrick@PatrickMBNeu ~ % which python /usr/bin/python patrick@PatrickMBNeu ~ % python --version 2.7.16 patrick@PatrickMBNeu ~ % which python3 /usr/bin/python3 patrick@PatrickMBNeu ~ % python3 --version 3.8.2
I was satisfied with the result. Both System Pythons in
/usr/bin are currently active on my system and this is exactly what I wanted to achieve with my clean up. I can now think about how I would like to manage other, different python versions on my system. And this is exactly where I come back to Pyenv.
Managing Python on Mac OS with Pyenv
First I install Pyenv with Homebrew.
patrick@PatrickMBNeu ~ % brew update patrick@PatrickMBNeu ~ % brew install pyenv
Then I create a
~/.zshrc file in my home directory and add the command
pyenv init at the end of the
~/.zshrc configuration file to enable pyenv shims in my
patrick@PatrickMBNeu ~ % echo -e 'eval "$(pyenv init -)" ' >> ~/.zshrc patrick@PatrickMBNeu ~ % cat .zshrc eval "$(pyenv init -)" patrick@PatrickMBNeu ~ %
It is very important to make sure that the command
eval "$(pyenv init -)" is placed at the end of the shell configuration because it manipulates the
PATH environment variable during the shell initialization.
Then I close the terminal and restart it again.
With the command
pyenv versions you can check the existing python versions on your Mac. After my cleanup the only version on my Mac is the System Python and marked with
patrick@PatrickMBNeu ~ % pyenv versions * system (set by /Users/patrick/.pyenv/version) patrick@PatrickMBNeu ~ % python --version Python 2.7.16 patrick@PatrickMBNeu ~ % python3 --version Python 3.8.2 patrick@PatrickMBNeu ~ %
When you check the version of the current System Python with the command
--``version you see that I still run the System Python Versions 2.7.16 and 3.8.2. Thats what I expected.
With Pyenv you can also install Python using the
pyenv install command. With the command
pyenv install --list you can list all available Python version that can be installed. This would be a long list but you can pipe it into grep to restrict the displayed list a little. Here I restrict the list to all available Python 3.8.* and 3.9.* versions.
patrick@PatrickMBNeu ~ % pyenv install --list | grep " 3\." 3.8.0 3.8-dev 3.8.1 3.8.2 3.8.3 3.8.4 3.8.5 3.8.6 3.8.7 3.9.0 3.9-dev 3.9.1 patrick@PatrickMBNeu ~ %
Then I install Python 3.9.1 which is the latest Python available at the time of writing this article.
patrick@PatrickMBNeu ~ % pyenv install 3.9.1
Then I check the existing pythons again with
pyenv versions and I can already see that the 3.9.1 version has been added. Now comes the real advantage of pyenv. With the command
pyenv global I set the 3.9.1 version as the currently active one.
patrick@PatrickMBNeu ~ % pyenv versions * system (set by /Users/patrick/.pyenv/version) 3.9.1 patrick@PatrickMBNeu ~ % pyenv global 3.9.1 patrick@PatrickMBNeu ~ % pyenv versions system * 3.9.1 (set by /Users/patrick/.pyenv/version) patrick@PatrickMBNeu ~ % python --version Python 3.9.1 patrick@PatrickMBNeu ~ % python3 --version Python 3.9.1 patrick@PatrickMBNeu ~ % python Python 3.9.1 (default, Mar 8 2021, 18:50:54) [Clang 12.0.0 (clang-118.104.22.168)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> exit() patrick@PatrickMBNeu ~ %
I check again if the default python is now set to 3.9.1 using the command
--``version. The output in the terminal shows the active version marked with a star. This is Python 3.9.1 and exactly what I expected. Then I login into the interactive python command interpreter and this also confirmed that it is currently the 3.9.1 version.
So I manage my Pythons on my Mac OS Big Sur from now on with Pyenv.
Make Pyenv working perfectly with Homebrew
When I finished installing Pyenv everything worked fine until I tried to use the brew doctor command to check my Homebrew installations.
patrick@PatrickMBNeu ~ % brew doctor Warning: "config" scripts exist outside your system or Homebrew directories. `./configure` scripts often look for *-config scripts to determine if software packages are installed, and what additional flags to use when compiling and linking. Having additional scripts in your path can confuse software installed via Homebrew if the config script overrides a system or Homebrew provided script of the same name. We found the following "config" scripts: /Users/patrick/.pyenv/shims/python-config /Users/patrick/.pyenv/shims/python3-config /Users/patrick/.pyenv/shims/python3.9-config patrick@PatrickMBNeu ~ %
I googled a little and I found out this. Very interesting Article about the
PATH settings I enforced in my
~/.zshrc file when I installed Pyenv (see above).
Then I checked my environment variables with the command
patrick@PatrickMBNeu ~ % env TMPDIR=/var/folders/zy/3s2n4mh502z3320837q80qdm0000gp/T/ __CFBundleIdentifier=com.apple.Terminal XPC_FLAGS=0x0 TERM=xterm-256color SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.ZIRHZWPkYD/Listeners XPC_SERVICE_NAME=0 TERM_PROGRAM=Apple_Terminal TERM_PROGRAM_VERSION=440 TERM_SESSION_ID=53DE5B52-55E2-4D61-A4E4-7F47D5C401AC SHELL=/bin/zsh HOME=/Users/patrick LOGNAME=patrick USER=patrick PATH=/Users/patrick/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin SHLVL=1 PWD=/Users/patrick OLDPWD=/Users/patrick PYENV_SHELL=zsh LANG=de_DE.UTF-8 _=/usr/bin/env patrick@PatrickMBNeu ~ %
The problem seems to be the
PATH which is set by the entry
eval "$ (pyenv init -)" in my
patrick@PatrickMBNeu ~ % cat .zshrc eval "$(pyenv init -)" patrick@PatrickMBNeu ~ %
This entry leads to that
/Users/patrick/.pyenv/shims is set at the beginning of the
PATH variable when
zsh is executed.
When you look at the complete
PATH you see that the command
brew doctor can be executed without specifying
/usr/local/bin/brew because the required path specification
/usr/local/bin is also available but further back. So brew reads first the path
/Users/patrick/.pyenv/shims and only then the path
/usr/local/bin and exactly there in
/Users/patrick/.pyenv/shims are the
*-config files used from Pyenv. So it seems to be that brew has problems with these config files.
Since these config files seems to confuse brew in some way, the best would be if brew could somehow ignore these files. This is possible if we define an
alias environment variable. We can define
brew as an alias and then instruct
zsh that in the event that
brew is called a specific
PATH applies namely the
PATH without the shims directory. For this we have to adapt the
~/.zshrc as follows.
patrick@PatrickMBNeu ~ % cat .zshrc alias brew='PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin brew' eval "$(pyenv init -)" patrick@PatrickMBNeu ~ %
After restarting the terminal I was able to run the
brew doctor command without this error message. The problem is solved.
patrick@PatrickMBNeu ~ % brew update Already up-to-date. patrick@PatrickMBNeu ~ % brew doctor Your system is ready to brew. patrick@PatrickMBNeu ~ %