Scientific Python on Mac OS X 10.9+ with homebrew

There are (too) many guides out there about how to install Python on Mac OS X. So why another one? Cause i found most of the other ways to lead to long-term maintenance debt that is unnecessary and can easily be avoided. Also many of the other guides aren’t very easy to extend in “but i also need that library”-cases. So here I’ll briefly explain how to set-up a scientific python stack based on homebrew. The advantages are that this stack will never conflict with your system’s core and homebrew opens up a whole new world of easy to access unix tools via a simple brew install ....

This step-by-step installation guide to setup a scientific python environment has been tested on Mac OS X Mavericks 10.9 / Yosemite 10.10 / El Capitain 10.11 / Sierra 10.12. It will probably also work in the following versions (if not, let me know in the comments).
An older version of this setup guide can be found here: Scientific Python on Mac OS X 10.8 with homebrew, main changes: rename of a tap + changes wrt. openblas as Accelerate was fixed in OS X 10.9

Needless to say: Make a backup (Timemachine)

First install homebrew.
Follow their instructions, then come back here.

If you don’t have a clean install, some of the following steps might need minor additional attention (like changing permissions chmod, chown, chgrp or overwriting existing files in the linking step with brew link --overwrite package_that_failed. In this case i can only recommend a backup again).

In general: execute the following commands one line at a time and read the outputs! If you read some warnings about “keg-only” that’s fine, it just means that brew won’t “hide” your system’s stuff behind the stuff it installed itself so it doesn’t cause problems… brewed stuff will still use it.

# set up some taps and update brew
brew tap homebrew/science # a lot of cool formulae for scientific tools
brew tap homebrew/python # numpy, scipy, matplotlib, ...
brew update && brew upgrade

# install a brewed python
brew install python

# and/or if you want to give python3 a go (it's about time):
#brew install python3  # repeat/replace the following pip commands with pip3

A word about brewed python: this is what you want!

It’s more up to date than the system python, it will come with pip and correctly install in the brew directories, working together well with brewed python libs that aren’t installable with plain pip. This also means that pip by default will work without sudo as all of homebrew, so if you ever run or have to run sudo pip ... because of missing permissions, then you’re doing it wrong! Also, don’t be afraid of multiple pythons on your system: you have them anyhow (python2 and python3) and it’s an advantage, as we’ll make sure that nothing poisons your system python and that you as a user & developer will use the brewed python:

hash -r python  # invalidate bash's lookup cache for python
which python
# should say /usr/local/bin/python

echo $PATH
# /usr/local/bin should appear in front of /usr/bin

If this is not the case you’d probably end up not using brewed python. Please check your brew install with brew doctor, it will probably tell you that you should consider updating your paths in ~/.bashrc. You can either follow its directions or create a ~/.profile file like this one: ~/.profile. If you performed these steps, please close your terminal windows and open a new one for the changes to take effect. Test the above again.

Even if the above check worked, run the following anyhow and read through its output (no output is good):

brew doctor

Pay special attention if this tells you to install XQuartz, and if it does, install it! You’ll need it anyhow…

Now after these preparations, let’s really start installing stuff… below you’ll mostly find one package / lib per line. For each of them and for their possible options: they’re a recommendation that might save you some trouble, so i’d recommend to install all of them as i write here, even if specifying some of the options will compile brew packages from source and take a bit longer…

# install PIL, imagemagick, graphviz and other
# image generating stuff
brew install libtiff libjpeg webp little-cms2
pip install Pillow
brew install imagemagick --with-fftw --with-librsvg --with-x11
brew install graphviz --with-librsvg --with-x11
brew install cairo
brew install py2cairo # this will ask you to download xquartz and install it
brew install qt pyqt5

# install virtualenv, nose (unittests & doctests on steroids)
pip install virtualenv
pip install nose

# install numpy and scipy
# nowadays there are two good ways to install numpy and scipy: via pip or via brew.
# PICK ONE:
# - i prefer pip for proper virtualenv support and more up-to-date versions.
# - brew builds are a bit older, but handy if you need a source build
pip install numpy
pip install scipy
# OR:
# (if you want to run numpy and scipy with openblas also remove comments below:)
#brew install openblas
#brew install numpy # --with-openblas
#brew install scipy # --with-openblas

# test the numpy & scipy install
python -c 'import numpy ; numpy.test();'
python -c 'import scipy ; scipy.test();'

# some cool python libs (if you don't know them, look them up)
# matplotlib: generate plots
# pandas: time series stuff
# nltk: natural language toolkit
# sympy: symbolic maths in python
# q: fancy debugging output
# snakeviz: cool visualization of profiling output (aka what's taking so long?)
#brew install Caskroom/cask/mactex  # if you want to install matplotlib with tex support and don't have mactex installed already
brew install matplotlib --with-cairo --with-tex  # cairo: png ps pdf svg filetypes, tex: tex fonts & formula in plots
pip install pandas
pip install nltk
pip install sympy
pip install q
pip install snakeviz

# ipython/jupyter with parallel and notebook support
brew install zmq
pip install ipython[all]

# html stuff (parsing)
pip install html5lib cssselect pyquery lxml BeautifulSoup

# webapps / apis (choose what you like)
pip install Flask Django tornado

# semantic web stuff: rdf & sparql
pip install rdflib SPARQLWrapper

# graphs (graph metrics, social network analysis, layouting)
pip install networkx

# maintenance: updating of pip libs
pip list --outdated  # see Updating section below

Have fun 😉

Updating

OK, let’s say it’s been a while since you installed things with this guide and you now want to update all the installed libs. To do this you should first upgrade everything installed with brew like this:

brew update && brew outdated && brew upgrade

Afterwards for upgrading pip packages (globally or in a virtualenv) you can just run

pip list --outdated

to get a list of outdated packages and then manually update them with:

pip install -U package1 package2 ...

If you want a tiny bit more comfort you can use the pip-review package to do this interactively:

pip install pip-review

Once installed you should be able to run the following either in a virtualenv or globally for your whole system:

pip-review -i # for interactive mode, -a to upgrade all which is dangerous

It will check your installed packages for new versions and give you a list of outdated packages. I’d recommend to run it with the -i option to interactively install the upgrades.

A word of warning about the brewed packages: If i recommended to install a package with brew above that’s usually for a good reason like the pip version not working properly. If you’re a bit more advanced, you can try to upgrade them with pip, but i’d recommend to properly unlink them with brew unlink <package> before, as some pip packages might run into problems otherwise. If you find the pip package works like a charm then, please let me know in the comments below so i can update this guide. In general i prefer the pip packages as they’re more up to date, work in virtual environments and can then easily be updated with pip-review.

As always: If you liked this, think something is missing or wrong leave a comment.

Updates to this guide:

2014-03-02: include checking of $PATH for Mike
2015-03-17: enhanced many explanations, provided some useful options for packages, general workover
2015-04-15: included comment about installing mactex via cask if not there already, thanks to William
2015-06-05: Pillow via pip and Updating section
2015-11-01: pip-review (was detached from pip-tools) + alternative
2016-02-15: hash -r python (invalidate bash python bin lookup cache)
2016-12-21: updates for sierra, brew upgrade, python3 and some more comments
2017-03-30: updated pyqt’s package name to pyqt5

47 thoughts on “Scientific Python on Mac OS X 10.9+ with homebrew

  1. Pingback: Scientific Python on Mac OS X 10.8 with homebrew | Jörn's Blog

  2. Pingback: 20140714 Monday Numpy, PyQt4, Pyaudio | YOUNG

  3. Pingback: Python | Programmer Tea Time

  4. Lil

    This is really good thank you! But I had problems with Scipy,

    [cc_bash]
    python -c import scipy; assert not scipy.test().failures
    Python version 2.7.9 (default, Jan 7 2015, 11:50:42) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)]
    nose version 1.3.3
    Traceback (most recent call last):
    File “”, line 1, in
    AssertionError
    Error: scipy: failed
    Failed executing: python -c import\ scipy;\ assert\ not\ scipy.test().failures
    /usr/local/Library/Homebrew/formula.rb:707:in `block in system’
    /usr/local/Library/Homebrew/formula.rb:664:in `open’
    /usr/local/Library/Homebrew/formula.rb:664:in `system’
    /usr/local/Library/Taps/homebrew/homebrew-python/scipy.rb:88:in `block (2 levels) in ‘
    /usr/local/Library/Homebrew/language/python.rb:25:in `call’
    /usr/local/Library/Homebrew/language/python.rb:25:in `block in each_python’
    /usr/local/Library/Homebrew/language/python.rb:17:in `each’
    /usr/local/Library/Homebrew/language/python.rb:17:in `each_python’
    /usr/local/Library/Taps/homebrew/homebrew-python/scipy.rb:87:in `block in ‘
    /usr/local/Library/Homebrew/formula.rb:617:in `block in run_test’
    /usr/local/Library/Homebrew/extend/fileutils.rb:21:in `mktemp’
    /usr/local/Library/Homebrew/formula.rb:614:in `run_test’
    /usr/local/Library/Homebrew/cmd/test.rb:38:in `block (2 levels) in test’
    /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/timeout.rb:66:in `timeout’
    /usr/local/Library/Homebrew/cmd/test.rb:37:in `block in test’
    /usr/local/Library/Homebrew/cmd/test.rb:15:in `each’
    /usr/local/Library/Homebrew/cmd/test.rb:15:in `test’
    /usr/local/Library/brew.rb:135:in `’
    [/cc_bash]

    Can you suggest anything to remedy this?

    Reply
    1. joern Post author

      after updating to the recent version i had a similar problem identified by the
      [cc_bash]ImportError: No module named _csr[/cc_bash]
      Following the discussion in https://github.com/Homebrew/homebrew-python/issues/185#issuecomment-67534979 it seems this is caused by a leftover from previous installs and can be fixed with:
      [cc_bash]find /usr/local/lib/python2.7/site-packages/scipy -name ‘*.pyc’ -delete[/cc_bash]

      After this [cci_bash]ipython -c ‘import scipy; scipy.test()'[/cci_bash] seems to only report one failure, namely this one: https://github.com/python-pillow/Pillow/issues/1074 . Seems this is fixed in >0.15.0 and a false positive, so either wait for brew to pick up scipy 0.15 or just ignore 😉

      Reply
  5. Pingback: Coding theano under remote ubuntu server from local Mac (在本地mac机器上,写、跑、调试、看-远程ubuntu上的theano代码) | 挖沙啦挖沙啦

  6. Pingback: Python Scientific Computing on Mac OS X Yosemite | Tung-Wei Lin

  7. Sabina

    I’ve been struggling with this the whole day, until stumbling upon this. Seriously amazing – thank you!

    Reply
  8. William Burke

    I just ran into a problem with this. You need LaTeX installed before you can install `matplotlib`. That’s easy to fix with `brew install Caskroom/cask/mactex`. Although it appears to take forever!

    Reply
    1. Jim Corson

      I finally got mactex to install, but still can’t get matplotlib to install –with-tex. I just removed the –with-tex to install without latex and it worked fine. I hardly ever use tex anyway.

      Reply
        1. jaust

          I had the same problem. Installed MacTex using:
          brew install Caskroom/cask/mactex

          Installed with no errors:
          brew install Caskroom/cask/mactex
          ==> brew cask install Caskroom/cask/mactex
          ==> Downloading http://tug.org/cgi-bin/mactex-download/MacTeX.pkg
          ######################################################################## 100.0%
          ==> No checksum defined for Cask mactex, skipping verification
          ==> Running installer for mactex; your password may be necessary.
          ==> Package installers may write to any location; options such as --appdir are ignored.
          Password:
          ==> installer: Package name is MacTeX-2015
          ==> installer: Installing at base path /
          ==> installer: The install was successful.

          Then when installing matlib it said:

          brew install matplotlib --with-cairo --with-tex
          ==> Installing matplotlib from homebrew/python
          matplotlib: A LaTeX distribution is required for Homebrew to install this formula.

          Make sure that "/usr/texbin", or the location you installed it to, is in
          your PATH before proceeding.

          You can install with Homebrew Cask:
          brew install Caskroom/cask/mactex

          You can download from:
          https://www.tug.org/mactex/
          matplotlib: `dvipng` not found. This is optional for Matplotlib.

          You can install with Homebrew Cask:
          brew install Caskroom/cask/matctex
          Error: An unsatisfied requirement failed this build.

          Reply
          1. joern Post author

            this looks like a path problem to me… can you run [cci]brew doctor[/cci] and [cci]echo $PATH[/cci]?

          2. jaust

            OK – I solved this problem. It’s necessary to open a new shell. The $PATH variable didn’t contain the Tex path in it. When you open a new shell, it should be there.

          3. jaust

            joern, I didn’t notice your reply before I posted mine. Yes – that seems to have been the issue. I opened a new shell window and all of a sudden there were a lot more entries in echo $PATH than there were before. And it included /Library/TeX/texbin.

    1. joern Post author

      yes, i think you can just run [cci_bash]brew install numpy –with-openblas –with-python3[/cci_bash].
      You can see these flags with [cci_bash]brew info numpy[/cci_bash].

      Reply
  9. Ric

    Thank you, it helps me a lot. Now I’m able to use theano with python. However, when I try to import theano with ipython I got the following error:

    /Library/Python/2.7/site-packages/theano/gof/compiledir.py in ()
    1 from __future__ import print_function
    —-> 2 import six.moves.cPickle as pickle
    3 import errno
    4 import logging
    5 import os

    ImportError: No module named cPickle

    I have tried to change line 2 in the cmodule.py from
    import six.moves.cPickle as pickle to import six.moves.pickle as pickle
    and also to
    import six.moves.pickle

    Any idea of what is wrong?

    Reply
    1. joern Post author

      I’m not exactly sure, but it looks a lot like you didn’t install theano after this guide / didn’t properly follow this guide. If you had, the path would not be [cci]/Library/Python2.7/site-packages[/cci] (that is the system path), but more like [cci]/usr/local/lib/python2.7/site-packages/[/cci] (that is the brew python’s system wide path).

      I’d uninstall all packages that you installed under [cci]/Library/Python2.7/site-packages[/cci] and then re-install them with brew python’s pip. Before you do so, make sure that you’re actually running brew python’s pip: [cci]which pip[/cci] on your commandline should return [cci]/usr/local/bin/pip[/cci]. Also when running [cci]which python[/cci] it should return [cci]/usr/local/bin/python[/cci].

      Reply
  10. jl

    Thanks for the information, good job.
    I have a problem with the graph-tool installation. As you can see all is prepared and allready to install:

    | => brew info graph-tool
    homebrew/science/graph-tool: stable 2.7 (bottled), HEAD
    http://graph-tool.skewed.de/
    Not installed
    From: https://github.com/Homebrew/homebrew-science/blob/master/graph-tool.rb
    ==> Dependencies
    Build: pkg-config ✔
    Required: boost ✔, boost-python ✔, cairomm ✔, cgal ✔, py2cairo ✔, gnome-icon-theme ✔, librsvg ✔, pygobject3 ✔
    Recommended: google-sparsehash ✔, gtk+3 ✔, homebrew/python/numpy ✔, homebrew/python/scipy ✔, homebrew/python/matplotlib ✔

    But the installation never finish:

    | => brew install graph-tool
    ==> Installing graph-tool from homebrew/homebrew-science
    ==> Installing dependencies for homebrew/science/graph-tool: librsvg
    ==> Installing homebrew/science/graph-tool dependency: librsvg
    ==> Downloading https://download.gnome.org/sources/librsvg/2.40/librsvg-2.40.10.tar.xz
    Already downloaded: /Library/Caches/Homebrew/librsvg-2.40.10.tar.xz
    ==> ./configure –prefix=/usr/local/Cellar/librsvg/2.40.10_1 –disable-Bsymbolic –enable-tools=yes –enable-pixbuf-loader=yes –enable-introspection=yes
    ==> make install gdk_pixbuf_binarydir=/usr/local/Cellar/librsvg/2.40.10_1/lib/gdk-pixbuf-2.0/2.10.0/loaders gdk_pixbuf_moduledir=/usr/local/Cellar/librsvg/2.40.10_1/lib/gdk-pixbuf-2.0/2.10.0/loaders

    It never finish the “make install”. Do you know how to solve it?

    Thanks

    Reply
    1. joern Post author

      hmmm, what you posted doesn’t seem to include an error message, sorry… maybe open an issue in github?

      Reply
  11. Steve

    This worked perfectly on El Capitan. Make sure you’ve updated Xcode (via the App Store) and your Xcode command line tools:

    xcode-select –install

    I picked the pip install path for scipy and numpy; you will need to run `brew link numpy`

    Reply
  12. Yannis Haralambous

    It worked fine, and thanks for the useful advice.
    But although I installed pip-tools, I don’t have access to any pip-review command…
    But I do have a pip-compile and a pip-sync (as explained in https://github.com/nvie/pip-tools)
    On that Web site there is no mention of pip-review whatsoever, doesn’t it exist?
    Thanks again

    Reply
    1. joern Post author

      yes, the pip-tools were updated since writing this guide and don’t provide the pip-review command anymore. I updated the guide.

      Reply
  13. JJ

    Thanks for the fantastic guide! However, pip’s numpy gave the following errors during the test, any ideas how to fix this:

    ERROR: test_scripts.test_f2py
    ———————————————————————-
    Traceback (most recent call last):
    File “/usr/local/lib/python2.7/site-packages/nose/case.py”, line 197, in runTest
    self.test(*self.arg)
    File “/usr/local/lib/python2.7/site-packages/numpy/testing/decorators.py”, line 146, in skipper_func
    return f(*args, **kwargs)
    File “/usr/local/lib/python2.7/site-packages/numpy/tests/test_scripts.py”, line 68, in test_f2py
    code, stdout, stderr = run_command([f2py_cmd, ‘-v’])
    File “/usr/local/lib/python2.7/site-packages/numpy/tests/test_scripts.py”, line 48, in run_command
    proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
    File “/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py”, line 710, in __init__
    errread, errwrite)
    File “/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py”, line 1335, in _execute_child
    raise child_exception
    OSError: [Errno 2] No such file or directory

    ———————————————————————-
    Ran 5919 tests in 34.415s

    FAILED (KNOWNFAIL=3, SKIP=8, errors=1)

    Scipy test output gave:
    Ran 18421 tests in 125.988s

    OK (KNOWNFAIL=97, SKIP=1183)

    Is this fine?

    Reply
  14. Frisbee

    Thank you SO MUCH for clearing this up. I like playing with all the new python packages as they come out and I had like 8 versions of python installed that weren’t all working with my pip installed libraries. This is the grunt work that isn’t so fun to blog about, but THANK YOU again.

    Reply
  15. Dmitry

    As of now, on some newer Mac systems you can stumble upon breaked python install in brew. When runninng numpy/scipy test, you can get this error:

    ImportError: dlopen(/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
    Referenced from: /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
    Expected in: flat namespace

    The easy way to handle this is to downgrade to 2.7.10. For this you’ll need to do following:
    $ brew unlink python
    $ cd /usr/local/Library/
    $ git checkout 10e9b5bf267010a34bad60a392206571909fedef /usr/local/Library/Formula/python.rb
    $ brew install python

    and voila!

    $ python –version
    Python 2.7.10

    Reply
    1. joern Post author

      didn’t have that problem after upgrading to Python 2.7.11 myself, but thanks for the feedback. Note that the fix you mention is only a temporary solution and might cause conflicts when updating brew… if you really need an older python version for whatever reason, i’d recommend installing pyenv instead (brew install pyenv) and configure it as you wish.

      Reply
      1. ozymandias

        I also ran into this problem. The root cause is explained here in superbatfish’s response to one of the comments: http://stackoverflow.com/questions/34386527/symbol-not-found-pycodecinfo-getincrementaldecoder-trying-to-register-a-packa

        The relevant comment: “The fundamental problem here is that bash’s hash cache can become incorrect immediately after you install a new version of python. (Try type python to verify that’s the problem.) The person you quoted may have accidentally fixed this via all the uninstalling/reinstalling they described, but the straightforward fix is to type hash -r python into the terminal. – superbatfish”

        rehashing python fixed this for me without having to downgrade to 2.10.
        Hope this helps someone out there

        Reply
  16. Isabelle Phan

    This helped me greatly fix my python environment after upgrading to El Capitan.

    Thank you.

    Reply
  17. bum

    Thank you !!!!
    hash -r python saved my day. In the middle of writing a research paper I though I screwed up everything.

    Thanks again.

    Reply
  18. Abdul Basit Ahmad

    I think this installs python2.7, I want to install python3.
    Would simply saying:
    brew tap homebrew/python3
    be sufficient? Can you tell me how to install this setup with python 3?

    Reply
  19. Vishal Kumar Singh

    I m not able to install mathplotlib.I have installed mactex but still i am getting the error. Error is given below:
    ==> Installing matplotlib from homebrew/python
    matplotlib: A LaTeX distribution is required for Homebrew to install this formula.

    Make sure that “/usr/texbin”, or the location you installed it to, is in
    your PATH before proceeding.
    TeXRequirement unsatisfied!

    You can install with Homebrew-Cask:
    brew cask install mactex

    You can download from:
    https://www.tug.org/mactex/
    matplotlib: dvipng not found. This is optional for Matplotlib.
    DvipngRequirement unsatisfied!

    You can install with Homebrew-Cask:
    brew cask install matctex
    Error: An unsatisfied requirement failed this build.

    My system had Mac OS Sierra. v 10.12

    Can u help me with this please.

    Reply
    1. joern Post author

      if you don’t have mactex i guess you don’t want latex support, just remove the –with-tex option then

      Reply
  20. Kanpi

    Looks like you don’t need –all in the brew update:

    myname-MBP:~ myname$ brew update && brew outdated && brew upgrade –all
    Warning: We decided to not change the behaviour of brew upgrade so
    brew upgrade --all is equivalent to brew upgrade without any other
    arguments (so the --all is a no-op and can be removed).

    Also, is there any particular reason you suggest brew for numpy and scipy? It looks like they install older versions. Also what is the advantage to openblas? I got an error when using “brew install numpy –with openblas” and just went with pip.

    Reply
    1. joern Post author

      thanks for the feedback. Updated the --all and clarified brewing numpy and scipy

      Reply
  21. Charlie Munro

    Thanks for the excellent step-by-step instructions and keeping this page up to date! And also for the previous post – John Cleese on Creativity – which made excellent viewing while the installs did their thing. I commented on John Cleese post with new URL for the video.

    Only one minor change to your steps:
    $ brew install qt pyqt
    Error: No available formula with the name “pyqt”
    ==> Searching for similarly named formulae…
    This similarly named formula was found:
    pyqt5
    To install it, run:
    brew install pyqt5

    I ran brew install qt pyqt5 and all seems well.

    Thanks again!

    Reply
    1. joern Post author

      yes, seems this is the updated module’s name. thanks for reporting 🙂

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

 

This site uses Akismet to reduce spam. Learn how your comment data is processed.