Friday, June 10, 2016

Telling Python to prefer your user's private directory over the global library directory

Just a quick note since this caused quite a bit of frustration for me over the last couple of days:

I usually install Python modules using 'pip install --user <module>' since this is nice and safe, and I don't need to sudo to do it. These modules are installed, by default, to ${HOME}/.local/lib/python<version>/site-packages. This has served me well over the years, and it's something that worked before pip going back to easy_install and even before when running 'python install --user' manually.

Recently, trying to debug problems I am having installing TensorFlow, I noticed that the global packages were taking priority over my user-local modules in the cases where the module was installed in both. eg if I
    >>> import six
    >>> six.__file__
even though I have six installed locally. Note, this is the debian-custom global location for Python modules. I don't think the fact that I am using a Debian-derivative modifies this story, but I am not completely sure.

After a lot of Googling and looking through, this ended up being due to:

  1. Having ENABLE_USER_SITE = None in /usr/lib/python2.7/ although I don't remember ever having changed this before and this used to work ok. Anyway, set this to True
  2. More importantly, there was a row in ${HOME}/.local/lib/python2.7/easy-install.pth saying '/usr/lib/python2.7/dist-packages'. All these directories get pre-pended to sys.path, so this sneaks the global path above my user-local. Removing this line fixes the problem.
/usr/lib/python2.7/dist-packages gets added to sys.path anyway later in, but it gets appended instead of pre-prended, so all works out well.