Winterflags Winterflags - 5 months ago 46
Python Question

Messy Python install? (OS X)

I am a complete beginner to both Python and the OS X Terminal, and I have attempted to install some packages for both Python 2.7.3 and Python 3.4.

I can't get mechanize to work with neither Python 2 or Python 3 after install. I get:

>>> from mechanize import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/XXX/Desktop/mechanize-0.2.5/mechanize/__init__.py", line 119, in <module>
from _version import __version__
ImportError: No module named '_version'


In trying to understand why it's searching for the module location on my Desktop (where I downloaded the source code and ran setup.py from), I'm wondering if my Python installs are messy.

I have located Python stuff at these three different locations I know of:


Macintosh HD/Library/Frameworks/Python.framework/3.4/ – [This is where Python 3 and its site-packages are stored]

Macintosh HD/System/Library/Frameworks/Python.framework/Versions/2.7/ – [I can't find any site-packages here]

Macintosh HD/Library/Python/2.7/ – [Only site-packages in this folder, nothing else]


Is the last one misplaced?

I should perhaps mention that I'm utilizing a non-admin/non-sudo user in Terminal per default. So when I'm installing packages, I do:

su -l admin
sudo easy_install xxx or sudo python3 setup.py install


Ideally, I would like to clean up my Python install, remove the packages I have installed, and place everything virtually with virtualenv instead.

Answer

OS X comes with Python pre-installed. Exactly which version(s) depends on your OS X version (e.g., 10.10 comes with 2.6 and 2.7, 10.8 comes with 2.5-2.7 plus a partial installation of 2.3, and 10.3 comes with 2.3).

These are installed in /System/Library/Frameworks/Python.framework/Versions/2.*, and their site-packages are inside /Library/Python/2.*. (The reason they're in different places is that /System/Library should only be written to by OS installs/upgrades.)

You can't remove the pre-installed versions of Python without potentially breaking the OS (and, even if you could, they'd just get reinstalled at the next OS update).

But if you're only planning to use Python 3.4, you can just ignore the 2.x versions that Apple gave you. All versions of at least Python 3.2+ that come from python.org or other major sources like Homebrew will follow PEP 394, meaning that you'll get python3, pip3, etc. commands that do not collide with the python, etc. commands installed by Apple.

Also, you mentioned using virtual environments in your question. This is a good idea. Whether you use the stdlib's venv or the third-party virtualenv, you can create separate Python 3.4 environments. When you're inside an active virtual environment, its pip will install into its site-packages instead of the global one. And if you really screw up an environment, you can clean it up very easily just by deactivating and doing rm -rf path/to/environment and recreating it.

As a side note, you almost never want to use easy_install. If you install Python 3.4, it comes with pip, so the proper way to install packages for it is:

[sudo] pip3 install xxx

Even when you have a setup.py, unless it's non pip-compatible (most of them are nowadays, but a few aren't), you probably want to use it instead of running setup.py manually, like this:

[sudo] pip3 install .

Also, the python.org installers have an option to make the site-packages directory group-writable. If you've enabled that, you should not use sudo, because then you'll end up with a site-packages that's a mix of writable and non-writable, and uninstalling or upgrading will become a nightmare. (One nice advantage of using this option is that never using sudo means you will never accidentally install anything for Apple's Python 2.7, because you'd get a permissions error if you tried. But the disadvantages are obvious. That's why it's an option.)


So, how do you get there from here?

  • You may want to pip list > requirements or pip freeze > requirements to get a list of all the packages you have installed. (The former is more human-readable, the latter can be used directly with pip install -r requirements to automatically reinstall the same set of packages once you've cleaned up.) You may want to do this with both 2.7 and 3.4 if you've pip installed for both and gotten things confused enough.
  • Uninstall your Python 3.4.
  • Uninstall any site-packages you installed for Apple's Python 2.7. (You already know where these are, so, even though you didn't install them with pip, you can uninstall them manually. Just be aware that they may have left behind scripts like ipython or similar in /usr/local/bin.)
  • Install the latest Python 3.4 from python.org.
  • [sudo] pip3 install all the packages you want globally. (This may be nothing at all, or just virtualenv.)
  • Use the venv or virtualenv to create a virtual environment for each project you want to keep separate, and use its pip rather than the global one to install packages for that specific environment.