wildthing wildthing - 1 year ago 252
Python Question

Install Scrapy on OS X El Capitan

I tried to install Scrapy on El Capitan but have not been successful yet. This happens when I use

pip install Scrapy

#include <openssl/opensslv.h>


1 error generated.

error: command 'cc' failed with exit status 1

Cleaning up...
Command /<scrapy_project>/venv/bin/python -c "import setuptools, tokenize;__file__='/<scrapy_project>/venv/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/p6/jvf54l7d5c7dntzm6d3rfc3w0000gn/T/pip-D2QIZq-record/install-record.txt --single-version-externally-managed --compile --install-headers /<scrapy_project>/venv/include/site/python2.7 failed with error code 1 in /<scrapy_project>/venv/build/cryptography

My Xcode and Xcode command tools are up to date.
I tried installing Scrapy with and without the approach via homebrew according to http://doc.scrapy.org/en/latest/intro/install.html#intro-install

Edit: I did the following:

  • brew install openssl && brew link openssl --force
    according to
    Craicerjack's suggestion

  • pip install cryptography

  • pip install scrapy

all worked without any errors. But
scrapy --version
throws this error:

ImportError: dlopen(/<scrapy_project>/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so, 2): Symbol not found: _BIO_new_CMS
Referenced from: /<scrapy_project>/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so
Expected in: flat namespace
in /<scrapy_project>/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

Answer Source

The issue here is with installation of the dependencies needed for Scrapy

first of all you should upgrade to the latest version of pip:

pip install --upgrade pip

If that doesnt work, to build cryptography and dynamically link it:

brew install openssl
env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography  

further information on installation can be found in the cryptography docs

Other issues may be solved by using the command

LDFLAGS="-L/usr/local/opt/openssl/lib" pip install cryptography --no-use-wheel

This shouldnt be necessary however, if all your software (latest pip and cryptography), is up to date. More on this issue can be found in the issues in the cryptography repo on github