bababa bababa - 1 year ago 155
Python Question

Access django models inside of Scrapy

Is it possible to access my django models inside of a Scrapy pipeline, so that I can save my scraped data straight to my model?

I've seen this, but I don't really get how to set it up?

Answer Source

If anyone else is having the same problem, this is how I solved it.

I added this to my scrapy file:

def setup_django_env(path):
    import imp, os
    from import setup_environ

    f, filename, desc = imp.find_module('settings', [path])
    project = imp.load_module('settings', f, filename, desc)       



Note: the path above is to your django project folder, not the file.

Now you will have full access to your django models inside of your scrapy project.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download