bababa bababa - 1 year ago 129
Python Question

Access django models inside of Scrapy

Is it possible to access my django models inside of a Scrapy pipeline, so that I can save my scraped data straight to my model?

I've seen this, but I don't really get how to set it up?

Answer Source

If anyone else is having the same problem, this is how I solved it.

I added this to my scrapy file:

def setup_django_env(path):
    import imp, os
    from import setup_environ

    f, filename, desc = imp.find_module('settings', [path])
    project = imp.load_module('settings', f, filename, desc)       



Note: the path above is to your django project folder, not the file.

Now you will have full access to your django models inside of your scrapy project.