Ajay Choudhary Ajay Choudhary - 1 year ago 120
Python Question

Scrapy for dynamic contect

Can we use Scrapy for getting content from a web page which is loaded by java script ?

I'm trying to scrap usage examples from this page
but since they are loaded using java script as a Jason object I'm not able to get them Scrapy.

Could u suggest what is the best way to deal with such issues?

Answer Source

Open your browser's developer tools and look at the Network tab. If you hit the "next" button on that page enough, it'll send out a new request:


After removing the JSONP paramter, the URL is pretty straightforward:


By making the minimal number of requests, your spider will be fast.

If you want to just emulate a full browser and execute the JavaScript, you can use something like Selenium or Scrapinghub's Splash (and its corresponding Scrapy plugin).

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download