Rahul Sharma Rahul Sharma - 24 days ago 8
Python Question

Scrapy Pagination XHR 400 Bad Request

I am trying to fetch all url from

https://www.magzter.com/magazines/listAllIssues/503


In one set, Page show 12 magzines and scroll paginate and proceed with next 12 magazines

After Debugging, Upcoming request are as follows

https://www.magzter.com/magazines/listAllIssues/503/12
https://www.magzter.com/magazines/listAllIssues/503/24


xml Request

But get request to
https://www.magzter.com/magazines/listAllIssues/503/12
through

400 Bad Request


Is there any implementation of this scenario in scrapy please provide a sample script.

or any other library which stimulate infinite scrolling and work with scrapy framework

Answer Source

The issue is that the request is a AJAX request and not sending it X-Requested-With: XMLHttpRequest header makes it a 400 bad request. There is no way to send headers directly from shell command line, so you need to launch shell and type commands to fetch the request with headers

$ scrapy shell --nolog

>>> from scrapy import Request
>>> req = Request("https://www.magzter.com/magazines/listAllIssues/146/12", headers = {"X-Requested-With" : "XMLHttpRequest"})
>>> fetch(req)
>>> response.body
b'\r\n\t<div class="mag-wrap">\r\n    \t<h3></h3>\r\n    \t<ul class="mag-list main-magz">\r\n            <li>\r\n                        <div class="cover-wrap"><a href="https://www.magzter.com/IN/EFY-Enterprises-Pvt-Ltd/Electronics-For-You/Science/183025" onclick="ga(