Rahul Sharma Rahul Sharma - 24 days ago 8
Python Question

Scrapy Pagination XHR 400 Bad Request

I am trying to fetch all url from


In one set, Page show 12 magzines and scroll paginate and proceed with next 12 magazines

After Debugging, Upcoming request are as follows


xml Request

But get request to

400 Bad Request

Is there any implementation of this scenario in scrapy please provide a sample script.

or any other library which stimulate infinite scrolling and work with scrapy framework

Answer Source

The issue is that the request is a AJAX request and not sending it X-Requested-With: XMLHttpRequest header makes it a 400 bad request. There is no way to send headers directly from shell command line, so you need to launch shell and type commands to fetch the request with headers

$ scrapy shell --nolog

>>> from scrapy import Request
>>> req = Request("https://www.magzter.com/magazines/listAllIssues/146/12", headers = {"X-Requested-With" : "XMLHttpRequest"})
>>> fetch(req)
>>> response.body
b'\r\n\t<div class="mag-wrap">\r\n    \t<h3></h3>\r\n    \t<ul class="mag-list main-magz">\r\n            <li>\r\n                        <div class="cover-wrap"><a href="https://www.magzter.com/IN/EFY-Enterprises-Pvt-Ltd/Electronics-For-You/Science/183025" onclick="ga(