user1497530 user1497530 - 1 year ago 127
Ajax Question

Scraping next page XHR request

I want to scrape the second page of this user reviews.

However the next button executes a XHR request, and while I can see it using Chrome developer tools, I cannot replicate it.

Answer Source

It's not so easy task. First of all you should install this extension. It helps you to test own requests based on captured data, i.e. catch and simulate requests with captured data.

As I see they send a token in this XHR request, so you need to get it in from html page body(stores in source code, js variable "taSecureToken" ).

Next you need to do four steps:

  1. Catch POST request with plugin
  2. Change token to saved before
  3. Set limit and offset variables in POST request data
  4. Generate request with resulted body

Note: on this request server returns json data(not the html with next page) containing info about loaded objects on next page.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download