ysaditya ysaditya - 25 days ago 7x
Java Question

Download in java a webpage which makes AJAX call to load

I am trying to get the Hotel reviews from different websites.

For Simple plain HTML Web Pages(like TripAdvisor ) i used JSoup and did like this


For Pages which used Javascript for Loading (like Expedia ) i used a Selenium WebDriver and did something like


They were fine because they had links and i could use those links to crawl and get more reviews.
like this


and like this

The problem i face is when Downloading from pages which make AJAX calls (like MakeMyTrip ).

Here i do not know how to download the web page as the hotels list which is there on the page and keeps coming as we scroll down. sample

Any Suggestions would be of great help.


Solved it by using the url to which ajax call is done.


For a hotel with Id 200703241029455940 ( which comes from the main page) we get the review from the url.


A GET request to the URL returns back a JSON array of reviews and thus i could extract the hotel reviews .