ysaditya ysaditya - 2 months ago 18
Java Question

Download in java a webpage which makes AJAX call to load

I am trying to get the Hotel reviews from different websites.

For Simple plain HTML Web Pages(like TripAdvisor ) i used JSoup and did like this

Jsoup.connect("foo.html").get();


For Pages which used Javascript for Loading (like Expedia ) i used a Selenium WebDriver and did something like

driver.get("foo.html");
driver.manage().timeouts().pageLoadTimeout(10,TimeUnit.SECONDS)


They were fine because they had links and i could use those links to crawl and get more reviews.
like this

and

and like this

The problem i face is when Downloading from pages which make AJAX calls (like MakeMyTrip ).

Here i do not know how to download the web page as the hotels list which is there on the page and keeps coming as we scroll down. sample

Any Suggestions would be of great help.

Answer

Solved it by using the url to which ajax call is done.

Example:

For a hotel with Id 200703241029455940 ( which comes from the main page) we get the review from the url.

http://hotelz.makemytrip.com/makemytrip/site/hotels/detail/responsive/hotelMmtReviews?hotelId=200703241029455940&start=10&rows=10&reviewsType=all

A GET request to the URL returns back a JSON array of reviews and thus i could extract the hotel reviews .

Comments