Waqas Shah Waqas Shah - 1 year ago 49
Ruby Question

Ruby on Rails- How to scrape data which is loaded on page after scrolling down?

I am trying to get the title and price of all the products in the following link by scraping.

I am able to get the title and price of products but the problem is new products are added as we scroll down the page and URL of page is not changed.So I am not able to scrape info of products that are loaded after the page is scrolled down.
This is the code that I am using

url = "https://homeshopping.pk/categories/Mobile-Phones-Price-Pakistan"

open('home.out', 'w') { |f|

url_temp = url
doc = Nokogiri::HTML(open(url_temp))
doc.css(".product-box").each do |name|

$Name = name.at_css(".ProductDetails").text
$Price = "Rs. " + name.at_css(".price").text

f.puts $Name
f.puts $Price


Answer Source

Inspect the ajax requests, that are called when new part of data is loaded. They use pagination (page=2):


Just iterate throught pagination by incrementing page param.