user3576036 user3576036 - 3 months ago 18
Ruby Question

How do I save the scraped data from Nokogiri to a rails model database?

I want to save the scraped data to the database so that I can implement search and sorting functionality on it. I tried creating a new rake task and updating attributes but for that I need to run rake fetch-data every time the data is scraped. Anyway it didnt work. Is there any other way to achieve this? I m new to rails. Any help would be much appreciated. Thank You.


def show
url = @scrapper.url
data = Nokogiri::HTML(open(url))
@doc= data.css(".s-item-container")


<% @doc.each do |item| %>
<td><%= item.css(".s-access-title").text %></td>
<td><%= item.css(".s-price").text %></td>
<td><%= item.css("span+ .a-text-normal").text %></td>
<% end %>

The data I m getting

  1. Create a database rake db:create
  2. Crete 'Product' model rails g model Product title:string price:decimal rating:float
  3. Create a rake task. Parse data and save it in the database.
doc = Nokogiri::HTML(open(@scrapper.url ))
items = doc.css(".s-item-container")
items.each do |item|
      title: item.css(".s-access-title").text.strip,
      price: item.css(".s-price").text.to_d,
      rating: item.css("span+ .a-text-normal").text.to_f)

to prevent duplicates

items.each do |item|
    title = item.css(".s-access-title").text.strip
    product = Product.find_or_initialize(title: title)
    product.price = item.css(".s-price").text.to_d
    product.rating = item.css("span+ .a-text-normal").text.to_f!
  1. Get data from Product model in your controller and show it in the view