Lukus Lukus - 28 days ago 12
Ruby Question

How to scrape and import data to existing db?

I just finished my first rails app, a yelp clone. Now I want to scrape data from yelp or use a webcrawler to add their data to my database. I have looked into scraping it and saving it to a excel then using the gem roo to read the excel file but I'm unsure this would work. I have looked into Nokogiri but I'm unsure if this will work. I have seen some web crawlers gems too, but I am unsure if they are what I need. Getting the images I have no idea how to do, also to get data from several pages maybe 20. I want to add all restaurants in Atlanta but I'm unsure if I can make that many calls without getting in trouble or having to pay.
The required fields I need are.
Name Address Phone Website Image

Here is my app: https://yelping-it-up.herokuapp.com/

Any additional information is appreciated.

Answer

You're looking for a gem called watir.

https://github.com/watir/watir

You don't have to deal with XLS files, you should probably just use .csv and the CSV library if you're going to go that route.

Honestly though, you should just use sqlite or whatever other database you're using to store the information. You can export that data to CSV later.

Comments