Ravshan Kh Ravshan Kh - 2 months ago 7
Javascript Question

How to continue running Rails app from where I left last time?

I've been working on a scraper on Ruby on Rails which uses Capybara/Poltergeist to save data into model attributes in a MongoDB database. However the dataset is huge and will take a lot of hours and sometimes I might have distractions like Internet connection and have to start from scratch. Could you please help me to find a method for the app to continue inserting elements to database from where I left last time.

Simplified excerpt of my code is:

naming.each_with_index do |namee, j|


sleep 2

@session.within_frame('content_consist') do
sleep 3
@session.within("form[name=ConsistList]") do
@session.all('input').each do |z|
@component[:partname] << z.value




And each such @komponent is saved in MongoDB as:

{ "_id" : ObjectId("57ad5b7f678208560bcb21bd"), "component" : "2U-9747 - WARNING GP-WIDE VEHICLE", "partname" : [ "PLATE-MARKER WIDTH ", "BOLT ", "NYLOC NUT ", "WASHER ", "PLATE MARKER FRONT ", "PLATE MARKER FRONT ", "SCREW DRIVE PAN " ]}

I want the app look at the latest database entry, and keep on adding new elements after the last one, not starting the scraper from the beggining.

Thank you.


I have come up with the following solution: added a boolean attribute :visited, default=false to a model through which I have been looping. And at the end of each iteration I have set :visited=true. And my loop was not Class.all each do |x| but rather Class.where(visited=false).map do |x|; Therefore it would continue from where it stopped last time.