user3281831 user3281831 - 2 months ago 15
Ruby Question

How to save results to file after every API call with ruby?

I use this short ruby script to scrape twitter. Problem is that scraping large accounts takes huge amount of time and because Im still testing it, I cannot see how it works with big accounts. For example I let it run for whole day and it did not save anything, so I need to save the results after every API call and not only after the whole script finishes. Thanks to anyone who can help me, or at least point me in right direction. here is the code:

def fetch_all_followers(twitter_username)
CSV.open("#{twitter_username}_friends_list.txt", 'w') do |csv|
@client.follower_ids(twitter_username).each_slice(5000).with_index do |slice, i|
@client.users(slice).each_with_index do |f, j|
csv << [i * 5000 + j + 1,f.screen_name]
sleep 5
end
end
end
end

fetch_all_followers("SOME_TWITTER_ACCOUNT")

Answer

The rule of thumb: do not use redundant 3rd parties, it’s being called “leftpad syndrome.”

def fetch_all_followers(twitter_username)
  fname = "#{twitter_username}_friends_list.txt"
  @client.follower_ids(twitter_username)
         .each_slice(5000)
         .with_index do |slice, i|
    @client.users(slice)
           .each_with_index do |f, j|
              File.open(fname, "a+") do |file|
      file.write [i * 5000 + j + 1, f.screen_name, "\n"].join(',')
      sleep 5
    end
  end
end

fetch_all_followers("SOME_TWITTER_ACCOUNT")
Comments