I'm building an app that parses the contents of a CSV into a number of "readings". It then posts these readings to an REST API server where it's added to a database.
So far so good (and the above is working). However, I'm conscious that there may occasionally be connectivity issues with the server (delays, or if the API goes down for any reason).
To address this, I planned on using Sidekiq to perform the transmission asynchronously. However, I have a couple of questions:
There are two scenarios
1) you are going to put the complete processing of the CSV file in on Job, in which case, I think it will not work perfectly as the if the job fails it will again loop over all the rows. unless you mark the rows by adding another column in the CSV say read.
2) You add each row as a separate Job in Sidekiq, I can only think one downside with this is that you may end up creating too many jobs, in case you have large CSV files, but it will save a lot of processing on CSV side.