Djerro Neth Djerro Neth - 2 months ago 6
JSON Question

Is there any value in using async await functionality when pulling JSON and store it in database?

I recently have started programming in C# (after having some experience with PHP and JavaScript), and i built a simple console program that downloads a JSON string and stores certain values in a database. The data in question is approx. 70.000 sets (converted into rows into my database). Due to a limitation on the server where I download this JSON from (Quandl), it was recommended to download it with 100 datasets per request, so I have 700 requests to make.

With every request, I download the JSON string, deserialize it and loop through it a 100 times to store the respective values in the database. I am using

to make the request and I utilize for the deserialization.

Currently, with the setup I have, it takes approx. 7 seconds for every request and including inserting the data into the database, it takes about one and half hour to finish.

The question then becomes; is there anyway to speed this up with the async/await method? Everything I read is more on the UI side of things (i.e. the UI is not frozen while a request is processed), but I was wondering if it were possible to start the requests maybe simultaneously (or, per 10 at the time or something). For completion, I have added a sanitized version of my code (made it a bit shorter but no logic has been removed).


async/await are for asynchronous operations. Asynchronous execution does not equal parallel execution. Asynchronous execution does not block the caller, and parallel execution allows for concurrent execution. You need parallel execution. To do this, you can use the Task Parallel Library. There is also a patterns and practices book that is a great read. Here is a simplified implementation:

var httpClient = new HttpClient();
httpClient.BaseAddress = new Uri("/path/to/data");

var tasks = new Task<Task<HttpResponseMessage>>[5];

for (var i = 0; i < tasks.Length; i++)
    tasks[i] = Task<Task<HttpResponseMessage>>.Factory.StartNew(async () => await httpClient.GetAsync("?updatedFilterParams"));

Task.WhenAll(tasks); // wait for them to complete

foreach (var task in tasks)
    var data = task.Result.Result.Content.ReadAsStringAsync();
        // do something

Some things to note: WebClient is not capable of concurrent requests so you'll either have to new up another one for every request or use HttpClient as I have. Also, there are multiple things in between your code and the data that can and often do impose limits on concurrent requests for the same origin, so you'll want to throttle how many requests you fire off at a time.