Autumn Autumn - 1 month ago 10
HTTP Question

My loop of GET requests with HttpClient falls into a mysterious flurry of AggregateExceptions

I'm new to this language, new to HttpClient, and bad with multithreading, so try to bear with me. I have a Windows Forms application that allows a user to input an ID. When they press Start, it will begin repeatedly making requests to a certain website's web API (until they press Stop), incrementing the ID by 1 each time. The ID is included with the requests; it makes the site return a JSON table of a virtual object associated with that ID. Thousands of these objects are uploaded to the site per minute.

Here is the method I am using to make the requests:

private async Task<string> GetResponseText(HttpClient client, string address)
{
return await client.GetStringAsync(address);
}


So in the part of my code where I handle what happens when the program is going to start scanning, I first create a new HttpClient, like this:

HttpClient client = new HttpClient();


Then I create a brand new thread and start looping:

System.Threading.Thread scanThread = new System.Threading.Thread(() =>
{
while (scanning)
{
string assetUrl = "https://api.example.com/blah?id=" + assetId.ToString();
string response = GetResponseText(client, assetUrl).Result;
dynamic assetJson = JsonConvert.DeserializeObject(response);
if (assetJson != null)
{
this.Invoke((MethodInvoker)delegate
{
try
{
// omitted lots of irrelevant stuff
assetId += 1;
}
catch (RuntimeBinderException)
{
// this probably means the ID doesn't exist yet so I do nothing
}
});
}
}
System.Threading.Thread.CurrentThread.Abort();
});
scanThread.IsBackground = true;
scanThread.Start();


The first time I ran that, it worked fine. It succeeded with every single ID, and it went at least 5 or 10 requests per second. There were no weird exceptions or issues behind the scenes or anything like that. But then, out of the blue, after maybe a hundred or two requests, I got my first problem:

A first chance exception of type 'System.Net.Http.HttpRequestException' occurred in mscorlib.dll
A first chance exception of type 'System.AggregateException' occurred in mscorlib.dll
An unhandled exception of type 'System.AggregateException' occurred in mscorlib.dll
Additional information: One or more errors occurred.


The stack trace narrowed down the problem to this line:

string response = GetResponseText(client, assetUrl).Result;


"That's okay," I thought to myself. I didn't know what was wrong, so I set up a try/catch block for that line. And this is the part that really makes me scratch my head.

With a try/catch in place, as soon as this error hits, it simply keeps happening over and over for the entire loop. It never recovers or continues normally. And there are two ways to fix it, the second of which is the weirdest:

1) Simply close the program and open it again

2) Stop scanning, wait patiently for about 6 lines to appear in Visual Studio's debug output about threads exiting, then start scanning again

That second scenario I feel like is a clue about why this is happening, but I'm not good enough to figure it out. What's happening? And why does waiting for some threads to close (whatever they are to begin with) fix the problem? Can I just get rid of those manually every time the loop runs?

Answer

The HTTP-Statuscode 400 means that your Request is invalid. So you have to check your HTTP-Request in detail. Maybe the API allows just a limited number of access or sth.

Further i will give you some code example which is more modern:

   var ts = new CancellationTokenSource();
   CancellationToken ct = ts.Token;

    private void Start()
    {
       var task = Task.Factory.StartNew(() => {

         var client = new HttpClient();
         while(true)
         {
            if (ct.IsCancellationRequested)
            {
               break;
            }

            string assetUrl = "https://api.example.com/blah?id=" + assetId;
            string response = await GetResponseText(client, assetUrl);
            dynamic assetJson = JsonConvert.DeserializeObject(response);

            if (assetJson != null)
            {
               assetId++;
            }
         }
       }, ct).ContinueWith(ex => Console.WriteLine(ex.Message), TaskContinuationOptions.OnlyOnFaulted);
     }


     private void Stop()
     {
        ct.Cancel();
     }
Comments