mholberger mholberger - 3 months ago 40
C# Question

C# Async Multiple Web Requests to DataTable - Work appears to stop for unknown reasons

I am developing a single form app that, on button press, makes a simple inventory database API query, and checks each returned ItemID# for a corresponding image which may or may not exist at a URL formed from the ID#. I am currently doing this by sending HttpWebRequest.Method = "HEAD" requests for each URL, returning true unless the catch block is triggered.

The database query may return 50 - 150 part numbers, and sending HEAD requests to each one individually in this way takes about 5 minutes and this is not productive.

I am trying to multi-task this process using async and await. When I click the button, it works fine, asynchronously loading rows into my DataGridView one by one, at a rate of about 2/second (which isnt bad, but I would still like to speed this up if possible).

HOWEVER: After finding 2 successful URL responses, it stops loading rows and appears to just give up, for reasons unknown to me??? And the syncContext block which re-enables the UI is never executed, because the work is never completed. Can anyone see what might be causing this to happen?

I have been working loosely based off this doc:

"How to: Make Multiple Web Requests in Parallel by Using async and await (C#)"
https://msdn.microsoft.com/en-us/library/mt674880.aspx

namespace ImageTableTest
{
public partial class ImageTableTestForm : Form
{
//P21 Authentication Variables
private static Token P21token = null;
private static RestClientSecurity rcs;

//Create Tables and bindingSource
DataTable itemDataIMG = new DataTable();
DataTable itemDataNOIMG = new DataTable();
DataTable itemDataComplete = new DataTable();
BindingSource bindingSource = new BindingSource();

private readonly SynchronizationContext synchronizationContext;


public ImageTableTestForm()
{
InitializeComponent();

//Create syncContexct on UI thread for updating UI
synchronizationContext = SynchronizationContext.Current;

//authenticate database API function
authenticateP21();

//Designing DataTables
itemDataIMG.Columns.Add("MPN#", typeof(string));
itemDataIMG.Columns.Add("IMG", typeof(bool));
itemDataIMG.Columns[1].ReadOnly = true;

itemDataNOIMG.Columns.Add("MPN#", typeof(string));
itemDataNOIMG.Columns.Add("IMG", typeof(bool));
itemDataNOIMG.Columns[1].ReadOnly = true;

itemDataComplete.Columns.Add("MPN#", typeof(string));
itemDataComplete.Columns.Add("IMG", typeof(bool));
itemDataComplete.Columns[1].ReadOnly = true;

//bind to DataGridView itemView
bindingSource.DataSource = itemDataComplete;
itemView.DataSource = bindingSource;
itemView.AutoGenerateColumns = false;
}



private async void testBtn_Click(object sender, EventArgs e)
{
//When button is clicked, disable UI and
//start background work:
testBtn.Enabled = false;
loadSpinner.Visible = true;

await Task.Run(() =>
{
getItemView();
});
}


private async void getItemView()
{
try
{
//This executes the query and returns an array of Part objects:
PartResourceClient prc = new PartResourceClient(ConfigurationManager.AppSettings["P21.BaseURI"], rcs);
prc.QueryFilter("add_to_ebay eq 'Y'");
Part[] pResults = prc.Resource.GetParts();

int numParts = pResults.Length;
Task<bool>[] taskArray = new Task<bool>[numParts];
bool[] IMGboolArray = new bool[numParts];

//For each part, create CheckImageURL task and add to task Array
//Then Await execution
for (int i = 0; i < numParts; i++)
{
taskArray[i] = CheckImageURL(pResults[i].ItemId);
IMGboolArray[i] = await taskArray[i];
}
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
}

//When all Tasks finish, remove loadSpinner, re-enable UI
//(This never executes for unknown reasons.)
synchronizationContext.Post(new SendOrPostCallback(o =>
{
loadSpinner.Visible = false;
testBtn.Enabled = true;
}), null);

MessageBox.Show("<DONE>");
}


async Task<bool> CheckImageURL(string MPN)
{
//Here I am forming and executing the web HEAD request,
//If there is there is a 'NOT FOUND' response it goes to 'catch' block:
string URL = "https://s3-us-west-2.amazonaws.com/www.crosscreektractor.com/ebay-images/" + MPN + "_e.png";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL);
request.Method = "HEAD";
try
{
await request.GetResponseAsync();
synchronizationContext.Post(new SendOrPostCallback(o =>
{
addDataRows(MPN, true);
}), null);

return true;
}
catch
{
synchronizationContext.Post(new SendOrPostCallback(o =>
{
addDataRows(MPN, false);
}), null);

return false;
}
}

private void addDataRows(string MPN, bool IMG)
{
//Add data to respective table:
if (IMG)
{
itemDataIMG.Rows.Add(MPN, IMG);
}
else
{
itemDataNOIMG.Rows.Add(MPN, IMG);
}

//Here I am sorting the IMG and NOIMG tables,
//then merging them into the Complete table which
//The DataGridView is bound to, so that IMG entries are on top:
itemDataIMG.DefaultView.Sort = ("MPN# DESC");
itemDataNOIMG.DefaultView.Sort = ("MPN# DESC");

itemDataComplete.Clear();
itemDataComplete.Merge(itemDataIMG);
itemDataComplete.Merge(itemDataNOIMG);
itemView.Refresh();
}

Answer

Thanks for the advice on my TAP patterns, definitely learning a lot about TAP.

The solution to my problem was the HttpWebRequest connection limit. On the successful requests it does not then auto-close the connection, and you must do so by grabbing the WebResponse and closing it (only needed on the successful connections):

WebResponse response = await request.GetResponseAsync();
{do stuff}
response.Close();