Shiny Shiny - 3 months ago 22
Vb.net Question

ThreadPool Execution Delay

It seems the more you are queuing periodically basically the slower ThreadPool actually executes threads.

For my application (Proxy Checker) this is a huge issue that makes the application extremely slow.

In other questions the answers are basically "Ditch ThreadPool, use tasks" but when I looked into Tasks it seems to only be able to execute 1 task and not a Queueing system that's loopable, or am I wrong?




If I add 14,000 items to the "Proxies" variable and Queue each of them using QueueUserWorkItem with 500 MinThreads it will get to 13,000 fast then after it gets to 13,000 (Ish) it starts to get EXTREMELY slow and pretty much fails to continue.

Tested with 1000 threads and 14,000 items and it resulted in 9,000 ish checked before it froze. So im assuming its to do with the ThreadPool exhaustion as usual.

When I tested on 14,000 items with 400 MinThreads and it got to 13,000 ish threads so it didnt seem to differ much to the 500 threads apart from it taking longer to get stuck at 13,000.

Update: It seems to get stuck between 90-92% completion not X thousand finished threads. I didn't include any Progress tracking code but you can make out how to calculate it Online.




Here's a MCV Example.
Project File or Form1.vb
It gets to 13,000 ish and freezes (The number differs depending on how many threads you set, And yes it does still happen when you dont mess with SetMin/MaxThreads.)




Here's the code for ease of access:

Imports System.IO, System.Net, System.Threading, System.Text.RegularExpressions

Public Class Form1

Dim totalproxies As Integer
Dim checkedproxies As Integer
Dim ActionList As New List(Of String)

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

'Create a OpenFileDialog;
Dim ofd = New OpenFileDialog()

'If the user chose a file to Import;
If (ofd.ShowDialog() = DialogResult.OK And ofd.CheckFileExists = True) Then

'Set the Thread Count;
ThreadPool.SetMinThreads(500, 500)
ServicePointManager.MaxServicePointIdleTime = 8000
ServicePointManager.DefaultConnectionLimit = 500 / 3

'Read the File;
Using sr As New StreamReader(ofd.FileName)

'Get all the Proxies found from the Regex Condition;
Dim MatchedRegex As MatchCollection = New Regex("[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}:[0-9]{1,4}").Matches(sr.ReadToEnd)

'Add all of the Matched Proxies to the List;
For Each proxy As Match In MatchedRegex

'Increment the Total Proxies count;
totalproxies += 1

'Check the Proxy;
ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf CheckProxy), proxy.ToString)

Next

End Using

End If

End Sub

Private Function CheckProxy(ByVal Proxy As String) As String

Try

'Set the WebRequest URL to a Proxy Judge and set the Proxy with Timeout settings;
Dim init As HttpWebRequest = WebRequest.Create("http://azenv.net")
init.Timeout = 4500
init.ReadWriteTimeout = 2500
init.KeepAlive = False
init.Proxy = New WebProxy(Proxy)

'Initiate the WebRequest and see if the Proxy can connect fine;
Dim response As HttpWebResponse = init.GetResponse()

'Since no Exception has Occured as the Proxy as a Working Proxy;
ActionList.Add(Proxy)

Return True

Catch ex As Exception

Return False

Finally

checkedproxies += 1

'Increment the CheckedProxies count;
Invoke(New MethodInvoker(Sub() Label1.Text = "Checked: " + checkedproxies.ToString + "/" + totalproxies.ToString))
Application.DoEvents()

End Try

End Function

End Class

Answer

Try adding "System.Reactive.Windows.Forms" using NuGet and then do this code:

Option Strict On

Imports System.IO
Imports System.Net
Imports System.Threading
Imports System.Text.RegularExpressions
Imports System.Reactive.Linq

Public Class Form1

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

        'Create a OpenFileDialog;
        Dim ofd = New OpenFileDialog()

        'If the user chose a file to Import;
        If (ofd.ShowDialog() = DialogResult.OK And ofd.CheckFileExists = True) Then

            ThreadPool.SetMinThreads(500, 500)
            ServicePointManager.MaxServicePointIdleTime = 8000
            ServicePointManager.DefaultConnectionLimit = 500 \ 3

            Dim text = File.ReadAllText(ofd.FileName)
            Dim regex = New Regex("[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}:[0-9]{1,4}")

            Dim proxies =
                regex _
                    .Matches(text) _
                    .OfType(Of Match)() _
                    .Select(Function(x) x.ToString()) _
                    .ToArray()

            Dim count = 0
            Dim query =
                proxies _
                    .ToObservable() _
                    .SelectMany(Function(p) Observable.Start(Function() New With {Key .proxy = p, Key .check = CheckProxy(p)})) _
                    .Do(Sub(x) count += 1) _
                    .Select(Function(x) New With {x.proxy, x.check, count}) _
                    .ObserveOn(Me) _
                    .Do(Sub(x) Label1.Text = "Checked: " + x.count.ToString() + "/" + proxies.Length.ToString()) _
                    .Where(Function(x) x.check) _
                    .Select(Function(x) x.proxy) _
                    .ToArray()

            query _
                .ObserveOn(Me) _
                .Subscribe(
                    Sub(ActionList)
                        ' Do something with `ActionList` array of successful proxies
                    End Sub)
        End If

    End Sub

    Private Function CheckProxy(proxy As String) As Boolean
        Dim init As HttpWebRequest = DirectCast(WebRequest.Create("http://azenv.net"), HttpWebRequest)
        init.Timeout = 4500
        init.ReadWriteTimeout = 2500
        init.KeepAlive = False
        init.Proxy = New WebProxy(proxy)
        Try
            Using response As HttpWebResponse = DirectCast(init.GetResponse(), HttpWebResponse)
                Return True
            End Using
        Catch ex As WebException
            Return False
        End Try
    End Function

End Class

It's worked like a treat in all of my tests.

And it's also possible that your code hasn't frozen - it just gets a lot of exceptions near the end and just doesn't display the progress counter for the last few percent. That would explain why it stops so close to 100%.

Comments