Th3Wolf Th3Wolf - 11 days ago 4
C# Question

c# Multi Threading Processing large amount of data fast

Hey there i have like +15000000 record on a text file
and i need to process each line of them
is using multi threading will be useful
any suggestions ?
like for example creating 10000 thread and dividing 15000000 / 10000 to give each thread small ammount of data ?

Answer

You can play with Parallel Linq (PLinq):

var result = File
  .ReadLines(@"C:\MyFile.txt")
  .AsParallel()
  //.AsOrdered() // if you want to preserve the order of file lines
  .WithDegreeOfParallelism(10) // let's try 10 threads
  .Select(line => ProcessLine(line)) // whatever
  ...   

You can easy compare parallel version with sequential one: either comment out .AsParallel() and .WithDegreeOfParallelism(10) or add .AsSequential()

Comments