Th3Wolf Th3Wolf - 5 months ago 79
C# Question

c# Multi Threading Processing large amount of data fast

Hey there i have like +15000000 record on a text file
and i need to process each line of them
is using multi threading will be useful
any suggestions ?
like for example creating 10000 thread and dividing 15000000 / 10000 to give each thread small ammount of data ?


You can play with Parallel Linq (PLinq):

var result = File
  //.AsOrdered() // if you want to preserve the order of file lines
  .WithDegreeOfParallelism(10) // let's try 10 threads
  .Select(line => ProcessLine(line)) // whatever

You can easy compare parallel version with sequential one: either comment out .AsParallel() and .WithDegreeOfParallelism(10) or add .AsSequential()