M Crozier M Crozier - 21 days ago 11
C# Question

Read double data from a text file - differences in read time

Using C# I'm reading data from text files into a 2D list for further processing. Each file is 256 doubles, space delimited in 256 lines, each line is read into a list of doubles and each list is added to a list of lines. All files have 256x256 = 65,536 data points.

I've got code that reads the files and works well for some files but for others it takes a really long time. Since all files are formatted in the same way and contain the same number of data points I don't understand the difference in read time, any one got any ideas?

How can I speed up the read time of file 2?

Here is the code I'm using:

private Data ReadData (string name, string file)
{
List<List<Double>> data_points = new List<List<Double>>();

String input = File.ReadAllText( file );

foreach (string row in input.Split('\n'))
{
List<Double> line_list = new List<double>();
foreach (string col in row.Trim().Split(' '))
{
if(row != "")
{
line_list.Add(double.Parse(col.Trim()));
}

}
if(line_list.Count > 1)
{
data_points.Add(line_list);
}
}

Data temp_data = new Data(name, data_points);
return temp_data;
}


Example text files are here:

https://www.dropbox.com/s/diindi2qjlgoxep/FOV2_t1.txt?dl=0 => reads fast

https://www.dropbox.com/s/4xrgdz0nq24ypz8/FOV2_t2.txt?dl=0 => reads slow

In answer to some of the comments:
@AntDC - What constitutes a valid double? I tried replacing Parse.Double with Convert.ToDouble with no improvement.

@Henk Holterman - the difference in read time is very noticeable <1s for the first file and approx. 50s for the second file. It appears to be repeatable.

@Slai - I moved both files to other locations and it had no impact on read time. Both files were exported from the same program within seconds of one another.

Answer

Performance wise you can optimize your code, instead reading the whole file and after that splitting it. Just read line by line.

List<Double> line_list = new List<double>();
foreach (string line in File.ReadLines("c:\\file.txt"))
{
    string[] rows = line.Trim().Split(' ');

    foreach(string el in rows)
    {
        line_list.Add(double.Parse(el.Trim()));
    }
}
Comments