Ams1 Ams1 - 21 days ago 8
C# Question

Writing many rows into a file efficently

I have a process that gets many rows of a table. This rows need to be reported in a report sheet. What I'm doing for the moment is too slow, because there are so many rows (600.000 <= x)-Rows.
I'll show what I'm doing now to give you an idea of what I'm doing:

private void ShowNotFoundRows() {

report += "NOT FOUND - Rows";
report += "\r\n";
report += "In Current:";
report += "\r\n";
string Columns = "|";
foreach (string header in currentModel.Columns) {
Columns += header + "|";
}
report += Columns;
report += "\r\n";
foreach (string row in newModel.Keys) {
report += row + "\r\n";
}
report += "\r\n";
report += "In New:";
report += "\r\n";
report += Columns;
report += "\r\n";
foreach (string row in currentModel.Keys) {
report += row + "\r\n";
}
}


and after:

private void CreateReportFile() {
MyLog.WriteToLog("Creating ReportFile "+newModel.TableName, MyLog.Messages.Info);
string dir = Settings.Default.ReportFolder + "\\" + directoryName + " " + DateTime.Now.ToString("dd-MM-yyyy");
Directory.CreateDirectory(dir);

try {
File.WriteAllText(dir + "\\Report " + newModel.TableName, report);
} catch (DirectoryNotFoundException e) {

Console.WriteLine(e.Message + "\n" + e.StackTrace);
}
}

Answer

Start by using StringBuilder and see where that gets you. You're copying in a lot of memory with stringx = stringx+ stringy because it creates a new object and copies all of it into the new object. I have a feeling that'll get you where you need to be. Stringbuilder by contrast preallocates memory.

Beyond that, you could do things like divide the data into partitions, and multithread that, and then combine the results at the end.