Razort4x Razort4x - 15 days ago 3
C# Question

why do I get System.OutOfMemoryException even when there is about 700Mb of RAM free?

I was running some test, to see the how my logging would perform is instead of doing

File.AppendAllText
I would first write to a memory stream and then copy to file. So, just to see how fast memory operation is I did this..

private void button1_Click(object sender, EventArgs e)
{
using (var memFile = new System.IO.MemoryStream())
{
using (var bw = new System.IO.BinaryWriter(memFile))
{
for (int i = 0; i < Int32.MaxValue; i++)
{
bw.Write(i.ToString() + Environment.NewLine);
}
bw.Flush();
}
memFile.CopyTo(new System.IO.FileStream(System.IO.Path.Combine("C", "memWriteWithBinaryTest.log"), System.IO.FileMode.OpenOrCreate));
}
}


When
i
reached
25413324
I got a
Exception of type 'System.OutOfMemoryException' was thrown.
even though my Process Explorer says I have about 700Mb of free ram???

Here are the screen shots (just in case)

Process Explorer
enter image description here

Here's the winform

enter image description here

EDIT : For the sake of more objects being created on heap, I rewrote the
bw.write
to this

bw.Write(i);

Answer

First of all, you run out of memory because you accumulate data in the MemoryStream, instead of writing it directly to the FileStream. Use the FileStream directly and you won't need much RAM at all (but you will have to keep the file open).

The amount of physical memory unused is not directly relevant to this exception, as strange as that might sound.

What matters is:

  • that you have a contiguous chunk of memory available in the process' virtual address space
  • that the system commit does not exceed the total RAM size + page file size

When you ask the Windows memory manager to allocate you some RAM, it needs to check not how much is available, but how much it has promised to make available to every other process. Such promising is done through commits. To commit some memory means that the memory manager offered you a guarantee that it will be available when you finally make use of it.

So, it can be that the physical RAM is completely used up, but your allocation request still succeeds. Why? Because there is lots of space available in the page file. When you actually start using the RAM you got through such an allocation, the memory manager will just simply page out something else. So 0 physical RAM != allocations will fail.

The opposite can happen too; an allocation can fail despite having some unused physical RAM. Your process sees memory through the so-called virtual address space. When your process reads memory at address 0x12340000, that's a virtual address. It might map to RAM at 0x78650000, or at 0x000000AB12340000 (running a 32-bit process on a 64-bit OS), it might point to something that only exists in the page file, or it might not even point at anything at all.

When you want to allocate a block of memory with contiguous addresses, it's in this virtual address space that the RAM needs to be contiguous. For a 32-bit process, you only get 2GB or 3GB of usable address space, so it's not too hard to use it up in such a way that no contiguous chunk of a sufficient size exists, despite there being both free physical RAM and enough total unused virtual address space.