Brad Boyce Brad Boyce - 1 month ago 11
C# Question

System.OutOfMemory exception when calling MemoryStream.ToArray() after serializing object with DataContractSerializer

I am getting an intermittent "out of memory" exception at this statement:

return ms.ToArray();


In this method:

public static byte[] Serialize(Object inst)
{
Type t = inst.GetType();
DataContractSerializer dcs = new DataContractSerializer(t);
MemoryStream ms = new MemoryStream();
dcs.WriteObject(ms, inst);
return ms.ToArray();
}


How can I prevent it? Is there a better way to do this?

The length of ms is 182,870,206 bytes (174.4 MB)

I am putting this into a byte array so that I can then run it through compression and store it to disk. The data is (obviously) a large list of a custom class that I am downloading from a WCF server when my silverlight application starts. I am serializing it and compressing it so it uses only about 6MB in isolated storage. The next time the user visits and runs the silverlight application from the web, I check the timestamp, and if good I just open the file from isolated, decompress it, deserialize it, and load my structure. I am keeping the entire structure in memory because the application is mostly geared around manipulating the contents of this structure.

@configurator is correct. The size of the array was too big. I rolled by own serailizer, by declaring a byte array of [list record count * byte count per record], then stuffed it directly myself using statements like this to stuff it:

Buffer.BlockCopy(
BitConverter.GetBytes(node.myInt),0,destinationArray,offset,sizeof(int));
offset += sizeof(int);


and this to get it back:

newNode.myInt= BitConverter.ToInt32(sourceByteArray,offset);
offset += sizeof(int);


Then I compressed it and stored it to isolated storage.

My size went from 174MB with the DataContractSerializer to 14MB with mine.
After compression it went from a 6MB to a 1MB file in isolated storage.

Thanks to Configurator and Filip for their help.

Answer

The problem seems to be that you're expecting to return a 180MB byte array. That means the framework would need to find and allocate a consecutive 180MB of free memory to copy the stream data into, which is usually quite hard - hence the OutOfMemoryException. If you need to continue handling this amount of memory, use the memory stream itself (reading and writing to it as you need) to hold the buffer; otherwise, save it to a file (or to whatever other place you need it, e.g. serving it over a network) directly instead of using the memory stream.

I should mention that the memory stream has a 180MB array of its own in there as well, so is also in a bit of trouble and could cause OutOfMemory during serialization - it would likely be better (as in, more robust) if you could serialize it to a temporary file. You might also want to consider a more compact - but possibly less readable - serialization format, like json, binary serialization, or protocol buffers.


In response to the comment: to serialize directly to disk, use a FileStream instead of a MemoryStream:

public static void Serialize(Object inst, string filename)
{
    Type t = inst.GetType();
    DataContractSerializer dcs = new DataContractSerializer(t);
    using (FileStream stream = File.OpenWrite(filename)) {
        dcs.WriteObject(ms, inst);
    }
}