Rajasuba Subramanian Rajasuba Subramanian - 6 months ago 17
Java Question

What is the root cause of getting Out Of Memory Exception? How can we overcome this?

Here is my sample snippet for reading and writing by output stream and I am getting out of memory exception.

public static void readFileContent(InputStream in, OutputStream out) throws IOException {
byte[] buf = new byte[500000];
int nread;
int navailable;
int total = 0;
synchronized (in) {
try {
while((nread = in.read(buf, 0, buf.length)) >= 0) {
out.write(buf, 0, nread);
total += nread;
}
}
finally {
if (in != null) {
try {
in.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}

out.flush();
buf = null;
}




  1. What are the possible scenarios with the above snippet to get "out of memory exception" ?

  2. Is it necessary to close the output stream here? And does stream, flush is enough or do we need to close the stream always? If so why?

  3. How could I avoid Out of memory exception in general?




Please clarify me.

Answer

I think it's obvious that the problem is that you allocate 500000 bytes at once, and they may not be available in the heap at runtime.

Explanation: I would not suggest it, but you could increment the heap size of your program. The default heap size for a java program is determined at runtime, but it can also be parameterized.

Recommendation: As far as I can see by the provided snippet, it's not absolutely necessary to read 500000 bytes at once. So, you can initialize your byte array with a smaller number that would result in having more reading loops. But if it's not a problem for your program... I guess.

Conclusion: Try by setting the initial byte array size to 5000, or even 1000.

EDIT:

An extra point to take into consideration is that in the above code snippet you only flush once at the end. The bytes you are writting to the OutputStream are kept in memory, and their size may cause an OutOfMemoryException also.

In order to overcome this, you should flush more often. It will affect your performance if you flush too often, but you can always experiment with a condition in your loop e.g.

...
if (total % 5000 == 0) {
    out.flush();
}
...

EDIT 2:

As the InputStream and OutputStream objects are passed to the given method as parameters, so, in my opinion this method is not responsible for closing them. The method that initializes the Streams is also responsible for close them gracefully. Flush is enough for this method. But consider doing it in smaller chunks.

EDIT 3:

To summarize the suggested tweaks:

public static void readFileContent(InputStream in, OutputStream out) throws IOException {
    byte[] buf = new byte[1000];
    // wrap your OutputStream in a BufferedOutputStream
    BufferedOutputStream bos = new BufferedOutputStream(out, 5000);
    int nread;
    int navailable;
    int total = 0;
    synchronized (in) {
        try {
            while((nread = in.read(buf, 0, buf.length)) >= 0) {
                // use the BufferedOutputStream to write data
                // you don't need to flush regularly as it is handled automatically every time the buffer is full
                bos.write(buf, 0, nread);
                total += nread;
            }
        }
        finally {
            if (in != null) {
                try {
                    in.close();
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }
    }

    // flush the last contents of the BufferedOutputStream
    bos.flush();
    buf = null;
}

Please note also that BufferedOutputStream will automatically call flush() when you will close it gracefully.

EDIT 4:

Example calling the above method:

public static void main(String[] args) {
    String filename = "test.txt";
    String newFilename = "newtest.txt";

    File file = new File(filename);
    File newFile = new File(newFilename);

    try (InputStream fis = new FileInputStream(file);
            OutputStream fout = new FileOutputStream(newFile)) {
        readFileContent(fis, fout);
    }
    catch(IOException ioe) {
        System.out.println(ioe.getMessage());
    }
}
Comments