Judah Flynn Judah Flynn - 1 year ago 175
Java Question

Reduce memory usage in java

I have a web service Java program which reads 13,000,000 dates like '08-23-2016 12:54:44' as strings from database. My developing environment is Java 8, MySQL 5.7 and tomcat 8. I have declare a string array String[] data to store it. I use Guice to inject the initial values of data array to empty. However, the memory usage is still huge. This is my code:

String[] data;//size is 1,000,000
void generateDataWrapper(String params) {
//read over 13000000 dates string
ResultSet rs = mySQLCon.readData(params);
clearData(data);//set to empty string
int index = 0;
while(rs.next()) {
data[index++] = rs.getString("date");
if (index == (size - 1)) {//calculate every 1,000,000 total 13 times
//calculate statistics
//reset all to empty string
index = 0;
//mySQLCon. readData function
ResultSet readData(String params) {
try {
String query = generateQuery(params);
Statement postStmt = connection.createStatement();
ResultSet rs = postStmt.executeQuery(query);
return rs;
} catch (Exception e) {
return null;

If I call this function once, the memory is reached 12G, If I call it again, the memory goes to 20G, on the third time the memory will goes to 25G and throw a 'java.lang.OutOfMemoryError: GC overhead limit exceeded' error in com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2174)

This is part of the error message:

java.lang.OutOfMemoryError: GC overhead limit exceeded
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)

I have changed the garbage collection algorithms to:
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
But it's not helping.
I have tried change the data to static variables, still will have this problem.
Currently the JVM heap is 8g, the tomcat memory is 24g, however, I don't think increase the memory will solve the problem.

I don't understand why my memory is still increasing every time I call this function, Could someone give me some suggestion?

Answer Source
  1. Used resources like a ResultSet have to be closed to release the underlying system-resources. This can be done automatically declaring the resources in a try-block like try (ResultSet resultSet =...).

  2. You can try to fetch only a limited number of rows from database when they are requested from ResultSet and not all of them immediately.

  3. Objects get eligible for garbage collection when they are not referenced any more. So, your array-object keeps in memory with it's whole size as long as it is referenced. If it's not referenced any more and the VM is running out of memory it will be able to dispose the array-object possibly avoiding an OutOfMemoryError.

  4. Unexpectedly high memory usage can be analyzed by creating a heap dump and exploring it in the tool jvisualvm of the JDK.