Roland Illig Roland Illig - 1 month ago 16
Java Question

Tool for analyzing large Java heap dumps

I have a HotSpot JVM heap dump that I would like to analyze. The VM ran with

-Xmx31g
, and the heap dump file is 48 GB large.


  • I won't even try
    jhat
    , as it requires about five times the heap memory (that would be 240 GB in my case) and is awfully slow.

  • Eclipse MAT crashes with an
    ArrayIndexOutOfBoundsException
    after analyzing the heap dump for several hours.



What other tools are available for that task? A suite of command line tools would be best, consisting of one program that transforms the heap dump into efficient data structures for analysis, combined with several other tools that work on the pre-structured data.

Answer

Normally, what I use is ParseHeapDump.sh included in Eclipse Memory Analyzer, and I do that on one our more beefed up servers. The shell script needs less resources than parsing the heap from the GUI, plus you can run it on your beefy server with more resources (you can allocate more resources by adding something like -vmargs -Xmx40g -XX:-UseGCOverheadLimit to the end of the last line of the script. For instance, it might look like this after modification

./MemoryAnalyzer -consolelog -application org.eclipse.mat.api.parse "$@" -vmargs -Xmx40g -XX:-UseGCOverheadLimit

When it succeeds, it creates a number of "index" files.

After getting the indices, I try to generate reports from that as well and scp those to my local machines and try to see if I can find the culprit just by that (not just the reports, not the indices). Here's a tutorial on creating the reports.

If those reports are not enough and if I need some more digging (i.e. let's say via oql), I scp the indices as well as hprof file to my local machine, and then open the heap dump (with the indices in the same directory as the heap dump) with my Eclipse MAT. From there, it does not need too much memory to run.

EDIT: I just liked to add two notes :

  • As far as I know, only the generation of the indices is the memory intensive part of Eclipse MAT. After you have the indices, most of your processing from Eclipse MAT would not need that much memory.
  • Doing this on a shell script means I can do it on a headless server (and I normally do it on a headless server as well, because they're normally the most powerful ones). And if you have a server that can generate a heap dump of that size, chances are, you have another server out there that can process that much of a heap dump as well.

Cheers,

Franz See

Comments