Pretty much anyone who has written a Java web application has grappled with the dreaded “java.lang.OutOfMemoryError: PermGen space” errors when reloading web applications multiple times. It’s a problem so long running, low-level and common that you would think it would have been solved for good by now… but alas Java developers still come up against it daily.
Frank Kieviet of Sun has written some good blogs describing the problem and how to go about finding and fixing them. The latter article describes a method of finding PermGen leaks using standard JDK jmap (Memory Map) and jhat (Heap Analysis Tool) tools, stating that other existing profilers were not able to find these leaks.
Well, the article is a bit old now, and I’ve discovered that version 7 of the excellent (but boringly named) YourKit commercial profiler is now able to hunt down these leaks very quickly indeed. There’s not too much specific info on how to do this though, so here’s how…
1. Enable profiling in your container
This is pretty straight-forward, but how you do it depends on which JVM, OS and container you’re using, and how you’re running it. It’s explained in detail on the YourKit site, but generally speaking you just need to:
- Add the YourKit binaries directory to your system path
- Add “
-agentlib:yjpagent” to your JVM startup parameters. I’m using Tomcat on Windows (standalone, not through my IDE), so this simply meant adding this to my JAVA_OPTS environment variable.
You can verify that all this is working by starting your container and attempting to connect to it with YourKit (Select “Connect to locally running profiled application…” on the “Welcome” screen).
2. Start, use and undeploy your web application
Simply load up and use your web application as normal. Once you’ve exercised it a little, undeploy it from your container in the usual way. I prefer to undeploy it completely rather than simply stopping it to make sure that the container properly disposes of all associated resources.
3. Capture a memory snapshot
Connect YourKit to your application if you haven’t already, then simply click the “Capture Memory Snapshot” button in the YourKit toolbar:
When prompted, open the resulting memory dump in YourKit.
4. Find “Classes Without Instances”
On the “Inspections” tab of the memory dump, choose “Classes without Instances” and then “Run This Inspection Only”:
5. Find an application class that should have been unloaded
The results for the inspection will show a list of Java ClassLoaders that have loaded classes that now have no instances. The trick now is to find the ClassLoader that loaded your application (if you’re running Tomcat this will be an instance of org.apache.catalina.loader.WebAppClassLoader). You can usually tell by expanding a few of the ClassLoader entries and looking for classes that specifically belong to your application.
Once you’ve found one of your classes, right click on it and choose “Paths from GC Roots”. In the example below I’ve found a MySQL JDBC connection that was used by my application that should have been unloaded.
6. Analyse the GC paths
This is where things get interesting. The “GC Roots” screen shows you all the paths from the various Garbage Collector roots to the class in question. Consider the example below:
This screen shows the shortest GC path to the JDBC connection class we were looking at in the previous step. We can see that the GC path starts from the threadLocals property of one of the container threads (unremoved ThreadLocals are notorious for causing PermGen leaks). Moving up the chain we can see that the dom4j library is involved, appearing just before the ClassLoader itself. The ClassLoader in turn references the JDBC connection class.
From this I would guess that the dom4j library is inserting some data into a ThreadLocal at some point, but never cleaning it up. This results in the container thread (which never dies) holding on to a dom4j class definition that was loaded by the web application’s ClassLoader. This in turn prevents the ClassLoader from being garbage collected, and hence none of the classes loaded by the ClassLoader are collected either!
A bit of investigation revealed this to be the case, and the solution was simply to upgrade dom4j to version 1.6.1.
Unfortunately it’s not always obvious from these GC roots exactly what the problem is. A bit of investigation and sometimes guesswork is required, but with a little patience and experience you should get the hang of it. One pattern that I have noticed is that frequently the class/library just before the ClassLoader in the reference chain is the cause of the problem (this is not always the case).
Frustratingly, finding the solution can be frequently just as challenging as finding the cause. Solutions range from upgrading libraries, adding explicit shutdown code to clean things up or getting in touch with library developers for bug fixes. I’ll cover some specific solutions that I have come across in a future entry.
Good luck!