Roman Makhlin Roman Makhlin - 3 years ago 140
Java Question

tomcat: Waiting on condition thread

I'm investigating the strange issue with tomcat shutdown process: after runnig the java process still appears(I check it by using

ps -ef|grep tomcat
The situation is a bit complicated, because I have very limitted access to the server(no debug, for example)

I took thread dump(by using
kill -3 <PID>
) and heap dump by using remote
and Hotspot features.

After looking into thread dump I found this:

"pool-2-thread-1" #74 prio=5 os_prio=0 tid=0x00007f3354359800 nid=0x7b46 waiting on condition [0x00007f333e55d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c378d330> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(
at java.util.concurrent.ThreadPoolExecutor.getTask(
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$

So, my understanding of the problem is follows: There is a resource(DB connection or something else) which is used in CachedThreadpool, and this resource is now locked,
and prevent to thread
to stop. Assuming that this thread isn't deamon - JVM cannot gracefully stop.

Is there a way to find out which resource is locked, from where is it locked and how to avoid that? Another question is - how to prevent this situation?
Another this is: what the adress 0x00007f333e55d000 is for?


Answer Source

After a wail of struggling I found out, that the freezing thread always have the same name pool-2-thread-1. I grep all the sources of the project to find any places where any scheduled thread pool is started: Executors#newScheduledThreadPool. After a huge amount of time I logged all that places, with thread name in thread pool. One more restart server and gotcha!

I found out one thread pool, that is started with only one thread and used following code:

private void shutdownThreadpool(int tries) {
    try {
        boolean terminated;
        LOGGER.debug("Trying to shutdown thread pool in {} tries", tries);
        do {
            terminated = pool.awaitTermination(WAIT_ON_SHUTDOWN,TimeUnit.MILLISECONDS);
            if (--tries == 0
                    && !terminated) {
                LOGGER.debug("After 10 attempts couldn't shutdown thread pool, force shutdown");
                terminated = pool.awaitTermination(WAIT_ON_SHUTDOWN, TimeUnit.MILLISECONDS);
                if (!terminated) {
                    LOGGER.debug("Cannot stop thread pool even with force");
                    LOGGER.trace("Some of the workers doesn't react to Interruption event properly");
                    terminated = true;
            } else {
      "After {} attempts doesn't stop", tries);
        } while (!terminated);
        LOGGER.debug("Successfully stop thread pool");
    } catch (final InterruptedException ie) {
        LOGGER.warn("Thread pool shutdown interrupted");

After that the issue was solved.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download