froi froi - 2 months ago 13
Groovy Question

Gpars.withPool threads eventually stops executing after iterating numerous times

Was trying to execute concurrent operations with Gpars.

Gpars.withPool(6) {
someList.eachParallel {
println "${Thread.currentThread}"
}
}


Initially it seems to work

Thread[ForkJoinPool-1-worker-1,5,main]
Thread[ForkJoinPool-1-worker-6,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-5,5,main]
Thread[ForkJoinPool-1-worker-3,5,main]
Thread[ForkJoinPool-1-worker-4,5,main]
Thread[ForkJoinPool-1-worker-1,5,main]
Thread[ForkJoinPool-1-worker-6,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]


But after iterating for sometime, some of the threads stops executing.

Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-3,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-3,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-3,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]


Until eventually we are left with only one.

Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]
Thread[ForkJoinPool-1-worker-2,5,main]


Any idea why this happens? Any solution to keep all threads active?

In my case, it might be worth mentioning that we are iterating at around 50k-200k.

Answer

Just in case other people have the same issue. Their documentation says:

Because GParsPool uses a Fork/Join pool (with work stealing), threads may not be applied to a waiting processing task even though they may appear idle. With a work-stealing algorithm, worker threads that run out of things to do can steal tasks from other threads that are still busy.

if you use GParsExecutorsPool , which doesn't use Fork/Join, you get the thread allocation behavior that you would naively expect.

Base on that, I opt to use GParsExecutorsPool instead. With this, all threads executed consistently until the end of the whole eachParallel process.

GParsExecutorsPool.withPool(6) {
  someList.eachParallel {
    println "${Thread.currentThread}"
  }
}