Hibernate "APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!"
I stumbled into the same exception and the reason was a wrong password of the db user...
c3p0 you're a funny guy
So, directly the issue is that the Connection pool was trying to acquire new Connections, but the tasks attempting to acquire those tasks froze for a long period of time, so long that c3p0 decided the tasks must be deadlocked and then discarded and replaced the Thread pool. Later, the Thread pool was hung on idle connection test tasks.
Normally, "hung" tasks tend to look like the second Thread underneath the first "Pool thread stack traces:" label above: performing network IO that fails to complete. Your circumstances are odd in that two of the three threads are not stuck in IO. They have barely begun to do anything, yet they aren't live. Then you experience an OutOfMemoryError, and you get another APPARENT DEADLOCK on idle connection test tasks that also look like they mostly should be live.
Maybe your application is very close to some kind of resource limits that are causing things to run very sluggishly? Straightforwardly, you might increase the amount of memory available to this application (or modify it to have a lower memory footprint). You experience an OOME the second time the Thread pool tries to flush and recreate hung Threads, not directly provoked by that, but quite possibly caused by the growing Thread footprint. (In your logs, are there lots of these APPARENT DEADLOCKs previously? If you force a JVM Thread dump, do you see lots of com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
instances around, still hung on old, not completing tasks?
Some general comments: If your load is no more than 5-ish simultaneous queries, why such a heavy pool? why not min_size
5-ish, max 10-ish? Your max_statements setting is way too small for the number of Connections you permit in the pool. I'd omit this entirely until you get things working smoothly. Then, to get a bit better performance, you might set the more-straightforward-to-reason-about maxStatementsPerConnection
parameter if you want.
Mostly you need to keep your application's footprint (memory? Threads?) well below the resources alotted to it, either by increasing resources, reducing its footprint, or fixing any issue that might exist that cause its resource footprint to increase into limits. I'd start by making the pool smaller, the available memory larger, and configuring the pool to be much smaller.
Authentication issue for me too. I just added the domain part to the database server name and it all worked. Very misleading error.