Java performance problem with LinkedBlockingQueue

I would generally recommend not using a LinkedBlockingQueue in a performance sensitive area of code, use an ArrayBlockingQueue. It will giving a much nicer garbage collection profile and is more cache friendly than the LinkedBlockingQueue.

Try the ArrayBlockingQueue and measure the performance.

The only advantage of the LinkedBlockingQueue is that it can be unbounded, however this is rarely what you actually want. If you have a case where a consumer fails and queues start backing up, having bounded queues allows the system to degrade gracefully rather risk OutOfMemoryErrors that may occur if queues are unbounded.

Your producer thread simply puts more elements than the consumer consumes, so the queue eventually hits its capacity limit, thus the producer waits.

Consolidating my original answer since now we have basically the full picture:

• You hit the inherent throughput limit of the LinkedBlockingQueue (every queue has one) by doing extremely fast put()s, where even continual take()s, with zero further processing, cannot keep up. (By the way this shows that in this structure, on your JVM and machine anyway, put()s are at least slightly more costly than the reads).
• Since there is a particular lock that consumers lock, putting more consumer threads could not possibly help (if your consumer was actually doing some processing and that was bounding the throughput, then adding more consumers would help. There are better queue implementations for a scenario with more than one consumers (or producers), you could try SynchronousQueue, ConcurrentLinkedQueue, and the upcoming TransferQueue of jsr166y).

Some suggestions:

• Try to make more coarse-grained objects, so that the overhead of queueing each is balanced with the actual work that is offloaded from the producing thread (in your case, it seems you create much communication overhead for objects that represent negligible amounts of work)
• You could also have the producer help the consumer by offloading some consuming work (not much point in waiting idly when there is work to be done).

/updated after John W. rightly pointed out my original answer was misleading

It's hard to say what happens without knowing something about the filling process.

If addMyMessage is called less frequently - perhaps because of a performance problem in a whole different part of your application - the take method has to wait.

That way it looks like take is the culprit, but actually it's the filling part of your application.

Here are a couple of things to try:

Replace the LinkedBlockingQueue with an ArrayBlockingQueue. It has no dangling references and so is better behaved when the queue fills up. Specifically, given the 1.6 implementation of LinkedBlockingQueue, full GC of the elements will not happen until the queue actually becomes empty.

If the producer side is consistently out performing the consumer side, consider using drain or drainTo to perform a "bulk" take operation.

Alternatively, have the queue take arrays or Lists of message objects. The the producer fills a List or array with message objects and each put or take moves multiple messages with the same locking overhead. Think of it as a secretary handing you a stack of "While you were out" messages vs. handing them to you one at a time.