Tips for debugging hard-to-reproduce concurrency bugs?

This might not help you but will probably help someone seeing this question in the future.

If you're using a .Net language you can use the CHESS project from Microsoft research. It runs unit tests with every kind of thread interleaving and shows you which ones cause the bug to happen.

There may be a similar tool for the language you're using.

Here is my technique : I generally use a lot of assert() to check the data consistency/validity as often as possible. When one assert fails, the program crashes generating a core file. Then I use a debugger with the core file to understand what thread configuration led to data corruption.