valgrind stalls in multithreaded socket program

While other answers focus on insisting that you take the standard synchronization approach (something I fully agree with), I thought instead I should answer your question regarding Valgrind.

As far as I know there are no issues with Valgrind running in multi-threaded environment. I believe Valgrind forces the application to run on a single core, but other than that it should not affect your threads.

What Valgrind is probably doing to your application is altering the timings and interactions between your threads in ways that might be exposing bugs and race conditions in your code that you don't normally see while running stand-alone.

The same logic you applied to decide that the bug could not be in the open source threading framework you are using also applies to Valgrind in my opinion. I recommend that you consider these hangs as bugs in your code and debug them as such, because that is most likely what they are.

As a side note, using a mutex is probably overkill for the problem you described. You should investigate semaphores or condition variables instead.

Good luck.

Reading/writing a boolean is not an atomic operation on x86.

See my question here: Is volatile a proper way to make a single byte atomic in C/C++?

I just had a similar problem. Like the OP I had one thread doing a busy wait. In my case the problem was that the busy wait was taking almost all the CPU cycles and causing the other threads to run many thousands of times slower. At first I fixed this by putting a usleep(1) in the busy wait loop (only for Valgrind builds). Then I read the Valgrind manual and learned of the --fair-sched=yes option, which also fixed the problem and allowed me to remove the usleep(1).