pthread_exit vs. return

Not sure if you're still interested in this, but I am currently debugging a similar situation. Threads that use pthread_exit cause valgrind to report reachable blocks. The reason seems to be fairly well explained here:

https://bugzilla.redhat.com/show_bug.cgi?id=483821

Essentially it seems pthread_exit causes a dlopen which is never cleaned up explicitly when the process exits.

The following minimal test case exhibits the behaviour you describe:

#include <pthread.h>
#include <unistd.h>

void *app1(void *x)
{
    sleep(1);
    pthread_exit(0);
}

int main()
{
    pthread_t t1;

    pthread_create(&t1, NULL, app1, NULL);
    pthread_join(t1, NULL);

    return 0;
}

valgrind --leak-check=full --show-reachable=yes shows 5 blocks allocated from functions called by pthread_exit() that is unfreed but still reachable at process exit. If the pthread_exit(0); is replaced by return 0;, the 5 blocks are not allocated.

However, if you test creating and joining large numbers of threads, you will find that the amount of unfreed memory in use at exit does not increase. This, and the fact that it is still reachable, indicates that you're just seeing an oddity of the glibc implementation. Several glibc functions allocate memory with malloc() the first time they're called, which they keep allocated for the remainder of the process lifetime. glibc doesn't bother to free this memory at process exit, since it knows that the process is being torn down anyway - it'd just be a waste of CPU cycles.

Are you actually using C++, by any chance? To clarify - your source file ends with a .c extension, and you are compiling it with gcc, not g++?

It seems reasonably likely that your function is allocating resources that you expect to be cleaned up automatically when the function returns. Local C++ objects like std::vector or std::string do this, and their destructors probably won't be run if you call pthread_exit, but would be cleaned up if you just return.

My preference is to avoid low-level APIs such as pthread_exit, and always just return from the thread function, where possible. They're equivalent, except that pthread_exit is a de-facto flow-control construct that bypasses the language you're using, but return doesn't.