Check to see if a pthread mutex is locked or unlocked (After a thread has locked itself)

You can use pthread_mutex_trylock. If that succeeds, the mutex was unclaimed and you now own it (so you should release it and return "unheld", in your case). Otherwise, someone is holding it.

I have to stress though that "check to see if a mutex is unclaimed" is a very bad idea. There are inherent race conditions in this kind of thinking. If such a function tells you at time t that the lock is unheld, that says absolutely nothing about whether or not some other thread acquired the lock at t+1.

In case this is better illustrated with a code example, consider:

bool held = is_lock_held();

if (!held)
{
  // What exactly can you conclude here?  Pretty much nothing.
  // It was unheld at some point in the past but it might be held
  // by the time you got to this point, or by the time you do your
  // next instruction...
}

Mutexes are not the right primitive for the scheme that you want to implement. You should be using condition variables:

int run_thread_a = 0;
pthread_mutex_t run_lock_a = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t run_cond_a = PTHREAD_COND_INITIALIZER;

int run_thread_b = 0;
pthread_mutex_t run_lock_b = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t run_cond_b = PTHREAD_COND_INITIALIZER;

/* thread A */
while (1) {
    /* Wait for Thread A to be runnable */
    pthread_mutex_lock(&run_lock_a);
    while (!run_thread_a)
        pthread_cond_wait(&run_cond_a, &run_lock_a);
    run_thread_a = 0;
    pthread_mutex_unlock(&run_lock_a);

    /* Do some work */

    /* Now wake thread B */
    pthread_mutex_lock(&run_lock_b);
    run_thread_b = 1;
    pthread_cond_signal(&run_cond_b);
    pthread_mutex_unlock(&run_lock_b);
}

/* thread B */
while (1) {
    /* Wait for Thread B to be runnable */
    pthread_mutex_lock(&run_lock_b);
    while (!run_thread_b)
        pthread_cond_wait(&run_cond_b, &run_lock_b);
    run_thread_b = 0;
    pthread_mutex_unlock(&run_lock_b);

    /* Do some work */

    /* Now wake thread A */
    pthread_mutex_lock(&run_lock_a);
    run_thread_a = 1;
    pthread_cond_signal(&run_cond_a);
    pthread_mutex_unlock(&run_lock_a);
}

Each thread will block in pthread_cond_wait() until the other thread signals it to wake up. This will not deadlock.

It can easily be extended to many threads, by allocating one int, pthread_cond_t and pthread_mutex_t per thread.

You can't compare a pthread_mutex_t with a int.

You can use

int pthread_mutex_trylock(pthread_mutex_t *mutex);

to check that.