How do I detect if a thread died, and then restart it?

I had a similar issue and stumbled across this question. I found that join takes a timeout argument, and that is_alive will return False once the thread is joined. So my audit for each thread is:

def check_thread_alive(thr):
    thr.join(timeout=0.0)
    return thr.is_alive()

This detects thread death for me.

You could potentially put in an a try except around where you expect it to crash (if it can be anywhere you can do it around the whole run function) and have an indicator variable which has its status.

So something like the following:

class MyThread(threading.Thread):
    def __init__(self, pass_value):
        super(MyThread, self).__init__()
        self.running = False
        self.value = pass_value
        self.RUNNING = 0
        self.FINISHED_OK  = 1
        self.STOPPED = 2
        self.CRASHED = 3
        self.status = self.STOPPED

    def run(self):
        self.running = True    
        self.status = self.RUNNING


        while self.running:
            time.sleep(0.25)

            rand = random.randint(0,10)
            print threading.current_thread().name, rand, self.value

            try:
                if rand == 4:
                    raise ValueError('Returned 4!')
            except:
                self.status = self.CRASHED

Then you can use your loop:

while True:
    # Create a copy of our groups to iterate over, 
    # so that we can delete dead threads if needed
    for m in group1[:]:
        if m.status == m.CRASHED:
            value = m.value
            group1.remove(m)
            group1.append(MyThread(value))

    for m in group2[:]:
        if m.status == m.CRASHED:
            value = m.value
            group2.remove(m)
            group2.append(MyThread(value))

time.sleep(5.0)