Difference in Python thread.join() between Python 3.7 and 3.8

There is an undocumented change in the behavior of threading _shutdown() from Python version 3.7.3 to 3.7.4.

Here's how I found it:

To trace the issue, I first used the inspect package to find out who join()s the thread in the Python 3.7.3 runtime. I modified the join() function to get some output:

...
    def join(self, *args, **kwargs):
        self._stop_event.set()
        c = threading.current_thread()
        print(f"join called from thread {c}")
        print(f"calling function: {inspect.stack()[1][3]}")
        super(StoppableWorker, self).join(*args, **kwargs)
...

When executing with Python 3.7.3, this prints:

main done.
join called from thread <_MainThread(MainThread, stopped 139660844881728)>
calling function: _shutdown
hi

So the MainThread, which is already stopped, invokes the join() method. The function responsible in the MainThread is _shutdown().

From the CPython source for Python 3.7.3 for _shutdown(), lines 1279-1282:

    t = _pickSomeNonDaemonThread()
    while t:
        t.join()
        t = _pickSomeNonDaemonThread()

That code invokes join() on all non-daemon threads when the MainThread exits!

That implementation was changed in Python 3.7.4.

To verify these findings I built Python 3.7.4 from source. It indeed behaves differently. It keeps the thread running as expected and the join() function is not invoked.

This is apparently not documented in the release notes of Python 3.7.4 nor in the changelog of Python 3.8.

-- EDIT:

As pointed out in the comments by MisterMiyagi, one might argue that extending the join() function and using it for signaling termination is not a proper use of join(). IMHO that is up to taste. It should, however, be documented that in Python 3.7.3 and before, join() is invoked by the Python runtime on system exit, while with the change to 3.7.4 this is no longer the case. If properly documented, it would explain this behavior from the get-go.

What's New only lists new features. This changes looks to me like a bug fix. https://docs.python.org/3.7/whatsnew/3.7.html has a changelog link near the top. Given the research in @Felix's answer, we should look at bugfixes released in 3.7.4. https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-4-release-candidate-1

This might be the issue: https://bugs.python.org/issue36402 bpo-36402: Fix a race condition at Python shutdown when waiting for threads. Wait until the Python thread state of all non-daemon threads get deleted (join all non-daemon threads), rather than just wait until non-daemon Python threads complete.