A call to CancellationTokenSource.Cancel never returns

This is the expected behavior of CancellationToken/Source.

Somewhat similar to how TaskCompletionSource works, CancellationToken registrations are executed synchronously using the calling thread. You can see that in CancellationTokenSource.ExecuteCallbackHandlers that gets called when you cancel.

It's much more efficient to use that same thread than to schedule all these continuations on the ThreadPool. Usually this behavior isn't a problem, but it can be if you call CancellationTokenSource.Cancel inside a lock as the thread is "hijacked" while the lock is still taken. You can solve such issues by using Task.Run. You can even make it an extension method:

public static void CancelWithBackgroundContinuations(this CancellationTokenSource)
{
    Task.Run(() => CancellationTokenSource.Cancel());
    cancellationTokenSource.Token.WaitHandle.WaitOne(); // make sure to only continue when the cancellation completed (without waiting for all the callbacks)
}

CancellationTokenSource.Cancel doesn't simply set the IsCancellationRequested flag.

The CancallationToken class has a Register method, which lets you register callbacks that will be called on cancellation. And these callbacks are called by CancellationTokenSource.Cancel.

Let's take a look at the source code:

public void Cancel()
{
    Cancel(false);
}

public void Cancel(bool throwOnFirstException)
{
    ThrowIfDisposed();
    NotifyCancellation(throwOnFirstException);            
}

Here's the NotifyCancellation method:

private void NotifyCancellation(bool throwOnFirstException)
{
    // fast-path test to check if Notify has been called previously
    if (IsCancellationRequested)
        return;

    // If we're the first to signal cancellation, do the main extra work.
    if (Interlocked.CompareExchange(ref m_state, NOTIFYING, NOT_CANCELED) == NOT_CANCELED)
    {
        // Dispose of the timer, if any
        Timer timer = m_timer;
        if(timer != null) timer.Dispose();

        //record the threadID being used for running the callbacks.
        ThreadIDExecutingCallbacks = Thread.CurrentThread.ManagedThreadId;

        //If the kernel event is null at this point, it will be set during lazy construction.
        if (m_kernelEvent != null)
            m_kernelEvent.Set(); // update the MRE value.

        // - late enlisters to the Canceled event will have their callbacks called immediately in the Register() methods.
        // - Callbacks are not called inside a lock.
        // - After transition, no more delegates will be added to the 
        // - list of handlers, and hence it can be consumed and cleared at leisure by ExecuteCallbackHandlers.
        ExecuteCallbackHandlers(throwOnFirstException);
        Contract.Assert(IsCancellationCompleted, "Expected cancellation to have finished");
    }
}

Ok, now the catch is that ExecuteCallbackHandlers can execute the callbacks either on the target context, or in the current context. I'll let you take a look at the ExecuteCallbackHandlers method source code as it's a bit too long to include here. But the interesting part is:

if (m_executingCallback.TargetSyncContext != null)
{

    m_executingCallback.TargetSyncContext.Send(CancellationCallbackCoreWork_OnSyncContext, args);
    // CancellationCallbackCoreWork_OnSyncContext may have altered ThreadIDExecutingCallbacks, so reset it. 
    ThreadIDExecutingCallbacks = Thread.CurrentThread.ManagedThreadId;
}
else
{
    CancellationCallbackCoreWork(args);
}

I guess now you're starting to understand where I'm going to look next... Task.Delay of course. Let's look at its source code:

// Register our cancellation token, if necessary.
if (cancellationToken.CanBeCanceled)
{
    promise.Registration = cancellationToken.InternalRegisterWithoutEC(state => ((DelayPromise)state).Complete(), promise);
}

Hmmm... what's that InternalRegisterWithoutEC method?

internal CancellationTokenRegistration InternalRegisterWithoutEC(Action<object> callback, Object state)
{
    return Register(
        callback,
        state,
        false, // useSyncContext=false
        false  // useExecutionContext=false
     );
}

Argh. useSyncContext=false - this explains the behavior you're seeing as the TargetSyncContext property used in ExecuteCallbackHandlers will be false. As the synchronization context is not used, the cancellation is executed on CancellationTokenSource.Cancel's call context.


Because of the reasons already listed here, I believe you want to actually utilize the CancellationTokenSource.CancelAfter method with a zero millisecond delay. This will allow the cancellation to propagate in a different context.

The source code for CancelAfter is here.

Internally it uses a TimerQueueTimer to make the cancel request. This is not documented but should resolve op's issue.

Documentation here.