Reentrant (Recursive) Async Lock is Impossible in C#

• The standard way to achieve lock reentry (i.e. thread affinity) is unavailable for async locks.
• An ExecutionContext seems like a valid alternative to thread affinity, but actually cannot guarantee mutual exclusion.
• If you need a reentrant async lock — you are out of luck and would have to get rid of lock reentry in your code-base instead.
• If you are using some 3rd party reentrant async lock — you are advised to get rid of it as well.

Lock reentry

Reentrant locks are controversial even without asynchrony mixed in, but with synchronous code they do work correctly (as one would expect). By tying a lock to the thread that first acquired it and then allowing that thread to re-acquire the lock as many times as it pleases (e.g. Mutex, Monitor), we still preserve the fundamental guarantee of any lock — mutual exclusion (no parallel access).

Async locks

Things become more complicated once we try to lock around an await.

Since the code before and after an await can execute on different threads (in the general case), thread-affine locks can no longer be used. Trying to release such a lock on any thread other than the one that acquired it will result in an exception.

In fact, the C# compiler will even generate a build error if you try to await inside a lock statement to save you from trouble. Of course, the lock statement is but a syntactic sugar, and nothing is stopping you from using synchronization primitives directly and shooting yourself in the foot.

Another reason to avoid traditional synchronization primitives inside your async methods is that synchronously blocking (especially on the calling thread) may defeat the purpose of using asynchrony in the first place.

Enter the AsyncLock (and its variants):

These locks work with async code by:

1. Allowing you to wait asynchronously (await)
2. Not tying a lock to any thread
3. Not allowing reentry to ensure mutual exclusion

The last point is of particular importance for this article. Of course, if a lock is not thread-affine then how could it allow reentry, from which thread? And if we cannot tie a lock to any particular thread, what other options do we have?

ExecutionContext

ExecutionContext seems like a good candidate to replace thread affinity and allow reentry for async locks. We can store our own data on an ExecutionContext using AsyncLocal (or previously using CallContext.LogicalSetData), and it will flow to whichever thread continues execution after an await throughout the whole async call chain (unless manually suppressed). It can also be removed from the thread on which we have started the execution of an async method, after an await:

The problem is that by default ExecutionContext flows to all the threads to which your work might be moved to down the call chain¹. This is true when you manually move your work to another thread, e.g. using Task.Run, Thread.Start, ThreadPool.QueueUserWorkItem, etc. Perhaps less obvious, this is true even if you have multiple async continuations executing in parallel:

So what we have here is multiple threads “inheriting” ExecutionContext in parallel. This means that ExecutionContext cannot substitute thread affinity and does not help us build a lock, which first and foremost should be about exclusive access. If multiple threads can have an equal claim to the lock and would be allowed to reenter it in parallel then this is no lock at all.

AsyncLocal ValueChangedHandler

We have established that even in the simplest case with an await we could have multiple threads executing within an async lock, therefore no one thread could claim exclusive ownership of the lock. But what if we had a way to track ExecutionContext flow from thread to thread and pass lock ownership as a baton?

“Please, pass the ownership”

AsyncLocal has a constructor which accepts a delegate to be invoked when the AsyncLocal's value is changed on any thread, including when it's changed because of the ExecutionContext flow:

Thus, we could track the ExecutionContext flow and, therefore, track which threads are currently executing within a lock. Let's see what happens if we implement an async lock that tries to track thread ownership in this way and allows reentry only from the thread which currently owns the lock:

This example is a little more involved, so let’s break down what could happen here in chronological order:

1. MainAsync starting on thread 1
2. ChildAsync starting on thread 1
3. ChildAsync continuing on thread 2
4. MainAsync continuing on thread 3
5. ChildAsync finished NonThreadSafe
6. MainAsync finished NonThreadSafe

After step 2 there is a 100 ms delay, when no thread is executing within the lock and, therefore, no thread can own it.

At step 3 ChildAsync method continues its execution on the thread-pool thread 2. Since this is the only thread within the lock it is assigned the ownership of the lock, and allowed to reenter the lock to perform NonThreadSafe operation.

1 ms later at step 4 MainAsync method continues execution on thread-pool thread 3. It also proceeds to perform NonThreadSafe operation since it is already within the lock.

At this moment (between steps 4 and 5) we have 2 threads executing NonThreadSafe operation in parallel, even though in both cases the operation is protected by the lock. At best this will end with an exception, at worst with silent data corruption.

Even though we managed to track which threads are executing within the lock, it does not allow us to give ownership of the lock to any thread. And without lock ownership we cannot allow lock reentry if we want to guarantee mutual exclusion and thread safety of lock protected operations.

Conclusion

In the end, reentrant async lock is just impossible in C#. I concentrated on examining ExecutionContext as a way to replace thread affinity for async locks mostly because it is the way every single attempt at implementing² reentrant async lock I've seen tried to do it. It just does not work and introduces subtle, hard-to-diagnose bugs.

The problem here is not with ExecutionContext, but with async/await itself, which often leads to code being executed on multiple threads. In this situation there is simply no way (even in theory) to assign lock ownership to any thread, and without that no way to implement lock reentry in a thread-safe (i.e. mutually exclusive) manner.

If you find yourself looking for a reentrant async lock then do not waste your time on this. Your options are:

1. Get rid of lock reentry (or even locks altogether) by refactoring your code.
2. Consider if the code you are trying to protect with locks really needs to be asynchronous. Despite popular belief, async/await does not automatically make everything better and may be unnecessary in some cases.

Updates

1

After publishing this article another reentrant AsyncLock implementation came to my attention. It seems to have successfully eliminated some of the problems I talked about here (i.e. WaitAllAsync example). The last example/problem in the article still stands.

2

Another challenge to my claims here was made by Matthew A. Thomas. The challenge was serious enough to make me consider putting it at the top of this article and withdrawing my claims. In the end, I decided against it and the main reason is ConfigureAwait(false).

The approach taken by Matthew relies on SynchronizationContext to control the scheduling of async continuations. The problem is that ConfigureAwait(false) will opt out of using this context, effectively escaping from the proposed lock. And most codebases should actually use ConfigureAwait(false) for every awaited task (incidentally my examples above were already using it in the important places).

The proposed implementation only works if:

1. You have control over the code you want to lock (i.e. you don’t want to impose the lock on an async code in a 3rd party library).
2. You are willing to break encapsulation boundaries of the class in which you will be using the lock in, and enforce that ConfigureAwait(false) is not used anywhere in the code that you need to lock. Of course, this is only a concern if the lock should apply its mutual exclusion guarantee across multiple classes.

So my claim now changes to: reentrant async lock is impossible in the general case, but in some special cases it might work. (There are also other limitations to this approach, but none are as serious as ConfigureAwait(false), so I'll leave it at that).

Footnotes:

1. ExecutionContext will also flow up the synchronous call chain (i.e. if your method is not async ), and from there it can spread to even more unexpected places.
2. Not trying to throw any shade on Stephen Cleary here — he never claimed that this particular lock is ready to be used by anyone, and it has never actually seen the light of day.