Since C# 8, developers can use IAsyncEnumerable to asynchronously iterate over a collection. However, I found out recently through hours of blood and tears that combining asynchronous iteration with lock can lead to unexpected and dangerous behavior. In this post, we will explore one of the reasons why using yield return inside a lock statement is problematic.

Problem

Consider the following minimal example, where a producer-consumer pattern is implemented using a queue and protected by a lock:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;

class Program
{
    private static readonly Queue<int> _pq = new();
    private static readonly object _lock = new object();

    static async Task Main()
    {
        var producerTask = Task.Run(async () =>
        {
            while (true)
            {
                lock (_lock)
                {
                    Console.WriteLine("Enqueueing on thread {0}", Environment.CurrentManagedThreadId);
                    _pq.Enqueue(1);
                }

                await Task.Delay(50);
            }
        });

        var consumerTask = Task.Run(async () =>
        {
            try
            {
                await foreach (var item in DequeueWithYield())
                {
                    Console.WriteLine($"Processing item: {item} on thread {Environment.CurrentManagedThreadId}");
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex); // 5
            }
        });

        await Task.WhenAll(producerTask, consumerTask);
    }

    static async IAsyncEnumerable<int> DequeueWithYield()
    {
        while (true)
        {
            Console.WriteLine($"Before yield {Environment.CurrentManagedThreadId}"); // 1
            await Task.Yield();
            Console.WriteLine($"After yield {Environment.CurrentManagedThreadId}"); // 2
            lock (_lock)
            {
                Console.WriteLine($"Before dequeuing on thread {Environment.CurrentManagedThreadId}"); // 3
                yield return _pq.TryDequeue(out var value) ? value : 0;
                Console.WriteLine($"After dequeuing on thread {Environment.CurrentManagedThreadId}"); // 4
            }
        }
    }
}

What Happens Here:

1. Producer Task: Enqueues an item into _pq every 50ms inside a lock.
2. Consumer Task: Uses an async IAsyncEnumerable<int> method (DequeueWithYield) to dequeue items.
3. Inside DequeueWithYield:
   • It calls await Task.Yield() before acquiring the lock which effectively returns the control back to the Task scheduler, allowing other tasks to run before running the continuation, which can possibly run in another thread. Task.Yield (or any async method that will yield) is absolutely necessary for this contrived experiment.

Danger: Synchronization Lock Exception

1. The continuation after await Task.Yield() is not guaranteed to run on the same thread since there is no SynchronizationContext (i.e. null) by default. Therefore, it’s possible that Point 1 and Point 2 will run on different threads, let’s denote this as X and Y. Point 2 and Point 3 will definitely run in the same thread since they are run synchronously.
2. Now the magic comes after Point 3. Since the consumer task is awaiting from IAsyncEnumerable using await foreach syntax, which is almost equivalent to var e = DequeueWithYield(); while (await e.MoveNextAsync()) { var curr = e.Current; ... };. This can be verified by looking at low-level IL C# using Rider decompiler.
3. Initially, the first iteration of e.MoveNextAsync() will try to retrieve the first value, running at thread X. Note that just because we are calling an async method, doesn’t mean it automatically will run the method asynchronously. The method needs to explicitly yield to release the control back to the original caller, which is why we absolutely need Task.Yield(), without it, this error won’t be observed.
4. Since the continuation of Task.Yield() possibly runs on thread Y, acquiring the lock and yielding the first value from the queue and then eventually blocks until the caller calls MoveNextAsync again.
5. But the problem is the original caller still runs on thread X, and eventually after the first loop will call MoveNextAsync again to get the second value, but the caller runs on thread X and it will cause point 4 to run on thread X, and eventually causing thread X to be the one releasing the lock.
6. Since lock in C# can only be acquired and released by the same thread, it will throw System.Threading.SynchronizationLockException: Object synchronization method was called from an unsynchronized block of code.

The Correct Approach

static async IAsyncEnumerable<int> DequeueSafely()
{
    while (true)
    {
        int item;
        lock (_lock)
        {
            item = _pq.TryDequeue(out var value) ? value : 0;
        }
        
        yield return item; // Outside the lock
        await Task.Yield();
    }
}

Do not call yield return or fancy magic method calls within a lock block. Always minimize the critical section and perform the fancy stuffs after exiting the lock.

Conclusion

Yielding while holding a lock is problematic in general case. For example, in Linux, it can lead to Priority Inversion and Deadlock Potential. In C# and in the context of single lock usage, yielding inside a lock might not be problematic until concurrency hits you hard and this can lead to hours of hell to debug this nasty concurrency bug. So please, do not yield return within a lock block!