Write to file from multiple threads async with C# and .NET Core

There are several patterns on how to allow multiple threads to write to the same file. the ReaderWriterLock class is invented for this purpose. Another classic is using semaphors and the lock statement to lock a shared resource.

This article explains how to use a ConcurrentQueue and a always running Task to accomplish the same feat.

The theory behind this is:

• Threads deliver what to write to the file to the ConcurrentQueue.
• A task running in the background will read from the ConcurrentQueue and do the actual file writing.

This allows the shared resource to be access from one thread only (the task running in the background) and everyone else to deliver their payload to a thread-safe queue.

But enough talk, lets code.

THE FILE WRITER CLASS

using System.Collections.Concurrent;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace MyCode
{
  public class MultiThreadFileWriter
  {
    private static ConcurrentQueue<string> _textToWrite = new ConcurrentQueue<string>();
    private CancellationTokenSource _source = new CancellationTokenSource();
    private CancellationToken _token;

    public MultiThreadFileWriter()
    {
      _token = _source.Token;
      // This is the task that will run
      // in the background and do the actual file writing
      Task.Run(WriteToFile, _token);
    }

    /// The public method where a thread can ask for a line
    /// to be written.
    public void WriteLine(string line)
    {
      _textToWrite.Enqueue(line);
    }

    /// The actual file writer, running
    /// in the background.
    private async void WriteToFile()
    {
      while (true)
      {
        if (_token.IsCancellationRequested)
        {
          return;
        }
        using (StreamWriter w = File.AppendText("c:\\myfile.txt"))
        {
          while (_textToWrite.TryDequeue(out string textLine))
          {
            await w.WriteLineAsync(textLine);
          }
          w.Flush();
          Thread.Sleep(100);
        }
      }
    }
  }
}

// Somewhere in the startup.cs or the Main.cs file
services.AddSingleton<MultiThreadFileWriter>();
// Now you can add the class using constructor injection
// and call the WriteLine() function from any thread without
// worrying about thread safety

Nothice that my code introduces a Thread.Sleep(100) statement. This is not needed, but it can be a good idea to give your application a little breathing space, especially if there are periods where nothing is written. Remove the line if your code requires an instant file write pattern.

MORE TO READ:

• ReaderWriterLock from Microsoft
• Multiple threads writing to 1 file from daniweb
• ConcurrentQueue from Microsoft