Handle exception inside background async Task

I've developed a queuing system in C#.NET and I'm trying to catch exceptions and retry errors from inside the tasks of the queue. I cannot use await to start-up the queue or run tasks as it will block execution. I need to catch and handle exceptions inside FileQueueTest.ProcessFile().

Below is the code I've written to demonstrate the queue. I've purposely put a non-existent file to force an exception. However, the exception is not handled in the try/catch block of the async task and crashing the program. I've included the program's output and a screenshot of the unhandled exception in Visual Studio.

Any help would be greatly appreciated! Thanks!

public class FileQueueTest : List<string>
{
    SemaphoreSlim mutex = new SemaphoreSlim(1);
    List<string> doneFiles = new List<string>();

    public async Task Start()
    {
        // This call cannot block as files will be queued outside this class at any time.
        Task.Factory.StartNew(WatchQueue);
    }

    async Task WatchQueue()
    {
        while (true)
        {
            await mutex.WaitAsync();
                
            var nextFile = this.FirstOrDefault();

            if (nextFile != null)
            {
                Task.Factory.StartNew(() => QueueAndWait(nextFile));
                RemoveAt(0);
            }

            mutex.Release();

            await Task.Delay(100);
        }
    }

    async Task QueueAndWait(string filename)
    {
        await ProcessFile(filename);

        await mutex.WaitAsync();
        doneFiles.Add(filename);
        mutex.Release();
    }

    public async Task QueueFile(string filename)
    {
        await mutex.WaitAsync();
        Add(filename);
        mutex.Release();
    }

    async Task ProcessFile(string filename)
    {
        Console.WriteLine($"Opening {filename}");

        FileStream stream;
        int retry = 0;

        while (true)
        {
            try
            {
                stream = File.OpenRead(filename);

                // do dummy work.
                Console.WriteLine($"Processing {filename}");
                await Task.Delay(1000);
                break;
            }
            catch (Exception ex)
            {
                // I want to catch exceptions here within the context of processing this file.
                if (++retry == 5)
                {
                    throw ex;
                }

                Console.WriteLine($"Retry #{retry} for {filename}: {ex.Message}");

                await Task.Delay(1000);
            }
        }

        if (stream != null)
        {
            stream.Dispose();
        }

        Console.WriteLine($"Closed {filename}");
    }
}

var queue = new FileQueueTest();
await queue.Start();

var files = new string[]
{
    @"C:\Work\test1.txt",
    @"C:\Work\test2.txt",
    @"C:\Work\test3.txt",
    @"C:\Work\does_not_exist.txt"
};

// simulate randomly queueing files in the background.
foreach (var file in files)
{
    await Task.Delay(new Random().Next(50, 1000));
    await queue.QueueFile(file);
}

Program output:

Opening C:\Work\test1.txt
Processing C:\Work\test1.txt
Opening C:\Work\test2.txt
Processing C:\Work\test2.txt
Closed C:\Work\test1.txt
Opening C:\Work\test3.txt
Processing C:\Work\test3.txt
Closed C:\Work\test2.txt
Opening C:\Work\does_not_exist.txt
Exception thrown: 'System.IO.FileNotFoundException' in mscorlib.dll
Could not find file 'C:\Work\does_not_exist.txt'.

Unhandled exception in Visual Studio