SourceLink Authentication error when using Microsoft.SourceLink.AzureRepos.Git

I am trying to enable SourceLink for our internal NuGet packages and am running into a problem where I don't see how I can solve it.

My setup is:

• Visual Studio 2019 Enterprise (16.3.8)
• An Azure DevOps git repository
• A .NET Framework 4.7.2 class library with an SDK style project containing a single HelloWorld class
• A PackageReference to Microsoft.SourceLink.AzureRepos.Git, using the latest Version (1.0.0-beta2-19554-01)
• This project generates a package that includes the DLL and the PDB (I know this is discouraged, but I would like to get it to work this way rather than using a symbol server; this also does not seem to be the source of the problem, since the debugger finds the correct PDB)

The project file

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net472</TargetFramework>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <AllowedOutputExtensionsInPackageBuildOutputFolder>$(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb</AllowedOutputExtensionsInPackageBuildOutputFolder>
    <RepositoryUrl>https://<host>.visualstudio.com/<project>/_git/Lib</RepositoryUrl>
    <RepositoryType>git</RepositoryType>
    <AssemblyVersion>2.0.0.0</AssemblyVersion>
    <FileVersion>2.0.0.0</FileVersion>
    <Version>2.0.0</Version>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.SourceLink.AzureRepos.Git" Version="1.0.0-beta2-19554-01">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>
</Project>

• I commit my changes to the repository and build the package. The package builds fine. When I check the generated .nuspec file, it contains the correct commit hash.
• I publish it to a local NuGet feed using nuget add Lib.2.0.0.nupkg -Source C:\NuGet.
• I delete my local HelloWorld sources so the debugger does not find these sources and tries to download them from the git repository
• I create a console application, reference this NuGet package, and call the HelloWorld function on the HelloWorld class. In the debugger, I hit F11 to step into the HelloWorld call. The "SourceLink will download [...]" dialog box appears and I select "Download Source and Continue Debugging".

This does not work. I get the following error message

Source Link Error:
ERROR: Azure DevOps: Authentication failed for all accounts. Use 'File -> Account Settings...' to add a new account or refresh credentials.

Source Link URL: https://<host>.visualstudio.com/<project>/_apis/git/repositories/Lib/items?api-version=1.0&versionType=commit&version=cf94d5a0f224b6983c3885a2a6ac29aa52cec418&path=/HelloWorld/HelloWorld.cs

Now, the error message is pretty clear, I just don't get it. When I go to the URL stated in the error message, I see the file content I am expecting. I used a browser I never use to check, and when navigating to this page, it asked me to sign in, and I do so using the same account I use in Visual Studio. I also tries signing out of Visual Studio, restarting it, and signing in again to Visual Studio using the same account I used in the browser. Still the same error.

I am kind of lost. Does anybody have any idea of what else I could try?