Azure DevOps SourceLink and Symbol Server with Release DLL in Nuget package

I have a .NET Standard 2.0 TestSouceLink project with the following configuration in the .csproj:

<PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <RestoreProjectStyle>PackageReference</RestoreProjectStyle>
    <Authors>Jérôme MEVEL</Authors>
    <Description>Just a test package for SourceLink</Description>
    <Version>1.1.1</Version>

    <PublishRepositoryUrl>true</PublishRepositoryUrl>
    <EmbedUntrackedSources>true</EmbedUntrackedSources>
</PropertyGroup>

<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'">
    <DebugType>full</DebugType>
    <DebugSymbols>true</DebugSymbols>
</PropertyGroup>

<ItemGroup>
    <PackageReference Include="Microsoft.SourceLink.Vsts.Git" Version="1.0.0-beta2-18618-05">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
    </PackageReference>
</ItemGroup>

We have an Azure DevOps server which generates our Nuget packages in build pipelines.

I added a Publish symbols path task at the end to push the .pdb files to the Symbol Server and I managed to get Source Link to work along with the Azure DevOps Symbol Server with the Debug build configuration only.

The problem is that I don't want my DLLs in the Nuget packages to be generated in Debug mode. I want them to be Release. But when I change the dotnet pack task to use Release in the Configuration to package text box, then Source Link doesn't work anymore.

Is there a way to generate a Nuget package with a Release DLL inside? and to still have Source Link working with the Symbol Server?

I don't mind using a .nuspec file along with the Nuget CLI if this is the only way however I don't want to include the .pdb files inside the Nuget packages.

Thanks