Entity Framework Core doesn't automatically fix-up navigation

I have two entities, an entity Contact that has a navigation property Buyer that may exist, and an entity Buyer that has a navigation property Contact that must exist. All Buyer's have exactly one Contact, all Contact's may have zero or one Buyer's.

The problem that occurs is that when a Contact (that has a Buyer) is loaded, the Buyer cannot be loaded either through Eager or Explicit loading.

public class Contact
{
    public int ContactID { get; set; }
    public string FirstName { get; set; } = null!;
    public string LastName { get; set; } = null!;
    public string Email { get; set; } = null!;
    public virtual Buyer? Buyer { get; set; }
}
public class Buyer
{
    public int BuyerID { get; set; }
    public string CompanyName { get; set; } = default!;
    public string ProductName { get; set; } = default!;
    public int ContactID { get; set; }
    public virtual Contact Contact { get; set; } = new Contact();
}

When I create the entities:

 // existing Contact already initialized with Buyer == null and added
 var newBuyer = new Buyer() { CompanyName = "Acme", ProductName = "Anvil" };
 newBuyer.ContactID = contactID;
 // Load the reference to the Contact
 newBuyer.Contact = await _context.Contacts.SingleOrDefaultAsync(c => c.ContactID == contactID);
 // error checking elided (but in this test it is not failing)
 // newBuyer.Contact.Buyer is null if examined
 _context.Buyers.Add(newBuyer);
 // newBuyer.Contact.Buyer is now newBuyer, automatic fix-up
 await _context.SaveChangesAsync();

Looking at the underlying database everything is as expected.

Now I attempt to load the Contact and navigation properties two different ways expecting automatic fix-ups:

 Contact = await _context.Contacts.FindAsync(id);
 // The Contact.Buyer is null here as expected, so explicitly Load
 _context.Entry(Contact).Reference(c => c.Buyer).Load();
 // The Contact.Buyer is still null here, so try DetectChanges
 _context.ChangeTracker.DetectChanges();
 // The Contact.Buyer is still null here, so try again with Eager Loading
 Contact = await _context.Contacts.Include(c => c.Buyer).FirstOrDefaultAsync(m => m.ContactID == id);
 // The Contact.Buyer is still null here! What is wrong?

When tracing in a debugger, the first explicit Load() sees Buyer as a navigation property and successfully loads it into memory. Also looking at _contacts.Buyers shows that it is in memory.
The DetectChanges was added just in case, it makes no difference.
The Eager loading using Include also is not causing the fix-ups.
Lazy loading was also tried and failed.

Does anyone have any idea how to get the automatic fixup to work?

The fluent API:

 modelBuilder.Entity<Contact>()
             .HasKey("ContactID");
 modelBuilder.Entity<Buyer>()
             .HasKey(p => p.BuyerID);
 modelBuilder.Entity<Buyer>()
             .HasOne<Contact>(p => p.Contact)
             .WithOne("Buyer")
             .HasForeignKey("Buyer", "ContactID")
             .OnDelete(DeleteBehavior.Cascade)
             .IsRequired();

Notes: EF Core 3.1.3 Net Core API 3.1.0 Nullable Enable

[Edit] By adding the following line of code before the FindAsync it causes all Buyer's to be loaded into memory/cache, the Contact.Buyer buyer is then automatically fixed up after the first FindAsync(). This shows that fixups can occur. But I don't want to forcibly load the whole table.

var test = _context.Buyers.ToList();