Azure Static Web App with custom authentication(Azure AD) has a login Loop

I'm trying to build custom authentication for a Azure Static Web App. I've followed the tutorial of assigning roles via Graph API. I've it working for two users, everyone else get's a login loop.

Added custom authentication in the staticwebapp.config.

{
  "navigationFallback": {
    "rewrite": "/index.html",
    "exclude": ["/images/*.{png,jpg,gif}", "/css/*"]
  },
  "routes": [
    {
      "route": "/logout",
      "redirect": "/.auth/logout"
    },
    {
      "route": "/.auth/login/twitter",
      "statusCode": 404
    },
    {
      "route": "/.auth/login/github",
      "statusCode": 404
    },
    {
      "route": "/profile-management",
      "allowedRoles": ["manager"]
    },
    {
      "route": "/*",
      "allowedRoles": ["authenticated", "consultant", "manager"]
    }
  ],
  "auth": {
    "rolesSource": "/api/auth/getroles",
    "identityProviders": {
      "azureActiveDirectory": {
        "userDetailsClaim": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name",
        "registration": {
          "openIdIssuer": "https://login.microsoftonline.com/{tenant_id}",
          "clientIdSettingName": "Aad_Client_Id",
          "clientSecretSettingName": "Aad_Client_Secret"
        },
        "login": {
          "loginParameters": ["resource=https://graph.microsoft.com"]
        }
      }
    }
  },
  "responseOverrides": {
    "401": {
      "redirect": "/.auth/login/aad",
      "statusCode": 302
    }
  }
}

The client id from the app registration is set in the application settings of the Azure Static Web App and the secret is set in the application settings as a Key Vault Reference. The Azure Static Web App is added to the Azure Key Vault via managed identity.

Azure Static Web App - Application Settings

Azure Static Web App - App Registrations