Google Directory API - Error 400 when using a Service Account

I am using the latest Beta of the Directory API and when trying to get a list of users I am getting the following:

"The remote server returned an error: (400) Bad Request."

I am able to execute this functionality from the API Explorer (https://developers.google.com/apis-explorer/#p/admin/directory_v1/directory.users.list) and I have all the appropriate permissions and scope set.

C# code below:

            X509Certificate2 certificate = new X509Certificate2(
                SERVICE_ACCOUNT_PKCS12_FILE_PATH,
                "KEY_PASSWORD", X509KeyStorageFlags.Exportable);

            var provider = new AssertionFlowClient(GoogleAuthenticationServer.Description, certificate)

            var provider = new AssertionFlowClient(GoogleAuthenticationServer.Description, certificate)
            {
                ServiceAccountId = SERVICE_ACCOUNT_EMAIL,
                Scope = Utilities.GetStringValue(
                    DirectoryService.Scopes.AdminDirectoryUser),
                ServiceAccountUser = SERVICE_ACCOUNT_USER,
            };
            var auth = new OAuth2Authenticator<AssertionFlowClient>(provider, AssertionFlowClient.GetState);

            var service = new DirectoryService(new BaseClientService.Initializer()
            {
                Authenticator = auth,
                ApplicationName = "API Project Name",                
            });

            var usersList = service.Users.List();
            usersList.Domain = "mydomain";

            Users results = usersList.Execute();