Authentification issue to BigQuery API REST using Python

Question

I would like to make a HTTP call to this resource : https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/jobs

As I read to the documentation I use an API key generated from my GCP project to be authenticated. So with requests I make a simple call like this:

import requests

params = {'key': 'MY_API_KEY'}
base_url = 'https://bigquery.googleapis.com'
project_id = 'MY_PROJECT_ID'
r = requests.get(f'{base_url}/bigquery/v2/projects/{project_id}/jobs', params=params)

Unfortunately it returns a response 401 and I can't figure out why.

Thanks a lot and have a nice day !

Update code after guillaume blaquiere reply :

from google.auth.transport.requests import AuthorizedSession
from google.oauth2 import service_account

base_url = 'https://bigquery.googleapis.com'
project_id = 'project_id'

credentials = service_account.Credentials.from_service_account_file(
    'service_account.json',
    scopes=['https://www.googleapis.com/auth/bigquery',
            'https://www.googleapis.com/auth/cloud-platform'],
)

authed_session = AuthorizedSession(credentials)
response = authed_session.request('GET', f'{base_url}/bigquery/v2/projects/{project_id}/jobs')
print(response.json())

# this returns : {'etag': 'tAZvk1k2f2GY8yHaQF7how==', 'kind': 'bigquery#jobList'}

guillaume blaquiere · Accepted Answer · 2020-07-10 09:59:55Z

2

The API Key no longer works for a large number of Google API. Only some legacy continue to accept an API key.

Now, you need an authenticated request. You can find exemple in the google-auth python library documentation. Look at Refresh and Authorized_session.

Don't hesitate to comment if you need help about the credential obtention, I can also help you on this.

EDIT

When you perform the request, it's, by default, only on the current user. In your case, it's the service account when you use the Python code, and your User account when you use the API Explorer (the swagger like in the Google Documentation).

In your case, I guess that your service account has never performed a job (query or load job) and thus, there is no entry for it.

According with the documentation, is you want to see all the user jobs, you have to add the param ?allUsers=true at the end of your URL

response = authed_session.request('GET', f'{base_url}/bigquery/v2/projects/{project_id}/jobs?allUsers=true')

edited Jul 10, 2020 at 9:59

answered Jul 9, 2020 at 15:34

guillaume blaquiere

76.5k3 gold badges65 silver badges102 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Baptiste Over a year ago

This now works better I don't have anymore a 401. But the response is not the expected one : {'etag': 'tAZvk1k2f2GY8yHaQF7how==', 'kind': 'bigquery#jobList'}. To print this I used response.json(). I can't show my code in comment. I followed the instructions from your link.

guillaume blaquiere Over a year ago

Did you already create a job? What the answer of this command

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" https://bigquery.googleapis.com/bigquery/v2/projects/PROJECT_ID/jobs

(change the project ID with yours)

Baptiste Over a year ago

The command returns me a 401. I used the "swagger" of Google from the doc and it returns a list of jobs. So yes I have created jobs :)

guillaume blaquiere Over a year ago

Did you remove the API key in your request?

guillaume blaquiere Over a year ago

I updated my answer. In summary: Yes YOU have created jobs, not your service account. Add a param to view the jobs of all users!!

Collectives™ on Stack Overflow

Authentification issue to BigQuery API REST using Python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related