Making a request to a RESTful API using Python

Question

I have a RESTful API that I have exposed using an implementation of Elasticsearch on an EC2 instance to index a corpus of content. I can query the search by running the following from my terminal (MacOSX):

curl -XGET 'http://ES_search_demo.com/document/record/_search?pretty=true' -d '{
  "query": {
    "bool": {
      "must": [
        {
          "text": {
            "record.document": "SOME_JOURNAL"
          }
        },
        {
          "text": {
            "record.articleTitle": "farmers"
          }
        }
      ],
      "must_not": [],
      "should": []
    }
  },
  "from": 0,
  "size": 50,
  "sort": [],
  "facets": {}
}'

How do I turn above into a API request using python/requests or python/urllib2 (not sure which one to go for - have been using urllib2, but hear that requests is better...)? Do I pass as a header or otherwise?

Yacc · Accepted Answer · 2020-02-05 13:25:55Z

403

Using requests:

import requests
url = 'http://ES_search_demo.com/document/record/_search?pretty=true'
data = '''{
  "query": {
    "bool": {
      "must": [
        {
          "text": {
            "record.document": "SOME_JOURNAL"
          }
        },
        {
          "text": {
            "record.articleTitle": "farmers"
          }
        }
      ],
      "must_not": [],
      "should": []
    }
  },
  "from": 0,
  "size": 50,
  "sort": [],
  "facets": {}
}'''
response = requests.post(url, data=data)

Depending on what kind of response your API returns, you will then probably want to look at response.text or response.json() (or possibly inspect response.status_code first). See the quickstart docs here, especially this section.

edited Feb 5, 2020 at 13:25

Yacc

1,12812 silver badges16 bronze badges

answered Jun 25, 2013 at 19:55

andersschuller

14k2 gold badges45 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

danio Over a year ago

@ParveenShukhala "Requests officially supports Python 2.6–2.7 & 3.3–3.5, and runs great on PyPy." -- pypi.python.org/pypi/requests

Mark Chorley Over a year ago

As it's JSON you are sending, you can use the json parameter rather than data like so: response = requests.post(url, json=data)

HVS · Accepted Answer · 2015-09-22 16:23:57Z

123

Using requests and json makes it simple.

Call the API
Assuming the API returns a JSON, parse the JSON object into a Python dict using json.loads function
Loop through the dict to extract information.

Requests module provides you useful function to loop for success and failure.

if(Response.ok): will help help you determine if your API call is successful (Response code - 200)

Response.raise_for_status() will help you fetch the http code that is returned from the API.

Below is a sample code for making such API calls. Also can be found in github. The code assumes that the API makes use of digest authentication. You can either skip this or use other appropriate authentication modules to authenticate the client invoking the API.

#Python 2.7.6
#RestfulClient.py

import requests
from requests.auth import HTTPDigestAuth
import json

# Replace with the correct URL
url = "http://api_url"

# It is a good practice not to hardcode the credentials. So ask the user to enter credentials at runtime
myResponse = requests.get(url,auth=HTTPDigestAuth(raw_input("username: "), raw_input("Password: ")), verify=True)
#print (myResponse.status_code)

# For successful API call, response code will be 200 (OK)
if(myResponse.ok):

    # Loading the response data into a dict variable
    # json.loads takes in only binary or string variables so using content to fetch binary content
    # Loads (Load String) takes a Json file and converts into python data structure (dict or list, depending on JSON)
    jData = json.loads(myResponse.content)

    print("The response contains {0} properties".format(len(jData)))
    print("\n")
    for key in jData:
        print key + " : " + jData[key]
else:
  # If response code is not ok (200), print the resulting http error code with description
    myResponse.raise_for_status()

answered Sep 22, 2015 at 16:23

HVS

2,5073 gold badges21 silver badges20 bronze badges

5 Comments

Denis The Menace Over a year ago

Last portion with iteration over keys will not always work because JSON document may have array as a top level element. So, it would be an error to try to get jData[key]

qasimalbaqali Over a year ago

@DenisTheMenace if it is an array, how would I loop around it?

Denis The Menace Over a year ago

@qasimalbaqali the same way you loop over dictionary. But array elements will be simply jData, not jData[key]

Marco Over a year ago

Sidenote: If your API returns a large JSON response, you can pretty print it like this: print(json.dumps(jData, indent=4, sort_keys=True))

Mirkules Over a year ago

Under python3, the following was spit out 'JSON must be str not bytes'. This is fixed by decoding the output, i.e. json.loads(myResponse.content.decode('utf-8')). Also you should wrap key and jData key with str() so when the RESTful API returns integers, it doesn't complain.

Shashank G · Accepted Answer · 2017-05-18 10:12:47Z

14

Below is the program to execute the rest api in python-

import requests
url = 'https://url'
data = '{  "platform": {    "login": {      "userName": "name",      "password": "pwd"    }  } }'
response = requests.post(url, data=data,headers={"Content-Type": "application/json"})
print(response)
sid=response.json()['platform']['login']['sessionId']   //to extract the detail from response
print(response.text)
print(sid)

answered May 18, 2017 at 10:12

Shashank G

9922 gold badges17 silver badges27 bronze badges

1 Comment

KWC Over a year ago

Not work for me. Instead I use the advice here. Seems work: realpython.com/api-integration-in-python

gvir · Accepted Answer · 2018-05-27 18:43:07Z

14

So you want to pass data in body of a GET request, better would be to do it in POST call. You can achieve this by using both Requests.

Raw Request

GET http://ES_search_demo.com/document/record/_search?pretty=true HTTP/1.1
Host: ES_search_demo.com
Content-Length: 183
User-Agent: python-requests/2.9.0
Connection: keep-alive
Accept: */*
Accept-Encoding: gzip, deflate

{
  "query": {
    "bool": {
      "must": [
        {
          "text": {
            "record.document": "SOME_JOURNAL"
          }
        },
        {
          "text": {
            "record.articleTitle": "farmers"
          }
        }
      ],
      "must_not": [],
      "should": []
    }
  },
  "from": 0,
  "size": 50,
  "sort": [],
  "facets": {}
}

Sample call with Requests

import requests

def consumeGETRequestSync():
data = '{
  "query": {
    "bool": {
      "must": [
        {
          "text": {
            "record.document": "SOME_JOURNAL"
          }
        },
        {
          "text": {
            "record.articleTitle": "farmers"
          }
        }
      ],
      "must_not": [],
      "should": []
    }
  },
  "from": 0,
  "size": 50,
  "sort": [],
  "facets": {}
}'
url = 'http://ES_search_demo.com/document/record/_search?pretty=true'
headers = {"Accept": "application/json"}
# call get service with headers and params
response = requests.get(url,data = data)
print "code:"+ str(response.status_code)
print "******************"
print "headers:"+ str(response.headers)
print "******************"
print "content:"+ str(response.text)

consumeGETRequestSync()

edited May 27, 2018 at 18:43

user3956566

answered Jan 19, 2016 at 17:01

gvir

2643 silver badges4 bronze badges

2 Comments

Nick Over a year ago

got a dead link there

Markus Meyer Over a year ago

headers variable should be used: requests.get(...headers = headers, ....)

Nokados · Accepted Answer · 2023-06-11 13:46:03Z

For now, httpx is a better option than requests. It has broadly request-compatible API, supports async, HTTP/2 and has a good performance. It is used by, for example, python-telegram-bot - a popular client for Telegram bots.

import httpx
url = 'http://ES_search_demo.com/document/record/_search?pretty=true'
data = {
  "query": {
    "bool": {
      "must": [
        {
          "text": {
            "record.document": "SOME_JOURNAL"
          }
        },
        {
          "text": {
            "record.articleTitle": "farmers"
          }
        }
      ],
      "must_not": [],
      "should": []
    }
  },
  "from": 0,
  "size": 50,
  "sort": [],
  "facets": {}
}
response = httpx.post(url, json=data)
print(response.json())

However, if you want to request Elasticsearch, it's better to use specific elasticsearch API clients: elasticsearch or high-level elasticsearch-dsl

Collectives™ on Stack Overflow

Making a request to a RESTful API using Python

5 Answers 5

2 Comments

5 Comments

1 Comment

2 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

5 Comments

1 Comment

2 Comments

Comments

Linked

Related