APIs and the Python Requests library

The Requests library for Python is one of the most downloaded Python packages of all time and probably one of my favourite libraries. It's one of those packages that abstracts so much away from what's actually happening underneath that you'll almost be fooled into thinking you're the one doing all the hard work.

You can use Requests to send human-readable HTTP/1.1 requests without the need for loads of configuration beforehand. Here's an example of just how simple it is to use:

1
2
3
4
5
6

import requests

r = requests.get('https://api.github.com', auth=('user', 'pass'))

print r.status_code  # 200
print r.headers['content-type']  # 'application/json'

Requests makes querying and interacting with APIs much simpler than previous methods. The Requests repository has an example of how the same request could have been made using urllib2:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import urllib2

gh_url = 'https://api.github.com'

req = urllib2.Request(gh_url)

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')

auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)

urllib2.install_opener(opener)

handler = urllib2.urlopen(req)

print handler.getcode()  # 200
print handler.headers.getheader('content-type')  # 'application/json'

I think I know which I'd prefer to have to use everyday (which isn't too far from the truth!). Let's try using it ourselves and see how we get on.

Types of HTTP methods

Before we move on to making our own requests, lets quickly make sure we're all on the same page. There are four common HTTP methods that you are likely to use on a regular basis. These are:

• GET - retrieves information from the chosen source.
• POST - sends new information to the chosen source.
• PUT - updates existing information of the chosen source.
• DELETE - removes existing information from the chosen source.

Most of the time you will probably find yourself using GET requests because you'll just be asking for information from the database of a specific source, like the titles of every film released in 2017 from a well-known internet movie database.

You are more likely to use PUSH, PUT and DELETE requests for APIs where you own an account, such as Twitter or Github. These APIs will allow you to send requests to add or delete tweets/repositories using your account credentials.

If you're still unsure about what all this translates to in real-world programming, don't worry, I was clueless too at first (and no doubt so was everyone at some point!). Luckily the Requests library makes all of this really easy, so by the end of this tutorial you should have a solid understanding of how you can using HTTP requests in your own projects.

The structure of a HTTP request

As well as the four HTTP methods, it's important to understand the anatomy of the HTTP request itself. There are three key parts of a HTTP request:

1. The request line - this part contains the request method (i.e. GET, POST, etc) and the URL of the resource the request is searching for.
2. The header - this part sends additional information to the chosen source, such as the type of encoding (e.g. UTF-8) and the desired content type (e.g. 'application/json').
3. The body - this part is optional, but when using the POST or PUT methods the data required for the update is contained here.

Now let's tie all this information together to make our own request.

Making our first HTTP request

If you don't already have the Requests library installed, you can install it from the console with pip (if you want to use easy_install or tarbell instead go right ahead):

pip install requests

Now type python into the console to open the Python shell, and then import the Requests library:

>>> import requests

We are going to start playing with the Requests package by using the Star Wars API (affectionately known as 'swapi') to send some simple GET requests. Swapi is a huge collection of data from the Star Wars universe organised in such a way that all the information can be accessed through an API.

How do you work out what the API call should be? The API documentation. Every API will have it's own unique URL structure and parameters that it expects in a HTTP request in order for you to access it's data, and the best way to check this is the documentation. The documentation for Swapi can be found here.

Using the Swapi documentation as a guide, let's make our first request:

>>> r = requests.get('http://swapi.co/api/planets/1/')

The first thing to confirm is whether the request was successful, which you can do by checking the status_code of the response:

>>> r.status_code 200

A status code of 200 is the standard response of a successful HTTP request. If you get a code other than 200 and you aren't getting the response from the API you were expecting, it's worth finding out what the status code means to help you figure out what you need to do to fix your request. A helpful list of HTTP status codes and their meanings can be found in the same place as everything else nowadays—Wikipedia.

Examining the HTTP response

Headers

Our HTTP request was successful, so we can now look at what the API sent back to us. You can view the response headers using .headers:

>>> r.headers {'Via': '1.1 vegur', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Set-Cookie': '__cfduid=d92acebf1060281dbc7a56a74b79e0f101494793395; expires=Mon, 14-May-18 20:23:15 GMT; path=/; domain=.swapi.co; HttpOnly', 'Vary': 'Accept, Cookie', 'Server': 'cloudflare-nginx', 'Connection': 'keep-alive', 'Etag': 'W/"167b2ef894493c72d59ba5a45bab88cf"', 'Allow': 'GET, HEAD, OPTIONS', 'Date': 'Sun, 14 May 2017 20:23:15 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Type': 'application/json', 'CF-RAY': '35f09301b6326a37-LHR'}

This gives us useful information like the content-type and the allowed request methods, which in this case are application/json and GET, HEAD, OPTIONS, respectively.

Encoding

You can view the encoding of the response with .encoding, however, Swapi doesn't use encoding for its responses, so to test this we are going to send a GET request to httpbin.org instead:

>>> r2 = requests.get('http://httpbin.org/') >>> r2.encoding 'utf8'

By default the Request library will guess the encoding type of the response, which will most often be utf8 and decode it accordingly, but you can change the encoding type yourself like the following:

>>> r2.encoding = 'ISO 8859-1'

NOTE - ISO 8859-1 is used for encoding characters common in Western Europe, such as é and ü. Most sites will be using UTF8 as standard.

Contents

Now for the juicy bit of the response, the content. You can check the contents of the response using .text, or if the response is in the format of JSON, the .json() method:

>>> r.text u'{"name": "Tatooine", "rotation_period": "23", "orbital_period": "304", "diameter": "10465", "climate": "arid", "gravity": "1 standard", "terrain": "desert"...[remainder of response truncated]}' >>> r.json() {u'diameter': u'10465', u'climate': u'arid', u'surface_water': u'1', u'name': u'Tatooine', u'created': u'2014-12-09T13:50:49.641000Z'...[remainder of response truncated]}

The key (and possibly obvious) difference between the above is that .text will return the contents as a unicode string, whilst the .json() method will return a JSON object. This means with the second option, we can use Python dictionary notation to obtain specific content in the request:

>>> data = r.json() >>> data['name'] u'Tatooine'

This is useful if you are only interested in writing code that extracts certain information held by the API.

Making POST requests

If you're particularly sharp-eyed you might have noticed above in .headers that the only methods allowed by the Swapi API are GET, HEAD, OPTIONS. This isn't a mistake, if you read their documentation they explain why:

Swapi is a completely open API. No authentication is required to query and get data. This also means that we've limited what you can do to just GET-ing the data.

So for us to have a go at testing POST requests we need to use a different API. I personally like to use RequestBin. RequestBin allows you to send API requests and examine the response to check you have structured your requests properly.

Let's start by going to https://requestb.in/ and pressing the big green button labelled Create a RequestBin. The page will refresh to look something like the one below.

Copy the URL in the Bin URL field (mine looked something like https://requestb.in/15qihl81).

Now go back to your Python terminal and type the following:

>>> r = requests.post('https://requestb.in/15qihl81', data={'name': 'nemo'})

Here we've used the Requests library to send a POST request using the .post method, and as well as providing the URL we want to send the request to we have also included some data we want the request to include.

Just as we did for the GET request, let's test the status code of the response:

>>> r.status_code 200

And if we check the content of the response, we should see something like this:

>>> r.text u'ok'

Perfect. Now, what I like about RequestBin is that you can check the API received the POST request with the information you sent it really easily. If you refresh the page that you copied the URL from, you should see the page now displays information about the POST request we just sent. You can review that the data we sent was received correctly under the FORM/POST PARAMETERS section. If you see the key/value pair 'name: nemo', then you know everything has worked as expected.

POST requests in practice

So, now you've seen how you can make a POST request using the Requests library, how might you use this in practice to achieve something useful? Well, if you have a GitHub account you can use it to create a new repository. To do this you will need to add user credentials to your .post method using the auth parameter. The code snippet below demonstrates how you can do this.

>>> import requests, json >>> data = json.dumps({'name':'test', 'description':'some test repo'}) >>> r = requests.post('https://api.github.com/user/repos', data=data, auth=('user', '*')) print r.json

Or you could use it to send a Tweet using the Twitter API:

>>> import requests, json >>> data = json.dumps({'status': '#covfefe'}) >>> r = requests.post('https://api.twitter.com/1.1/statuses/update', data=data, auth=('user', '*'))

Summary

The possibilities are endless. Wherever a website uses an API you can interact with it easily using the Requests library. Hopefully this has given you a bit of an insight into how awesome the Requests library is and how it can help you make and send HTTP requests.