Python Requests Module Guide

We live in an interconnected world, applications are constantly exposing API’s to be consumed by others. Many times these are HTTP based such as Rest APIs. How can you consume HTTP based APIs? Python makes calling HTTP based APIs a breeze through the requests module. It’s motto is “HTTP for Humans”, and it makes consuming Rest APIs simple. In this article you’ll see how to use the python requests module to run various HTTP commands.

This guide is based on Python 3. 

Install Requests Module

As is the case with python, installing is made easy with pip. Run the following command to install the requests module.

pip install requests

You can also run the following command.

python -m pip install requests

Requests Get Function

The two most common HTTP methods are a GET and POST. GET is used to request data from a source. When you browse insightsbot.com for example, you are issuing a GET command.

Let’s run python requests get function to issue a GET.

import requests

r = requests.get('http://www.insightsbot.com')  
print(r.content)  

Straightforward right? Requests makes executing a GET very simple. The variable r is the response object which contains a server’s response to an HTTP request. It contains important properties such as content which contains the response in bytes. We print the content of the requests response and we see the Html that is returned by the server. 

In our code above, our variable r holds other important properties. Run the following command to see all that is available.

print(r.__dict__)

You’ll notice the reponse header also contains interesting information such as the content type, the server that is running the site and others.

print(r.headers)

Python Requests

Let's print a few of these: 

Done, let's continue to learn how to issue a POST.

Requests Post Function

HTTP Post is used when you want to submit data to the server for it to be stored or make a change. This is the second most commonly used HTTP verb.

In order to test this, we need a site we can submit a post method to. Luckily httpbin.org exists. It is a service to test HTTP requests and responses. Let’s now issue a post with python requests.

import requests

Data = {'key1': 'value1', 'key2': 'value2'}  
r = requests.post("http://httpbin.org/post", data=Data)   
print(r.text)  

As you can see above, we use the post function from the requests module. Data is submitted through the data parameter. In this case we are sending a dictionary of strings. The requests module encodes these into a form as you can see below.Python Requests Post

Python Requests Basic Authentication

In HTTP transactions, basic authentication is a validation method in which a client provides a user name and password when making a request. This is then validated by the service and if found valid, the service responds, otherwise a validation error occurs.

Using python requests for basic auth is very similar to issuing a regular get command. Only difference is you add an auth parameter with the username and password such as below.

import requests

r = requests.get('https://yoursite.com/resource', auth=('USER_NAME', 'PASSWORD'))
print(r.text)

Python Requests Download File

To download files with python requests, you will be issuing a python requests. If you recall, the content attribute contains the results in bytes. With this, all that is left is to write bytes to file with python.

First let’s issue a GET on the file you want.

import requests

url_of_file = "https://www.python.org/static/opengraph-icon-200x200.png"
r = requests.get(url_of_file)

With the requests object, now write the bytes to a file. We will call our downloaded file PythonImage.png and use the open() function to save the requests content bytes to a file.

fileName= "PythonImage.png"
with open(fileName, 'wb') as f:
    f.write(r.content)

Conclusion

You how now learned how to use the pythons requests module. This module is definitely made for humans as it makes calling http methods a breeze. You learned how to use the requests get and requests post functions and different variations such as basic authentication and using the module requests to download files from the internet.

MJ

Advanced analytics professional currently practicing in the healthcare sector. Passionate about Machine Learning, Operations Research and Programming. Enjoys the outdoors and extreme sports.

Related Articles

>