Learn to Use Proxy with Python Requests

In this guide, we are going to learn about proxy, why we need a proxy, and how to use proxy with the python requests module.

Table Of Contents

Introduction to Proxy
How to Use Proxy with Python Requests
Visualize the Difference between Proxy and Without Proxy
Conclusion

Introduction to Proxy

A proxy is a web server that allows you to access the internet by hiding your real IP address. Means you can access the internet through a proxy server and your real IP address will be hidden from the website you are visiting.

This is very useful when you are using a public network like a library or a cyber cafe. You can use a proxy server to hide your real IP address and access the internet.

You may have seen in some locations or country, some websites are blocked. You can use a proxy server to access those websites. Because the proxy server will hide your real IP address and request the website with its own IP address.

The proxy server behaves like a middleman between you and the website you are visiting. They request the website on your behalf and send the response back to you.

How to Use Proxy with Python requests module

The requests module in Python is a very powerful module. It allows you to handle HTTP requests in a very easy way.

Python requests module is very flexible and provides a way to use proxy with it.

Let's see how to use proxy with the python requests module.

Find a Proxy Server

First of all, you need to find a proxy server to use. You can find a free proxy server on many websites. Some of them are:

These websites provide a list of free proxy servers. You can use any of them but some of them may not work. So, you need to try multiple proxy servers to find a working one.

If your need goes around web scraping, then you can use proxy API.

Using Proxy with Python requests module

Now, we are going to see how to use proxy with the python requests module.

Python requests module provides a way to use proxy with it. You can use a proxy with the python requests module by passing a dictionary to the proxies parameter of any of the requests methods.

Let's see how to do this.

import requests

proxies = {
  "https": "8.219.97.248:80"
}

r = requests.get("https://www.ipinfo.io/json", proxies=proxies)

print(r.text)

Note: If this proxy server is not working, you can use any other proxy server from the list we provided above.

Now, let's see what is happening in the above code.

We are importing the requests module. Then we are creating a dictionary and store the proxy server in it. The key of the dictionary is https and the value is the proxy server.

Then we are making a GET request to https://www.ipinfo.io/json and pass the proxies parameter to it. The value of the proxies parameter is the dictionary we created above.

Proxy Authentication

Some proxy servers require authentication. You need to provide a username and password to use them.

You can use auth parameter of the requests method to provide authentication to the proxy server as a key-value pair.

Let's see how to do this.

import requests

proxies = {
  "https": "8.219.97.248:80"
}
auth = {
  "username": "username",
  "password": "password"
}

r = requests.get("https://www.example.com", proxies=proxies, auth=auth)

print(r.text)

Visualize the Difference between Proxy and Without Proxy

Now, let's see the difference between using a proxy and without a proxy.

Let's make a GET request to https://www.ipinfo.io/json without using a proxy.

import requests

url = "https://www.ipinfo.io/json"
r = requests.get(url)

print(r.text)

Output:

In the above output, you can that the current IP address and another detail are returned by the website.

Now, let's make a GET request to https://www.ipinfo.io/json using a proxy. The proxy server we are using is from Japan.

import requests

proxies = {
    'https': '133.18.173.81:8080'
}
url = "https://www.ipinfo.io/json"
r = requests.get(url, proxies=proxies)

print(r.text)

Output:

After using a proxy you can see that the website thinks that the request is coming from Japan.

So, this is how you can use proxy with the python requests module.

Conclusion

In concluding this brief guide, we have used the python requests module to make requests and learned how to use proxy with python.

Now, you can use a proxy with the python requests module to make requests.