In order to reach the scope of this article which is learning how to use cURL in Python, we first need to introduce cURL. Client URL(cURL) is, for short, an easy-to-use command line designed for developers to fetch data from a server.

As I already mentioned above, using cURL is pretty straightforward and can extract information with just a one-line command. Firstly you need to open an terminal and type curl followed by an website link, for example: 

Congratulations, you made your first request using cURL. This simple command requests information from the server just like a traditional browser does and returns the HTML of the page. Not every website will give you back HTML, there are endpoints which send back data as a JSON object. Take this example:

$ curl 'https://jsonplaceholder.typicode.com/todos/1'

Type this command in your terminal and you should get back this response:

{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}

Most API’s will give you back either HTML or JSON when you are running cURL commands against them. Well this is not everything cURL can do for us. In reality it is a very sophisticated tool. If you want to learn more about cURL, I strongly recommend you to take a look at cURL documentation for a better understanding of its parameters. Alternatively you can run the following command: 

This will show you some options you can set to cURL: 

Usage: curl [options...] <url>
 -d, --data <data>   HTTP POST data
 -f, --fail          Fail silently (no output at all) on HTTP errors
 -h, --help <category>  Get help for commands
 -i, --include       Include protocol response headers in the output
 -o, --output <file>  Write to file instead of stdout
 -O, --remote-name   Write output to a file named as the remote file
 -s, --silent        Silent mode
 -T, --upload-file <file>  Transfer local FILE to destination
 -u, --user <user:password>  Server user and password
 -A, --user-agent <name>  Send User-Agent <name> to server
 -v, --verbose       Make the operation more talkative
 -V, --version       Show version number and quit

This is not the full help, this menu is stripped into categories.
Use "--help category" to get an overview of all categories.
For all options use the manual or "--help all".

As you can probably see these are not even all the options you can set to cURL, it is a menu divided into categories. You probably guessed that in order to get all the options you would like to run:

For this step,there are 2 prerequisites. First one is pretty obvious, you should install Python on your machine. You can do that by navigating to Python’s official website and install the proper version for your operating system. Make sure to be a new version as older ones will likely not include pip, which is required for most packages we will want to use. Run the following command afterwards:

Upon successful installation, this should give you back the version of pip you have installed. 

Otherwise, you will more likely get this message:

"pip" is not considered to be an external or internal command. A batch file is a program to operate.

It is very important for you to have pip installed in order to proceed as you will need it to install packages.

The second prerequisite is that you should be familiar with Python’s syntax or at least a beginner's level of experience with any other programming language.

You are probably wondering: is it not enough just to use cURL from the command line? We are running just one line of command and the API gives us back the information.This is right, but in the real world we will want to process somehow data we are receiving back from the server and this is why we need a programming language.That is the moment where Python comes into play.

Python is a high level programming language used for many purposes.It’s simple syntax and simplicity makes it very easy for beginners to pick it up. On top of that it has a huge community ready to help you, so in case you encounter any issues you should not hesitate to ask a question. One great place where you can jump in and ask a question is StackOverflow and someone will surely reach out to you. 

Just like before cURL in Python is also very straightforward. Remember when we wrote one line command to get the data from the server. The difference is that now you need to write 2 lines of code for this simple call, for example:

import os
os.system(f'curl "https://www.webscrapingapi.com/product/"')

Now lets see the real advantage of using a programming language. We can build a function that can take a custom URL and run the curl command against it: 

import os

def cURL(url):
   return os.system(f'curl "{url}"')

cURL('https://www.webscrapingapi.com/blog/')

You can replace https://www.webscrapingapi.com/blog/ with any other website you want to fetch data from. Congratulations, at this point you created a script which takes an url , runs the cURL command and displays the result in the console. You can run Python directly in your terminal, but for a better programming experience I strongly recommend you to use an Integrated Development Environment. There are many choices you can make, but for Python I recommend PyCharm which you can download from here

As I already mentioned above cURL is nowhere limited to that.It can do a huge amount of things apart from sending GET requests. It can also download files or send POST,PUT or DELETE requests.Here it is an example of a Python function which sends a POST request to https://httpbin.org/post: 

import os

def cURL(method,url,data):
   return os.system(f'curl -X "{method}" --url "{url}" --data {data} ')

data = '{"foo":"bar"}'

cURL('POST', 'https://httpbin.org/post', data)

As you probably saw the command gets a little bit different when sending a POST request. In the previous request you did not have to use -X and –data parameters because cURL default method is GET. After running this command you should get back from the httpbin API a response containing your request, IP address, params and body you have submitted, which would be something like that.

At this point you have already built a simple tool which can send GET, POST, PUT, PATCH, DELETE etc.. requests to fetch some data from an API. We can now just set the URL, method and body directly from our Python script instead of typing it manually. However, this is a minor benefit we have by using cURL in Python. The major benefit is that we can now process the data in the way we want , even if the API does not give us an option to get it in the way we would like. For instance, let’s say that we are given a list of all the users, but we want to split them into 2 groups and get the users from the second group.

We can do that thanks to Python by storing the response in a variable called users which we can convert in an array of JSON objects by using json.loads() method for which you can find more information about here. After that we can iterate through the users array and display only the users in the 2nd half of the list or with their id numbers greater than middle of the list. For a better understanding, this is how it translates in code: 

import subprocess
import json

def cURL(url):
   return subprocess.check_output(['curl',url])

users = json.loads(cURL('https://jsonplaceholder.typicode.com/users'))

for user in users:
   if(user['id'] > len(users) / 2): print(user)

You probably noticed that instead of os, we are now using subprocess and json libraries. We are using the subprocess library because we want to be able to save the output of the command , unlike the os module which just runs the command without storing it. The code is a simple get request to an API which gives us back a list of users under the form of JSON objects.. The output should be:

{'id': 6, 'name': 'Mrs. Dennis Schulist', 'username': 'Leopoldo_Corkery', 'email': 'Karley_Dach@jasper.info', 'address': {'street': 'Norberto Crossing', 'suite': 'Apt. 950', 'city': 'South Christy', 'zipcode': '23505-1337', 'geo': {'lat': '-71.4197', 'lng': '71.7478'}}, 'phone': '1-477-935-8478 x6430', 'website': 'ola.org', 'company': {'name': 'Considine-Lockman', 'catchPhrase': 'Synchronised bottom-line interface', 'bs': 'e-enable innovative applications'}}

{'id': 7, 'name': 'Kurtis Weissnat', 'username': 'Elwyn.Skiles', 'email': 'Telly.Hoeger@billy.biz', 'address': {'street': 'Rex Trail', 'suite': 'Suite 280', 'city': 'Howemouth', 'zipcode': '58804-1099', 'geo': {'lat': '24.8918', 'lng': '21.8984'}}, 'phone': '210.067.6132', 'website': 'elvis.io', 'company': {'name': 'Johns Group', 'catchPhrase': 'Configurable multimedia task-force', 'bs': 'generate enterprise e-tailers'}}

{'id': 8, 'name': 'Nicholas Runolfsdottir V', 'username': 'Maxime_Nienow', 'email': 'Sherwood@rosamond.me', 'address': {'street': 'Ellsworth Summit', 'suite': 'Suite 729', 'city': 'Aliyaview', 'zipcode': '45169', 'geo': {'lat': '-14.3990', 'lng': '-120.7677'}}, 'phone': '586.493.6943 x140', 'website': 'jacynthe.com', 'company': {'name': 'Abernathy Group', 'catchPhrase': 'Implemented secondary concept', 'bs': 'e-enable extensible e-tailers'}}

{'id': 9, 'name': 'Glenna Reichert', 'username': 'Delphine', 'email': 'Chaim_McDermott@dana.io', 'address': {'street': 'Dayna Park', 'suite': 'Suite 449', 'city': 'Bartholomebury', 'zipcode': '76495-3109', 'geo': {'lat': '24.6463', 'lng': '-168.8889'}}, 'phone': '(775)976-6794 x41206', 'website': 'conrad.com', 'company': {'name': 'Yost and Sons', 'catchPhrase': 'Switchable contextually-based project', 'bs': 'aggregate real-time technologies'}}

{'id': 10, 'name': 'Clementina DuBuque', 'username': 'Moriah.Stanton', 'email': 'Rey.Padberg@karina.biz', 'address': {'street': 'Kattie Turnpike', 'suite': 'Suite 198', 'city': 'Lebsackbury', 'zipcode': '31428-2261', 'geo': {'lat': '-38.2386', 'lng': '57.2232'}}, 'phone': '024-648-3804', 'website': 'ambrose.net', 'company': {'name': 'Hoeger LLC', 'catchPhrase': 'Centralized empowering task-force', 'bs': 'target end-to-end models'}}

This is only one of the uses of cURL in Python. We do not depend on the API anymore to send data back in the way we want, we can process it in many different ways such as displaying whether a person is in a response list based on their name , email , phone number or any other unique property of a person. 

For example let us say that we would like to check whether or not a name with a specific name exists into a list. For that we can create a function which can take the response which we converted into a JSON array ready for Python to parse.

import subprocess
import json

def cURL(url):
   return subprocess.check_output(['curl',url])

users = json.loads(cURL('https://jsonplaceholder.typicode.com/users'))

def check_if_user_exists(users,name):
   for user in users:
       if(user['name'] == name): print(f'An user called {name} exists in the list and has the id of {user["id"]}')

check_if_user_exists(users,'Clementina DuBuque')

This code block will then give us back this output: 

An user called Clementina DuBuque exists in the list and has the id of 10

Congratulations. You made a script which fetch data from a server , store it and parse it afterwards. Advantages of using Python do not stop here, Python even has a special interface designed for using cURL which is called PycURL and we are going to discuss about it  now.

As we mentioned before, PycURL is, for short, a Python tool which helps us to use cURL more naturally. One great advantage is that PycURL is heavily optimized and supports concurrency, meaning that it is very fast (faster than the popular Python request library). On the other hand, using PycURL is not as easy to use like what we have seen before, being a tool targeted for an advanced developer.However you should not feel intimidated by that because in the end you will gain a deeper understanding of networking and you will become more comfortable with Python.

Just like any other package, you can install it with pip: 

You would also like to install certifi this time for security purposes when using pycurl. Certifi is a tool that validates SSL certificates while verifying the identity of TLS host.To learn more about certifi, I strongly suggest you to check their docs.  You can install it in the same way: 

You can test now successful installation by running the following script: 

The output should be the location where the package got installed:

/usr/local/lib/python3.10/site-packages/certifi/cacert.pem

The difference from what we did before is that now we are only writing code without running commands directly. For short, we create an instance of pycurl.Curl() class , we fetch the data and write it in a buffer which we are later going to decode in order to be able to read the data you receive: 

import pycurl
import certifi
from io import BytesIO

buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.URL, 'https://docs.webscrapingapi.com/')
c.setopt(c.WRITEDATA, buffer)
c.setopt(c.CAINFO, certifi.where())
c.perform()
c.close()

body = buffer.getvalue()
print(body.decode('iso-8859-1'))

You guessed it, just like our previous examples, this fetch the HTML content of WebScrapingAPI webpage and print it in the command line.

A POST request is not much different, except that you need to tell to your instance of pycurl.Curl() class that you are going to use a post method, set your body and headers, if it is the case.This is how it looks like: 

​​import pycurl
import certifi
import json
from io import BytesIO

buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.URL, 'https://httpbin.org/post')
c.setopt(pycurl.HTTPHEADER, [ 'Content-Type: application/json' , 'Accept: application/json'])
data = json.dumps({"foo": "bar"})

c.setopt(pycurl.POST, 1)
c.setopt(pycurl.POSTFIELDS, data)
c.setopt(c.WRITEDATA, buffer)
c.setopt(c.CAINFO, certifi.where())

c.perform()
c.close()

body = buffer.getvalue()

print(body.decode('iso-8859-1'))

And we should get back,  just like before a response response containing your request, IP address, params and body: 

As I already mentioned, there are just some basic things pycURL can do for us.It is a very complex and sophisticated tool which we can write many articles about. If you want to explore more things you can do with it, I strongly encourage you to check their docs.

In conclusion, using cURL in Python is very effective and saves a lot of time and on top of that can be a starting point for some interesting projects in topics such as data analysis or web scraping.The approach I recommend is first to get comfortable with cURL and Python and then move on and use pycURL. I hope you found this resource useful to learn how to use cURL in Python and you will play around with it and build some scripts.

Find out how to use cURL in Python

In order to reach the scope of this article which is learning how to use cURL with a Proxy, we first need to introduce cURL. Client URL(cURL) is, for short, an easy-to-use command line designed for developers to fetch some data from a server.

$ curl 'https://jsonplaceholder.typicode.com/todos/3'

{
  "userId": 1,
  "id": 3,
  "title": "fugiat veniam minus",
  "completed": false
}

Most API’s will give you back either HTML or JSON when you are running cURL commands against them. Well this is not everything cURL can do for us. In reality it is a very sophisticated tool which requires a lot of time to master. If you want to learn more about cURL, I strongly recommend you to take a look at cURL documentation for a better understanding of its parameters. Alternatively you can run the following command: 

However, using cURL alone has some limitations regarding the number of servers we can choose to fetch data from. For example, some servers can use geolocalization and refuse to give us the data we are looking for because of our location. This is the moment we need a proxy, which acts like a middleman between us and the target server.

The concept of a proxy server is not hard to understand at all. As already mentioned above , proxy server is like an intermediary between a client requesting a resource and the server providing that resource.Proxies are designated for us to be able to get data from anywhere. In order to understand better this concept , lets assume that we have a server called Bob which has some data we are interested in, but Bob provide that data only if we are in Europe, but we are in United States.

How we deal with that? We send our request to a proxy server which is located in Europe and not to Bob and tell the proxy that we want to get some data from Bob. The proxy will send itself the request to Bob and Bob will return to the proxy server data since the proxy is in Europe. Then the proxy server will send us back the data from Bob.

This is the main flow of how proxies work. Another great use case for a proxy is for example when we want to get data which contains prices in a specific currency in order to avoid confusion. For a further understanding of proxies, I strongly recommend you to have a look on Wikipedia.

In order to use a proxy , you will most likely need a host , a port , an user , a password and a target URL you want to get data from. For this example , I will use a proxy provided by WebScrapingAPI for making requests for which you can find more information about it here. WebScrapingAPI is not a proxy provider, it is a web scraping service which provides proxies instead .In our examples, our setup will be the following:

Proxy username: webscrapingapi.proxy_type=datacenter.device=desktop

Proxy password: <YOUR-API-KEY-HERE> // you can get one by registering here

Please note that there may be some proxy providers which require other schema of authentication.

Since we have covered cURL and proxies , now we are ready to combine them together and make requests by using a proxy which is a pretty straightforward process.We first need to authenticate and then we can use the proxy.

Proxy authentication in cURL is pretty simple and can be done for our example from above as following :

$ curl -U webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR-API-KEY> --proxy @proxy.webscrapingapi.com:80 http://httpbin.org/get

Running that command , httpbin will give us back our IP address , and some other properties:

​​{
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.5173.0 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-633af93b-35faf8637f37f7f075f185ec"
  }, 
  "origin": "45.9.120.69", 
  "url": "http://httpbin.org/get"
}

As you can already probably see, the origin you receive back is not your IP address and it is the address of the proxy server instead. Furthermore, you can run the command even without revealing your password in the terminal. This can be done as following: 

$ curl -U webscrapingapi.proxy_type=datacenter.device=desktop --proxy @proxy.webscrapingapi.com:80 http://httpbin.org/get

And then you will get a prompt to enter your password: 

Enter proxy password for user 'webscrapingapi.proxy_type=datacenter.device=desktop':

Now you can type your API key there without exposing it in the terminal which makes the whole process more secure.Furthermore, typing your credentials , host and port every single time you want to run a cURL command via a proxy may not feel that ideal , especially when you want to run many commands via a proxy and you are using the same proxy provider. 

Of course, you can store your credentials on a separate file stored on your machine and copy paste them everytime, but there is a more natural approach you can take which is passing them via environment variables which we will talk about below.

Using cURL with a proxy via environment variables 

An environment variable is like an object which stores an editable value in the memory which can be used by one or more softwares. In this particular case we can pass to cURL a variable called http_proxy or https_proxy which contains our proxy details and we will not need to specify on every run of the command. You can do that by running this command:

$ export http_proxy="http://webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR-API-KEY>@proxy.webscrapingapi.com:80"

Please note that you have to call your variable http_proxy or https_proxy in order for cURL to understand what you are talking about. That is it, now you do not need to pass your credentials on every run anymore and now you can just run cURL as simple as this: 

As you probably see the IP address is the proxy’s address which confirms that you have made a great job setting your proxy. At this point we can run any cURL command without specifying the proxy details, cURL will take care of that for us.

Disabling the proxy for a specific command

However, if you need to send a specific request without a proxy, you do not need to worry about deleting the value of http_proxy variable. Being a sophisticated tool with a lot of options , cURL can take care of that for us by its noproxy parameter which tells it to not use any proxy when making the request. It can be done as following:

$ curl --noproxy "*" http://httpbin.org/get

And that will give us back our own IP address and not the proxy’s one.

In conclusion, using cURL with a proxy is a great way to bypass geolocation filters, extends the amount of resources we can fetch from webservers and it is a good starting point for getting into topics such as web-scraping where we need to use proxies in order to be able to get certain data or to receive it in the format we want.I hope you found this article useful for you to learn how to use cURL with a proxy and you will play around with it and build your own scripts which extracts data from servers which use geolocation filters.

Andrei Ogiolan