If curiosity pushes you to learn about APIs, this is the right article for you. Discover definitions, API types, documentation, and more.

The Beginner’s Guide to APIs: Everything You Need to Know

Just as humans communicate with each other by word of mouth, letters, or gestures, other living creatures have their form of communication as well. But did you ever ask yourself how an app does that?

Applications talk to each other using an intermediary software, for example, an Application Programming Interface or API for short. Each time you listen to music via Spotify or binge-watch something on Netflix, you are indirectly using an API.

This APIs for Dummies guide will present just how big an impact APIs have in our lives, even though most know absolutely nothing about them. Let’s dive in!

An API (Application Programming Interface) is a set of functions that allows applications to access data and interact with external software components, operating systems, or microservices. To simplify, an API delivers a user request to a system and sends the system’s response back to a user.

APIs are highly used because they can make things way easier and speed up the development process of other systems and applications. For example, if you are building your own software and wish to integrate face recognition or payment processing, it is best to use an API. You just have to connect your software with another one. These are provided with the API’s documentation, a manual for developers to help implement the desired functionalities.

Using an API, you don’t need to deal with any source code or even understand how it works. Sounds good, right? Cool, let’s take a look at just how APIs do that.

Think of an API as the “middleman” between your application and the server. You tell your app to do something for you; then the app will use an API to communicate with the web server, telling it what to do. This action is called a request. Afterward, the server will send a response to be delivered by the “middleman” to your app.

Let’s see a less abstract example. You walk into a restaurant, pick a table with a view and wait for the waiter to get your order. In this case, the waiter is your “middleman” because the cook cannot take your order directly, as he has other orders to prepare in the kitchen. You make a request to the waiter, the waiter sends said request to the kitchen, then gets a response, in this case, your order, and brings it to your table. Enjoy your food!

Even the API is an abstraction of the web server, as the application doesn’t know what is going on in the “kitchen,” just what the final result will be. The end-user knows that if he orders a cake from the menu, he will get a cake. There are many types of APIs, depending on what they’re built to do or their release policy. Let’s take a look at these differentiators.

APIs are handy. As we mentioned above, communication between various software and apps is essential in order to achieve a particular goal, so an API is designed based on what you want to accomplish. Let’s see a few examples:

Database APIs make communication between an application and a database management system easier. For example, Firebase is a mobile web application platform owned by Google that lets developers simply sync and store data in real-time without having to manage servers or write server-side code.


Operating systems APIs. Even operating systems have a fair number of APIs, for example, Windows API or the Linux Kernel User-Space API. You can also find the APIs provided by Apple for macOS and iOS and presented in their developer documentation.


Remote APIs interact through a communications network. The most common communications network is the Internet, so APIs are usually designed based on web standards. But that doesn’t mean that all remote APIs are web APIs. Still, Web APIs are generally remote, typically use HTTP for request messages, and define the structure of response messages. These messages are preferred to be in XML or JSON format as it is easier for other apps to parse the information.


The most commonly used APIs are Web APIs because they send requests from a web application and responses from servers using HTTP (Hypertext Transfer Protocol). Developers make use of such web APIs in their apps or sites, increasing their functionality and making them more practical for the user, like integrating a login method via Facebook or Google or adding a map using Google Map API.

Some APIs are built only for internal use. This is particularly common for large enterprises. Others are designed as products meant to integrate with whatever software customers use. These are the main categories:

Private APIs are mainly used inside one’s organization, integrating them in systems and apps used by their employees and improving the internal operation. Even if they appear open to the public, the interface remains available only for those within the organization, which has complete control over the API. It also reduces some costs for the organization.

Partner APIs can provide additional revenue streams as they are shared with one or more business partners. An agreement is made between said partner and the publisher. A partner API also adds value to the service and opens up a channel for up-selling.

Public APIs are available for any third-party developers. These types of APIs don’t need to have a close relationship between the user and the publisher, they can increase the revenue stream, help reach larger audiences and build traffic, and support many causes and a variety of business initiatives. Public APIs can be either:

Open: These APIs can be used by somebody however they please, which means that there are no restrictive terms or conditions, you don’t need the punisher’s consent to use the API and you don’t have to pay for it. They provide public documentation for developers to access and can be utilized to test and create applications.

Commercial: In order to use such APIs, you will have to pay a subscription fee. In order for users to test and review the product, publishers often offer a free trial so that the customer knows whether or not to purchase a subscription.

Of particular interest are SOAP APIs and REST APIs since these are very widespread. Let’s take a look at the concepts behind the acronyms:

SOAP or Simple Object Access Protocol APIs are web communication protocols and are used for exchanging information and structured data over HTTP/HTTPS. Unlike REST APIs, SOAP supports the XML data format only. Most web data exchange is realized over REST exchange, but that doesn’t mean that SOAP will have its downfall anytime soon, as it is highly standardized, more secure, and sometimes allows for automation.

XML or eXtensible Markup Language is a tool independent of software or hardware for storing and transporting data. It has a set of rules for structuring messages as both human and machine-readable records. As you may see below, its structure is similar to HTML, but the XML language has no predefined tags, so the author must define both tags and the document structure.

The Representational State Transfer is an architecture style created by Roy Fielding in 2000 for designing coupled applications over HTTP and is often used in API development. For a service to be referred to as RESTful, it mustn’t violate these constraints.

Developers prefer using REST as an alternative to SOAP because it doesn’t involve writing a lot of code to achieve simple tasksUnlike SOAP, which needs you to create the required XML structure every time. RESTful web service uses a different approach. It treats information as a resource represented by a unique URL. So if you want to request a certain resource, you will have to provide its URL using HTTP methods such as GET, POST, PUT, HEAD, DELETE, PATCH, and OPTIONS. You can find more about these methods here! Sometimes you might need to send additional data along with the URL in order to get what you need, like an access key.

Unlike SOAP, REST isn’t limited to using XML to provide the response. You can obtain the information in different formats like JSON, XML, CSV, plain text, and HTML, which are parsed easier based on the language you’re using for your application.

Here is a comparison between JSON and XML:

You’ll often hear developers talking about the functionalities of different APIs, but there’s another important, less discussed aspect: the documentation. Without the knowledge in it, an API might as well be a bunch of hieroglyphs.

An APIs documentation isn’t some videogame tutorial you can skip with nonchalance and learn its mechanics along the way. The documentation contains instructions on how to effectively use and integrate said API.

Even if an API is revolutionary for developers, poor documentation or lack of knowledge on how it works would make it useless.

Well-written documentation can help save support time and costs. Onboarding new users become easier and they won’t need technical support for using the API. A poorly written documentation can be rather frustrating for developers and the publishers’ support team.

Some documentation may provide a demo section or a playground where you can test things out and see live results. This helps developers visualize what the API does and have a better understanding.

I hope our APIs for Dummies helped you better understand how APIs work and just how important they are in the world of information technology.

The growing number of apps and software products, both corporate and for personal use, need communication and resource exchange methods. APIs satisfy that need and are key in helping the expansion of technology.

Check out this article on web scraping APIs to see a few examples of web APIs in action!

APIs for Dummies: Everything You Need to Know

We all hate the moments when we try to access a page and the site blocks our request for no good reason. Geoblocking, for example, can be passed only by using a proxy. 

Node-unblocker helps us to create a custom proxy and have it up and running in a matter of minutes.

Node unblocker is a general-purpose library to create a web proxy, intercept and alter requests and responses. 

This library is also used in web scraping for bypassing restrictions implemented by the site like geo-blocking, hiding the IP address and rate limiting, or for sending authentication tokens. 

To make it shorter, while using this library, you can kiss blocked and censored content goodbye.

In this article, we create an Express application with a custom proxy using Node Unblocker, we add a middleware that changes the user agent for each request, discuss the proxy limitations, deploy it to Heroku and compare it to a managed service like WebScrapingAPI.

Before we start, make sure you have the latest version of Node.JS installed. Installing Node.JS for each platform (Windows, Linux, Mac) would be the subject of a separate article so instead of going into details, check the official website and follow the instructions.

We start by creating a directory for our project named unblocked and we initialize a Node.JS project inside of it:

For this application, we install two libraries: Express, a minimalist framework for Node.JS, and Node Unblocker.

Because of node.unblocker running inside an Express instance, we have to set up the Hello World example in our application.

Create an index.js file and paste the following code:

const express = require('express') const app = express() const port = 8080  app.get('/', (req, res) => {     res.send('Hello World!') })  app.listen(process.env.PORT || 8080, () => {     console.log(`Example app listening on port ${port}`) })

We can run the application with this command:

If we access http://localhost:8080 we see a Hello World message. This means our application is up and running.

It’s time to import the Node Unblock library into our application:

We create a node unbloker instance and pass the proxy parameter. You can find the full list of available parameters here.

var unblocker = new Unblocker({prefix: '/proxy/'})

We register the node unblocker library to Express as a middleware so requests are intercepted:

We update the Express app listener to add support for websockets:

app.listen(process.env.PORT || 8080, () => {     console.log(`Example app listening on port ${port}`) }).on('upgrade', unblocker.onUpgrade)

After following all these steps, our application should look like this:

const express = require('express') const Unblocker = require('unblocker') const app = express() const port = 8080  const unblocker = new Unblocker({prefix: '/proxy/'})  app.use(unblocker)  app.get('/', (req, res) => {     res.send('Hello World!') })  app.listen(process.env.PORT || 8080, () => {     console.log(`Example app listening on port ${port}`) }).on('upgrade', unblocker.onUpgrade)

Restart the application and access the following URL in your browser:

http://localhost:8080/proxy/https://webscrapingapi.com

To make sure the proxy works as expected, we open the Developer Tools in the browser and check the Network tab. All requests should go through the proxy.

For any issues the proxy might throw, it’s recommended to enable the debug mode by setting the DEBUG environment variable. Use this command to start the proxy in debug mode:

It’s never a good idea to enable this in production so let’s keep it only for the development environment.

Nodeunblocker is not just a custom proxy solution but allows the interception and alteration of outgoing and incoming requests through middleware.

We can use this feature to block specific resources from loading based on the resource type or domain, update the user agent, replace returned content, or inject authentication tokens in request headers.

You can find a full list of examples here.

Let’s start by creating a middleware for setting a custom user agent. Create a file called user-agent.js and add this code to it:

module.exports = function(userAgent) {
    function setUserAgent(data) {
        data["headers"]["user-agent"] = userAgent
    }
    return setUserAgent
}

This function accepts the custom user agent with the userAgent parameter and registers it in the data object using the setUserAgent function. Node unblocked will call the setUserAgent function with each request.

const userAgent = require('./user-agent')

We set the requestMiddleware parameter in the Unblocker constructor and we should be good to go.

const unblocker = new Unblocker({
    prefix: '/proxy/',
    requestMiddleware: [userAgent("nodeunblocker/1.5")]
})

const express = require('express')
const Unblocker = require('unblocker')
const userAgent = require('./user-agent')
const app = express()
const port = 8080

const unblocker = new Unblocker({
    prefix: '/proxy/',
    requestMiddleware: [userAgent("nodeunblocker/1.5")]
})

app.use(unblocker)

app.get('/', (req, res) => {
    res.send('Hello World!')
})

app.listen(process.env.PORT || 8080, () => {
    console.log(`Example app listening on port ${port}`)
}).on('upgrade', unblocker.onUpgrade)

It’s time to check if our code works. We have to change the node-unblocker URL to make sure the headers are properly updated. 

Restart the application and open this URL in your browser:

http://localhost:8080/proxy/https://www.whatsmyua.info/

If the site displays nodeunblocker/1.5, our middleware works.

With our proxy running, it’s time to deploy it to Heroku, a platform as a service (PaaS) that allows us to create, launch, and manage apps fully in the cloud.

Keep in mind that not all providers allow proxies and web scraping apps on their infrastructure. Heroku accepts these kinds of apps as long as the rules from robots.txt are not ignored.

With the legal part discussed, let’s prepare our project for deployment.

We have to set the start script and the engines in the package.json file.

The `engines` property tells Heroku we require the latest version of Node.JS 16 installed in our environment. The start script is executed when the environment is set and our application is ready to run.

unblocked

 Heroku made the deployment of a Node.JS application a bliss. Before moving to the next section, make sure you have installed the Heroku CLI and the git tools.

Use this command to authenticate with Heroku from your local terminal:

Create a new Heroku application by running this command:

This command will return the app’s ID and a git repository. Let’s use the ID to set the remote origin for our repository:

git init
heroku git:remote -a [YOUR_APP_ID]

Because versioning the node_modules folder is never a good idea, let’s create a .gitignore file and add the folder to it.

The last step before our code will reach production is to commit it and deploy it. Let’s add all the files, create a commit and merge the master branch in the heroku one.

git add .
git commit -am "Initial commit"
git push heroku master

After a few seconds, the application will be deployed to Heroku. Congratulations! It’s time to access it in our browser and make sure it works.

Use the following URL structure to build the Heroku URL:

[HEROKU_DYNO_URL]/proxy/https://webscrapingapi.com

If you forgot or lost the Dyno URL, you can use this command to get the available information about the current app:

The ease of setup for this custom proxy comes with a caveat: it works well only for simple sites and it fails for advanced tasks. Some of these limitations are impossible to overcome and will require the use of a different library or third-party services.

A managed service like WebScrapingAPI implements fixes for all these limitations and adds a few extra features like automatic captcha solving, residential proxies, and advanced evasions to prevent services like Akamai, Cloudflare, and Datadome to detect your request. 

Node Unblocker for Web Scraping

Rather than using commercial tools, many developers prefer to create their own web scrapers. While available products have more fleshed out features, we can’t deny the results these bots can bring or the fun of making your own.

In the following article, you’ll find out the steps you have to take to build your own web scraper using Node.js and Puppeteer. We’ll code an app that loads a website, snaps a screenshot, log in to the website using a headless browser and scrape some data across multiple pages. Your app will grow in complexity as you progress.

An overview of Web Scraping with Puppeteer

Google designed Puppeteer to provide a simple yet powerful interface in Node.js for automating tests and various tasks using the Chromium browser engine. It runs headless by default, but it can be configured to run full Chrome or Chromium.

The API build by the Puppeteer team uses the DevTools Protocol to take control of a web browser, like Chrome, and perform different tasks, like:

Snap screenshots and generate PDFs of pages

UI testing (clicking buttons, keyboard input, etc.)

Scrape a SPA and generate pre-rendered content (Server-Side Rendering)

Most actions that you can do manually in the browser can also be done using Puppeteer. Furthermore, they can be automated so you can save more time and focus on other matters.

Puppeteer was also built to be developer-friendly. People familiar with other popular testing frameworks, such as Mocha, will feel right at home with Puppeteer and find an active community offering support for Puppeteer. This led to a massive growth in popularity amongst the developers.

Of course, Puppeteer isn’t suitable only for testing. After all, if it can do anything a standard browser can do, then it can be extremely useful for web scrapers. Namely, it can help with executing javascript code so that the scraper can reach the page’s HTML and imitating normal user behavior by scrolling through the page or clicking on random sections.

These much-needed functionalities make headless browsers a core component for any commercial data extraction tool and all but the most simple homemade web scrapers.

First and foremost, make sure you have up-to-date versions of Node.js and Puppeteer installed on your machine. If that isn’t the case, you can follow the steps below to install all prerequisites.

You can download and install Node.js from here. Node’s default package manager npm comes preinstalled with Node.js.

To install the Puppeteer library, you can run the following command in your project root directory:

npm install puppeteer
# or "yarn add puppeteer"

Note that when you install Puppeteer, it also downloads the latest version of Chromium that is guaranteed to work with the API.

There are many different things you can do with the library. Since our main focus is web scraping, we’ll talk about the use cases that are the most likely to interest you if you want to extract web data.

Let’s start with a basic example. We’ll write a script that will snap a screenshot of a website of our choice. 

Keep in mind that Puppeteer is a promise-based library (it performs asynchronous calls to the headless Chrome instance under the hood). So let’s keep the code clean by using async/await.

First, create a new file called index.js in your project root directory.

Inside that file, we need to define an asynchronous function and wrap it around all the Puppeteer code.

const puppeteer = require('puppeteer')

async function snapScreenshot() {
	try {
		const URL = 'https://old.reddit.com/'
		const browser = await puppeteer.launch()
		const page = await browser.newPage()

		await page.goto(URL)
		await page.screenshot({ path: 'screenshot.png' })

		await browser.close()
	} catch (error) {
		console.error(error)
	}
}

snapScreenshot()

First, an instance of the browser is started using the puppeteer.launch() command. Then, we create a new page using the browser instance. For navigating to the desired website, we can use the goto() method, passing the URL as a parameter. To snap a screenshot, we’ll use the screenshot() method. We also need to pass the location where the image will be saved.

Note that Puppeteer sets an initial page size to 800×600px, which defines the screenshot size. You can customize the page size using the setViewport() method.

Don’t forget to close the browser instance. Then all you have to do is run node index.js in the terminal.

It really is that simple! You should now see a new file called screenshot.png in your project folder.

If, for some reason, the website you want to scrape doesn’t show the content unless you’re logged in, you can automate the login process with Puppeteer.

Firstly we need to inspect the website that we’re scraping and find the login fields. We can do that by right-clicking on the element and choosing the Inspect option.

In my case, the inputs are inside a form with the class login-form. We can enter the login credentials using the type() method.

Also, if you want to make sure that it does the correct actions, you can add the headless parameter and set it to false when you launch the Puppeteer instance. You’ll then see how Puppeteer does the entire process for you.

const puppeteer = require('puppeteer')

async function login() {
   try {
       const URL = 'https://old.reddit.com/'
       const browser = await puppeteer.launch({headless: false})
       const page = await browser.newPage()

       await page.goto(URL)

       await page.type('.login-form input[name="user"]', 'EMAIL@gmail.com')
       await page.type('.login-form input[name="passwd"]', 'PASSWORD')

       await Promise.all([
           page.click('.login-form .submit button'),
           page.waitForNavigation(),
       ]);
      
       await browser.close()

   } catch (error) {
       console.error(error)
   }
}

login()

To simulate a mouse click we can use the click() method. After we click the login button, we should wait for the page to load. We can do that with the waitForNavigation() method.

If we entered the correct credentials, we should be logged in now!

I will use the /r/learnprogramming subreddit for this article. So we want to navigate to the website, grab the title and URL for every post. We’ll use the evaluate() method for that.

const puppeteer = require('puppeteer')

async function tutorial() {
   try {
       const URL = 'https://old.reddit.com/r/learnprogramming/'
       const browser = await puppeteer.launch()
       const page = await browser.newPage()

       await page.goto(URL)
       let data = await page.evaluate(() => {
           let results = []
           let items = document.querySelectorAll('.thing')
           items.forEach((item) => {
               results.push({
                   url: item.getAttribute('data-url'),
                   title: item.querySelector('.title').innerText,
               })
           })
           return results
       })

       console.log(data)
       await browser.close()

   } catch (error) {
       console.error(error)
   }
}

tutorial()

Using the Inspect method presented earlier, we can grab all the posts by targeting the .thing selector. We iterate through them, and for each one, we get the URL and the title and push them into an array.

After the entire process is completed, you can see the result in your console.

Great, we scraped the first page. But how do we scrape multiple pages of this subreddit?

It’s simpler than you think. Here’s the code:

We need a variable to know how many pages we want to scrape and another variable for the current page. While the current page is less than or equal to the number of pages that we want to scrape, we grab the URL and title for each post on the page. After each page is harvested, we concatenate the new results with the ones already scraped. 

Then we click the next page button and repeat the scraping process until we reach the desired number of extracted pages. We also need to increment the current page after each page.

Congratulations! You have successfully built your own web scraper with Puppeteer. I hope you enjoyed the tutorial!

Even so, the script we created in this guide cannot do a lot of hard work. It is missing a few key aspects that make web scraping feel flawless. Using mobile or residential proxies and solving CAPTCHAs are just a few of the missing functionalities.

If you're looking for a more professional way to extract data, take a look at what WebScrapingAPI can accomplish and see if it's a good fit. There's a free package, so all you have to invest in is 30 minutes of your time.

Web Scraping with Puppeteer Advanced Node.JS

In order to reach the scope of this article which is learning how to use cURL with a Proxy, we first need to introduce cURL. Client URL(cURL) is, for short, an easy-to-use command line designed for developers to fetch some data from a server.

As I already mentioned above, using cURL is pretty straightforward and can extract information with just a one-line command. Firstly you need to open an terminal and type curl followed by an website link, for example: 

Congratulations, you made your first request using cURL. This simple command requests information from the server just like a traditional browser does and returns the HTML of the page. Not every website will give you back HTML, there are endpoints which send back data as a JSON object. Take this example:

$ curl 'https://jsonplaceholder.typicode.com/todos/3'

Type this command in your terminal and you should get back this response:

{
  "userId": 1,
  "id": 3,
  "title": "fugiat veniam minus",
  "completed": false
}

Most API’s will give you back either HTML or JSON when you are running cURL commands against them. Well this is not everything cURL can do for us. In reality it is a very sophisticated tool which requires a lot of time to master. If you want to learn more about cURL, I strongly recommend you to take a look at cURL documentation for a better understanding of its parameters. Alternatively you can run the following command: 

This will show you some options you can set to cURL: 

Usage: curl [options...] <url>
 -d, --data <data>   HTTP POST data
 -f, --fail          Fail silently (no output at all) on HTTP errors
 -h, --help <category>  Get help for commands
 -i, --include       Include protocol response headers in the output
 -o, --output <file>  Write to file instead of stdout
 -O, --remote-name   Write output to a file named as the remote file
 -s, --silent        Silent mode
 -T, --upload-file <file>  Transfer local FILE to destination
 -u, --user <user:password>  Server user and password
 -A, --user-agent <name>  Send User-Agent <name> to server
 -v, --verbose       Make the operation more talkative
 -V, --version       Show version number and quit

This is not the full help, this menu is stripped into categories.
Use "--help category" to get an overview of all categories.
For all options use the manual or "--help all".

As you can probably see these are not even all the options you can set to cURL, it is a menu divided into categories. You probably guessed that in order to get all the options you would like to run:

However, using cURL alone has some limitations regarding the number of servers we can choose to fetch data from. For example, some servers can use geolocalization and refuse to give us the data we are looking for because of our location. This is the moment we need a proxy, which acts like a middleman between us and the target server.

The concept of a proxy server is not hard to understand at all. As already mentioned above , proxy server is like an intermediary between a client requesting a resource and the server providing that resource.Proxies are designated for us to be able to get data from anywhere. In order to understand better this concept , lets assume that we have a server called Bob which has some data we are interested in, but Bob provide that data only if we are in Europe, but we are in United States.

How we deal with that? We send our request to a proxy server which is located in Europe and not to Bob and tell the proxy that we want to get some data from Bob. The proxy will send itself the request to Bob and Bob will return to the proxy server data since the proxy is in Europe. Then the proxy server will send us back the data from Bob.

This is the main flow of how proxies work. Another great use case for a proxy is for example when we want to get data which contains prices in a specific currency in order to avoid confusion. For a further understanding of proxies, I strongly recommend you to have a look on Wikipedia.

In order to use a proxy , you will most likely need a host , a port , an user , a password and a target URL you want to get data from. For this example , I will use a proxy provided by WebScrapingAPI for making requests for which you can find more information about it here. WebScrapingAPI is not a proxy provider, it is a web scraping service which provides proxies instead .In our examples, our setup will be the following:

Proxy username: webscrapingapi.proxy_type=datacenter.device=desktop

Proxy password: <YOUR-API-KEY-HERE> // you can get one by registering here

Please note that there may be some proxy providers which require other schema of authentication.

Since we have covered cURL and proxies , now we are ready to combine them together and make requests by using a proxy which is a pretty straightforward process.We first need to authenticate and then we can use the proxy.

Proxy authentication in cURL is pretty simple and can be done for our example from above as following :

$ curl -U webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR-API-KEY> --proxy @proxy.webscrapingapi.com:80 http://httpbin.org/get

Running that command , httpbin will give us back our IP address , and some other properties:

​​{
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.5173.0 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-633af93b-35faf8637f37f7f075f185ec"
  }, 
  "origin": "45.9.120.69", 
  "url": "http://httpbin.org/get"
}

As you can already probably see, the origin you receive back is not your IP address and it is the address of the proxy server instead. Furthermore, you can run the command even without revealing your password in the terminal. This can be done as following: 

$ curl -U webscrapingapi.proxy_type=datacenter.device=desktop --proxy @proxy.webscrapingapi.com:80 http://httpbin.org/get

And then you will get a prompt to enter your password: 

Enter proxy password for user 'webscrapingapi.proxy_type=datacenter.device=desktop':

Now you can type your API key there without exposing it in the terminal which makes the whole process more secure.Furthermore, typing your credentials , host and port every single time you want to run a cURL command via a proxy may not feel that ideal , especially when you want to run many commands via a proxy and you are using the same proxy provider. 

Of course, you can store your credentials on a separate file stored on your machine and copy paste them everytime, but there is a more natural approach you can take which is passing them via environment variables which we will talk about below.

Using cURL with a proxy via environment variables 

An environment variable is like an object which stores an editable value in the memory which can be used by one or more softwares. In this particular case we can pass to cURL a variable called http_proxy or https_proxy which contains our proxy details and we will not need to specify on every run of the command. You can do that by running this command:

$ export http_proxy="http://webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR-API-KEY>@proxy.webscrapingapi.com:80"

Please note that you have to call your variable http_proxy or https_proxy in order for cURL to understand what you are talking about. That is it, now you do not need to pass your credentials on every run anymore and now you can just run cURL as simple as this: 

As you probably see the IP address is the proxy’s address which confirms that you have made a great job setting your proxy. At this point we can run any cURL command without specifying the proxy details, cURL will take care of that for us.

Disabling the proxy for a specific command

However, if you need to send a specific request without a proxy, you do not need to worry about deleting the value of http_proxy variable. Being a sophisticated tool with a lot of options , cURL can take care of that for us by its noproxy parameter which tells it to not use any proxy when making the request. It can be done as following:

$ curl --noproxy "*" http://httpbin.org/get

And that will give us back our own IP address and not the proxy’s one.

In conclusion, using cURL with a proxy is a great way to bypass geolocation filters, extends the amount of resources we can fetch from webservers and it is a good starting point for getting into topics such as web-scraping where we need to use proxies in order to be able to get certain data or to receive it in the format we want.I hope you found this article useful for you to learn how to use cURL with a proxy and you will play around with it and build your own scripts which extracts data from servers which use geolocation filters.

Recent articles

Sign up for our newsletter

News and updates