Long gone are the days when you manually collected and processed the data helping you kickstart your projects. Whether an e-commerce website or a lead generation algorithm, one thing is sure: the data-gathering process was tedious and time-consuming.

In this article, you will learn how Cheerio can come to your help with its extensive features of parsing markup languages, first with some trivial examples and then with a real-life use case.

“But what is Cheerio?” you may ask yourself. Well, trying to clarify a common misconception, I will start with what Cheerio is not: a browser.

The confusion may start from the fact that Cheerio parses documents written in a markup language and then offers an API to help you manipulate the resulting data structure. But unlike a browser, Cheerio will not visually render the document, load CSS files or execute Javascript.

So basically what Cheerio does is to receive an HTML or XML input, parse the string and return the API. This makes it incredibly fast and easy to use, hence its popularity among Node.js developers.

Now, let’s see some hands-on examples of what Cheerio can do. First things first, you need to make sure your environment is all set up.

It goes without saying that you must have Node.js installed on your machine. If you don’t, simply follow the instructions from their official website, according to your operating system.

Make sure to download the Long Term Support version (LTS) and don’t forget about the Node.js Package Manager (NPM). You can run these commands to make sure the installation went okay:

Now, regarding the IDE debate: for this tutorial, I will be using Visual Studio Code, as it is pretty flexible and easy to use, but you are welcome to use any IDE you prefer.

Just create a folder that will hold your little project and open a terminal. Run the following command to set up a Node.js project:

This will create a default version of the package.json file, which can be modified at any time.

Next step: I will install TypeScript along with the type definitions for Node.js:

npm install typescript @types/node -save-dev

I chose TypeScript in this tutorial for its optional static typing to JavaScript objects, which makes the code more bulletproof when it comes to type errors. 

This is the same advantage that steadily increased its popularity among the JavaScript community, according to a recent CircleCI survey regarding the most popular programming languages.

To verify the correct installation of the previous command, you can run:

Now I will create the tsconfig.json configuration file at the root of the project directory, which should define the compiler options. If you want a better understanding of this file and its properties, the official TypeScript documentation got your back. 

If not, then simply copy and paste the following:

Almost done! Now you have to install Cheerio (obviously):

Last but not least, create the src directory which will hold the code files. And speaking of the code file, create and place the index.ts file in the src directory.

For the moment I will illustrate some basic Cheerio features using a static HTML document. Simply copy and paste the content below into a new static.html file within your project:

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Page Name - Static HTML Example</title>
</head>
<body>
    <div class="page-heading">
        <h1>Page Heading</h1>
    </div>
    <div class="page-container">
        <div class="page-content">
            <ul>
                <li>
                    <a href="#">Item 1</a>
                      <p class="price">$100</p>
                      <p class="stock">12</p>
                </li>
                <li>
                    <a href="#">Item 2</a>
                    <p class="price">$200</p>
                    <p class="stock">422</p>
                </li>
                <li>
                    <a href="#">Item 3</a>
                    <p class="price">$150</p>
                    <p class="stock">5</p>
                </li>
            </ul>
        </div>
    </div>
    <footer class="page-footer">
        <p>Last Updated: Friday, September 23, 2022</p>
    </footer>
</body>
</html>

Next, you have to serve the HTML file as input to Cheerio, which will then return the resulting API:

import fs from 'fs'
import * as cheerio from 'cheerio'

const staticHTML = fs.readFileSync('static.html')
const $ = cheerio.load(staticHTML)

If you receive an error at this step, make sure that the input file contains a valid HTML document, as from Cheerio version 1.0.0 this criterion is verified as well.

Now you can start experimenting with what Cheerio has to offer. The NPM package is well-known for its jQuery-like syntax and the use of CSS selectors to extract the nodes you are looking for. You can check out their official documentation to get a better idea.

Let’s say you want to extract the page title:

const title = $('title').text()
console.log("Static HTML page title:", title)

We should test this, right? You are using Typescript, so you have to compile the code, which will create the dist directory, and then execute the associated index.js file. For simplicity, I will define the following script in the package.json file:

"scripts": {
    "test": "npx tsc && node dist/index.js",
}

and the script will handle both steps that I just described.

Okay, but what if the selector matches more than one HTML element? Let’s try extracting the name and the stock value of the items presented in the unordered list:

const itemStocks = {}
$('li').each((index, element) => {
    const name = $(element).find('a').text()
    const stock = $(element).find('p.stock').text()
    itemStocks[name] = stock
})
console.log("All items stock:", itemStocks)

Now run the shortcut script again, and the output from your terminal should look like this:

Static HTML page title: Page Name - Static HTML Example
All items stock: { 'Item 1': '12', 'Item 2': '422', 'Item 3': '5' }

So that was basically the tip of the iceberg. Cheerio is also capable of parsing XML documents, extracting the style of the HTML elements and even altering the nodes’ attributes.

But how can Cheerio help in a real use case?

Let’s say that we want to gather some data to train a Machine Learning model for a future bigger project. Usually, you would search Google for some training files and download them, or use the website’s API.

But what do you do when you can’t find some relevant files or the website you are looking at does not provide an API, has a rate-limiting over the data or does not offer the whole data that you are seeing on a page?

Well, this is where web scraping comes in handy. If you are curious about more practical use cases of web scraping, you can check out this well-written article from our blog.

Back to our sheep, for the sake of the example let’s consider that we are in this precise situation: wanting data, and data is nowhere to be found. Remember that Cheerio does not handle the HTML extraction nor the CSS loading or JS execution.

Thus, in our tutorial, I am using Puppeteer to navigate to the website, grab the HTML and save it to a file. Then I will repeat the process from the previous section.

To be more specific, I want to gather some public opinions from Reddit on a popular drum module and centralise the data in a single file that will be further fed to a potential ML training model. What happens next can also vary: sentiment analysis, market research and the list can go on.

Let’s see how this use case will look put into code. First, you need to install the Puppeteer NPM package:

I will also create a new file reddit.ts, to keep the project more organised, and define a new script in the package.json file:

"scripts": {
    "test": "npx tsc && node dist/index.js",
    "parse": "npx tsc && node dist/reddit.js"
},

To get the HTML document, I will define a function looking like this:

import fs from 'fs'
import puppeteer from 'puppeteer'
import * as cheerio from 'cheerio'

async function getContent(url: string): Promise<void> {

    // Open the browser and a new tab
    const browser = await puppeteer.launch()
    const page = await browser.newPage()

    // Navigate to the URL and write the content to file
    await page.goto(url)
    const pageContent = await page.content()
    fs.writeFileSync("reddit.html", pageContent)

    // Close the browser
    await browser.close()
    console.log("Got the HTML. Check the reddit.html file.")
}

To quickly test this, add an entry point in your code and call the function:

async function main() {


    const targetURL = 'https://old.reddit.com/r/Drumming/comments/r3tidc/yamaha_ead10/'
    await getContent(targetURL)
}

main()
    .then(() => {console.log("All done!")})
    .catch(e => {console.log("Unexpected error occurred:", e.message)})

The reddit.html file should appear in your project tree, which will contain the HTML document we want.

Now, a bit of a more challenging part: you have to identify the nodes that are of interest for our use case. Go back to your browser (the real one) and navigate to the target URL. Go with your mouse cursor over the comments section, right-click and then choose the “Inspect” option.

The Developer Tools tab will open, showing you the exact same HTML document you previously saved on your machine.

To extract only the comments, you have to identify the selectors unique to this section of the page. You can notice that the whole list of comments is within a div container with a sitetable nestedlisting class.

Going deeper, each individual comment has a form element as a parent, with the usertext warn-on-unload class. Then, at the bottom, you can see that every comment’s text is divided between multiple p elements.

function parseComments(): void {

    // Load the HTML document
    const staticHTML = fs.readFileSync('reddit.html')
    const $ = cheerio.load(staticHTML)

    // Get the comments section
    const commentsSection = $('div.sitetable.nestedlisting')

    // Iterate each comment
    const comments = []


    $(commentsSection).find('form.usertext.warn-on-unload').each((index, comment) => {
        let commentText = ""

          // Iterate each comment section and concatenate them
          $(comment).find('p').each((index, piece) => {
            commentText += $(piece).text() + '\n'
          })

          comments.push(commentText)
    })

    // Write the results to external file
    fs.writeFileSync("comments.json", JSON.stringify({comments}))
}

Alright, and now let’s update the entry point with the newly defined function and see how this code is working together:

async function main() {

    const targetURL = 'https://old.reddit.com/r/Drumming/comments/r3tidc/yamaha_ead10/'
    await getContent(targetURL)
    parseComments()
}

main()
    .then(() => {console.log("All done. Check the comments.csv file.")})
    .catch(e => {console.log("Unexpected error occurred:", e.message)})

Execute the code with the script defined before:

It will take around 5 to 10 seconds for the headless browser to open and navigate to our target URL. If you are more curious about it, you can add timestamps at the beginning and the end of each of our functions to really see how fast Cheerio is.

The comments.json file should be our final result:

This use case can be easily extended to parsing the number of upvotes and downvotes for every comment, or to get the comments’ nested replies. The possibilities are endless.

Thank you for making it to the end of this tutorial. I hope you grasped how indispensable Cheerio is to the process of data extraction and how to quickly integrate it into your next scraping project.

We are also using Cheerio in our product, WebScrapingAPI. If you ever find yourself tangled by the many challenges encountered in web scraping (IP blocks, bot detection, etc.), consider giving it a try.

Scraping with Cheerio: How to Easily Collect Data from Web Pages

Developers are using web scrapers for all kinds of data fetching. Let us show you how to build your own web scraper using JavaScript.

The Ultimate Guide to Web Scraping with JavaScript and Node.Js

Let’s be honest. From here on out, the Internet’s data volume will only get bigger and bigger. There’s really nothing we can do about it. This is where Web Scrapers come into the picture. 

In the following article, we will show you how to build your own Web Scraper using JavaScript as the main programming language.

Understanding Web Scraping with JavaScript

A web scraper is a piece of software that helps you automate the tedious process of collecting useful data from third-party websites. Usually, this procedure involves making a request to a specific web page, reading the HTML code, and breaking down that code to gather some data.

Let's say you want to create a price comparison platform. You need the prices of several items from a couple of online shops. A web scraping tool can help you manage this in a couple of minutes. 

Maybe you're trying to get some new leads for your company or even get the most favorable flight or hotel prices. While we were crawling the web, researching for this article we stumbled upon Brisk Voyage. 

Brisk Voyage is a web application that helps its users find cheap, last-minute weekend trips. Using some sort of web scraping technology, they manage to constantly check flight and hotel prices. When the web scraper finds a trip that’s a low-priced outlier, the user receives an email with the booking instructions.

Developers are using web scrapers for all kinds of data fetching but the most used cases are the following:

Collection training and testing datasets for Machine Learning

What are the challenges of Web Scraping with JavaScript & Node.Js?

You know those little tick boxes that make you admit you're not a robot? Oh well, they don't always succeed in keeping the bots away.

But most of the time they do, and when search engines find out you're trying to scrape their website without permission, they restrict your access. 

Another obstacle web scrapers face is represented by the changes in a website's structure. One small change in the website's structure can make us waste a lot of time. Web scraping tools require frequent updates to adapt and get the job done.

One other challenge web scrapers face is called geo-blocking. Based on your physical location, a website can completely ban your access if requests come from untrustworthy regions.

To combat these challenges and help you focus on building your product we created WebScrapingAPI. It’s an easy-to-use, enterprise-grade scaled API that helps you collect and manage HTML data. We’re obsessed with speed, we use a global rotating proxy network and we have more than 10.000 clients already using our services. If you feel like you don’t have the time to build the web scraper from scratch, you can give WebScrapingAPI a try by using the free tier.

Most web applications supply an API that allows users to get access to their data in a predetermined, organized way. The user will make a request to a specific endpoint and the application responds with all the data the user specifically asked for. More often than not, the data will already be formatted as a JSON object.

When using an Application Programming Interface you normally don't have to worry about the previously presented obstacles. Be that as it may, APIs can also receive updates. In this situation, the user has to always keep an eye on the API he's using and update the code accordingly to not lose its functionality. 

Furthermore, the documentation of an API matters a lot. If an API functionality it's not clearly documented the user finds itself wasting a lot of time.

A good understanding of the Internet requires a lot of knowledge. Let’s go through a brief introduction to all the terms you need to better understand web scraping.

HTTP or HyperText Transfer Protocol is the foundation of any data exchange on the web. As the name suggests, HTTP is a client-server convention. An HTTP client like a web browser, opens a connection to an HTTP server and sends a message, like: "Hey! What's up? Do you mind passing me those images?". The server typically offers a response, as the HTML code, and closes the connection.

Let's say you need to visit Google. If you type the address in the web browser and hit enter, the HTTP client (the browser) will send the following message to the server:

GET / HTTP/1.1
Host: google.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/web\p,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch, br
Connection: keep-aliveUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36

The first line of the message contains the request method (GET), the path we made the request to (in our case is just '/' because we only accessed www.google.com), the version of the HTTP protocol, and multiple headers, like Connection or User-Agent.

Let's talk about the most important header fields for the process:

Host: The domain name of the server you accessed after you typed the address in the web browser and hit enter.

User-Agent: Here we can see details regarding the client that made the request. I use a MacBook, as you can see from the __(Macintosh; Intel Mac OS X 10_11_6)__ part, and Chrome as the web browser __(Chrome/56.0.2924.87)__.

Accept: By using this header, the client constrains the server to only send him some type of responses, like application/JSON or text/plain.

Referrer:  This header field contains the address of the page making the request. Websites use this header to change their content based on where the user comes from.

HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu) Content-Type: text/html; charset=utf-8 
<!DOCTYPE html>
<html>
<head>
<title>Title of the document</title>
</head>
<body>The content of the document</body>
</html>

As you can see, on the first line there's the HTTP response code: **200 OK. This means that the scraping action was successful. 

Now, if we would have sent the request using a web browser, it would have parsed the HTML code, obtained all the other assets like CSS, JavaScript files, images, and rendered the final version of the web page. In the steps below, we are going to try to automate this process.

JavaScript was initially created to help its users add dynamic content to websites. In the beginning, it couldn't interact directly with a computer or its data. When you access a website, the JavaScript is read by the browser and changed to a couple of lines of code that the computer can process. 

Introducing Node.Js, the tool that helps Javascript run not only client-side but also server-side. Node.Js can be defined as a free, open-source JavaScript for server-side programming. It helps its users build and run network applications quickly.

const http = require('http');
const port = 8000;
const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello world!');
});
server.listen(port, () => {
  console.log(`Server running on port 8000.`);
});

If you don't have Node.Js installed, check the next step for the following instructions. Else create a new index.js file and run it by typing in a terminal node index.js, open up a browser and navigate to localhost:8000. You should see the following string: "Hello world!".

Chrome - Please follow the installation guide here.

VSCode - You can download it from this page and follow the instructions to install it on your device.

Node.Js - Before starting using Axios, Cheerio, or Puppeteer, we need to install Node.Js and the Node Package Manager. The easiest way to install Node.Js and NPM is to get one of the installers from the Node.Js official source and run it.

After the installation is done, you can verify if Node.Js is installed by running node -v and npm -v in a terminal window. The version should be higher than v14.15.5. If you're having issues with this process, there's an alternative way of installing Node.Js. 

Now, let's create a new NPM project. Create a new folder for the project and run npm init -y. Now let's get to installing the dependencies.

Axios - a JavaScript library used to make HTTP requests from Node.Js. Run npm install axios in the newly created folder.

Cheerio - an open-source library that helps us to extract useful information by parsing markup and providing an API for manipulating the resulting data. To use Cheerio you can select tags of an HTML document by making use of selectors. A selector looks like this:  $("div"). This specific selector helps us pick all <div> elements on a page. 

To install Cheerio, please run npm install cheerio in the projects' folder.

Puppeteer - a Node.Js library that used to get control of Chrome or Chromium by providing a high-level API. 

To use Puppeteer you have to install it using a similar command: npm install puppeteer. Note that when you install this package, it will also install a recent version of Chromium that is guaranteed to be compatible with your version of Puppeteer.

First, you need to access the website you want to scrape using Chrome or any other web browser. To successfully scrape the data you need, you have to understand the website's structure. 

Once you managed to access the website, use it just as a normal user would do. If you accessed the /r/dundermifflin subreddit, you can check out the posts on the main page by clicking on them, checking out the commentaries, and the upvotes, and even sorting the posts by the number of votes during a specific timeframe.

As you can see, the website contains a list of posts, and each post has some upvotes and some comments. 

You can understand a lot of the data of a website just by looking at its URL. In this case, https://www.old.reddit.com/r/DunderMifflin represents the base URL, the path that gets us to the "The Office" Reddit community. As you are starting to sort the posts by the number of votes, you can see that the base URL is changing to https://www.old.reddit.com/r/DunderMifflin/top/?t=year.

The query parameters are extensions of the URL that are used to help define specific content or actions based on the data being passed. In our case "?t=year" represents the time frame selected for which we will want to see the most upvoted posts.

As long as you're still on this specific subreddit, the base URL will stay the same. The only things that will change are the query parameters. We can look at them as the filters applied to the database to retrieve the data that we want. You can switch the timeframe to only see the top upvoted posts in the last month or week just by changing the URL.

In the following steps, you are going to learn more about how the information is organized on the page. You'll need to do this to have a better understanding of what can we actually scrape from our source.

The developer tools help you interactively explore the website's Document Object Model (DOM). We are going to use the developer tools in Chrome but you can use any web browser you're comfortable with. In Chrome, you can open it by right-clicking anywhere on the page and selecting the "Inspect" option.

In the new menu that appeared on the screen, please select the "Elements" tab. This will present the interactive HTML structure of the website. 

You can interact with the website by editing its structure, expanding and collapsing elements, or even deleting them. Note that these changes will only be visible to you.

Regular expressions, also known as RegEx, help you create rules that allow you to find and manage different strings. If you ever need to parse large amounts of information, mastering the world of regular expressions can save you a lot of time.

When you first start using regex, it seems to be a little too complicated, but the truth is they're pretty easy to use. Let's take the following example: \d. By using this expression, you can easily grab any digit from 0 to 9. Of course, there are a lot more complex ones, like: ^($\d{3}$|^\d{3}[.-]?)?\d{3}[.-]?\d{4}$. This matches a phone number, with or without parentheses around the area code, or with or without dots to separate the numbers.

As you can see regular expressions are pretty easy to use and can be very powerful if you spend enough time mastering them.

After you've successfully installed all the previously presented dependencies and inspected the DOM using the developer tools, you can get to the actual scraping.

Web Scraping with JavaScript and Node.Js

We personally cannot see a bright future for data extraction without proxies in the picture. Most of the time, you either run the risk of being blocked by the website or scrape at a snail's pace. In short, without multiple IPs, web scrapers lose most of their luster.

You may be wondering why that is. Simple - bot detection tools.

Bot detection software has dramatically advanced, so kudos to them. Captchas, IP blacklists, and request throttling are examples of features that help protect the Internet from malicious bots. Unfortunately, these tools also make it difficult for friendly web scrapers to do their job.

Proxies serve a variety of purposes, but what role do they play in web scraping? Frankly, quite a large one.

Proxies, at their core, are meant to mask your real IP. Additionally, they're crucial for accessing geo-restricted content since websites think that your requests come from different regions. The best-known example of this are streaming sites. The Office, for example, is available on Netflix only in the UK and Ireland, but if you get a UK proxy, you can watch the show from anywhere.

In web scraping, the proxy pool is one of the most critical components. They're used to make it look like the bot's requests come from different locations and at different times. This is your first line of defense from IP blocks. Additionally, if it fails, your real IP won't be barred from accessing the website.

Web scrapers, and bots in general, can send tons of requests very quickly. That's what makes them so desirable for data gathering. But, this speed is often their downfall. Websites can determine if the requests are coming from a legitimate person or a bot by their behavior. For example, a human won't ever request 25 pages in less than a second.

Add a proxy in the middle, or better yet, a whole proxy pool, and suddenly you have more options. By distributing the requests to a handful of proxies, you're changing what the website sees. Namely, instead of one IP sending 100 requests at once, it's now 10 IPs sending 10 requests each. Ideally, you send each request through a different proxy.

You may think that constantly switching IPs is a huge chore, and you'd be right. That's why proxy service providers came up with proxy rotation - an automated system that changes IPs for you. Rotating proxies is the best way of making sure you're using all your IPs to their full potential.

In fact, widespread scraping is nearly impossible without a large, rotating proxy pool.  Pacing yourself matters, and paying attention to how many requests you are allowed to submit in a certain amount of time can make or break your progress. The broader your proxy pool is, the more requests you can send without drawing suspicion. The result is clear - you're much less likely to be blocked.

Essentially, a residential IP address is an address issued by an ISP to a household. When you set up the Internet in a new house or apartment, you receive a residential IP, and every time you view a webpage, you're accessing it thanks to that IP. Use these IPs as proxies, and you're officially cooking with residential proxies.

Since these IPs come from Internet Service Providers, they are much more trustworthy than other kinds of proxies. For example, datacenter proxies are created in bulk on cloud-hosted, virtual servers and enjoy much less trust from websites.

Another big plus for residential IPs is that service providers usually have proxies scattered across the globe. As a result, they can provide you with access to just about any content. Geo-restrictions stop being a problem once you have proxies in dozens of different countries.

So, in essence, residential IPs are the top-quality, highly anonymous proxies that get the job done where other IPs fail. The best solution (both in functionality and ease of use) would be a backconnect proxy that automatically switches between IPs at a fixed interval or after every request.

Now that you got the gist of how residential proxies work, you're probably wondering which provider you should be going for. I have compiled a clear list of what I personally consider the best available options:

We'll start with WebScrapingAPI for a simple reason: it's a proxy provider and a web scraping service provider at the same time. With over 100 million proxies available, you get the proxy pools you need at any price point - be it the free trial, business, or custom plan. The free trial lasts for 14 days, and in that time, you can try all kinds of different geolocation options. After that, the prices start at $20/month for 200,000 API calls, but you can still use the limited free-forever package.

Each package has a different number of allowed API calls, and you can use those as you please with unlimited bandwidth. Additionally, only successful calls are counted. If a request fails, you can try again without worrying that you'll run out of API calls.

As you might have noticed, WebScrapingAPI can be much more than just your trusted proxy partner. The API can scrape any page and handle everything from CAPTCHAs to Javascript rendering to request throttling. In short, you have an all-in-one web scraping tool.

Oxylabs is a well-known residential proxy provider with over 70 million IP addresses available worldwide. They allow you to filter by region, which will help you find the finest proxies for your project. With a big proxy pool, you'll have access to Residential IP addresses from all over the world, making it easy to get over geo-blocks.

Additionally, concurrent sessions are not limited, so you can simply scale up your web scraping tasks as needed.

The costs, however, may be a dealbreaker for some. The cheapest residential IP package costs $300/month for 20GB of traffic. You also have the option to incorporate machine learning that should raise your success rate. In that case, the price is a bit higher, namely $360/month for the same bandwidth.

GeoSurf is a residential proxy network with over 2 million residential IP addresses in 192 countries. With that much variety, it's unlikely that you'll face any problems with geolocation.

Most of their proxies (close to a million) are located in Asia. So, if you're going to primarily target websites hosted in Asia, GeoSurf is a good choice.

They also provide a toolbar browser plugin that lets you see online material through different IPs from across the world. This is particularly useful for people who use their proxies for ad verification. The GeoSurf Toolbar is compatible with Internet Explorer, Chrome, Firefox, and Firefox for Mac.

The starter plan may seem steep, but you also get quite a bit of bandwidth for the price: $450/month for 38GB of traffic through residential IPs in +130 countries. You should consider this option if your scraping project is considerably large. Otherwise, you might want to stick to less expensive providers.

Bright Data claims to be the largest data-collecting platform and proxy service provider on the globe. With more than 72 million IP addresses and excellent load speeds, this company deserves its spot on the list.

Their geological coverage is quite impressive. In fact, they seem to have 14 residential IPs in North Korea. I didn't think that was possible, and yet here we are.

With datacenter proxies, it's pretty common to have the option of using shared or dedicated proxies. With residential IPs, the choice is rarer, but Bright Data does give you the opportunity. Our advice is to stick to dedicated proxies unless you're trying to lower costs as much as possible.

Their 'experimenting' plan is the cheapest option, with each GB of bandwidth costing $15. Alternatively, they have a pretty complex pricing calculator that you can use to create a custom plan.

Smartproxy is a premium proxy service that offers servers at a low cost. It's a secure and dependable proxy service that provides a money-back guarantee if you don't like it. They have over 40 million IPs in more than 195 locations.

All proxies in the network are anonymous, and their servers use complex rotation, which means you'll obtain a live and tested proxy after each rotation. It's your choice if it should be completely random or from a specific country. This is one of the simplest residential proxy networks to deploy, and it eliminates the need for proxy maintenance.

The Micro plan is a good option if you're in the experimentation phase of your project. It costs 75 dollars and provides you with 5GB of bandwidth, with the possibility of going over the limit for $15 per GB.

At first glance, NetNut's 20M+ residential proxies may seem few compared to some of the other providers. That may be true, but their solid infrastructure ensures that the IPs you do have access to are always available and operational.

As far as we can tell, their IPs are spread across approximately 50 different locations. While not ideal, it does give you viable geolocation options.

Their pricing model is quite interesting. Most clients will opt for a price based on bandwidth, the same as many other options. But, if you have a big project in the works, you can also pay based on the number of API calls you get for their own API.

The starter plan costs $300 per month and nets you 20GB of bandwidth. The lowest request-based package is called Plus and costs a whopping $7500 but provides you with 50M API calls.

Compared to other service providers on this list, StormProxies seem to be more focused on the needs of the lone developer. Their prices are considerably lower, which makes them a good choice if you're learning how to make your first scraper as well.

You can choose between a proxy pool of 40,000 residential proxies or a network of 70,000 IPs with both datacenter and residential mixed in. They also have datacenter-only options, but that's not the focus of today's article.

If you opt for the rotating proxy packages, prices start at $50 per month, and you get access to 5 residential proxy ports. If you'd instead have dedicated IPs, you can get 5 private proxies for just $10.

Sadly, their geographical coverage, geo-targeting choices, and authentication mechanism, among other things, are severely constrained.

RSocks is very transparent with their stats: 8M residential proxies and 68 personal proxy countries. Compared to providers with 195 geolocation options, it may seem a bit limited, but they can be a great provider, depending on your use case.

They have a large number of different packages up for purchase. You can choose one based on geolocation, rotation options, or even themes (for specific platforms like Youtube and Twitch).

Telling you a price here won't do much good since the price heavily depends on what's being offered. The criteria that will determine that price are:

Whether they have rotating proxies implemented;

So, while it's difficult to draw a clear conclusion, we found their prices acceptable. They might not be the cheapest option, but they're far from the most expensive.

Shifter, which claims to have the largest pool of peer-to-peer connections on the Internet, with 31 million IP addresses, has won the Internet vote of confidence for many users.

Their packages are split into two main categories: basic backconnect proxy plans and special backconnect proxy plans. The main difference is in how many extra functionalities you get. Special proxy plans let you choose the location through which to send your request and allows you access to high-demand websites.

So, if you know you'll have to deal with geo-restricted content, make sure you get the right package. If not, they have a three-day money-back guarantee.

10 special backconnect proxies (which have access to many more IPs) would cost you $250. Alternatively, you can get 25 basic proxies for the same price. You can also choose how often the IP pool behind your backconnect proxy should refresh. The minimum amount is 5 minutes, while the maximum is an hour.

Scraping the web without a care in the world

With so many benefits to using residential proxies, the actual issue is, which one is best suited for every particular need?

Not tackling the best proxy service might result in the scraper being banned or restricted, so take your time and examine all of the options above before making a decision.

While proxy providers are a valuable resource to integrate with a separate web scraper, keeping track of both can be difficult. We built WebScrapingAPI to be the perfect bridge between the two. So, my closing question for you is this: Why not start your free trial and see what the API can do for you?

The Top 9 Residential Proxy Service Providers

We all hate the moments when we try to access a page and the site blocks our request for no good reason. Geoblocking, for example, can be passed only by using a proxy. 

Node-unblocker helps us to create a custom proxy and have it up and running in a matter of minutes.

Node unblocker is a general-purpose library to create a web proxy, intercept and alter requests and responses. 

This library is also used in web scraping for bypassing restrictions implemented by the site like geo-blocking, hiding the IP address and rate limiting, or for sending authentication tokens. 

To make it shorter, while using this library, you can kiss blocked and censored content goodbye.

In this article, we create an Express application with a custom proxy using Node Unblocker, we add a middleware that changes the user agent for each request, discuss the proxy limitations, deploy it to Heroku and compare it to a managed service like WebScrapingAPI.

Before we start, make sure you have the latest version of Node.JS installed. Installing Node.JS for each platform (Windows, Linux, Mac) would be the subject of a separate article so instead of going into details, check the official website and follow the instructions.

We start by creating a directory for our project named unblocked and we initialize a Node.JS project inside of it:

For this application, we install two libraries: Express, a minimalist framework for Node.JS, and Node Unblocker.

Because of node.unblocker running inside an Express instance, we have to set up the Hello World example in our application.

Create an index.js file and paste the following code:

const express = require('express') const app = express() const port = 8080  app.get('/', (req, res) => {     res.send('Hello World!') })  app.listen(process.env.PORT || 8080, () => {     console.log(`Example app listening on port ${port}`) })

We can run the application with this command:

If we access http://localhost:8080 we see a Hello World message. This means our application is up and running.

It’s time to import the Node Unblock library into our application:

We create a node unbloker instance and pass the proxy parameter. You can find the full list of available parameters here.

var unblocker = new Unblocker({prefix: '/proxy/'})

We register the node unblocker library to Express as a middleware so requests are intercepted:

We update the Express app listener to add support for websockets:

app.listen(process.env.PORT || 8080, () => {     console.log(`Example app listening on port ${port}`) }).on('upgrade', unblocker.onUpgrade)

Recent articles

Sign up for our newsletter

News and updates