Let's not sugarcoat this, as the years drag on, the Internet's data volume will only continue to rise. The situation is beyond anyone's control, but is that really a bad thing?

Particularly in the past decade, web scraping has gained enormous popularity. To succeed in today's business world, companies need proper strategies, which require ample information within a short amount of time. Plus, it could be a neat way for developers to improve their coding skills AND help the company.

If you play for team Ruby but don't have much to do with web scraping, this article will provide you with a new niche to explore: building your own web scraper.

Let's take a price comparison platform as an example. Its job is to get the cost of a multitude of items from several online retailers. But why stop at physical products? The airline and hospitality industries have also become a lot more consumer-friendly, thanks to comparison sites. So, how do these magical websites or apps function? Through web scraping, of course!

You are likely wondering now, "How else will I use this data?". So let's have a look at some of the practical applications of web scraping:

Real estate agencies require data on properties, neighborhoods, and whole areas to make the best deals

Sales and marketing leads need branding and pricing data from their competitors to create competitive strategies

Companies must generate leads to find clients if it hopes to stay in business

This well-written article that discusses web scraping provides detailed descriptions and additional use cases.

Web scrapers are not easy to create, even if you understand how it works and the potential benefits it can provide. There are many ways through which websites can identify and block bots from accessing their data.

Geo-blocking: You might be shown regionally specific results when asking for information from another area (e.g., airplane ticket prices).

IP Blocking: A website can block or slow you down when it determines you are making repeated requests from a particular IP address;

Honeypots: Humans cannot see honeypot links, but bots can; once the robots fall for the trap, their IP address is blocked;

CAPTCHAs: People can solve these simple logical problems relatively quickly, but scrapers often find them difficult;

It is no easy task to overcome all these obstacles. It isn't all that difficult to build a simple bot, but creating an excellent web scraper is a bit more challenging. Therefore, over the past decade, APIs for web scraping have become one of the hottest topics.

HTML content from any website can be collected using WebScrapingAPI, and all the problems we mentioned earlier will be automatically addressed. Additionally, we use Amazon Web Services to ensure speed and scalability. Sounds interesting, right? Well, don't just stop at sound! Try it for yourself with the 5000 free API calls you get as part of the free WebScrapingAPI trial.

It is necessary to understand Hypertext Transfer Protocol (HTTP) to understand the Web. It explains how a server and a client communicate. A message contains information that describes the client and how it handles data: method, HTTP version, and headers.

For HTTP requests, web scrapers use the GET method to retrieve data from the server. Additionally, there are some advanced methods such as POST and PUT. The HTTP methods are detailed here for your reference.

In HTTP headers, a variety of additional information can be found about requests and responses. For web scraping, these are the ones that matter:

User-Agent: web scrapers rely on this header to make their requests seem more realistic; it contains information such as the application, operating system, software, and version.

Cookie: the server and request can exchange confidential information (such as authentication tokens).

Referrer: contains the source site the user visited; accordingly, it is essential to consider this fact.

Host: it identifies the host that you are connecting to.

Accept: provides a response type for the server (e.g., text/plain, application/json).

Ruby is a high-level multiparadigm programing language that is also fully interpretable. This means that the program code is stored in plain text, which is transmitted to the interpreter that executes it. 

In 1995, Yukihiro Matsumoto (also known as Matz in the Ruby community) combined features of different programming languages such as Perl, Lisp, and Smalltalk to create a new one that focuses on simplicity and productivity.

It's a niche programming language, its natural area being web applications. Here are the significant advantages you benefit from using this language in your projects:

It gets results, fast. In combination with the Rails framework, you can create software relatively quickly; that is why startups mainly prefer Ruby to promptly build their MVPs (Minimum Viable Product).

It's well-developed and maintained by the growing Ruby community.

Helpful tools and libraries (called gems) ensure that it's easy to follow the best coding practices in almost any situation.

On the other hand, these advantages do not make Ruby a magical universal solution to every new software. You may also consider these characteristics of the language before making a decision:

As their size grows, applications built with Ruby get slower, which causes problems to scalability.

Its natural area is web applications. Therefore, it is not well suited for desktop/mobile applications.

Because it uses an interpreter, a possible Object-Oriented code will be slower.

Now we can start talking about extracting data. First things first, we need a website that provides valuable information. 

To build our Ruby web scraper, we need first to make sure that we have all the prerequisites:

The latest stable version of Ruby: check out their official installation guide to choose the best method for your operating system.

An IDE: in this guide, we will use Visual Studio Code, as it is lightweight and there is no need for additional configurations, but you can choose whatever IDE you prefer.

Bundler: a dependency management Ruby tool (also called a gem);

Watir: a Selenium powered gem used for automatic testing, as it can imitate user's behavior on a browser;

Webdrivers: a gem recommended by Watir that will automatically download the latest driver for a browser instance;

Nokogiri: a gem well-known for the ability to make web pages analysis easy. It can parse HTML, XML, detects broken HTML documents, and offers access to elements by XPath and CSS3 selectors.

After setting up the Ruby environment, create a new directory anywhere on your computer and open it with your IDE of choice. Then run the following command in a terminal window to install our first gem:

Now create a file called Gemfile in your project root directory. Here we are going to add the rest of the gems as dependencies:

source 'https://rubygems.org'

gem 'watir', '~> 6.19', '>= 6.19.1'
gem 'webdrivers', '~> 4.6'
gem 'nokogiri', '~> 1.11', '>= 1.11.7'

Now get back to the terminal window and run the following command to install the gems we declared:

Cool setup! Finally, just create a “scraper.rb” file to hold our web scraper’s code. Whatever we write here, we can execute with the command:

Step 2: Inspect the page you want to scrape

Great, let’s move on! Navigate to the page you want to scrape and right-click anywhere on it, then hit “Inspect element”. The developer console will pop up, where you should see the website’s HTML.

Step 3: Send an HTTP request and scrape the HTML

Now, to get that HTML on our local machine, we have to send an HTTP request using Watir to return the document. Let’s get back to the IDE and put this idea into code.

require 'watir'
require 'webdrivers'
require 'nokogiri'

Then we initialize a browser instance and navigate to the website that we want to scrape. We then access the HTML and pass it to the Nokogiri constructor, which will help us to parse the result.

browser = Watir::Browser.new
browser.goto 'https://blog.eatthismuch.com/latest-articles/'
parsed_page = Nokogiri::HTML(browser.html)

File.open("parsed.txt", "w") { |f| f.write "#{parsed_page}" }

browser.close

We also wrote the result in a text file called “parsed.txt” to take a look over the HTML. It’s important to close the connection after receiving the response, as the process will continue running.

So, we have an HTML document, but we want data, which means that we should parse the previous response into human-readable information.

Starting with baby steps, let’s extract the title of the website. A remarkable fact about Ruby is that everything is an object with very few exceptions, meaning that even a simple string can have attributes and methods.

Therefore, we can simply access the value of the website’s title through the attributes of the parsed_page object.

Moving forward, let’s extract all the links from the website. For this, we will use a more generic method that parses specific tags, the css method.

links = parsed_page.css('a')
links.map {|element| element["href"]}

puts links

We also use the map method to keep only the links with a href attribute from the HTML.

Let’s take a more realistic example. We need to extract the articles from the blog, their title, address, and meta description.

If you inspect one of the article cards, you can see that we can get the address and the article’s title through the link’s attributes. Also, the meta description is under a <div> tag with a specific class name.

Of course, there are many ways to perform this search. The one we’ll use will consist of looking for all the <div> tags with the td_module_10 class name and then iterating through each one of them to extract the <a> tags and the inner <div> tags with the td-excerpt class name.

article_cards = parsed_page.xpath("//div[contains(@class, 'td_module_10')]")

article_cards.each do |card|
    title = card.xpath("div[@class='td-module-thumb']/a/@title")
    link = card.xpath("div[@class='td-module-thumb']/a/@href")
    meta = card.xpath("div[@class='item-details']/div[@class='td-excerpt']")
end

Yes, as you may have already guessed, an XPath expression is what does the trick because we are looking for HTML elements by their class names and their ascendants.

This type of extraction can be beneficial when the data should pass to another application, an article aggregator in our case. So, to do that, we need to export the parsed data to an external file.

We will create a CSV file, as it can be easily read by another application and opened with Excel for further processing. First, just one more import:

Then we will create the CSV in “append” mode, and wrap the previous code, so now our scraper will look like this:

CSV.open("articles.csv", "a+") do |csv|
    csv << ["title", "link", "meta"]

    article_cards = parsed_page.xpath("//div[contains(@class, 'td_module_10')]")
    article_cards.each do |card|

        title = card.xpath("div[@class='td-module-thumb']/a/@title")
        link = card.xpath("div[@class='td-module-thumb']/a/@href")
        meta = card.xpath("div[@class='item-details']/div[@class='td-excerpt']")

        csv << [title.first.value, link.first.value, meta.first.text.strip]
    end
end

Cool, that’s all! Now we can see all the parsed data in a clean, non-scary, and easy to forward way.

We have now completed our tutorial. Congrats! Hopefully, this article gave you a lot of information about web scraping and helped you better understand it.

Obviously, this technology can do a lot more than power article aggregators. The key is finding the correct data and analyzing it to come up with new possibilities.

However, as I stated at the start of the article, web scrapers face numerous challenges. In addition to boosting your business, it is a great learning opportunity for developers to solve issues using their own web scrapers. Nevertheless, you may want to cut costs if you need to complete a project (time, money, people).

An API dedicated to solving these problems will always be easier to use. Even when there may be obstacles such as Javascript rendering, proxies, CAPTHAs, and other blocking factors, WebScrapingAPI overcomes all of them and provides a customizable experience. If you are still unsure, why not try out the free trial option?

Web Scraping in Ruby: The Ultimate Tutorial

When you started learning the basics of web scraping, did you have an “oh damn, I’ll need proxies for this” moment? Well, prepare to have that kind of epiphany all over again because rotating proxies are the next level in web scraping functionality.

Don’t believe me? Let’s look at the benefits proxies bring if you don’t rotate them and then at the extra advantages you get through rotation. Here’s what non-rotating IPs bring:

You can retry scraping a page/website after the initial IP is blocked.

You can manually switch addresses to get past websites that use geolocation to restrict content.

You don’t give away your actual location.

By adding a request delay, you can extract data from websites with anti-scraping countermeasures.

Those are definitely good things to have, but let’s see what happens when you add a new spin (pun intended):

The scraper continually sends requests until it manages to get the information you need.

You can easily access geo-restricted content from any country where you have a proxy.

Websites don’t even know that bots are visiting them.

You can get data from any website without having to slow down requests.

That’s just a quick rundown of the differences, but it’s pretty impressive, no? So let’s dive in and learn how these benefits come to be!

By rotating proxies, we mean the process of automatically switching from one IP to another with each request sent. In theory, this can be done manually, but it would mean that you couldn’t queue a list of URLs for scraping. Instead, you’d have to assign a proxy, send a request, assign another proxy, send another request, and so on. I hope you see how inefficient that would be.

For automatic proxy rotation, you’ll need a new intermediary, a proxy to your proxy, if you will. This new middle-man is a server that has access to your whole proxy pool. So instead of you searching through the IP list, the server does that for you. All you have to do is give it the command.

Here’s a play-by-play of the scraping process with a program that rotates proxies:

The user sends a request to the proxy management server.

The server picks a proxy at random (unless otherwise specified) and sends the request to it.

The proxy then sends the request to the final target, the webpage with data to extract.

The response takes the same route back to the user.

These four steps are repeated until the user has all the information he or she wants. Every request goes through a new proxy, simulating the actions of many different users connecting to a website.

So, in essence, rotating proxies take no extra work on your part. That’s the beauty of it. It’s more automation for a tool that’s already meant to gather tonnes of information without effort from the user.

Of course, for IP rotation, you need a proxy pool first and foremost. However, since not all proxies are the same, you need to gather the right IPs, too. Here are the two leading contenders:

There are several ways to classify proxies. You can go by anonymity, access, or origin. That latter factor is the most important for web scraping projects. In that sense, proxies are generally either datacenter or residential. Let’s look at each.

Data centers are named so because of their unrivaled capacity to store and share information online. The term “stored in the cloud” is just a more concise way of saying “stored in a data center, and you can access it as long as you have Internet.” The shorter version rolls off the tongue a lot better.

Data centers are essentially huge collections of interconnected servers with a colossal combined storage capacity and the infrastructure to keep them running. Datacenter proxies, as you may have guessed, are hosted by these structures. To get proxies, you have to create a virtual server, install an operating system on it, then install specialized software to set up IPs as proxies. 

The key to efficient datacenter proxies is finding the right balance of servers and IPs. A server can hold multiple IPs, but each one creates more overhead. At some point, you'll get diminishing returns from the server and have to make a new one. Server and IP juggling is a lot of work, so most developers prefer to just rent or buy these proxies from specialized companies.

The IPs aren’t associated with an Internet Service Provider. Instead, you’re dealing with the data center owners or a third party that uses storage space to set up proxies and distribute them to customers.

They’re a popular choice for web scrapers because:

They’re inexpensive compared to residential IPs due to the way they’re set up en masse;

Their superior speed means that you can gather more data in less time;

They’re very reliable thanks to the data centers’ solid infrastructure;

It’s easy to buy or rent in bulk from the same server farm.

Of course, everything has its downsides. For datacenter proxies, those are:

Their IPs are easier to spot by vigilant websites;

All proxies from the same data center share a subnet identifier, making them more prone to blanket banning.

It’s more challenging to build a proxy pool with IPs from every country as data centers are few in number.

By using rotating datacenter proxies, you can successfully access and scrape most websites. Since every new request comes from a different IP, it’s relatively difficult to track and block the scraper.

Advanced and popular websites are a different story. Amazon, Google, and other significant names face scrapers and other bots daily. As a result, it’s much more likely to be noticed. Moreover, datacenter proxies always run the risk of already being banned since all IPs from the same data center share a subnet. 

What matters most is who you choose as your proxy service provider and how they manage their IPs. For example, WebScrapingAPI datacenter proxies are private and guarantee little to no blacklisting.

If data centers are the new and high-tech alternative, residential IPs are the tried and true option. Residential IPs are real devices connected to the web through an ISP. These proxies are virtually indistinguishable from normal users because, in a sense, they are just that.

Setting up a datacenter proxy pool takes some CS knowledge, money, and the right software. On the other hand, gathering residential IPs is a lot more complicated. First, you’d have to convince people to let you install specialized software on their device that gives you access from afar. Then, they’d have to always leave the machines on. Otherwise, the proxy would sometimes be unusable. 

Due to the inconvenience and the needed degree of trust, it’s supremely easier for developers to rent residential proxies from dedicated service providers.

Compared to datacenter proxies, residential IPs have a few key advantages:

Residential IPs are backed by ISPs and look like regular visitors while surfing the web, making the scraper harder to detect;

Each IP is unique, so even if one proxy is identified and blocked, all others are still usable;

It’s easier to prepare a proxy pool from a wide range of locations, assuring easier access to geo-restricted content.

Despite these substantial benefits which make residential proxies the best of the best in terms of effectiveness, they also have a few drawbacks:

Due to location and the device’s Internet connection, request speed varies from IP to IP and are generally slower than datacenter proxies;

You have to carefully choose a residential proxy provider, making sure that they offer a good number of IPs and that they have proxies in the countries you require for your project.

Residential proxies combined with an IP rotation system and a script that cycles request headers (especially user-agent) provide the best cover. With those, your web scraper can gather data without running into barriers, such as IP blocks or CAPTCHAs.

Due to their authenticity, residential proxies are often used for scraping more complex websites, like search engines, large eCommerce sites, or social media platforms. However, if you log in on these sites, turn proxy rotation off so that all requests come from the same IP. Otherwise, it will look like the same user is sending requests from all over the globe in a matter of seconds, proving that it’s a bot.

Why you should use rotating proxies for web scraping

Some websites tend to be popular targets for web scrapers. Google, Amazon, and Facebook come to mind. These platforms expect bots, so they implement methods to both slow down and stop them. With the increase in popularity that web scrapers are experiencing, these countermeasures are being used by more and more websites.

In this context, it’s becoming more and more vital that you rotate your IPs. Otherwise, you risk running into constant IP blocks, captchas, and your proxy pool slowly becomes ineffectual.

Request throttling is a technique through which websites decrease the speed at which bots can navigate. Simply put, it limits the number of requests that a visitor can make in a specified amount of time. 

Once the limit is reached, the bot is redirected to a CAPTCHA page. If your web scraping tool doesn’t have CAPTCHA solving capabilities or they fail to work, the IP can’t continue scraping on that website.

The key takeaway here is that websites monitor the number of requests sent from each IP. If your requests are sent from multiple addresses, the load is shared between them. Here’s an example:

If a website is set up so that after 10 requests, the 11th will trigger a CAPTCHA, that means you can scrape 10 pages before you have to stop and manually change the IP. If you’re collecting price intelligence, that might be a drop in the bucket compared to the total number of pages you want to scrape.

By rotating your proxies, you eliminate the need to intervene manually. You’ll be able to scrape ten times the number of proxies you have. Then, the only limit is the proxy pool’s size, each IP being used to its full potential.

The best part is that request throttling limits the number of visits over a set period. With a large enough proxy pool, the timer for the proxies you already used will expire before you have to call on them again, giving you virtually unlimited scraping capabilities.

Evenly distribute requests across the proxy pool

By the same logic as the previous point, proxy rotation ensures that you’re not overusing the same IPs while others sit idly. In the most basic terms, you’re using your resources as efficiently as possible. The result — you may get the same results with a smaller number of IPs.

Without proxy rotation, you’re using the same IP until it gets blocked while all other proxies are gathering dust. When one stops working, you go onto the next. This method makes proxies seem like a finite resource that’s used up to collect data. That’s not necessary.

With good location spread and data center/residential distribution, a decent proxy pool can go on indefinitely if you rotate IPs. All you need to make sure is that no one proxy stands out and risks getting blocked.

In the same vein, proxy rotation can exponentially speed up the scraping process. The key here is concurrent requests. Most web scraping APIs can send several requests simultaneously, the number depending on your chosen package.

Sending 50 requests at the same time towards the same website is bound to raise some red flags. In fact, it’s so far removed from normal user behavior that you’ll most likely be blocked immediately. You can probably guess where this is going. Rotating your proxies is a must, as it lets you push the scraper to its limits without getting blocked.

While on the subject of being blocked, one of the most significant advantages of rotating proxies is the anonymity it ensures. Let’s elaborate!

Websites are not too keen on being visited by bots. They may be trying to stop you from collecting data or just trying to ensure that no malicious program is trying to crash their servers. That’s why they implement several countermeasures to detect and block bots. Because web scrapers surf the Internet faster than regular users, they stand out. Think back to the example with 50 concurrent requests. According to ContentSquare’s 2020 Digital Experience Benchmark, the average user spends 62 seconds on a single page. The difference in behavior is plain to see.

With rotating proxies, you constantly swap between IPs and avoid sending numerous requests from a single address. The result — the website sees the traffic you generate as a group of separate visitors with no connection between them.

A proxy rotator shouldn’t just switch IPs at random. Sure, sometimes that’s enough, but a more advanced functionality you should look for is the option to rotate proxies from a specific region.

This feature is critical when you’re trying to collect information that pertains to a single region. Depending on the geographical source of a request, some websites may:

Display data specific to the IP’s origin

Restrict access because the request is coming from a blocked country 

So, by rotating a diverse proxy pool, you can both mitigate the downsides while capitalizing on the benefits. Applied to the previous points, that means:

Gain a better overview of foreign markets by obtaining data specific to several countries;

Extract information that would otherwise be inaccessible due to certain countries being blocked from the website.

It’s important to stay vigilant and notice when websites customize their content based on the request’s origin. These sites have the potential to offer great insights into a country’s data. Still, if you don’t know you’re receiving custom information, you might end up with significant inaccuracies in your data.

Like web scrapers, you can build, manage and use a proxy rotator all on your own. For that, you’ll need programming knowledge (Python is ideal as it has many valuable frameworks and an active community), some general CS knowledge, a list of proxies, and a whole lot of patience.

The most basic form would be a script that receives a variable containing your proxy list and assigns random IPs for each request. For example, you could use the random.sample() function to pick one IP at complete random each time, but that means that the same proxy might be used several consecutive times. In that case, you could make it so that after an IP is used, it’s taken out of the proxy list, so it won’t be used again until all other addresses are used as well.

import random
import requests

proxy_pool = ["191.5.0.79:53281", "202.166.202.29:58794", "51.210.106.217:443", "5103.240.161.109:6666"]
URL = 'https://httpbin.org/get'

while len(proxy_pool) >0:
	random_proxy_list = random.sample(proxy_pool, k=1)
	random_proxy = {
		'http': 'http://' + random_proxy_list[0],
	}
	response = requests.get(URL, proxies=random_proxy)
	print(response.json())
	proxy_pool.remove(random_proxy_list[0])

The code only cycles the proxy pool once and does it for a single URL, but it should illustrate the logic well. I grabbed the IPs from https://free-proxy-list.net/, by the way. Unsurprisingly, they didn’t work.

That’s kind of the problem with building your own rotator, in fact. You’ll still need good dedicated or at least shared IPs. Once you’re at the point of buying proxies, you might as well look for a solution that rotates the IPs for you as well. This way, you don’t spend extra time building it or extra money outsourcing it. Also, you get more goodies like:

A quick option to rotate only IPs from a specific region;

The chance to choose what kinds of proxies to cycle (datacenter or residential; regular or mobile; etc.)

Setting up static IPs for when you’re scraping behind a login screen;

Rotating Proxies: Everything You Need To Know

Long gone are the days when you manually collected and processed the data helping you kickstart your projects. Whether an e-commerce website or a lead generation algorithm, one thing is sure: the data-gathering process was tedious and time-consuming.

In this article, you will learn how Cheerio can come to your help with its extensive features of parsing markup languages, first with some trivial examples and then with a real-life use case.

“But what is Cheerio?” you may ask yourself. Well, trying to clarify a common misconception, I will start with what Cheerio is not: a browser.

The confusion may start from the fact that Cheerio parses documents written in a markup language and then offers an API to help you manipulate the resulting data structure. But unlike a browser, Cheerio will not visually render the document, load CSS files or execute Javascript.

So basically what Cheerio does is to receive an HTML or XML input, parse the string and return the API. This makes it incredibly fast and easy to use, hence its popularity among Node.js developers.

Now, let’s see some hands-on examples of what Cheerio can do. First things first, you need to make sure your environment is all set up.

It goes without saying that you must have Node.js installed on your machine. If you don’t, simply follow the instructions from their official website, according to your operating system.

Make sure to download the Long Term Support version (LTS) and don’t forget about the Node.js Package Manager (NPM). You can run these commands to make sure the installation went okay:

Now, regarding the IDE debate: for this tutorial, I will be using Visual Studio Code, as it is pretty flexible and easy to use, but you are welcome to use any IDE you prefer.

Just create a folder that will hold your little project and open a terminal. Run the following command to set up a Node.js project:

This will create a default version of the package.json file, which can be modified at any time.

Next step: I will install TypeScript along with the type definitions for Node.js:

npm install typescript @types/node -save-dev

I chose TypeScript in this tutorial for its optional static typing to JavaScript objects, which makes the code more bulletproof when it comes to type errors. 

This is the same advantage that steadily increased its popularity among the JavaScript community, according to a recent CircleCI survey regarding the most popular programming languages.

To verify the correct installation of the previous command, you can run:

Now I will create the tsconfig.json configuration file at the root of the project directory, which should define the compiler options. If you want a better understanding of this file and its properties, the official TypeScript documentation got your back. 

If not, then simply copy and paste the following:

Almost done! Now you have to install Cheerio (obviously):

Last but not least, create the src directory which will hold the code files. And speaking of the code file, create and place the index.ts file in the src directory.

For the moment I will illustrate some basic Cheerio features using a static HTML document. Simply copy and paste the content below into a new static.html file within your project:

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Page Name - Static HTML Example</title>
</head>
<body>
    <div class="page-heading">
        <h1>Page Heading</h1>
    </div>
    <div class="page-container">
        <div class="page-content">
            <ul>
                <li>
                    <a href="#">Item 1</a>
                      <p class="price">$100</p>
                      <p class="stock">12</p>
                </li>
                <li>
                    <a href="#">Item 2</a>
                    <p class="price">$200</p>
                    <p class="stock">422</p>
                </li>
                <li>
                    <a href="#">Item 3</a>
                    <p class="price">$150</p>
                    <p class="stock">5</p>
                </li>
            </ul>
        </div>
    </div>
    <footer class="page-footer">
        <p>Last Updated: Friday, September 23, 2022</p>
    </footer>
</body>
</html>

Next, you have to serve the HTML file as input to Cheerio, which will then return the resulting API:

import fs from 'fs'
import * as cheerio from 'cheerio'

const staticHTML = fs.readFileSync('static.html')
const $ = cheerio.load(staticHTML)

If you receive an error at this step, make sure that the input file contains a valid HTML document, as from Cheerio version 1.0.0 this criterion is verified as well.

Now you can start experimenting with what Cheerio has to offer. The NPM package is well-known for its jQuery-like syntax and the use of CSS selectors to extract the nodes you are looking for. You can check out their official documentation to get a better idea.

Let’s say you want to extract the page title:

const title = $('title').text()
console.log("Static HTML page title:", title)

We should test this, right? You are using Typescript, so you have to compile the code, which will create the dist directory, and then execute the associated index.js file. For simplicity, I will define the following script in the package.json file:

"scripts": {
    "test": "npx tsc && node dist/index.js",
}

and the script will handle both steps that I just described.

Okay, but what if the selector matches more than one HTML element? Let’s try extracting the name and the stock value of the items presented in the unordered list:

const itemStocks = {}
$('li').each((index, element) => {
    const name = $(element).find('a').text()
    const stock = $(element).find('p.stock').text()
    itemStocks[name] = stock
})
console.log("All items stock:", itemStocks)

Now run the shortcut script again, and the output from your terminal should look like this:

Static HTML page title: Page Name - Static HTML Example
All items stock: { 'Item 1': '12', 'Item 2': '422', 'Item 3': '5' }

So that was basically the tip of the iceberg. Cheerio is also capable of parsing XML documents, extracting the style of the HTML elements and even altering the nodes’ attributes.

But how can Cheerio help in a real use case?

Let’s say that we want to gather some data to train a Machine Learning model for a future bigger project. Usually, you would search Google for some training files and download them, or use the website’s API.

But what do you do when you can’t find some relevant files or the website you are looking at does not provide an API, has a rate-limiting over the data or does not offer the whole data that you are seeing on a page?

Raluca Penciuc