Long gone are the days when you manually collected and processed the data helping you kickstart your projects. Whether an e-commerce website or a lead generation algorithm, one thing is sure: the data-gathering process was tedious and time-consuming.

In this article, you will learn how Cheerio can come to your help with its extensive features of parsing markup languages, first with some trivial examples and then with a real-life use case.

“But what is Cheerio?” you may ask yourself. Well, trying to clarify a common misconception, I will start with what Cheerio is not: a browser.

The confusion may start from the fact that Cheerio parses documents written in a markup language and then offers an API to help you manipulate the resulting data structure. But unlike a browser, Cheerio will not visually render the document, load CSS files or execute Javascript.

So basically what Cheerio does is to receive an HTML or XML input, parse the string and return the API. This makes it incredibly fast and easy to use, hence its popularity among Node.js developers.

Now, let’s see some hands-on examples of what Cheerio can do. First things first, you need to make sure your environment is all set up.

It goes without saying that you must have Node.js installed on your machine. If you don’t, simply follow the instructions from their official website, according to your operating system.

Make sure to download the Long Term Support version (LTS) and don’t forget about the Node.js Package Manager (NPM). You can run these commands to make sure the installation went okay:

Now, regarding the IDE debate: for this tutorial, I will be using Visual Studio Code, as it is pretty flexible and easy to use, but you are welcome to use any IDE you prefer.

Just create a folder that will hold your little project and open a terminal. Run the following command to set up a Node.js project:

This will create a default version of the package.json file, which can be modified at any time.

Next step: I will install TypeScript along with the type definitions for Node.js:

npm install typescript @types/node -save-dev

I chose TypeScript in this tutorial for its optional static typing to JavaScript objects, which makes the code more bulletproof when it comes to type errors. 

This is the same advantage that steadily increased its popularity among the JavaScript community, according to a recent CircleCI survey regarding the most popular programming languages.

To verify the correct installation of the previous command, you can run:

Now I will create the tsconfig.json configuration file at the root of the project directory, which should define the compiler options. If you want a better understanding of this file and its properties, the official TypeScript documentation got your back. 

If not, then simply copy and paste the following:

Almost done! Now you have to install Cheerio (obviously):

Last but not least, create the src directory which will hold the code files. And speaking of the code file, create and place the index.ts file in the src directory.

For the moment I will illustrate some basic Cheerio features using a static HTML document. Simply copy and paste the content below into a new static.html file within your project:

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Page Name - Static HTML Example</title>
</head>
<body>
    <div class="page-heading">
        <h1>Page Heading</h1>
    </div>
    <div class="page-container">
        <div class="page-content">
            <ul>
                <li>
                    <a href="#">Item 1</a>
                      <p class="price">$100</p>
                      <p class="stock">12</p>
                </li>
                <li>
                    <a href="#">Item 2</a>
                    <p class="price">$200</p>
                    <p class="stock">422</p>
                </li>
                <li>
                    <a href="#">Item 3</a>
                    <p class="price">$150</p>
                    <p class="stock">5</p>
                </li>
            </ul>
        </div>
    </div>
    <footer class="page-footer">
        <p>Last Updated: Friday, September 23, 2022</p>
    </footer>
</body>
</html>

Next, you have to serve the HTML file as input to Cheerio, which will then return the resulting API:

import fs from 'fs'
import * as cheerio from 'cheerio'

const staticHTML = fs.readFileSync('static.html')
const $ = cheerio.load(staticHTML)

If you receive an error at this step, make sure that the input file contains a valid HTML document, as from Cheerio version 1.0.0 this criterion is verified as well.

Now you can start experimenting with what Cheerio has to offer. The NPM package is well-known for its jQuery-like syntax and the use of CSS selectors to extract the nodes you are looking for. You can check out their official documentation to get a better idea.

Let’s say you want to extract the page title:

const title = $('title').text()
console.log("Static HTML page title:", title)

We should test this, right? You are using Typescript, so you have to compile the code, which will create the dist directory, and then execute the associated index.js file. For simplicity, I will define the following script in the package.json file:

"scripts": {
    "test": "npx tsc && node dist/index.js",
}

and the script will handle both steps that I just described.

Okay, but what if the selector matches more than one HTML element? Let’s try extracting the name and the stock value of the items presented in the unordered list:

const itemStocks = {}
$('li').each((index, element) => {
    const name = $(element).find('a').text()
    const stock = $(element).find('p.stock').text()
    itemStocks[name] = stock
})
console.log("All items stock:", itemStocks)

Now run the shortcut script again, and the output from your terminal should look like this:

Static HTML page title: Page Name - Static HTML Example
All items stock: { 'Item 1': '12', 'Item 2': '422', 'Item 3': '5' }

So that was basically the tip of the iceberg. Cheerio is also capable of parsing XML documents, extracting the style of the HTML elements and even altering the nodes’ attributes.

But how can Cheerio help in a real use case?

Let’s say that we want to gather some data to train a Machine Learning model for a future bigger project. Usually, you would search Google for some training files and download them, or use the website’s API.

But what do you do when you can’t find some relevant files or the website you are looking at does not provide an API, has a rate-limiting over the data or does not offer the whole data that you are seeing on a page?

Well, this is where web scraping comes in handy. If you are curious about more practical use cases of web scraping, you can check out this well-written article from our blog.

Back to our sheep, for the sake of the example let’s consider that we are in this precise situation: wanting data, and data is nowhere to be found. Remember that Cheerio does not handle the HTML extraction nor the CSS loading or JS execution.

Thus, in our tutorial, I am using Puppeteer to navigate to the website, grab the HTML and save it to a file. Then I will repeat the process from the previous section.

To be more specific, I want to gather some public opinions from Reddit on a popular drum module and centralise the data in a single file that will be further fed to a potential ML training model. What happens next can also vary: sentiment analysis, market research and the list can go on.

Let’s see how this use case will look put into code. First, you need to install the Puppeteer NPM package:

I will also create a new file reddit.ts, to keep the project more organised, and define a new script in the package.json file:

"scripts": {
    "test": "npx tsc && node dist/index.js",
    "parse": "npx tsc && node dist/reddit.js"
},

To get the HTML document, I will define a function looking like this:

import fs from 'fs'
import puppeteer from 'puppeteer'
import * as cheerio from 'cheerio'

async function getContent(url: string): Promise<void> {

    // Open the browser and a new tab
    const browser = await puppeteer.launch()
    const page = await browser.newPage()

    // Navigate to the URL and write the content to file
    await page.goto(url)
    const pageContent = await page.content()
    fs.writeFileSync("reddit.html", pageContent)

    // Close the browser
    await browser.close()
    console.log("Got the HTML. Check the reddit.html file.")
}

To quickly test this, add an entry point in your code and call the function:

async function main() {


    const targetURL = 'https://old.reddit.com/r/Drumming/comments/r3tidc/yamaha_ead10/'
    await getContent(targetURL)
}

main()
    .then(() => {console.log("All done!")})
    .catch(e => {console.log("Unexpected error occurred:", e.message)})

The reddit.html file should appear in your project tree, which will contain the HTML document we want.

Now, a bit of a more challenging part: you have to identify the nodes that are of interest for our use case. Go back to your browser (the real one) and navigate to the target URL. Go with your mouse cursor over the comments section, right-click and then choose the “Inspect” option.

The Developer Tools tab will open, showing you the exact same HTML document you previously saved on your machine.

To extract only the comments, you have to identify the selectors unique to this section of the page. You can notice that the whole list of comments is within a div container with a sitetable nestedlisting class.

Going deeper, each individual comment has a form element as a parent, with the usertext warn-on-unload class. Then, at the bottom, you can see that every comment’s text is divided between multiple p elements.

function parseComments(): void {

    // Load the HTML document
    const staticHTML = fs.readFileSync('reddit.html')
    const $ = cheerio.load(staticHTML)

    // Get the comments section
    const commentsSection = $('div.sitetable.nestedlisting')

    // Iterate each comment
    const comments = []


    $(commentsSection).find('form.usertext.warn-on-unload').each((index, comment) => {
        let commentText = ""

          // Iterate each comment section and concatenate them
          $(comment).find('p').each((index, piece) => {
            commentText += $(piece).text() + '\n'
          })

          comments.push(commentText)
    })

    // Write the results to external file
    fs.writeFileSync("comments.json", JSON.stringify({comments}))
}

Alright, and now let’s update the entry point with the newly defined function and see how this code is working together:

async function main() {

    const targetURL = 'https://old.reddit.com/r/Drumming/comments/r3tidc/yamaha_ead10/'
    await getContent(targetURL)
    parseComments()
}

main()
    .then(() => {console.log("All done. Check the comments.csv file.")})
    .catch(e => {console.log("Unexpected error occurred:", e.message)})

Execute the code with the script defined before:

It will take around 5 to 10 seconds for the headless browser to open and navigate to our target URL. If you are more curious about it, you can add timestamps at the beginning and the end of each of our functions to really see how fast Cheerio is.

The comments.json file should be our final result:

This use case can be easily extended to parsing the number of upvotes and downvotes for every comment, or to get the comments’ nested replies. The possibilities are endless.

Thank you for making it to the end of this tutorial. I hope you grasped how indispensable Cheerio is to the process of data extraction and how to quickly integrate it into your next scraping project.

We are also using Cheerio in our product, WebScrapingAPI. If you ever find yourself tangled by the many challenges encountered in web scraping (IP blocks, bot detection, etc.), consider giving it a try.

Scraping with Cheerio: How to Easily Collect Data from Web Pages

Nowadays, web scraping has become a really challenging task. If you ever built a web scraper with a headless browser, most definitely you came across some anti-bot systems. And bypassing cloudflare, datadome, or any other anti-bot provider for that matter, is no easy task. You have to first think of evasion strategies and then implement them in production. And even so, there might be scenarios that you never accounted for. 

Yet without at least a minimal implementation of these evasion techniques, your chances of getting caught and blocked are very high. That is why, the topic for today’s article is how to bypass cloudflare detection. And most of the techniques we’re going to discuss apply to other detection vendors as well. As a reference, we’ll focus on Selenium to see if we can make it stealthy. Throughout  the article, among others, I’ll discuss:

How does Cloudflare detect headless browsers?

Cloudflare is a tech company with a giant network. They focus on services like CDN, DNS, and various online security systems. Their Web Application Firewall is usually designed to protect against attacks such as DDoS or cross site scripting. In recent years, Cloudflare added and other providers in the field introduced fingerprinting systems, capable of detecting headless browsers. As you might guess, one of the first affected by these techniques is Selenium. And since the web scraping industry relies heavily on this technology, scrapers are directly affected as well. 

Before moving forward to anti-bot techniques, I think it is important to discuss how cloudflare detects Selenium. Well, the system can be very complex. For example, there are properties in a browser that a web driver lacks. The `navigator` interface in a browser even has a property called `webdriver` that indicates if a browser is controlled by automation. And that is an instant giveaway. If you want to experiment with it:

Type the following command: `navigator.webdriver`

In your case, it should return `false`. But if you try it with Puppeteer or Selenium, you will get `true`. If you’re wondering how Cloudflare leverages this to detect bots, well it’s pretty simple. All they need to do is inject a script like the one below on their partner's website:

// detection-script.js

const webdriver = navigator.webdriver

// If webdriver returns true, display a reCaptcha

// In this example, I am transferring the user to a Cloudflare challenge page.

// But you get the idea

if ( webdriver ) location.replace('https://cloudflarechallenge.com')

Of course, in real life, there are many more levels of detection these providers use. Even the size of the screen, the keyboard layout, or the plugins used by the browsers are used to specifically fingerprint a browser. If you’re interested in how browserless detection works, check out my simple service worker test. And that is just if you stick to the browser. You can also detect bot activity by looking into the IP address from where the request originates. For example, if you’re using datacenter IPs, your chances of getting blocked increase with every request. That is why it’s recommended to use residential or ISP proxies when you’re building a web scraper.

Fortunately, the web scraping community is really active. And because there is such demand to bypass Cloudflare and other anti-bot providers, there are open source solutions in that area. Great things can be achieved when programming communities work together! Moving forward, I suggest we follow these steps:

Run some tests to see if default Selenium can bypass Cloudflare 

Add some extra evasions to make our scripts stealthier

So let’s get started with our first step:

#1: Can default Selenium bypass Cloudflare?

I am not the type to make assumptions. And that’s especially because we don’t know for sure how Cloudflare systems work. They use all sorts of obfuscation with their code, which makes it harder to reverse engineer. That’s why, throughout my experience as a developer, I learned that testing is the best way to understand how a system works. So let us build a basic scraper and see how it behaves on a real target, protected by Cloudflare.

With Python, it’s best we isolate our projects inside one single directory. So let’s create a new folder, open a terminal window and navigate to it:

# Create a new virtual environment and activate it

~ » python3 -m venv env && source env/bin/activate

# Install dependencies

~ » python3 -m pip install selenium

# Create a new .py file and open the project inside your IDE

~ » touch app.py && code .

1.2. Build a simple web scraper with Selenium

Now that you have successfully set up your project, it’s time to add some code. We’re not going to build anything fancy here. We just need this script for testing purposes. If you want to learn about advanced scraping, check out this [LINK] tutorial on Pyppeteer.

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

# Set Chrome to open in headless mode

options = Options()

options.headless = True

# Create a new Chrome instance and navigate to target

driver = webdriver.Chrome(options=options)

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('screenshot.png')

# Close browser

driver.quit()

Now check out the screenshot. Here’s what I got:

I think we’re safe to conclude that the test failed. The targeted website is protected by Cloudflare and as you can see, we’re getting blocked. So by default, Selenium is not able to bypass Cloudflare. I am not going to go in depth and check with other bot detection providers. If you want to test further, here are some targets and their vendors:

#2: Can stealthy selenium bypass Cloudflare?

First of all, let me clarify the terms. By stealth Selenium I mean a version of Selenium that can go undetected and bypass Cloudflare. I am not referring to any specific stealthiness technique. There are a couple of ways you can implement evasion techniques into Selenium. There are packages that handle it, or you can use the `execute_cdp_cmd` to interact directly with the Chrome API. The latter allows you more control but requires more work. Here is an example of how you could use it to change the user agent’s value:

driver.execute_cdp_cmd('Emulation.setUserAgentOverride', {

               "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win32; x86) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36",

               "platform": "Win32",

               "acceptLanguage":"ro-RO"

       })

But you’d have to go through CDP and identify the APIs that allow you to make all the necessary changes. So for the time being, let’s test with some packages.

There are at least two packages you can use to make Selenium stealthy. Up to this point though, none of them is guaranteed to bypass Cloudflare. Yet again, we need to test and see if either of them works. First of all, let’s look into `selenium-stealth`. This package is a wrapper around `puppeteer-extra-plugin-stealth`, making it possible to use Puppeteer’s evasions with Python’s Selenium. To use it, you have to install it first. Open a terminal window and enter this command:

# Install selenium-stealth

~ » python3 -m pip install selenium-stealth

You’re all set now. We can use it to make our previous scraper stealthier:

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

from selenium_stealth import stealth

import time

# Set Chrome to open in headless mode

options = Options()

options.headless = True

# Create a new Chrome instance

driver = webdriver.Chrome(options=options)

# Apply stealth to your webdriver

stealth(driver,

   languages=["en-US", "en"],

   vendor="Google Inc.",

   platform="Win32",

   webgl_vendor="Intel Inc.",

   renderer="Intel Iris OpenGL Engine",

   fix_hairline=True,

)

# Navigate to target

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('stealth.png')

# Close browser

driver.quit()

Running the script returned some different outcomes for me this time, as opposed to the default Selenium settings:

The second option you may use is `undetected_chromedriver`. This one is described as an ‘optimized Selenium chromedriver. Let’s test it out:

# Install undetected_chromedriver

~ » python3 -m pip install undetected_chromedriver

The code is very similar to our default script. The main difference stands in the package name. Here is a basic scraper with `undetected_chromedriver` and let’s see if it can bypass Cloudflare:

import undetected_chromedriver as uc

import time

# Set Chrome to open in headless mode

options = uc.ChromeOptions()

options.headless = True

# Create a new Chrome instance and maximize the window

driver = uc.Chrome(options=options, executable_path='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome')

driver.maximize_window()

# Navigate to target

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('stealth-uc.png')

# Close browser

driver.quit()

Yet again, running the script turns out well for me. It seems that at least these two packages can successfully bypass Cloudflare protection. At least in the short term. Truth be told, chances are, if you use these scripts extensively, Cloudflare will catch up on your IP address and block it. So let me introduce you to a third option: Web Scraping API. 

Web Scraping API has this amazing feature called the Proxy Mode. You can read more about it here. But what I want to note here is that our Proxy Mode can successfully be integrated with Selenium. This way, you get access to all the evasion features we’ve implemented. And let me tell you that we have a dedicated team working on custom evasion techniques. In technical terms, we’re handling IP rotations, we’re using various proxies, we’re solving captchas and we’re using Chrome’s API to continuously change our fingerprint. In non-technical terms, this translates to less hassle on your side and a greater success rate. You basically get the stealthiest version of Selenium there is. And here’s how it’s done:

# Install selenium-wire

~ » python3 -m pip install selenium-wire

We’re using `selenium-wire` in order to use Selenium with a proxy. Now here’s the script:

from seleniumwire import webdriver

import time

# Method to encode parameters

def get_params(object):

   params = ''

   for key,value in object.items():

       if list(object).index(key) < len(object) - 1:

           params += f"{key}={value}."

       else:

           params += f"{key}={value}"

   return params

# Your WSA API key

API_KEY = '<YOUR_API_KEY>'

# Default proxy mode parameters

PARAMETERS = {

   "proxy_type":"datacenter",

   "device":"desktop",

   "render_js":1

}

# Set Selenium to use a proxy

options = {

   'proxy': {

       "http": f"http://webscrapingapi.{ get_params(PARAMETERS) }:{ API_KEY }@proxy.webscrapingapi.com:80",

   }

}

# Create a new Chrome instance

driver = webdriver.Chrome(seleniumwire_options=options)

# Navigate to target

driver.get('https://www.httpbin.org/get')

# Retrieve the HTML documeent from the page

html = driver.page_source

print(html)

# Close browser

driver.quit()

If you run this script a couple of times, you’ll see how the IP address changes every time. That’s our IP rotation system. In the background, it also adds evasions techniques. You don’t even need to worry about them. We handle the Cloudflare bypassing part so that you can focus more on parsing the data. 

If you want to build a scraper that can bypass Cloudflare, you need to account for a lot of things. A dedicated team can work 24/7 and there’s still no guarantee the evasions will work every time. That’s because, with every browser version release, there is a chance new features are added to the API. And some of these features can be used to fingerprint and detect bots.

I’d even say that the best browser to bypass Cloudflare and other providers is the one you build yourself. And we built one at Web Scraping API. Now we’re sharing it with you. So enjoy scraping!

Learn How To Bypass Cloudflare Detection With The Best Selenium Browser

Web scraping is the process of extracting data from a website and using it for a variety of purposes. It can be a useful tool for businesses looking to gather information on their competitors, for individuals looking to track changes on a website, or for researchers looking to gather data for a study.

There are many different ways to scrape a website, but one of the most popular methods is using an API. An API, or application programming interface, is a set of rules and protocols that allow different software programs to communicate with each other.

SERP stands for Search Engine Results Page. It is the page that is displayed to a user after they enter a query into a search engine, such as Google, Bing, or Yahoo. The SERP contains a list of results, also known as "search results," that are relevant to the user's query. These results can include web pages, images, videos, and other types of content. The SERP also includes other elements such as sponsored ads, local packs, featured snippets, and more.

SERPs are constantly changing and evolving as search engines continue to improve their algorithms and add new features. The layout, format, and content of SERPs can vary depending on the search engine, the query, and the location of the user. Understanding the SERP and what appears on it is crucial for SEO and SEM professionals, as it helps them to understand how their website is performing in search engine results and what they can do to improve their visibility and performance.

When it comes to scraping Google's search engine results pages (SERPs), there are several APIs available, both free and paid. In this article, we'll take a look at the top 7 Google SERP APIs and compare their pros and cons.

The first option we are going to explore is WebScrapingAPI as it brings the best value for money compared to others in the list.

WebScrapingAPI can be used by developers to scrape the web and has a lot of great features such as residential proxies, JavaScript rendering, CSS extracting, CAPTCHA bypassing, IP rotation, and evasions which gets them rid of the stress of being banned. It is very easy to use and all you need to do is to send your API KEY and a search query in an API call, just like this: https://serpapi.webscrapingapi.com/v1?engine=google&api_key=<YOUR-API-KEY>&q=Steve%20Jobs and that’s it, you are now ready to receive your results back. On top of that, this service offers a lot of dedicated scrapers for Google or Amazon, making it a great option for businesses or individuals who want to affordably scrape data from these websites and use it for different purposes.

When it comes to pricing, WebScrapingAPI has something for everybody with pricing starting from as little as 49$ per month for 100.000 API calls to an Enterprise plan for those who want to scrape the web at a large scale and need more than 10.000.000 API calls. The cost per request may vary depending on whether you use rendering JavaScript or residential proxies from 1 to 25 API credits.

This service also offers you the chance to try all products and features for free for 7 days with no credit card required, thus enabling you to find out how closely it satisfies your needs before you purchase a subscription. In case you encounter any issue with the product, you can easily get in touch with their technical support and someone will get back to you very soon, usually within a couple of hours.

Pros: By far best value for money from this list. This API is feature-rich, has great customer care and it is suitable for every budget.

Cons: Does not provide the response in many different formats such as CSV as some other options from the list, but you can easily convert the data using an JSON to CSV parser.

The second option we are going to explore is Apify which provides a great Google Scraping service.

The API allows users to extract data from Google search results pages in real time and offers a variety of options for customizing the scrape, such as the ability to search for specific keywords, retrieve data on specific SERP elements, and scrape results from multiple countries and languages. Additionally, the Apify platform offers an actor library of pre-made scraping and automation tasks, which can save time and effort for users who need to perform repetitive scraping tasks. The platform also offers a cloud storage for the scraped data which can be easily exported to different formats and a feature called Autoscaled Pool of Headless Browsers that allows to run multiple concurrent scraping tasks. Apify's Google scraping API is a valuable tool for businesses, researchers, and individuals looking to gather data on search trends, track changes in SERPs, and gather data on competitors.

When it comes to pricing, Apify has a pay-as-you-go pricing model, which means that users are charged based on the number of page loads and the amount of data scraped. The exact pricing can vary depending on the specific scraping needs of the user. It's worth mentioning that Apify offers a free trial with a certain amount of free page loads and storage, so you can test the platform and see if it meets your scraping needs before committing to a paid plan.

Pros: Offers a lot of scrapers and flexibility, thus being suitable for a lot of different scraping needs.

Cons: Price and monitoring systems can become confusing, especially at a greater scale.

Another option we are going to explore is SerpAPI which is a service that allows users to scrape Google's search engine results pages (SERPs) in real time. 

SERP API is a powerful web scraping tool specifically designed to scrape Google's search engine results pages (SERPs) in real-time. One of the key features of SERP API is its ability to scrape SERPs from multiple countries and languages, which allows users to gather data on search trends in different regions. Additionally, SERP API offers an extensive list of endpoints, which allows users to retrieve specific types of data from SERPs, such as organic search results, paid search results, featured snippets, local packs, images, videos, etc.

Another great feature of the service is its ability to handle CAPTCHAs, which can be a common issue when scraping Google SERPs. This allows users to continue scraping even when faced with CAPTCHAs, without having to manually solve them.

SerpAPI also offers a user-friendly dashboard that allows users to monitor the usage and track the progress of the scrapes and also offers support for users who need help getting started or troubleshooting issues.

This API stands out by being extremely feature-rich having scrapers for very many different engines such as: 

While most of the engines offer high quality data fastly and provide great documentation which is very easy to follow, at the time of writing this article there are a few engines that do not work as intended, but their team is currently working on fixing these issues, so you can expect that to improve soon. 

When it comes to pricing, it starts from $50 for 50,000 Google searches per month and can get expensive at a larger scale so it is definitely not as cheap as some of the other options from this list.

Pros: It is the option with most features from this list with great documentation making them very easy to use. This service offers a beautiful playground for you to use and test if the product satisfies your needs before committing to a paid subscription.

Cons: The pricing is quite expensive and some engines are currently unreliable.

The fourth option we are going to talk about is SerpWow which provides an easy-to-use Google Search API.

This service is powerful and it enables the developer to retrieve SERP data in real time, which can be used to build SEO tools, track keyword rankings, and analyze search engine performance. SerpWow uses a wide variety of proxies, thus getting the users rid of the worry of being banned and giving the data back in either JSON, CSV, or HTML. On the other hand, at the time of writing this article, they are not as feature-rich as other options from this list and provide scrapers only for a few Google engines. 

When it comes to pricing, it starts from $45 for 5000 Google searches, and as you may already have noticed it is very expensive compared to other options. SerpWow is intended more for Big Data purposes, having the most plans designed for Big Data from this list. In this case, the prices are starting from 500$ per month for 100.000 searches and get as expensive as 6300$ per month for 3.000.000 searches.

Pros: This service offers an easy to use Google Scraping API which returns the data back in various formats.

Cons: It is very expensive and not as feature-rich as other options.

The next option we are going to talk about is OxyLabs which is a professional scraping company that can help businesses and individuals to extract data and insights from the search engines to improve their online performance.

OxyLabs is a very well-known company in the scraping world and they developed a SERP solution that is high quality and reliability that uses proxy rotation in order to avoid getting banned. This service offers a fast SERP API and they provide in-depth tutorials by providing many code samples showing how to integrate their API into your product. 

But, same as the last option on the list it is very expensive with plans starting from 99$ per month for 29.000 requests (limited to 15 results per request) and getting as expensive as 10.000$ per month for more than 10.000.000 searches (no search limits per results).  

Pros: This service offers a fast, reliable and scalable Google SERP solution.

The sixth option we are going to explore is SerpStack, a product developed by Apilayer, a software company based in London. 

While this product was designed first for internal needs, it has grown to become one of the most reliable products in the marketplace. The Serpstack API allows automated retrieval of SERP data from search engines by utilizing a proxy network. With solid documentation, they make scraping Google an easy and straightforward process for developers.

When it comes to pricing it is more affordable than other options from the list, starting from 30$ for 5.000 searches and getting as expensive as 200$ per month for 50.000 searches. It is worth mentioning that if you need scraping at a larger scale this service also offers a custom Enterprise plan designed for such needs. In order to learn more about you need to get in touch with their sales team. 

Pros: This service offers a cheap and reliable Google SERP solution.

Cons: It is not as feature-rich as other candidates.

Lastly, we are going to talk about the SERP API offered by DataForSEO, a provider that offers a variety of tools and services that allow businesses and individuals to gather data on search engine rankings, keywords, backlinks, and other important SEO metrics.

The SERP API from DataForSEO is very flexible and suitable for those who are not sure how many requests are going to consume in a month, having a pay-as-you-go policy. This API is great and lets you scrape Google Search Engine and receive back high-quality data in a reasonable amount of time.

Having a pay-as-you-go payment system makes the price quite confusing and expensive for those who need Google scraping at a larger scale. It is more suitable for those who do not need as many API calls and do not have a constant number of calls per month.

Pros: Flexible and great for digital marketers or SEO consultants.

Cons: Gets expensive when scraping at scale and it is not as feature-rich as other options from the list.

In conclusion, I hope this article gives you an overview of the Google SERP available services in the marketplace. The ability to scrape Google's search engine results pages (SERPs) can be a valuable tool for businesses, researchers, and individuals looking to gather data on search trends, track changes in SERPs, and gather data on competitors. 

There are several options available for scraping Google SERPs and it is extremely important to consider the specific needs of your business or project and to carefully evaluate the features and limitations of each service.

Top 7 Best Google SERP APIs (Free & Paid)

Data parsing is the process of analyzing and transforming structured or unstructured data into a more specific format that various software systems can use and understand.

To make this concept easier to understand, here are some examples of parsers:

CSV parsers: for parsing and converting CSV (Comma Separated Values) files into more usable formats

JSON parsers: for parsing and converting JSON (JavaScript Object Notation) data into more usable formats

Regex parsers: for parsing and extracting specific patterns of text using regular expressions

Compilers: for parsing and converting code written in one programming language into machine-readable code in another programming language

SQL Parsers: for analyzing and interpreting a SQL query, carrying out the command, and returning the results

When it comes to web scraping, data parsing is essential. Websites are made up of HTML, a great markup language for displaying information on a screen but not so great for machines to read.

When we scrape a website, we're grabbing a big string of HTML. To do anything useful with that information, we need to parse it.

This will cover the importance of data parsing in web scraping, as well as the different types of data parsers available, such as HTML parsing libraries for various programming languages, regular expressions, and building your parser.

A good data parser can extract relevant information from an HTML document based on pre-defined rules, regardless of the type of parser used. The parsing process consists of two main steps: lexical analysis and syntactic analysis.

Lexical analysis is the process of analyzing individual words and symbols in a document and breaking them down into smaller, more manageable pieces. 

This involves tokenization, which is the process of breaking a document down into individual tokens, such as keywords, symbols, and numbers.

Let’s take a look at this simple HTML document:

<html>

  <head>

	<title>Scraping</title>

  </head>

  <body>

	<h1>Welcome to my scraping page</h1>

	<p>This is a paragraph.</p>

	<ul>

  	<li>First Scraping Item</li>

  	<li>Second Scraping Item</li>

	</ul>

  </body>

</html>

The lexical analysis process would tokenize this document into individual elements such as:

This way, each element of the HTML document gets split down into smaller, more manageable tokens that can be further analyzed and processed.

Syntactic analysis is the process of analyzing the structure of a document and determining how the individual tokens relate to each other. This involves identifying patterns and structures in the data and using this information to create a tree-like structure called a parse tree.

For example, the <html> tag is the root element, and it contains the <head> and <body> elements. Within the <head> element, there is a <title> element, and within the <body> element, there are <h1>, <p>, and <a> elements. 

By identifying these elements and their relationships, you can construct a parse tree, with the <html> element as the root, <head> and <body> as its children, and so on.

You can use the parse tree to extract specific data from the HTML document, such as the text within the <title> element, or the href attribute of the <a> element.

In this section, we will explore some of the most popular HTML parsing libraries available for different programming languages. These libraries make it easy to extract structured data from an HTML document and can be a great starting point for your web scraping project. 

From Python's Scrapy and BeautifulSoup, to NodeJS's Cheerio and Java's JSoup, we will take a look at each library and provide examples of how to use them. 

Whether you are a beginner or an experienced developer, this section will give you a solid understanding of the options available to you when working with HTML data.

Cheerio is a JavaScript library that allows developers to parse, manipulate, and navigate the DOM of an HTML or XML document, like how jQuery works. This article goes into much more detail about Cheerio and talks about different use cases.

Here’s a simple cheerio implementation:

const cheerio = require('cheerio');

const $ = cheerio.load('<h2 class="title">Hello, World!</h2>');

console.log($('h2').text())

Running this code will return the following output:

Scrapy and BeautifulSoup are libraries for web scraping in Python. 

Scrapy is a powerful web scraping framework that allows you to extract structured data from websites by using selectors or XPath expressions. 

import scrapy

class QuotesSpider(scrapy.Spider):

    name = "quotes"

    def start_requests(self):

        urls = [

            'https://quotes.toscrape.com/page/1/',

            'https://quotes.toscrape.com/page/2/',

        ]

        for url in urls:

            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):

        page = response.url.split("/")[-2]

        filename = f'quotes-{page}.html'

        with open(filename, 'wb') as f:

            f.write(response.body)

        self.log(f'Saved file {filename}')

BeautifulSoup is a library that allows you to parse HTML and XML documents and extract data from them in a way like how a web browser does. 

Here’s a simple BeautifulSoup implementation:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>Scraper</title></head>

<body>

<h1 class="title">Hello, World!</h1>

</body>"""

soup = BeautifulSoup(html_doc, 'html.parser')

print(soup.title.string)

Both of these libraries provide a simple and convenient API for traversing, searching, and modifying the content of web pages, and have a perfect use in web scraping projects.

If your programming language of choice is Java, JSoup is a data parser that provides a convenient API for extracting and manipulating data, using the best of DOM, CSS, and jQuery-like methods. 

This allows you to parse and navigate HTML documents, and change the contents of a document using a simple, intuitive API. The library is a perfect fit for web scraping, web crawling, and data extraction projects.

Here’s a simple implementation of JSoup for extracting the text from the title tag:

import org.jsoup.Jsoup;

import org.jsoup.nodes.Document;

public class JSoupExample {

	public static void main(String[] args) {

    	String html = "<html><head><title>Example Title</title></head>"

            	+ "<body>Hello, World!</body></html>";

    	Document doc = Jsoup.parse(html);

    	// Get the title of the document

    	String title = doc.title();

    	System.out.println("Title: " + title);

	}

}

Nokogiri is a library for Ruby that provides an easy-to-use interface for parsing and searching XML and HTML documents and supports XPath and CSS selectors, making it used for web scraping and data extraction tasks.

For a more comprehensive analysis of the data parser libraries in the Ruby ecosystem, you can read this article.

Use this command to install the nokogiri gem:

The next code sample is a simple nokogiri implementation:

require "nokogiri"

html = "<!DOCTYPE html><html><head><title>Hello, World!</title></head><body>Hello, World!</body></html>"

parsed_data = Nokogiri::HTML.parse(html)

puts parsed_data.title

Regular expressions, also known as regex, are a powerful tool for matching patterns in strings. They are often used for text-processing tasks such as searching, validating, and extracting information from a document.

Recent articles

Sign up for our newsletter

News and updates