Before writing a web scraper, you need to understand the data you are going to scrape and how to access that data. There are many ways in which you can access data in a web page, the most common one is to use CSS Selectors. Another alternative is to use XPath. You can find the XPath Cheat Sheet here.

In the process of parsing an HTML file, the browser creates a data representation in its memory that looks like a tree. This representation is called the DOM (Document Object Model). For every HTML tag, there is a node paired with it in the DOM. A node has properties like name, content, child nodes, styles, events etc. You can find more information about how browser rendering works on this article How browser rendering works — behind the scenes.

When we say that we want to access data from a web page, we only want to iterate through the DOM to a specific set of nodes and extract the content inside them. In this article I will tell you different tips on how to access those nodes quickly using CSS Selectors.

Why are they named CSS (Cascading Style Sheets) Selectors in the first place? 

CSS is used to define the appearance of nodes on a page. With CSS you can write rules about what a node's appearance should be and how it should interact with other nodes. A rule is composed of a selector and a list of styles to overwrite.

So, these selectors are associated with CSS because this is their most common use, but we don’t need to use them with just CSS. With CSS you want to select a node and change its style property. If you think about it, we want to do the same thing: select a node and do something with it, like read its contents or trigger an event. 

It will help you greatly if you visualize the selection happening. Let’s say that you want to scrape all paragraphs from a website. You want to get all nodes that have the name `p`. You can do that by hand. You just need to iterate through every node on the DOM and select only the nodes that have  node.tagName === 'P' (tag names are uppercase).

Here is a short code snippet that you can use:

function scrapeByTagName(node, tagName) {
    if (node === null)
        return;
 
    node.childNodes.forEach(node => {
        //console.log(node.tagName)

        if (node.tagName?.toLowerCase() === tagName.toLowerCase()) {
            console.log(node)
            return
        }

        scrapeByTagName(node, tagName)
    });
}

I made a dummy web page that looks like this:

<!DOCTYPE html>
<html lang="en">

<head>
    <link rel="stylesheet" href="styles.css">
    <script src="script.js"></script>
</head>

<body>
    <div id="wrapper">
        <h1 custom-attr="some data">Some Title</h1>
        <h2 custom-attr="some other data">Some Subtitle</h2>
        <div id="container">
            <p custom-attr>paragraph
                <span> subparagraph</span>
            </p>
            <p id="text">paragraph with id #text</p>
            <p class="bold">paragraph with class .bold</p>
            <p class="text">paragraph with class .text</p>
            <p class="text bold">paragraph with class .text.bold</p>
            <p class="text italic">paragraph with class .text.italic</p>
        </div>
    </div>
</body>

</html>

After I ran the function in the browser console, I got this response:

As you can see, the function logged all the p tags.

To see the browser console you need to open devTools and go to the console tab or press escape. You can open devTools by right clicking an element and choosing inspect from the menu or by using the keyboard shortcut control + shift + i.

We will use two methods: querySelector and querySelectorAll. These methods appear on every object with type Element. The nodes we are trying to scrape have type HTMLElement which inherits from type Element.

querySelector will return the first node that matches the selector. querySelectorAll will return a list with all the nodes that match the selector. To replicate the example previously shown we just need to call querySelectorAll and iterate through the returned list.

document.querySelectorAll('p').forEach(node => console.log(node))

You can see that I used document.querySelectorAll, that’s because document is defined in the window context to be the root of the web page a.k.a. the correspondent of the html tag. You can use querySelector methods with every node, not only with the root node.

To actually scrape something you will need to use a library that can open a browser window and go to an url. Only then your code will execute, in the context of that window. To learn more about how to do this I recommend this article The Ultimate Guide to Web Scraping with JavaScript and Node.Js.

Here at WebScrapingAPI we use Puppeteer. Puppeteer is a library that lets us control instances of headless Chromium browsers. You can use our API to extract data from a website without building a custom scraper. We actually have a parameter named extract_rules that uses CSS Selectors to extract data from a given URL.

This selector specifies all elements from the tree. It does not have much usage but is good to know.

You can get a node with a specific class by using .class. It is mostly used when you have a list of items. Because the items in a list are likely to look the same, they might have the same class. Let’s search for the .text class.

Maybe you want to select the node that has the .bold class.

It looks like there is another element that has the .bold class. You can be more specific with the class selector by using multiple classes concatenated.

Please note that there are no spaces between classes.

document.querySelectorAll('.text .bold').forEach(node => console.log(node))

This query does not return anything from the HTML above, because it looks for an element with class .text that has a child with .bold class (not necessarily a direct child). The query would return the child element if found. 

What if an element doesn't have a class or if the class is used too frequently in the document? You can use the ID attribute to achieve a deeper level of specificity. The drawback of using the id selector is that, in most cases, the id is unique in the HTML page so you can’t get a list of nodes with it.

Every node has a name. It is the exact name of the paired tag in the HTML. You can get all nodes that have a specific name by using their name in the selector.

You might encounter situations in which you would want to select all the nodes that have some specific attribute.

You can also specify the attribute value.

Or even what the attribute value should contain. You can use tilde ~ before the equals sign to define that the attribute value should contain a list of words.

The attribute selector will be the most used if you decide to build a scrapper. It is very powerful and it has a lot more use cases than what  I showed here. You can find more information about how to use the attribute selector here W3 Attribute Selectors.

Select all span nodes that are child of a p node.

Get all div nodes that are direct childs of the body node.

The options of grouping these selectors are endless. Try to copy the HTML code from above and add more nodes to it. Then try different selectors combinations. If you want to learn more about CSS selectors in general, Mozilla offers a fantastic article that explains how CSS selectors work for web development.

If you want to learn something new I advise you to learn how that thing works first. Yes, it is an optional step, but it will give you some information that others don’t have. 

In the field of software development this information will help you search for the right answer to your problem/error. You could take the matter into your own hands and even create a custom solution.

If you really want to understand CSS selectors you need to understand the DOM. It is just a tree (a connected acyclic undirected graph) with nodes that have a name and some attributes. That’s it. When you write a selector, you just write a string that is parsed and used to query the DOM.

CSS Selectors Cheat Sheet - How to scrape the web tips and tricks

Nowadays, web scraping has become a really challenging task. If you ever built a web scraper with a headless browser, most definitely you came across some anti-bot systems. And bypassing cloudflare, datadome, or any other anti-bot provider for that matter, is no easy task. You have to first think of evasion strategies and then implement them in production. And even so, there might be scenarios that you never accounted for. 

Yet without at least a minimal implementation of these evasion techniques, your chances of getting caught and blocked are very high. That is why, the topic for today’s article is how to bypass cloudflare detection. And most of the techniques we’re going to discuss apply to other detection vendors as well. As a reference, we’ll focus on Selenium to see if we can make it stealthy. Throughout  the article, among others, I’ll discuss:

How does Cloudflare detect headless browsers?

Cloudflare is a tech company with a giant network. They focus on services like CDN, DNS, and various online security systems. Their Web Application Firewall is usually designed to protect against attacks such as DDoS or cross site scripting. In recent years, Cloudflare added and other providers in the field introduced fingerprinting systems, capable of detecting headless browsers. As you might guess, one of the first affected by these techniques is Selenium. And since the web scraping industry relies heavily on this technology, scrapers are directly affected as well. 

Before moving forward to anti-bot techniques, I think it is important to discuss how cloudflare detects Selenium. Well, the system can be very complex. For example, there are properties in a browser that a web driver lacks. The `navigator` interface in a browser even has a property called `webdriver` that indicates if a browser is controlled by automation. And that is an instant giveaway. If you want to experiment with it:

Type the following command: `navigator.webdriver`

In your case, it should return `false`. But if you try it with Puppeteer or Selenium, you will get `true`. If you’re wondering how Cloudflare leverages this to detect bots, well it’s pretty simple. All they need to do is inject a script like the one below on their partner's website:

// detection-script.js

const webdriver = navigator.webdriver

// If webdriver returns true, display a reCaptcha

// In this example, I am transferring the user to a Cloudflare challenge page.

// But you get the idea

if ( webdriver ) location.replace('https://cloudflarechallenge.com')

Of course, in real life, there are many more levels of detection these providers use. Even the size of the screen, the keyboard layout, or the plugins used by the browsers are used to specifically fingerprint a browser. If you’re interested in how browserless detection works, check out my simple service worker test. And that is just if you stick to the browser. You can also detect bot activity by looking into the IP address from where the request originates. For example, if you’re using datacenter IPs, your chances of getting blocked increase with every request. That is why it’s recommended to use residential or ISP proxies when you’re building a web scraper.

Fortunately, the web scraping community is really active. And because there is such demand to bypass Cloudflare and other anti-bot providers, there are open source solutions in that area. Great things can be achieved when programming communities work together! Moving forward, I suggest we follow these steps:

Run some tests to see if default Selenium can bypass Cloudflare 

Add some extra evasions to make our scripts stealthier

So let’s get started with our first step:

#1: Can default Selenium bypass Cloudflare?

I am not the type to make assumptions. And that’s especially because we don’t know for sure how Cloudflare systems work. They use all sorts of obfuscation with their code, which makes it harder to reverse engineer. That’s why, throughout my experience as a developer, I learned that testing is the best way to understand how a system works. So let us build a basic scraper and see how it behaves on a real target, protected by Cloudflare.

With Python, it’s best we isolate our projects inside one single directory. So let’s create a new folder, open a terminal window and navigate to it:

# Create a new virtual environment and activate it

~ » python3 -m venv env && source env/bin/activate

# Install dependencies

~ » python3 -m pip install selenium

# Create a new .py file and open the project inside your IDE

~ » touch app.py && code .

1.2. Build a simple web scraper with Selenium

Now that you have successfully set up your project, it’s time to add some code. We’re not going to build anything fancy here. We just need this script for testing purposes. If you want to learn about advanced scraping, check out this [LINK] tutorial on Pyppeteer.

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

# Set Chrome to open in headless mode

options = Options()

options.headless = True

# Create a new Chrome instance and navigate to target

driver = webdriver.Chrome(options=options)

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('screenshot.png')

# Close browser

driver.quit()

Now check out the screenshot. Here’s what I got:

I think we’re safe to conclude that the test failed. The targeted website is protected by Cloudflare and as you can see, we’re getting blocked. So by default, Selenium is not able to bypass Cloudflare. I am not going to go in depth and check with other bot detection providers. If you want to test further, here are some targets and their vendors:

#2: Can stealthy selenium bypass Cloudflare?

First of all, let me clarify the terms. By stealth Selenium I mean a version of Selenium that can go undetected and bypass Cloudflare. I am not referring to any specific stealthiness technique. There are a couple of ways you can implement evasion techniques into Selenium. There are packages that handle it, or you can use the `execute_cdp_cmd` to interact directly with the Chrome API. The latter allows you more control but requires more work. Here is an example of how you could use it to change the user agent’s value:

driver.execute_cdp_cmd('Emulation.setUserAgentOverride', {

               "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win32; x86) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36",

               "platform": "Win32",

               "acceptLanguage":"ro-RO"

       })

But you’d have to go through CDP and identify the APIs that allow you to make all the necessary changes. So for the time being, let’s test with some packages.

There are at least two packages you can use to make Selenium stealthy. Up to this point though, none of them is guaranteed to bypass Cloudflare. Yet again, we need to test and see if either of them works. First of all, let’s look into `selenium-stealth`. This package is a wrapper around `puppeteer-extra-plugin-stealth`, making it possible to use Puppeteer’s evasions with Python’s Selenium. To use it, you have to install it first. Open a terminal window and enter this command:

# Install selenium-stealth

~ » python3 -m pip install selenium-stealth

You’re all set now. We can use it to make our previous scraper stealthier:

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

from selenium_stealth import stealth

import time

# Set Chrome to open in headless mode

options = Options()

options.headless = True

# Create a new Chrome instance

driver = webdriver.Chrome(options=options)

# Apply stealth to your webdriver

stealth(driver,

   languages=["en-US", "en"],

   vendor="Google Inc.",

   platform="Win32",

   webgl_vendor="Intel Inc.",

   renderer="Intel Iris OpenGL Engine",

   fix_hairline=True,

)

# Navigate to target

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('stealth.png')

# Close browser

driver.quit()

Running the script returned some different outcomes for me this time, as opposed to the default Selenium settings:

The second option you may use is `undetected_chromedriver`. This one is described as an ‘optimized Selenium chromedriver. Let’s test it out:

# Install undetected_chromedriver

~ » python3 -m pip install undetected_chromedriver

The code is very similar to our default script. The main difference stands in the package name. Here is a basic scraper with `undetected_chromedriver` and let’s see if it can bypass Cloudflare:

import undetected_chromedriver as uc

import time

# Set Chrome to open in headless mode

options = uc.ChromeOptions()

options.headless = True

# Create a new Chrome instance and maximize the window

driver = uc.Chrome(options=options, executable_path='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome')

driver.maximize_window()

# Navigate to target

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('stealth-uc.png')

# Close browser

driver.quit()

Yet again, running the script turns out well for me. It seems that at least these two packages can successfully bypass Cloudflare protection. At least in the short term. Truth be told, chances are, if you use these scripts extensively, Cloudflare will catch up on your IP address and block it. So let me introduce you to a third option: Web Scraping API. 

Web Scraping API has this amazing feature called the Proxy Mode. You can read more about it here. But what I want to note here is that our Proxy Mode can successfully be integrated with Selenium. This way, you get access to all the evasion features we’ve implemented. And let me tell you that we have a dedicated team working on custom evasion techniques. In technical terms, we’re handling IP rotations, we’re using various proxies, we’re solving captchas and we’re using Chrome’s API to continuously change our fingerprint. In non-technical terms, this translates to less hassle on your side and a greater success rate. You basically get the stealthiest version of Selenium there is. And here’s how it’s done:

# Install selenium-wire

~ » python3 -m pip install selenium-wire

We’re using `selenium-wire` in order to use Selenium with a proxy. Now here’s the script:

from seleniumwire import webdriver

import time

# Method to encode parameters

def get_params(object):

   params = ''

   for key,value in object.items():

       if list(object).index(key) < len(object) - 1:

           params += f"{key}={value}."

       else:

           params += f"{key}={value}"

   return params

# Your WSA API key

API_KEY = '<YOUR_API_KEY>'

# Default proxy mode parameters

PARAMETERS = {

   "proxy_type":"datacenter",

   "device":"desktop",

   "render_js":1

}

# Set Selenium to use a proxy

options = {

   'proxy': {

       "http": f"http://webscrapingapi.{ get_params(PARAMETERS) }:{ API_KEY }@proxy.webscrapingapi.com:80",

   }

}

# Create a new Chrome instance

driver = webdriver.Chrome(seleniumwire_options=options)

# Navigate to target

driver.get('https://www.httpbin.org/get')

# Retrieve the HTML documeent from the page

html = driver.page_source

print(html)

# Close browser

driver.quit()

If you run this script a couple of times, you’ll see how the IP address changes every time. That’s our IP rotation system. In the background, it also adds evasions techniques. You don’t even need to worry about them. We handle the Cloudflare bypassing part so that you can focus more on parsing the data. 

If you want to build a scraper that can bypass Cloudflare, you need to account for a lot of things. A dedicated team can work 24/7 and there’s still no guarantee the evasions will work every time. That’s because, with every browser version release, there is a chance new features are added to the API. And some of these features can be used to fingerprint and detect bots.

I’d even say that the best browser to bypass Cloudflare and other providers is the one you build yourself. And we built one at Web Scraping API. Now we’re sharing it with you. So enjoy scraping!

Learn How To Bypass Cloudflare Detection With The Best Selenium Browser

Web scraping is the process of extracting data from a website and using it for a variety of purposes. It can be a useful tool for businesses looking to gather information on their competitors, for individuals looking to track changes on a website, or for researchers looking to gather data for a study.

There are many different ways to scrape a website, but one of the most popular methods is using an API. An API, or application programming interface, is a set of rules and protocols that allow different software programs to communicate with each other.

SERP stands for Search Engine Results Page. It is the page that is displayed to a user after they enter a query into a search engine, such as Google, Bing, or Yahoo. The SERP contains a list of results, also known as "search results," that are relevant to the user's query. These results can include web pages, images, videos, and other types of content. The SERP also includes other elements such as sponsored ads, local packs, featured snippets, and more.

SERPs are constantly changing and evolving as search engines continue to improve their algorithms and add new features. The layout, format, and content of SERPs can vary depending on the search engine, the query, and the location of the user. Understanding the SERP and what appears on it is crucial for SEO and SEM professionals, as it helps them to understand how their website is performing in search engine results and what they can do to improve their visibility and performance.

When it comes to scraping Google's search engine results pages (SERPs), there are several APIs available, both free and paid. In this article, we'll take a look at the top 7 Google SERP APIs and compare their pros and cons.

The first option we are going to explore is WebScrapingAPI as it brings the best value for money compared to others in the list.

WebScrapingAPI can be used by developers to scrape the web and has a lot of great features such as residential proxies, JavaScript rendering, CSS extracting, CAPTCHA bypassing, IP rotation, and evasions which gets them rid of the stress of being banned. It is very easy to use and all you need to do is to send your API KEY and a search query in an API call, just like this: https://serpapi.webscrapingapi.com/v1?engine=google&api_key=<YOUR-API-KEY>&q=Steve%20Jobs and that’s it, you are now ready to receive your results back. On top of that, this service offers a lot of dedicated scrapers for Google or Amazon, making it a great option for businesses or individuals who want to affordably scrape data from these websites and use it for different purposes.

When it comes to pricing, WebScrapingAPI has something for everybody with pricing starting from as little as 49$ per month for 100.000 API calls to an Enterprise plan for those who want to scrape the web at a large scale and need more than 10.000.000 API calls. The cost per request may vary depending on whether you use rendering JavaScript or residential proxies from 1 to 25 API credits.

This service also offers you the chance to try all products and features for free for 7 days with no credit card required, thus enabling you to find out how closely it satisfies your needs before you purchase a subscription. In case you encounter any issue with the product, you can easily get in touch with their technical support and someone will get back to you very soon, usually within a couple of hours.

Pros: By far best value for money from this list. This API is feature-rich, has great customer care and it is suitable for every budget.

Cons: Does not provide the response in many different formats such as CSV as some other options from the list, but you can easily convert the data using an JSON to CSV parser.

The second option we are going to explore is Apify which provides a great Google Scraping service.

The API allows users to extract data from Google search results pages in real time and offers a variety of options for customizing the scrape, such as the ability to search for specific keywords, retrieve data on specific SERP elements, and scrape results from multiple countries and languages. Additionally, the Apify platform offers an actor library of pre-made scraping and automation tasks, which can save time and effort for users who need to perform repetitive scraping tasks. The platform also offers a cloud storage for the scraped data which can be easily exported to different formats and a feature called Autoscaled Pool of Headless Browsers that allows to run multiple concurrent scraping tasks. Apify's Google scraping API is a valuable tool for businesses, researchers, and individuals looking to gather data on search trends, track changes in SERPs, and gather data on competitors.

When it comes to pricing, Apify has a pay-as-you-go pricing model, which means that users are charged based on the number of page loads and the amount of data scraped. The exact pricing can vary depending on the specific scraping needs of the user. It's worth mentioning that Apify offers a free trial with a certain amount of free page loads and storage, so you can test the platform and see if it meets your scraping needs before committing to a paid plan.

Pros: Offers a lot of scrapers and flexibility, thus being suitable for a lot of different scraping needs.

Cons: Price and monitoring systems can become confusing, especially at a greater scale.

Another option we are going to explore is SerpAPI which is a service that allows users to scrape Google's search engine results pages (SERPs) in real time. 

SERP API is a powerful web scraping tool specifically designed to scrape Google's search engine results pages (SERPs) in real-time. One of the key features of SERP API is its ability to scrape SERPs from multiple countries and languages, which allows users to gather data on search trends in different regions. Additionally, SERP API offers an extensive list of endpoints, which allows users to retrieve specific types of data from SERPs, such as organic search results, paid search results, featured snippets, local packs, images, videos, etc.

Another great feature of the service is its ability to handle CAPTCHAs, which can be a common issue when scraping Google SERPs. This allows users to continue scraping even when faced with CAPTCHAs, without having to manually solve them.

SerpAPI also offers a user-friendly dashboard that allows users to monitor the usage and track the progress of the scrapes and also offers support for users who need help getting started or troubleshooting issues.

This API stands out by being extremely feature-rich having scrapers for very many different engines such as: 

While most of the engines offer high quality data fastly and provide great documentation which is very easy to follow, at the time of writing this article there are a few engines that do not work as intended, but their team is currently working on fixing these issues, so you can expect that to improve soon. 

When it comes to pricing, it starts from $50 for 50,000 Google searches per month and can get expensive at a larger scale so it is definitely not as cheap as some of the other options from this list.

Pros: It is the option with most features from this list with great documentation making them very easy to use. This service offers a beautiful playground for you to use and test if the product satisfies your needs before committing to a paid subscription.

Cons: The pricing is quite expensive and some engines are currently unreliable.

The fourth option we are going to talk about is SerpWow which provides an easy-to-use Google Search API.

This service is powerful and it enables the developer to retrieve SERP data in real time, which can be used to build SEO tools, track keyword rankings, and analyze search engine performance. SerpWow uses a wide variety of proxies, thus getting the users rid of the worry of being banned and giving the data back in either JSON, CSV, or HTML. On the other hand, at the time of writing this article, they are not as feature-rich as other options from this list and provide scrapers only for a few Google engines. 

When it comes to pricing, it starts from $45 for 5000 Google searches, and as you may already have noticed it is very expensive compared to other options. SerpWow is intended more for Big Data purposes, having the most plans designed for Big Data from this list. In this case, the prices are starting from 500$ per month for 100.000 searches and get as expensive as 6300$ per month for 3.000.000 searches.

Pros: This service offers an easy to use Google Scraping API which returns the data back in various formats.

Cons: It is very expensive and not as feature-rich as other options.

The next option we are going to talk about is OxyLabs which is a professional scraping company that can help businesses and individuals to extract data and insights from the search engines to improve their online performance.

OxyLabs is a very well-known company in the scraping world and they developed a SERP solution that is high quality and reliability that uses proxy rotation in order to avoid getting banned. This service offers a fast SERP API and they provide in-depth tutorials by providing many code samples showing how to integrate their API into your product. 

But, same as the last option on the list it is very expensive with plans starting from 99$ per month for 29.000 requests (limited to 15 results per request) and getting as expensive as 10.000$ per month for more than 10.000.000 searches (no search limits per results).  

Pros: This service offers a fast, reliable and scalable Google SERP solution.

The sixth option we are going to explore is SerpStack, a product developed by Apilayer, a software company based in London. 

While this product was designed first for internal needs, it has grown to become one of the most reliable products in the marketplace. The Serpstack API allows automated retrieval of SERP data from search engines by utilizing a proxy network. With solid documentation, they make scraping Google an easy and straightforward process for developers.

When it comes to pricing it is more affordable than other options from the list, starting from 30$ for 5.000 searches and getting as expensive as 200$ per month for 50.000 searches. It is worth mentioning that if you need scraping at a larger scale this service also offers a custom Enterprise plan designed for such needs. In order to learn more about you need to get in touch with their sales team. 

Pros: This service offers a cheap and reliable Google SERP solution.

Cons: It is not as feature-rich as other candidates.

Lastly, we are going to talk about the SERP API offered by DataForSEO, a provider that offers a variety of tools and services that allow businesses and individuals to gather data on search engine rankings, keywords, backlinks, and other important SEO metrics.

The SERP API from DataForSEO is very flexible and suitable for those who are not sure how many requests are going to consume in a month, having a pay-as-you-go policy. This API is great and lets you scrape Google Search Engine and receive back high-quality data in a reasonable amount of time.

Having a pay-as-you-go payment system makes the price quite confusing and expensive for those who need Google scraping at a larger scale. It is more suitable for those who do not need as many API calls and do not have a constant number of calls per month.

Pros: Flexible and great for digital marketers or SEO consultants.

Cons: Gets expensive when scraping at scale and it is not as feature-rich as other options from the list.

In conclusion, I hope this article gives you an overview of the Google SERP available services in the marketplace. The ability to scrape Google's search engine results pages (SERPs) can be a valuable tool for businesses, researchers, and individuals looking to gather data on search trends, track changes in SERPs, and gather data on competitors. 

There are several options available for scraping Google SERPs and it is extremely important to consider the specific needs of your business or project and to carefully evaluate the features and limitations of each service.

Top 7 Best Google SERP APIs (Free & Paid)

Data parsing is the process of analyzing and transforming structured or unstructured data into a more specific format that various software systems can use and understand.

To make this concept easier to understand, here are some examples of parsers:

CSV parsers: for parsing and converting CSV (Comma Separated Values) files into more usable formats

JSON parsers: for parsing and converting JSON (JavaScript Object Notation) data into more usable formats

Regex parsers: for parsing and extracting specific patterns of text using regular expressions

Compilers: for parsing and converting code written in one programming language into machine-readable code in another programming language

SQL Parsers: for analyzing and interpreting a SQL query, carrying out the command, and returning the results

When it comes to web scraping, data parsing is essential. Websites are made up of HTML, a great markup language for displaying information on a screen but not so great for machines to read.

When we scrape a website, we're grabbing a big string of HTML. To do anything useful with that information, we need to parse it.

This will cover the importance of data parsing in web scraping, as well as the different types of data parsers available, such as HTML parsing libraries for various programming languages, regular expressions, and building your parser.

A good data parser can extract relevant information from an HTML document based on pre-defined rules, regardless of the type of parser used. The parsing process consists of two main steps: lexical analysis and syntactic analysis.

Lexical analysis is the process of analyzing individual words and symbols in a document and breaking them down into smaller, more manageable pieces. 

This involves tokenization, which is the process of breaking a document down into individual tokens, such as keywords, symbols, and numbers.

Let’s take a look at this simple HTML document:

<html>

  <head>

	<title>Scraping</title>

  </head>

  <body>

	<h1>Welcome to my scraping page</h1>

	<p>This is a paragraph.</p>

	<ul>

  	<li>First Scraping Item</li>

  	<li>Second Scraping Item</li>

	</ul>

  </body>

</html>

The lexical analysis process would tokenize this document into individual elements such as:

This way, each element of the HTML document gets split down into smaller, more manageable tokens that can be further analyzed and processed.

Syntactic analysis is the process of analyzing the structure of a document and determining how the individual tokens relate to each other. This involves identifying patterns and structures in the data and using this information to create a tree-like structure called a parse tree.

For example, the <html> tag is the root element, and it contains the <head> and <body> elements. Within the <head> element, there is a <title> element, and within the <body> element, there are <h1>, <p>, and <a> elements. 

By identifying these elements and their relationships, you can construct a parse tree, with the <html> element as the root, <head> and <body> as its children, and so on.

You can use the parse tree to extract specific data from the HTML document, such as the text within the <title> element, or the href attribute of the <a> element.

In this section, we will explore some of the most popular HTML parsing libraries available for different programming languages. These libraries make it easy to extract structured data from an HTML document and can be a great starting point for your web scraping project. 

From Python's Scrapy and BeautifulSoup, to NodeJS's Cheerio and Java's JSoup, we will take a look at each library and provide examples of how to use them. 

Whether you are a beginner or an experienced developer, this section will give you a solid understanding of the options available to you when working with HTML data.

Cheerio is a JavaScript library that allows developers to parse, manipulate, and navigate the DOM of an HTML or XML document, like how jQuery works. This article goes into much more detail about Cheerio and talks about different use cases.

Here’s a simple cheerio implementation:

const cheerio = require('cheerio');

const $ = cheerio.load('<h2 class="title">Hello, World!</h2>');

console.log($('h2').text())

Running this code will return the following output:

Scrapy and BeautifulSoup are libraries for web scraping in Python. 

Scrapy is a powerful web scraping framework that allows you to extract structured data from websites by using selectors or XPath expressions. 

import scrapy

class QuotesSpider(scrapy.Spider):

    name = "quotes"

    def start_requests(self):

        urls = [

            'https://quotes.toscrape.com/page/1/',

            'https://quotes.toscrape.com/page/2/',

        ]

        for url in urls:

            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):

        page = response.url.split("/")[-2]

        filename = f'quotes-{page}.html'

        with open(filename, 'wb') as f:

            f.write(response.body)

        self.log(f'Saved file {filename}')

BeautifulSoup is a library that allows you to parse HTML and XML documents and extract data from them in a way like how a web browser does. 

Here’s a simple BeautifulSoup implementation:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>Scraper</title></head>

<body>

<h1 class="title">Hello, World!</h1>

</body>"""

soup = BeautifulSoup(html_doc, 'html.parser')

print(soup.title.string)

Both of these libraries provide a simple and convenient API for traversing, searching, and modifying the content of web pages, and have a perfect use in web scraping projects.

If your programming language of choice is Java, JSoup is a data parser that provides a convenient API for extracting and manipulating data, using the best of DOM, CSS, and jQuery-like methods. 

This allows you to parse and navigate HTML documents, and change the contents of a document using a simple, intuitive API. The library is a perfect fit for web scraping, web crawling, and data extraction projects.

Here’s a simple implementation of JSoup for extracting the text from the title tag:

import org.jsoup.Jsoup;

import org.jsoup.nodes.Document;

public class JSoupExample {

	public static void main(String[] args) {

    	String html = "<html><head><title>Example Title</title></head>"

            	+ "<body>Hello, World!</body></html>";

    	Document doc = Jsoup.parse(html);

    	// Get the title of the document

    	String title = doc.title();

    	System.out.println("Title: " + title);

	}

}

Nokogiri is a library for Ruby that provides an easy-to-use interface for parsing and searching XML and HTML documents and supports XPath and CSS selectors, making it used for web scraping and data extraction tasks.

For a more comprehensive analysis of the data parser libraries in the Ruby ecosystem, you can read this article.

Use this command to install the nokogiri gem:

The next code sample is a simple nokogiri implementation:

require "nokogiri"

html = "<!DOCTYPE html><html><head><title>Hello, World!</title></head><body>Hello, World!</body></html>"

parsed_data = Nokogiri::HTML.parse(html)

puts parsed_data.title

Regular expressions, also known as regex, are a powerful tool for matching patterns in strings. They are often used for text-processing tasks such as searching, validating, and extracting information from a document.

You can use regular expressions to extract information from HTML by searching for specific patterns, like extracting email addresses or headings from an HTML document.

For example, to extract all the URLs from an HTML document, you can use the following regular expression:

This expression will match any string that starts with "http" or "https" followed by a ":" and two slashes, then any combination of word characters, dots, and hyphens, followed by a dot and one or more lowercase letters. The "gi" flag makes the search global and case-insensitive.

Using regular expressions can be a bit tricky, as the syntax can be complex, and getting the pattern exactly right can take some trial and error. There are also some sites like Regex101 or Rubular, that can help you test and debug your regular expressions.

If you don't want to use the available libraries or regular expressions, you can always build your parser. 

Building your parser can be a great way to gain a deeper understanding of the data you're working with and can also be a good option if you have specific requirements that aren't met by existing libraries or tools.

Building your parser can be a challenging task, but it can also be a rewarding one. The process of building a parser involves creating a set of rules and instructions that specify how the data gets parsed and organized.

You can do this by using a variety of techniques, such as regular expressions, state machines, and recursive descent parsing.

When building a parser, it's crucial to have a thorough understanding of the data's structure and format to be able to design an appropriate set of rules and instructions for the parser. Choosing an appropriate programming language is also an important consideration.

One of the advantages of building your parser is that you can tailor it to the data and use case at hand. This can result in a more efficient and effective parser, compared to using a generic library or tool. 

Additionally, building your parser can also be a great learning experience, as it allows you to gain a deeper understanding of the underlying concepts and techniques of data parsing.

Building a parser from scratch comes with disadvantages as well:

It can be time-consuming and need a significant amount of effort to design and implement, especially if you're not familiar with parsing algorithms and data structures.

It can be difficult to get your parser to perform as well as existing libraries or tools that have been optimized for performance.

It can be hard to maintain and update the parser over time if the data format or structure changes.

It might be hard to debug or find errors in your code, especially if you are not familiar with the parsing process.

It could be prone to errors and bugs, which could lead to the parser not working as expected.

In the case of complex parsing, it could be hard to implement all the rules and edge cases.

it might not be as efficient as pre-existing libraries and tools, as they have been optimized and used by many people.

In summary, building a custom parser from scratch has its own set of disadvantages, such as high development time, high maintenance cost, and high risk of errors. It's generally recommended to use existing libraries or tools or to use regular expressions if they can meet the specific requirements of your use case.

Parsing schema.org metadata is a way to extract structured data from web pages using web schema standards. The community behind schema.org manages these standards and promotes the use of schema for structured data on the web.

Parsing schema metadata can be useful for various reasons, such as finding updated information on events, or for researchers gathering data for studies. Additionally, websites that aggregate data like real-estate listings, job postings, and weather forecasts can also benefit from parsing schema data.

There are different formats of schema you can use, including JSON-LD, RDFa, and Microdata. 

JSON-LD (JavaScript Object Notation for Linked Data) is a format for encoding linked data using JSON. The design of this standard makes it easy for humans to read and write and for machines to parse and generate. 

Here’s how JSON-LD would look for a web page about a book:

Recent articles

Sign up for our newsletter

News and updates