Did you ever need to write a CSS selector that is class-independent? If your answer is no, well, you can consider yourself lucky. If the answer is yes, then our XPath Cheat Sheet is what you need. The web is crawling with data. Entire businesses depend on putting some of it together to bring new services to the world. APIs are of great use, but not every website has open APIs. Sometimes, you'll have to get what you need the old way. You'll have to build a scraper for the website. Modern websites circumvent scraping by renaming their CSS classes. As a result, it is better to write selectors that rely on something more stable. In this article, you'll learn how to write selectors based on the DOM node layout of the page.

XPath stands for XML Path Language. It uses a path notation (as in URLs) to provide a flexible way of pointing to any part of an XML document. 

XPath is mainly used in XSLT, but can also be used as a much more powerful way of navigating through the DOM of any XML-like language document using XPathExpression, such as HTML and SVG, instead of relying on the Document.getElementById() or Document.querySelectorAll() methods, the Node.childNodes properties, and other DOM Core features. XPath | MDN (mozilla.org)

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Nothing to see here</title>
</head>
<body>
    <h1>My First Heading</h1>
    <p>My first paragraph.</p>
    <div>
        <h2>My Second Heading</h2>
        <p>My second paragraph.</p>
        <div>
            <h3>My Third Heading</h3>
            <p>My third paragraph.</p>
        </div>
    </div>
</body>
</html>

There are two types of paths: relative and absolute

The unique path ( or absolute path ) to My third paragraph. is /html/body/div/div/p

A relative path to My third paragraph. is //body/div/div/p
For My Second Heading. => //body/div/h2
For My first paragraph. => //body/p

Notice that I'm using //body. Relative paths use // to skip right to the desired element.

The usage of //<path> also implies that it should look for all occurrences of <path> in the document, regardless of what came before <path>.

For example, //div/p returns both My second paragraph. and My third paragraph.

You can test this example in your browser to get a better overview!

Paste the code into a .html file and open it with your browser. Open the developer tools and hit control + F. Paste the XPath locator into the small input bar and press enter.

You can also get the XPath of any tag by right-clicking on it in the Elements tab and selecting “Copy XPath”

Notice how I’m switching between “My second paragraph.” and “My third paragraph.”

Also, another important thing to know is that it is not necessary for a path to contain // in order to return multiple elements. Let's see what happens when I add another <p> in the last <div>.

/html/body/div/div/p is no longer an absolute path.

If you followed me this far, congratulations, you’re on the right track to XPath mastery. You are now ready to dive into the fun stuff.

You can use the square brackets to select specific elements.

 In this case, //body/div/div[2]/p[3] only selects the last <p> tag.

You can also use attributes to select your elements.

//body//p[@class="not-important"] => select all the <p> tags that are inside a <body> tag and have the "not-important" class.

//div[@id] => select all the <div> tags that have an id attribute.

//div[@class="p-children"][@id="important"]/p[3] => select the third <p> that is within a <div> tag that has both class="p-children" and id="important"

//div[@class="p-children" and @id="important"]/p[3] => same as above

//div[@class="p-children" or @id="important"]/p[3] => select the third <p> that is within a <div> that has class="p-children" or id="important"

XPath provides a set of useful functions that you can use inside the square brackets.

position() => returns the index of the element
Ex: //body/div[position()=1] selects the first <div> in the <body>

last() => returns the last element
Ex: //div/p[last()] selects all the last <p> children of all the <div> tags

count(element) => returns the number of elements
Ex: //body/count(div) returns the number of child <div> tags inside the <body> 

node() or * => returns any element
Ex: //div/node() and //div/*=> selects all the children of all the <div> tags

text() => returns the text of the element
Ex: //p/text() returns the text of all the <p> elements

concat(string1, string2) => merges string1 with string2

contains(@attribute, "value") => returns true if @attribute contains "value" 
Ex:
 //p[contains(text(),"I am the third child")] selects all the <p> tags that have the "I am the third child" text value.

starts-with(@attribute, "value") => returns true if @attribute starts with "value" 
ends-with(@attribute, "value") => returns true if @attribute ends with "value" 

substring(@attribute,start_index,end_index)] => returns the substring of the attribute value based on two index values
Ex:
//p[substring(text(),3,12)="am the third"] => returns true if text() = "I am the third child"

normalize-space() => acts like text(), but it removes the trailing spaces
Ex: normalize-space(" example ") = "example"

string-length() => returns the length of the text
Ex: //p[string-length()=20] returns all the <p> tags that have the text length of 20

The functions can be a little tricky to remember. Luckily, The Ultimate Xpath Cheat Sheet provides helpful examples: 

//p[text()=concat(substring(//p[@class="not-important"]/text(),1,15), substring(text(),16,20))]

//p[text()=<expression_return_value>] will select all the <p> elements that have the text value equal to the return value of the condition.

//p[@class="not-important"]/text() returns the text values of all the <p> tags that have class="not-important".

If there is only one <p> tag that satisfies this condition, then we can pass the return_value to the substring function.

substring(return_value,1,15) will return the first 15 characters of the return_value string.

substring(text(),16,20) will return the last 5 characters of the same 

text() value that we used in //p[text()=<expression_return_value>].

Finally, concat() will merge the two substrings and create the return value of <expression_return_value>.

XPath supports path nesting. That’s cool, but what exactly do I mean by path nesting?

Let's try something new: /html/body/div[./div[./p]]

You can read it as "Select all the <div> sons of the <body> that have a <div> child. Also, the children must also be parents to a <p> element."

If you don't care about the father of the <p> element, you can write: /html/body/div[.//p]

This now translates to "Select all the div children of the body that have a <p> descendant"

In this particular example, /html/body/div[./div[./p]] and /html/body/div[.//p] yield the same result.

By now, I'm sure that you are wondering what is up with those dots in ./ and .// 

The dot represents the self element. When used in a pair of brackets, it references the specific tag that opened them. Let's dive a little deeper.

In our example, /html/body/div returns two divs:
<div class="no-content"> and <div class="content">

/html/body/div[.//p] translates to:

    /html/body/div[1][/html/body/div[1]//p]
and /html/body/div[2][/html/body/div[2]//p]

/html/body/div[2][/html/body/div[2]//p] is true, so it returns /html/body/div[2] 

In our case, the dot ensures that /html/body/div and /html/body/div//p refer to the same <div>

Now let's look at what would have happened if it didn't.

/html/body/div[/html/body/div//p] would return both 
<div class="no-content">  and <div class="content">

Why? Because /html/body/div//p is true for both /html/body/div[1] and /html/body/div[2].

/html/body/div[/html/body/div//p] actually translates to "Select all the div children of the <body> if /html/body/div//p is true. 

/html/body/div//p is true if the body has a <div> child, and that child has a <p> descendent". In our case, this statement is always true.

It’s a shame that other Xpath Cheat Sheets don’t mention anything about nesting. I consider it amazing.  It enables you to scan the document for different patterns and come back to return something else. The only downside is that writing queries this way can become hard to follow. The good news is, there are other ways of doing it.

You can use axes to locate nodes relative to other context nodes.

//p/ancestor::div => selects all the divs that are ancestors of <p>

How I read it: Get all the <p> tags, for each <p> look through its ancestors. If you find <div> tags, select them.

//p/parent::div => selects all the <div> tags that are parents of <p> 

How I read it: Get all the <p> tags and of all their parents, if the parent is a <div>, select it.

//div/child::p=> selects all the <p> tags that are children of <div> tags.

How I read it: Get all the <div> tags and their children, if the child is a <p>, select it.

//div/descendant::p => selects all the <p> tags that are descendants of <div> tags.

How I read it: Get all the <div> tags and their descendants, if the descendant is a <p>, select it.

Now it’s time to rewrite the previous expression:

/html/body/div[./div[./p]] is equivalent to /html/body/div/div/p/parent::div/parent::div

But /html/body/div[.//p] is NOT equivalent to /html/body/div//p/ancestor::div

The good news is that we can tweak it a little bit.

/html/body/div//p/ancestor::div[last()] is equivalent to /html/body/div[.//p]

//p/following-sibling::span => for each <p> tag, select its following <span> siblings.

//p/preceding-sibling::span => for each <p> tag, select its preceding <span> siblings.

The ultimate XPath Cheat Sheet. How to easily write powerful selectors.

Rust is a programming language designed for speed and efficiency. Unlike C or C++, Rust features an integrated package manager and build tool. It also has excellent documentation and a friendly compiler with helpful error messages. It does take a while to get used to the syntax. But once you do, you'll realize that you can write complex functionalities with just a few lines of code. Web scraping with Rust is an empowering experience. You gain access to powerful scraping libraries that do most of the heavy lifting for you. As a result, you get to spend more time on the fun parts, like designing new features. In this article, I will walk you through the process of building a web scraper with Rust. 

Installing Rust is a pretty straightforward process. Visit Install Rust - Rust Programming Language (rust-lang.org) and follow the recommended tutorial for your operating system. The page displays different contents based on the operating system you are using. At the end of the installation, make sure you open a brand new terminal and run a rustc --version. If everything went right, you should see the version number of the installed Rust compiler.

Since we will be building a web scraper, let’s create a Rust project with Cargo. Cargo is Rust’s build system and package manager. If you used the official installers provided by rust-lang.org, Cargo should be already installed. Check whether Cargo is installed by entering the following into your terminal:  cargo --version.  If you see a version number, you have it! If you see an error, such as command not found, look at the documentation for your method of installation to determine how to install Cargo separately. To create a project, navigate to the desired project location and run cargo new <project name>.

  You manage dependencies in the Cargo.toml file.

  Visit crates.io: Rust Package Registry to find packages for Rust.

Now let’s have a look at how you could use Rust to build a scraper. The first step is defining a clear purpose. What do I want to extract? The next one is deciding how you want to store the scraped data. Most people save it as .json, but you should generally consider the format that better suits your individual needs. With these two requirements figured out, you can confidently move forward with implementing any scraper. To better illustrate this process, I propose we build a small tool that extracts Covid data from the COVID Live - Coronavirus Statistics - Worldometer (worldometers.info) website. It should parse the reported cases tables and store the data as .json. We will create this scraper together in the following chapters.

To extract the tables, you'll first need to get the HTML that’s inside the web page. We will use the “reqwest” crate/library to fetch raw HTML from the website.

First, add it as a dependency in the Cargo.toml file:

reqwest = { version = "0.11", features = ["blocking", "json"] }

Then define your target url and send your request:

let url = "https://www.worldometers.info/coronavirus/";
let response = reqwest::blocking::get(url).expect("Could not load url.");

The "blocking" feature ensures that the request is synchronous. As a result, the program will wait for it to complete and then continue with the other instructions. 

let raw_html_string = response.text().unwrap();

You got all the necessary raw data. Now you have to find a way to locate the reported cases tables. The most popular Rust library for this type of task is called “scraper”. It enables HTML parsing and querying with CSS selectors.

Add this dependency to your Cargo.toml file:

use scraper::Selector;
use scraper::Html;

Now use the raw HTML string to create an HTML fragment:

let html_fragment = Html::parse_fragment(&raw_html_string);

We’ll select the tables that display the reported cases for today, yesterday, and two days ago.

Open the developer console and identify the table ids:

At the time of writing this article, the id for today is: “main_table_countries_today”.

The other two table id’s are:  
“main_table_countries_yesterday” and “main_table_countries_yesterday2”

let table_selector_string = "#main_table_countries_today, #main_table_countries_yesterday, #main_table_countries_yesterday2";

let table_selector = Selector::parse(table_selector_string).unwrap();

let head_elements_selector = Selector::parse("thead>tr>th").unwrap();
 
let row_elements_selector = Selector::parse("tbody>tr").unwrap();
 
let row_element_data_selector = Selector::parse("td, th").unwrap();

Pass the table_selector_string to the html_fragment select method to get the references of all the tables:

let all_tables = html_fragment.select(&table_selector);

Using the tables references, create a loop that parses the data from each table.

for table in all_tables{
   let head_elements = table.select(&head_elements_selector);
    for head_element in head_elements{
        //parse the header elements
    }

   let head_elements = table.select(&head_elements_selector);
   for row_element in row_elements{
    for td_element in row_element.select(&row_element_data_selector){
        //parse the individual row elements
    }
   }
}

The format you store the data in dictates the way you parse it. For this project, it is .json. Consequently, we need to put the table data in key-value pairs. We can use the table header names as keys and the table rows as values. 

Use the .text() function to extract the headers and store them in a Vector:

//for table in tables loop
let mut head:Vec<String> = Vec::new();

let head_elements = table.select(&head_elements_selector);

for head_element in head_elements{
    let mut element = head_element.text().collect::<Vec<_>>().join(" ");
    element = element.trim().replace("\n", " ");
    head.push(element);
}


//head
["#", "Country, Other", "Total Cases", "New Cases", "Total Deaths", ...]

Extract the row values in a similar manner:

//for table in tables loop
let mut rows:Vec<Vec<String>> = Vec::new();

let row_elements = table.select(&row_elements_selector);

for row_element in row_elements{
 let mut row = Vec::new();
 for td_element in row_element.select(&row_element_data_selector){
     let mut element = td_element.text().collect::<Vec<_>>().join(" ");
     element = element.trim().replace("\n", " ");
     row.push(element);
 }
 rows.push(row)

}
//rows
[...
["", "World", "625,032,352", "+142,183", "6,555,767", ...]
...
["2", "India", "44,604,463", "", "528,745", ...]
...]

Use the zip() function to create a matching between header and row values:

for row in rows {
    let zipped_array = head.iter().zip(row.iter()).map(|(a, b)|                        
    (a,b)).collect::<Vec<_>>();
 }

//zipped_array
[
...
[("#", ""), ("Country, Other", "World"), ("Total Cases", "625,032,352"), ("New Cases", "+142,183"), ("Total Deaths", "6,555,767"), ...]
...
]

Now store the zipped_array (key, value) pairs in an IndexMap:

serde = {version="1.0.0",features = ["derive"]}

indexmap = {version="1.9.1", features = ["serde"]}  (add these dependencies)

use indexmap::IndexMap;

//use this to store all the IndexMaps 
let mut table_data:Vec<IndexMap<String, String>> = Vec::new();
for row in rows {
    let zipped_array = head.iter().zip(row.iter()).map(|(a, b)|                        
    (a,b)).collect::<Vec<_>>();
    let mut item_hash:IndexMap<String, String> = IndexMap::new();
    for pair in zipped_array{
        //we only want the non empty values
        if !pair.1.to_string().is_empty(){
            item_hash.insert(pair.0.to_string(), pair.1.to_string());
        }
    }
table_data.push(item_hash);

//table_data
[
...
{"Country, Other": "North America", "Total Cases": "116,665,220", "Total Deaths": "1,542,172", "Total Recovered": "111,708,347", "New Recovered": "+2,623", "Active Cases": "3,414,701", "Serious, Critical": "7,937", "Continent": "North America"}
,
{"Country, Other": "Asia", "Total Cases": "190,530,469", "New Cases": "+109,009", "Total Deaths": "1,481,406", "New Deaths": "+177", "Total Recovered": "184,705,387", "New Recovered": "+84,214", "Active Cases": "4,343,676", "Serious, Critical": "10,640", "Continent": "Asia"}
...
]

IndexMap is a great choice for storing the table data because it preserves the insertion order of the (key, value) pairs.


Now that you can create json-like objects with table data, it is time to serialize them to .json. Before we begin, make sure that you have all these dependencies installed:

serde = {version="1.0.0",features = ["derive"]}
serde_json = "1.0.85"
indexmap = {version="1.9.1", features = ["serde"]}

Store each table_data in a tables_data Vector:

let mut tables_data: Vec<Vec<IndexMap<String, String>>> = Vec::new();

For each table:
    //fill table_data (see previous chapter)
    tables_data.push(table_data);

Define a struct container for the tables_data:

 #[derive(Serialize)]
  struct FinalTableObject {
         tables: IndexMap<String, Vec<IndexMap<String, String>>>,
 }

 let final_table_object = FinalTableObject{tables: tables_data};

let serialized = serde_json::to_string_pretty(&final_table_object).unwrap();

Write the serialized .json string to a .json file:

use std::fs::File;
use std::io::{Write};

let path = "out.json";

    let mut output = File::create(path).unwrap();

    let result = output.write_all(serialized.as_bytes());

    match result {

          Ok(()) => println!("Successfully wrote to {}", path),

          Err(e) => println!("Failed to write to file: {}", e),

    }

Aaaand, you’re done. If everything went right, your output .json should look like:

{
  "tables": [
    [ //table data for #main_table_countries_today
      { 
       "Country, Other": "North America",
       "Total Cases": "116,665,220",
       "Total Deaths": "1,542,172",
       "Total Recovered": "111,708,347",
       "New Recovered": "+2,623",
       "Active Cases": "3,414,701",
       "Serious, Critical": "7,937",
       "Continent": "North America"
      },
      ...
    ],
    [...table data for #main_table_countries_yesterday...],
    [...table data for #main_table_countries_yesterday2...],
  ]
}

You can find the whole code for the project at [Rust][A simple <table> scraper] (github.com)

Making adjustments to fit other use cases

If you followed me this far, you've probably realized that you can use this scraper on other websites. The scraper is not bound to a specific table column count or naming convention. Also, it does not rely on many CSS selectors. So it should not take a lot of tweaking to make it work for other tables, right? Let's test this theory.

If class="wikitable sortable jquery-tablesorter", you could change the table_selector to:

let table_selector_string = ".wikitable.sortable.jquery-tablesorter";
let table_selector = Selector::parse(table_selector_string).unwrap();

This table has the same <thead> <tbody> structure, so there is no reason to change the other selectors.

The scraper should work now. Let’s give it a test run:

Webscraping with Rust is fun, isn’t it? 

The easiest way to figure what went wrong is to look at the HTML that is returned from the GET request:

let url = "https://en.wikipedia.org/wiki/List_of_countries_by_population_in_2010";


let response = reqwest::blocking::get(url).expect("Could not load url.");

et raw_html_string = response.text().unwrap();

let path = "debug.html";


let mut output = File::create(path).unwrap();

let result = output.write_all(raw_html_string.as_bytes());

The HTML returned from the GET request is different from the one we see on the actual website. The browser offers an environment for Javascript to run and alter the page's layout. In the context of our scraper, we get the unaltered version of it.

Our table_selector did not work because the “jquery-tablesorter” class is injected dynamically by Javascript. Also, you can see that the <table> structure is different. The <thead> tag is missing. The table head elements are now found in the first <tr> of the <tbody>. Thus, they will be picked up by the row_elements_selector.

Removing “jquery-tablesorter” from the table_selector is not enough, we also need to handle the missing <tbody> case:

let table_selector_string = ".wikitable.sortable";

 if head.is_empty() {
    head=rows[0].clone();
    rows.remove(0);
 }// take the first row values as head if there is no <thead>

{
  "tables": [
    [
      {
        "Rank": "--",
        "Country / territory": "World",
        "Population 2010 (OECD estimate)": "6,843,522,711"
      },
      {
        "Rank": "1",
        "Country / territory": "China",
        "Population 2010 (OECD estimate)": "1,339,724,852",
        "Area (km 2 ) [1]": "9,596,961",
        "Population density (people per km 2 )": "140"
      },
      {
        "Rank": "2",
        "Country / territory": "India",
        "Population 2010 (OECD estimate)": "1,182,105,564",
        "Area (km 2 ) [1]": "3,287,263",
        "Population density (people per km 2 )": "360"
      },
      ...
     ]
]

I hope that this article provides a good reference point for web scraping with Rust. Even though Rust’s rich type system and ownership model can be a little overwhelming, it is by no means unfit for web scraping. You get a friendly compiler that constantly points you in the right direction. You also encounter a lot of well-written documentation: The Rust Programming Language - The Rust Programming Language (rust-lang.org).

Building a web scraper is not always a straightforward process. You will face Javascript rendering, IP blocks, captchas, and many other setbacks. At WebScraping API, we provide you with all the necessary tools to combat these common issues. Are you curious to find out how it works? You can try our product for free at WebScrapingAPI - Product. Or you can contact us at WebScrapingAPI - Contact. We are more than happy to answer all your questions!

Mihai Maxim