Long gone are the days when you manually collected and processed the data helping you kickstart your projects. Whether an e-commerce website or a lead generation algorithm, one thing is sure: the data-gathering process was tedious and time-consuming.

In this article, you will learn how Cheerio can come to your help with its extensive features of parsing markup languages, first with some trivial examples and then with a real-life use case.

“But what is Cheerio?” you may ask yourself. Well, trying to clarify a common misconception, I will start with what Cheerio is not: a browser.

The confusion may start from the fact that Cheerio parses documents written in a markup language and then offers an API to help you manipulate the resulting data structure. But unlike a browser, Cheerio will not visually render the document, load CSS files or execute Javascript.

So basically what Cheerio does is to receive an HTML or XML input, parse the string and return the API. This makes it incredibly fast and easy to use, hence its popularity among Node.js developers.

Now, let’s see some hands-on examples of what Cheerio can do. First things first, you need to make sure your environment is all set up.

It goes without saying that you must have Node.js installed on your machine. If you don’t, simply follow the instructions from their official website, according to your operating system.

Make sure to download the Long Term Support version (LTS) and don’t forget about the Node.js Package Manager (NPM). You can run these commands to make sure the installation went okay:

Now, regarding the IDE debate: for this tutorial, I will be using Visual Studio Code, as it is pretty flexible and easy to use, but you are welcome to use any IDE you prefer.

Just create a folder that will hold your little project and open a terminal. Run the following command to set up a Node.js project:

This will create a default version of the package.json file, which can be modified at any time.

Next step: I will install TypeScript along with the type definitions for Node.js:

npm install typescript @types/node -save-dev

I chose TypeScript in this tutorial for its optional static typing to JavaScript objects, which makes the code more bulletproof when it comes to type errors. 

This is the same advantage that steadily increased its popularity among the JavaScript community, according to a recent CircleCI survey regarding the most popular programming languages.

To verify the correct installation of the previous command, you can run:

Now I will create the tsconfig.json configuration file at the root of the project directory, which should define the compiler options. If you want a better understanding of this file and its properties, the official TypeScript documentation got your back. 

If not, then simply copy and paste the following:

Almost done! Now you have to install Cheerio (obviously):

Last but not least, create the src directory which will hold the code files. And speaking of the code file, create and place the index.ts file in the src directory.

For the moment I will illustrate some basic Cheerio features using a static HTML document. Simply copy and paste the content below into a new static.html file within your project:

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Page Name - Static HTML Example</title>
</head>
<body>
    <div class="page-heading">
        <h1>Page Heading</h1>
    </div>
    <div class="page-container">
        <div class="page-content">
            <ul>
                <li>
                    <a href="#">Item 1</a>
                      <p class="price">$100</p>
                      <p class="stock">12</p>
                </li>
                <li>
                    <a href="#">Item 2</a>
                    <p class="price">$200</p>
                    <p class="stock">422</p>
                </li>
                <li>
                    <a href="#">Item 3</a>
                    <p class="price">$150</p>
                    <p class="stock">5</p>
                </li>
            </ul>
        </div>
    </div>
    <footer class="page-footer">
        <p>Last Updated: Friday, September 23, 2022</p>
    </footer>
</body>
</html>

Next, you have to serve the HTML file as input to Cheerio, which will then return the resulting API:

import fs from 'fs'
import * as cheerio from 'cheerio'

const staticHTML = fs.readFileSync('static.html')
const $ = cheerio.load(staticHTML)

If you receive an error at this step, make sure that the input file contains a valid HTML document, as from Cheerio version 1.0.0 this criterion is verified as well.

Now you can start experimenting with what Cheerio has to offer. The NPM package is well-known for its jQuery-like syntax and the use of CSS selectors to extract the nodes you are looking for. You can check out their official documentation to get a better idea.

Let’s say you want to extract the page title:

const title = $('title').text()
console.log("Static HTML page title:", title)

We should test this, right? You are using Typescript, so you have to compile the code, which will create the dist directory, and then execute the associated index.js file. For simplicity, I will define the following script in the package.json file:

"scripts": {
    "test": "npx tsc && node dist/index.js",
}

and the script will handle both steps that I just described.

Okay, but what if the selector matches more than one HTML element? Let’s try extracting the name and the stock value of the items presented in the unordered list:

const itemStocks = {}
$('li').each((index, element) => {
    const name = $(element).find('a').text()
    const stock = $(element).find('p.stock').text()
    itemStocks[name] = stock
})
console.log("All items stock:", itemStocks)

Now run the shortcut script again, and the output from your terminal should look like this:

Static HTML page title: Page Name - Static HTML Example
All items stock: { 'Item 1': '12', 'Item 2': '422', 'Item 3': '5' }

So that was basically the tip of the iceberg. Cheerio is also capable of parsing XML documents, extracting the style of the HTML elements and even altering the nodes’ attributes.

But how can Cheerio help in a real use case?

Let’s say that we want to gather some data to train a Machine Learning model for a future bigger project. Usually, you would search Google for some training files and download them, or use the website’s API.

But what do you do when you can’t find some relevant files or the website you are looking at does not provide an API, has a rate-limiting over the data or does not offer the whole data that you are seeing on a page?

Well, this is where web scraping comes in handy. If you are curious about more practical use cases of web scraping, you can check out this well-written article from our blog.

Back to our sheep, for the sake of the example let’s consider that we are in this precise situation: wanting data, and data is nowhere to be found. Remember that Cheerio does not handle the HTML extraction nor the CSS loading or JS execution.

Thus, in our tutorial, I am using Puppeteer to navigate to the website, grab the HTML and save it to a file. Then I will repeat the process from the previous section.

To be more specific, I want to gather some public opinions from Reddit on a popular drum module and centralise the data in a single file that will be further fed to a potential ML training model. What happens next can also vary: sentiment analysis, market research and the list can go on.

Let’s see how this use case will look put into code. First, you need to install the Puppeteer NPM package:

I will also create a new file reddit.ts, to keep the project more organised, and define a new script in the package.json file:

"scripts": {
    "test": "npx tsc && node dist/index.js",
    "parse": "npx tsc && node dist/reddit.js"
},

To get the HTML document, I will define a function looking like this:

import fs from 'fs'
import puppeteer from 'puppeteer'
import * as cheerio from 'cheerio'

async function getContent(url: string): Promise<void> {

    // Open the browser and a new tab
    const browser = await puppeteer.launch()
    const page = await browser.newPage()

    // Navigate to the URL and write the content to file
    await page.goto(url)
    const pageContent = await page.content()
    fs.writeFileSync("reddit.html", pageContent)

    // Close the browser
    await browser.close()
    console.log("Got the HTML. Check the reddit.html file.")
}

To quickly test this, add an entry point in your code and call the function:

async function main() {


    const targetURL = 'https://old.reddit.com/r/Drumming/comments/r3tidc/yamaha_ead10/'
    await getContent(targetURL)
}

main()
    .then(() => {console.log("All done!")})
    .catch(e => {console.log("Unexpected error occurred:", e.message)})

The reddit.html file should appear in your project tree, which will contain the HTML document we want.

Now, a bit of a more challenging part: you have to identify the nodes that are of interest for our use case. Go back to your browser (the real one) and navigate to the target URL. Go with your mouse cursor over the comments section, right-click and then choose the “Inspect” option.

The Developer Tools tab will open, showing you the exact same HTML document you previously saved on your machine.

To extract only the comments, you have to identify the selectors unique to this section of the page. You can notice that the whole list of comments is within a div container with a sitetable nestedlisting class.

Going deeper, each individual comment has a form element as a parent, with the usertext warn-on-unload class. Then, at the bottom, you can see that every comment’s text is divided between multiple p elements.

function parseComments(): void {

    // Load the HTML document
    const staticHTML = fs.readFileSync('reddit.html')
    const $ = cheerio.load(staticHTML)

    // Get the comments section
    const commentsSection = $('div.sitetable.nestedlisting')

    // Iterate each comment
    const comments = []


    $(commentsSection).find('form.usertext.warn-on-unload').each((index, comment) => {
        let commentText = ""

          // Iterate each comment section and concatenate them
          $(comment).find('p').each((index, piece) => {
            commentText += $(piece).text() + '\n'
          })

          comments.push(commentText)
    })

    // Write the results to external file
    fs.writeFileSync("comments.json", JSON.stringify({comments}))
}

Alright, and now let’s update the entry point with the newly defined function and see how this code is working together:

async function main() {

    const targetURL = 'https://old.reddit.com/r/Drumming/comments/r3tidc/yamaha_ead10/'
    await getContent(targetURL)
    parseComments()
}

main()
    .then(() => {console.log("All done. Check the comments.csv file.")})
    .catch(e => {console.log("Unexpected error occurred:", e.message)})

Execute the code with the script defined before:

It will take around 5 to 10 seconds for the headless browser to open and navigate to our target URL. If you are more curious about it, you can add timestamps at the beginning and the end of each of our functions to really see how fast Cheerio is.

The comments.json file should be our final result:

This use case can be easily extended to parsing the number of upvotes and downvotes for every comment, or to get the comments’ nested replies. The possibilities are endless.

Thank you for making it to the end of this tutorial. I hope you grasped how indispensable Cheerio is to the process of data extraction and how to quickly integrate it into your next scraping project.

We are also using Cheerio in our product, WebScrapingAPI. If you ever find yourself tangled by the many challenges encountered in web scraping (IP blocks, bot detection, etc.), consider giving it a try.

Scraping with Cheerio: How to Easily Collect Data from Web Pages

Olawale Daniel, the principal consultant and lead trainer with Olda Consult, once famously said: “The government wants to fool its people by doubling the minimum wage but at the same time printing more than enough money into circulation. We can't solve problems by creating more problem for generations to come, which is why Bitcoin and some other decentralized finance applications powered by blockchain technology looks to be our only hope of creating a better future.”

Daniel's words are resonating with the cryptocurrency community as decentralized networks become increasingly popular. In the cryptocurrency space, decentralization is the key to success. Decentralized Autonomous Organizations (DAOs) provide users with a voting right on the network and make it difficult for malicious actors to seize control of the project. This decentralized structure makes it difficult for malicious actors to seize control of the project and manipulate its features or assets.  Two projects that demonstrate this decentralization are the ice Network and Pi Network. In this article, we will explore what prominent financial and tech institutions have to say about these networks and why decentralization is so important for the future of finance. We will look at Amazon and Nasdaq’s views on the subject, as well as insights from Ledger. 

What Exactly is Decentralized in The Crypto Space?

The most important aspect of decentralization in cryptocurrency is the removal of a central authority. Amazon explains the concept of decentralization in the following way:  "In blockchain, decentralization refers to the transfer of control and decision-making from a centralized entity (individual, organization, or group thereof) to a distributed network. Decentralized networks strive to reduce the level of trust that participants must place in one another, and deter their ability to exert authority or control over one another in ways that degrade the functionality of the network."

Amazon adds that though blockchain technologies utilize decentralized networks, a blockchain application cannot be easily identified as being either centralized or decentralized.

"Rather, decentralization is a sliding scale and should be applied to all aspects of a blockchain application. By decentralizing the management of and access to resources in an application, greater and fairer service can be achieved. Decentralization typically has some tradeoffs such as lower transaction throughput, but ideally, the tradeoffs are worth the improved stability and service levels they produce," it says.

The ice Network As A Decentralized Platform

The ice Network, which allows users to mine its crypto asset on mobile phones and tablets, is a fully decentralized platform. The ice Network's distributed consensus protocol is designed to provide users with trustless security that no single entity can control. This means that the network cannot be shut down or manipulated by any single individual or organization.

The ice Network allows users to vote on various decisions regarding the network’s governance, from how funds are allocated to which projects should receive funding from the community pool. This allows the network to remain secure and highly resistant to any malicious actors attempting to manipulate it.

The Pi Network As A Decentralized Platform

Like the ice Network is a cryptocurrency platform that allows users to mine its crypto asset on mobile phones and tablets. A key difference, however, is that the Pi Network is a centralized platform, meaning that there is a centralized entity in control of the network. In other words, it does not give users any voting rights over its decision-making processes or operations.

Instead, all decisions are made by an appointed team of developers, meaning that users do not have any say in how their investments are used or managed. This lack of decentralization can make it easier for malicious actors to take control of the project and manipulate its features or assets without detection.

Nasdaq, the world's second-largest stock exchange, recently published an article calling decentralization "the future of finance, at least for the world’s cryptocurrency investors." It provided seven reasons why the concept of decentralization is critical:

1) Independent control and decision-making:

Decentralized platforms put users in control of their finances and allow them to make decisions without relying on a centralized entity. " The direction and future of your system aren’t dependent on the integrity of any single authority," it wrote. "If a single authority ever becomes corrupt, damaged, or compromised, the entire organization has the potential to collapse. But a system without any corruptible authority can maintain better protection and independence."

With decentralization, users can be more confident that their investments are safe and secure from malicious actors. Think of it as a financial security blanket without which risk is higher. It's no wonder that more and more investors are turning to decentralized platforms for their investments. 

Decentralized networks remove the need for participants to trust one another in order to transact.   "Instead, you can trust the emergent wisdom of thousands or millions of devices working together to keep each other honest. No individual bad actor can possibly compromise this system," Nasdaq wrote.

In today's digital world, trust is a huge issue. Decentralized networks mitigate the risk of trust issues by providing users with an efficient and secure way to transact without having to worry about their investments being compromised.  When users have the trustless assurance of decentralized networks, it emboldens them to invest in the project and support its growth.

Decentralized networks mean that users don’t need to rely on a single data source. Instead, they can pull from multiple sources and trust that the information is accurate because it has been verified by numerous independent participants.

"Data reliability in a traditional financial system is questionable, as each stage of the data storage and exchange process opens the door to opportunities for fraud, corruption, and decay," the Nasdaq article read.  "For example, when one company sends data to another, they typically store it in a data silo. And, it only emerges when someone needs to transfer it. Distributed networks, by contrast, allow for more transparency and data integrity."

In other words, decentralization gives users the confidence they need to know that their investments are safe and secure. And, since data reliability is improved by these networks, it makes them more trustworthy for investors. Imagine how much easier it is to make an investment decision when you know that the data behind it is reliable. 

When it comes to security, decentralized networks have fewer vulnerabilities than centralized networks. Since data is spread out between multiple nodes and no central authority holds the power to make decisions or control information, it is much harder for bad actors to manipulate the system and gain access to sensitive user data.

"As stated in a previous point, the system of trust-less connection inherent in most cryptocurrencies disallows for the possibility of a single bad actor forging a transaction and corrupting the system; instead, more than 50 percent of the nodes in the network would need to cooperate unanimously to gain influence over the system," Nasdaq wrote.

The fewer vulnerabilities of decentralized networks make them attractive to both investors and developers. With fewer potential threats, users can feel more secure about their investments and less likely to have their assets stolen or misused by malicious actors. It's an added layer of protection that gives users peace of mind when investing in a project.

One of the major advantages of decentralized networks is that they offer better resource distribution. Decentralized platforms are designed to spread resources evenly among participants, instead of only a few central authorities. This means that more users have access to resources, and that no single entity can monopolize them.

"Distribution has the potential to improve resource allocation within the system as well. Distributed networks can engage nodes, as needed, to produce the most consistent performance," according to Nasdaq.

The improved resource distribution of decentralized networks allows projects to scale more effectively and users to access better services. By providing resources to all participants, it also ensures that no single user is left out or disadvantaged by the system. Why would you settle for a centralized network that largely benefits a few users when you can access the same resources through a decentralized platform? 

Decentralized networks offer users the financial independence that other traditional systems often don’t. With these platforms, users can send and receive payments without relying on third-party entities like banks or payment processors. Cryptocurrencies like Bitcoin and ice make it possible for users to transact without relying on a third-party intermediary.

"They don’t necessarily have to go through a bank to finance a major purchase, nor do they have to create an audit trail that infringes on their privacy. Individuals within the network can exercise much more independent decision-making," Nasdaq wrote.

When users have more control over their finances, it allows them to make better investment decisions and take advantage of opportunities that may not otherwise be available. Decentralized networks offer users a level of independence that other traditional systems lack and can provide many benefits for investors who are looking for freedom from the constraints of centralized platforms. This is especially true for users in developing countries where access to traditional banking services may be limited or nonexistent. 

Finally, decentralized networks offer greater possibilities for competition. Since there is no central authority or monopoly, new entrants can enter the network and add value without having to compete with a much larger entity. This means that smaller players have a chance to succeed and grow, leading to a healthier overall environment.

"In our modern financial system," Nasdaq wrote, "it’s nearly impossible to start a new institution (such as a bank) to threaten influential players or provide better services to consumers. But, in the world of decentralized digital currency, it’s the Wild West."

The open and competitive nature of decentralized networks inspires innovation and creates opportunities for businesses to reach new, untapped markets. With the right resources, users can capitalize on these opportunities and launch projects with a much higher chance of success than they would have in a more centralized system. In the end, this leads to a healthier and more diverse economic landscape. 

In an article on his Medium blog, Spencer Bogart, general partner at Blockchain Capital, summed up the advantages of decentralized networks succinctly: “The reality, however, is that without decentralization these crypto networks lose their most important qualities of being “permissionless” and “censorship-resistant” — that is, that anyone can use the network and anyone can build on top of them."

When Nasdaq, as a prominent financial institution,  and Amazon, as a digital commerce giant,  and prominent venture capitalists, like Spencer Bogart, recognize the potential of decentralized networks in building a secure, transparent and democratic financial system, it is clear that these networks are here to stay. When you look at the ice Network, as a decentralized system, and the Pi Network, as a centralized one, there is no doubt that decentralized networks have many benefits over their centralized counterparts. Decentralization offers users more control over their data, improved resource distribution and financial independence, among other things. As such, it is easy to see why these networks are so popular in the cryptocurrency space. 

Kirsty Moreland of Ledger explains where her organization stands on the issue: “At Ledger, we strongly believe in the decentralized prowess of cryptocurrencies. We see how it’s a catalyst for bringing power and financial freedom back to individuals. Indeed, thanks to the decentralized nature of crypto assets, you can take back control of your own, hard-earned money."

If you want to be part of the ice Network, use the following invitation code because every new user must have an invitation to join: WebscrapingAPI.


We hope this article helps you understand the differences between decentralized and centralized networks. The most important thing is that you, as the user, make an informed decision about which network is right for you. Regardless of what you decide, the possibilities that decentralized networks offer are undeniable and will continue to shape the future of finance.

Decentralization at its Best: ice Network's DAO Model vs. Pi Network's Lack of Control for Users

Navigating the world of web scraping can be a bit overwhelming. You have to choose the right programming language, the right library, and deal with many unforeseen setbacks. It quickly becomes a lot to take in. But don't let that discourage you! In this article, I've answered some of the most asked questions about web scraping. You'll learn what other people are doing and the challenges they've faced. This will help guide your own decision-making process. Whether you're new to the field or a seasoned pro, there's something here for everyone.

Why can't my scraper see the same data as my browser?

You've written a script to fetch HTML from a website, but you're not getting the full data. You've tested your selectors in the browser and they should work, right? Not always. Websites that rely on JavaScript to render won't work with a simple GET request. There are libraries like Puppeteer and Selenium that use headless browsers to render JavaScript. They allow you to make the request in the context of a browser and wait for JavaScript to finish executing. This way, you can get the full HTML. You may not always need a headless browser to get the missing data. Search for <script> tags in the HTML. The missing data could be hidden inside <script> tags as JavaScript variables.

How can I scrape a website that uses generated CSS classes?

Some websites use libraries that automatically create unique class names for different page components. This can make it difficult to use traditional CSS selectors to target specific elements.

One solution is to use XPath expressions instead. XPath selectors rely on the layout of the page, rather than specific class names. This means that even if the class names change, the XPath selector will still be able to locate the desired element.

For example, if you have an HTML component that looks like this:

<div class="container">

    <div class="subcontainer_af21">

        <ul class="ul_ax1">

            <li class="li_adef">

                <a href="https://link1">Winter Storm</a>

            </li>

        </ul>

        <ul class="ul_cgt4">

            <li class="li_ocv2">

                <a href="https://lin2">SpaceX</a>

            </li>

        </ul>

    </div>

</div>

You can select the second <a> element with:

Yes, Cheerio is generally considered to be faster than Puppeteer. This is because Cheerio is a server-side library that works directly with the HTML content. Puppeteer is a browser automation library that controls a headless browser to load web pages and interact with them. Cheerio is limited in the sense that it can only work with static pages, it doesn't have the ability to interact with the browser as Puppeteer does

Are XPath selectors better than CSS selectors?

It depends on the context. If you are looking to extract data based on the position of elements, XPath is the better choice. However, if you are looking to extract data based on properties such as class or id, CSS selectors are a better option.

Both of them offer similar functionalities, but. Playwright supports multiple browsers, including Chrome, Firefox, and Safari. Puppeteer supports only Chrome and Chromium.

Playwright has better support for working with multiple tabs and windows. It also has built-in support for handling browser contexts, cookies, and storage. Playwright is better suited for complex projects.

In general, you can try to space out your requests. Use different IPs. Use proxies. Try to alter the browser fingerprint. For most people, this is a never ending battle. The good news is that it does not have to be this way. You can use our solution, WebScrapingAPI. WebScrapingAPI provides an API that will handle all the heavy lifting for you. It can execute JavaScript, rotate proxies, and even handle CAPTCHAs. You'll never have to worry about getting your IP banned. But don’t take our word for it. You can try it for free.

How to extract text from HTML with BeautifulSoup?

You can use the BeautifulSoup library. Here is an example of extracting text using the .get_text() function:

from bs4 import BeautifulSoup

html_doc = """

<html>

 <head>

   <title>title of the page</title>

 </head>

 <body>

   <p>a paragraph</p>

   <a href='https://link.com'>a link</a>

 </body>

</html>

"""

soup = BeautifulSoup(html_doc, 'html.parser')

paragraph_text = soup.find('p').text

print(paragraph_text)

#Prints 'a paragraph'

link_text = soup.find('a').text

print(link_text)

#Prints 'a link'

all_text = soup.get_text()

print(all_text)

"""

title of the page

a paragraph

a link

"""

How to extract text from HTML with Selenium?

from selenium import webdriver

from selenium.webdriver.common.by import By

DRIVER_PATH = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

# get all the h2 elements

content = driver.find_element(By.TAG_NAME, "h2")

print(content.text)

# Prints 'From today's featured article'

How to select HTML elements by text with BeautifulSoup?

With BeautifulSoup, you can use the soup.find method with the the text=re.compile("<text>") parameter:

from bs4 import BeautifulSoup

import re

html_doc = """

<html>

<body>

  <p class="my_paragraph">a paragraph.</p>

  <p class="my_paragraph">another paragraph.</p>

</body>

</html>

"""

soup = BeautifulSoup(html_doc, 'html.parser')

# find the first pTag that contains the text 'a par'

pTag = soup.find("p", text=re.compile("a par"))

print(pTag)

How to select HTML elements by text with Selenium?

from selenium import webdriver

from selenium.webdriver.common.by import By

DRIVER_PATH = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

# get all the elements with class vector-body

span = driver.find_element(By.XPATH, "//span[contains(text(), 'Did')]")

print(span.text)

# Prints 'Did you know ...'

driver.quit()

How to find HTML elements with CSS selectors in BeautifulSoup?

Here’s how you can do it with BeautifulSoup and the find and find_all methods:

from bs4 import BeautifulSoup

html_doc = """

<html>

  <body>

    <p class="my_paragraph">First paragraph.</p>

    <p class="my_paragraph">Second paragraph..</p>

    <p>Last paragraph.</p>

  </body>

</html>

"""

soup = BeautifulSoup(html_doc, 'html.parser')

# find all elements with class 'my_paragraph

elements = soup.find_all(class_="my_paragraph")

for element in elements:

    print(element.text)

# prints 'First paragraph.' and 'Second paragraph..'

How to find HTML elements by class with Selenium?

from selenium import webdriver

from selenium.webdriver.common.by import By

DRIVER_PATH = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

# get all the elements with class vector-body

elements = driver.find_elements(By.CLASS_NAME, "vector-body")

for element in elements:

   print(element.text)

driver.quit()

import requests

from bs4 import BeautifulSoup

from lxml import etree

response = requests.get("https://en.wikipedia.org/wiki/Main_Page")

soup = BeautifulSoup(response.content, 'html.parser')

dom = etree.HTML(str(body))

xpath_str = '//h1//text()'

print(dom.xpath(xpath_str))

#Prints ['Main Page', 'Welcome to ', 'Wikipedia']



How to wait for the page to load in Selenium?

If you simply want to wait for a certain time before timing out when trying to find any element, you can use the driver.implicitly_wait(time_in_secods) function:

from selenium import webdriver

from selenium.webdriver.common.by import By

DRIVER_PATH = 'C:/Users/Michael/Desktop/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.implicitly_wait(10)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

element = driver.find_element(By.ID, "not_found_id")

# the element does not exist, but it waits 10 seconds for it

text = element.text

print(text)

# Close the browser

driver.quit()

You can also choose to wait until a certain condition is met:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

DRIVER_PATH = 'C:/Users/Michael/Desktop/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

# Wait for the element with id 'content' to be present on the page

wait = WebDriverWait(driver, 10)

element = wait.until(EC.presence_of_element_located((By.ID, "content")))

element = driver.find_element(By.ID, "content")

text = element.text

print(text)

# Close the browser

driver.quit()

How to find HTML elements with CSS selectors in Puppeteer?

In Puppetter, you can use the page.$() and page.$$() functions to select elements with CSS selectors. The page.$() function is used to find the first element that matches the selector. The page.$$() function is used to find all the elements that match the selector.

How to find HTML elements with CSS selectors in Playwright?

Here is how you can do it with Playwright. It is very similar to Puppeteer:

const { chromium } = require('playwright');

(async () => {

  const browser = await chromium.launch({

    headless: false,

  });

  const context = await browser.newContext();

  const page = await context.newPage();

  await page.goto('https://www.scrapethissite.com/pages/simple/');

  // Extract the first odd row element

  const firstOddRow = await page.$('.container .row');

  console.log(await firstOddRow.textContent());

  // Extract all the odd rows

  const allOddRows = await page.$$('.container .row');

  for (const oddRow of allOddRows ) {

    console.log(await oddRow.textContent());

  }

  await browser.close();

})();

How to find HTML elements with CSS selectors in cheerio?

With cheerio, you’ll have to fetch the HTML (I used the request library to do that) and then pass it to the cheerio library:

With Puppetter, you can use the page.$x() function to select elements with XPath selectors:

const puppeteer = require('puppeteer');

(async () => {

  const browser = await puppeteer.launch();

  const page = await browser.newPage();

  await page.goto('https://www.scrapethissite.com/pages/forms/');

  // Extract the table header elements

  const allTableHeaders = await page.$x('//table/tbody/tr[1]//th');

  for(let i = 0; i < allTableHeaders.length; i++) {

    const header = await page.evaluate(el => el.textContent, allTableHeaders[i]);

    console.log(header.trim());

  }

  await browser.close();

})();

// Output:

// Team Name

// Year

// Wins

// Losses

// OT Losses

// Win %

// Goals For (GF)

// Goals Against (GA)

// + / -

const { chromium } = require('playwright');

(async () => {

  const browser = await chromium.launch({

    headless: false,

  });

  const context = await browser.newContext();

  const page = await context.newPage();

  await page.goto('https://www.scrapethissite.com/pages/forms/');

  // Extract the table header elements

  const allTableHeaders = await page.locator('xpath=//table/tbody/tr[1]//th').all();

 

  for (let i = 0; i < allTableHeaders.length; i++) {

    const headerText = await allTableHeaders[i].innerText();

    console.log(headerText);

  }

  await browser.close();

})();

Any selector string starting with // or .. are assumed to be an xpath selector. For example, Playwright converts '//html/body' to 'xpath=//html/body'. 

How to find HTML elements by text in Puppeteer?

In Puppeteer, the simplest way to find elements by text is to use the XPath text() function:

How to find HTML elements by text in Playwright?

If you want to find elements by text in Playwright, you can use the allInnerTexts() function in combination with XPath.

const { chromium } = require('playwright');

(async () => {

  const browser = await chromium.launch({

    headless: false,

  });

  const context = await browser.newContext();

  const page = await context.newPage();

  await page.goto('https://en.wikipedia.org/wiki/Web_scraping');

  // Select all the p tags texts that contain the word "prevent" 

  const pTags  = await page.locator('//p[contains(text(), "prevent")]').allInnerTexts();

 

  for (let i = 0; i < pTags.length; i++) {

    console.log(pTags[i], "\n");

  }

  await browser.close();

})();

How to find HTML elements by text in cheerio?

In Puppeteer, you can use the page.waitForSelector() function to wait for a specific element to appear on the page before continuing with the script. You can use it with both CSS and XPath selectors:

await page.waitForSelector('.basic-element', { timeout: 10000 });

await page.waitForXPath("//div[@class='basic-element']"), { timeout: 10000 });

The timeout parameter specifies the maximum wait time in ms.

You can also wait for an element to reach a certain state:

await page.waitForSelector('.basic-element', { visible: true });

// wait until the element becomes visible

Playwright is similar to Puppeteer. You can use the page.waitForSelector() method to wait for a specific element to appear on the page.

await page.waitForSelector('.element-class', { timeout: 10000 });

 await page.waitForSelector('.basic-element', { state: 'visible' });

 // wait for element to become visible

Web scraping is a vast subject and this article only covers the surface. Choosing the right tool for your specific use case is crucial. For example, if you want to scrape a website using JavaScript, the cheerio library is a good option. However, if the website requires JavaScript to load fully, Puppeteer or Playwright are better options. Web scraping is challenging but understanding the tools can save you a lot of headache. I hope this article broadened your perspective and I wish you all the best in your web scraping endeavors.

Common Questions About Web Scraping - Answers & Tips

Idealista is one of the leading real estate websites in Southern Europe, providing a wealth of information on properties for sale and rent. It’s available in Spain, Portugal and Italy, listing millions of homes, rooms and apartments.

For businesses and individuals looking to gain insights into the Spanish property market, the website can be a precious tool. Web scraping Idealista can help you extract this valuable information and use it in various ways such as market research, lead generation, and creating new business opportunities.

In this article, we will provide a step-by-step guide on how to scrape the website using TypeScript. We will cover the prerequisites, the actual scraping of properties data and how to improve the process, and why using a professional scraper is better than creating your own.

By the end of the article, you will have the knowledge and tools to extract data from Idealista and make good use of it for your business.

Before we begin, let's make sure we have the necessary tools in place.

First, download and install Node.js from the official website, making sure to use the Long-Term Support (LTS) version. This will also automatically install Node Package Manager (NPM) which we will use to install further dependencies.

For this tutorial, we will be using Visual Studio Code as our Integrated Development Environment (IDE) but you can use any other IDE of your choice. Create a new folder for your project, open the terminal and run the following command to set up a new Node.js project:

This will create a package.json file in your project directory, which will store information about your project and its dependencies.

Next, we need to install TypeScript and the type definitions for Node.js. TypeScript offers optional static typing which helps prevent errors in the code. To do this, run in the terminal:

npm install typescript @types/node --save-dev

You can verify the installation by running:

TypeScript uses a configuration file called tsconfig.json to store compiler options and other settings. To create this file in your project, run the following command:

Make sure that the value for “outDir” is set to “dist”. This way we will separate the TypeScript files from the compiled ones. You can find more information about this file and its properties in the official TypeScript documentation.

Now, create an “src” directory in your project, and a new “index.ts” file. Here is where we will keep the scraping code. To execute TypeScript code you have to compile it first, so to make sure that we don’t forget this extra step, we can use a custom-defined command.

Head over to the “package. json” file, and edit the “scripts” section like this:

"scripts": {

    "test": "npx tsc && node dist/index.js"

}

This way, when you will execute the script, you just have to type “npm run test” in your terminal.

Finally, to scrape the data from the website we will be using Puppeteer, a headless browser library for Node.js that allows you to control a web browser and interact with websites programmatically. To install it, run this command in the terminal:

It is highly recommended when you want to ensure the completeness of your data, as many websites today contain dynamic-generated content. If you’re curious, you can check out before continuing the Puppeteer documentation to fully see what it’s capable of.

Now that you have your environment set up, we can start looking at extracting the data. For this article, I chose to scrape the list of houses and apartments available in a region from Toledo, Spain: https://www.idealista.com/pt/alquiler-viviendas/toledo/buenavista-valparaiso-la-legua/.

We’re going to extract the following data from each listing on the page:

the details (number of rooms, surface, etc.);

You can see all this information highlighted in the screenshot below:

By opening the Developer Tools on each of these elements you will be able to notice the CSS selectors that we will use to locate the HTML elements. If you’re fairly new to how CSS selectors work, feel free to reach out to this beginner guide.

To start writing our script, let’s verify that the Puppeteer installation went alright:

import puppeteer from 'puppeteer';

async function scrapeIdealistaData(idealista_url: string): Promise<void> {

    

    // Launch Puppeteer

    const browser = await puppeteer.launch({

        headless: false,

    	  args: ['--start-maximized'],

    	  defaultViewport: null

    })

    // Create a new page

    const page = await browser.newPage()

    // Navigate to the target URL

    await page.goto(idealista_url)

    // Close the browser

    await browser.close()

}

scrapeIdealistaData("https://www.idealista.com/pt/alquiler-viviendas/toledo/buenavista-valparaiso-la-legua/")

Here we open a browser window, create a new page, navigate to our target URL and then close the browser. For the sake of simplicity and visual debugging, I open the browser window maximized in non-headless mode.

Since all listings have the same structure and data, we can extract all of the information for the entire properties list in our algorithm. After running the script, we can loop through all of the results and compile them into a single list.

To get the URL of all the properties, we locate the anchor elements with the “item-link” class. Then we convert the result to a JavaScript array and map each element to the value of the “href” attribute.

// Extract listings location

const listings_location = await page.evaluate(() => {

    const locations = document.querySelectorAll('a.item-link')

    const locations_array = Array.from(locations)

    return locations ? locations_array.map(a => a.getAttribute('href')) : []

})

console.log(listings_location.length, listings_location)

Recent articles

Sign up for our newsletter

News and updates