Idealista is one of the leading real estate websites in Southern Europe, providing a wealth of information on properties for sale and rent. It’s available in Spain, Portugal and Italy, listing millions of homes, rooms and apartments.

For businesses and individuals looking to gain insights into the Spanish property market, the website can be a precious tool. Web scraping Idealista can help you extract this valuable information and use it in various ways such as market research, lead generation, and creating new business opportunities.

In this article, we will provide a step-by-step guide on how to scrape the website using TypeScript. We will cover the prerequisites, the actual scraping of properties data and how to improve the process, and why using a professional scraper is better than creating your own.

By the end of the article, you will have the knowledge and tools to extract data from Idealista and make good use of it for your business.

Before we begin, let's make sure we have the necessary tools in place.

First, download and install Node.js from the official website, making sure to use the Long-Term Support (LTS) version. This will also automatically install Node Package Manager (NPM) which we will use to install further dependencies.

For this tutorial, we will be using Visual Studio Code as our Integrated Development Environment (IDE) but you can use any other IDE of your choice. Create a new folder for your project, open the terminal and run the following command to set up a new Node.js project:

This will create a package.json file in your project directory, which will store information about your project and its dependencies.

Next, we need to install TypeScript and the type definitions for Node.js. TypeScript offers optional static typing which helps prevent errors in the code. To do this, run in the terminal:

npm install typescript @types/node --save-dev

You can verify the installation by running:

TypeScript uses a configuration file called tsconfig.json to store compiler options and other settings. To create this file in your project, run the following command:

Make sure that the value for “outDir” is set to “dist”. This way we will separate the TypeScript files from the compiled ones. You can find more information about this file and its properties in the official TypeScript documentation.

Now, create an “src” directory in your project, and a new “index.ts” file. Here is where we will keep the scraping code. To execute TypeScript code you have to compile it first, so to make sure that we don’t forget this extra step, we can use a custom-defined command.

Head over to the “package. json” file, and edit the “scripts” section like this:

"scripts": {

    "test": "npx tsc && node dist/index.js"

}

This way, when you will execute the script, you just have to type “npm run test” in your terminal.

Finally, to scrape the data from the website we will be using Puppeteer, a headless browser library for Node.js that allows you to control a web browser and interact with websites programmatically. To install it, run this command in the terminal:

It is highly recommended when you want to ensure the completeness of your data, as many websites today contain dynamic-generated content. If you’re curious, you can check out before continuing the Puppeteer documentation to fully see what it’s capable of.

Now that you have your environment set up, we can start looking at extracting the data. For this article, I chose to scrape the list of houses and apartments available in a region from Toledo, Spain: https://www.idealista.com/pt/alquiler-viviendas/toledo/buenavista-valparaiso-la-legua/.

We’re going to extract the following data from each listing on the page:

the details (number of rooms, surface, etc.);

You can see all this information highlighted in the screenshot below:

By opening the Developer Tools on each of these elements you will be able to notice the CSS selectors that we will use to locate the HTML elements. If you’re fairly new to how CSS selectors work, feel free to reach out to this beginner guide.

To start writing our script, let’s verify that the Puppeteer installation went alright:

import puppeteer from 'puppeteer';

async function scrapeIdealistaData(idealista_url: string): Promise<void> {

    

    // Launch Puppeteer

    const browser = await puppeteer.launch({

        headless: false,

    	  args: ['--start-maximized'],

    	  defaultViewport: null

    })

    // Create a new page

    const page = await browser.newPage()

    // Navigate to the target URL

    await page.goto(idealista_url)

    // Close the browser

    await browser.close()

}

scrapeIdealistaData("https://www.idealista.com/pt/alquiler-viviendas/toledo/buenavista-valparaiso-la-legua/")

Here we open a browser window, create a new page, navigate to our target URL and then close the browser. For the sake of simplicity and visual debugging, I open the browser window maximized in non-headless mode.

Since all listings have the same structure and data, we can extract all of the information for the entire properties list in our algorithm. After running the script, we can loop through all of the results and compile them into a single list.

To get the URL of all the properties, we locate the anchor elements with the “item-link” class. Then we convert the result to a JavaScript array and map each element to the value of the “href” attribute.

// Extract listings location

const listings_location = await page.evaluate(() => {

    const locations = document.querySelectorAll('a.item-link')

    const locations_array = Array.from(locations)

    return locations ? locations_array.map(a => a.getAttribute('href')) : []

})

console.log(listings_location.length, listings_location)

Then, for the titles, we can make use by the same anchor element, except that this time we will extract its “title” attribute.

// Extract listings titles

const listings_title = await page.evaluate(() => {

    const titles = document.querySelectorAll('a.item-link')

    const titles_array = Array.from(titles)

    return titles ? titles_array.map(t => t.getAttribute('title')) : []

})

console.log(listings_title.length, listings_title)

For the prices, we locate the “span” elements having 2 class names: “item-price” and “h2-simulated”. It’s important to identify the elements as unique as possible, so you don’t alter your final result. It needs to be converted to an array as well and then mapped to its text content.

// Extract listings prices

const listings_price = await page.evaluate(() => {

    const prices = document.querySelectorAll('span.item-price.h2-simulated')

    const prices_array = Array.from(prices)

    return prices ? prices_array.map(p => p.textContent) : []

})

console.log(listings_price.length, listings_price)

We apply the same principle for the property details, parsing the “div” elements with the “item-detail-char” class name.

// Extract listings details

const listings_detail = await page.evaluate(() => {

    const details = document.querySelectorAll('div.item-detail-char')

    const details_array = Array.from(details)

    return details ? details_array.map(d => d.textContent) : []

})

console.log(listings_detail.length, listings_detail)

And finally, the description of the properties. Here we apply an extra regular expression to remove all the unnecessary newline characters.

// Extract listings descriptions

const listings_description = await page.evaluate(() => {

    const descriptions = document.querySelectorAll('div.item-description.description')

    const descriptions_array = Array.from(descriptions)

    return descriptions ? descriptions_array.map(d => d.textContent.replace(/(\r\n|\n|\r)/gm, "")) : []

})

console.log(listings_description.length, listings_description)

Now you should have 5 lists, one for each piece of data we scraped. As I mentioned before, we should centralize them into a single one. This way, the information we gathered will be much easier to further process.

// Group the lists

const listings = []

for (let i = 0; i < listings_location.length; i++) {

    listings.push({

        url: listings_location[i],

        title: listings_title[i],

        price: listings_price[i],

        details: listings_detail[i],

        description: listings_description[i]

    })

}

console.log(listings.length, listings)

[

  {

    url: '/pt/inmueble/99004556/',

    title: 'Apartamento em ronda de Buenavista, Buenavista-Valparaíso-La Legua, Toledo',

    price: '750€/mês',

    details: '\n3 quart.\n115 m² área bruta\n2º andar exterior com elevador\nOntem \n',

    description: 'Apartamento para alugar na Ronda Buenavista, em Toledo.Três quartos e duas casas de banho, sala, cozinha, terraço, garagem e arrecadação....'

  },

  {

    url: '/pt/inmueble/100106615/',

    title: 'Moradia em banda em Buenavista-Valparaíso-La Legua, Toledo',

    price: '1.000€/mês',

    details: '\n4 quart.\n195 m² área bruta\nOntem \n',

    description: 'Magnífica casa geminada para alugar com 3 andares, 4 quartos aconchegantes, 3 banheiros, sala ampla e luminosa, cozinha totalmente equipa...'

  },

  {

    url: '/pt/inmueble/100099977/',

    title: 'Moradia em banda em calle Francisco Ortiz, Buenavista-Valparaíso-La Legua, Toledo',

    price: '800€/mês',

    details: '\n3 quart.\n118 m² área bruta\n10 jan \n',

    description: 'O REMAX GRUPO FV aluga uma casa mobiliada na Calle Francisco Ortiz, em Toledo.Moradia geminada com 148 metros construídos, distribuídos...'

  },

  {

    url: '/pt/inmueble/100094142/',

    title: 'Apartamento em Buenavista-Valparaíso-La Legua, Toledo',

    price: '850€/mês',

    details: '\n4 quart.\n110 m² área bruta\n1º andar exterior com elevador\n10 jan \n',

    description: 'Apartamento muito espaçoso para alugar sem móveis, cozinha totalmente equipada.Composto por 4 quartos, 1 casa de banho, terraço.Calefaç...'

  }

]

If you run your script at least 2 times during the course of this tutorial, you may have already noticed this annoying page:

Idealista uses DataDome as its antibot protection, which incorporates a GeeTest CAPTCHA challenge. You’re supposed to move the piece of the puzzle until the image is complete, and then you should be redirected back to your target page.

You can easily pause your Puppeteer script until you solve the challenge using this code:

await page.waitForFunction(() => {

    const pageContent = document.getElementById('main-content')

    return pageContent !== null

}, {timeout: 10000})

This tells our script to wait 10 seconds for a specified CSS selector to appear in the DOM. It should be enough for you to solve the CAPTCHA and then let the navigation to complete.

…Unless the Idealista page will block you anyway. 

At this point, the process became more complex and challenging, and you didn’t even scale up your project.

As I mentioned before, Idealista is protected by DataDome. They collect multiple browser data to generate and associate you with a unique fingerprint. If they are suspicious, you receive the CAPTCHA challenge above, which is pretty difficult to automatically solve.

Among the collected browser data we find:

properties from the Navigator object (deviceMemory, hardwareConcurrency, languages, platform, userAgent, webdriver, etc.)

inconsistencies between the User-Agent and your operating system

One way to overcome these challenges and continue scraping at large scale is to use a scraping API. These kinds of services provide a simple and reliable way to access data from websites like Idealista.com, without the need to build and maintain your own scraper.

WebScrapingAPI is an example of such a product. Its proxy rotation mechanism avoids CAPTCHAs altogether, and its extended knowledge base makes it possible to randomize the browser data so it will look like a real user.

The setup is quick and easy. All you need to do is register an account, so you’ll receive your API key. It can be accessed from your dashboard, and it’s used to authenticate the requests you send.

As you have already set up your Node.js environment, we can make use of the corresponding SDK. Run the following command to add it to your project dependencies:

Now all it’s left to do is to adjust the previous CSS selectors to the API. The powerful feature of extraction rules makes it possible to parse data without significant modifications.

import webScrapingApiClient from 'webscrapingapi';

const client = new webScrapingApiClient("YOUR_API_KEY");

async function exampleUsage() {

    const api_params = {

        'render_js': 1,

    	  'proxy_type': 'residential',

    	  'timeout': 60000,

    	  'extract_rules': JSON.stringify({

            locations: {

                selector: 'a.item-link',

                output: '@href',

                all: '1'

        	},

        	titles: {

                selector: 'a.item-link',

                output: '@title',

                all: '1'

        	},

        	prices: {

                selector: 'span.item-price.h2-simulated',

                output: 'text',

                all: '1'

        	},

        	details: {

                selector: 'div.item-detail-char',

                output: 'text',

                all: '1'

        	},

        	descriptions: {

                selector: 'div.item-description.description',

                output: 'text',

                all: '1'

        	}

        })

    }

    const URL = "https://www.idealista.com/pt/alquiler-viviendas/toledo/buenavista-valparaiso-la-legua/"

    const response = await client.get(URL, api_params)

    if (response.success) {

        // Group the lists

    	  const listings = []

    	  for (let i = 0; i < response.response.data.locations.length; i++) {

            listings.push({

                url: response.response.data.locations[i],

                title: response.response.data.titles[i],

                price: response.response.data.prices[i],

                details: response.response.data.details[i],

                description: response.response.data.descriptions[i].replace(/(\r\n|\n|\r)/gm, "")

        	})

    	  }

    	  console.log(listings.length, listings)

    } else {

        console.log(response.error.response.data)

    }

}

exampleUsage();

In this article, we have shown you how to scrape Idealista, a popular Spanish real estate website, using TypeScript and Puppeteer. We've gone through the process of setting up the prerequisites and scraping the data, and we discussed some ways to improve the code.

Web scraping Idealista can provide valuable information for businesses and individuals. By using the techniques outlined in this article, you can extract data such as property URLs, prices, and descriptions from the website.

Additionally, If you want to avoid the antibot measures and the complexity of the scraping process, using a professional scraper can be more efficient and reliable than creating your own. 

By following the steps and techniques outlined in this guide, you can unlock the power of web scraping Idealista and use it to support your business needs. Whether it's for market research, lead generation, or creating new business opportunities, web scraping Idealista can help you stay ahead of the competition.

How to Web Scrape Idealista: A Comprehensive Guide (2023 Update)

Yelp is a platform that allows users to search for businesses, read reviews, and even make reservations. It is a popular website with millions of monthly visitors, making it an ideal target for data scraping.

Knowing how to web scrape Yelp can be a powerful tool for businesses and entrepreneurs looking to gather valuable information about the local market.

In this article, we will explore the advantages of web scraping Yelp, including information on how to set up the environment, locate the data, and extract valuable information.

We will also look at the potential business ideas that can be created using this scraped data, and why using a professional scraper is better than creating your own. By the end of this article, you will have a solid understanding of how to web scrape Yelp.

Now that you have your environment set up, we can start looking at extracting the data. For this article, I chose to scrape the page of an Irish restaurant from Dublin: https://www.yelp.ie/biz/the-boxty-house-dublin?osq=Restaurants.

We’re going to extract the following data:

Before writing our script, let’s verify that the Puppeteer installation went alright:

import puppeteer from 'puppeteer';

async function scrapeYelpData(yelp_url: string): Promise<void> {

    // Launch Puppeteer

    const browser = await puppeteer.launch({

        headless: false,

    	  args: ['--start-maximized'],

    	  defaultViewport: null

    })

    // Create a new page

    const page = await browser.newPage()

    // Navigate to the target URL

    await page.goto(yelp_url)

    // Close the browser

    await browser.close()

}

scrapeYelpData("https://www.yelp.ie/biz/the-boxty-house-dublin?osq=Restaurants")

Now, let’s take a look at the website’s structure:

It seems that Yelp displays a somewhat difficult page structure, as the class names are randomly generated and very few elements have unique attribute values.

But fear not, we can get creative with the solution. Firstly, to get the restaurant name, we target the only “h1” element present on the page.

// Extract restaurant name

const restaurant_name = await page.evaluate(() => {

    const name = document.querySelector('h1')

    return name ? name.textContent : ''

})

console.log(restaurant_name)

Now, to get the restaurant rating, you can notice that beyond the star icons, the explicit value is present in the attribute “aria-label”. So, we target the “div” element whose “aria-label” attribute ends with the “star rating” string.

// Extract restaurant rating

const restaurant_rating = await page.evaluate(() => {

    const rating = document.querySelector('div[aria-label$="star rating"]')

    return rating ? rating.getAttribute('aria-label') : ''

})

console.log(restaurant_rating)

And finally (for this particular HTML section), we see that we can easily get the reviews number by targeting the highlighted anchor element.

// Extract restaurant reviews

const restaurant_reviews = await page.evaluate(() => {

    const reviews = document.querySelector('a[href="#reviews"]')

    return reviews ? reviews.textContent : ''

})

console.log(restaurant_reviews)

Easy peasy. Let’s take a look at the business information widget:

Unfortunately, in this situation, we cannot rely on CSS selectors. Luckily, we can make use of another method to locate the HTML elements: XPath. If you’re fairly new to how CSS selectors work, feel free to reach out to this beginner guide.

To extract the restaurant’s website: we apply the following logic:

locate the “p” element that has “Business website” as text content;

locate the anchor element and its “href” attribute.

// Extract restaurant website

const restaurant_website_element = await page.$x("//p[contains(text(), 'Business website')]/following-sibling::p/a/@href")

const restaurant_website = await page.evaluate(

    element => element.nodeValue,

    restaurant_website_element[0]

)

console.log(restaurant_website)

Now, for the phone number and the address we can follow the exact same logic, with 2 exceptions:

for the phone number, we stop the following sibling and extract its textContent property;

for the address, we target the following sibling of the parent element.

// Extract restaurant phone number

const restaurant_phone_element = await page.$x("//p[contains(text(), 'Phone number')]/following-sibling::p")

const restaurant_phone = await page.evaluate(

    element => element.textContent,

    restaurant_phone_element[0]

)

console.log(restaurant_phone)

// Extract restaurant address

const restaurant_address_element = await page.$x("//a[contains(text(), 'Get Directions')]/parent::p/following-sibling::p")

const restaurant_address = await page.evaluate(

    element => element.textContent,

    restaurant_address_element[0]

)

console.log(restaurant_address)

The Boxty House

4.5 star rating

948 reviews

/biz_redir?url=http%3A%2F%2Fwww.boxtyhouse.ie%2F&cachebuster=1673542348&website_link_type=website&src_bizid=EoMjdtjMgm3sTv7dwmfHsg&s=16fbda8bbdc467c9f3896a2dcab12f2387c27793c70f0b739f349828e3eeecc3

(01) 677 2762

20-21 Temple Bar Dublin 2

While scraping Yelp may seem easy at first, the process can become more complex and challenging as you scale up your project. The website implements various techniques to detect and prevent automated traffic, so your scaled-up scraper starts getting blocked.

Yelp collects multiple browser data to generate and associate you with a unique fingerprint. Some of these are:

properties from the Navigator object (deviceMemory, hardwareConcurrency, platform, userAgent, webdriver, etc.)

One way to overcome these challenges and continue scraping at large scale is to use a scraping API. These kinds of services provide a simple and reliable way to access data from websites like yelp.com, without the need to build and maintain your own scraper.

Now all it’s left to do is to send a GET request so we receive the website’s HTML document. Note that this is not the only way you can access the API.

import webScrapingApiClient from 'webscrapingapi';

const client = new webScrapingApiClient("YOUR_API_KEY");

async function exampleUsage() {

    const api_params = {

        'render_js': 1,

    	  'proxy_type': 'residential',

    }

    const URL = "https://www.yelp.ie/biz/the-boxty-house-dublin?osq=Restaurants"

    const response = await client.get(URL, api_params)

    if (response.success) {

        console.log(response.response.data)

    } else {

        console.log(response.error.response.data)

    }

}

exampleUsage();

By enabling the “render_js” parameter, we send the request using a headless browser, just like you previously did along with this tutorial.

After receiving the HTML document, you can use another library to extract the data of interest, like Cheerio. Never heard of it? Check out this guide to help you get started!

This article has presented you with a comprehensive guide on how to web scrape Yelp using TypeScript and Puppeteer. We have gone through the process of setting up the environment, locating and extracting data, and why using a professional scraper is a better solution than creating your own.

The data scraped from Yelp can be used for various purposes such as, identifying market trends, analyzing customer sentiment, monitoring competitors, creating targeted marketing campaigns, and many more.

Overall, web scraping Yelp.com can be a valuable asset for anyone looking to gain a competitive advantage in their local market and this guide has provided a great starting point to do so.

How to Web Scrape Yelp.com (2023 Update) - A Step-by-Step Guide

Have you ever had to work with POST requests while web scraping? I am sure you have! And it’s forms we have to handle most of the time. That is why today, I am going to talk about how to submit form with Puppeteer. If you don’t know what puppeteer is yet, don’t worry. You’ll find out in just a moment. Until then, let me set some expectations for today’s article. If you follow me in our learning path, today you should be able to learn:

How is form submission handled in Puppeteer

So, without further ado, let’s get to it!

What is Puppeteer and why is it important for web scraping?

In general, web scraping refers to the process of automating data extraction from various servers. Back in the days, a simple HTTP client would have been enough to perform this task. Nowaday though, websites rely more and more on JavaScript. And traditional HTTP clients are unable to render JavaScript files. That is where Puppeteer comes into play.

Puppeteer is a Node.js library that allows you to control a headless Chrome or Chromium browser via the DevTools Protocol. Long story short, it provides a high level API to automate Chrome. 

In terms of web scraping, Puppeteer is useful for scraping websites that require JavaScript to be rendered. Additionally, it can also be used to interact with web pages in a way similar to a human. For example, clicking buttons or our focus for today, filling out forms. This makes it ideal for scraping websites that use anti-scraping techniques.

I believe in taking things slow for better understanding of the overall process. Before going into how to submit form with Puppeteer, let’s talk simple Puppeteer. In this section, I am going to show you how to set up a Node project, install Puppeteer and use it to scrape data. So, first thing first, let’s create a new folder and open it inside our desired IDE. I prefer Visual Studio Code, but feel free to use whatever you want. 

You can ‘programmatically’ create a new folder from your terminal by typing the `mkdir` command. 

You can use the `npm init -y` command to setup a node project and accept default values

You can create a new file with the `touch` command. 

And you can also open VSCode with the `code .` command. 

If you want, you can combine the four and spin up a project in seconds like this:

~ » mkdir scraper && cd scraper && npm init -y && code . 

Inside your IDE, open up a new terminal (Terminal > New Terminal) and let’s install Puppeteer. Type `npm i puppeteer --save` inside your terminal. Also, I like using JS modules instead of CommonJS. Check out the differences between the two here. If you want to use modules too, open `package. json` and add `"type": "module"` to the JSON object.

Now that we’re all set up, we can start adding some code. Create a new `index.js` file and open it in the IDE. No need to do it from the terminal this time, but just as a hint, you could use the `touch` command.  Now let’s add the code:

import puppeteer, { executablePath } from 'puppeteer'

const scraper = async (url) => {

   const browser = await puppeteer.launch({

       headless: false,

       executablePath: executablePath(),

   })

   const page = await browser.newPage()

   await page.goto(url)

   const html = await page.content()

   await browser.close()

   return html

}

We’re importing Puppeteer and `executablePath` into our project

We’re defining a new function, that takes one `url` parameter

We’re launching a new browser using `puppeteer.launch`
     a. We’re specifying we want it to run headfull
     b. We’re using `executablePath` to get the Chrome path

We’re opening a new page and navigating to the `url`

We’re saving the `page.content()` in a constant

And finally we’re returning the `html` output of the page we just scraped

So far things are not complicated. This is the bare minimum of a web scraper implementation with Node JS and Puppeteer. If you want to run the code, simply give the `scraper` function a target and log its return value:

console.log(await scraper('https://webscrapingapi.com/'))

But remember our goal is to extract data upon submitting a form. Which means we have to think of a way to submit form with Puppeteer. Luckily, I’ve done it before and I know it’s not hard. So let’s see how you can do it too. 

Think of Puppeteer as the means to mimic human behavior on a given website. How do we, humans, submit forms? Well, we identify the form, we fill it in and usually click a button. That is the same logic used to submit form with Puppeteer. The only difference is how we perform these actions. Because humans rely on senses. Since puppeteer is software, we’ll do it programmatically, using puppeteer’s built in methods, like so:

First things first, we need to ‘visualize’ our form. In a website, all elements are grouped in an HTML block and every element has an identifier. Identifiers usually consist of CSS attributes of the element. Yet, you  may come across websites that don’t have such selectors. In such scenarios, you can use xPaths for example. But that is a subject for another talk. Let’s focus on identifying elements in Puppeteer using CSS.

To have some sort of background, let’s say we want to automate the login action on the Stack Overflow. So the target is https://stackoverflow.com/users/login. Open up your browser, navigate to the login page and open Developer Tools. You can right click on the page and select ‘Inspect’. You should see something like this:

On the left side, there is the graphical interface. On the right side, there is the HTML structure. If you look closely on the right side, you will see our form. It mainly consists of two inputs and one button. These are the three elements we are targeting. And as you can see, all three elements have an `id` as a CSS identifier. Let’s translate what we’ve learned so far in code:

import puppeteer, { executablePath } from 'puppeteer'

const scraper = async (target) => {

   const browser = await puppeteer.launch({

       headless: false,

       executablePath: executablePath(),

   })

   const page = await browser.newPage()

   await page.goto(target.url,{waitUntil: 'networkidle0'})

   await page.type(target.username.selector, target.username.value)

   await page.type(target.password.selector, target.password.value)

   await page.click(target.buttonSelector)

   const html = await page.content()

   await browser.close()

   return html

}

In order to keep it functional and reusable, I chose to replace my function’s parameter with an object. This object consists of the targeted URL, the inputs selectors and values and the selector for the submit button. So, to run the code, just create a new `TARGET` object that holds your data, and pass it to your `scraper` function:

const TARGET = {

   url: 'https://stackoverflow.com/users/login',

   username: {

       selector: 'input[id=email]',

       value: '<YOUR_USERNAME>'

   },

   password: {

       selector: 'input[id=password]',

       value: '<YOUR_PASSWORD>'

   },

   buttonSelector: 'button[id=submit-button]'

}

console.log(await scraper(TARGET))

Sometimes, web automation requires us to upload files, rather than submitting simple forms. If you come across such a task and need to attach files before you submit form with Puppeteer, you will want to make use of Puppeteer’s `uploadFile` method. To keep things simple, I suggest you create a new function for this action:

const upload = async (target) => {

   const browser = await puppeteer.launch({

       headless: false,

       executablePath: executablePath(),

   })

   const page = await browser.newPage()

   await page.goto(target.url,{waitUntil: 'networkidle0'})

   const upload = await page.$(target.form.file)

   await upload.uploadFile(target.file);

   await page.click(target.form.submit)

  

   await browser.close()

}

See how this time I am using `page.$` to first identify the element. And only after that I am calling the `uploadFile` method which only works on `ElementHandle` types. Parameter wise, just like previously, I am using an object to pass all data at once to my function. If you want to test the script, just add the following code and run `node index.js` in your terminal:

const TARGET = {

   url: 'https://ps.uci.edu/~franklin/doc/file_upload.html',

   form: {

       file: 'input[type=file]',

       submit: 'input[type=submit]'

   } ,

   file: './package.json'

}

upload(TARGET)

Wrapping everything up, I’d say it’s quite easy to submit form with Puppeteer. Moreover, I find that compared to its alternatives, Puppeteer fully handles this action. Basically, all the user has to do is properly identify elements.

Now, I should note that a real world scraper requires much more in order to be efficient. Most of the time, if you ‘abuse’ a server by submitting too many forms in a short period of time, you will probably get blocked.  That is why, if you want to automate the process of form submission, I recommend using a professional scraping service. At Web Scraping API, we do offer the option of sending POST & PUT requests You can read more about it in our documentation.

How To Create A Scraper And Submit A Form With Puppeteer

When it comes to Python and web automation, Selenium was pretty much the go-to. At least up until now. Because of Puppeteer’s success in the JavaScript community, Python developers started to look more and more into it. And that is how Pyppeteer came to existence. But what exactly is Pyppeteer? And why should we choose it over Selenium? Is it reliable enough to build a complex solution using it? To all these questions and many more, we will answer in today’s article. My aim for today is, if you read through this material, you should leave with at least:

A definition of Pyppeteer and its use cases

An understanding of how Pyppeteer compares to Selenium

An actual implementation of a web scraper using Pyppeteer

So get yourself ready, because today we’ll talk and get our hands on some coding!

What is Pyppeteer really and how can you use it?

If you’re reading this, chances are you are already familiar with what web scripting is in general. And you’ve probably already heard of Puppeteer or Selenium, depending on your favorite programming language. But Pyppeteer is indeed newer to the scene of web scraping. Well, to cut it short, Pyppeteer is much more like Puppeteer that it is like Selenium.

Puppeteer is a Node.js library that facilitates control of a headless version of Chrome via the DevTools protocol. Pyppeteer is a Python port of Puppeteer. Just like the original Puppeteer, Pyppeteer is a library, written in Python, that basically automates a browser. In other words, Pyppeteer is a Python implementation of the Puppeteer API, which allows you to use Puppeteer's features in a Python environment. The main difference between the two is the language being used.

Before moving forward, I think we should discuss some terms commonly used in the context of Pyppeteer:

Headless: It means starting a browser without a graphical user interface (GUI). In other words, it's running "behind the scenes" and you can't see it on the screen. It’s usually used to reduce resource usage while scraping.

Headful: Conversely, a “headful” browser is one that is running with a GUI. This is the opposite of a headless browser and It is often used for testing, debugging or interacting with web pages manually.

Browser Context: Is a state shared among all pages in a browser. It’s usually used to set browser-wide settings, such as cookies, HTTP headers, and geolocation.

DOM: The Document Object Model (DOM) is a programming interface for HTML and XML documents. It represents the structure of a web page in a tree-like format, with nodes that represent elements. Pyppeteer allows you to interact with the elements of a page by manipulating the DOM.

Elements: The building blocks of a web page. They are defined using tags, attributes, and values.

Of course, there’s more to it and you’ll learn some more along the way. But I wanted you to get a grasp of it so that we have a solid start. I’m satisfied that knowing these terms will help you better understand the essence of this article.

Why use Pyppeteer in your scraping project?

I think there are two aspects of this matter. The first one is why Pyppeteer is a good choice for web scraping in general. The second one is why use Pyppeteer over Selenium. Generally, some of the advantages of Pyppeteer account for:

Evaluating JavaScript: Pyppeteer provides a `page.evaluate()` function. It allows you to execute JavaScript code within the context of the page.

Network control: Pyppeteer provides a `page.on()` method. This allows you to listen for network events, such as requests and responses, that happen on a page.

Tracing and Logging: Pyppeteer allows you to trace the browser’s activity and log browser messages from a page. This makes it easy to debug, trace and understand what a website is doing.

Compared to Selenium, it is quite similar, in that both are used to automate a web browser. However, there are a few key differences and advantages that Pyppeteer has over Selenium:

Simplicity: Pyppeteer has a simpler and more consistent API than Selenium, which makes it easier to use for beginners. Pyppeteer API is built on top of the DevTools protocol which is close to the browser and it is easy to learn and use.

Performance: Pyppeteer can be faster than Selenium because it is built on top of the DevTools protocol. The protocol is designed for debugging web pages and it is much faster than Selenium WebDriver.

Better Network Control: Pyppeteer allows for greater control over the browser’s network settings such as request interception and request/response blocking. This makes it easier to test and diagnose network-related issues.

And of course there is also a matter of choice. Take me for example. On a day to day basis, I code in JavaScript. And I am quite familiar with Puppeteer. But my favorite programming language is Python on the other hand. So if I were to build a Scraper with a known technology in a language I prefer, I’d probably go with Pyppeteer.

And with that said, I think we covered the ‘talk’ aspects of this article. It’s time to start with some actual coding.

How to create a web scraper with Pyppeteer

Before we start coding, let me introduce you to the Pyppeteer official documentation. I am an advocate of using the official documentation whenever one feels stuck. That’s before asking questions in the community (like on Stackoverflow). I usually find that most answers can be found if you just read the docs first. So take this as a kind ask from me. Whenever you’re stuck, check the documentation, then search for answers and only ask questions as your last resort.

First things first, as a Python developer, you’re probably familiar with virtual environments. So the first thing we need to do is create a virtual environment for our project. This is generally the sequence of commands I use:

# Create a new directory and navigate into it

~ » mkdir py_project && cd py_project 

# Create the virtual environment

~ » python3 -m venv env 

# Activate the virtual environment

~ » source env/bin/activate

With regards to the virtual environment, you’re all set up now. It’s time to move on and install Pyppeteer. Since you’ve got your terminal open, just type:

# Install the package using pip

~ » python3 -m pip install pyppeteer

# Open the project in your IDE

~ » code .

The last command opens Visual Studio Code, or your preferred IDE. So now that you’re in the ‘development environment’, let’s create a new `.py` file that will hold our code. I’ll call my file `scraper.py`. Note that Pyppeteer natively supports asynchronous execution. So let’s import both `asyncio` and `pyppeteer` into our file:

import asyncio
from pyppeteer import launch

With this done, we can move ahead to more complicated code. In general, I am not the biggest advocate of functional programming. Yet, I think splitting code in small chunks allows for better learning. So let’s wrap our code inside a function:

async def scrape(url):

   browser = await launch()

   page = await browser.newPage()

   await page.goto(url)

   content = await page.content()

   await browser.close()

  

   return content

See how the syntax is almost identical to the one used in this (AICI TREBUIE POSTAT ARTICOLUL ASTA:https://docs.google.com/document/d/1oUekhOuW7dobPcWBv1mLqh044-e0Afh_IGqenvrrnTs/edit#heading=h.pj0e3c52extb) simple Puppeteer project: 

This function takes a URL as an input and it launches a headless browser using pyppeteer. Then, it navigates to the provided URL, retrieves the page's content, and closes the browser. The value it returns is nothing else than the HTML collected from the page. You can use this function to scrape almost any website. To use the function, you would call it in an `asyncio` event loop, like this:

async def main():

   content = await scrape('https://www.example.com')

   print(content)

loop = asyncio.get_event_loop()

loop.run_until_complete(main())

Up to this point, we have a working scraper. But that is pretty much all we have. If you want to build a more advanced web scraper with Pyppeteer, you will have to add more functionality to id. Spoiler alert: we’ll dive into the world of object oriented programming. But first let’s trace our objectives. What do we want our scraper to be able to do?

Initialize the browser with some custom values

Navigate and extract content from a web page

So let’s create a new `Scraper` class for now and we’ll add its methods afterward:

class Scraper:

   def __init__(self, launch_options: dict) -> None:

   	self.options = launch_options['options']

 	self.viewPort = launch_options['viewPort'] if 'viewPort' in launch_options else None

pass

The only argument we’re using for our Scraper is a `launch_options` dictionary. As you see, it holds two keys inside it. One key defines Pyppeteer’s launcher options. The second option is either `None` or a dictionary holding the `width` and `height` of the `viewPort`. The latter is used for this method.

If you look at the function we used before, you’ll see that we cover both navigating and extracting raw data from a specific URL. The only thing we need to do is to tweak and turn the function into a method for our Scraper:

async def goto(self, url: str) -> None:

       self.browser = await launch(options=self.options)

       self.page = await self.browser.newPage()

       await self.page.setViewport(self.viewPort) if self.viewPort != None else print('[i] Using default viewport')

       await self.page.goto(url)

This method is quite simple. First, it launches a new browser, with the custom options we set before. It then creates a new page and, if our `launch_options` dictionary has `viewPort`, then it sets the page’s viewPort. Otherwise, it logs a simple message. Last but not least, it takes us to the target. 

Again, we have the method in our initial `scraper` function. We will only await for the `page.content()` to load and return its value:

async def get_full_content(self) -> str:

       content = await self.page.content()

       return content

In order to write something to an input field using Pyppeteer, you’ll need two things. First, locate the element. Second, add some value to it. Luckily, Pyppeteer has methods for both these actions:

async def type_value(self, selector: str, value: str) -> None:

       element = await self.page.querySelector(selector)

       await element.type(value)

Remember we want to be able to extract either the value from a single element, or values from multiple elements. We could use one single method for both. But I usually like things separated. So for now, I’ll add two more methods:

async def extract_one(self, selector) -> str:

       element = await self.page.querySelector(selector)

       text = await element.getProperty("textContent")

       return await text.jsonValue()

Here, we are locating the element using the `querySelector` method. We then await for the `textContent` and return its `jsonValue()`. On the other hand, when we want to select many elements, we’ll use `querySelector`:

async def extract_many(self, selector) -> list:

       result = []

       elements = await self.page.querySelectorAll(selector)

       for element in elements:

           text = await element.getProperty("textContent")

           result.append(await text.jsonValue())

       return result

This method works in a similar way to `extract_one`. The only difference is its return value. This time we’re returning a list of all text inside the selected elements. And I guess with this added, we touched on all our goals. 

In web scraping, stealthiness can be described as the ability to go undetected. Of course, building a fully undetectable scraper takes a lot of work. For example, Web Scraping API’s Stealth Mode is maintained by a dedicated team. And the effort put into it makes our scraper’s fingerprint unique with every request. 

But my overall aim for this tutorial is to set you on the right path. And the right path for a complete web scraper with Pyppeteer implies adding some stealth functionality to it. Lucky enough, just like there is `puppeteer-extra-plugin-stealth` in Node, there is a package for Python too. And it's called intuitively, `pyppeteer-stealth`. To add it to your project, first install it using pip:

All Guides articles

News and updates