Have you ever had to work with POST requests while web scraping? I am sure you have! And it’s forms we have to handle most of the time. That is why today, I am going to talk about how to submit form with Puppeteer. If you don’t know what puppeteer is yet, don’t worry. You’ll find out in just a moment. Until then, let me set some expectations for today’s article. If you follow me in our learning path, today you should be able to learn:

How is form submission handled in Puppeteer

So, without further ado, let’s get to it!

What is Puppeteer and why is it important for web scraping?

In general, web scraping refers to the process of automating data extraction from various servers. Back in the days, a simple HTTP client would have been enough to perform this task. Nowaday though, websites rely more and more on JavaScript. And traditional HTTP clients are unable to render JavaScript files. That is where Puppeteer comes into play.

Puppeteer is a Node.js library that allows you to control a headless Chrome or Chromium browser via the DevTools Protocol. Long story short, it provides a high level API to automate Chrome. 

In terms of web scraping, Puppeteer is useful for scraping websites that require JavaScript to be rendered. Additionally, it can also be used to interact with web pages in a way similar to a human. For example, clicking buttons or our focus for today, filling out forms. This makes it ideal for scraping websites that use anti-scraping techniques.

I believe in taking things slow for better understanding of the overall process. Before going into how to submit form with Puppeteer, let’s talk simple Puppeteer. In this section, I am going to show you how to set up a Node project, install Puppeteer and use it to scrape data. So, first thing first, let’s create a new folder and open it inside our desired IDE. I prefer Visual Studio Code, but feel free to use whatever you want. 

You can ‘programmatically’ create a new folder from your terminal by typing the `mkdir` command. 

You can use the `npm init -y` command to setup a node project and accept default values

You can create a new file with the `touch` command. 

And you can also open VSCode with the `code .` command. 

If you want, you can combine the four and spin up a project in seconds like this:

~ » mkdir scraper && cd scraper && npm init -y && code . 

Inside your IDE, open up a new terminal (Terminal > New Terminal) and let’s install Puppeteer. Type `npm i puppeteer --save` inside your terminal. Also, I like using JS modules instead of CommonJS. Check out the differences between the two here. If you want to use modules too, open `package. json` and add `"type": "module"` to the JSON object.

Now that we’re all set up, we can start adding some code. Create a new `index.js` file and open it in the IDE. No need to do it from the terminal this time, but just as a hint, you could use the `touch` command.  Now let’s add the code:

import puppeteer, { executablePath } from 'puppeteer'

const scraper = async (url) => {

   const browser = await puppeteer.launch({

       headless: false,

       executablePath: executablePath(),

   })

   const page = await browser.newPage()

   await page.goto(url)

   const html = await page.content()

   await browser.close()

   return html

}

We’re importing Puppeteer and `executablePath` into our project

We’re defining a new function, that takes one `url` parameter

We’re launching a new browser using `puppeteer.launch`
     a. We’re specifying we want it to run headfull
     b. We’re using `executablePath` to get the Chrome path

We’re opening a new page and navigating to the `url`

We’re saving the `page.content()` in a constant

And finally we’re returning the `html` output of the page we just scraped

So far things are not complicated. This is the bare minimum of a web scraper implementation with Node JS and Puppeteer. If you want to run the code, simply give the `scraper` function a target and log its return value:

console.log(await scraper('https://webscrapingapi.com/'))

But remember our goal is to extract data upon submitting a form. Which means we have to think of a way to submit form with Puppeteer. Luckily, I’ve done it before and I know it’s not hard. So let’s see how you can do it too. 

Think of Puppeteer as the means to mimic human behavior on a given website. How do we, humans, submit forms? Well, we identify the form, we fill it in and usually click a button. That is the same logic used to submit form with Puppeteer. The only difference is how we perform these actions. Because humans rely on senses. Since puppeteer is software, we’ll do it programmatically, using puppeteer’s built in methods, like so:

First things first, we need to ‘visualize’ our form. In a website, all elements are grouped in an HTML block and every element has an identifier. Identifiers usually consist of CSS attributes of the element. Yet, you  may come across websites that don’t have such selectors. In such scenarios, you can use xPaths for example. But that is a subject for another talk. Let’s focus on identifying elements in Puppeteer using CSS.

To have some sort of background, let’s say we want to automate the login action on the Stack Overflow. So the target is https://stackoverflow.com/users/login. Open up your browser, navigate to the login page and open Developer Tools. You can right click on the page and select ‘Inspect’. You should see something like this:

On the left side, there is the graphical interface. On the right side, there is the HTML structure. If you look closely on the right side, you will see our form. It mainly consists of two inputs and one button. These are the three elements we are targeting. And as you can see, all three elements have an `id` as a CSS identifier. Let’s translate what we’ve learned so far in code:

import puppeteer, { executablePath } from 'puppeteer'

const scraper = async (target) => {

   const browser = await puppeteer.launch({

       headless: false,

       executablePath: executablePath(),

   })

   const page = await browser.newPage()

   await page.goto(target.url,{waitUntil: 'networkidle0'})

   await page.type(target.username.selector, target.username.value)

   await page.type(target.password.selector, target.password.value)

   await page.click(target.buttonSelector)

   const html = await page.content()

   await browser.close()

   return html

}

In order to keep it functional and reusable, I chose to replace my function’s parameter with an object. This object consists of the targeted URL, the inputs selectors and values and the selector for the submit button. So, to run the code, just create a new `TARGET` object that holds your data, and pass it to your `scraper` function:

const TARGET = {

   url: 'https://stackoverflow.com/users/login',

   username: {

       selector: 'input[id=email]',

       value: '<YOUR_USERNAME>'

   },

   password: {

       selector: 'input[id=password]',

       value: '<YOUR_PASSWORD>'

   },

   buttonSelector: 'button[id=submit-button]'

}

console.log(await scraper(TARGET))

Sometimes, web automation requires us to upload files, rather than submitting simple forms. If you come across such a task and need to attach files before you submit form with Puppeteer, you will want to make use of Puppeteer’s `uploadFile` method. To keep things simple, I suggest you create a new function for this action:

const upload = async (target) => {

   const browser = await puppeteer.launch({

       headless: false,

       executablePath: executablePath(),

   })

   const page = await browser.newPage()

   await page.goto(target.url,{waitUntil: 'networkidle0'})

   const upload = await page.$(target.form.file)

   await upload.uploadFile(target.file);

   await page.click(target.form.submit)

  

   await browser.close()

}

See how this time I am using `page.$` to first identify the element. And only after that I am calling the `uploadFile` method which only works on `ElementHandle` types. Parameter wise, just like previously, I am using an object to pass all data at once to my function. If you want to test the script, just add the following code and run `node index.js` in your terminal:

const TARGET = {

   url: 'https://ps.uci.edu/~franklin/doc/file_upload.html',

   form: {

       file: 'input[type=file]',

       submit: 'input[type=submit]'

   } ,

   file: './package.json'

}

upload(TARGET)

Wrapping everything up, I’d say it’s quite easy to submit form with Puppeteer. Moreover, I find that compared to its alternatives, Puppeteer fully handles this action. Basically, all the user has to do is properly identify elements.

Now, I should note that a real world scraper requires much more in order to be efficient. Most of the time, if you ‘abuse’ a server by submitting too many forms in a short period of time, you will probably get blocked.  That is why, if you want to automate the process of form submission, I recommend using a professional scraping service. At Web Scraping API, we do offer the option of sending POST & PUT requests You can read more about it in our documentation.

How To Create A Scraper And Submit A Form With Puppeteer

When it comes to Python and web automation, Selenium was pretty much the go-to. At least up until now. Because of Puppeteer’s success in the JavaScript community, Python developers started to look more and more into it. And that is how Pyppeteer came to existence. But what exactly is Pyppeteer? And why should we choose it over Selenium? Is it reliable enough to build a complex solution using it? To all these questions and many more, we will answer in today’s article. My aim for today is, if you read through this material, you should leave with at least:

A definition of Pyppeteer and its use cases

An understanding of how Pyppeteer compares to Selenium

An actual implementation of a web scraper using Pyppeteer

So get yourself ready, because today we’ll talk and get our hands on some coding!

What is Pyppeteer really and how can you use it?

If you’re reading this, chances are you are already familiar with what web scripting is in general. And you’ve probably already heard of Puppeteer or Selenium, depending on your favorite programming language. But Pyppeteer is indeed newer to the scene of web scraping. Well, to cut it short, Pyppeteer is much more like Puppeteer that it is like Selenium.

Puppeteer is a Node.js library that facilitates control of a headless version of Chrome via the DevTools protocol. Pyppeteer is a Python port of Puppeteer. Just like the original Puppeteer, Pyppeteer is a library, written in Python, that basically automates a browser. In other words, Pyppeteer is a Python implementation of the Puppeteer API, which allows you to use Puppeteer's features in a Python environment. The main difference between the two is the language being used.

Before moving forward, I think we should discuss some terms commonly used in the context of Pyppeteer:

Headless: It means starting a browser without a graphical user interface (GUI). In other words, it's running "behind the scenes" and you can't see it on the screen. It’s usually used to reduce resource usage while scraping.

Headful: Conversely, a “headful” browser is one that is running with a GUI. This is the opposite of a headless browser and It is often used for testing, debugging or interacting with web pages manually.

Browser Context: Is a state shared among all pages in a browser. It’s usually used to set browser-wide settings, such as cookies, HTTP headers, and geolocation.

DOM: The Document Object Model (DOM) is a programming interface for HTML and XML documents. It represents the structure of a web page in a tree-like format, with nodes that represent elements. Pyppeteer allows you to interact with the elements of a page by manipulating the DOM.

Elements: The building blocks of a web page. They are defined using tags, attributes, and values.

Of course, there’s more to it and you’ll learn some more along the way. But I wanted you to get a grasp of it so that we have a solid start. I’m satisfied that knowing these terms will help you better understand the essence of this article.

Why use Pyppeteer in your scraping project?

I think there are two aspects of this matter. The first one is why Pyppeteer is a good choice for web scraping in general. The second one is why use Pyppeteer over Selenium. Generally, some of the advantages of Pyppeteer account for:

Evaluating JavaScript: Pyppeteer provides a `page.evaluate()` function. It allows you to execute JavaScript code within the context of the page.

Network control: Pyppeteer provides a `page.on()` method. This allows you to listen for network events, such as requests and responses, that happen on a page.

Tracing and Logging: Pyppeteer allows you to trace the browser’s activity and log browser messages from a page. This makes it easy to debug, trace and understand what a website is doing.

Compared to Selenium, it is quite similar, in that both are used to automate a web browser. However, there are a few key differences and advantages that Pyppeteer has over Selenium:

Simplicity: Pyppeteer has a simpler and more consistent API than Selenium, which makes it easier to use for beginners. Pyppeteer API is built on top of the DevTools protocol which is close to the browser and it is easy to learn and use.

Performance: Pyppeteer can be faster than Selenium because it is built on top of the DevTools protocol. The protocol is designed for debugging web pages and it is much faster than Selenium WebDriver.

Better Network Control: Pyppeteer allows for greater control over the browser’s network settings such as request interception and request/response blocking. This makes it easier to test and diagnose network-related issues.

And of course there is also a matter of choice. Take me for example. On a day to day basis, I code in JavaScript. And I am quite familiar with Puppeteer. But my favorite programming language is Python on the other hand. So if I were to build a Scraper with a known technology in a language I prefer, I’d probably go with Pyppeteer.

And with that said, I think we covered the ‘talk’ aspects of this article. It’s time to start with some actual coding.

How to create a web scraper with Pyppeteer

Before we start coding, let me introduce you to the Pyppeteer official documentation. I am an advocate of using the official documentation whenever one feels stuck. That’s before asking questions in the community (like on Stackoverflow). I usually find that most answers can be found if you just read the docs first. So take this as a kind ask from me. Whenever you’re stuck, check the documentation, then search for answers and only ask questions as your last resort.

First things first, as a Python developer, you’re probably familiar with virtual environments. So the first thing we need to do is create a virtual environment for our project. This is generally the sequence of commands I use:

# Create a new directory and navigate into it

~ » mkdir py_project && cd py_project 

# Create the virtual environment

~ » python3 -m venv env 

# Activate the virtual environment

~ » source env/bin/activate

With regards to the virtual environment, you’re all set up now. It’s time to move on and install Pyppeteer. Since you’ve got your terminal open, just type:

# Install the package using pip

~ » python3 -m pip install pyppeteer

# Open the project in your IDE

~ » code .

The last command opens Visual Studio Code, or your preferred IDE. So now that you’re in the ‘development environment’, let’s create a new `.py` file that will hold our code. I’ll call my file `scraper.py`. Note that Pyppeteer natively supports asynchronous execution. So let’s import both `asyncio` and `pyppeteer` into our file:

import asyncio
from pyppeteer import launch

With this done, we can move ahead to more complicated code. In general, I am not the biggest advocate of functional programming. Yet, I think splitting code in small chunks allows for better learning. So let’s wrap our code inside a function:

async def scrape(url):

   browser = await launch()

   page = await browser.newPage()

   await page.goto(url)

   content = await page.content()

   await browser.close()

  

   return content

See how the syntax is almost identical to the one used in this (AICI TREBUIE POSTAT ARTICOLUL ASTA:https://docs.google.com/document/d/1oUekhOuW7dobPcWBv1mLqh044-e0Afh_IGqenvrrnTs/edit#heading=h.pj0e3c52extb) simple Puppeteer project: 

This function takes a URL as an input and it launches a headless browser using pyppeteer. Then, it navigates to the provided URL, retrieves the page's content, and closes the browser. The value it returns is nothing else than the HTML collected from the page. You can use this function to scrape almost any website. To use the function, you would call it in an `asyncio` event loop, like this:

async def main():

   content = await scrape('https://www.example.com')

   print(content)

loop = asyncio.get_event_loop()

loop.run_until_complete(main())

Up to this point, we have a working scraper. But that is pretty much all we have. If you want to build a more advanced web scraper with Pyppeteer, you will have to add more functionality to id. Spoiler alert: we’ll dive into the world of object oriented programming. But first let’s trace our objectives. What do we want our scraper to be able to do?

Initialize the browser with some custom values

Navigate and extract content from a web page

So let’s create a new `Scraper` class for now and we’ll add its methods afterward:

class Scraper:

   def __init__(self, launch_options: dict) -> None:

   	self.options = launch_options['options']

 	self.viewPort = launch_options['viewPort'] if 'viewPort' in launch_options else None

pass

The only argument we’re using for our Scraper is a `launch_options` dictionary. As you see, it holds two keys inside it. One key defines Pyppeteer’s launcher options. The second option is either `None` or a dictionary holding the `width` and `height` of the `viewPort`. The latter is used for this method.

If you look at the function we used before, you’ll see that we cover both navigating and extracting raw data from a specific URL. The only thing we need to do is to tweak and turn the function into a method for our Scraper:

async def goto(self, url: str) -> None:

       self.browser = await launch(options=self.options)

       self.page = await self.browser.newPage()

       await self.page.setViewport(self.viewPort) if self.viewPort != None else print('[i] Using default viewport')

       await self.page.goto(url)

This method is quite simple. First, it launches a new browser, with the custom options we set before. It then creates a new page and, if our `launch_options` dictionary has `viewPort`, then it sets the page’s viewPort. Otherwise, it logs a simple message. Last but not least, it takes us to the target. 

Again, we have the method in our initial `scraper` function. We will only await for the `page.content()` to load and return its value:

async def get_full_content(self) -> str:

       content = await self.page.content()

       return content

In order to write something to an input field using Pyppeteer, you’ll need two things. First, locate the element. Second, add some value to it. Luckily, Pyppeteer has methods for both these actions:

async def type_value(self, selector: str, value: str) -> None:

       element = await self.page.querySelector(selector)

       await element.type(value)

Remember we want to be able to extract either the value from a single element, or values from multiple elements. We could use one single method for both. But I usually like things separated. So for now, I’ll add two more methods:

async def extract_one(self, selector) -> str:

       element = await self.page.querySelector(selector)

       text = await element.getProperty("textContent")

       return await text.jsonValue()

Here, we are locating the element using the `querySelector` method. We then await for the `textContent` and return its `jsonValue()`. On the other hand, when we want to select many elements, we’ll use `querySelector`:

async def extract_many(self, selector) -> list:

       result = []

       elements = await self.page.querySelectorAll(selector)

       for element in elements:

           text = await element.getProperty("textContent")

           result.append(await text.jsonValue())

       return result

This method works in a similar way to `extract_one`. The only difference is its return value. This time we’re returning a list of all text inside the selected elements. And I guess with this added, we touched on all our goals. 

In web scraping, stealthiness can be described as the ability to go undetected. Of course, building a fully undetectable scraper takes a lot of work. For example, Web Scraping API’s Stealth Mode is maintained by a dedicated team. And the effort put into it makes our scraper’s fingerprint unique with every request. 

But my overall aim for this tutorial is to set you on the right path. And the right path for a complete web scraper with Pyppeteer implies adding some stealth functionality to it. Lucky enough, just like there is `puppeteer-extra-plugin-stealth` in Node, there is a package for Python too. And it's called intuitively, `pyppeteer-stealth`. To add it to your project, first install it using pip:

~ » python3 -m pip install pyppeteer_stealth

Then import it into your project and just add one extra line of code:

async def goto(self, url: str) -> None:

       self.browser = await launch(options=self.options)

       self.page = await self.browser.newPage()

	  # Make it stealthy

       await stealth(self.page)  

       await self.page.setViewport(self.viewPort) if self.viewPort != None else print('[i] Using default viewport')

       await self.page.goto(url)

And here is how you run your scraper. I added some comments to the code to highlight what each step is doing:

async def main():

   # Define the launch options dictionary

   launch_options = {

       'options': {

           'headless': False,

           'autoClose': True

       },

       'viewPort': {

           'width': 1600,

           'height': 900

       }

   }

   # Initialize a new scraper

   scraper = Scraper(launch_options)

   # Navigae to your target

   await scraper.goto('https://russmaxdesign.github.io/accessible-forms/accessible-name-input01.html')

   # Type `This is me` inside the input field

   await scraper.type_value(

       '#fish',

       'This is me')

   # Scrape the entire page

   content = await scraper.get_full_content()

   print(content)

   # Scrape one single element

   el = await scraper.extract_one('body > div:nth-child(14) > ul')

   print(el)

   # Scrape multiple elements

   els = await scraper.extract_many('p')

   print(els)

loop = asyncio.get_event_loop()

loop.run_until_complete(main())

Pyppeteer is an amazing tool for web scraping. It ports the entire Puppeteer API to Python, making it possible for the Python community to use this technology without learning JavaScript. Moreover, I don’t think it’s a replacement for Selenium, but it sure is a good alternative to it.

I hope today’s article added value to your learning path. And as I like to push everyone's limits, I challenge you to add more to what you learned today. The scraper we built together is a really good starting point and introduces a key element in programming: OOP. So, I challenge you to add more methods to `Scraper` and make it truly amazing.

The Ultimate Guide To Building a Web Scraper With Pyppeteer

Detection is a big issue in the web scraping industry. It’s like a constant cat and mouse game between the scrapers and the anti-bot systems. And using an anti detect browser became a requirement for anyone looking to scrape data. But anti detection browsers are not limited to web scraping. People have used these for things like managing multiple social media accounts, e-commerce and so on. Even users interested in privacy can benefit from an antidetect browser. 

Today’s article will cover exactly this area. We are going to discuss anti detect browsers, their use cases, effectiveness and which vendor to pick in 2023. I’m also going to introduce you to browser fingerprinting. So brace yourselves, there’s much to talk about today. 

Partially, the definition of an anti detect browser is given by the name itself. Each regular browser (say Chrome or Firefox) has its own online identity. We call this a fingerprint. The way your browser is fingerprinted is very complex. It has to do with things like:

The User Agent you’re sending with your requests

Now, the last one is the most important I would say. That is because anyone on the internet can build a relatively simple script to get certain properties of a browser. And what is most important is that almost every time, combining these values generates a unique system configuration. In practice, one can generate a hash of the stringified combined values of these properties and create a fingerprint. Take for example FingerprintJS or CreepJS. 

Furthermore, detection is not window scoped. I even created this tool that can detect evasions such as the ones implemented with `puppeteer-extra-plugin-stealth` by comparing values gathered from the window and workers. But that is a more complex discussion. For now, it seems that the best way to go undetected online is to use an anti detect browser.

And we’re back to the topic. Antidetect browsers are applications based on known browsers that have been modified to accept custom values for the properties we discussed earlier. Because projects like Chromium or Mozilla are open source, anyone can fork them and create their own version of a browser. And that is exactly what happens with anti detect browsers too. 

You too can clone Chromium for example and tweak it to serve random values for various properties of the browser. But in practice, this requires lots of knowledge and resources. That’s why there are some known vendors and we’ll get to them in just a moment. But first:

What is the use of an anti detect browser?

There are many use cases for anti detect browsers. Yet there are industries that rely on these browsers more than others. That’s why I am going to be specific about what are the use cases of anti detect browser and why use it. So understanding that its biggest strength is anonymity, here are the most common use cases for an anti detect web browser:

Marketing agencies:  Affiliate marketers or social media agencies usually have to manage multiple social media accounts. In practice, social media platforms ban access to such activities. Ideally, from their pov, one device should only handle one account. More could lead to abuse. Since anti detect browsers create more than a single identity, you could easily use them to manage such activity.

e-Commerce Vendors:  There is a common (yet unspoken) practice among Amazon sellers or drop-shippers to use multiple accounts to sell their products. This increases their reach and implicitly sales. Again, big e-commerce platforms don’t usually allow such practices. And yet again, the anti detection browsers can come into play.

Web scraping: In the past years, web scraping has become a real challenge. Fingerprinting a web driver (automated browser) is easier today than it was a couple of years ago. And as you’ve probably already guessed, almost any website is trying to avoid bots. Since web scrapers fall into this category, websites are usually trying to block their access. 

How efficient are anti detect browsers in practice?

Depending on the vendor, anti detect browsers can be quite efficient. Working on Web Scraping API’s stealth mode, I got to play around with various evasion techniques. One of which meant testing an anti detect browser. It might come as a surprise, but we were able to identify automated browsers for patched Puppeteer and Selenium, but not for anti detect browsers.

As an extra layer of efficiency, we discovered through testing that the values used to generate custom fingerprints are also important to detect web drivers. For example, randomized values are a ‘no-no’. Istead, when generating a fingerprint, try to:

Use device platforms that match the user agent

And that’s just to name a few. The main take is to use real, matching information. Also, should you pick a vendor, try to stick to their recommendations. 

In a related article [LINK](AICI TREBUIE POSTAT ARTICOLUL) I said that the best anti detect browser is the one you build yourself. But honestly speaking, building a browser, even when you have an open source project as a starting point, takes a lot of effort. So if you don’t have the resources to put up a team, you’re better off going with one of these vendors:

According to their website, Multilogin offers ‘full fingerprint customization’. This means that you get complete access to the Browser’s properties and can tweak them in numerous ways. Plus, they’re one of the few vendors that do not disable browser features. Disabling browser features (such as the service workers) is not an instant giveaway for an automated browser. However, regular users rarely if ever disable such features. And that can raise a red flag for bot detectors. 

Now back to Multilogin. What they did was to fork Chrome and Firefox and build their own customisable versions. Their GUI allows you to either generate a quick profile (with a fingerprint associated) or spoof certain values yourself. The only downside to Multilogin I’d say is the price. It starts at 99 USD per month and can go up to 399 USD per month. That is at the time I’m writing this. For updated prices, check out their pricing page. Now to wrap it up, here is some compressed info:

Just like Multilogin, AdsPower offers two anti detect browsers. One is Chromium based and is called Sun Browser. The second is Firefox based and is called Flower Browser. Both have the added capability of managing the browser fingerprint. What is interesting about this anti detect browser is that it has built in automation features. Little to no programming is required to automate tasks using AdsPower’s Robotic Process Automation.

The company started in 2019 in Hong Kong and quickly grew to become one of the main competitors of Multilogin. Part of their success stands in the amazing capabilities the browsers feature. The other part I think is driven by the low prices. AdsPower even has a free tier which gives you access to two profiles and their pro plan is only 50 USD per month. On personal note, I find it the most appealing solution. And with that, let’s wrap it:

GoLogin is new to the scene of anti detection browsers. Yet this doesn’t make it less efficient. What I first noticed about it was the support channels. They are available not only via email or chat, but also via WhatsApp, Telegram and Facebook Messenger. The second honorable mention about GoLogin is their proxy support. Their browser is packed with proxies by default. And the third is related to access. GoLogin has a Desktop app, a mobile app and a web app you can use. You can access their tool from the cloud and that is quite impressive.

The browser itself is based on Google Chrome. It was modified to generate new fingerprints for each profile. One downside is that instead of spoofing all values, they also add noise to some. And tests indicate that this can lead to more captchas, indicating that antibot systems can still pick up some automated activity. 

Price wise, GoLogin starts at 49 USD per month and goes up to 199 USD. They also offer a customisable tier. It goes up to 1999 USD per month for 10.000 cloud profiles and other benefits. And you also get a 50% discount if you pay yearly. You can check their pricing page for more information. Wrapping it up:

With X-Browser, the anti detect browser in itself is free. And that can be a pretty appealing feature. The browser is offered by smartproxy and is free to download. The ‘catch’ here is that you have to pay for the proxies. However, there is actually no downside to it. And that is because with either option, you need proxies to hide your IP and avoid getting blocked.

Bounding X-Browser to their own proxies can be a good and a bad thing at the same time. For once, the user can trust that smartproxy will offer an optimized solution for their proxies. On the other hand, you can only use smartproxy proxies. Being bound to a single provider means being bound to the quality of their service. And if you want to scale your activity, I think you need more flexibility. 

Since you only pay for proxies, pricing is pretty decent with X-Browser. Their datacenter proxies range between 10 and 45 USD per month. Residential proxies can be a bit expensive, but it’s all a matter of how much you use. Check out prices here. So let’s wrap up X-Browser:

Kameleo is quite a chameleon. It supports all major web browsers (Chrome, Safari, Firefox and Edge) and it can be used with the main automation frameworks (Puppeteer, Selenium and Playwright). And more than this, they offer fingerprint customization for mobile devices as well. According to their website, Kameleo Mobile can be used to spoof your fingerprint on any Android device. 

As opposed to their competitors, Kameleo’s service is not centered around browsers, but around browser profiles. Meaning, they don’t necessarily make changes to an open-source browser to create their own version. 

We did some experimenting with their platform and the main disadvantage we found was customer support. It can be really slow. Apart from that, pricing is good: 59 to 199 USD paid monthly. So let’s wrap this up:

Incognito is yet another anti detect browser with free tier. And as opposed to AdsPower, it offers access to 10 free browser profiles. Their browser is Chromium based and it pretty much offers the same features as the rest of anti detect browsers. The product also features an API that allows you to interact with the headless browser. Apart from the generous free tier, their price ranges from 29.99 to 149.99 USD per month. I’d say it’s a fair price for what they offer. Now wrapping it up:

Linken Sphere is an anti detect browser based on Chromium and is offered by the Russian group Tenebris. It was launched back in 2017, making it one of the oldest solutions on the market. The main downside with it is that it looks quite shady. They only accept payment in bitcoin at the moment and support it centered around their Telegram channel. Moreover, it’s known to be used in illegal activities, such as hacking or fraud. Apart from that it does its job pretty well. And with this, I’ll wrap things up for good:

Browser fingerprinting can make web scraping extremely hard. If you want to go undetected, you need to cover a lot of aspects. Proxies, captcha solvers, and a stealthy headless browser are just a few of the ‘ingredients’ of a reliable web scraper. Adding everything up can generate very high development and maintenance costs. The alternative is to use an established third party web scraping service. And Web Scraping API offers exactly that. And I’d even go to say it’s all at a fraction of the cost.

Best Anti-Detect Browsers for Safe and Effective Web Scraping

Nowadays, web scraping has become a real challenging task. If you ever built a web scraper with a headless browser, most definitely you came across some anti-bot systems. And bypassing cloudflare, datadome or any other anti-bot provider for that matter, is no easy task. You have to first think of evasion strategies and then implement them in production. And even so, there might be scenarios that you never accounted for. 

Yet without at least a minimal implementation of these evasion techniques, your chances of getting caught and blocked are very high. That is why, the topic for today’s article is how to bypass cloudflare detection. And most of the techniques we’re going to discuss apply to other detection vendors as well. As a reference, we’ll focus on Selenium to see if we can make it stealthy. Throughout  the article, among others, I’ll discuss:

How does Cloudflare detect headless browsers?

Cloudflare is a tech company with a giant network. They focus on services like CDN, DNS and various online security systems. Their Web Application Firewall is usually designed to protect against attacks such as DDoS or cross site scripting. In recent years, Cloudflare added and other providers in the field introduced fingerprinting systems, capable of detecting headless browsers. As you might guess, one of the first affected by these techniques is Selenium. And since the web scraping industry relies heavily on this technology, scrapers are directly affected as well. 

Before moving forward to anti-bot techniques, I think it is important to discuss how cloudflare detects Selenium. Well, the system can be very complex. For example, there are properties in a browser that a web driver lacks. The `navigator` interface in a browser even has a property called `webdriver` that indicates if a browser is controlled by automation. And that is an instant give away. If you want to experiment with it:

Type the following command: `navigator.webdriver`

In your case, it should return `false`. But if you try it with Puppeteer or Selenium, you will get `true`. If you’re wondering how Cloudflare leverages this to detect bots, well it’s pretty simple. All they need to do is inject a script like the one below on their partner's website:

// detection-script.js

const webdriver = navigator.webdriver

// If webdriver returns true, display a reCaptcha

// In this example, I am transferring the user to a Cloudflare challenge page.

// But you get the idea

if ( webdriver ) location.replace('https://cloudflarechallenge.com')

Of course, in real life, there are many more levels of detection these providers use. Even the size of the screen, the keyboard layout or the plugins used by the browsers are used to specifically fingerprint a browser. If you’re interested in how browserless detection works, check out my simple service worker test. And that is just if you stick to the browser. You can also detect bot activity by looking into the IP address from where the request originates. For example, if you’re using datacenter IPs, your chances of getting blocked increase with every request. That is why it’s recommended to use residential or ISP proxies when you’re building a web scraper.

Fortunately, the web scraping community is really active. And because there is such demand to bypass Cloudflare and other anti-bot providers, there are open source solutions in that area. Great things can be achieved when programming communities work together! Moving forward, I suggest we follow these steps:

Run some test to see if default Selenium can bypass Cloudflare 

Add some extra evasions to our make our scripts stealthier

So let’s get started with our first step:

#1: Can default Selenium bypass Cloudflare?

I am not the type to make assumptions. And that’s especially because we don’t know for sure how Cloudflare systems work. They use all sorts of obfuscation with their code, which makes it harder to reverse engineer. That’s why, throughout my experience as a developer, I learned that testing is the best way to understand how a system works. So let us build a basic scraper and see how it behaves on a real target, protected by Cloudflare.

With Python, it’s best we isolate our projects inside one single directory. So let’s create a new folder, open a terminal window and navigate to it:

# Create a new virtual environment and activate it

~ » python3 -m venv env && source env/bin/activate

# Install dependencies

~ » python3 -m pip install selenium

# Create a new .py file and open the project inside your IDE

~ » touch app.py && code .

1.2. Build a simple web scraper with Selenium

Now that you have successfully set up your project, it’s time to add some code. We’re not going to build anything fancy here. We just need this script for testing purposes. If you want to learn about advanced scraping, check out this [LINK](AICI TREBUIE POSTAT ARTICOLUL) tutorial on Pyppeteer.

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

# Set Chrome to open in headless mode

options = Options()

options.headless = True

# Create a new Chrome instance and navigate to target

driver = webdriver.Chrome(options=options)

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('screenshot.png')

# Close browser

driver.quit()

Now check out the screenshot. Here’s what I got:

I think we’re safe to conclude that the test failed. The targeted website is protected by Cloudflare and as you can see, we’re getting blocked. So by default, Selenium is not able to bypass Cloudflare. I am not going to go in depth and check with other bot detection providers. If you want to test further, here are some targets and their vendors:

#2: Can stealthy selenium bypass Cloudflare?

First of all, let me clarify the terms. By stealth Selenium I mean a version of Selenium that can go undetected and bypass Cloudflare. I am not referring to any specific stealthiness technique. There are a couple of ways you can implement evasion techniques into Selenium. There are packages that handle it, or you can use the `execute_cdp_cmd` to interact directly with the Chrome API. The latter allows you more control, but requires more work. Here is an example of how you could use it to change the user agent’s value:

driver.execute_cdp_cmd('Emulation.setUserAgentOverride', {

               "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win32; x86) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36",

               "platform": "Win32",

               "acceptLanguage":"ro-RO"

       })

But you’d have to go through CDP and identify the APIs that allow you to make all the necessary changes. So for the time being, let’s test with some packages.

There are at least two packages you can use to make Selenium stealthy. Up to this point though, none of them is guaranteed to bypass Cloudflare. Yet again, we need to test and see if either of them works. First of all, let’s look into `selenium-stealth`. This package is a wrapper around `puppeteer-extra-plugin-stealth`, making it possible to use Puppeteer’s evasions with Python’s Selenium. To use it, you have to install it first. Open a terminal window and enter this command:

# Install selenium-stealth

~ » python3 -m pip install selenium-stealth

You’re all set now. We can use it to make our previous scraper stealthier:

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

from selenium_stealth import stealth

import time

# Set Chrome to open in headless mode

options = Options()

options.headless = True

# Create a new Chrome instance

driver = webdriver.Chrome(options=options)

# Apply stealth to your webdriver

stealth(driver,

   languages=["en-US", "en"],

   vendor="Google Inc.",

   platform="Win32",

   webgl_vendor="Intel Inc.",

   renderer="Intel Iris OpenGL Engine",

   fix_hairline=True,

)

# Navigate to target

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('stealth.png')

# Close browser

driver.quit()

Running the script returned some different outcome for me this time, as opposed to the default Selenium settings:

The second option you may use is `undetected_chromedriver`. This one is described as an ‘optimized Selenium chromedriver’. Let’s test it out:

# Install undetected_chromedriver

~ » python3 -m pip install undetected_chromedriver

The code is very similar to our default script. The main difference stands in the package name. Here is a basic scraper with `undetected_chromedriver` and let’s see if it can bypass Cloudflare:

import undetected_chromedriver as uc

import time

# Set Chrome to open in headless mode

options = uc.ChromeOptions()

options.headless = True

# Create a new Chrome instance and maximize the window

driver = uc.Chrome(options=options, executable_path='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome')

driver.maximize_window()

# Navigate to target

driver.get('https://www.snipesusa.com/')

# Give it some time to load

time.sleep(10)

# Take screenshot of page

driver.get_screenshot_as_file('stealth-uc.png')

# Close browser

driver.quit()

Yet again, running the script turns out well for me. It seems that at least these two packages can successfully bypass Cloudflare protection. At least in the short term. Truth to be told, chances are, if you use these scripts extensively, Cloudflare will catch up on your IP address and block it. 

So let me introduce you to a third option: Web Scraping API. 

Web Scraping API has this amazing feature called the Proxy Mode. You can read more about it here. But what I want to note here is that our Proxy Mode can successfully be integrated with Selenium. This way, you get access to all the evasion features we’ve implemented. And let me tell you that we have a dedicated team working on custom evasion techniques. In technical terms, we’re handling IP rotations, we’re using various proxies, we’re solving captchas and we’re using Chrome’s API to continuously change our fingerprint. In non-technical terms, this translates to less hussle on your side and greater success rate. You basically get the stealthiest version of Selenium there is. And here’s how it’s done:

# Install selenium-wire

~ » python3 -m pip install selenium-wire

We’re using `selenium-wire` in order to use Selenium with a proxy. Now here’s the script:

from seleniumwire import webdriver

import time

# Method to encode parameters

def get_params(object):

   params = ''

   for key,value in object.items():

       if list(object).index(key) < len(object) - 1:

           params += f"{key}={value}."

       else:

           params += f"{key}={value}"

   return params

# Your WSA API key

API_KEY = '<YOUR_API_KEY>'

# Default proxy mode parameters

PARAMETERS = {

   "proxy_type":"datacenter",

   "device":"desktop",

   "render_js":1

}

# Set Selenium to use a proxy

options = {

   'proxy': {

       "http": f"http://webscrapingapi.{ get_params(PARAMETERS) }:{ API_KEY }@proxy.webscrapingapi.com:80",

   }

}

# Create a new Chrome instance

driver = webdriver.Chrome(seleniumwire_options=options)

# Navigate to target

driver.get('https://www.httpbin.org/get')

# Retrieve the HTML documeent from the page

html = driver.page_source

print(html)

# Close browser

driver.quit()

If you run this script a couple of times, you’ll see how the IP address changes every time. That’s our IP rotation system. In the background, it also adds evasions techniques. You don’t even need to worry about them. We handle the Cloudflare bypassing part so that you can focus more on parsing the data. 

If you want to build a scraper that can bypass Cloudflare, you need to account for a lot of things. A dedicated team can work 24/7 and there’s still no guarantee the evasions will work every time. That’s because with every browser version release, there is a chance new features are added to the API. And some of these features can be used to fingerprint and detect bots.

I’d even say that the best browser to bypass Cloudflare and other providers is the one you build yourself. And we built one at Web Scraping API. Now we’re sharing it with you. So enjoy scraping!

Step-by-Step: How to Bypass Cloudflare and Improve Your Web Scraping Efforts

Proxies are a very important aspect of web scraping. And there are mainly three types of proxies. There are datacenter, residential and ISP proxies. Each type has its own use cases. Yet, there is little to no information available on what they mean and when to use them in web scraping. And it’s especially ISP proxies that are left unhandled by tech writers. That is why today, we will focus our attention on this type of proxies. By the end of today’s article, you should have a solid understanding of:

What is a proxy in general and how does it work

What is the definition of an ISP proxy and what are its particularities

How and why to use ISP proxies for web scraping

In short, proxies are middleware's between clients and servers. A proxy will act as an intermediary for requests from clients seeking resources from other servers. The flow of a client - proxy - server relationship looks something like this:

A client connects to the proxy server, requesting some service from a destination server 

The proxy server evaluates the request, connects to the destination server and fetches the requested service

Once received, it then transfers the service back to the client, unaltered.

Apart from that, proxies are also used to:

bypass geographical restrictions on internet access

As we discussed in the introduction, there are mainly three types of proxies. For the first two, the definition is pretty straightforward. Datacenter proxies are proxies that are owned by data centers. Which means that their IP addresses are associated with the datacenter. Residential proxies have the IP address associated with a physical location. Moreover, these IP addresses are registered to a specific individual or organization. 

Now when it comes to IPS proxies, there is a bit of a confusion. First of all, ISP stands for Internet Service Provider. And as you can imagine, all residential IPs originate from ISPs. Well, this small aspect partially answers the question. ISP proxies fit somewhere between datacenter and residential proxies.

Most of the time, you’ll find that an ISP proxy is actually a residential proxy, hosted on a datacenter machine. Hence, these proxies inherit advantages from the other two. And the list of benefits include for the most part (but are not limited to):

Mihnea-Octavian Manolache