Navigating the world of web scraping can be a bit overwhelming. You have to choose the right programming language, the right library, and deal with many unforeseen setbacks. It quickly becomes a lot to take in. But don't let that discourage you! In this article, I've answered some of the most asked questions about web scraping. You'll learn what other people are doing and the challenges they've faced. This will help guide your own decision-making process. Whether you're new to the field or a seasoned pro, there's something here for everyone.

Why can't my scraper see the same data as my browser?

You've written a script to fetch HTML from a website, but you're not getting the full data. You've tested your selectors in the browser and they should work, right? Not always. Websites that rely on JavaScript to render won't work with a simple GET request. There are libraries like Puppeteer and Selenium that use headless browsers to render JavaScript. They allow you to make the request in the context of a browser and wait for JavaScript to finish executing. This way, you can get the full HTML. You may not always need a headless browser to get the missing data. Search for <script> tags in the HTML. The missing data could be hidden inside <script> tags as JavaScript variables.

How can I scrape a website that uses generated CSS classes?

Some websites use libraries that automatically create unique class names for different page components. This can make it difficult to use traditional CSS selectors to target specific elements.

One solution is to use XPath expressions instead. XPath selectors rely on the layout of the page, rather than specific class names. This means that even if the class names change, the XPath selector will still be able to locate the desired element.

For example, if you have an HTML component that looks like this:

<div class="container">

    <div class="subcontainer_af21">

        <ul class="ul_ax1">

            <li class="li_adef">

                <a href="https://link1">Winter Storm</a>

            </li>

        </ul>

        <ul class="ul_cgt4">

            <li class="li_ocv2">

                <a href="https://lin2">SpaceX</a>

            </li>

        </ul>

    </div>

</div>

You can select the second <a> element with:

Yes, Cheerio is generally considered to be faster than Puppeteer. This is because Cheerio is a server-side library that works directly with the HTML content. Puppeteer is a browser automation library that controls a headless browser to load web pages and interact with them. Cheerio is limited in the sense that it can only work with static pages, it doesn't have the ability to interact with the browser as Puppeteer does

Are XPath selectors better than CSS selectors?

It depends on the context. If you are looking to extract data based on the position of elements, XPath is the better choice. However, if you are looking to extract data based on properties such as class or id, CSS selectors are a better option.

Both of them offer similar functionalities, but. Playwright supports multiple browsers, including Chrome, Firefox, and Safari. Puppeteer supports only Chrome and Chromium.

Playwright has better support for working with multiple tabs and windows. It also has built-in support for handling browser contexts, cookies, and storage. Playwright is better suited for complex projects.

In general, you can try to space out your requests. Use different IPs. Use proxies. Try to alter the browser fingerprint. For most people, this is a never ending battle. The good news is that it does not have to be this way. You can use our solution, WebScrapingAPI. WebScrapingAPI provides an API that will handle all the heavy lifting for you. It can execute JavaScript, rotate proxies, and even handle CAPTCHAs. You'll never have to worry about getting your IP banned. But don’t take our word for it. You can try it for free.

How to extract text from HTML with BeautifulSoup?

You can use the BeautifulSoup library. Here is an example of extracting text using the .get_text() function:

from bs4 import BeautifulSoup

html_doc = """

<html>

 <head>

   <title>title of the page</title>

 </head>

 <body>

   <p>a paragraph</p>

   <a href='https://link.com'>a link</a>

 </body>

</html>

"""

soup = BeautifulSoup(html_doc, 'html.parser')

paragraph_text = soup.find('p').text

print(paragraph_text)

#Prints 'a paragraph'

link_text = soup.find('a').text

print(link_text)

#Prints 'a link'

all_text = soup.get_text()

print(all_text)

"""

title of the page

a paragraph

a link

"""

How to extract text from HTML with Selenium?

from selenium import webdriver

from selenium.webdriver.common.by import By

DRIVER_PATH = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

# get all the h2 elements

content = driver.find_element(By.TAG_NAME, "h2")

print(content.text)

# Prints 'From today's featured article'

How to select HTML elements by text with BeautifulSoup?

With BeautifulSoup, you can use the soup.find method with the the text=re.compile("<text>") parameter:

from bs4 import BeautifulSoup

import re

html_doc = """

<html>

<body>

  <p class="my_paragraph">a paragraph.</p>

  <p class="my_paragraph">another paragraph.</p>

</body>

</html>

"""

soup = BeautifulSoup(html_doc, 'html.parser')

# find the first pTag that contains the text 'a par'

pTag = soup.find("p", text=re.compile("a par"))

print(pTag)

How to select HTML elements by text with Selenium?

from selenium import webdriver

from selenium.webdriver.common.by import By

DRIVER_PATH = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

# get all the elements with class vector-body

span = driver.find_element(By.XPATH, "//span[contains(text(), 'Did')]")

print(span.text)

# Prints 'Did you know ...'

driver.quit()

How to find HTML elements with CSS selectors in BeautifulSoup?

Here’s how you can do it with BeautifulSoup and the find and find_all methods:

from bs4 import BeautifulSoup

html_doc = """

<html>

  <body>

    <p class="my_paragraph">First paragraph.</p>

    <p class="my_paragraph">Second paragraph..</p>

    <p>Last paragraph.</p>

  </body>

</html>

"""

soup = BeautifulSoup(html_doc, 'html.parser')

# find all elements with class 'my_paragraph

elements = soup.find_all(class_="my_paragraph")

for element in elements:

    print(element.text)

# prints 'First paragraph.' and 'Second paragraph..'

How to find HTML elements by class with Selenium?

from selenium import webdriver

from selenium.webdriver.common.by import By

DRIVER_PATH = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

# get all the elements with class vector-body

elements = driver.find_elements(By.CLASS_NAME, "vector-body")

for element in elements:

   print(element.text)

driver.quit()

import requests

from bs4 import BeautifulSoup

from lxml import etree

response = requests.get("https://en.wikipedia.org/wiki/Main_Page")

soup = BeautifulSoup(response.content, 'html.parser')

dom = etree.HTML(str(body))

xpath_str = '//h1//text()'

print(dom.xpath(xpath_str))

#Prints ['Main Page', 'Welcome to ', 'Wikipedia']



How to wait for the page to load in Selenium?

If you simply want to wait for a certain time before timing out when trying to find any element, you can use the driver.implicitly_wait(time_in_secods) function:

from selenium import webdriver

from selenium.webdriver.common.by import By

DRIVER_PATH = 'C:/Users/Michael/Desktop/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.implicitly_wait(10)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

element = driver.find_element(By.ID, "not_found_id")

# the element does not exist, but it waits 10 seconds for it

text = element.text

print(text)

# Close the browser

driver.quit()

You can also choose to wait until a certain condition is met:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

DRIVER_PATH = 'C:/Users/Michael/Desktop/chromedriver'

driver = webdriver.Chrome(executable_path=DRIVER_PATH)

driver.get("https://en.wikipedia.org/wiki/Main_Page")

# Wait for the element with id 'content' to be present on the page

wait = WebDriverWait(driver, 10)

element = wait.until(EC.presence_of_element_located((By.ID, "content")))

element = driver.find_element(By.ID, "content")

text = element.text

print(text)

# Close the browser

driver.quit()

How to find HTML elements with CSS selectors in Puppeteer?

In Puppetter, you can use the page.$() and page.$$() functions to select elements with CSS selectors. The page.$() function is used to find the first element that matches the selector. The page.$$() function is used to find all the elements that match the selector.

How to find HTML elements with CSS selectors in Playwright?

Here is how you can do it with Playwright. It is very similar to Puppeteer:

const { chromium } = require('playwright');

(async () => {

  const browser = await chromium.launch({

    headless: false,

  });

  const context = await browser.newContext();

  const page = await context.newPage();

  await page.goto('https://www.scrapethissite.com/pages/simple/');

  // Extract the first odd row element

  const firstOddRow = await page.$('.container .row');

  console.log(await firstOddRow.textContent());

  // Extract all the odd rows

  const allOddRows = await page.$$('.container .row');

  for (const oddRow of allOddRows ) {

    console.log(await oddRow.textContent());

  }

  await browser.close();

})();

How to find HTML elements with CSS selectors in cheerio?

With cheerio, you’ll have to fetch the HTML (I used the request library to do that) and then pass it to the cheerio library:

With Puppetter, you can use the page.$x() function to select elements with XPath selectors:

const puppeteer = require('puppeteer');

(async () => {

  const browser = await puppeteer.launch();

  const page = await browser.newPage();

  await page.goto('https://www.scrapethissite.com/pages/forms/');

  // Extract the table header elements

  const allTableHeaders = await page.$x('//table/tbody/tr[1]//th');

  for(let i = 0; i < allTableHeaders.length; i++) {

    const header = await page.evaluate(el => el.textContent, allTableHeaders[i]);

    console.log(header.trim());

  }

  await browser.close();

})();

// Output:

// Team Name

// Year

// Wins

// Losses

// OT Losses

// Win %

// Goals For (GF)

// Goals Against (GA)

// + / -

const { chromium } = require('playwright');

(async () => {

  const browser = await chromium.launch({

    headless: false,

  });

  const context = await browser.newContext();

  const page = await context.newPage();

  await page.goto('https://www.scrapethissite.com/pages/forms/');

  // Extract the table header elements

  const allTableHeaders = await page.locator('xpath=//table/tbody/tr[1]//th').all();

 

  for (let i = 0; i < allTableHeaders.length; i++) {

    const headerText = await allTableHeaders[i].innerText();

    console.log(headerText);

  }

  await browser.close();

})();

Any selector string starting with // or .. are assumed to be an xpath selector. For example, Playwright converts '//html/body' to 'xpath=//html/body'. 

How to find HTML elements by text in Puppeteer?

In Puppeteer, the simplest way to find elements by text is to use the XPath text() function:

How to find HTML elements by text in Playwright?

If you want to find elements by text in Playwright, you can use the allInnerTexts() function in combination with XPath.

const { chromium } = require('playwright');

(async () => {

  const browser = await chromium.launch({

    headless: false,

  });

  const context = await browser.newContext();

  const page = await context.newPage();

  await page.goto('https://en.wikipedia.org/wiki/Web_scraping');

  // Select all the p tags texts that contain the word "prevent" 

  const pTags  = await page.locator('//p[contains(text(), "prevent")]').allInnerTexts();

 

  for (let i = 0; i < pTags.length; i++) {

    console.log(pTags[i], "\n");

  }

  await browser.close();

})();

How to find HTML elements by text in cheerio?

In Puppeteer, you can use the page.waitForSelector() function to wait for a specific element to appear on the page before continuing with the script. You can use it with both CSS and XPath selectors:

await page.waitForSelector('.basic-element', { timeout: 10000 });

await page.waitForXPath("//div[@class='basic-element']"), { timeout: 10000 });

The timeout parameter specifies the maximum wait time in ms.

You can also wait for an element to reach a certain state:

await page.waitForSelector('.basic-element', { visible: true });

// wait until the element becomes visible

Playwright is similar to Puppeteer. You can use the page.waitForSelector() method to wait for a specific element to appear on the page.

await page.waitForSelector('.element-class', { timeout: 10000 });

 await page.waitForSelector('.basic-element', { state: 'visible' });

 // wait for element to become visible

Web scraping is a vast subject and this article only covers the surface. Choosing the right tool for your specific use case is crucial. For example, if you want to scrape a website using JavaScript, the cheerio library is a good option. However, if the website requires JavaScript to load fully, Puppeteer or Playwright are better options. Web scraping is challenging but understanding the tools can save you a lot of headache. I hope this article broadened your perspective and I wish you all the best in your web scraping endeavors.

Common Questions About Web Scraping - Answers & Tips

Detection is a big issue in the web scraping industry. It’s like a constant cat and mouse game between the scrapers and the anti-bot systems. And using an anti detect browser became a requirement for anyone looking to scrape data. But anti detection browsers are not limited to web scraping. People have used these for things like managing multiple social media accounts, e-commerce and so on. Even users interested in privacy can benefit from an antidetect browser. 

Today’s article will cover exactly this area. We are going to discuss anti detect browsers, their use cases, effectiveness and which vendor to pick in 2023. I’m also going to introduce you to browser fingerprinting. So brace yourselves, there’s much to talk about today. 

Partially, the definition of an anti detect browser is given by the name itself. Each regular browser (say Chrome or Firefox) has its own online identity. We call this a fingerprint. The way your browser is fingerprinted is very complex. It has to do with things like:

The User Agent you’re sending with your requests

Now, the last one is the most important I would say. That is because anyone on the internet can build a relatively simple script to get certain properties of a browser. And what is most important is that almost every time, combining these values generates a unique system configuration. In practice, one can generate a hash of the stringified combined values of these properties and create a fingerprint. Take for example FingerprintJS or CreepJS. 

Furthermore, detection is not window scoped. I even created this tool that can detect evasions such as the ones implemented with `puppeteer-extra-plugin-stealth` by comparing values gathered from the window and workers. But that is a more complex discussion. For now, it seems that the best way to go undetected online is to use an anti detect browser.

And we’re back to the topic. Antidetect browsers are applications based on known browsers that have been modified to accept custom values for the properties we discussed earlier. Because projects like Chromium or Mozilla are open source, anyone can fork them and create their own version of a browser. And that is exactly what happens with anti detect browsers too. 

You too can clone Chromium for example and tweak it to serve random values for various properties of the browser. But in practice, this requires lots of knowledge and resources. That’s why there are some known vendors and we’ll get to them in just a moment. But first:

What is the use of an anti detect browser?

There are many use cases for anti detect browsers. Yet there are industries that rely on these browsers more than others. That’s why I am going to be specific about what are the use cases of anti detect browser and why use it. So understanding that its biggest strength is anonymity, here are the most common use cases for an anti detect web browser:

Marketing agencies:  Affiliate marketers or social media agencies usually have to manage multiple social media accounts. In practice, social media platforms ban access to such activities. Ideally, from their pov, one device should only handle one account. More could lead to abuse. Since anti detect browsers create more than a single identity, you could easily use them to manage such activity.

e-Commerce Vendors:  There is a common (yet unspoken) practice among Amazon sellers or drop-shippers to use multiple accounts to sell their products. This increases their reach and implicitly sales. Again, big e-commerce platforms don’t usually allow such practices. And yet again, the anti detection browsers can come into play.

Web scraping: In the past years, web scraping has become a real challenge. Fingerprinting a web driver (automated browser) is easier today than it was a couple of years ago. And as you’ve probably already guessed, almost any website is trying to avoid bots. Since web scrapers fall into this category, websites are usually trying to block their access. 

How efficient are anti detect browsers in practice?

Depending on the vendor, anti detect browsers can be quite efficient. Working on Web Scraping API’s stealth mode, I got to play around with various evasion techniques. One of which meant testing an anti detect browser. It might come as a surprise, but we were able to identify automated browsers for patched Puppeteer and Selenium, but not for anti detect browsers.

As an extra layer of efficiency, we discovered through testing that the values used to generate custom fingerprints are also important to detect web drivers. For example, randomized values are a ‘no-no’. Istead, when generating a fingerprint, try to:

Use device platforms that match the user agent

And that’s just to name a few. The main take is to use real, matching information. Also, should you pick a vendor, try to stick to their recommendations. 

In a related article [LINK](AICI TREBUIE POSTAT ARTICOLUL) I said that the best anti detect browser is the one you build yourself. But honestly speaking, building a browser, even when you have an open source project as a starting point, takes a lot of effort. So if you don’t have the resources to put up a team, you’re better off going with one of these vendors:

According to their website, Multilogin offers ‘full fingerprint customization’. This means that you get complete access to the Browser’s properties and can tweak them in numerous ways. Plus, they’re one of the few vendors that do not disable browser features. Disabling browser features (such as the service workers) is not an instant giveaway for an automated browser. However, regular users rarely if ever disable such features. And that can raise a red flag for bot detectors. 

Now back to Multilogin. What they did was to fork Chrome and Firefox and build their own customisable versions. Their GUI allows you to either generate a quick profile (with a fingerprint associated) or spoof certain values yourself. The only downside to Multilogin I’d say is the price. It starts at 99 USD per month and can go up to 399 USD per month. That is at the time I’m writing this. For updated prices, check out their pricing page. Now to wrap it up, here is some compressed info:

Just like Multilogin, AdsPower offers two anti detect browsers. One is Chromium based and is called Sun Browser. The second is Firefox based and is called Flower Browser. Both have the added capability of managing the browser fingerprint. What is interesting about this anti detect browser is that it has built in automation features. Little to no programming is required to automate tasks using AdsPower’s Robotic Process Automation.

The company started in 2019 in Hong Kong and quickly grew to become one of the main competitors of Multilogin. Part of their success stands in the amazing capabilities the browsers feature. The other part I think is driven by the low prices. AdsPower even has a free tier which gives you access to two profiles and their pro plan is only 50 USD per month. On personal note, I find it the most appealing solution. And with that, let’s wrap it:

GoLogin is new to the scene of anti detection browsers. Yet this doesn’t make it less efficient. What I first noticed about it was the support channels. They are available not only via email or chat, but also via WhatsApp, Telegram and Facebook Messenger. The second honorable mention about GoLogin is their proxy support. Their browser is packed with proxies by default. And the third is related to access. GoLogin has a Desktop app, a mobile app and a web app you can use. You can access their tool from the cloud and that is quite impressive.

The browser itself is based on Google Chrome. It was modified to generate new fingerprints for each profile. One downside is that instead of spoofing all values, they also add noise to some. And tests indicate that this can lead to more captchas, indicating that antibot systems can still pick up some automated activity. 

Price wise, GoLogin starts at 49 USD per month and goes up to 199 USD. They also offer a customisable tier. It goes up to 1999 USD per month for 10.000 cloud profiles and other benefits. And you also get a 50% discount if you pay yearly. You can check their pricing page for more information. Wrapping it up:

With X-Browser, the anti detect browser in itself is free. And that can be a pretty appealing feature. The browser is offered by smartproxy and is free to download. The ‘catch’ here is that you have to pay for the proxies. However, there is actually no downside to it. And that is because with either option, you need proxies to hide your IP and avoid getting blocked.

Bounding X-Browser to their own proxies can be a good and a bad thing at the same time. For once, the user can trust that smartproxy will offer an optimized solution for their proxies. On the other hand, you can only use smartproxy proxies. Being bound to a single provider means being bound to the quality of their service. And if you want to scale your activity, I think you need more flexibility. 

Since you only pay for proxies, pricing is pretty decent with X-Browser. Their datacenter proxies range between 10 and 45 USD per month. Residential proxies can be a bit expensive, but it’s all a matter of how much you use. Check out prices here. So let’s wrap up X-Browser:

Kameleo is quite a chameleon. It supports all major web browsers (Chrome, Safari, Firefox and Edge) and it can be used with the main automation frameworks (Puppeteer, Selenium and Playwright). And more than this, they offer fingerprint customization for mobile devices as well. According to their website, Kameleo Mobile can be used to spoof your fingerprint on any Android device. 

As opposed to their competitors, Kameleo’s service is not centered around browsers, but around browser profiles. Meaning, they don’t necessarily make changes to an open-source browser to create their own version. 

We did some experimenting with their platform and the main disadvantage we found was customer support. It can be really slow. Apart from that, pricing is good: 59 to 199 USD paid monthly. So let’s wrap this up:

Incognito is yet another anti detect browser with free tier. And as opposed to AdsPower, it offers access to 10 free browser profiles. Their browser is Chromium based and it pretty much offers the same features as the rest of anti detect browsers. The product also features an API that allows you to interact with the headless browser. Apart from the generous free tier, their price ranges from 29.99 to 149.99 USD per month. I’d say it’s a fair price for what they offer. Now wrapping it up:

Linken Sphere is an anti detect browser based on Chromium and is offered by the Russian group Tenebris. It was launched back in 2017, making it one of the oldest solutions on the market. The main downside with it is that it looks quite shady. They only accept payment in bitcoin at the moment and support it centered around their Telegram channel. Moreover, it’s known to be used in illegal activities, such as hacking or fraud. Apart from that it does its job pretty well. And with this, I’ll wrap things up for good:

Browser fingerprinting can make web scraping extremely hard. If you want to go undetected, you need to cover a lot of aspects. Proxies, captcha solvers, and a stealthy headless browser are just a few of the ‘ingredients’ of a reliable web scraper. Adding everything up can generate very high development and maintenance costs. The alternative is to use an established third party web scraping service. And Web Scraping API offers exactly that. And I’d even go to say it’s all at a fraction of the cost.

Best Anti-Detect Browsers for Safe and Effective Web Scraping

Proxies are a very important aspect of web scraping. And there are mainly three types of proxies. There are datacenter, residential and ISP proxies. Each type has its own use cases. Yet, there is little to no information available on what they mean and when to use them in web scraping. And it’s especially ISP proxies that are left unhandled by tech writers. That is why today, we will focus our attention on this type of proxies. By the end of today’s article, you should have a solid understanding of:

What is a proxy in general and how does it work

What is the definition of an ISP proxy and what are its particularities

How and why to use ISP proxies for web scraping

In short, proxies are middleware's between clients and servers. A proxy will act as an intermediary for requests from clients seeking resources from other servers. The flow of a client - proxy - server relationship looks something like this:

A client connects to the proxy server, requesting some service from a destination server 

The proxy server evaluates the request, connects to the destination server and fetches the requested service

Once received, it then transfers the service back to the client, unaltered.

Apart from that, proxies are also used to:

bypass geographical restrictions on internet access

As we discussed in the introduction, there are mainly three types of proxies. For the first two, the definition is pretty straightforward. Datacenter proxies are proxies that are owned by data centers. Which means that their IP addresses are associated with the datacenter. Residential proxies have the IP address associated with a physical location. Moreover, these IP addresses are registered to a specific individual or organization. 

Now when it comes to IPS proxies, there is a bit of a confusion. First of all, ISP stands for Internet Service Provider. And as you can imagine, all residential IPs originate from ISPs. Well, this small aspect partially answers the question. ISP proxies fit somewhere between datacenter and residential proxies.

Most of the time, you’ll find that an ISP proxy is actually a residential proxy, hosted on a datacenter machine. Hence, these proxies inherit advantages from the other two. And the list of benefits include for the most part (but are not limited to):

IP legitimacy - using a residential IP address lowers the risk of bot detection

Speed - hosting the proxy on a datacenter server increases the performance of the service

The use of proxies in web scraping is quite a common need. But before discussing ISP proxies in particular, let me tell you about why proxies are important for scraping. To begin with, let’s define what web scraping is. On a high level, web scraping is accessing a server with the aim of extracting resources. And that is usually done using automated software. Moreover, web scraping typically involves sending a lot of requests to the targeted server in a short period of time. 

As you can imagine, this puts a lot of load on the server. And that is why web platforms are typically not ‘happy’ about scrapers accessing their servers. To prevent access from automated software, these platforms usually use some sort of detection and prevention system. And one of the methods of detection is as basic as it can be: checking the IP address. It’s common sense to think that IP addresses associated with data centers are more likely to host bots. 

And I think this answers the question quite well. If we account for the main advantages discussed previously, we’ll have a wider understanding of the answer. We primarily use ISP proxies in web scraping to increase our success rate while maintaining an optimal performance. But that is not it. Let’s explore other scenarios:

I am sure you’ve come across websites that target visitors from specific locations. In SEO, this concept is known as geo-location specific content. What happens is that websites first check the origin of the client’s IP address. And if it matches their pattern (say it’s a US website targeting US only clients) it will allow the client to connect. If on the other hand the client is from a foreign country, the website will lock its access. 

In web scraping, this is a very common scenario. So, as a workaround, we will use proxies from that specific country. You may first want to try out a datacenter proxy. If you still get locked, you can then try ISP proxies, which again, offer a higher success rate. 

When we want to access many resources on a server, we may put a lot of load on that server. And servers will usually see it as abuse and block the IP address that is sending all these requests. In web scraping, to avoid getting blocked, we would use a rotating system that switches between ISP proxies. This way, the server will ‘think’ there are different residential users accessing it. Hence, the bulk requests won’t get blocked. 

There are primarily two types of web scrapers:

The first type is not that complex and it’s usually faster, as it does not require rendering JavaScript. It’s no different than sending a `curl` command. However, most modern websites rely heavily on JavaScript. A good example is using a simple HTTP client to scrape a web platform built with React or Next JS. In such scenarios, you will probably get back an HTML file that says ‘please enable JavaScript`. Here is a solid example of an app I’ve built with React:

~ » curl https://<REACT_APP>.com

...<body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

So, if you want to build a modern scraper, you will most likely need to use an automated version of a web browser. This will help you render JavaScript and also perform different actions on your target. 

I usually use JavaScript and Puppeteer for my examples. But I think this time I will switch to my first and favorite programming language: Python. And the go-to solution for webdrivers in Python is Selenium. So let’s see how to build a web scraper with selenium and ISP proxies:

First things first, let’s create a new directory that will hold our files. Next, open the project in your favorite IDE (mine is Visual Studio Code) and open a new terminal. To open a new terminal from within VSCode, go to Terminal > New terminal. We’ll create a new virtual environment inside the project and activate it:

~ » python3 -m venv env && source env/bin/activate

In your project, let’s create a new ‘scraper.py’ file and add some code to it. The basic structure of a scraper with Selenium, from a functional programming perspective is:

from selenium import webdriver

def scrape_page(url):

   driver = webdriver.Chrome()

   driver.get(url)

   return driver.page_source

But remember we want to use ISP proxies with selenium. Such that our browser is not the stealthiest, but let’s say more undetectable. Luckily, things are quite simple in Python (and that’s why I love it). Here is how we introduce proxies in Selenium:

from selenium import webdriver

def scrape_page(url, proxy):

   options = webdriver.ChromeOptions()

   options.add_argument('--proxy-server=%s' % proxy)

   driver = webdriver.Chrome(options=options)

   driver.get(url)

   return driver.page_source

print(scrape_page('http://httpbin.org/ip', '75.89.101.60:80'))

We only added more lines inside the function. The last one is to call the function. If you run the script now, we should probably be able to see that the request originates from 75.89.101.60. For the purpose of this example, I’ve used a free proxy server from here. But if you want to build a real scraper, I suggest you look into more reliable sources. Preferable, proxy providers that offer ISP proxies as well.

How to check the origin of an IP address?

There are a few ways you can check to see if an IP address originates from an ISP proxy. But since we’re talking DevOps today, you should probably get yourself comfortable with using the terminal. Today I will introduce you to `whois`. 

In Unix and Unix-like operating systems, `whois` is built-in. It is a command line utility that allows users to look up information about targets. And the targets can be either domain names or IP addresses. So let’s fire up a new terminal window and test this command. 

First of all, let’s send a `curl` command to the API offered by ipify.org. This way, you can get your own IP address and perform the test using it. If you’re not familiar with `curl`, just go over my article on how to use curl. 

Now that we have an IP address to test on, just send your `whois` command. I’ve used my own IP, but feel free to replace <IP_ADDRESS> with yours:

~ » whois <IP_ADDRESS>

...

inetnum:        82.78.XX.0 - 82.78.XX.XX

netname:        RO-RESIDENTIAL

descr:          RCS & RDS Residential CGN

descr:          City: Bucuresti

remarks:        INFRA-AW

country:        RO

...

The output is larger, but I wanted you to have an overview of how easily a residential IP is detected. There are also public APIs that track datacenter IPs, such as the one offered by incolumitas.

Today we have explored both the DevOps and the Coding skills of building a web scraper. To wrap up, I’ll ask you a simple question. Can we say that ISP proxies are nothing more than datacenter proxies, hidden behind a residential IP address? I think this is not the most accurate definition, but it sure describes these proxies pretty well. 

At Web Scraping API, we’re using both residential and datacenter proxies. That is because some targets allow traffic for non residential, others don’t. If you want to learn more about how you can use proxies with our API, check out our documentation.

What Are ISP Proxies And How To Use It For Web Scraping


Web scraping is an incredible tool for extracting valuable information from the internet, but let's be real, it can be pretty frustrating when your scraping scripts get blocked.

It's like a game of cat and mouse, with website owners always coming up with new ways to keep you out. But, there's a secret weapon in your toolbox that can give you the upper hand: HTTP headers and cookies.

These two elements play a critical role in how your scraping scripts interact with websites, and mastering them can mean the difference between a blocked scrape and a successful one.

In this tutorial, we'll uncover the secrets of HTTP headers and cookies and show you how to use them to make your scraping efforts as human-like as possible.

You'll learn about the most common headers used in web scraping, how to grab headers and cookies from a real browser, and how to use custom headers to bypass security measures. So, let's dive in and see how we can take our scraping game to the next level!

HTTP headers are key-value pairs that you send as part of an HTTP request or response. They are separated by a colon and a space and their names (keys) are case-insensitive.

You can group HTTP headers into different categories, depending on their function and the direction in which you send them. These categories include:

general headers: apply to both request and response messages

request headers: contain more information about the resource you want to fetch, or about the client making the request

response headers: hold additional information about the response, such as its location or the server providing it

entity headers: contain information about the body of the resource, such as its size or MIME type

extension headers: used to provide backward compatibility

HTTP headers provide a wide range of information, including the type of request you are making, the browser you are using, and any additional information that the server needs to process the request.

They also allow you to provide authentication and security information, control caching and compression, and specify the language and character set of the request.

For example, here are the headers that Chrome sends when accessing youtube.com:

:authority: www.youtube.com

:method: GET

:path: /

:scheme: https

accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9

accept-encoding: gzip, deflate, br

accept-language: en-US,en;q=0.9

cache-control: no-cache

pragma: no-cache

referer: https://www.google.com/

sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"

sec-ch-ua-arch: "x86"

sec-ch-ua-bitness: "64"

sec-ch-ua-full-version: "109.0.5414.75"

sec-ch-ua-full-version-list: "Not_A Brand";v="99.0.0.0", "Google Chrome";v="109.0.5414.75", "Chromium";v="109.0.5414.75"

sec-ch-ua-mobile: ?0

sec-ch-ua-model: ""

sec-ch-ua-platform: "Windows"

sec-ch-ua-platform-version: "15.0.0"

sec-ch-ua-wow64: ?0

sec-fetch-dest: document

sec-fetch-mode: navigate

sec-fetch-site: same-origin

sec-fetch-user: ?1

upgrade-insecure-requests: 1

user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36

x-client-data: CIm2yQEIorbJAQjEtskBCKmdygEI5PDKAQiWocsBCKr2zAEI7oLNAQibiM0BCLqIzQEI9YjNAQ==

Decoded:

message ClientVariations {

  // Active client experiment variation IDs.

  repeated int32 variation_id = [3300105, 3300130, 3300164, 3313321, 3324004, 3330198, 3357482, 3359086, 3359771, 3359802, 3359861];

}

Web cookies, or HTTP cookies, are small text files that a website stores on the user’s browser. After that, every time the user sends a new request to the website, the cookies are included automatically.

They are uniquely identifiable, and you can find them as:

session cookies: they are temporary and expire when the user closes the browser

persistent cookies: they have a specific expiration date and remain on the user's device until they expire or are deleted by the user.

Websites use web cookies to personalize the user’s experience. This may include from remembering their login information and storing shopping cart contents to understanding how users interact with the website and delivering targeted advertising.

Taking youtube.com as an example here as well, we can notice what cookies stores:

cookie: CONSENT=YES+srp.gws-20210816-0-RC3.ro+FX+801; __Secure-3PAPISID=jG4abs_wYhyzcDG5/A2yfWlePlb1U9fglf; VISITOR_INFO1_LIVE=pJuwGIYiJlE; __Secure-3PSIDCC=AEf-XMRV_MjLL0AWdGWngxFHvNUF3OIpk3_jdeUwRiZ76WZ3XsSY0Vlsl1jM9n7FLprKTqFzvw; __Secure-3PSID=RAi8PYLbf3qLvF1oEav9BnHK_eOXwimNM-0xwTQPj1-QVG1Xwpz17T4d-EGzT6sVps1PjQ.; YSC=4M3JgZEwyiA; GPS=1; DEVICE_INFO=ChxOekU1TURJMk1URTBOemd5TWpJeU5qVTJOdz09EOvgo54GGOvgo54G; PREF=tz=Europe.Bucharest&f6=40000000&f4=4000000; CONSISTENCY=ACHmjUr7DnoYSMf5OL-vaunKYfoLGz1lWYRUZRepFyIBDRpp_jrEa85E4wgRJLJ2j15l688hk9IVQu7fIjYXo7sdsZamArxVHTMuChgHd22PkX_mbfifnMjyp4OX2swyQJRS-8PE6cOCt_6129fGyBs; amp_adc4c4=Ncu7lbhgeiAAYqecSmyAsS.MXVDWTJjd3BXdmRkQ3J0YUpuTkx3OE5JcXVKMw==..1gn4emd4v.1gn4en556.0.4.4

The importance of headers and cookies in web scraping

In web scraping, you can use headers and cookies to bypass security measures and access restricted content, as well as to provide information that can help identify your scraping script as a legitimate browser.

For example, by specifying the correct User-Agent header (more details in the following section), you can make your script appear as if it is a Chrome browser, which can help you avoid detection by the website.

Also, by storing and sending cookies, your scraper can access content that is only available to logged-in users. As the website uses cookies to provide targeted advertising, you can also use them to extract more accurate data and get a better understanding of the website.

Besides playing with HTTP headers and cookies, consider some pretty useful web scraping best practices listed in this guide.

There are many different headers that can be used in web scraping, but some of the most commonly used are:

It is used to identify the browser and operating system being used by the client making the request. This information is used by servers to determine which browser and operating system the client is using in order to provide the appropriate content and features.

The User-Agent string contains information such as the browser name, version, and platform. For example, a User-Agent string for Google Chrome on Windows might look like this:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36

In web scraping, by specifying a User-Agent that belongs to a common web browser, you can make your script appear as a browser that is commonly used by legitimate users, thus making it less likely to be blocked.

It is used to specify the types of content that the browser is willing to accept in response to an HTTP request, such as text, images, audio, or video. An Accept header may look like this:

Accept: 
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8

In web scraping, by specifying the correct Accept header, you can ensure that your script receives the correct data. For example, if you want to scrape an HTML page, you can specify the Accept header as text/html.

However, some websites can use this header to detect scrapers, so it's important to use it carefully and only when necessary.

This header specifies the preferred language of the content that the browser is willing to accept in response to an HTTP request. The Accept-Language header is used by servers to determine which language to send to the client.

A browser requesting an HTML page in English might send an Accept-Language header that looks like this:

Websites use this header to send cookies to a client after a request is made. A Cookie header may look like this:

Cookie: session_id=1234567890; user_id=johndoe

In web scraping, you can use this header to pass session cookie and access content that is available only to logged-in users. Another use case would be to use persistent cookies to get personalized results.

This header specifies the URL of the previous webpage from which a link to the current page was followed. It's used by servers to track the origin of the request and to understand the context of the request.

For example, if a user clicks on a link from a webpage to another webpage, the browser sends a request to the second webpage with the URL of the first webpage in the Referer header. So the request to the second webpage would have a Referer header that looks like this:

In web scraping, you can use the Referer header to access websites that do not allow direct requests.

Retrieving headers and cookies from a real browser

But enough theory. Let’s see how you can extract HTTP headers and web cookies that are sent to a website.

Back to our example, navigate to youtube.com in your real browser. Then right-click anywhere on the page, and choose the “Inspect” option. When the Developer Tools open, go to the “Network” tab.

After refreshing the page you should see the requests loading in real time. Click the first (and the main) one, and an additional tab will appear.

Here you can see all the details of the request you sent: URL, method, status code, remote address, and most importantly: the request and response headers that we were looking for.

use httpbin.org to show the headers sent by a browser

pick a programming language and send a basic GET request to see the headers

repeat the process for node.js and python

rewrite the section for WSA
      - link to API docs
      -describe the feature of custom headers

Now let’s see how we can make use of these headers to improve our scrapers. For this part, we’ll take as an example a simple mirroring website, which will directly show us the headers we send: https://httpbin.org/headers.

All Science of Web Scraping articles

News and updates