We all hate the moments when we try to access a page and the site blocks our request for no good reason. Geoblocking, for example, can be passed only by using a proxy. 

Node-unblocker helps us to create a custom proxy and have it up and running in a matter of minutes.

Node unblocker is a general-purpose library to create a web proxy, intercept and alter requests and responses. 

This library is also used in web scraping for bypassing restrictions implemented by the site like geo-blocking, hiding the IP address and rate limiting, or for sending authentication tokens. 

To make it shorter, while using this library, you can kiss blocked and censored content goodbye.

In this article, we create an Express application with a custom proxy using Node Unblocker, we add a middleware that changes the user agent for each request, discuss the proxy limitations, deploy it to Heroku and compare it to a managed service like WebScrapingAPI.

Before we start, make sure you have the latest version of Node.JS installed. Installing Node.JS for each platform (Windows, Linux, Mac) would be the subject of a separate article so instead of going into details, check the official website and follow the instructions.

We start by creating a directory for our project named unblocked and we initialize a Node.JS project inside of it:

For this application, we install two libraries: Express, a minimalist framework for Node.JS, and Node Unblocker.

Because of node.unblocker running inside an Express instance, we have to set up the Hello World example in our application.

Create an index.js file and paste the following code:

const express = require('express') const app = express() const port = 8080  app.get('/', (req, res) => {     res.send('Hello World!') })  app.listen(process.env.PORT || 8080, () => {     console.log(`Example app listening on port ${port}`) })

We can run the application with this command:

If we access http://localhost:8080 we see a Hello World message. This means our application is up and running.

It’s time to import the Node Unblock library into our application:

We create a node unbloker instance and pass the proxy parameter. You can find the full list of available parameters here.

var unblocker = new Unblocker({prefix: '/proxy/'})

We register the node unblocker library to Express as a middleware so requests are intercepted:

We update the Express app listener to add support for websockets:

app.listen(process.env.PORT || 8080, () => {     console.log(`Example app listening on port ${port}`) }).on('upgrade', unblocker.onUpgrade)

After following all these steps, our application should look like this:

const express = require('express') const Unblocker = require('unblocker') const app = express() const port = 8080  const unblocker = new Unblocker({prefix: '/proxy/'})  app.use(unblocker)  app.get('/', (req, res) => {     res.send('Hello World!') })  app.listen(process.env.PORT || 8080, () => {     console.log(`Example app listening on port ${port}`) }).on('upgrade', unblocker.onUpgrade)

Restart the application and access the following URL in your browser:

http://localhost:8080/proxy/https://webscrapingapi.com

To make sure the proxy works as expected, we open the Developer Tools in the browser and check the Network tab. All requests should go through the proxy.

For any issues the proxy might throw, it’s recommended to enable the debug mode by setting the DEBUG environment variable. Use this command to start the proxy in debug mode:

It’s never a good idea to enable this in production so let’s keep it only for the development environment.

Nodeunblocker is not just a custom proxy solution but allows the interception and alteration of outgoing and incoming requests through middleware.

We can use this feature to block specific resources from loading based on the resource type or domain, update the user agent, replace returned content, or inject authentication tokens in request headers.

You can find a full list of examples here.

Let’s start by creating a middleware for setting a custom user agent. Create a file called user-agent.js and add this code to it:

module.exports = function(userAgent) {
    function setUserAgent(data) {
        data["headers"]["user-agent"] = userAgent
    }
    return setUserAgent
}

This function accepts the custom user agent with the userAgent parameter and registers it in the data object using the setUserAgent function. Node unblocked will call the setUserAgent function with each request.

const userAgent = require('./user-agent')

We set the requestMiddleware parameter in the Unblocker constructor and we should be good to go.

const unblocker = new Unblocker({
    prefix: '/proxy/',
    requestMiddleware: [userAgent("nodeunblocker/1.5")]
})

const express = require('express')
const Unblocker = require('unblocker')
const userAgent = require('./user-agent')
const app = express()
const port = 8080

const unblocker = new Unblocker({
    prefix: '/proxy/',
    requestMiddleware: [userAgent("nodeunblocker/1.5")]
})

app.use(unblocker)

app.get('/', (req, res) => {
    res.send('Hello World!')
})

app.listen(process.env.PORT || 8080, () => {
    console.log(`Example app listening on port ${port}`)
}).on('upgrade', unblocker.onUpgrade)

It’s time to check if our code works. We have to change the node-unblocker URL to make sure the headers are properly updated. 

Restart the application and open this URL in your browser:

http://localhost:8080/proxy/https://www.whatsmyua.info/

If the site displays nodeunblocker/1.5, our middleware works.

With our proxy running, it’s time to deploy it to Heroku, a platform as a service (PaaS) that allows us to create, launch, and manage apps fully in the cloud.

Keep in mind that not all providers allow proxies and web scraping apps on their infrastructure. Heroku accepts these kinds of apps as long as the rules from robots.txt are not ignored.

With the legal part discussed, let’s prepare our project for deployment.

We have to set the start script and the engines in the package.json file.

The `engines` property tells Heroku we require the latest version of Node.JS 16 installed in our environment. The start script is executed when the environment is set and our application is ready to run.

unblocked

 Heroku made the deployment of a Node.JS application a bliss. Before moving to the next section, make sure you have installed the Heroku CLI and the git tools.

Use this command to authenticate with Heroku from your local terminal:

Create a new Heroku application by running this command:

This command will return the app’s ID and a git repository. Let’s use the ID to set the remote origin for our repository:

git init
heroku git:remote -a [YOUR_APP_ID]

Because versioning the node_modules folder is never a good idea, let’s create a .gitignore file and add the folder to it.

The last step before our code will reach production is to commit it and deploy it. Let’s add all the files, create a commit and merge the master branch in the heroku one.

git add .
git commit -am "Initial commit"
git push heroku master

After a few seconds, the application will be deployed to Heroku. Congratulations! It’s time to access it in our browser and make sure it works.

Use the following URL structure to build the Heroku URL:

[HEROKU_DYNO_URL]/proxy/https://webscrapingapi.com

If you forgot or lost the Dyno URL, you can use this command to get the available information about the current app:

The ease of setup for this custom proxy comes with a caveat: it works well only for simple sites and it fails for advanced tasks. Some of these limitations are impossible to overcome and will require the use of a different library or third-party services.

A managed service like WebScrapingAPI implements fixes for all these limitations and adds a few extra features like automatic captcha solving, residential proxies, and advanced evasions to prevent services like Akamai, Cloudflare, and Datadome to detect your request. 

Node Unblocker for Web Scraping

Web scraping is the process of extracting data from publicly available sites like forums, social media sites, news sites, e-commerce platforms etc. To give you a sneak peek for what you will build today, this article describes the creation of a web scraper in Elixir.

If the definition of web scraping is still not clear to you, let’s consider saving an image from a site as manual web scraping. If you want to save all the images from a site by hand, based on the site complexity, it might take hours or even days. 

You can automate this process by building a web scraper.

Maybe you’re wondering what are some use cases of a web scraper. Here are the common ones:

You can scrape the latest news from your favourite financial news site, run a sentiment analysis algorithm and know in what to invest minutes before the market will start and move

You can scrape the comments from your social media pages and analyse what your subscribers are talking about and how they are feeling about your product or service.

If your passion is collecting consoles and video games but you don’t want to spend a fortune on the latest PS5, you can make a web scraper that retrieves the eBay listings and sends a notification to you when a cheap console is on the market.

If you want to create a mobile application that can identify the breed of a cat in any picture, you’ll need a lot of training data; instead of saving hundreds of thousands of pictures with cats manually to train the model, you can use a web scraper to do it automatically.

We will build our web scraper in Elixir, a programming language built on top of Erlang, created by José Valim, member of the core Ruby on Rails team. The programming language borrows the Ruby syntax simplicity and combines it with Erlang's ability to build low-latency, distributed, and fault-tolerant systems.

Before you write the first line of code, make sure you have Elixir installed on your computer. Download the installer for your operating system and follow the instructions from the Install page. 

During the installation, you’ll notice the Erlang programming language is required as well. Keep in mind that Elixir runs in the Erlang VM so you need both.

In this article, you will learn how to create a web scraper in Elixir, scrape the products from eBay for PS5 listings and store locally the extracted data (name, url, price).

It’s time to inspect the search results on the eBay page and collect some selectors.

Go to ebay.com, enter the term PS5 in the Search input and click the Search Button. Once the search results page is loaded, open the Inspect tool from your browser (right click anywhere on the page and select Inspect).

You need to collect the following selectors:

Use the select element tool and search for the product item list (ul) and the product item (li):

Using these two elements, you can extract the classes needed for the crawler for data extraction:

.srp-results .s-item the child elements of the products list element (ul)

Let’s create an Elixir project using the mix command:

The --sup flag generates an OTP application skeleton including a supervision tree, a needed feature for an application that manages multiple concurrent processes like a crawler.

Change the current directory to the project root:

We use this directory to store the scraped items.

After the project is created, you need to add the two dependencies:

Crawly isan application framework for crawling sites and extracting structured data

Floki is an HTML parser that enables search for nodes using CSS Selectors

Open the mix.exs file and add the dependencies in the deps block:

defp deps do
    [
        {:crawly, "~> 0.14.0"},
        {:floki, "~> 0.33.1"}
    ]
end

Fetch the dependencies by running this command:

Create the config/config.exs file and paste this configuration in it:

import Config

config :crawly,
    closespider_timeout: 10,
    concurrent_requests_per_domain: 8,
    closespider_itemcount: 100,

    middlewares: [
            Crawly.Middlewares.DomainFilter,
            Crawly.Middlewares.UniqueRequest,
            {Crawly.Middlewares.UserAgent, user_agents: ["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"]}
    ],
    pipelines: [
            {Crawly.Pipelines.Validate, fields: [:url, :title, :price]},
            {Crawly.Pipelines.DuplicatesFilter, item_id: :title},
            Crawly.Pipelines.JSONEncoder,
            {Crawly.Pipelines.WriteToFile, extension: "jl", folder: "./temp"}
    ]

Let’s go through each property and make some sense of it:

closespider_timeout: integer, the maximum number of seconds the spider will stay open

concurrent_requests_per_domain: the maximum number of requests that will be made for each scraped domain

closespider_itemcount: the maximum number of items are passed by the item pipeline

By setting the user agent, you improve the scraping results by mimicking a real browser. Sites are not happy with scrapers and try to block any user agents that don’t seem real. You can use a tool like this to get your browser user agent.

WebScrapingAPI rotates the user agent and the IP address with each request and also implements countless evasions to prevent this kind of situation. Your requests will not get blocked and implementing a retry mechanism will give you amazon results.

The pipelines are commands processed from top to bottom and enable the manipulation of the processed items. We use the following pipelines:

Validate fields (title, price, url): checks if the item has the scraped fields set

Duplicate filter: checks for duplicate items by title

JSON Encoder: encodes the structures to JSON object

Write to File: writes the items to the ./temp folder

A web crawler, or a spider, is a type of bot that goes through a site, and extracts data using user defined fields through CSS selectors. A crawler can extract all the links from a page and use specific ones (like pagination links) to crawl more data. 

It’s time to set the base for the crawler: create the ebay_scraper.ex file in the lib/elixir_spider folder and paste the following code to it:

# lib/elixir_spider/ebay.ex
defmodule EbayScraper do
    use Crawly.Spider

    @impl Crawly.Spider
    def base_url(), do: ""

    @impl Crawly.Spider
    def init() do

    end

    @impl Crawly.Spider
    def parse_item(response) do

    end
 end

This is just the skeleton for the file and it won’t run or return any results. Let’s talk about each function first and then we fill them one by one.

The base_url() function is called once and it returns the base URL for the target website the crawler will scrape; it’s also used to filter external links and prevent the crawler from following them. You don’t want to scrape the entire Internet.

@impl Crawly.Spider
def base_url(), do: "https://www.ebay.com/"

The init() function is called once and is used to initialise the default state for the crawler; in this case, the function returns the start_url from where the scraping will begin.

Replace your blank function with this one:

@impl Crawly.Spider
def init() do
    [start_urls: ["https://www.ebay.com/sch/i.html?_nkw=ps5"]]
end

All the data extract magic happens in the parse_item(). This function is called for each scraped URL. Within this function, we use the Floki HTML parser to extract the fields we need: title, url and price.

@impl Crawly.Spider
def parse_item(response) do
    # Parse response body to document
    {:ok, document} = Floki.parse_document(response.body)

    # Create item (for pages where items exists)
    items =
        document
        |> Floki.find(".srp-results .s-item")
        |> Enum.map(fn x ->
         %{
           title: Floki.find(x, ".s-item__title span") |> Floki.text(),
           price: Floki.find(x, ".s-item__price") |> Floki.text(),
           url: Floki.find(x, ".s-item__link") |> Floki.attribute("href") |> Floki.text(),
         }
    end)

    %{items: items}
end

As you might have noticed, we’re using the classes we found in the Getting Started - Inspecting the target section to extract the data we need from the DOM elements.

It’s time to test the code and make sure it works. From the project root directory, run this command:

iex -S mix run -e "Crawly.Engine.start_spider(EbayScraper)"

If you’re using PowerShell, make sure you replace iex with iex.bat otherwise you will get an error for the non-existent -S parameter. Use this command for PowerShell:

iex.bat -S mix run -e "Crawly.Engine.start_spider(EbayScraper)"

Open the ./temp folder and check the .jl file. You should see a text file containing a list of JSON objects, one per line. Each object contains the information we needed from the eBay products list: title, price and URL.

Here’s how the product object should look like:

PS5 Sony PlayStation 5 Console Disc Version!  US VERSION!

We retrieved all the products from the first products list page but this is not enough. It’s time to implement pagination and let the crawler extract all the available products.

Let’s change the parse_item() function and add a new block that creates a requests struct with the next pagination link. Add this code after the items code:

# Extract the next page link and convert it to a request
requests =
  document
  |> Floki.find(".s-pagination a.pagination__next")
  |> Floki.attribute("href")
  |> Crawly.Utils.build_absolute_urls(response.request_url)
  |> Crawly.Utils.requests_from_urls()

Update the return statement from the parse_item() function to include the next requests as well. The struct will look like this:

%{
  :requests => requests,
  :items => items
}

Run the crawler again but this time prepare a coffee. Scraping all the pages for the PS5 listings will take some minutes.

The Ultimate Guide On How To Start Web Scraping With Elixir

The importance of information gathering has been known since ancient times, and people who used it to their advantage have prospered.

Today we can do that much easier and faster by using a scraping tool, and creating your own scraper isn’t difficult, either. The ability to gather leads faster, keep an eye on both the competition and your own brand, and learn more before investing in ideas is at your fingertips.

If you are interested in knowing more about web scraping or how to build your tool in C#, you should tag along!

Well, it is legal as long as the website you wish to scrape is ok with it. You can check that by adding “/robots.txt” to its URL address like so http://httpbin.org/robots.txt and reading the permissions, or by looking through their TOS section.

Web scraping is an automated technique used by companies of all sizes to extract data for various purposes, such as price optimization or email gathering. Researchers use web scraping to collect data reports and statistics, and developers get large amounts of data for machine learning. 

How does it work? Well, for most web scraping tools, all you need to do is specify the URL of the website you wish to extract data. Depending on the scraper's abilities, it will extract that web page’s information in a structured manner, ready for you to parse and manipulate in any way you like. 

Take into account that some scrapers only look at the HTML content of a page to see the information of a dynamic web page. In this case, a more sophisticated web scraping tool is needed to complete the job.

Using a web scraper is very useful as it can reduce the amount of time you’d normally spend on this task. Manually copying and pasting data doesn’t sound like a fun thing to do over and over again. Think about how much it would take to get vast amounts of data to train an AI! If you are interested in knowing more about why data extraction is useful, have a look!  

Let’s see how we can create our web scraping tool in just a few minutes.

For this tutorial, I will show you how a web scraper can be written in C#. I know that using a different programming language such as Python can be more advantageous for this task, but that doesn’t mean it is impossible to do it in C#. 

Coding in C# has its advantages, such as:

Has better integrity and interoperability;

First things first, you need to decide what web page to scrape. In this example, I will be scraping Greece on Wikipedia and see what subjects are presented in its Table of Contents. This is an easy example, but you can scale it to other web pages as well.

Using the developer tools, you can inspect each element to check and see under which tag the information you need is. Simply right-click on the web page and select “inspect”, and a “Browser Inspector Box” will pop up. 

You can search for the class directly in the elements section or using the inspect tool on the web page as shown below.

Thus, you found out that the data you need is located within the span tag which has the class toctext. What you’ll do next is extract the whole HTML of the page, parse it, and select only the data within that specific class. Let’s make some quick preparations first!

You can use whatever IDE is comfortable for you. In this example, I will use Visual Studio Code. You will also need to install .NET SDK.

Now you need to create your project. To do so, you obviously have to open up Visual Studio Code. Then, you will go to the extensions menu and install C# for Visual Studio Code.

You need a place to write and run our code. In the menu bar, you will select File > Open File (File > Open… on macOS) and in the dialog, you will create a folder that will serve as our workspace.

After you created the workplace, you can create a simple “Hello World” application template by entering the following command in our projects’ terminal:

Next, you need to install these two packages:

HtmlAgilityPack is an HTML parser written in C# to read/write DOM.

CsvHelper is a package that’s used to read and write CSV files.

You can install them using these command lines inside your projects’ terminal:

dotnet add package csvhelper
dotnet add package htmlagilitypack

Let’s import the packages we installed a few minutes ago, and some other helpful packages for later use:

using CsvHelper;
using HtmlAgilityPack;
using System.IO;
using System.Collections.Generic;
using System.Globalization;

Outside our Main function, you will create a public class for your table of contents titles.

public class Row
{
    public string Title {get; set;}
}

Now, coming back to the Main function, you need to load the page youwish to scrape. As I mentioned before, we will look at what Wikipedia is writing about Greece!

HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("https://en.wikipedia.org/wiki/Greece");

Our next step is to parse and select the nodes containing the information you are looking for, which is located in the span tags with the class toctext.

varHeaderNames = doc.DocumentNode.SelectNodes("//span[@class='toctext']");

What should you do with this information now? Let’s store it in a .csv file for later use. To do that, you first need to iterate over each node we extracted earlier and store its text into a list.

CsvHelper will do the rest of the job, creating and writing the extracted information into a file.

The code is done and now we just have to run it! Use this command line in the terminal. Be sure you saved your file first!

I hope this article helped you better understand the basics of web scraping using C#.

It is very useful to have a scraper of your own but keep in mind that you can only scrape one web page at a time and still have to manually select the tags inside the website’s HTML code.

If you want to scrape several other pages, it will be a lot faster to use a scraper than to select the information manually, yes, but not all websites can be scraped using this method. Some websites are dynamic, and this example won’t extract all their data.

Have you ever thought about using a pre-made and more advanced tool to help you scrape en masse? 

How about checking out what an API can do for you? Here is a guide written by WebScrapingAPI to help you choose an API that might fit your needs.

Suciu Dan