The Internet is a vast place with over 4.7 billion users, and it’s continuously growing. To put it into perspective, over 50% of the earth’s population uses the Internet as of 2018.

Of course, more users mean data as well. Right now, the Internet is so big that it’s estimated that Google, Amazon, Microsoft, and Facebook alone store somewhere around 1.2 million terabytes.

Even a fraction of that information can create new business opportunities. With that in mind, it’s no wonder how popular web scraping has become.

The Internet’s wealth of data is woefully spread thin across billions of websites. As such, developers need a way to collect and process it, to provide users with new innovative products. However, manual information gathering is definitely not a great idea as the volume is often too great, and the data changes continuously.

The solution is automatically extracting it. That’s what web scraping does in a nutshell.

With more information comes more ideas, opportunities, and benefits. Once processed, it can be invaluable to you or your customers. Here are just a few ways web scraping can be used:

Price comparison tools - Scraping multiple websites to get an overview of how a product type is priced.

Market research - Learning who are your most important competitors and what they are doing.

Machine Learning - Collecting training and testing data sets for a machine learning model.

Any idea that requires access to a considerable amount of data.

For example, a possible use case is building a nutritional app that allows users to add their meals. Ideally, the users will just open the app, search for the products they ate, add them to the tool, and keep track of how many more calories they can eat afterward.

However, the tool needs to provide an extensive list of all possible products and their nutritional values. This list can be created and automatically updated by scraping the nutritional information from multiple websites.

While web scraping is very handy for whoever is using the bot, sometimes websites aren’t happy to share their content, and they may try to stop you. Some of the ways they may choose to do so are:

Captcha Codes - Any page can use Captchas, even if it doesn’t show. When you make multiple requests, a captcha code may show up and break your web scraper.


IP Blocking - Some websites choose to block your IP once they see excessive traffic coming from your side.


Geo-Blocking - Some content may be available just in specific countries, or you might receive data particular to a region when you want information on another.


Javascript - Most of today’s websites use javascript in one way or another. Some may display their content dynamically, complicating matters since the page source is not the same as the rendered page content.

Overcoming these challenges can require a lot of work, but there are options. To help you, we created WebscrapingAPI, which takes care of all these problems while helping you build your solution faster and with fewer headaches.

Whenever an Internet user accesses a website, the browser will create an HTTP (Hypertext Transfer Protocol) request. You can think of a request as a message from the client (the user’s computer) to the server (the computer where the website is located) where the client specifies what he wants to receive.

For each request sent, you’ll get a response. The response can be successful, or an error, such as the famous ‘404 page not found error code. The content of a website is usually found in the body of the response received from the server.

Both the requests and the response contain a head, and a body used to exchange information. Also, the requests can be of multiple methods, the most common being GET (this is used when accessing a web page). These methods indicate the action the client wants to do. 

For example, when registering or updating your password on a website, you want your data to be hidden in the browser, and websites can use the method POST or PUT for this type of request.

The head of a request contains multiple properties. Let’s go over the most important ones:

User-Agent - Details about the client that made the request, such as browser and operating system.

Cookie - A set of properties associated with the client.

Accept - A parameter used to receive the response for the server in a specific type such as text/plain or application/json.

Requests are exclusive to web pages, though. They are made for images, styles, and javascript code as well, separately from the page. You can get a glance at all the requests your Google Chrome browser makes when accessing a web page by pressing F12 on a page, choosing the tab “Network” and refreshing the page you are on. You should see at the end something like this:

PHP is one of the oldest and most popular web programming languages used for application backends. It’s been around since 1995, and now it’s on its 8th version. 

Programmers choose this programming language because of its simple syntax and easy way to run, as all you need to run PHP code is a machine with PHP installed. Furthermore, since it’s been around for so long, there are many resources and support for solving and debugging PHP errors.

PHP also has many popular frameworks and CMSs (Content Management System) built on this programming language. Famous examples are WordPress, Drupal, Magento, and Laravel.

Still, there are some disadvantages as well. For example, it’s more difficult to scrape dynamic content compared to Python or Javascript. However, if you only need information from simple pages, PHP is definitely a good solution, and it can help you save or store the scraped data much easier.

All good so far? Ready to create your first web scraper? Before starting, you should have a way to run your PHP code. You can choose an Apache/Nginx server with PHP installed and run the code directly from your browser, or you can run the code from your command line.

Let’s make our lives easier by using a library to process the scraped content. Some of the popular PHP scraping libraries are Goutte, Simple HTML DOM, Panther, and htmlSQL. Alternatively, you may choose to process the content using regular expressions.

For this guide, we will use Simple HTML DOM. However, for more advanced requests, we will also use the PHP library called CURL.

Simple HTML Dom it’s a library developed for PHP versions from 5.6 upwards, and it allows us to access the page’s content in a much easier way — with selectors. You can download the library from here, and you should also read the documentation.

From the zip file in the download link, you will only need the simple_html_dom.php file, which you should put in the same folder where you’ll write the code for the scraper.

To include the library in the code, you only need this one line of code:

include 'simple_html_dom.php'; // If the library is in another folder you should do include 'path_to_library/simple_html_dom.php'

While it’s not always necessary, you’ll need to send different headers for more advanced requests. Using the PHP-CURL library will help.

To install it on a Ubuntu machine, you can use the following command:

After installing the library, don’t forget to restart your Apache/Nginx server.

Now that we have all that we need, it’s time to extract data! First, you should decide on the website and the content you want to scrape. For this article, we will scrape the contents from IMDB’s Top Rated Movies List.

Most web content is displayed using HTML. Since we need to extract specific content from the HTML source, understanding it is required as well. We need first to inspect what the page source looks like to know what elements to extract from the page. 

You can do this in Google Chrome with a right-click on the element you want to extract and then choosing “Inspect Element”. This should open a window in your browser with the page source and the rendered styles of the elements. From this window, the only tab that we need to check is “Elements”, which will show us how the page’s HTML dom is structured.

For example, the page contains a table with the class “chart” and “full-width” in the image above. In this table, each cell has its own class (posterColumn, titleColumn, and so on), which we can use to create a selector. Then, we can access only the needed data.

Confused? Don’t worry, the following steps will clarify things.

Sending a request, in this case, basically means accessing the HTML of a page directly with PHP code. There are two ways to do that.

First, we can use the PHP-CURL library, which also allows us to modify the headers and the body we send in our request.

<?php
 
header("Content-Type: text/plain"); // We choose to display the content as plain text
 
$ch = curl_init("https://www.imdb.com/chart/top/");
curl_setopt($ch, CURLOPT_HEADER, 0);
$response = curl_exec($ch); // Running the request
 
if (curl_error($ch)) {
    echo curl_error($ch); // Displaying possible errors from the request
} else {
    echo $response; // Displaying the content of the response
}
 
curl_close($ch);
?>

Another option is a one-liner, using the file_get_contents($url) method, but this can be insufficient for some instances. To send headers to this request, you need to use a context created with the stream_context_create method.

<?php
 
header("Content-Type: text/plain"); // We choose to display the content as plain text
 
echo file_get_contents('https://www.imdb.com/chart/top/'); // We retrieve and display the contents of the response in a single line
 
?>

You should decide on what method to use based on the complexity of the scraper that you want to build.

The two pieces of code from above will display the HTML source of the page that we are scraping, the same one that is visible when you inspect the website. We’ll use the first line of the code to display the results as a text/plain. Otherwise, it will be rendered directly as html.

If there are any differences in the HTML structure, then a javascript code is running on the website and changing the content once a user accesses it. We’ve prepared a tip on how to deal with that later in the article.

From our chosen page, we will extract only the title of the movies and the rating associated with each of them. As we saw earlier, the content is displayed in a table where each cell has its class. 

Using this, we can choose to extract all the rows of the table. Then we look through each individual row for the cells that interest us. 

The following piece of code should do just that:

<?php
 
header("Content-Type: text/plain"); // We choose to display the content as plain text
 
include 'simple_html_dom.php';
 
$html_dom = file_get_html('https://www.imdb.com/chart/top/'); // We retrieve the contents using file_get_html from simple_html_dom
$table_rows = $html_dom->find('table.chart tbody tr'); // Getting all of the table rows
foreach($table_rows as $table_row) {
    $title_element = $table_row->find('.titleColumn a', 0);
    $rating_element = $table_row->find('.ratingColumn strong', 0);
    if (!is_null($title_element) && !is_null($rating_element)) { // Checking if the row has a title and a rating column
        echo $title_element->innertext . ' has rating ' . $rating_element->innertext . PHP_EOL; // If it does then we print it
    }
}
 
?>

You can notice that we used the selector “table.chart tbody tr” to extract all of the rows of the table. It’s good to use selectors that are as specific as possible so that you can differentiate the elements you need from the rest. 

After retrieving the rows, we looped through them, looking for elements with the class titleColumn or ratingColumn. If the code found any, it displayed their innerText property.

It’s important to note that we used file_get_html instead of file_get_contents for this example. That’s because this function comes from the simple_html_dom library, and it works as a wrapper for the file_get_contents function.

In the above examples, we collected the site data and displayed it directly on the screen. However, you can also save the data in PHP quite easily.

You can save the scraped data in a .txt file, as a JSON, as a CSV, or even send it directly to a database. PHP it’s very good at that. We just have to store it in an array and put the array’s contents in a new file.

<?php
 
include 'simple_html_dom.php';
 
$scraped_data = [];
 
$html_dom = file_get_html('https://www.imdb.com/chart/top/'); // We retrieve the contents using file_get_html from simple_html_dom
$table_rows = $html_dom->find('table.chart tbody tr'); // Getting all of the table rows
foreach($table_rows as $table_row) {
    $title_element = $table_row->find('.titleColumn a', 0);
    $rating_element = $table_row->find('.ratingColumn strong', 0);
    if (!is_null($title_element) && !is_null($rating_element)) { // Checking if the row has a title and a rating column
        $scraped_data[] = [
            'title' => $title_element->innertext,
            'rating' => $rating_element->innertext,
        ];
    }
}
 
file_put_contents('file.json', json_encode($scraped_data)); // Saving the scraped data in a .json file
 
// Saving the scraped data as a csv
$csv_file = fopen('file.csv', 'w');
fputcsv($csv_file, array_keys($scraped_data[0]));
 
foreach ($scraped_data as $row) {
    fputcsv($csv_file, array_values($row));
}
 
fclose($csv_file);
 
?>

Web Scraping with PHP: The Ultimate Guide To Web Scraping

APIs come in many shapes and sizes. While plenty of developers have APIs to thank for making their jobs more manageable in a hundred different ways, not many actually take the time to learn more about these interfaces.

We get it. Time is limited, and understanding what defines a REST API might not be useful for everyone. We won’t deny that, but there’s a reason why just about anyone should learn this: it’s damn interesting. Plus, if you want to design or work with an API, it pays to know more about it.

So, in this article, we’ll take a look under the hood of REST APIs and understand how their design principles have made them the stape software they are today.

The famous abbreviation stands for “Representational State Transfer,” which is a software architectural style. Its most common use is to facilitate the use of Web services with a standardized, universal approach.

These Web services offer their Web resources in a textual representation and allow them to be read and processed with a stateless protocol. The operations that a client can do are predefined and well-known, generally based on the HTTP protocols.

RESTful APIs and software aren’t based on some technological or programming advancement. They aren’t even that new, being first thought up in 2000. The idea behind REST is to impose certain rules upon the API so that you get more performance, scalability, simplicity, modifiability, visibility, portability, and reliability.

Those are a lot of benefits, but how do REST APIs achieve them, you may ask? Simple, by following six constraints. In fact, these rules are the most defining features of the RESTful architectural style.

The six architectural constraints of REST APIs

An API’s job is to connect two pieces of software without limiting their own functionalities. This objective is one of the core restrictions of REST: the client (that makes requests) and the server (that gives responses) stay separate and independent.

When done properly, the client and server can update and evolve in different directions without having an impact on the quality of their data exchange. This is especially important in various cases where there are plenty of different clients a server has to cater to. Think about weather APIs — they have to send data to tons of different clients (all types of mobile devices are good examples) from a single database.

For an API to be stateless, it has to handle calls independently of each other. Each API call has to contain the data and commands necessary to complete the desired action.

An example of a non-stateless API would be if, during a session, only the first call has to contain the API key, which is then stored server-side. The following API calls depend on that first one since it provides the client’s credentials.

In the same case, a stateless API will ensure that each call contains the API key and the server expects to see proof of access each time.

Stateless APIs have the advantage that one bad or failed call doesn’t derail the ones that follow.

While the client and the server change in different ways, it’s important that the API can still facilitate communication. To that end, REST APIs impose a uniform interface that can easily accommodate all connected software.

In most cases, that interface is based on the HTTP protocols. Besides the fact that it sets rules as to how the clients and the server may interact, it also has the advantage of being widely known and used on the Internet. Data is stored and exchanged through JSON files because of their versatility.

To keep the API easy to understand and scale, RESTful architecture dictates that the design is structured into layers that operate together.

With a clear hierarchy for these layers, executing a command means that each layer does its function and then sends the data to the next one. Connected layers communicate with each other, but not with every component of the program. This way, the overall security of the API is also improved.

If the scope of the API changes, layers can be added, modified, or taken out without compromising other components of the interface.

It’s not uncommon for a stateless API’s requests to have large overhead. In some cases, that’s unavoidable, but for repeated requests that need the same data, caching said information can make a huge difference.

The concept is simple: the client has the option to locally store certain pieces of data for a predetermined period of time. When they make a request for that data, instead of the server sending it again, they use the stored version.

The result is simple: instead of the client sending several difficult or costly requests in a short span of time, they only have to do it once.

Unlike the other constraints we talked about up to this point, the last one is optional. The reason for making “code on demand” optional is simple: it can be a large security risk.

The concept is to allow code or applets to be sent through the API and used for the application. As you can imagine, unknown code from a shady source could do some damage, so this constraint is best left for internal APIs where you have less to fear from hackers and people with bad intentions. Another drawback is that the code has to be in the appropriate programming language for the application, which isn’t always the case.

The upside is that “code on demand” can help the client implement their own features on the go, with less work being necessary on the API or server. In essence, it permits the whole system to be much more scalable and agile.

The focus of RESTful design is on efficiency and scalability in a web-dominated world. As you might imagine, this has made architecture quite popular, and we’ll most likely see it become even more widespread in the coming years.

The primary concern for many developers and security experts is how exploitable web APIs can be if created without proper care. Of course, hackers have been and will continue to be a problem for APIs and software in general. Still, it’s not like we’ll stop using them. Rather, it’s up to security experts to find new ways to protect our programs.

That way of thinking, together with the goal of making developers lives easier is what lead us to develop WebScrapingAPI, a data extraction REST API that handles everything from proxy rotation to Javascript rendering and solving CAPTCHAs. Check it out to see what a REST API can do!

Rest API architectural constraints

Making HTTP Requests is one of the most important features of any modern programming language. Node.js is no exception to this rule, but until now this feature was in the hands of way too many npm packages that are out there. Node-Fetch offers an alternative to this, by using the original fetch API that is currently supported by most of the browsers out there.

Before I start telling you about the Node-Fetch API, I must give you a brief information about what is an HTTP Request. An HTTP Request's purpose is to retrieve information from urls all over the internet. A simple example of an HTTP Request is accessing a website.

While long ago, HTTP Requests were done using XMLHttpRequest or XHR Objects, nowadays all modern browsers support the Fetch API from javascript. This allows programmers to make the requests with a much simpler and cleaner syntax. However the fetch API was missing for a long time from the Node.JS server side language, leaving room for other custom made packages to take care of this feature, such as: Axios, GOT and many others.

Node-Fetch is the equal of the Fetch API from javascript, and now it's finally available in Node.JS as well.

First of all since this is a Node.JS tutorial, you will need of course to have Node.JS installed. If you don’t have it already, you can download and install it from this link.

Node.js has released experimental support for the Fetch API only starting with the version 17.5. So you will need to have at least the Node 17.5 version. Also you will need to use the –experimental-fetch flag when running your scripts.

If you have a lower version of Node.JS, to jump to the latest version you can use the n package. N is a npmjs package with the sole purpose of allowing you to switch between node and npm versions. To install it and switch to the latest version, follow these steps:

The n latest command will install the latest version of Node.js. To verify your Node.js version simply run this command:

Making an HTTP Request in any programming language is an asynchronous operation, since receiving the response from the request will take time. There are two approaches when it comes to how you can use asynchronous operations. You can either decide to wait for the response and then continue with your code, or run your code in parallel.

Node-Fetch supports both synchronous and asynchronous function calls.

To make a simple get request and extract the body from the response you can use the piece of code bellow:

fetch('https://www.webscrapingapi.com/')

    .then((response) => response.text())

    .then((body) => {

        console.log(body);

    });

To run this code you should save it in a file called node-fetch-example.js and run it from the same folder using this command: node --experimental-fetch node-fetch-example.js. Note that when running it, you will get a warning saying that “The Fetch API is an experimental feature”. This is normal since this is an experimental feature at the time of writing this article.

The previous piece of code does not wait for the request to finish before continuing its execution. This means that any code below this one will start executing immediately, without waiting for the fetch to finish. For example if you would add a console.log(“Something”); below the code, when executing the script the output will look like this:

To further explain the code from above, you may notice that we use the “then” function twice. The first “then” will execute when we receive a response from the HTTP Request, and what it does is to map that response with the contents from the response.text() method (which returns the body of the response). But, the response.text() method is asynchronous as well, and thus we need to wait for its response in the second “then”, where body is the equal of the response.text() promise’s result.

You can also call the fetch API using await like we do in the following example:

(async () => {

    const response = await fetch('https://webscrapingapi.com');

    const body = await response.text();

    console.log(body);

})();

This gives an even better explanation on how the fetch API works and what are the promises that you need to wait for. Further in this article we will use the fetch api with await, as this allows us to keep a cleaner syntax for the code.

Another feature you will need when sending requests is to be able to set the headers of the request you’re making. To do that you may add the headers in the second parameter of the fetch API, like this:

(async () => {

    const response = await fetch('http://httpbin.org/headers', {

        headers: {

            'my-custom-header': 'my-header-value'

        }

    });

    const body = await response.text();

    console.log(body);

})();

Along with the headers there are a lot more options that you may send in the second parameter of the fetch API. To see them all  check the documentation of the fetch API (the one used client side).

Another important option from the fetch API is the method option. This specifies the method you’re using for the HTTP Request. There are 5 methods you can use: GET, POST, PUT, PATCH and DELETE, but out of them, the first two are the ones most commonly used (GET and POST). By default if no method is specified, Node-Fetch will use GET.

To make a POST request using Node-Fetch you can use this code snippet:

(async () => {

    const response = await fetch('http://httpbin.org/post', {

        method: 'POST',

        body: JSON.stringify({

            'key': 'value'

        })

    });

    const body = await response.text();

    console.log(body);

})();

What you may notice here is that we use JSON.stringify to send the body of the request. This is because the fetch api sends the body as a string instead of an object, like other packages such as axios do.

The fetch API covers all the other request methods as well.

Handling errors when making HTTP requests is a must, since you can never count on a third party service to be always available. As a best practice you should always handle errors, to prevent your app or script from going down alongside the url that you’re making the request to.

Error handling in node-fetch can be done by surrounding the code with a simple try catch syntax. Here is a sample on how to do that when using await:

(async () => {

    try {

        const response = await fetch('[INVALID_URL]');

        const responseBody = await response.text();

    } catch (error) {

        console.log(error.message);

    }

})();

If you prefer to use fetch without await instead, then you can add a catch to your code like this:

fetch('[INVALID_URL]')

    .then((response) => response.text())

    .then((body) => {

        console.log(body);

    })

    .catch((error) => {

        console.log(error.message);

    });

Making HTTP Requests can be useful in many ways as it allows getting new information from different services and extracting data in quite an elegant and easy manner. There are a few use cases for this that we will explore in the following paragraphs.

When programming, oftentimes you may need to use an API. The reason for this is that you may need to get specific data from a different backend source, and then process it or update it. A good example here would be an API with 4 endpoints that allows you to create, read, update and delete users (CRUD operations) in a backend database from another server.

Oftentimes, such an API would need authentication to prevent unauthorized sources from using it and changing the data to their advantage. HTTP Requests have many methods for authentication. One of the most common ones is using an API Key, where the API provider gives you a key that only you should know, and the endpoints of the API work only when sending the correct key.

Another method an API may be protected through, is Basic Authentication. This means that to access the API you will need to send a header with a base64 encoded string of the format “username:password”. Here’s an example of how you may use basic authentication in a POST request:

(async () => {

    const response = await fetch('http://httpbin.org/post', {

        method: 'POST',

        headers: {

            "Authorization": `Basic ${btoa('login:password')}`

        },

        body: JSON.stringify({

            'key': 'value'

        })

    });

    const body = await response.text();

    console.log(body);

})();

Web Scraping is a way of getting content from websites and parsing it, so that you keep only the needed data, which you may use as you wish. A good npmjs library that you can use to make the parsing of the data easier is cheerio. This library allows you to query static html, once obtained from the fetch api, the same way as you would query it from javascript or jquery.

Here’s an example of how to get the title of a page using the fetch API and cheerio:

const cheerio = require("cheerio");

(async () => {

    const response = await fetch('https://www.webscrapingapi.com/');

    const responseBody = await response.text();

    const $ = cheerio.load(responseBody);

    console.log($('title').first().text());

})();

The example above should return “WebScrapingAPI | All-In-One Scraping API” as this is the title of the page (the text written on top of the window in your browser). To break it down, we use fetch to get the HTML page source from https://www.webscrapingapi.com/ and we use cheerio to parse that content. To find out more about cheerio you can check out their documentation from here 

Scraping information from other websites can be useful in many ways. For example scraped information can be used to create a training dataset for a machine learning model or to create a price comparison tool that extracts data from many sources and then compares it.

While the example above works fine, the fetch api may not always be the best option when it comes to scraping. That is because modern websites nowadays display their content through javascript and use captchas or other methods to prevent their data from being scraped. The fetch API works as a simple CURL to the given url and it retrieves the static content that the page displays when it’s loaded, without any javascript rendering at all.

To scrape data while also executing the javascript code on the page you may want to look into alternatives such as puppeteer, as described in this article about advanced scraping. If you don’t want to go through this whole trouble however, you can use WebScrapingAPI, an API made especially for this task, that takes care of all these problems (including antibot detections) and it comes with a free trial with all the features included. 

To summarize, the good news is that the long awaited fetch api is finally available in node.js even though for now it’s only in its experimental feature stage (at the time of writing this article). While making requests in node.js was possible before, the only way to do them was through either the XMLHttpRequest/XHR objects or one of the many packages out there such as Axios or GOT.

This change will make client side javascript and server side nodejs more alike, since this feature was already available and supported by all modern browsers on the client side javascript.

Making HTTP requests can be useful in many ways, such as using an API or scraping data from a website. While the other npm packages will remain an option and will continue to be used on legacy projects, using fetch is the best solution for the future. This will improve nodejs code readability and it will make the switch from frontend to backend even easier.

Learn How To Use Node-Fetch NPM To Make HTTP Requests In Node.js

Web scraping with Go it's a great way of creating a fast and powerful scraper. This is because GoLang is one of the best programming languages you can use for concurrency. But before jumping right into it, I must first tell you more about what web scraping is, and how it can help you.

Web scraping is the process of extracting data from websites. This process can be done manually, but this approach is not recommended when dealing with a lot of data. In this article we will explore how you can build your own automated web scraper from scratch with Go.

If you’re new to this, you may wonder what are some of the use cases that Web Scraping has. Here is a small list with a few of the most common ones:

Price Comparison Tools - You can build many tools using a web scraper. One of the most common and useful ones is a price comparison tool. Such a tool would scrape the prices for a product from many sources, and display the best deal possible.

Machine Learning - If you want to build a model using machine learning, you will need a training dataset. While sometimes you may find existing datasets that you can use, oftentimes you will need to do some extra work, and get the data that you need yourself.

Market Research - A third use case is scraping information from the internet to find out who your competitors are and what they are doing. This way you can keep up or stay ahead of the competition, being aware of any new feature that they might have added.

What will you need for scraping data with go

Before we begin you will need to be able to run GoLang code on your machine. To do that all you need is to install Go, if you hadn’t already. You can find out more details on how to install Go, and how to check if you have it installed over here.

The other thing you will need is an IDE or a text editor of your choice where we will write the code. I prefer to use Visual Studio Code, but feel free to use whatever you may see fit.

And that’s it. Pretty simple, no? Now let’s dive into the main topic of this article, web scraping with go.

For building our scraper we will first need a purpose, a set of data, that we want to collect from a specific source. Thus as the topic for our scraper I chose scraping the weekly downloads of the first packages from npmjs.com that are using the keyword “framework”. You can find them on this page: https://www.npmjs.com/search?q=keywords:framework&page=0&ranking=optimal)

Inspect the contents of the page you want to scrape

To do scraping right, before you start actually extracting the data, you will need to see where the data is. By this I mean that you will need to build the HTML selectors to query the data, based on the HTML structure of the page.

To see the HTML structure of the page, you can use the developer tools available in most of the modern browsers. In Chrome, you can do this on the page by right clicking on the element you want to extract and click “Inspect Page”. Once you do that you will see something like this:

Based on the HTML that you can see on the right (the inspect window), we can now build the selectors we will use. From this page we only need the URLs of each of the packages.

By looking over the HTML, we can see that the css classes used by the website are code generated. This makes them not reliable for scraping so we will use the html tags instead. On the page we can see that the packages are  in <section> tags and that the link to the package is in the first div from the first div of the section.

Knowing this we can build the following selector to extract the links of all the packages: section > div:first-child > div:first-child a. Before trying it in code, we can test the selector from the developer tools of the browser. To do this go to the console tab and run document.querySelectorAll("{{ SELECTOR }}"):

By hovering over each of the nodelist elements returned, we can see that they are exactly the ones we were looking for, and so we can use this selector.

We’re finally starting to build the scraper! To do that you should first create a folder where we will put all our code. Next, you need to open a terminal window, either from your IDE or from your operating system and go to our folder.

To open a terminal in the folder, using Visual Studio Code you can click on Terminal -> New Terminal (from the top bar).

Now that we have our terminal opened, it’s time to initialize the project. You can do this by running the command: 

go mod init webscrapingapi.com/my-go-scraper

This will create in your folder a file called go.mod with the following contents:

module webscrapingapi.com/my-go-scraper
go 1.19

To make the request to the page and extract the selectors from the HTML, we will use Colly, a GoLang package (check out Colly documentation for more info). To install this package you have to run 

Now that we have everything prepared, all we need to do is create our main.go file, and write down some code. The code for extracting all the links from the first page of npmjs frameworks, is the following:

package main

import (
    "fmt"
    "github.com/gocolly/colly"
)

func scrape() {
    c := colly.NewCollector()

    // Find and print all links
    c.OnHTML("section > div:first-child > div:first-child a", func(e *colly.HTMLElement) {
        fmt.Println(e.Attr("href"))
    })
    c.Visit("https://www.npmjs.com/search?q=keywords:framework&page=0&ranking=optimal")
}

func main() {
    scrape()
}

If this seems hard to read at first, don’t worry, we will break it down in the following paragraphs and explain it. 

Every golang file should start with the package name and the imports that Go will use. In this case the two packages we use are “fmt” to print the links we scrape, and "Colly" (for the actual scraping).

In the next part, we created the function scrape() which takes care of scraping the links we need. To do that the function visits the first page, and waits for finding the selector we decided on. When an element from that selector shows up, it goes ahead and prints the href attribute of that element.

The last part is the main function which is the function that is called everytime we run a golang script. To execute the previous code run go run main.go from your terminal, and you should get the following output:

As you can see, the href attribute has relative paths to the links, so we will need to prefix it with the npmjs url.

Use GoLang’s concurrency for efficiency

One of the coolest features of GoLang are the GoRoutines. GoRoutines are simple lightweight threads managed by the Go runtime. What’s great about this is that Go can help us scrape many urls at the same time with a lightning speed.

Before we extracted the links for the first 20 packages under the keyword “framework” on npmjs.com. Now we will attempt to scrape all these links at the same time and extract the weekly downloads for each of them. To do that we will be using GoRoutines and WaitGroups.

Here is the final code to extract the weekly downloads using the goroutines:

package main

import (
    "fmt"
    "github.com/gocolly/colly"
    "sync"
)

func scrapeWeeklyDownloads(url string, wg *sync.WaitGroup) {
    defer wg.Done()

    c := colly.NewCollector()

    // Find and print the weekly downloads value
    c.OnHTML("main > div > div:last-child > div:not([class]) p", func(e *colly.HTMLElement) {
        fmt.Println(fmt.Sprintf("%s - %s", url, e.Text))
    })
    c.Visit(url)
}

func scrape() {
    c := colly.NewCollector()

    var wg sync.WaitGroup

    // Find and print all links
    c.OnHTML("section > div:first-child > div:first-child a", func(e *colly.HTMLElement) {
        wg.Add(1)
        go scrapeWeeklyDownloads(fmt.Sprintf("%s%s", "https://www.npmjs.com", e.Attr("href")), &wg)
    })
    c.Visit("https://www.npmjs.com/search?q=keywords:framework&page=0&ranking=optimal")

    wg.Wait()
}

func main() {
    scrape()
}

Now let’s discuss what was added to our previous code. First you will notice that we have a new package imported called “sync”. This will help us use golang routines and wait for the threads to finish, before stopping the execution of the program.

The next thing added is the new function called “scrapeWeeklyDownloads”. This function accepts two parameters: the url of the link we’re gonna scrape and a WaitGroup pointer. What this function does is to visit the url given and extract the weekly downloads (using the selector main > div > div:last-child > div:not([class]) p).

Last changes you will notice were in the scrape function where we created a WaitGroup using var wg sync.WaitGroup. Here, for each link from the packages page, we used wg.Add(1) and then created a GoRoutine that calls the scrapeWeeklyDownloads function. At the end of the function, the instruction wg.Wait() makes the code wait until all the GoRoutines finish executing. 

For more information on WaitGroups please check out this example from golang. 

By using concurrency in golang with GoRoutines and WaitGroups we can create a very fast scraper. Running the previous code sample will return the page and the weekly downloads of each package. But, because we’re using multi threading, the order in which these informations will show, it’s unknown (as threads execute at different speeds)

If you’re running the code using linux or windows subsystem linux (wsl), you can use time go run main.go to see the execution time of the whole script. For me, the execution time is somewhere around 5 to 6 seconds. This is very fast, considering we’re scraping 21 pages (First the page with the packages and then the pages of each of the packages).

Most scrapers usually count on making a simple HTTP Request to the page, to get the content they need. This solution is good but sometimes websites display their information through javascript rendering. That means that the website will first show you only a part of its content, loading the rest of it dynamically through javascript.

To scrape such pages you will need to use chromedriver and control a real chrome browser. While there are some options to do this in Golang, you will need to do some extra research for this topic.

Even with javascript rendering covered there are still some extra obstacles when scraping a website. Some websites may use antibot detections, ip blocks or captchas to prevent bots from scraping their content. To still scrape those websites you may try to use some tips and tricks for web scraping, such as making your scraper slower and to act more human-like.

But, if you want to keep your scraper fast and to tackle these obstacles in an easy way, you may use WebScrapingAPI. WebScrapingAPI is an API designed to help you with scraping by rotating your IP and avoiding antibot detections. This way you can continue to use the lightning speed that GoLang provides and scrape your data in no time.

Web Scraping is a nice and fast way to extract data from the internet, and it can be used for many different use cases. You can choose to extract data for your machine learning model or to build an application from scratch using the data you scraped.

GoLang it's one of the best solutions out there when it comes to concurrency. Using GoLang and Colly you can build a fast scraper that will bring your data in no time. This makes web scraping with Go very easy and efficient once you get used to Go’s syntax

The Ultimate Guide On How To Start Web Scraping With Go

Web scrapers are valuable tools that help you extract specific information from a website. Theoretically, you could do this manually, but web scraping allows you to process vast amounts of data more efficiently and productively.

One of the most popular programming languages for web scraping is Python. This language comes with the library BeautifulSoup, which simplifies the process. Together, this duo makes web scraping a lot easier than in other languages.

In my personal opinion, using BeautifulSoup is the easiest way to build a simple web scraper from scratch. If you want to learn more about this, read on because I’ll show you how to create your own web scraper using Python and BeautifulSoup.

BeautifulSoup, as stated in their documentation, is a python library for pulling data out of HTML and XML files. So, you can use Python to extract the HTML content from a website and then use BeautifulSoup to parse that HTML to get just the relevant information.

The main advantage of using BeautifulSoup it’s the simple syntax that it offers. With this library, you can navigate the DOM tree, search for specific elements, or modify HTML content. All of these benefits made it the most popular python library for parsing HTML and XML documents.

To install BeautifulSoup, you should check the guide from here, as the installation is different based on the machine you use. In this article, I’m using a Linux system and only need to run the following command:

If you use python3, you might need to install the library by using the following command instead:

Keep in mind that my machine already has python3 installed. If you’re new to Python, you can find a guide on how to install it here. Also, you should check out our ultimate guide to building a web scraper with python for even more information on the topic.

Now, if everything went smoothly, we’re ready to begin building our own scraper. For this article, I chose to retrieve the top 100 movies of all time from RottenTomatoes and save everything in both JSON and CSV formats.

To warm up and get familiar with BeautifulSoup, we will first retrieve the full HTML of the page and save it in a new file called “page.txt”. 

Sorin-Gabriel Marica