In order to reach the scope of this article which is learning how to use cURL with a Proxy, we first need to introduce cURL. Client URL(cURL) is, for short, an easy-to-use command line designed for developers to fetch some data from a server.

As I already mentioned above, using cURL is pretty straightforward and can extract information with just a one-line command. Firstly you need to open an terminal and type curl followed by an website link, for example: 

Congratulations, you made your first request using cURL. This simple command requests information from the server just like a traditional browser does and returns the HTML of the page. Not every website will give you back HTML, there are endpoints which send back data as a JSON object. Take this example:

$ curl 'https://jsonplaceholder.typicode.com/todos/3'

Type this command in your terminal and you should get back this response:

{
  "userId": 1,
  "id": 3,
  "title": "fugiat veniam minus",
  "completed": false
}

Most API’s will give you back either HTML or JSON when you are running cURL commands against them. Well this is not everything cURL can do for us. In reality it is a very sophisticated tool which requires a lot of time to master. If you want to learn more about cURL, I strongly recommend you to take a look at cURL documentation for a better understanding of its parameters. Alternatively you can run the following command: 

This will show you some options you can set to cURL: 

Usage: curl [options...] <url>
 -d, --data <data>   HTTP POST data
 -f, --fail          Fail silently (no output at all) on HTTP errors
 -h, --help <category>  Get help for commands
 -i, --include       Include protocol response headers in the output
 -o, --output <file>  Write to file instead of stdout
 -O, --remote-name   Write output to a file named as the remote file
 -s, --silent        Silent mode
 -T, --upload-file <file>  Transfer local FILE to destination
 -u, --user <user:password>  Server user and password
 -A, --user-agent <name>  Send User-Agent <name> to server
 -v, --verbose       Make the operation more talkative
 -V, --version       Show version number and quit

This is not the full help, this menu is stripped into categories.
Use "--help category" to get an overview of all categories.
For all options use the manual or "--help all".

As you can probably see these are not even all the options you can set to cURL, it is a menu divided into categories. You probably guessed that in order to get all the options you would like to run:

However, using cURL alone has some limitations regarding the number of servers we can choose to fetch data from. For example, some servers can use geolocalization and refuse to give us the data we are looking for because of our location. This is the moment we need a proxy, which acts like a middleman between us and the target server.

The concept of a proxy server is not hard to understand at all. As already mentioned above , proxy server is like an intermediary between a client requesting a resource and the server providing that resource.Proxies are designated for us to be able to get data from anywhere. In order to understand better this concept , lets assume that we have a server called Bob which has some data we are interested in, but Bob provide that data only if we are in Europe, but we are in United States.

How we deal with that? We send our request to a proxy server which is located in Europe and not to Bob and tell the proxy that we want to get some data from Bob. The proxy will send itself the request to Bob and Bob will return to the proxy server data since the proxy is in Europe. Then the proxy server will send us back the data from Bob.

This is the main flow of how proxies work. Another great use case for a proxy is for example when we want to get data which contains prices in a specific currency in order to avoid confusion. For a further understanding of proxies, I strongly recommend you to have a look on Wikipedia.

In order to use a proxy , you will most likely need a host , a port , an user , a password and a target URL you want to get data from. For this example , I will use a proxy provided by WebScrapingAPI for making requests for which you can find more information about it here. WebScrapingAPI is not a proxy provider, it is a web scraping service which provides proxies instead .In our examples, our setup will be the following:

Proxy username: webscrapingapi.proxy_type=datacenter.device=desktop

Proxy password: <YOUR-API-KEY-HERE> // you can get one by registering here

Please note that there may be some proxy providers which require other schema of authentication.

Since we have covered cURL and proxies , now we are ready to combine them together and make requests by using a proxy which is a pretty straightforward process.We first need to authenticate and then we can use the proxy.

Proxy authentication in cURL is pretty simple and can be done for our example from above as following :

$ curl -U webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR-API-KEY> --proxy @proxy.webscrapingapi.com:80 http://httpbin.org/get

Running that command , httpbin will give us back our IP address , and some other properties:

​​{
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.5173.0 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-633af93b-35faf8637f37f7f075f185ec"
  }, 
  "origin": "45.9.120.69", 
  "url": "http://httpbin.org/get"
}

As you can already probably see, the origin you receive back is not your IP address and it is the address of the proxy server instead. Furthermore, you can run the command even without revealing your password in the terminal. This can be done as following: 

$ curl -U webscrapingapi.proxy_type=datacenter.device=desktop --proxy @proxy.webscrapingapi.com:80 http://httpbin.org/get

And then you will get a prompt to enter your password: 

Enter proxy password for user 'webscrapingapi.proxy_type=datacenter.device=desktop':

Now you can type your API key there without exposing it in the terminal which makes the whole process more secure.Furthermore, typing your credentials , host and port every single time you want to run a cURL command via a proxy may not feel that ideal , especially when you want to run many commands via a proxy and you are using the same proxy provider. 

Of course, you can store your credentials on a separate file stored on your machine and copy paste them everytime, but there is a more natural approach you can take which is passing them via environment variables which we will talk about below.

Using cURL with a proxy via environment variables 

An environment variable is like an object which stores an editable value in the memory which can be used by one or more softwares. In this particular case we can pass to cURL a variable called http_proxy or https_proxy which contains our proxy details and we will not need to specify on every run of the command. You can do that by running this command:

$ export http_proxy="http://webscrapingapi.proxy_type=datacenter.device=desktop:<YOUR-API-KEY>@proxy.webscrapingapi.com:80"

Please note that you have to call your variable http_proxy or https_proxy in order for cURL to understand what you are talking about. That is it, now you do not need to pass your credentials on every run anymore and now you can just run cURL as simple as this: 

As you probably see the IP address is the proxy’s address which confirms that you have made a great job setting your proxy. At this point we can run any cURL command without specifying the proxy details, cURL will take care of that for us.

Disabling the proxy for a specific command

However, if you need to send a specific request without a proxy, you do not need to worry about deleting the value of http_proxy variable. Being a sophisticated tool with a lot of options , cURL can take care of that for us by its noproxy parameter which tells it to not use any proxy when making the request. It can be done as following:

$ curl --noproxy "*" http://httpbin.org/get

And that will give us back our own IP address and not the proxy’s one.

In conclusion, using cURL with a proxy is a great way to bypass geolocation filters, extends the amount of resources we can fetch from webservers and it is a good starting point for getting into topics such as web-scraping where we need to use proxies in order to be able to get certain data or to receive it in the format we want.I hope you found this article useful for you to learn how to use cURL with a proxy and you will play around with it and build your own scripts which extracts data from servers which use geolocation filters.

Find out how to use cURL with Proxy

Have you ever wondered how businesses keep track of the competition? Of course, double agents, private investigators, and binoculars may sound fun, but real life isn’t like the movies. Actual competitor analysis is a tedious process that’s better left to the software. This way, you can focus on delivering the most value to your customers.

Nowadays, your competitors already have all the information you need in the open. With social media gaining a lot of traction in the last couple of decades, businesses are trying to be more present in their customer’s life by creating an online presence. The information is already there. We just have to find a way to automate the data gathering process.

In the following article, we will build a script that will help us gather all the information we need regarding our business competitors by taking advantage of their public Yelp profile.

Investigating competitors' product pricing, overseeing their product line and categories, or analyzing their social media strategy are just three of the most valuable tactics you can use to improve your business.

Finding and comparing product prices is not a very challenging task, but it is time-consuming. Using a web scraping tool can help you automatically get updated information regarding your competitor’s pricing strategy and flash sales.

Scraping your competitors’ product lines and categories helps you better understand the trends that define the market. For example, if you own a restaurant, you can find out that a specific dish like the vegan burger is trending in your city, so you can make adjustments to your menu to drive more sales.

Social media isn’t useful just for strengthening the bond between business and customer. By analyzing your competitors' social media strategy, you can find out what works and what doesn’t in just a couple of minutes. You can easily find answers to questions like “How many times do they post per week?” or “What visuals keep engagement at a high level?” just by scraping.

Why Yelp is the perfect place to scrape competitor data

Yelp is one of the most popular online platforms where users can share their experiences while helping others make informed decisions about local businesses. If you own a business or are looking to start one, finding out valuable information about the competition can improve your success rate by a wide margin. Pieces of information about the exact location, working hours, delivery services, or business type are just a couple of the ones you can find by browsing Yelp.

What’s even better is that you can decrease the time you spend analyzing competition by using a web scraping tool and focus more on building a better business. This strategy will pay off in the long term as automating the data gathering process can become one of the most tedious and time-consuming tasks.

How to use WebScrapingAPI to extract data from Yelp

In the following section, we will use Node.js and a couple of libraries like got and jsdom to create the script that will get all the data from a Yelp page and format it nicely to be as understandable as possible. Let’s see how all the information presented above can be converted to tangible results:

Let’s assume we are interested in opening up a restaurant in New York City. We would want to create a list of competitors that includes helpful information like:

The URL we are going to scrape is the following: https://www.yelp.ie/search?find_desc=restaurant&find_loc=New+York%2C+NY%2C+United+States&ns=1.

You can scrape in tandem with the article if you start your free trial with 5000 requests and access to all functionalities to test our product. You can use whatever scraping tool you feel most comfortable with. Just know that from here on, the tutorial will focus on how to get the results using WebScrapingAPI.

After successfully creating a free account, access the API Playground page by hitting the “Use API Playground” button on the Dashboard page. The page should look like this:

Just as the name suggests, this is the place where we can test the scraping tool before creating our script. Let’s copy the URL presented above in the URL input (left column), scroll down a little bit, and hit the “Send API Request” button. This action should return a result that looks like this:

Now let’s build the script that is going to do most of the work for us.

Create the project’s folder, open up a terminal window and navigate to it. Just as we have said before, we will use the ‘got’ library to make the requests and ‘jsdom’ for our parsing needs. After you have accessed the project’s folder, run the following commands in the terminal:

The project should now include the ‘package.json’ files. Create a new file called ‘index.js’ and open it up using your preferred IDE. If you do not have any installed, we recommend downloading Visual Studio Code, as it’s considered the most popular IDE at the moment.

Let’s set the parameters, make the request and parse the HTML. Write the following lines in the newly created ‘index.js’ file:

const {JSDOM} = require("jsdom")

const got = require("got")

(async () => {
const params = {
    api_key: "YOUR_API_KEY,”
    url:"https://www.yelp.ie/search?find_desc=restaurant&find_loc=New+York%2C+NY%2C+United+States&ns=1"
}

const response = await got('https://api.webscrapingapi.com/v1', {searchParams: params})


const {document} = new JSDOM(response.body).window

const competitors = document.querySelectorAll('.container__09f24__sxa9-')

console.log(competitors)

})()

Be sure to replace the “YOUR_API_KEY” string with the API key provided to you by our service. You can find it on the dashboard page.

By running the script, we make a request to the WebScrapingAPI service to get the Yelp page HTML. We then collect all the elements on the page containing information about our competitors and log it to the screen using the ‘console.log’ function.

Let’s get back to the Yelp page and find out how we can select only the information we need. Right-click on the first restaurant’s name and click ‘Inspect.’

You’ll get a new window containing the HTML source code:

Let’s find all the information we need by looking for the HTML elements that contain it. In the picture presented above, we can easily see that the element containing the restaurant’s name has a CSS class of ‘css-166la90’. To get all our competitors’ names, we have to select all the elements on the page with that class. We will do the same to all the details presented before, like the number of reviews, restaurant type, pricing, and location.

We will now get all the information we need from the resulting API response. We will fetch and format the restaurant’s name, review score, restaurant type, price range, and location. Add the following code lines to the ‘index.js’ file.

competitors.forEach((competitor) => {
    if (competitor) {
        const name = competitor.querySelector('.css-166la90')
        if (name) competitor.name = name.innerHTML

        const reviewScore = competitor.querySelector('.reviewCount__09f24__EUXPN')
        if (reviewScore) competitor.review_score = `${reviewScore.innerHTML}/100`

        const types = competitor.querySelectorAll('.css-1hx6l2b')
        if (types) {
            competitor.types = []
            for (type of types) competitor.types.push(type.innerText)
       }

       const priceRange = competitor.querySelector('.priceRange__09f24__2O6le')
       if (priceRange) competitor.price_range = priceRange.innerHTML

       const locationContainer = competitor.querySelector('.priceCategory__09f24__Ylk7h')
       if (locationContainer) {
	     let location = locationContainer.querySelector('.css-e81eai')
	     competitor.location = location
       }

       results.push(competitor)
    }
})

console.log(results)

In the end, we will have an array of objects, and each of them will contain every competitor on the page and its specific data.

As you can see, scraping Yelp data using WebScrapingAPI is quite easy. We have to use a scraping API to get the HTML content, parse the response, get the relevant information from each element on the page, and add everything to a list.

Getting to know your competition without any extra work

Trying to create a new business nowadays takes a lot more than having a great product. There are endless opportunities depending on how creative you can be. Some of the most valuable strategies business owners should pay attention to are:

working on providing the most value for the lowest price

having a complete understanding of the competition’s advantage

All of these strategies can prove vital in one’s business. It feels good to know that web scrapers offer a huge help in tackling these problems. Adding automation to the data gathering process may be the easiest step to improve their business.

We try to offer a helping hand by creating the necessary tools for these kinds of jobs. Thank you for reading all the way through and remember that WebScraping API has a free trial after which users are downgraded to the free plan so you can test the API to your heart’s content without spending a dime.

How to use WebScrapingAPI to gather Yelp data

The property market is constantly expanding, and with it, real estate agents and businesses try to find new solutions to pinpoint what the future holds. While real estate, in general, doesn’t change drastically overnight, it’s affected by way too many factors for one person or even an organization to keep track of.

So, will the prices rise or will they go down? What neighborhoods are in high demand? Are there properties that just need a makeover to skyrocket their value? These are just a few of the questions real estate agents are frequently asking themselves.

To answer these questions, one needs loads of research data for comparison, and to manually gather such amounts of information would be like a wild goose chase. Here is where web scraping comes in handy, it collects and structures data as fast as you can say:

As we all know by now, web scraping is the powerhouse of data extraction! So, if you want to know more about why anybody would want to scrape real estate data from the Internet and how to do it properly, let’s continue our journey together. We’ve prepared both a DIY solution and a step-by-step guide on how WebScrapingAPI can do it.

Scraping the web will ensure that the extracted information about real estate is precise, credible, and up to date. This way, one can predict if the real estate market will skyrocket any time soon or see in what price range their property will compete.

For businesses, web data is valuable because it leads to better decisions, better pricing, and a more significant profit margin. However, the catch is that each bit of information needs to be as fresh as possible, making web scraping the obvious solution.

The most commonly extracted types of real estate data are the following:

The information listed above can make or break a real estate agency. It makes a huge difference in communication, strategy, and efficiency, but the biggest advantage is how well agents get to know their properties and market. After that, it’s just a matter of finding the right client.

Let’s take a look at a few scenarios that illustrate the value of web scraping:

Decision-making: Taking risks is part of the job, but that doesn’t mean you must do it blindly. Researching before buying or selling something is mandatory to work, and more info means better deals.

Predicting the market: It is crucial to know when to buy and sell properties to get the best and most profitable outcome. Some types of properties soar in popularity while others lose their luster.  Some areas flourish while others stagnate. Knowing what’s around the corner is the key to longevity as a business.

Web scraping isn’t all about helping businesses. Actually, part of what makes it so popular is how easy it is for a single person to use. Sure, you need some knowledge of computer science, but there are plenty of tutorials to help. Heck, this is one of them!

Buying and selling: You need to accurately deduce the property’s value before buying or selling it. It would be a shame to sell your childhood home and see it a week later on a real estate website at double the price, wouldn’t it?

Investing: If you like to invest in properties, either by buying at a small price to sell it later for profit or simply rent the property, it is highly recommended to know fast you’ll break even and what returns you should expect.

Ok, that’s enough on use cases. Let’s look at some code!

For starters, let’s assume we are searching for a new home in New York City. We want to buy a property with at least two bedrooms and, of course, a bathroom. So, we’ll start our search on Realtor, extract data from there and compare it to find the best deal.

There are various ways one can extract content from web pages. This article will explain two methods: one in which we create our web scraper from scratch and one in which we use an already existing tool.

First, let’s try to do it ourselves. The code will later prove helpful once we use a professional web scraping tool.  

Building a web scraper to extract real estate data

I chose to write in Python because of how popular it is in web scraping.  We have a general-purpose tutorial for extracting web data in Python that you should check out!

The data we need to extract can be found in the nested tags of the said webpage. Before we start scraping, we need to find it. To do this, simply right-click on the element and select “Inspect.”

A “Browser Inspector Box” window will pop up, like this:

In this window, we will navigate to find the tags and classes under which our essential data can be found. It might seem a bit intimidating at first, but it only gets easier with experience!

We can see that everything we need to extract is within the <li> tag with the class ‘component_property-card’. If we go even deeper in the tag, we observe that the data referring to the number of beds and bathrooms are under the attribute ‘data-label’ with the values ‘pc-meta-beds’ and ‘pc-beta-baths’, respectively. Knowing this, we can proceed with writing our code!

As mentioned before, we will use Python as our programming language, so you need to download and install it.

You can use whichever IDE you feel comfortable with, but I recommend using PyCharm.

After you’ve created a new project, make your work easier by using these libraries:

Selenium: Used for web testing and automating browser activities.

BeautifulSoup: Used for parsing HTML and XML documents.

Pandas: Used for data manipulation. The extracted data will be stored in a structured format.

Installing them within the project is quite simple. Just use this command line in the project’s terminal: python -m pip install selenium beautifulsoup4 pandas

Let’s start by importing the libraries we’ve installed earlier:

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

To extract the data from the website, we have to load it by configuring the webdriver to use the Chrome browser. To do this, we simply need to specify the path where the chromedriver is located. Don’t forget to add the name of the executable at the end - not just its location!

driver = webdriver.Chrome('your/path/here/chromedriver')

Besides the number of beds and bathrooms, we can also extract the address, price, and, why not, the size of the property? The more information we have, the easier it will be to decide on a new home. 

Declare the variables and set the URL of the to be scraped website.

prices = []
beds = []
baths = []
sizes = []
addresses = []
driver.get('https://www.realtor.com/realestateandhomes-search/New-York_NY')

We need to extract the data from the website, which is located in the nested tags as explained earlier. Find the tags with the previously mentioned attributes and store the data in the variables declared above. Remember that we only want to save properties with at least two beds and one bathroom!

content = driver.page_source
soup = BeautifulSoup(content, features='html.parser')
for element in soup.findAll('li', attrs={'class': 'component_property-card'}):
   price = element.find('span', attrs={'data-label': 'pc-price'})
   bed = element.find('li', attrs={'data-label': 'pc-meta-beds'})
   bath = element.find('li', attrs={'data-label': 'pc-meta-baths'})
   size = element.find('li', attrs={'data-label': 'pc-meta-sqft'})
   address = element.find('div', attrs={'data-label': 'pc-address'})

   if bed and bath:
       nr_beds = bed.find('span', attrs={'data-label': 'meta-value'})
       nr_baths = bath.find('span', attrs={'data-label': 'meta-value'})

       if nr_beds and float(nr_beds.text) >= 2 and nr_baths and float(nr_baths.text) >= 1:
           beds.append(nr_beds.text)
           baths.append(nr_baths.text)

           if price and price.text:
               prices.append(price.text)
           else:
               prices.append('No display data')

           if size and size.text:
               sizes.append(size.text)
           else:
               sizes.append('No display data')

           if address and address.text:
               addresses.append(address.text)
           else:
               addresses.append('No display data')

Great! We have all the information we need, but where should we store it? This is where the pandas library comes in handy and helps structure the data into a csv file for us to use in the future.

df = pd.DataFrame({'Address': addresses, 'Price': prices, 'Beds': beds, 'Baths': baths, 'Sizes': sizes})
df.to_csv('listings.csv', index=False, encoding='utf-8')

If we run the code, a file named ‘listings.csv’ will be created, and in it, our precious data!

We did it! We created our own web scraping tool! Now let’s jump right into it and see what steps we need to follow and which lines of code we need to modify to use a scraping tool.

For this scenario, we will use WebScrapingAPI, of course.

To make use of WebScrapingAPI, you need to create an account. Don’t worry, the first 5000 API calls are free, and you don’t need to share any personal data, like credit card info. After you successfully create your account and validate your email, we can move to the next step.

To use WebScrapingAPI, you will need to authenticate via the private API Key, which you can find on your account dashboard. Note that you mustn’t share this key with anyone, and if you suspect that it has been compromised, you can always reset the key by pressing the “Reset API Key” button.

Perfect! Now that you have the API Key, let’s make the necessary changes.

We won’t be using a webdriver anymore. Instead, the ‘requests’ library will send the request to WebScrapingAPI and retrieve the website’s HTML code as a response.

import requests
from bs4 import BeautifulSoup
import pandas as pd

Next, we have to prepare a few parameters for the request: the url of the website we wish to extract data from (realtor) and our API Key.

url = "https://api.webscrapingapi.com/v1"
params = {
 "api_key": "XXXXXXX",
 "url": "https://www.realtor.com/realestateandhomes-search/New-York_NY"
}
response = requests.request("GET", url, params=params)

Don’t forget to change which content beautifulsoup is parsing. Instead of the source from the chromedriver, we will use the response received from the API.

From this point on, you can use the same code from the previous scenario. The data will still be stored in a CVS file named ‘listings.csv.’

And that’s pretty much it; you can run the code. WebScrapingAPI will do the job, and you’ll get the necessary data to find the perfect home. But you might ask yourself: “What is the difference between using WebScrapingAPI and the scraper we built ourselves?”. Well, allow me to explain.

One of the most significant advantages of using WebScrapingAPI is its proxies. The service has a huge rotating proxy pool that ensures its users’ anonymity while surfing the web.

This feature is also helpful when someone wishes to scrape a website en masse. Making multiple requests on a website in a short amount of time will surely block your IP, thinking it is a grief attempt or a bot with bad intentions.

Using a rotating proxy pool will make the website think that multiple users are interacting with it, so you remain undetected and can scrape all day long.

Many more obstacles can come your way when scraping the web, such as CAPTCHAs or browser fingerprinting. As you might expect, we built WebScrapingAPI to side-step all those hurdles and make data extractions as easy as possible for you. If you want to know more about this topic, check out our article on the most common problems web scrapers encounter.

We can all agree that scraping the web is an excellent solution for the real estate industry, but you can use it for other purposes as well. Here are just a few examples: monitoring your competition, comparing product prices, and training machine learning algorithms.

I could go on, but that’s already a whole new subject. I won’t drag this article on forever, so I recommend you check out these seven use cases for web scraping tools.

Creating a web scraping tool in your free time sounds pretty neat, but there are many things to consider, things that will burn a considerable amount of development time. Here you can find an in-depth discussion about DIY vs. pre-made web scraping tools.

If we are talking about scraping a few web pages, building the tool yourself can be a fast solution. Still, a professional job needs a professional tool, ideally an API, WebScrapingAPI. Did I mention the free trial?

The Complete Guide to Web Scraping Real Estate Data

Web scraping, a well-known subject for this blog’s readers, has multiple benefits for businesses. It is involved in a suite of processes and decisions, and the marketing area makes no exception.

Using web scraping to extract data from public websites tends to be a common practice between businesses in multiple industries. We have talked about frequent use cases of web scraping, but these topics need to be narrowed down and detailed for those who want to go the extra mile. That’s why we wrote about issues like using web scraping for price intelligence and competitors’ research using the different tools available on the market.

But it is time to go further with another marketing process that can be greatly improved. So, in today’s focus, we have SEO and how web scraping can change your entire marketing game and strategy.

Stay tuned to learn more about smart decisions and how to make them.

What SEO means, and how it affects your business

Search Engine Optimization refers to the process of improving one’s website in terms of keywords to increase visibility on the search results page. Increased visibility means a greater possibility to attract prospective and existing customers to your business, which in the end will result in a more profitable outcome.

Google and Bing, or any other search engine, use bots that crawl from website to website, collect information, and index them. Then, all this information is analyzed based on multiple ranking factors or signals. This way, algorithms determine the order pages should appear in the search results for a given keyword.

Search ranking factors cover multiple aspects connected to user experience like content and website architecture. Content quality and keywords used on the website and blog posts are critical factors of content optimization, and website architecture is essential for the ability to crawl and friendliness.

Optimizing the UX and content of your websites are the main on-site methods of boosting its rank on SERPs. Unlike paid ads, search engine results are focused on organic rank growth.

So giving just the right amount of attention to SEO might be an excellent idea if you are trying to hit the first page in search results and organically maximize your website’s visibility.

And let’s be honest. Who doesn’t want to be on the first page when customers search for particular products and services? The competition is high, and the spots on the first page are limited. That’s why you need to use every advantage technology gives you and make sure you maximize your chances.

SEO and web scraping, a love-love relationship

Every relationship needs to be nurtured, even though we are talking about a business one. There are three combinations for maximized results: SEO and web scraping work like fire and gasoline to give you the boost you need for your business.

First, you can track the page ranks by scraping various search engine results pages for particular keywords. You can easily do that by searching for an important keyword that describes your product or service.

For example, we know that WebSrapingAPI is a web scraping tool, so by searching after “web scraping” on Google and scraping the results page, we can learn where WebScrapingAPI ranks for this targeted keyword and which are the direct competitors for this tool. Extracting data from the search result page gives you the opportunity to create personalized reports and charts meant to inform you which keywords are worth investing in.

Second, if you aim to rank higher for more exposure and clicks, our suggestion is to look at what your competitors are using in terms of keywords and phrases.

Using web scraping tools like WebScrapingAPI helps you extract their site content. Having this on your computers makes it easier to run a text analysis and have an accurate insight into the titles, keywords, descriptions, and links. This information offers the opportunity to create high-quality content for your website and blog so that you generate organic traffic from search engines. For this, running the first step can be a great start.

Finally, ranking changes all the time because even the Internet is a living ecosystem in which things take different shapes. That’s why it is crucial to keep an eye on the updated data so that you know how your keywords change in relationship with others. This option of real-time data is an alternative that most web scrapers provide via cloud services, or you can use your programming skills to build the web scraper to check these data for yourself.

Of course, all these steps and procedures have no power if you don’t take actions that match the web scraping process results.

This means that after gathering and analyzing all the needed data, you will probably have to upgrade your marketing strategy, change some keywords and phrases, and pay attention constantly to everyday changes. This is the safest way to the first search results page, which in the end will bring you more clients and more revenue.

Luckily, WebScrapingAPI is here to guide and support you in your SEO journey.

Our product can sustain every step described ahead so that you don’t have to worry about anything. Did we mention that you will receive 1000 free API calls when you sign up?

Using web scraping as an advantage in creating strategies will transform the way business owners make decisions and how they will implement changes into the processes, making the environment faster and faster every day.

So keep your mind open and take every opportunity that web scraping gives you!

How to Use Web Scraping for Better SEO

Have you ever found yourself in the situation of trying to discover the perfect place to spend the holiday? Or maybe you just want to find out how your listing competes with your neighbours’. Either way, why not use the power of web scraping to do this?

A web scraper is a piece of software that helps you automate the tedious process of collecting useful data from third-party websites. Most of the online services offer developers access to an API in order to easily read information from their website. Unfortunately, Airbnb is not one of them. This is where web scrapers come into play.

Airbnb is a platform that gives people the opportunity to rent their properties using just an internet connection. It was founded in 2008 by Brian Chesky, Nathan Blecharczyk, and Joe Gebbia and it has seen massive success even during the pandemic.

Everyone can find the listings on the platform just by accessing Airbnb and searching for a place, but there’s no easy way to find a meaningful dataset with the following information:

Of course, you have your own reasons for why you would want to get this information and I’m sure we can help with this.

To be able to extract all the necessary data, make sure you follow the next steps.

Check out the elements you are interested in scraping off Airbnb’s website. By right-clicking anywhere on the page and selecting the “Inspect” option you will get to see the Developer Tools.

Let’s say we want to get the price, image, type, and rating of the places we will scrape.

First, we are going to find the common element in the DOM. It seems like _gigle7 is what we are looking for.

To get the best results, we recommend using our service, WebScrapingAPI, as this is the one we will base our tutorial on. You can give it a try for free by accessing this link. Create an account and get back to this page after you are done.

After you have logged in, go to the dashboard page. Here, you can find your private API access key, which we’ll use to make the requests, the API playground where you can test out our product, and the documentation.

After you have created a folder for the project, run the following commands:

To make the requests we are going to install the got module, and for our HTML parsing needs, we will use the jsdom package.

Create a new file called “index.js” and open it up.

Let’s set the parameters and make the request and parse the HTML. Write the following lines in the previously created file:

const {JSDOM} = require("jsdom")
const got = require("got")

(async () => {
const params = {
       api_key: "YOUR_API_KEY",
       url: "https://www.airbnb.com/s/Berlin/homes?tab_id=home_tab&refinement_paths%5B%5D=%2Fhomes&flexible_trip_dates%5B%5D=april&flexible_trip_dates%5B%5D=may&flexible_trip_lengths%5B%5D=weekend_trip&date_picker_type=calendar&source=structured_search_input_header&search_type=filter_change&place_id=ChIJAVkDPzdOqEcRcDteW0YgIQQ&checkin=2021-04-01&checkout=2021-04-08"
   }

   const response = await got('https://api.webscrapingapi.com/v1', {searchParams: params})
const {document} = new JSDOM(response.body).window

   const places = document.querySelectorAll('._gig1e7')

})()

As we have previously stated, all the relevant information can be found under the _gigle7 element, so we are going to fetch all the elements that are assigned the _gigle7 class. You can log it to the screen by adding a console.log() action just right after the line where we define the places constant.

From here on, we will dig deeper to get the specific elements containing the price, image type, and rating information.

After the previously presented lines of code, copy the following:

const results = []

   places.forEach(place => {

       if (place) {
           const price = place.querySelector('._ls0e43')
           if (price) place.price = price.querySelector('._krjbj').innerHTML

           const image = place.querySelector('._91slf2a')
           if (image) place.image = image.src

           const type = place.querySelector('._b14dlit')
           if (type) place.type = type.innerHTML

           const rating = place.querySelector('._10fy1f8')
           if (rating) place.rating = rating.innerHTML

           results.push(place)
       }

   })

console.log(results)

As you can see, for each listing we get on the first page, we fetch the price tag element, the image source location, the type of the listing, and the rating. In the end, we will have an array of objects, and each of them will contain every element in this list.

Now that we have written all the code necessary to scrape the Airbnb information, the index.js file should look something like this:

const {JSDOM} = require("jsdom");
const got = require("got");

(async () => {
   const params = {
       api_key: "YOUR_API_KEY",
       url: "https://www.airbnb.com/s/Berlin/homes?tab_id=home_tab&refinement_paths%5B%5D=%2Fhomes&flexible_trip_dates%5B%5D=april&flexible_trip_dates%5B%5D=may&flexible_trip_lengths%5B%5D=weekend_trip&date_picker_type=calendar&source=structured_search_input_header&search_type=filter_change&place_id=ChIJAVkDPzdOqEcRcDteW0YgIQQ&checkin=2021-04-01&checkout=2021-04-08"
   }

   const response = await got('https://api.webscrapingapi.com/v1', {searchParams: params})

   const {document} = new JSDOM(response.body).window

   const places = document.querySelectorAll('._gig1e7')
   const results = []

   places.forEach(place => {

       if (place) {
           const price = place.querySelector('._ls0e43')
           if (price) place.price = price.querySelector('._krjbj').innerHTML

           const image = place.querySelector('._91slf2a')
           if (image) place.image = image.src

           const type = place.querySelector('._b14dlit')
           if (type) place.type = type.innerHTML

           const rating = place.querySelector('._10fy1f8')
           if (rating) place.rating = rating.innerHTML

           results.push(place)
       }

   })

   console.log(results)

})()

As you can see, scraping Airbnb data using WebScrapingAPI it’s pretty simple.

Make a request to WebScrapingAPI using the necessary parameters: the API key and the URL we need to scrape data from.

Select all the listings by finding the specific class.

For each listing, get the price tag, image, listing type, and rating.

Add every place to a new array called results.

Log the newly created results array to the screen.

The response should look something like this:

[
  HTMLDivElement {
    price: '$47 per night, originally $67',
    image: 'https://a0.muscache.com/im/pictures/miso/Hosting-46812239/original/c56d6bb5-3c2f-4374-ac01-ca84a50d31cc.jpeg?im_w=720',
    type: 'Room in serviced apartment in Friedrichshain',
    rating: '4.73'
  },
  HTMLDivElement {
    price: '$82 per night, originally $109',
    image: 'https://a0.muscache.com/im/pictures/miso/Hosting-45475252/original/f6bd7cc6-f72a-43ef-943e-deba27f8253d.jpeg?im_w=720',
    type: 'Entire serviced apartment in Mitte',
    rating: '4.80'
  },
  HTMLDivElement {
    price: '$97 per night, originally $113',
    image: 'https://a0.muscache.com/im/pictures/92966859/7deb381e_original.jpg?im_w=720',
    type: 'Entire apartment in Mitte',
    rating: '4.92'
  },
  HTMLDivElement {
    price: '$99 per night, originally $131',
    image: 'https://a0.muscache.com/im/pictures/f1b953ca-5e8a-4fcd-a224-231e6a92e643.jpg?im_w=720',
    type: 'Entire apartment in Prenzlauer Berg',
    rating: '4.90'
  },
  HTMLDivElement {
    price: '$56 per night, originally $61',
    image: 'https://a0.muscache.com/im/pictures/bb0813a6-e9fe-4f0a-81a8-161440085317.jpg?im_w=720',
    type: 'Entire apartment in Tiergarten',
    rating: '4.67'
  },
...
]

One of the limitations that we are currently facing is that we only scrape the information from one page of our search. This can be fixed by using some type of a headless browser, like Puppeteer, or a browser automation tool like Selenium. This will help us do most of the things we can manually do in a web browser, like completing a form or clicking a button.

We encourage you to check out our Ultimate Guide to Web Scraping with JavaScript and Node.Js if you want to find more about these technologies.

As you can see, we have managed to build a basic web scraper in just a couple of minutes. From here on, your imagination is the limit. If you’re ambitious enough you can even use the data you gather to visualize the distribution and concentration of the properties on a map. Airbnb provides you all the necessary information in the head of any listing page.

All Use Cases articles

News and updates