C and C++ Web Scraping with libcurl and lexbor

Yes, when the scraper must live inside an existing C or C++ codebase, run on constrained hardware, or process large volumes of HTML with little memory. For one-off jobs, Python is quicker to write. C++ has no official Playwright binding, so JavaScript-heavy sites need headless Chrome or a scraping API.

C++ is a good fit for:

• Embedded and edge devices, where a small native binary beats shipping a Python runtime.
• Existing native services, such as trading systems, search indexers, and monitoring agents, where the scraper is one more module instead of a separate process in another language.
• High-volume parsing, where lower CPU and memory use per page turns into real savings on servers.
• Teams that want the same parser in C and C++, because libcurl and lexbor are C libraries with C++ wrappers on top.

C++ is a poor fit for:

• Quick one-off scripts, where a Python notebook gets you to a CSV faster.
• Heavy browser automation with logins, clicks, and waits. Playwright supports JavaScript, Python, Java, and .NET, but not C++.
• Teams without anyone who reviews C++ code regularly.

Manual memory management is the main risk in native scrapers, because every C library hands you raw pointers to free. This guide wraps every C handle in a small RAII class and shows how to check the result with sanitizers, so a missed cleanup shows up in a test run instead of a production leak.

You need a C++20 compiler, CMake, and three libraries: libcurl to download pages, lexbor to parse HTML5 and run CSS selectors, and nlohmann/json to write JSON. libcurl and lexbor are plain C libraries, so the same stack works in C. vcpkg installs all three on Windows, macOS, and Linux.

Older tutorials still recommend libraries that are archived or can't handle real-world HTML.

C and C++ scraping libraries (vcpkg versions, November 2026)

This guide uses libcurl, lexbor, and nlohmann/json. Both core libraries are written in C, are actively maintained, and are packaged in vcpkg and Conan. lexbor has no external dependencies at all, and libcurl already ships with nearly every operating system. You get a browser-grade parser and a battle-tested HTTP client with nothing exotic to install. The code you write for C++ also carries over to plain C almost line for line.

The project uses a vcpkg manifest, a small JSON file that lists the libraries. When CMake configures the project with vcpkg's toolchain file, vcpkg reads the manifest and builds every dependency for your platform. See vcpkg manifest mode for the details. Here is the layout you'll end up with:

bookscraper/
├── CMakeLists.txt
├── vcpkg.json
├── src/
│   ├── http.hpp      # libcurl client: timeouts, retries, Retry-After
│   ├── html.hpp      # lexbor RAII wrapper: parse + CSS selectors
│   ├── books.hpp     # Book struct, listing and detail parsers, URL resolution
│   ├── csv.hpp       # RFC 4180 CSV writer
│   ├── robots.hpp    # robots.txt check
│   └── main.cpp      # crawl, JavaScript, and Scraper API commands
├── tests/parser_test.cpp
└── fixtures/         # saved pages for offline tests

vcpkg.json

{
  "name": "bookscraper",
  "version": "1.0.0",
  "dependencies": ["curl", "lexbor", "nlohmann-json"]
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.21)
project(bookscraper CXX)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(CURL REQUIRED)
find_package(lexbor CONFIG REQUIRED)
find_package(nlohmann_json CONFIG REQUIRED)
# lexbor exports lexbor::lexbor for shared builds and lexbor::lexbor_static for static ones
if(TARGET lexbor::lexbor)
  set(LEXBOR_TARGET lexbor::lexbor)
else()
  set(LEXBOR_TARGET lexbor::lexbor_static)
endif()

add_executable(bookscraper src/main.cpp)
target_link_libraries(bookscraper PRIVATE CURL::libcurl ${LEXBOR_TARGET} nlohmann_json::nlohmann_json)

enable_testing()
add_executable(parser_test tests/parser_test.cpp)
target_include_directories(parser_test PRIVATE src)
target_compile_definitions(parser_test PRIVATE FIXTURES="${CMAKE_CURRENT_SOURCE_DIR}/fixtures")
target_link_libraries(parser_test PRIVATE CURL::libcurl ${LEXBOR_TARGET})
add_test(NAME parser COMMAND parser_test)

The if(TARGET ...) block matters. vcpkg builds lexbor as a static library on Linux by default, and the static build exports a different CMake target name than the shared one. Keeping both lines saves you a confusing link error.

Clone and bootstrap vcpkg once, then configure, build, and test. On Windows, run bootstrap-vcpkg.bat and use backslashes:

git clone https://github.com/microsoft/vcpkg.git
./vcpkg/bootstrap-vcpkg.sh
cmake -B build -S . -DCMAKE_TOOLCHAIN_FILE=./vcpkg/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build

Output (tail)

-- Found nlohmann_json: …/vcpkg_installed/x64-linux/share/nlohmann_json/nlohmann_jsonConfig.cmake (found version "3.12.0")
-- Configuring done (67.9s)
[100%] Built target bookscraper
1/1 Test #1: parser ...........................   Passed    0.00 sec
100% tests passed, 0 tests failed out of 1

The first configure builds the libraries from source, so it takes a minute. Later configures reuse them.

If you'd rather use system packages, Debian and Ubuntu have sudo apt install libcurl4-openssl-dev nlohmann-json3-dev. lexbor isn't packaged in Debian stable, so install it with vcpkg or build it from source. On macOS, brew install curl lexbor nlohmann-json installs all three.

Visual Studio: open the project folder in Visual Studio 2022 or later, which detects CMakeLists.txt automatically. In CMakePresets.json, set CMAKE_TOOLCHAIN_FILE to vcpkg's scripts/buildsystems/vcpkg.cmake, and the manifest installs libcurl, lexbor, and nlohmann/json on the first configure. You don't need to set include paths, library paths, or environment variables by hand.

Download the page with libcurl into a std::string, parse it with lexbor, and select elements with CSS selectors such as article.product_pod. Read each element's text or attributes, then convert prices and relative links into clean values. Wrap the C handles in small RAII classes so every parser and handle is freed automatically.

Start with the parser, because it's the part you'll touch most. lexbor's API is plain C: you create objects, initialize them, use them, and destroy them. The Html class below hides that lifecycle. Its destructor frees everything, even when an exception is thrown halfway through a page.

// html.hpp: a small RAII layer over lexbor's HTML5 parser and CSS selector engine
#pragma once
#include <lexbor/html/html.h>
#include <lexbor/css/css.h>
#include <lexbor/selectors/selectors.h>
#include <stdexcept>
#include <string>
#include <string_view>
#include <unordered_map>
#include <vector>

using Node = lxb_dom_node_t*;

class Html {
public:
    explicit Html(std::string_view html)
        : doc_(lxb_html_document_create()), css_(lxb_css_parser_create()), mem_(lxb_css_memory_create()),
          css_sel_(lxb_css_selectors_create()), sel_(lxb_selectors_create()) {
        if (!doc_ || !css_ || !mem_ || !css_sel_ || !sel_
            || lxb_html_document_parse(doc_, reinterpret_cast<const lxb_char_t*>(html.data()), html.size()) != LXB_STATUS_OK
            || lxb_css_parser_init(css_, nullptr) != LXB_STATUS_OK
            || lxb_css_memory_init(mem_, 128) != LXB_STATUS_OK
            || lxb_css_selectors_init(css_sel_) != LXB_STATUS_OK
            || lxb_selectors_init(sel_) != LXB_STATUS_OK) {
            release();
            throw std::runtime_error("lexbor: could not parse the document");
        }
        // Every compiled selector lives in one memory pool that is freed with the document.
        lxb_css_parser_memory_set(css_, mem_);
        lxb_css_parser_selectors_set(css_, css_sel_);
    }
    ~Html() { release(); }
    Html(const Html&) = delete;
    Html& operator=(const Html&) = delete;

    // All elements under `scope` (the whole document by default) that match a CSS selector.
    std::vector<Node> select(std::string_view css, Node scope = nullptr) const {
        std::vector<Node> found;
        lxb_css_selector_list_t* list = compile(css);
        lxb_selectors_find(sel_, scope ? scope : lxb_dom_interface_node(doc_), list,
            [](Node node, lxb_css_selector_specificity_t, void* out) -> lxb_status_t {
                static_cast<std::vector<Node>*>(out)->push_back(node);
                return LXB_STATUS_OK;
            }, &found);
        return found;
    }

    Node first(std::string_view css, Node scope = nullptr) const {
        auto nodes = select(css, scope);
        return nodes.empty() ? nullptr : nodes.front();
    }

    std::string text(Node node) const {
        if (!node) return {};
        size_t len = 0;
        lxb_char_t* raw = lxb_dom_node_text_content(node, &len);   // allocated by lexbor
        std::string out(reinterpret_cast<char*>(raw), len);
        lxb_dom_document_destroy_text(lxb_dom_interface_document(doc_), raw);
        return out;
    }

    static std::string attr(Node node, std::string_view name) {
        if (!node) return {};
        size_t len = 0;
        const lxb_char_t* value = lxb_dom_element_get_attribute(lxb_dom_interface_element(node),
            reinterpret_cast<const lxb_char_t*>(name.data()), name.size(), &len);
        return value ? std::string(reinterpret_cast<const char*>(value), len) : std::string{};
    }

private:
    // Compile each selector once per document; "h3 a" inside 20 cards is parsed a single time.
    lxb_css_selector_list_t* compile(std::string_view css) const {
        auto it = cache_.find(std::string(css));
        if (it != cache_.end()) return it->second;
        auto* list = lxb_css_selectors_parse(css_, reinterpret_cast<const lxb_char_t*>(css.data()), css.size());
        if (!list || css_->status != LXB_STATUS_OK) throw std::invalid_argument("bad CSS selector: " + std::string(css));
        return cache_.emplace(std::string(css), list).first->second;
    }
    void release() {
        if (sel_) lxb_selectors_destroy(sel_, true);
        if (css_sel_) lxb_css_selectors_destroy(css_sel_, true);
        if (css_) lxb_css_parser_destroy(css_, true);
        if (mem_) lxb_css_memory_destroy(mem_, true);     // frees every compiled selector list
        if (doc_) lxb_html_document_destroy(doc_);
        sel_ = nullptr; css_sel_ = nullptr; css_ = nullptr; mem_ = nullptr; doc_ = nullptr;
    }
    lxb_html_document_t* doc_;
    lxb_css_parser_t* css_;
    lxb_css_memory_t* mem_;
    lxb_css_selectors_t* css_sel_;
    lxb_selectors_t* sel_;
    mutable std::unordered_map<std::string, lxb_css_selector_list_t*> cache_;
};

Three lexbor details are worth knowing before you change this class:

1. The document, the CSS parser, and the selector engine are separate objects. You need all three to run a query, and the class creates and destroys them in one place, so none can outlive the others.
2. Compiled selectors share one memory pool. The class compiles each selector string once and caches it for the life of the document, so h3 a inside 20 product cards is parsed a single time. The pool also explains a classic lexbor crash: if you free one selector's memory and then parse another selector with the same parser, lexbor reads freed memory. One shared pool, released in the destructor, avoids that.
3. Text content is allocated for you. lxb_dom_node_text_content returns a new buffer, and you must give it back with lxb_dom_document_destroy_text. The text() helper copies it into a std::string and releases it immediately.

The lexbor selectors module documentation lists every supported selector, including :has() and :nth-child().

Next, turn the HTML into typed records. Each book card on Books to Scrape holds a title, price, star rating, availability line, and a link to its product page.

// books.hpp: turn Books to Scrape HTML into typed records
#pragma once
#include "html.hpp"
#include <curl/curl.h>
#include <charconv>
#include <memory>
#include <optional>
#include <string>
#include <vector>

struct Book {
    std::string title;
    long price_pence = 0;     // integer money: 51.77 GBP is stored as 5177
    int rating = 0;           // 1 to 5 stars
    bool in_stock = false;
    std::string url;          // absolute product URL
    std::string upc;          // filled from the product page
    int stock_count = 0;      // filled from the product page
};

// Resolve an href against the URL of the page it came from, using libcurl's URL API.
inline std::string resolve_url(const std::string& page_url, const std::string& href) {
    std::unique_ptr<CURLU, decltype(&curl_url_cleanup)> u(curl_url(), &curl_url_cleanup);
    if (!u || curl_url_set(u.get(), CURLUPART_URL, page_url.c_str(), 0) != CURLUE_OK
           || curl_url_set(u.get(), CURLUPART_URL, href.c_str(), 0) != CURLUE_OK)  // relative to the page
        return {};
    char* full = nullptr;
    if (curl_url_get(u.get(), CURLUPART_URL, &full, 0) != CURLUE_OK) return {};
    std::string out(full);
    curl_free(full);
    return out;
}

// "£51.77" -> 5177. Locale-independent: std::from_chars ignores the user's decimal separator.
inline std::optional<long> parse_pence(std::string_view text) {
    const auto start = text.find_first_of("0123456789");
    if (start == std::string_view::npos) return std::nullopt;
    text.remove_prefix(start);
    long whole = 0, frac = 0;
    auto [p, ec] = std::from_chars(text.data(), text.data() + text.size(), whole);
    if (ec != std::errc{}) return std::nullopt;
    if (p != text.data() + text.size() && *p == '.') {
        const char* f = p + 1;
        auto [q, ec2] = std::from_chars(f, text.data() + text.size(), frac);
        if (ec2 != std::errc{} || q - f != 2) return std::nullopt;
    }
    return whole * 100 + frac;
}

// <p class="star-rating Three"> -> 3
inline int parse_rating(const std::string& css_class) {
    static constexpr const char* words[] = {"One", "Two", "Three", "Four", "Five"};
    for (int i = 0; i < 5; ++i)
        if (css_class.find(words[i]) != std::string::npos) return i + 1;
    return 0;
}

struct ListingPage {
    std::vector<Book> books;
    std::string next_url;     // empty on the last page
};

inline ListingPage parse_listing(std::string_view html, const std::string& page_url) {
    Html doc(html);
    ListingPage page;
    for (Node card : doc.select("article.product_pod")) {
        Node link = doc.first("h3 a", card);
        Book b;
        b.title = Html::attr(link, "title");                   // the visible text is truncated with "..."
        b.url = resolve_url(page_url, Html::attr(link, "href"));
        b.price_pence = parse_pence(doc.text(doc.first(".price_color", card))).value_or(-1);
        b.rating = parse_rating(Html::attr(doc.first("p.star-rating", card), "class"));
        b.in_stock = doc.text(doc.first(".availability", card)).find("In stock") != std::string::npos;
        page.books.push_back(std::move(b));
    }
    if (Node next = doc.first("li.next a")) page.next_url = resolve_url(page_url, Html::attr(next, "href"));
    return page;
}

// Product page: UPC and "In stock (22 available)" from the information table.
inline void parse_detail(std::string_view html, Book& b) {
    Html doc(html);
    for (Node row : doc.select("table.table-striped tr")) {
        const std::string key = doc.text(doc.first("th", row));
        const std::string value = doc.text(doc.first("td", row));
        if (key == "UPC") b.upc = value;
        if (key == "Availability") {
            const auto open = value.find('(');
            if (open != std::string::npos) std::from_chars(value.data() + open + 1, value.data() + value.size(), b.stock_count);
        }
    }
}

Prices stay as integer pence until the moment you print them. Binary floating point can't represent 51.77 exactly, so summing or comparing double prices drifts by fractions of a penny. long values don't. The title comes from the link's title attribute instead of its text, because the visible text is cut short with "..." on long titles. parse_detail reads the product page's information table later, when the crawler visits each book.

main.cpp ties the headers together. It starts with the includes, an honest User-Agent, and a small file helper, then dispatches on a command name:

// main.cpp: Books to Scrape crawler (libcurl + lexbor), plus the JavaScript and Scraper API paths
#include "books.hpp"
#include "csv.hpp"
#include "http.hpp"
#include "robots.hpp"
#include <nlohmann/json.hpp>
#include <atomic>
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <fstream>
#include <iostream>
#include <sstream>
#include <thread>

static const std::string kUserAgent = "bookscraper/1.0 (+https://example.com/contact)";

static std::string read_file(const std::string& path) {
    std::ifstream in(path, std::ios::binary);
    if (!in) throw std::runtime_error("cannot open " + path);
    std::ostringstream ss; ss << in.rdbuf(); return ss.str();
}

int main(int argc, char** argv) try {
    CurlGlobal curl;
    const std::string cmd = argc > 1 ? argv[1] : "crawl";
    // if (cmd == "fixture") { … } else if (cmd == "crawl") { … } … : the commands in this guide
    return 0;
} catch (const std::exception& e) {
    std::cerr << "error: " << e.what() << '\n';
    return 1;
}

Run the parser against a saved copy of the page first. If something breaks here, you know it's a parsing problem and not a network one. This is the fixture command:

    if (cmd == "fixture") {                       // offline: parse saved HTML
        auto page = parse_listing(read_file(argv[2]), "https://books.toscrape.com/catalogue/page-1.html");
        for (size_t i = 0; i < 3 && i < page.books.size(); ++i) {
            const Book& b = page.books[i];
            std::cout << b.title << " | " << format_price(b.price_pence) << " | " << b.rating << " stars | " << b.url << '\n';
        }
        std::cout << page.books.size() << " books, next: " << page.next_url << '\n';
    }

curl -s https://books.toscrape.com/catalogue/page-1.html -o page1.html
./build/bookscraper fixture page1.html

Output

A Light in the Attic | 51.77 | 3 stars | https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet | 53.74 | 1 stars | https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
Soumission | 50.10 | 1 stars | https://books.toscrape.com/catalogue/soumission_998/index.html
20 books, next: https://books.toscrape.com/catalogue/page-2.html

lexbor follows the WHATWG HTML parsing rules that browsers use, has a built-in CSS selector engine, and is actively maintained. libxml2's HTML parser predates HTML5, so its tree can differ from what DevTools shows. Google archived its Gumbo repository. A community fork continues, but neither version has a selector engine.

The difference shows up on messy markup, which real sites produce all the time. Here is one misnested input, <table><b>Sale</b><tr><td>9.99</td></tr></table>, parsed by lexbor, by libxml2 2.9 (the version most Linux distributions ship), and by Chrome:

Output

lexbor : <body><b>Sale</b><table><tbody><tr><td>9.99</td></tr></tbody></table></body>
libxml2: <body><table>
<b>Sale</b><tr><td>9.99</td></tr>
</table></body>
chrome  : <html><head></head><body><b>Sale</b><table><tbody><tr><td>9.99</td></tr></tbody></table></body></html>

lexbor and Chrome build the same tree: the <b> element moves in front of the table, and a <tbody> appears around the row. libxml2 keeps <b> inside the table and adds no <tbody>. So a selector you copy from DevTools, such as table > tbody > tr, matches on lexbor and silently returns nothing on that libxml2 tree. The rules lexbor follows are in the WHATWG HTML parsing specification. Newer libxml2 releases tokenize more like HTML5, but they still don't build the full HTML5 tree.

If your project already links libxml2, or you think in XPath, you can parse the same listing page with it. Wrap each libxml2 object in a std::unique_ptr with its matching free function, pass HTML_PARSE_NONET so the parser never fetches anything from the network, and release every string libxml2 gives you with xmlFree, not free(). If you're deciding between the two query languages, XPath vs CSS selectors compares them side by side.

// libxml2 alternative: the same listing page through libxml2's HTML parser and XPath.
#include <libxml/HTMLparser.h>
#include <libxml/xpath.h>
#include <fstream>
#include <iostream>
#include <memory>
#include <sstream>

int main(int argc, char** argv) {
    std::ifstream in(argc > 1 ? argv[1] : "fixtures/books-page1.html", std::ios::binary);
    std::ostringstream ss; ss << in.rdbuf(); const std::string html = ss.str();

    std::unique_ptr<xmlDoc, decltype(&xmlFreeDoc)> doc(
        htmlReadMemory(html.data(), static_cast<int>(html.size()), "https://books.toscrape.com/catalogue/page-1.html",
                       "UTF-8", HTML_PARSE_NOERROR | HTML_PARSE_NOWARNING | HTML_PARSE_NONET),
        &xmlFreeDoc);
    if (!doc) { std::cerr << "parse failed\n"; return 1; }
    std::unique_ptr<xmlXPathContext, decltype(&xmlXPathFreeContext)> ctx(xmlXPathNewContext(doc.get()), &xmlXPathFreeContext);
    std::unique_ptr<xmlXPathObject, decltype(&xmlXPathFreeObject)> res(
        xmlXPathEvalExpression(BAD_CAST "//article[contains(@class,'product_pod')]//h3/a/@title", ctx.get()),
        &xmlXPathFreeObject);
    if (!res || !res->nodesetval) return 1;
    for (int i = 0; i < res->nodesetval->nodeNr && i < 3; ++i) {
        xmlChar* title = xmlNodeGetContent(res->nodesetval->nodeTab[i]);
        std::cout << reinterpret_cast<char*>(title) << '\n';
        xmlFree(title);                         // libxml2 memory goes back through xmlFree, not free()
    }
    std::cout << res->nodesetval->nodeNr << " titles\n";
}

g++ -std=c++20 libxml2_xpath.cpp $(pkg-config --cflags --libs libxml-2.0) -o libxml2_xpath
./libxml2_xpath page1.html

Output

A Light in the Attic
Tipping the Velvet
Soumission
20 titles

Open the page in your browser, right-click the data, and choose Inspect. Pick short selectors built on stable classes or attributes, such as article.product_pod h3 a, and test them in the DevTools console with document.querySelectorAll. Then check the raw HTML too, because DevTools shows the parsed DOM, including tbody elements that browsers insert.

On Books to Scrape, every book sits in an <article class="product_pod">. Inside it, the selectors you need are short and tied to class names that describe the content, which makes them less likely to break than long paths copied with "Copy selector":

Test each selector in the console before you write C++: document.querySelectorAll("article.product_pod").length should return 20. Then press Ctrl+U to view the source and confirm the same elements exist in the HTML the server sends, not just in the DOM after scripts run. That one check catches JavaScript-built pages early. Our CSS selectors cheat sheet covers attribute, pseudo-class, and combinator selectors when the simple ones aren't enough.

Create one easy handle per thread and reuse it so connections stay open. Set a descriptive User-Agent, connect and total timeouts, a redirect limit, and CURLOPT_ACCEPT_ENCODING to an empty string for compression. Check the HTTP status yourself, retry only 408, 429, 5xx, and network errors, and honor the Retry-After header.

The HttpClient class wraps one libcurl easy handle with those defaults. CurlGlobal runs curl_global_init once, in main, before any thread starts, because that call isn't thread-safe.

// http.hpp: one reusable libcurl handle with timeouts, an honest User-Agent, and polite retries
#pragma once
#include <curl/curl.h>
#include <algorithm>
#include <chrono>
#include <stdexcept>
#include <string>
#include <thread>

// Call curl_global_init() once, in main(), before any thread starts.
struct CurlGlobal {
    CurlGlobal() {
        if (curl_global_init(CURL_GLOBAL_DEFAULT) != CURLE_OK)
            throw std::runtime_error("curl_global_init failed");
    }
    ~CurlGlobal() { curl_global_cleanup(); }
    CurlGlobal(const CurlGlobal&) = delete;
    CurlGlobal& operator=(const CurlGlobal&) = delete;
};

struct Response {
    long status = 0;
    std::string body;
};

class HttpClient {
public:
    explicit HttpClient(const std::string& user_agent, long timeout_seconds = 30)
        : curl_(curl_easy_init()) {
        if (!curl_) throw std::runtime_error("curl_easy_init failed");
        curl_easy_setopt(curl_, CURLOPT_USERAGENT, user_agent.c_str());  // libcurl copies the string
        curl_easy_setopt(curl_, CURLOPT_FOLLOWLOCATION, 1L);
        curl_easy_setopt(curl_, CURLOPT_MAXREDIRS, 5L);
        curl_easy_setopt(curl_, CURLOPT_CONNECTTIMEOUT, 10L);
        curl_easy_setopt(curl_, CURLOPT_TIMEOUT, timeout_seconds);
        curl_easy_setopt(curl_, CURLOPT_ACCEPT_ENCODING, "");  // accept every compression this libcurl supports
        curl_easy_setopt(curl_, CURLOPT_WRITEFUNCTION, &HttpClient::append);
    }
    ~HttpClient() { curl_easy_cleanup(curl_); }
    HttpClient(const HttpClient&) = delete;             // one handle per thread; never share it
    HttpClient& operator=(const HttpClient&) = delete;

    // GET a URL. Retries network errors, 408, 429, and 5xx; waits for Retry-After when the server sends it.
    Response get(const std::string& url, int max_retries = 3) {
        for (int attempt = 0;; ++attempt) {
            Response r;
            curl_easy_setopt(curl_, CURLOPT_URL, url.c_str());
            curl_easy_setopt(curl_, CURLOPT_WRITEDATA, &r.body);
            const CURLcode rc = curl_easy_perform(curl_);
            if (rc == CURLE_OK) curl_easy_getinfo(curl_, CURLINFO_RESPONSE_CODE, &r.status);

            const bool retryable = rc != CURLE_OK || r.status == 408 || r.status == 429 || r.status >= 500;
            if (!retryable || attempt == max_retries) {
                if (rc != CURLE_OK)
                    throw std::runtime_error(std::string("GET ") + url + " failed: " + curl_easy_strerror(rc));
                return r;
            }

            curl_off_t retry_after = 0;                   // seconds, parsed by libcurl; 0 when absent
            curl_easy_getinfo(curl_, CURLINFO_RETRY_AFTER, &retry_after);
            const auto backoff = std::chrono::seconds(1LL << attempt);   // 1 s, 2 s, 4 s
            const auto wait = retry_after > 0 ? std::chrono::seconds(std::min<curl_off_t>(retry_after, 120)) : backoff;
            std::this_thread::sleep_for(wait);
        }
    }

private:
    static size_t append(char* data, size_t size, size_t count, void* out) {
        static_cast<std::string*>(out)->append(data, size * count);
        return size * count;
    }
    CURL* curl_;
};

Four choices in this class matter:

• No CURLOPT_FAILONERROR. That option turns HTTP errors into transfer errors and throws away the body. The scraper wants both the status code and the body, so it checks the status itself.
• Retries only where they can help. A 404 won't fix itself, so the client retries network errors, 408, 429, and 5xx responses, up to three times.
• CURLINFO_RETRY_AFTER does the parsing. libcurl reads the Retry-After header in either form (seconds or an HTTP date) and hands you seconds. The client caps the wait at 120 seconds, so a hostile or broken header can't stall the crawl for hours. Without the header, it backs off for 1, 2, then 4 seconds.
• The handle is reused. Each get call keeps the same connection open when the server allows it, which is faster for you and lighter for the site.

To see the retry logic work, point the client at a tiny local server that answers 429 with Retry-After: 2 twice, then 200:

# Local test server: the first two requests get 429 + Retry-After: 2, then 200.
import http.server, sys
hits = 0
class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        global hits; hits += 1
        if hits <= 2:
            self.send_response(429); self.send_header("Retry-After", "2"); self.end_headers(); self.wfile.write(b"slow down")
        else:
            self.send_response(200); self.end_headers(); self.wfile.write(b"ok after %d tries" % hits)
    def log_message(self, fmt, *a): sys.stderr.write("server: " + fmt % a + "\n")
http.server.HTTPServer(("127.0.0.1", 8429), H).serve_forever()

#include "../src/http.hpp"
#include <iostream>
int main() {
    CurlGlobal g;
    HttpClient http("bookscraper/1.0");
    auto t0 = std::chrono::steady_clock::now();
    Response r = http.get("http://127.0.0.1:8429/");
    auto s = std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
    std::cout << "status " << r.status << ", body \"" << r.body << "\", waited " << s << " s\n";
}

Output (server log lines, then the program)

server: "GET / HTTP/1.1" 429 -
server: "GET / HTTP/1.1" 429 -
server: "GET / HTTP/1.1" 200 -
status 200, body "ok after 3 tries", waited 4.00422 s

The client waited two seconds twice, exactly as the server asked, and succeeded on the third try.

User-Agent: identify your scraper and give site owners a way to reach you, such as bookscraper/1.0 (+https://example.com/contact). Pretending to be a desktop browser makes your traffic harder to tell apart from abuse and doesn't make it more welcome. Our guide to HTTP headers for web scraping covers the other headers worth setting.

Proxies: add CURLOPT_PROXY (for example socks5h://host:1080) and CURLOPT_PROXYUSERPWD read from environment variables to route through a proxy. The option maps to curl's -x flag, so the same settings from using curl with a proxy carry over.

cpr: if you prefer a requests-style C++ API, cpr wraps libcurl, and the rest of this guide works the same.

Parse the next-page link on each listing page, resolve it against the current URL, and loop until no link remains or a page limit is hit. libcurl's URL API resolves relative links correctly. Pause between requests, and fetch detail pages with a small fixed pool of worker threads, each owning its own libcurl handle.

The crawl loop is short because the parser already returns the next URL:

// 1. Follow "next" links until the last page or a page limit, pausing between requests.
static std::vector<Book> crawl(HttpClient& http, std::string url, int max_pages) {
    std::vector<Book> all;
    for (int page = 1; !url.empty() && page <= max_pages; ++page) {
        Response r = http.get(url);
        if (r.status != 200) { std::cerr << "HTTP " << r.status << " for " << url << '\n'; break; }
        ListingPage lp = parse_listing(r.body, url);
        std::cerr << "page " << page << ": " << lp.books.size() << " books\n";
        all.insert(all.end(), lp.books.begin(), lp.books.end());
        url = lp.next_url;
        std::this_thread::sleep_for(std::chrono::milliseconds(500));
    }
    return all;
}

The page limit protects you from loops and from accidentally crawling a 1,000-page catalog when you meant to test two pages. The half-second pause keeps the request rate modest.

The tricky part is resolve_url in books.hpp. Books to Scrape links are relative: a card on /catalogue/page-1.html links to a-light-in-the-attic_1000/index.html, and the same href on the home page means something else, because the home page lives one directory up. Gluing strings together breaks on cases like that and on ../ paths. libcurl's URL API applies the standard resolution rules instead: set the page URL as the base, set the href on top of it, and read back the absolute URL. The offline tests later in this guide check both a plain relative path and a ../ link.

The crawl command checks robots.txt (explained in the responsible scraping section), crawls the listing pages, visits each product page, and writes both export files:

    if (cmd == "crawl") {                  // live: listing pages, product pages, CSV + JSON
        const int max_pages = argc > 2 ? std::atoi(argv[2]) : 2;
        HttpClient http(kUserAgent);
        Response robots_txt = http.get("https://books.toscrape.com/robots.txt");
        Robots robots(robots_txt.status == 200 ? robots_txt.body : "", kUserAgent);
        std::cerr << "robots.txt: HTTP " << robots_txt.status << ", /catalogue/ allowed: "
                  << std::boolalpha << robots.allowed("/catalogue/page-1.html") << '\n';
        if (!robots.allowed("/catalogue/page-1.html")) return 1;
        auto books = crawl(http, "https://books.toscrape.com/catalogue/page-1.html", max_pages);
        enrich(books, 4);
        write_csv("books.csv", books);
        write_json("books.json", books);
        std::cout << "saved " << books.size() << " books to books.csv and books.json\n";
    }

cd build && ./bookscraper crawl 2

Output (about 7 seconds for 2 listing pages, 40 product pages, and the pauses)

robots.txt: HTTP 404, /catalogue/ allowed: true
page 1: 20 books
page 2: 20 books
saved 40 books to books.csv and books.json

When the crawl itself becomes the project, with thousands of pages, link discovery, and scheduling, WebScrapingAPI's Crawl API collects related pages through one bounded crawl.

Start a fixed number of std::jthread workers, around four, and let each claim the next index from a std::atomic counter. Give every worker its own libcurl easy handle, because one handle must never be used by two threads at once. Write each result into its own slot, so no mutex is needed.

Product pages are independent of each other, so they're a natural fit for a small pool:

// 2. Visit product pages with a small, fixed number of workers. Each worker owns its own
//    libcurl handle, and each result goes into its own slot, so no mutex is needed.
static void enrich(std::vector<Book>& books, unsigned workers) {
    std::atomic<size_t> next{0};
    std::vector<std::jthread> pool;
    for (unsigned w = 0; w < workers; ++w)
        pool.emplace_back([&] {
            HttpClient http(kUserAgent);
            for (size_t i = next++; i < books.size(); i = next++) {
                try {
                    Response r = http.get(books[i].url);
                    if (r.status == 200) parse_detail(r.body, books[i]);
                } catch (const std::exception& e) { std::cerr << e.what() << '\n'; }
            }
        });
}   // jthread joins on destruction

Each worker creates its own HttpClient inside the thread, then loops: take the next index, fetch that book's page, and fill in upc and stock_count on that one Book. Because no two workers ever touch the same element, and the vector never changes size during the loop, there's nothing to lock. std::jthread joins automatically when it's destroyed, so the function can't return while a worker is still writing. There are no join calls to forget.

Four workers is a polite default for a single site. More threads mostly add load on the target. If you ever need hundreds of transfers in parallel, look at libcurl's multi interface, which drives many transfers from one thread, instead of adding threads.

Store prices as integer cents or pence, then write CSV that follows RFC 4180: quote fields that contain commas, quotes, or line breaks, double any inner quotes, and end rows with CRLF. For JSON, nlohmann/json's ordered_json keeps fields in the order you add them. Both formats open cleanly in spreadsheets and data tools.

Book titles break naive CSV quickly. Books to Scrape has titles with commas and parentheses on the first pages, and joining fields with plain commas would shift every column after them. The writer below follows RFC 4180:

// csv.hpp: RFC 4180 output, so titles with commas or quotes stay in one column
#pragma once
#include "books.hpp"
#include <cstdio>
#include <fstream>
#include <string>
#include <vector>

inline std::string csv_field(const std::string& s) {
    if (s.find_first_of(",\"\r\n") == std::string::npos) return s;
    std::string out = "\"";
    for (char c : s) {
        if (c == '"') out += '"';   // a quote inside a field is written twice
        out += c;
    }
    return out + '"';
}

inline std::string format_price(long pence) {
    char buf[32];
    std::snprintf(buf, sizeof buf, "%ld.%02ld", pence / 100, pence % 100);
    return buf;
}

inline void write_csv(const std::string& path, const std::vector<Book>& books) {
    std::ofstream out(path, std::ios::binary);       // binary: "\r\n" stays exactly as written on Windows
    out << "title,price_gbp,rating,in_stock,stock_count,upc,url\r\n";
    for (const Book& b : books)
        out << csv_field(b.title) << ',' << format_price(b.price_pence) << ',' << b.rating << ','
            << (b.in_stock ? "true" : "false") << ',' << b.stock_count << ',' << csv_field(b.upc) << ','
            << csv_field(b.url) << "\r\n";
    if (!out) throw std::runtime_error("could not write " + path);
}

The file opens in binary mode on purpose. In text mode, Windows turns every \n into \r\n, so the \r\n row endings would come out as \r\r\n. Binary mode writes exactly the bytes you asked for on every platform.

The JSON writer uses ordered_json, so the fields come out in the same order as the CSV columns instead of alphabetically:

static void write_json(const std::string& path, const std::vector<Book>& books) {
    nlohmann::ordered_json arr = nlohmann::ordered_json::array();   // keeps field order
    for (const Book& b : books)
        arr.push_back({{"title", b.title}, {"price_gbp", b.price_pence / 100.0}, {"rating", b.rating},
                       {"in_stock", b.in_stock}, {"stock_count", b.stock_count}, {"upc", b.upc}, {"url", b.url}});
    std::ofstream(path) << arr.dump(2) << '\n';
}

Output: books.csv (first rows, plus the row that needs quoting)

title,price_gbp,rating,in_stock,stock_count,upc,url
A Light in the Attic,51.77,3,true,22,a897fe39b1053632,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet,53.74,1,true,20,90fa61229261140a,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
…
"Starving Hearts (Triangular Trade Trilogy, #1)",13.99,2,true,19,0312262ecafa5a40,https://books.toscrape.com/catalogue/starving-hearts-triangular-trade-trilogy-1_990/index.html

Output: books.json (first record)

{
  "title": "A Light in the Attic",
  "price_gbp": 51.77,
  "rating": 3,
  "in_stock": true,
  "stock_count": 22,
  "upc": "a897fe39b1053632",
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
}

The Starving Hearts title stays in one column because the comma sits inside quotes. Prices were parsed with std::from_chars, which ignores the user's locale, so a system set to use commas as decimal separators can't change the result. The price is parsed as two integers, pounds and pence, because floating-point from_chars support still differs across standard libraries.

Use the same two libraries without the C++ wrappers: libcurl writes the response into a growable buffer through a write callback, and lexbor parses the buffer and runs CSS selectors. Free every handle on every path. A single cleanup label with goto keeps that manageable. The complete C program is under 80 lines.

Everything above also works in C17. The program below fetches the first Books to Scrape page and prints every title:

/* scrape.c: plain C17. Fetch one Books to Scrape page with libcurl, print titles and prices with lexbor. */
#include <curl/curl.h>
#include <lexbor/html/html.h>
#include <lexbor/css/css.h>
#include <lexbor/selectors/selectors.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct { char *data; size_t len; } buffer;

static size_t on_data(char *ptr, size_t size, size_t n, void *userdata) {
    buffer *buf = userdata;
    size_t add = size * n;
    char *grown = realloc(buf->data, buf->len + add + 1);
    if (!grown) return 0;                       /* returning less than add makes libcurl abort the transfer */
    buf->data = grown;
    memcpy(buf->data + buf->len, ptr, add);
    buf->len += add;
    buf->data[buf->len] = '\0';
    return add;
}

static lxb_status_t on_match(lxb_dom_node_t *node, lxb_css_selector_specificity_t spec, void *ctx) {
    (void) spec; (void) ctx;
    size_t len = 0;
    const lxb_char_t *title = lxb_dom_element_get_attribute(lxb_dom_interface_element(node),
                                                            (const lxb_char_t *) "title", 5, &len);
    if (title) printf("%.*s\n", (int) len, (const char *) title);
    return LXB_STATUS_OK;
}

int main(void) {
    const char *url = "https://books.toscrape.com/catalogue/page-1.html";
    static const char selector[] = "article.product_pod h3 a";
    buffer page = {0};
    long status = 0;
    int rc = EXIT_FAILURE;

    if (curl_global_init(CURL_GLOBAL_DEFAULT) != CURLE_OK) return EXIT_FAILURE;
    CURL *curl = curl_easy_init();
    if (!curl) goto cleanup_global;
    curl_easy_setopt(curl, CURLOPT_URL, url);
    curl_easy_setopt(curl, CURLOPT_USERAGENT, "bookscraper-c/1.0 (+https://example.com/contact)");
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
    curl_easy_setopt(curl, CURLOPT_TIMEOUT, 30L);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, on_data);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, &page);

    CURLcode res = curl_easy_perform(curl);
    if (res != CURLE_OK) { fprintf(stderr, "request failed: %s\n", curl_easy_strerror(res)); goto cleanup_curl; }
    curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &status);
    if (status != 200) { fprintf(stderr, "HTTP %ld\n", status); goto cleanup_curl; }

    lxb_html_document_t *doc = lxb_html_document_create();
    lxb_css_parser_t *parser = lxb_css_parser_create();
    lxb_selectors_t *sel = lxb_selectors_create();
    lxb_css_selector_list_t *list = NULL;
    if (doc && parser && sel
        && lxb_html_document_parse(doc, (const lxb_char_t *) page.data, page.len) == LXB_STATUS_OK
        && lxb_css_parser_init(parser, NULL) == LXB_STATUS_OK
        && lxb_selectors_init(sel) == LXB_STATUS_OK
        && (list = lxb_css_selectors_parse(parser, (const lxb_char_t *) selector, sizeof selector - 1)) != NULL) {
        lxb_selectors_find(sel, lxb_dom_interface_node(doc), list, on_match, NULL);
        rc = EXIT_SUCCESS;
    }
    if (list) lxb_css_selector_list_destroy_memory(list);
    if (sel) lxb_selectors_destroy(sel, true);
    if (parser) lxb_css_parser_destroy(parser, true);
    if (doc) lxb_html_document_destroy(doc);
cleanup_curl:
    curl_easy_cleanup(curl);
cleanup_global:
    curl_global_cleanup();
    free(page.data);
    return rc;
}

gcc -std=c17 -Wall -Wextra scrape.c $(pkg-config --cflags --libs libcurl lexbor) -o scrape
./scrape

Output (first 3 of 20 lines)

A Light in the Attic
Tipping the Velvet
Soumission
…

If you installed the libraries with vcpkg, point PKG_CONFIG_PATH at vcpkg_installed/<triplet>/lib/pkgconfig first, because vcpkg ships libcurl.pc and lexbor.pc files there.

Two traps catch most C scrapers:

• realloc can fail. Assign its result to a temporary pointer and check it before overwriting buf->data. Writing buf->data = realloc(buf->data, …) leaks the old buffer when memory runs out.
• The write callback must return the number of bytes it handled. Any other value tells libcurl something went wrong, and it aborts the transfer. That's also how the callback reports an allocation failure: it returns 0.

The goto labels give the function one exit path per resource, so libcurl and lexbor objects are released in reverse order no matter where an error happens.

Fetch the page with libcurl and count the elements you expect. If the count is zero but your browser shows data, the page builds its content with JavaScript. Before reaching for a browser, search the raw HTML's script tags and the network tab. The data is often already there as JSON you can parse directly.

The /js/ version of Quotes to Scrape looks identical to the normal one in a browser. Fetch it with libcurl, though, and the quote elements aren't there. The quotes live in a JavaScript array inside a <script> tag, and the page's own script builds the HTML from it after load.

That means you can often skip the browser entirely. This function finds the script that defines the array and hands the array to nlohmann/json:

// 3. JavaScript pages, option A: the data is often already in the HTML, inside a <script>.
static nlohmann::json quotes_from_script(const std::string& html) {
    Html doc(html);
    for (Node script : doc.select("script")) {
        const std::string js = doc.text(script);
        const auto start = js.find("var data = [");
        if (start == std::string::npos) continue;
        const auto open = js.find('[', start);
        const auto close = js.find("];", open);   // first "];" after the array start ends the literal
        return nlohmann::json::parse(js.substr(open, close - open + 1));
    }
    return nlohmann::json::array();
}

    if (cmd == "js") {                     // live: plain fetch of a JavaScript page
        HttpClient http(kUserAgent);
        const std::string html = http.get("https://quotes.toscrape.com/js/").body;
        Html doc(html);
        std::cout << "div.quote elements in raw HTML: " << doc.select("div.quote").size() << '\n';
        auto quotes = quotes_from_script(html);
        std::cout << "quotes found in <script> JSON: " << quotes.size() << '\n'
                  << "first: " << quotes[0]["text"].get<std::string>() << " - "
                  << quotes[0]["author"]["name"].get<std::string>() << '\n';
    }

Output

div.quote elements in raw HTML: 0
quotes found in <script> JSON: 10
first: “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” - Albert Einstein

Zero elements in the HTML, ten quotes in the script. The extraction looks for var data = [, then takes everything up to the first ]; after it. That's a pattern for this page, not a general JavaScript parser. On other sites, look for application/json or application/ld+json script tags, or open the network tab and call the JSON endpoint the page itself uses.

Run headless Chrome with --dump-dom to execute the page's scripts and print the final HTML, then parse that output with lexbor. For clicks, logins, or waits, drive Chrome through WebDriver or the DevTools Protocol over HTTP. Playwright has no official C++ binding. At scale, a rendering API removes the browser fleet entirely.

When the data isn't sitting in a script tag, let a real browser build the page. A headless browser runs without a window, and Chrome's --dump-dom flag prints the rendered DOM to standard output, which C++ can read through a pipe:

// 3. JavaScript pages, option B: let headless Chrome run the scripts, then parse the DOM it prints.
//    POSIX popen; the URL is a constant here. Never paste untrusted input into a shell command.
static std::string render_with_chrome(const std::string& url) {
    const char* chrome = std::getenv("CHROME");
    const std::string cmd = std::string(chrome ? chrome : "google-chrome") +
        " --headless --disable-gpu --dump-dom '" + url + "' 2>/dev/null";
    std::unique_ptr<FILE, int (*)(FILE*)> pipe(popen(cmd.c_str(), "r"), pclose);
    if (!pipe) throw std::runtime_error("could not start Chrome");
    std::string dom; char buf[8192];
    for (size_t n; (n = std::fread(buf, 1, sizeof buf, pipe.get())) > 0;) dom.append(buf, n);
    return dom;
}

    if (cmd == "render") {                 // live: headless Chrome, then lexbor
        Html doc(render_with_chrome("https://quotes.toscrape.com/js/"));
        auto quotes = doc.select("div.quote");
        std::cout << "div.quote elements after rendering: " << quotes.size() << '\n';
        if (!quotes.empty())
            std::cout << "first author: " << doc.text(doc.first("small.author", quotes[0])) << '\n';
    }

Output (a few seconds, including Chrome startup)

div.quote elements after rendering: 10
first author: Albert Einstein

The same Html class and the same CSS selectors work on Chrome's output, because lexbor and Chrome parse HTML the same way. Chrome's headless mode documentation covers the other flags.

Keep three limits in mind:

• Never build the command from untrusted input. The URL here is a constant. A URL taken from a page or a user could contain shell syntax, and popen would run it. On Windows, use _popen and _pclose with double quotes around the URL.
• --dump-dom prints the DOM after the load event. It doesn't wait for data that arrives later through extra network calls. When a page loads content after that, or you need clicks, logins, or scrolling, drive Chrome through WebDriver or the Chrome DevTools Protocol, both of which speak HTTP or WebSocket that libcurl can reach.
• Set the CHROME environment variable to the browser's path on macOS and Windows, where google-chrome isn't on the PATH.

For many browser sessions with logins and interactions, WebScrapingAPI's Browser API runs the browsers for you.

Send a GET request with libcurl to WebScrapingAPI's Scraper API endpoint, passing your API key, the URL-encoded target URL, and options such as render_js=1 and country=us. The response body is the page's HTML, so your existing lexbor parser works unchanged. Read the key from the WSA_API_KEY environment variable.

Running Chrome yourself is fine for a few pages. When you need JavaScript rendering, a specific country, or access to sites that block scrapers across thousands of requests, a scraping API takes over the browsers and proxies. Because it returns HTML, the change on your side is one URL:

// 4. Hand-off: the same libcurl client, pointed at WebScrapingAPI's Scraper API.
static std::string scraper_api_url(const std::string& api_key, const std::string& target,
                                   bool render_js, const std::string& country) {
    std::unique_ptr<CURL, decltype(&curl_easy_cleanup)> c(curl_easy_init(), &curl_easy_cleanup);
    auto escape = [&](const std::string& s) {
        char* e = curl_easy_escape(c.get(), s.c_str(), static_cast<int>(s.size()));
        std::string out(e); curl_free(e); return out;
    };
    std::string url = "https://api.webscrapingapi.com/v2?api_key=" + escape(api_key) + "&url=" + escape(target);
    if (render_js) url += "&render_js=1";
    if (!country.empty()) url += "&country=" + escape(country);
    return url;
}

    if (cmd == "api") {                    // Scraper API: needs WSA_API_KEY; prints a dry run without it
        const char* key = std::getenv("WSA_API_KEY");
        const std::string url = scraper_api_url(key ? key : "YOUR_API_KEY",
                                                "https://quotes.toscrape.com/js/", true, "us");
        if (!key) { std::cout << "dry run (set WSA_API_KEY to send it):\n" << url << '\n'; return 0; }
        HttpClient http(kUserAgent, 90);          // rendered requests take longer
        Response r = http.get(url, 1);
        if (r.status != 200) { std::cerr << "Scraper API returned HTTP " << r.status << ": " << r.body << '\n'; return 1; }
        Html doc(r.body);
        std::cout << "div.quote elements via Scraper API: " << doc.select("div.quote").size() << '\n';
    }

curl_easy_escape encodes the target URL, so its own ? and & characters can't be mistaken for the API's parameters. The client gets a 90-second timeout because rendering a page takes longer than fetching it, and only one retry, so a wrong key fails fast. Keep the key out of source code and shell history files you share:

export WSA_API_KEY="your-key-from-the-dashboard"   # PowerShell: $env:WSA_API_KEY = "..."
./bookscraper api

Without the variable, the command prints the request it would send:

Output without WSA_API_KEY

dry run (set WSA_API_KEY to send it):
https://api.webscrapingapi.com/v2?api_key=YOUR_API_KEY&url=https%3A%2F%2Fquotes.toscrape.com%2Fjs%2F&render_js=1&country=us

With a wrong key, the API answers 401 with a JSON error, and the program prints it instead of trying to parse it:

Output with an invalid key

Scraper API returned HTTP 401: {"error":"Invalid API key"}

Sample output with a valid key (simulated for illustration)

div.quote elements via Scraper API: 10

The rendered page should hold the same ten quotes Chrome produced, and the same selector should find them. WebScrapingAPI also reports the credits each request used in response headers prefixed with WSA, which you can read with CURLOPT_HEADERFUNCTION if you want to track usage from C++.

Use libcurl and lexbor for static pages, and look in script tags before adding a browser. Use headless Chrome when the page needs JavaScript and the volume is small. Switch to a scraping API when you need JavaScript rendering, geotargeting, or proxy rotation at scale without running browsers yourself.

Save real pages as fixtures and test your parser against them offline with CTest, so failures point to parsing, not the network. Build test binaries with AddressSanitizer and UndefinedBehaviorSanitizer to catch leaks and memory errors around C library calls, and run ThreadSanitizer on the concurrent crawler. Refresh fixtures when the site changes.

Save the pages your parser depends on with curl -o into fixtures/: a listing page and a product page are enough here. The test below checks the parser, URL resolution, price parsing, CSV quoting, and the robots.txt rules, without touching the network:

// parser_test.cpp: offline checks against saved pages, run with ctest
#include "books.hpp"
#include "csv.hpp"
#include "robots.hpp"
#include <fstream>
#include <iostream>
#include <sstream>

static int failures = 0;
#define CHECK(cond) do { if (!(cond)) { std::cerr << "FAIL line " << __LINE__ << ": " #cond "\n"; ++failures; } } while (0)

static std::string slurp(const std::string& p) { std::ifstream f(p, std::ios::binary); std::ostringstream s; s << f.rdbuf(); return s.str(); }

int main() {
    curl_global_init(CURL_GLOBAL_DEFAULT);
    const std::string base = "https://books.toscrape.com/catalogue/page-1.html";
    auto page = parse_listing(slurp(FIXTURES "/books-page1.html"), base);
    CHECK(page.books.size() == 20);
    CHECK(page.books[0].title == "A Light in the Attic");
    CHECK(page.books[0].price_pence == 5177);
    CHECK(page.books[0].rating == 3);
    CHECK(page.books[0].in_stock);
    CHECK(page.books[0].url == "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html");
    CHECK(page.next_url == "https://books.toscrape.com/catalogue/page-2.html");

    // The same href means a different URL on the home page: resolve against the page you fetched.
    CHECK(resolve_url("https://books.toscrape.com/index.html", "catalogue/page-2.html")
          == "https://books.toscrape.com/catalogue/page-2.html");
    CHECK(resolve_url("https://books.toscrape.com/catalogue/page-2.html", "../index.html")
          == "https://books.toscrape.com/index.html");

    Book b; parse_detail(slurp(FIXTURES "/book-detail.html"), b);
    CHECK(b.upc == "a897fe39b1053632");
    CHECK(b.stock_count == 22);

    CHECK(parse_pence("£0.99") == 99);
    CHECK(!parse_pence("free").has_value());
    CHECK(csv_field("Starving Hearts (Triangular Trade Trilogy, #1)") == "\"Starving Hearts (Triangular Trade Trilogy, #1)\"");
    CHECK(csv_field("6\" ruler") == "\"6\"\" ruler\"");

    // robots.txt: a 404 means no rules; otherwise the most specific matching rule wins.
    CHECK(Robots("", "bookscraper").allowed("/catalogue/page-1.html"));
    const std::string txt = "User-agent: *\nDisallow: /private/\nAllow: /private/public-page\n\n"
                            "User-agent: bookscraper\nDisallow: /search\n";
    CHECK(Robots(txt, "bookscraper/1.0").allowed("/private/x"));          // our own group replaces "*"
    CHECK(!Robots(txt, "bookscraper/1.0").allowed("/search?q=c"));
    CHECK(!Robots(txt, "otherbot").allowed("/private/x"));
    CHECK(Robots(txt, "otherbot").allowed("/private/public-page"));

    curl_global_cleanup();
    std::cout << (failures ? "FAILED" : "all parser checks passed") << '\n';
    return failures ? 1 : 0;
}

Run it through CTest, then build the same test with sanitizers. ThreadSanitizer needs its own build, because it can't be combined with AddressSanitizer:

ctest --test-dir build --output-on-failure

# Memory and undefined-behavior checks (GCC or Clang)
g++ -std=c++20 -g -fsanitize=address,undefined -Isrc -DFIXTURES='"fixtures"' tests/parser_test.cpp $(pkg-config --cflags --libs libcurl lexbor) -o parser_test_asan && ./parser_test_asan

# Data races in the crawler (separate build; TSan can't be combined with ASan)
g++ -std=c++20 -g -fsanitize=thread src/main.cpp $(pkg-config --cflags --libs libcurl lexbor) -o bookscraper_tsan && ./bookscraper_tsan crawl 1

Output

all parser checks passed

100% tests passed, 0 tests failed out of 1

A clean sanitizer run prints only the program's own output. Any leak, out-of-bounds read, or data race prints a report with the exact line.

Two debugging habits save the most time. First, set CURLOPT_VERBOSE to 1 on the handle to see every request and response header, which quickly shows redirects, compression, and status codes you didn't expect. Second, when a selector returns nothing, write r.body to a file and open it. The HTML you received is often a consent page, an error page, or a JavaScript shell, not the page you saw in the browser.

Read robots.txt before crawling and skip disallowed paths, identify your scraper with an honest User-Agent that includes a contact URL, and space out requests. Collect only the public data you need, avoid personal data unless you have a lawful basis, and respect the site's terms. Back off immediately when the server answers 429.

The crawler already does most of this: it sends an identifying User-Agent, pauses between listing pages, limits itself to four workers, and waits as long as Retry-After asks. The last piece is robots.txt. This small checker follows RFC 9309:

// robots.hpp: a small RFC 9309 check for one user agent (longest matching rule wins; Allow wins ties)
#pragma once
#include <algorithm>
#include <cctype>
#include <sstream>
#include <string>
#include <vector>

class Robots {
public:
    // body = robots.txt contents; pass an empty string when the file returns 404 (everything is allowed)
    Robots(const std::string& body, std::string agent) {
        std::transform(agent.begin(), agent.end(), agent.begin(), tolower_ascii);
        std::istringstream in(body);
        std::string line;
        bool in_group = false, group_has_rules = false, matched_specific = false;
        std::vector<Rule> star, mine;
        std::vector<Rule>* target = nullptr;
        while (std::getline(in, line)) {
            line = trim(line.substr(0, line.find('#')));
            const auto colon = line.find(':');
            if (colon == std::string::npos) continue;
            std::string key = trim(line.substr(0, colon)), value = trim(line.substr(colon + 1));
            std::transform(key.begin(), key.end(), key.begin(), tolower_ascii);
            if (key == "user-agent") {
                if (!in_group || group_has_rules) { target = nullptr; group_has_rules = false; }
                in_group = true;
                std::transform(value.begin(), value.end(), value.begin(), tolower_ascii);
                if (value == "*") { if (!target) target = &star; }
                else if (agent.find(value) != std::string::npos) { target = &mine; matched_specific = true; }
            } else if (key == "allow" || key == "disallow") {
                group_has_rules = true;
                if (target && !value.empty()) target->push_back({value, key == "allow"});
            }
        }
        rules_ = matched_specific ? mine : star;
    }

    bool allowed(const std::string& path) const {
        size_t best = 0; bool allow = true;
        for (const Rule& r : rules_)
            if (path.compare(0, r.prefix.size(), r.prefix) == 0 &&
                (r.prefix.size() > best || (r.prefix.size() == best && r.allow))) {
                best = r.prefix.size(); allow = r.allow;
            }
        return allow;
    }

private:
    struct Rule { std::string prefix; bool allow; };
    static char tolower_ascii(char c) { return (c >= 'A' && c <= 'Z') ? char(c - 'A' + 'a') : c; }
    static std::string trim(const std::string& s) {
        const auto b = s.find_first_not_of(" \t\r"), e = s.find_last_not_of(" \t\r");
        return b == std::string::npos ? "" : s.substr(b, e - b + 1);
    }
    std::vector<Rule> rules_;
};

It picks the group for your user agent, or falls back to the * group when there's none, and the longest matching rule wins, with Allow winning ties. A robots.txt that returns 404 means there are no rules, which is the case for Books to Scrape:

robots.txt: HTTP 404, /catalogue/ allowed: true

This checker is deliberately small. It doesn't handle the * and $ wildcards that RFC 9309 allows inside paths, so use a full parser for production crawls of sites you don't control.

Robots.txt is a crawling convention, not a license. What you may collect also depends on the data, the site's terms, and the law where you operate. Read is web scraping legal? before you scrape anything for a commercial project.

For parsing and processing, usually yes: compiled C++ with lexbor uses less CPU and memory per page than most Python stacks. For most scrapers, though, network latency and polite delays dominate total time, so the real gain is smaller than benchmarks suggest. Choose C++ when parsing volume, memory limits, or an existing codebase justify it.

C and C++ compile to native code and use C libraries such as libcurl and lexbor, so one stack serves both. C# runs on .NET and has its own tools, including HttpClient, AngleSharp, and Playwright for .NET. The names look alike, but the libraries don't overlap. C# developers should follow our separate C# guide.

lexbor is the strongest default in 2026: it implements the WHATWG HTML5 parsing algorithm, includes a CSS selector engine, has no external dependencies, and installs through vcpkg on Windows, macOS, and Linux. Use libxml2 when you need XPath or already depend on it, and pugixml only for well-formed XML, not real-world HTML.

Either works, because cpr is a C++ wrapper around libcurl with a friendlier, requests-style API. Raw libcurl gives you every option, such as built-in Retry-After parsing and the URL API, with one fewer dependency, and the same calls work in plain C. Pick cpr if you prefer shorter code and only write C++.

Yes. The vcpkg port of lexbor builds on Windows desktop, macOS, and Linux. Only Universal Windows Platform apps are excluded. In Visual Studio, open the project folder, point CMAKE_TOOLCHAIN_FILE at vcpkg's toolchain file, and the manifest installs libcurl, lexbor, and nlohmann/json on the first configure. No manual library paths are needed.

Yes, but not with libcurl alone, because libcurl doesn't run JavaScript. First check whether the data already sits in a script tag as JSON. Otherwise, run headless Chrome with --dump-dom, drive a browser over WebDriver or the DevTools Protocol, or call a scraping API with JavaScript rendering enabled and parse the returned HTML.

Send fewer requests, more slowly, with an honest User-Agent, and honor Retry-After when a server answers 429. Reuse connections instead of opening new ones, and don't run more workers than the site can handle. For protected sites, route requests through rotating proxies with CURLOPT_PROXY, or use a scraping API that manages proxies and browsers.

Scraping public data is legal in many jurisdictions, but the answer depends on what you collect, how you collect it, and the site's terms. Avoid personal data without a lawful basis, never bypass logins or paywalls, follow robots.txt, and get legal advice for commercial projects. The programming language you use doesn't change the rules.

Set CURLOPT_PROXY to the proxy URL, such as http://host:port or socks5h://host:port, and CURLOPT_PROXYUSERPWD to the credentials, read from environment variables rather than source code. libcurl supports HTTP, HTTPS, and SOCKS proxies. To rotate IP addresses, point the option at a provider's rotating gateway and keep the rest of your scraper unchanged.

Start with about four concurrent requests per site and measure. Network waits, not CPU, limit most scrapers, so extra threads mainly add load on the target and raise block rates. Give each thread its own libcurl handle. For thousands of parallel transfers, use libcurl's multi interface on a few threads instead of one thread per request.

Keep your libcurl and lexbor code and let WebScrapingAPI handle rendering and geotargeting. Add render_js=1 and country to one GET request, and parse the HTML you already know. JavaScript rendering and geotargeting cost no extra credits. Failed requests aren't billed, and the free trial includes 5,000 requests. Try Scraper API