Skip to content
Start free
Back to Blog

Web Scraping with JavaScript and Node.js: A Practical Guide

Robert SfichiLast updated on 22 min read
Web Scraping with JavaScript and Node.js: A Practical Guide
TL;DR: Web scraping with JavaScript and Node.js needs Node.js 22.12 or later, the built-in fetch for requests, and Cheerio to parse HTML. Use Cheerio when the data is in the HTML your request receives, and Playwright when JavaScript builds the page or it needs clicks and waits. This guide builds one scraper, from pagination and hidden JSON to offline tests and a scraper API fallback.

Web scraping with JavaScript starts with a simple job: the data sits on a web page, you write JavaScript, and you want that data as clean JSON without fighting blocks or empty arrays. If the idea is new to you, read what web scraping is first.

This guide uses books.toscrape.com, a practice bookstore built for scrapers, as its running example. You'll fetch and parse a page with native fetch and Cheerio, crawl a paginated catalog politely, check whether JavaScript builds a page, render and click through pages with Playwright, test your parser offline, and move blocked pages to a scraper API without changing the parser.

Quick start

  • Use Node.js 22.12 or later (24 or 26 LTS recommended)
  • npm install cheerio for pages whose data is in the HTML
  • npm install playwright for pages that need a real browser
  • Check robots.txt and the site's terms before you crawl

What do you need to start web scraping with JavaScript?

You need Node.js 22.12 or later, a code editor, and two packages: Cheerio to parse HTML and Playwright to render pages that need a browser. Node's built-in fetch handles requests, so you don't need Axios. Learn your browser's DevTools too, because finding the right selector and the page's data source is half the job.

Node.js suits scrapers well. It speaks the same language as the page's own scripts, so reading a site's JavaScript feels familiar. Its async I/O model handles many network requests without threads. And the leading browser automation tools, Playwright and Puppeteer, are JavaScript-first.

The table below covers the tools this guide uses. For a broader roundup, see six Node.js scraping libraries compared.

Tool

Job

Use it when

Latest on npm (Oct 2026)

Node floor

fetch (built in)

HTTP requests

Always; no install

Part of Node

Stable since Node 21

Cheerio

Parse HTML with CSS selectors

The data is in the HTML

1.2.0

20.18.1

jsdom

A browser-like DOM in Node

You need DOM APIs, not just selectors

30.1.3

22.22.2, 24.15, or 26

Playwright

Drive Chromium, Firefox, WebKit

Pages need clicks, waits, or rendering

1.64.1

20

Puppeteer

Drive Chrome or Firefox

Existing Puppeteer code, Chrome tooling

25.16.0

22.12.0

p-limit

Cap concurrent promises

Many URLs on sites that allow parallel requests

7.3.3

20

Axios / Got

HTTP clients with extras

You already use them

1.21.0 / 16.1.0

— / 22

jsdom builds a fuller DOM than Cheerio, but it's slower and runs page scripts only with an option its own docs call unsafe, so Cheerio is the better default. Axios is fine if your project already uses it. The Node floors explain the version advice: Puppeteer needs 22.12, and Node 20 is past end of life on the Node.js release schedule.

How do you set up a Node.js scraping project?

Create a folder, switch the project to ES modules so import and top-level await work, and install the two main packages. Playwright downloads its own Chromium build with one extra command.

mkdir node-scraper && cd node-scraper
npm init -y
npm pkg set type=module
npm install cheerio playwright
npx playwright install chromium
mkdir fixtures out
# Optional, for the Puppeteer example:
npm install puppeteer

The fixtures/ folder holds saved pages for offline tests, and out/ holds results. Puppeteer downloads Chrome during install, and npm 11 may print an install-scripts warning for it. You can review those scripts with npm install-scripts ls. If you write TypeScript, the same code runs in .ts files through Node's built-in type stripping or a runner such as tsx.

Can you scrape a website with JavaScript in the browser?

Only on pages from the same site. Browsers block JavaScript on one site from reading responses from another unless that server allows it with CORS headers, and most don't. That's why scrapers run in Node.js, which has no such rule. Use browser JavaScript for testing selectors in the DevTools console, not for collecting data.

The rule behind this is the same-origin policy. A script loaded from one origin can send a request to another, but the browser hides the response unless the other server opts in with Access-Control-Allow-Origin headers. MDN's CORS guide explains the details.

Node.js isn't a browser, so it reads any response it receives. That makes it the right place for a scraper, and it keeps credentials such as API keys off visitors' machines. Browser extensions and bookmarklets are the one exception: they can read the page you're already viewing, which suits one-off copying but not scheduled collection.

How do you scrape a web page with Node.js fetch and Cheerio?

Fetch the page's HTML with fetch(), check that the response is OK, and load the text into Cheerio with cheerio.load(). Select each repeating card or row with a CSS selector, read text or attributes from it, and convert numbers as you go. Return plain objects, and resolve relative links with new URL(href, pageUrl).

books.toscrape.com lists 1,000 books across 50 catalog pages, 20 per page, with prices, star ratings, and stock status. A banner on the site says it's a demo for scraping practice and that prices and ratings are random, so it's a safe place to learn.

Start with a fetch helper you'll reuse everywhere. It sends a descriptive user agent, gives up after 15 seconds with AbortSignal.timeout(), checks res.ok, and retries rate limits and server errors. When the server sends the Retry-After header, it waits that long. Otherwise, it backs off exponentially.

// fetch-html.js: one polite GET with a timeout, a status check, and retries
const HEADERS = {
  'User-Agent': 'node-scraper-tutorial/1.0 (+https://example.com/contact)',
  'Accept': 'text/html,application/xhtml+xml',
  'Accept-Language': 'en-US,en;q=0.9',
};
const RETRY = new Set([429, 500, 502, 503, 504]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function fetchHtml(url, { retries = 3, timeoutMs = 15_000 } = {}) {
  for (let attempt = 0; ; attempt++) {
    let res;
    try {
      res = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(timeoutMs) });
    } catch (err) {                                   // DNS, reset, or timeout
      if (attempt >= retries) throw err;
      await sleep(1000 * 2 ** attempt);
      continue;
    }
    if (res.ok) return res.text();
    if (!RETRY.has(res.status) || attempt >= retries) {
      throw new Error(`GET ${url} failed: HTTP ${res.status}`);
    }
    const retryAfter = Number(res.headers.get('retry-after'));  // seconds, if the server says so
    await sleep(retryAfter > 0 ? retryAfter * 1000 : 1000 * 2 ** attempt);
  }
}

Replace the contact URL with your own, so a site owner can reach you instead of blocking you. fetch is built into Node, and the Fetch API works the same way it does in the browser. If you're weighing a library instead, see native fetch, Axios, Got, and Ky compared.

The parser takes HTML and the page's URL and returns typed records. Keeping it free of network code makes it easy to test and easy to reuse when you change how you fetch.

// parse-books.js: turn one books.toscrape.com listing page into typed records
import * as cheerio from 'cheerio';

const RATINGS = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };

export function parseBooks(html, pageUrl) {
  const $ = cheerio.load(html);

  const books = $('article.product_pod').map((_, card) => {
    const $card = $(card);
    const link = $card.find('h3 a');
    const priceText = $card.find('.price_color').text();          // "£51.77"
    const ratingWord = ($card.find('.star-rating').attr('class') ?? '')
      .replace('star-rating', '').trim();                          // "Three"

    return {
      title: link.attr('title'),                    // full title; the link text is cut off
      price: priceText ? Number(priceText.replace(/[^0-9.]/g, '')) : null,
      currency: priceText.includes('£') ? 'GBP' : null,
      rating: RATINGS[ratingWord] ?? null,
      inStock: $card.find('.instock.availability').length > 0,
      url: new URL(link.attr('href'), pageUrl).href, // hrefs are relative, and differ by page
    };
  }).get();

  const nextHref = $('li.next a').attr('href');
  const nextUrl = nextHref ? new URL(nextHref, pageUrl).href : null;
  return { books, nextUrl };
}

The output needs no second cleanup pass. The price becomes a number, not the string "£51.77". The star rating lives in a class name (star-rating Three), so a small map turns it into 1 to 5, and anything unexpected becomes null instead of a wrong guess. And every link goes through new URL(), because the site's hrefs are relative.

Save a copy of the first catalog page and run the parser on it with no network involved:

curl -o fixtures/books-page-1.html "https://books.toscrape.com/catalogue/page-1.html"
// static-fixture.js: parse a saved page, no network
import { readFile } from 'node:fs/promises';
import { parseBooks } from './parse-books.js';

const html = await readFile('fixtures/books-page-1.html', 'utf8');
const { books, nextUrl } = parseBooks(html, 'https://books.toscrape.com/catalogue/page-1.html');
console.log(books.length, 'books');
console.log(books[0]);
console.log('next:', nextUrl);

Output:

20 books
{
  title: 'A Light in the Attic',
  price: 51.77,
  currency: 'GBP',
  rating: 3,
  inStock: true,
  url: 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html'
}
next: https://books.toscrape.com/catalogue/page-2.html

Save each page type you plan to scrape the same way, one curl -o per page with a second between them. Those files become your test fixtures later. Cheerio has more to offer than this, including extract() and reading JSON from script tags. Both are covered in our full Cheerio guide, and Cheerio's documentation lists the full API.

How do you find the right CSS selectors?

Open the page in your browser, right-click the data you want, and choose Inspect. Find the element that repeats once per record, such as a product card with a class, and test your selector in the DevTools console with document.querySelectorAll(). Prefer short class-based selectors over long paths copied from DevTools, which break when the layout shifts.

On books.toscrape.com, right-click any book title and choose Inspect. Each book sits in an <article class="product_pod">, so that's the repeating element. Test it in the Console tab:

document.querySelectorAll('article.product_pod').length   // 20
document.querySelector('article.product_pod h3 a').textContent   // "A Light in the ..."
document.querySelector('article.product_pod h3 a').title         // "A Light in the Attic"

The second line shows a trap that's easy to miss. Long titles are shortened in the visible link text, but the full title sits in the link's title attribute. Always compare what you extract against what the page means, not just what it shows. The rating works the same way: there's no "3" anywhere in the card, only the class star-rating Three.

Know the difference between View Source and Inspect, too. View Source shows the HTML the server sent, which is exactly what fetch receives. Inspect shows the live DOM after scripts have run. When a selector works in Inspect but returns nothing in Node.js, JavaScript built that content, and the sections below show what to do.

Anchor selectors on classes and attributes rather than position (li.next a, not ul > li:nth-child(2) > a). Also, CSS IDs can't start with a digit unless escaped, so an element with id="2015" needs [id="2015"] rather than #2015. For everything else, keep a full selector reference open.

How do you scrape multiple pages politely?

Check robots.txt first, then follow each page's Next link instead of guessing URL patterns. Keep a set of visited URLs so a bad link can't loop, dedupe records by a stable key, and wait between requests. Honor the site's own rate rules and Retry-After headers, and cap total pages while you develop.

Start with robots.txt. This minimal reader collects the Disallow rules for the User-agent: * group. It handles simple prefix rules, while the robots.txt standard (RFC 9309) defines wildcards and Allow precedence, so use a full parser on sites that rely on them. Under that standard, a 4xx response means no rules apply, and a 5xx means you should treat the whole site as off-limits.

// robots.js: a minimal robots.txt check for the "User-agent: *" group.
// Good enough for simple prefix rules; use a full RFC 9309 parser for wildcards or Allow precedence.
export async function disallowedPaths(origin) {
  const res = await fetch(new URL('/robots.txt', origin), { signal: AbortSignal.timeout(10_000) });
  if (res.status >= 500) throw new Error(`robots.txt HTTP ${res.status}: treat the site as off-limits`);
  if (!res.ok) return [];                       // 4xx: no robots.txt, so nothing is disallowed
  let inStar = false;
  const rules = [];
  for (const raw of (await res.text()).split('\n')) {
    const line = raw.split('#')[0].trim();
    const [key, ...rest] = line.split(':');
    const value = rest.join(':').trim();
    if (/^user-agent$/i.test(key)) inStar = value === '*';
    else if (inStar && /^disallow$/i.test(key) && value) rules.push(value);
  }
  return rules;
}

export const isAllowed = (url, rules) => !rules.some((p) => new URL(url).pathname.startsWith(p));

The crawler starts at the first catalog page, follows the next link until there isn't one, waits a second between requests, and saves JSON and CSV.

// crawl-books.js: follow "next" links politely and save JSON + CSV
import { writeFile } from 'node:fs/promises';
import { fetchHtml } from './fetch-html.js';
import { parseBooks } from './parse-books.js';
import { disallowedPaths, isAllowed } from './robots.js';

const START = 'https://books.toscrape.com/catalogue/page-1.html';
const MAX_PAGES = Number(process.env.MAX_PAGES ?? 5);
const DELAY_MS = 1000;                           // no published limit, so stay at one request per second

const rules = await disallowedPaths(START);
const seen = new Set();                          // stops loops if a "next" link points backward
const byUrl = new Map();                         // dedupe records, not just pages

for (let url = START, page = 1; url && page <= MAX_PAGES && !seen.has(url); page++) {
  if (!isAllowed(url, rules)) { console.log('robots.txt disallows', url); break; }
  seen.add(url);
  const { books, nextUrl } = parseBooks(await fetchHtml(url), url);
  for (const b of books) byUrl.set(b.url, b);
  console.log(`page ${page}: ${books.length} books (${url})`);
  url = nextUrl;
  if (url) await new Promise((r) => setTimeout(r, DELAY_MS));
}

const books = [...byUrl.values()];
await writeFile('out/books.json', JSON.stringify(books, null, 2));
const cols = Object.keys(books[0]);
const cell = (v) => `"${String(v ?? '').replaceAll('"', '""')}"`;
await writeFile('out/books.csv',
  [cols.join(','), ...books.map((b) => cols.map((c) => cell(b[c])).join(','))].join('\n') + '\n');
console.log(`saved ${books.length} unique books; robots.txt disallows: ${rules.join(', ') || 'nothing'}`);

Output with the default five-page cap:

page 1: 20 books (https://books.toscrape.com/catalogue/page-1.html)
page 2: 20 books (https://books.toscrape.com/catalogue/page-2.html)
page 3: 20 books (https://books.toscrape.com/catalogue/page-3.html)
page 4: 20 books (https://books.toscrape.com/catalogue/page-4.html)
page 5: 20 books (https://books.toscrape.com/catalogue/page-5.html)
saved 100 unique books; robots.txt disallows: nothing

Run it with MAX_PAGES=100 node crawl-books.js and it stops on its own after page 50, because the last page has no next link:

page 49: 20 books (https://books.toscrape.com/catalogue/page-49.html)
page 50: 20 books (https://books.toscrape.com/catalogue/page-50.html)
saved 1000 unique books; robots.txt disallows: nothing

This site shows why each safeguard exists. The next link on the home page is catalogue/page-2.html, but on page 2 it's just page-3.html. Gluing that onto the site root gives https://books.toscrape.com/page-3.html, which returns 404. new URL(href, pageUrl) resolves each link against the page it came from, so it's always right.

The home page and catalogue/page-1.html also serve the same 20 books under two URLs, and both point their next link at page 2. A crawler that seeds from both, or follows a "previous" link back to page 1, would save 40 rows for 20 books. The visited set stops repeat pages, and keying records by book URL stops repeat rows. For more habits like these, read our guide to scraping best practices across the project lifecycle.

How many requests can you run at once?

Concurrency is a per-site decision. When a site publishes a rate limit, follow it, even if that means one request at a time. When it doesn't, cap parallel requests per host with a small limit, keep a pause, and back off on 429. More speed rarely helps if it gets you blocked.

books.toscrape.com doesn't publish a limit, so this script fetches book detail pages two at a time with p-limit (npm install p-limit) to add each book's UPC and exact stock count:

// details.js: enrich books from their detail pages, at most 2 requests at a time
import { readFile, writeFile } from 'node:fs/promises';
import * as cheerio from 'cheerio';
import pLimit from 'p-limit';
import { fetchHtml } from './fetch-html.js';

const books = JSON.parse(await readFile('out/books.json', 'utf8')).slice(0, 10);
const limit = pLimit(2);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

const results = await Promise.allSettled(books.map((book) => limit(async () => {
  const $ = cheerio.load(await fetchHtml(book.url));
  await sleep(500);                               // keep a pause even with two workers
  return {
    ...book,
    upc: $('th:contains("UPC")').next('td').text(),
    stockCount: Number($('p.instock.availability').text().match(/\d+/)?.[0] ?? 0),
  };
})));

const ok = results.filter((r) => r.status === 'fulfilled').map((r) => r.value);
console.log(`${ok.length} enriched, ${results.length - ok.length} failed`);
console.log(ok[0]);
await writeFile('out/books-details.jsonl', ok.map((b) => JSON.stringify(b)).join('\n') + '\n');

Output:

10 enriched, 0 failed
{
  title: 'A Light in the Attic',
  price: 51.77,
  currency: 'GBP',
  rating: 3,
  inStock: true,
  url: 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html',
  upc: 'a897fe39b1053632',
  stockCount: 22
}

Promise.allSettled() keeps one failed page from throwing away the other nine results.

How do you save scraped data to JSON and CSV?

JSON keeps your types: numbers stay numbers and null stays null. CSV is what spreadsheets want, but it needs quoting, so the crawler wraps every cell in quotes and doubles any quote inside a value. Empty values become empty cells.

title,price,currency,rating,inStock,url
"A Light in the Attic","51.77","GBP","3","true","https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
"Tipping the Velvet","53.74","GBP","1","true","https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"

For bigger jobs, write one JSON object per line (JSON Lines), as details.js does, so a crash halfway through doesn't cost you the whole file. If your source is a table rather than cards, see our guide to exporting HTML tables to clean CSV.

How do you know if a page needs JavaScript rendering?

Fetch the page with plain fetch() and run your selector. If it finds nothing but the browser shows the data, JavaScript builds that content after load. Then open DevTools, select the Network tab, filter by Fetch/XHR, and reload or click: the request that returns your data is usually easier to use than a browser.

You can automate that check. This script counts your selector's matches in the raw HTML and in a rendered page, and lists every fetch or XHR request the page makes while loading:

// check-js.js: does the selector match in the raw HTML and in the rendered page?
import * as cheerio from 'cheerio';
import { chromium } from 'playwright';
import { fetchHtml } from './fetch-html.js';

const url = process.argv[2] ?? 'https://books.toscrape.com/catalogue/page-1.html';
const selector = process.argv[3] ?? 'article.product_pod';

const $ = cheerio.load(await fetchHtml(url));
const inHtml = $(selector).length;

const browser = await chromium.launch();
try {
  const page = await browser.newPage();
  const dataRequests = [];
  page.on('request', (req) => {
    if (['xhr', 'fetch'].includes(req.resourceType())) dataRequests.push(req.url());
  });
  await page.goto(url, { waitUntil: 'networkidle' });
  const rendered = await page.locator(selector).count();

  console.log(`raw HTML: ${inHtml} | rendered: ${rendered} | fetch/XHR requests: ${dataRequests.length}`);
  dataRequests.forEach((u) => console.log('  ', u));
  console.log(inHtml > 0 ? 'The data is in the HTML: use fetch + Cheerio.'
    : dataRequests.length ? 'JavaScript loads the data: try calling the request above.'
    : 'JavaScript builds the data: render the page or look for JSON in a <script>.');
} finally {
  await browser.close();
}

Output for the books catalog:

raw HTML: 20 | rendered: 20 | fetch/XHR requests: 0
The data is in the HTML: use fetch + Cheerio.

That's the answer you hope for: the server sends every book in the HTML, so a browser would only add time. Run the same script with another URL and selector (node check-js.js <url> "<selector>") before you write a scraper for a new site.

When the raw count is 0, the data is hiding in one of two other places. It may be JSON inside a <script> tag, which Cheerio can read and JSON.parse() can decode. Or JavaScript requests it after load, and the request shows up in the script's list and in the Network tab. Either way, you can often skip the browser. For background on what a rendering browser does, see how headless browsers work.

How do you scrape JavaScript data without a browser?

Find the request the page makes for its data in the DevTools Network tab, copy its URL and parameters, and call it yourself with fetch(). You get JSON with no HTML parsing and no browser to run. Keep the same rate limit and user agent, and only call endpoints the page itself uses for public data.

The steps are the same on any site:

  1. Open DevTools, select the Network tab, and filter by Fetch/XHR.
  2. Reload the page, or click the button, tab, or "load more" link that brings in the data.
  3. Click each request and check its Response tab until you find the one holding your records.
  4. Note the URL, the query parameters that change (often page, offset, or cursor), and any field that tells you when to stop, such as has_next or an empty array.
  5. Rebuild the URL in Node.js with URLSearchParams and fetch it in a loop.

A paged JSON endpoint usually ends up looking like this. Swap in the endpoint and fields you found:

// fetch-json-pages.js: call a paged JSON endpoint you found in the Network tab
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function fetchAllPages(endpoint, { maxPages = 50, delayMs = 1000 } = {}) {
  const items = [];
  for (let page = 1; page <= maxPages; page++) {
    const url = `${endpoint}?${new URLSearchParams({ page })}`;
    const res = await fetch(url, {
      headers: { 'User-Agent': 'node-scraper-tutorial/1.0 (+https://example.com/contact)' },
      signal: AbortSignal.timeout(15_000),
    });
    if (!res.ok) throw new Error(`HTTP ${res.status} for ${url}`);
    const data = await res.json();
    items.push(...data.items);           // the array that holds the records
    if (!data.has_next) break;           // or: if (data.items.length === 0) break;
    await sleep(delayMs);
  }
  return items;
}

When it exists, this is the fastest and most stable path: JSON fields rarely change when a site redesigns its HTML. Normalize as you go, the same way the Cheerio parser does: trim strings, convert numbers, and turn missing keys into null or false on purpose.

Know where to stop, too. An endpoint that needs a login, a session token, or signed parameters isn't public data, and working around those checks isn't scraping anymore. Treat it as a stop sign, not a puzzle.

How do you scrape JavaScript-rendered pages with Playwright?

Launch Chromium with Playwright, open the page, and do what a user would do, such as clicking a link. Wait for the data instead of sleeping: waitForURL() catches the navigation a click starts, and a locator's waitFor() waits until rows exist. Then read the rows inside the page with evaluateAll() and close the browser in a finally block.

Use a browser when the data only appears after clicks, scrolling, typing, or other script-driven steps. This script browses books.toscrape.com like a reader would: it opens the home page, clicks the Mystery category, reads every card, and clicks "next" until the category runs out.

// playwright-books.js: open a category like a user, read every page of results
import { chromium } from 'playwright';

const browser = await chromium.launch();               // headless by default
try {
  const page = await browser.newPage({ userAgent: 'node-scraper-tutorial/1.0 (+https://example.com/contact)' });
  await page.route('**/*.{jpg,jpeg,png}', (route) => route.abort());   // skip images you won't use
  await page.goto('https://books.toscrape.com/');

  await page.getByRole('link', { name: 'Mystery', exact: true }).click();
  await page.waitForURL('**/category/books/mystery_3/**');

  const books = [];
  for (;;) {
    const cards = page.locator('article.product_pod');
    await cards.first().waitFor();                     // wait for the data, not a fixed sleep
    books.push(...await cards.evaluateAll((els) => els.map((el) => ({
      title: el.querySelector('h3 a').getAttribute('title'),
      price: Number(el.querySelector('.price_color').textContent.replace(/[^0-9.]/g, '')),
      url: el.querySelector('h3 a').href,               // the browser resolves it for you
    }))));

    const next = page.locator('li.next a');
    if (await next.count() === 0) break;
    const current = page.url();
    await next.click();
    await page.waitForURL((u) => u.href !== current);  // the click navigates to the next page
    await page.waitForTimeout(1000);                   // politeness delay between pages
  }
  console.log(books.length, 'mystery books; last page:', page.url());
  console.log(books[0]);
} finally {
  await browser.close();
}

Output:

32 mystery books; last page: https://books.toscrape.com/catalogue/category/books/mystery_3/page-2.html
{
  title: 'Sharp Objects',
  price: 47.82,
  url: 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html'
}

A few choices here matter on any site. getByRole('link', { name: 'Mystery', exact: true }) finds the link the way a user sees it, so it survives class-name changes. waitForURL() after each click makes sure you read the new page, not the old one. cards.first().waitFor() waits until data exists, which beats a fixed sleep that's either too short on a slow day or wasted time on a fast one. Playwright's auto-waiting also checks that an element is visible and stable before it clicks.

Inside evaluateAll(), the code runs in the page, so a.href is already absolute. Blocking images with page.route() saves bandwidth for you and the site. The one fixed wait left is a deliberate politeness delay, not a guess about load time.

When a click triggers a data request instead of a navigation, wrap the click and page.waitForResponse() in one Promise.all() so the response can't arrive before you start listening. For proxies, infinite scroll, and request interception, see our in-depth Playwright guide.

What does the same scraper look like in Puppeteer?

Puppeteer covers the same flow with slightly different calls: page.click() paired with page.waitForNavigation(), then page.waitForSelector() and page.$$eval() to read the cards. Puppeteer 25 needs Node.js 22.12 or later.

// puppeteer-books.js: the same category in Puppeteer
import puppeteer from 'puppeteer';

const browser = await puppeteer.launch();
try {
  const page = await browser.newPage();
  await page.setUserAgent('node-scraper-tutorial/1.0 (+https://example.com/contact)');
  await page.goto('https://books.toscrape.com/');

  await Promise.all([
    page.waitForNavigation(),
    page.click('a[href*="category/books/mystery_3"]'),
  ]);
  await page.waitForSelector('article.product_pod');

  const books = await page.$$eval('article.product_pod h3 a', (links) => links.map((a) => ({
    title: a.getAttribute('title'),
    url: a.href,
  })));
  console.log(books.length, 'books on page 1;', 'first:', books[0].title);
} finally {
  await browser.close();
}

Output:

20 books on page 1; first: Sharp Objects

Note the Promise.all() around the click: start waiting for the navigation before you click, or a fast page can finish loading before Puppeteer starts listening. For screenshots, PDFs, and more Puppeteer patterns, see our Puppeteer and Node.js guide. If a page offers files for download, our Puppeteer file download guide compares four ways to save them.

Should you use Playwright or Puppeteer for web scraping?

Use Playwright for new scrapers: it waits for elements automatically, drives Chromium, Firefox, and WebKit from one API, and handles multiple isolated sessions well. Use Puppeteer when your team already has Puppeteer code or wants Chrome-specific tooling. For scraping, both render the same pages, so choose by API comfort, not speed claims.


Playwright

Puppeteer

Browsers

Chromium, Firefox, WebKit

Chrome, Firefox

Waiting

Locators auto-wait for elements to be ready

Explicit waitForSelector() and similar

Isolated sessions

Browser contexts, built in

Browser contexts, built in

Catch the data request

page.waitForResponse()

page.waitForResponse()

Languages

JavaScript/TypeScript, Python, Java, .NET

JavaScript/TypeScript

Latest on npm (Oct 2026)

1.64.1

25.16.0

Pick it when

New projects, cross-browser checks

Existing Puppeteer code, Chrome DevTools Protocol work

Both projects are open source and actively maintained, and both download a matching browser for you. The practical difference shows up in day-to-day code: Playwright's locators wait and retry by themselves, so scripts need fewer explicit waits. If you already run Puppeteer in production, there's no scraping reason to rewrite it.

How do you scrape blocked pages from Node.js with a scraper API?

Keep your parser and change only the fetch. Send the target URL to WebScrapingAPI's Scraper API with your key from an environment variable, add render_js=1 when the page needs JavaScript, and add country for location-specific content. You get the page's HTML back and pass it to the same Cheerio function you already wrote.

You don't need this for a practice site. You need it when a real target starts answering with 403s or challenge pages that retries don't fix, when content changes by country, or when you'd rather not run a fleet of browsers to render pages at volume. Try other ways to reduce blocks first if you're only seeing occasional errors.

The Scraper API takes a URL and returns the page, while WebScrapingAPI handles proxies, retries, and optional JavaScript rendering. wsaFetch() is a drop-in sibling of fetchHtml():

// wsa-fetch.js: same job as fetchHtml(), but WebScrapingAPI's Scraper API does the fetching
const ENDPOINT = 'https://api.webscrapingapi.com/v2';
const RETRY = new Set([429, 500, 503]);          // documented as "retry later"
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export function buildWsaUrl(url, { renderJs = false, country } = {}) {
  const apiKey = process.env.WSA_API_KEY;
  if (!apiKey) throw new Error('Set WSA_API_KEY in your environment first.');
  const params = new URLSearchParams({ api_key: apiKey, url });   // encodes the target URL
  if (renderJs) params.set('render_js', '1');
  if (country) params.set('country', country);                     // two-letter code, e.g. "us"
  return `${ENDPOINT}?${params}`;
}

export async function wsaFetch(url, { renderJs = false, country, retries = 2 } = {}) {
  for (let attempt = 0; ; attempt++) {
    const res = await fetch(buildWsaUrl(url, { renderJs, country }), {
      signal: AbortSignal.timeout(130_000),   // the API's own timeout tops out at 120 s
    });
    if (res.ok) return res.text();
    if (res.status === 422 && !renderJs) {    // scrape not completed: try once with rendering
      renderJs = true;
      continue;
    }
    if (!RETRY.has(res.status) || attempt >= retries) {
      throw new Error(`WebScrapingAPI HTTP ${res.status}: ${await res.text()}`);
    }
    await sleep(2000 * 2 ** attempt);
  }
}

The key comes from process.env.WSA_API_KEY and never appears in code. URLSearchParams encodes the target URL, which matters: if a target URL carried its own query string, an unencoded ? or & would be read as an API parameter. The client timeout sits just above the API's 120-second maximum, and only the statuses the API documents as temporary get retried. JavaScript rendering and geotargeting cost no extra credits, but rendering is slower, so turn it on only when the HTML lacks your data.

Swap the fetch layer and keep parseBooks() exactly as it was:

// crawl-books-wsa.js: swap the fetch layer, keep parseBooks() unchanged
import { wsaFetch, buildWsaUrl } from './wsa-fetch.js';
import { parseBooks } from './parse-books.js';

const url = 'https://books.toscrape.com/catalogue/page-1.html';
const opts = { country: 'us' };                    // add renderJs: true only if the HTML lacks the data

if (process.argv.includes('--dry-run')) {
  const redact = (u) => u.replace(/api_key=[^&]+/, 'api_key=REDACTED');
  console.log(redact(buildWsaUrl(url, opts)));
  console.log(redact(buildWsaUrl(url, { ...opts, renderJs: true })));
} else {
  const { books, nextUrl } = parseBooks(await wsaFetch(url, opts), url);
  console.log(books.length, 'books; first:', books[0]?.title, '; next:', nextUrl);
}

Put the key in a .env file containing WSA_API_KEY=your_key_here, add .env to .gitignore, and load it with Node's built-in --env-file flag, so you don't need the dotenv package:

node --env-file=.env crawl-books-wsa.js --dry-run
node --env-file=.env crawl-books-wsa.js

The dry run prints the request URLs with the key redacted, so you can see the encoding:

https://api.webscrapingapi.com/v2?api_key=REDACTED&url=https%3A%2F%2Fbooks.toscrape.com%2Fcatalogue%2Fpage-1.html&country=us
https://api.webscrapingapi.com/v2?api_key=REDACTED&url=https%3A%2F%2Fbooks.toscrape.com%2Fcatalogue%2Fpage-1.html&render_js=1&country=us

The real run hands the returned HTML to the same parser:

20 books; first: A Light in the Attic ; next: https://books.toscrape.com/catalogue/page-2.html

Errors stay explicit. Without the environment variable, the script stops with Error: Set WSA_API_KEY in your environment first. before sending anything, and a wrong key comes back as WebScrapingAPI HTTP 401: {"error":"Invalid API key"} instead of an empty result. For every parameter and error code, see our full Scraper API walkthrough. Pages that need clicks or typing fit a hosted browser better than a fetch, which the next section covers.

Which JavaScript scraping method should you use?

Use fetch and Cheerio when the data is in the HTML. Call the JSON endpoint when JavaScript loads data from one. Use Playwright when you must click, scroll, or wait on the page. Use a scraper API when blocks, location, or rendering at volume make local fetching unreliable. Most projects combine two of these.

Situation

Use

Why

Data is in View Source

fetch + Cheerio

Fastest, fewest moving parts

Many pages

fetch + Cheerio with robots.txt, delay, retries, dedupe

Polite and restartable

JavaScript loads data from an XHR you can call

fetch on the JSON endpoint

No parsing, no browser

Data is JSON inside a <script>

Cheerio + JSON.parse

No browser

Clicks, scrolling, or waiting on the page

Playwright (or Puppeteer)

Real browser, real events

403s, challenge pages, country-specific content, or rendering at volume

Scraper API from fetch

Managed proxies, retries, and rendering; parser unchanged

Hosted browser sessions with actions

Browser API

Sessions without running Chrome yourself

Run through these checks in order for each new site:

  1. Is the data in View Source? Use fetch and Cheerio.
  2. Does it arrive through a Fetch/XHR request? Call that request.
  3. Do you need clicks, scrolling, or waits? Use Playwright.
  4. Are you blocked, or is the content location-specific? Route the fetch through the Scraper API.
  5. Do you need hosted browser sessions that click and type? Use the Browser API.

How do you test a Node.js scraper so it doesn't break silently?

Save a copy of each page type as an HTML file and test your parser against it with Node's built-in test runner. Assert the row count, the types of key fields, and the next-page link. When the site changes its layout, a test fails loudly instead of your scraper quietly saving empty or wrong data.

Scrapers rarely crash when a site changes. They keep running and save empty strings, zeros, or nothing at all. Fixture tests catch that before bad data reaches your database. Save the last catalog page and a category page next to the first page you saved earlier:

curl -o fixtures/books-page-50.html "https://books.toscrape.com/catalogue/page-50.html"
sleep 1
curl -o fixtures/mystery-page-1.html "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"

Then write the tests with node:test and node:assert/strict. No extra packages are needed:

// parse-books.test.js: run with `node --test`; uses saved pages, no network
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { readFile } from 'node:fs/promises';
import { parseBooks } from './parse-books.js';

const BASE = 'https://books.toscrape.com/catalogue/';
const load = (name) => readFile(`fixtures/${name}`, 'utf8');

test('parses 20 typed records and the next link', async () => {
  const { books, nextUrl } = parseBooks(await load('books-page-1.html'), `${BASE}page-1.html`);
  assert.equal(books.length, 20);
  for (const b of books) {
    assert.ok(b.title.length > 0);
    assert.ok(Number.isFinite(b.price) && b.price > 0, b.title);
    assert.ok(b.rating >= 1 && b.rating <= 5, b.title);
  }
  assert.equal(nextUrl, `${BASE}page-2.html`);
});

test('resolves ../../../ links on category pages', async () => {
  const url = `${BASE}category/books/mystery_3/index.html`;
  const { books } = parseBooks(await load('mystery-page-1.html'), url);
  assert.equal(books[0].url, `${BASE}sharp-objects_997/index.html`);
});

test('the last page has no next link', async () => {
  const { books, nextUrl } = parseBooks(await load('books-page-50.html'), `${BASE}page-50.html`);
  assert.equal(books.length, 20);
  assert.equal(nextUrl, null);
});

Run node --test:

✔ parses 20 typed records and the next link
✔ resolves ../../../ links on category pages
✔ the last page has no next link
ℹ tests 3
ℹ pass 3
ℹ fail 0

The second test guards the relative-link trap: category pages link books as ../../../sharp-objects_997/index.html, and a parser that mishandles that would still return 20 records, just with broken URLs. Re-save your fixtures on a schedule and diff the parsed output, so you notice layout changes before your users do. Node's built-in test runner also supports watch mode, coverage, and snapshots.

Is JavaScript or Python better for web scraping?

Both work well. Choose JavaScript if your team already writes it, if you'll automate browsers, or if the scraper lives inside a Node.js service. Choose Python if the data feeds pandas, notebooks, or machine learning, or if you want Scrapy's crawling framework. The site's defenses matter far more than the language you pick.

The building blocks map almost one to one. Native fetch matches Python's HTTP clients, Cheerio plays the role of a fast HTML parser, and Playwright exists in both languages with nearly the same API. JavaScript's edge is browser automation and a single language from the page's scripts to your backend. Python's edge is the analysis stack downstream and a mature crawling framework for very large jobs. If your team lives in Python, read our guide to web scraping with Python. The decisions in this guide carry over unchanged. If you work in a C++ codebase, read our guide to web scraping in C++.

How do you scrape responsibly with JavaScript?

Read the site's terms and robots.txt before you crawl, keep request rates low, identify your scraper with an honest user agent, and collect only the fields you need. Avoid personal data unless you have a lawful basis, never scrape behind a login without permission, and use the site's official API when one exists.

Most of that is already in your code. robots.js checks the rules before the first page. The one-second delay and the two-worker cap keep your load small. The User-Agent header names your scraper and gives the site a way to reach you. And the parser keeps only the fields you need, typed and clean, instead of storing whole pages.

The rest is judgment. Public pages aren't a free-for-all: terms of service, copyright, and privacy law still apply, especially to personal data. Read our responsible web data principles and our overview of whether web scraping is legal, and get legal advice before you scrape for a commercial project.

Frequently asked questions

Is Node.js good for web scraping?

Yes. Node.js handles many network requests at once without threads, ships fetch built in, and runs the leading browser automation tools, Playwright and Puppeteer. Cheerio parses HTML quickly with familiar CSS selectors. It's a strong fit when your team already writes JavaScript or the scraper runs inside a Node.js service.

Why does Cheerio return an empty array?

Usually because the data isn't in the HTML your request received. JavaScript adds it in the browser after the page loads, so Cheerio never sees it. Check View Source in your browser: if the data is missing there, find the JSON request in the Network tab or render the page with Playwright.

What is the difference between Cheerio and Playwright?

Cheerio parses HTML you already fetched. It's fast and light, but it can't run JavaScript or click anything. Playwright drives a real browser, so it sees the page after scripts run and can click, scroll, and wait. Use Cheerio by default and reach for Playwright only when the page needs a browser.

Can JavaScript scrape dynamic websites?

Yes. For pages that load data with JavaScript, first look for the JSON request the page makes and call it with fetch(), which is faster and more stable. When you need clicks, scrolling, or waiting on the page, use Playwright or Puppeteer to load the page in a headless browser and read the rendered elements.

How do I avoid getting blocked when scraping with Node.js?

Slow down, send an honest user agent, retry 429 and 503 responses after the Retry-After delay, and cache pages so you don't fetch them twice. If a site still returns 403 errors or challenge pages, route the requests through a scraper API that manages proxies and rendering, and keep your Cheerio parser unchanged.

Do I still need Axios or node-fetch for web scraping?

No. Node.js 18 and later include fetch, and it's stable in every supported release, so a new scraper needs no HTTP library. Axios and Got still work and add conveniences such as interceptors and built-in retries. Keep them if your project already uses them. Otherwise, native fetch plus a small retry helper is enough.

How should I store an API key in a Node.js scraper?

Keep the key in an environment variable such as WSA_API_KEY and read it with process.env. For local work, put it in a .env file, run node --env-file=.env, and add .env to .gitignore. Never paste keys into code, URLs you share, or front-end JavaScript, where any visitor could read them.

Can I scrape websites with TypeScript?

Yes. Every example in this guide works in TypeScript, because Cheerio, Playwright, and Puppeteer ship their own type definitions. Recent Node.js versions can run .ts files directly by stripping types, or you can use a runner such as tsx. Typing your record shapes also catches parser mistakes before you save bad data.

How do I scrape a page with infinite scroll?

Open DevTools, scroll the page, and watch the Network tab for the request that loads each new batch. It usually takes a page number or cursor, so you can call it with fetch() in a loop until it reports no more results. If there's no usable request, scroll with Playwright and read the rows.

The language doesn't change the legal picture. Scraping public data is often lawful, but it depends on the site's terms, the kind of data, and local law, especially for personal data and copyrighted content. Respect robots.txt, avoid personal data without a lawful basis, and get legal advice for commercial projects.

Keep your Node.js parser, hand off the hard pages

Your Cheerio parser doesn't need to change when a site starts blocking you or needs JavaScript. Point your fetch at WebScrapingAPI's Scraper API and keep parsing in Node.js. JavaScript rendering and geotargeting cost no extra credits, and failed requests aren't billed. Compare Scraper API plans, starting with a free trial of 5,000 requests.

About the Author

Robert Sfichi, Full-Stack Developer @ WebScrapingAPI

Robert Sfichi

Full-Stack Developer

Robert Sfichi is a team member at WebScrapingAPI, contributing to the product and helping build reliable solutions that support the platform and its users.

Start Building

Ready to Scale Your Data Collection?

Join 2,000+ companies using WebScrapingAPI to extract web data at enterprise scale with zero infrastructure overhead.