Web Scraping in R: A Practical rvest and httr2 Guide

You need R 4.1 or later and two packages: rvest to parse HTML and httr2 to fetch pages with retries and rate limits. Add chromote when pages are built by JavaScript, and a scraper API when sites block you. Everything else, from dplyr to readr, is the tidyverse you already use.

R suits scraping because scraped rows land directly in a tibble, so cleaning and analysis happen in the same session.

rvest parses: it selects elements with CSS selectors or XPath and pulls out text, attributes, and tables. httr2 fetches: it sets headers, waits between requests, retries temporary failures, and runs requests in parallel. rvest can still fetch a URL by itself, which is fine for one page. Once you crawl, fetch with httr2 and parse with rvest.

Rcrawler and polite were both archived on CRAN (Rcrawler in April 2025, polite in September 2026), so older tutorials that depend on them won't install cleanly. You don't need them: httr2's req_throttle() handles request pacing, and the robots.txt check later in this guide covers the rest of what polite did.

Download R from CRAN if you don't have it. The current release is R 4.6.2, and every snippet here runs on R 4.1 or later. Any editor works: RStudio, Positron, or VS Code. Then install the packages this guide uses in one line:

install.packages(c("rvest", "httr2", "jsonlite", "dplyr", "readr", "chromote", "testthat"))

read_html_live() also needs Google Chrome on your machine. chromote finds it automatically.

Create a project folder with a fixtures/ subfolder for saved pages. Save the three Goodreads pages this guide parses offline:

download.file("https://www.goodreads.com/list/show/1.Best_Books_Ever", "fixtures/list-page1.html")
download.file("https://www.goodreads.com/list/show/1.Best_Books_Ever?page=2", "fixtures/list-page2.html")
download.file("https://www.goodreads.com/book/show/5470.1984", "fixtures/book-1984.html")

Load the page with read_html(), select repeating elements with html_elements() and a CSS selector, then pull fields from each with html_element() plus html_text2() or html_attr(). Build a tibble from those vectors. Because rvest functions are vectorized, you rarely need a loop, and missing fields come back as NA instead of breaking.

The Goodreads "Best Books Ever" list shows 100 books per page. Each book is one row of a table with the class tableList, and each row holds a rank, a title link, an author link, and a line like "4.36 avg rating — 10,338,532 ratings". This function turns one list page into a tibble:

# parse_books.R: turn one Goodreads list page into a tibble
library(rvest)
library(tibble)

parse_books <- function(page, page_url) {
  rows  <- html_elements(page, "table.tableList tr")
  links <- html_element(rows, "a.bookTitle")
  stats <- html_text2(html_element(rows, "span.minirating"))   # "4.36 avg rating — 10,338,532 ratings"
  tibble(
    rank       = as.integer(html_text2(html_element(rows, "td.number"))),
    title      = html_text2(links),
    author     = html_text2(html_element(rows, "a.authorName")),
    avg_rating = readr::parse_number(stats),
    ratings    = readr::parse_number(sub(".*avg rating", "", stats)),
    url        = xml2::url_absolute(html_attr(links, "href"), page_url)
  )
}

Line by line:

• html_elements(page, "table.tableList tr") returns every row: one node per book.
• html_element(rows, "a.bookTitle"), singular, returns exactly one result per row. That's what keeps the columns aligned. If a row were missing an author, you'd get NA in that position, not a shorter vector that shifts every value below it.
• html_text2() returns text the way a browser displays it, collapsing line breaks and runs of spaces. The older html_text() returns the raw text, indentation included.
• readr::parse_number() reads the first number in a string, so it turns the rating line into 4.36. Dropping everything up to "avg rating" first leaves "— 10,338,532 ratings", which parses to 10338532. Both columns are numbers you can sort and average, not strings.
• xml2::url_absolute() turns each relative href, such as /book/show/2767052-the-hunger-games, into a full URL. Resolve links against the page they came from, not a site root you typed by hand. A link can be relative to the page's folder, and the wrong base produces URLs that 404. The url_absolute() reference covers the rules.

Run it on the saved page:

# 01_static.R: parse a saved page (offline); the same code works on a live URL
source("parse_books.R")
page  <- read_html("fixtures/list-page1.html")
books <- parse_books(page, "https://www.goodreads.com/list/show/1.Best_Books_Ever")
print(books, n = 5, width = 100)
cat("rows:", nrow(books), "\n")

Output:

# A tibble: 100 × 6
   rank title                                                       
  <int> <chr>                                                       
1     1 The Hunger Games (The Hunger Games, #1)                     
2     2 Pride and Prejudice                                         
3     3 To Kill a Mockingbird                                       
4     4 Harry Potter and the Order of the Phoenix (Harry Potter, #5)
5     5 The Book Thief                                              
  author          avg_rating  ratings url                                       
  <chr>                <dbl>    <dbl> <chr>                                     
1 Suzanne Collins       4.36 10338532 https://www.goodreads.com/book/show/27670…
2 Jane Austen           4.3   5042082 https://www.goodreads.com/book/show/1885.…
3 Harper Lee            4.26  7157438 https://www.goodreads.com/book/show/2657.…
4 J.K. Rowling          4.5   3954056 https://www.goodreads.com/book/show/2.Har…
5 Markus Zusak          4.39  2994493 https://www.goodreads.com/book/show/19063…
# ℹ 95 more rows
rows: 100

A saved copy lets you iterate without sending a request on every run. read_html() also takes a URL, so read_html("https://www.goodreads.com/list/show/1.Best_Books_Ever") parses the live list.

Open the page in Chrome or Firefox, right-click the data you want, and choose Inspect. Look for a class or tag that repeats once per record, test it in the DevTools console with document.querySelectorAll(), then paste the same selector into html_elements(). Prefer stable class names over long generated paths copied from DevTools.

On the Goodreads list, inspecting a book title shows <a class="bookTitle"> inside a <tr> of table.tableList. Typing document.querySelectorAll("table.tableList tr").length in the console returns 100, one per book, which confirms it's the right repeating element.

Six selector patterns appear in this guide, and they cover most scraping work:

• Tag.class: a.bookTitle matches <a> elements with the class bookTitle.
• Descendant: table.tableList tr matches rows anywhere inside that table.
• Class only: .minirating matches any element with that class (we qualify it as span.minirating to be explicit).
• ID: script#__NEXT_DATA__ matches the one element with that id.
• Attribute value: script[type="application/ld+json"] matches scripts of one type.
• Attribute on a control: button[aria-label="Book details and editions"] finds a button by its accessible name, which tends to outlive styling changes.

Avoid the long paths that DevTools' "Copy selector" produces, such as #all_votes > table > tbody > tr:nth-child(1) > td:nth-child(3). They break as soon as the layout shifts. Generated class names with random suffixes are fragile for the same reason.

For pointing and clicking, the SelectorGadget browser extension suggests a selector when you click the elements you want. Check its suggestion in the console first.

rvest also accepts XPath: html_elements(page, xpath = "//a[@class='bookTitle']") does the same job as the CSS version. See when XPath beats CSS for the cases where it's the better tool, and keep the full selector reference open while you work.

Tip: If a selector works in DevTools but returns zero elements in R, the data may not be in the HTML the server sends. DevTools shows the page after JavaScript runs, and read_html() sees it before.

Select the table with html_element() and pass it to html_table(), which returns a tibble with one column per table column. Set header = FALSE when the table has no header row, then rename the columns. Clean numbers afterward with readr::parse_number(), because html_table() keeps currency symbols and text exactly as shown.

The Goodreads list is a real HTML table, so html_table() reads it in one call:

# 03_table.R: read the same list with html_table()
library(rvest)
page <- read_html("fixtures/list-page1.html")
raw  <- page |> html_element("table.tableList") |> html_table(header = FALSE)
dim(raw)
raw$X3 <- trimws(gsub("\\s+", " ", raw$X3))   # collapse the cell's line breaks
print(raw[, c("X1", "X3")], n = 3, width = 100)

Output:

[1] 100   4
# A tibble: 100 × 2
     X1
  <int>
1     1
2     2
3     3
  X3                                                                            
  <chr>                                                                         
1 The Hunger Games (The Hunger Games, #1) by Suzanne Collins 4.36 avg rating — …
2 Pride and Prejudice by Jane Austen 4.30 avg rating — 5,042,082 ratings score:…
3 To Kill a Mockingbird by Harper Lee 4.26 avg rating — 7,157,438 ratings score…

You get 100 rows and four columns. The first column is the rank, already typed as an integer. The third column shows the limit: Goodreads packs the title, author, rating, and vote score into a single cell, so html_table() gives you all of it as one string.

html_table() is the fastest option when each cell holds one value, as in a price list or a statistics table. When a cell packs several fields, select inside each row instead, as parse_books() does, to get clean typed columns.

Calling html_table() on a whole page returns a list of every table, so select the one you want first.

Scrape the first page, read the next-page link, resolve it with xml2::url_absolute(), and repeat until the link disappears or you hit a page limit. Fetch through httr2 with req_throttle() to cap your request rate and req_retry() to retry 429 and 503 responses, which also honors the Retry-After header.

read_html(url) is fine for one page. For a crawl, fetch with httr2 instead, because you control the user agent, timeout, request rate, and retries in one pipeline:

# 02_paginate.R: polite live crawl with httr2 retries, throttling, and next-page links
library(httr2)
source("parse_books.R")

fetch_html <- function(url) {
  request(url) |>
    req_user_agent("r-scraping-tutorial/1.0 (you@example.com)") |>
    req_throttle(capacity = 1, fill_time_s = 2) |>   # at most ~1 request every 2 seconds
    req_retry(max_tries = 3) |>                      # retries 429 and 503, honors Retry-After
    req_timeout(30) |>
    req_perform() |>
    resp_body_html()
}

scrape_list <- function(start_url, max_pages = 3) {
  url <- start_url
  out <- list()
  for (i in seq_len(max_pages)) {
    page     <- fetch_html(url)
    out[[i]] <- parse_books(page, url)
    next_href <- html_attr(html_element(page, "a.next_page"), "href")
    if (is.na(next_href)) break
    url <- xml2::url_absolute(next_href, url)
  }
  dplyr::bind_rows(out, .id = "page")
}

books <- scrape_list("https://www.goodreads.com/list/show/1.Best_Books_Ever", max_pages = 3)
print(dplyr::count(books, page))
print(range(books$rank))

Output:

# A tibble: 3 × 2
  page      n
  <chr> <int>
1 1       100
2 2       100
3 3       100
[1]   1 300

What each part of fetch_html() does:

• req_user_agent() identifies your scraper and gives the site a way to contact you. Put a real address there.
• req_throttle(capacity = 1, fill_time_s = 2) is a token bucket that holds one request and refills every two seconds, so the crawl sends at most about one request every two seconds to a host, however fast the loop runs.
• req_retry() retries transient failures. By default, those are HTTP 429 Too Many Requests and 503 Service Unavailable, and when the server sends a Retry-After header, httr2 waits that long before trying again.
• req_timeout(30) stops a hung request from stalling the whole crawl.

scrape_list() follows the "Next" link (a.next_page) instead of guessing page numbers, resolves it against the current URL, and stops when the link disappears or it reaches max_pages. bind_rows(.id = "page") records each row's source page, and the rank range of 1 to 300 confirms no page was skipped or repeated.

Keep max_pages small while you develop. Three pages teach you as much as a hundred.

httr2's req_perform_parallel() takes a list of requests and runs several at once, with max_active capping how many are in flight (the default is 10). Build the list with lapply() over your URLs, and add req_throttle() to each request so the combined rate against a single host stays polite. Don't start one worker per CPU core against the same site. It multiplies your traffic and gets you blocked fast. For thousands of URLs, the network is the bottleneck, not R: parsing a page takes milliseconds. Spread parallel work across different hosts, not across one.

Fetch the raw HTML with read_html() and check whether your selector finds anything. If it returns zero elements but the browser shows the data, JavaScript builds that content. Before reaching for a browser, look inside script tags: many sites ship the records as JSON that jsonlite can parse directly.

The 1984 page on Goodreads shows a "Book details and editions" button. Click it in a browser and a panel opens with the edition's format, publisher, and language. In the raw HTML, that panel doesn't exist yet. But the data behind it does, inside two script tags:

# 04_js_check.R: is the data in the HTML, or built by JavaScript?
library(rvest)
page <- read_html("fixtures/book-1984.html")
cat("edition details in raw HTML:", length(html_elements(page, "div.EditionDetails")), "\n")

# 1. JSON-LD: the structured summary many sites publish for search engines
ld <- page |> html_element('script[type="application/ld+json"]') |> html_text2() |> jsonlite::fromJSON()
book <- tibble::tibble(
  title      = ld$name,
  author     = ld$author$name[1],
  pages      = ld$numberOfPages,
  avg_rating = ld$aggregateRating$ratingValue,
  ratings    = ld$aggregateRating$ratingCount
)
print(book)

# 2. __NEXT_DATA__: the app state the page's JavaScript renders from
next_data <- page |> html_element("script#__NEXT_DATA__") |> html_text2() |>
  jsonlite::fromJSON(simplifyVector = FALSE)
state     <- next_data$props$pageProps$apolloState
this_book <- Filter(\(x) identical(x$`__typename`, "Book") && identical(x$legacyId, 5470L), state)[[1]]
cat("publisher:", this_book$details$publisher, "| language:", this_book$details$language$name, "\n")

Output:

edition details in raw HTML: 0 
# A tibble: 1 × 5
  title author        pages avg_rating ratings
  <chr> <chr>         <int>      <dbl>   <int>
1 1984  George Orwell   328       4.21 5812589
publisher: New American Library | language: English 

The selector finds nothing, so JavaScript builds the panel, yet the page still carries the data.

JSON-LD (script[type="application/ld+json"]) is a structured summary that many sites publish for search engines. It's small, documented by schema.org, and changes less often than a page's layout, so check for it first on product, book, recipe, and event pages.

__NEXT_DATA__ is the state the page's JavaScript framework renders from. Goodreads stores records in a flat list keyed by type, so the code keeps the Book entry whose ID matches this book. It holds fields the visible HTML doesn't, such as the publisher, but its structure is internal to the site and can change without notice, so cover it with tests.

A third source is the site's own JSON endpoints: in DevTools, open Network, filter by Fetch/XHR, and reload. httr2 can call an endpoint you find there, and resp_body_json() parses the answer, as long as robots.txt allows its path.

All three are usually faster and more stable than rendering the page in a browser.

Use rvest's read_html_live(), which loads the page in headless Chrome through chromote and lets you run the usual rvest functions on the rendered DOM. It can also click, type, and scroll. Install Google Chrome first. For bigger jobs or blocked sites, render remotely with a scraper API instead of running Chrome yourself.

read_html_live() opens the 1984 page, clicks the details button, and reads the panel that appears:

# 05_live.R: render the page in headless Chrome and click a button (rvest >= 1.0.4, chromote + Chrome)
library(rvest)
options(chromote.timeout = 30)   # heavy pages can take longer than the 10-second default
page <- read_html_live("https://www.goodreads.com/book/show/5470.1984")
Sys.sleep(2)   # simple wait; see the note on waiting for a selector
page$click('button[aria-label="Book details and editions"]')
Sys.sleep(1)
items <- page$html_elements("div.EditionDetails div.DescListItem")
details <- tibble::tibble(
  field = items |> html_element("dt") |> html_text2(),
  value = items |> html_element("dd") |> html_text2()
)
print(details)
invisible(page$session$close())

Output:

# A tibble: 3 × 2
  field     value                               
  <chr>     <chr>                               
1 Format    328 pages, Mass Market Paperback    
2 Published July 1, 1950 by New American Library
3 Language  English                             

read_html_live() arrived in rvest 1.0.4 and returns a live page object. Its $html_elements() method works like the function you already know, and the result feeds straight into html_element() and html_text2(). The same object can $click() a selector, $type() into a field, and $scroll_to() a position, which covers "load more" buttons and infinite scroll. The read_html_live() reference lists every method.

Sys.sleep() is the simplest possible wait. On slow pages, use a short loop that checks length(page$html_elements("div.EditionDetails")) every half second until it's above zero or a time limit passes. options(chromote.timeout = 30) gives heavy pages longer to load. Close the session when you're done, because each one keeps a Chrome process running.

New to running a browser without a window? Read how headless browsers work.

RSelenium is the older route. It's still on CRAN and useful if your team already runs a Selenium Grid or needs Firefox. The cost is setup: Java, a Selenium server or driver, and browser and driver versions that must match.

Local Chrome works well for a handful of pages. When you need managed browser sessions with waits and actions running in the cloud, use a hosted automated browser instead.

Send the target URL to a scraper API from httr2 and parse the response with rvest as usual. With WebScrapingAPI's Scraper API, you call one endpoint with your api_key and the URL. Add render_js=1 for JavaScript pages. Keep the key in .Renviron, never in code. Your parsing functions don't change.

Retries and throttling solve most problems on friendly sites. These symptoms mean local fetching is the bottleneck:

• 403 or 429 responses that keep coming back after polite retries
• Challenge or consent pages where the content should be
• Content that changes by country, when you need a specific location's version
• More JavaScript pages to render than local Chrome can handle

Before you change your fetch layer, try the basics. There are other ways to reduce blocks, and four matter most:

• Slow down: raise fill_time_s in req_throttle().
• Send an honest user agent with contact details.
• Cache pages you've already fetched instead of requesting them again.
• Don't scrape what you don't need, such as images or pages outside your question.

If the symptoms persist, swap the fetch layer. WebScrapingAPI's Scraper API takes a URL and returns the page, handling proxies, retries, and optional JavaScript rendering on its side. There's no R SDK and you don't need one: one httr2 function is the whole integration.

# 06_scraper_api.R: call WebScrapingAPI's Scraper API from R with httr2
library(httr2)
library(rvest)

wsa_get <- function(url, render_js = FALSE, country = NULL) {
  key <- Sys.getenv("WSA_API_KEY")
  if (key == "") stop("Set WSA_API_KEY in your environment (for example in ~/.Renviron).")
  request("https://api.webscrapingapi.com/v2") |>
    req_url_query(
      api_key   = key,
      url       = url,                       # httr2 URL-encodes query values
      render_js = if (render_js) 1 else NULL,
      country   = country
    ) |>
    req_retry(max_tries = 3) |>
    req_timeout(120) |>
    req_perform()
}

# See exactly what you'll send, without sending it
req <- request("https://api.webscrapingapi.com/v2") |>
  req_url_query(api_key = "YOUR_KEY",
                url = "https://www.goodreads.com/list/show/1.Best_Books_Ever?page=2",
                country = "us")
req$url

Output:

[1] "https://api.webscrapingapi.com/v2?api_key=YOUR_KEY&url=https%3A%2F%2Fwww.goodreads.com%2Flist%2Fshow%2F1.Best_Books_Ever%3Fpage%3D2&country=us"

The printed URL shows why req_url_query() matters. The target URL has its own ?page=2. Pasted raw into another URL, that ? and any & would be read as the API's parameters, and the API would fetch the wrong page. httr2 encodes them as %3F and %26, so the target arrives intact. NULL arguments are dropped, so render_js and country only appear when you set them.

Keep the key out of your code. Add this line to the .Renviron file in your home directory, then restart R:

WSA_API_KEY=your_api_key_here

Sys.getenv("WSA_API_KEY") reads it, and the R manual explains how R reads .Renviron at startup. If the variable isn't set, wsa_get() stops with Error: Set WSA_API_KEY in your environment (for example in ~/.Renviron). An invalid key comes back as HTTP 401 with the body {"error":"Invalid API key"}, and httr2 turns that into an R error, so a bad key never passes silently as an empty page.

render_js = TRUE runs the page's JavaScript before the HTML comes back. country = "us" fetches the page as a visitor from that country would see it. JavaScript rendering and geotargeting cost no extra credits.

The response goes into the same parse_books() you wrote at the start:

source("parse_books.R")
source("wsa_get.R")   # the wsa_get() function from the previous snippet

target <- "https://www.goodreads.com/list/show/1.Best_Books_Ever?page=2"
page   <- wsa_get(target, country = "us") |> httr2::resp_body_html()
books  <- parse_books(page, target)
print(books, n = 5, width = 100)
cat("rows:", nrow(books), "\n")

# Ask the API to run the page's JavaScript before it returns the HTML
book_page <- wsa_get("https://www.goodreads.com/book/show/5470.1984", render_js = TRUE) |>
  httr2::resp_body_html()
html_text2(html_element(book_page, 'h1[data-testid="bookTitle"]'))

Output:

# A tibble: 100 × 6
   rank title                                                       
  <int> <chr>                                                       
1   101 The Alchemist                                               
2   102 Divergent (Divergent, #1)                                   
3   103 Fahrenheit 451                                              
4   104 The Catcher in the Rye                                      
5   105 The Handmaid's Tale                                         
  author           avg_rating ratings url                                       
  <chr>                 <dbl>   <dbl> <chr>                                     
1 Paulo Coelho           3.91 3382145 https://www.goodreads.com/book/show/18144…
2 Veronica Roth          4.13 3905122 https://www.goodreads.com/book/show/13335…
3 Ray Bradbury           3.97 2488011 https://www.goodreads.com/book/show/13079…
4 J.D. Salinger          3.8  3600144 https://www.goodreads.com/book/show/5107.…
5 Margaret Atwood        4.13 2145887 https://www.goodreads.com/book/show/38447…
# ℹ 95 more rows
rows: 100
[1] "1984"

The parser didn't change, only the fetch line did, so every test you write for parse_books() still covers this path.

Use rvest alone for static pages and practice sites. Add httr2 when you crawl many pages and need rate limits and retries. Parse script-tag JSON or the site's own JSON endpoint when JavaScript loads the data. Use read_html_live() for a few interactive pages, and a scraper API when blocks or volume make local fetching unreliable.

When you're not sure, work down this list and stop at the first yes:

1. Is the data in the raw HTML? Use rvest.
2. Do you need more than a few pages? Fetch with httr2 and parse with rvest.
3. Is the data in a script tag or a JSON endpoint? Parse it with jsonlite.
4. Does the page need a browser, and do you have only a few pages? Use read_html_live().
5. Are you blocked, do you need a specific country, or are you rendering at volume? Use a scraper API.

Save a copy of each page type as an HTML file and write testthat tests that parse it offline, so a site redesign fails a test instead of silently returning NA. Then treat the scraped tibble like any other data: summarize it with dplyr, write it with readr::write_csv(), and record when you collected it.

Scrapers rarely crash when a site changes. They return empty columns. Tests against saved pages catch that. You already saved the fixtures with download.file(). If you fetch with httr2, writeLines(resp_body_string(resp), "fixtures/page.html") does the same. Commit them to the project so the tests run the same way on every machine.

This testthat file checks what matters about the list parser:

# tests/test-parse.R: offline tests against saved HTML, run with testthat::test_file()
library(testthat)
source("../parse_books.R")

test_that("list page parses into 100 clean rows", {
  books <- parse_books(read_html("../fixtures/list-page1.html"),
                       "https://www.goodreads.com/list/show/1.Best_Books_Ever")
  expect_equal(nrow(books), 100)
  expect_false(anyNA(books$avg_rating))
  expect_true(all(books$avg_rating >= 1 & books$avg_rating <= 5))
  expect_true(all(books$ratings > 0))
  expect_match(books$url, "^https://www\\.goodreads\\.com/book/show/")
})

Run it from the tests/ folder with testthat::test_file("test-parse.R"):

[ FAIL 0 | WARN 0 | SKIP 0 | PASS 5 ]

When the site changes, save a fresh copy of the page and rerun the test. A failure tells you which field broke before bad data reaches your analysis. Apply the same "fail loudly" rule in production: if a selector returns zero rows, stop the run with an error instead of writing an empty file.

Do books with more ratings also rate higher? Two saved pages give you 200 books to look at:

# 07_analysis.R: from scraped rows to an answer
library(dplyr, warn.conflicts = FALSE)
source("parse_books.R")
list_url <- "https://www.goodreads.com/list/show/1.Best_Books_Ever"
books <- bind_rows(
  parse_books(read_html("fixtures/list-page1.html"), list_url),
  parse_books(read_html("fixtures/list-page2.html"), paste0(list_url, "?page=2"))
)
books |>
  mutate(popularity = cut(ratings, c(0, 1e6, 3e6, Inf),
                          labels = c("under 1M", "1M to 3M", "over 3M"))) |>
  group_by(popularity) |>
  summarize(books = n(), median_rating = median(avg_rating), .groups = "drop") |>
  print()
books |> count(author, sort = TRUE) |> head(3) |> print()
readr::write_csv(books, "books.csv")
cat("wrote books.csv with", nrow(books), "rows\n")

Output:

# A tibble: 3 × 3
  popularity books median_rating
  <fct>      <int>         <dbl>
1 under 1M      84          4.1 
2 1M to 3M      79          4.14
3 over 3M       37          4.3 
# A tibble: 3 × 2
  author           n
  <chr>        <int>
1 J.K. Rowling    10
2 Rick Riordan     5
3 Riley Sager      4
wrote books.csv with 200 rows

In this sample, the 37 books with more than three million ratings have a median rating of 4.3, against 4.1 for books with fewer than one million. That's a pattern in one reader-voted list, not a law of publishing. From here, the data is ordinary R data: a ggplot2 scatter of ratings against avg_rating on a log scale is the obvious next chart.

Because the list changes daily, add a scraped_at = Sys.time() column before you save. When someone reruns the analysis next month and gets different numbers, the timestamp explains why.

Python has more scraping libraries and browser tooling, so it suits large crawlers and production services. R is the better fit when the scraped data feeds analysis, reports, or models in R, because rvest returns tibbles and httr2 covers polite fetching. For most analysts, staying in one language beats switching for slightly faster scraping. If the data ends up in an R analysis, a Quarto report, or a model, scraping in R removes a handoff: no export, and no second environment to keep in sync.

The fetch layer is language-neutral, which makes mixing easy. An HTTP client or a scraper API returns the same HTML to R and Python alike, so a team can crawl in one language and analyze in the other. For the Python side, see our guide to web scraping with Python.

Read the site's terms and robots.txt before you crawl, keep request rates low, identify your scraper with a user agent, and collect only the fields you need. Avoid personal data unless you have a lawful basis, and never scrape behind a login without permission. When in doubt, ask the site owner or use their API.

robots.txt is how a site tells crawlers which paths it doesn't want them to visit. The robots.txt standard (RFC 9309) defines the rules: find the group for your user agent, and the most specific matching rule wins. This function applies them to a list of paths:

# 08_robots.R: check paths against a site's robots.txt before you crawl (RFC 9309)
library(httr2)

robots_allows <- function(site, paths, agent = "*") {
  resp <- request(site) |> req_url_path("/robots.txt") |>
    req_error(is_error = \(resp) FALSE) |> req_perform()
  if (resp_status(resp) == 404) return(rep(TRUE, length(paths)))   # no robots.txt, no rules
  lines <- trimws(sub("#.*", "", strsplit(resp_body_string(resp), "\n")[[1]]))
  lines <- lines[lines != ""]
  field <- tolower(trimws(sub(":.*", "", lines)))
  value <- trimws(sub("^[^:]*:", "", lines))

  # Keep the Allow/Disallow rules from groups whose User-agent line matches `agent`
  keep <- logical(length(lines)); in_group <- FALSE
  for (i in seq_along(lines)) {
    if (field[i] == "user-agent") {
      if (i == 1 || field[i - 1] != "user-agent") in_group <- FALSE
      in_group <- in_group || value[i] == agent
    } else {
      keep[i] <- in_group && field[i] %in% c("allow", "disallow") && value[i] != ""
    }
  }
  type <- field[keep]; rule <- value[keep]
  pattern <- paste0("^", gsub("*", ".*", gsub("([.?+^$()|{}\\[\\]\\\\])", "\\\\\\1", rule, perl = TRUE), fixed = TRUE))

  vapply(paths, function(path) {
    hits <- which(vapply(pattern, grepl, logical(1), x = path))
    if (length(hits) == 0) return(TRUE)
    best <- hits[nchar(rule[hits]) == max(nchar(rule[hits]))]   # longest match wins
    any(type[best] == "allow")                                  # a tie goes to Allow
  }, logical(1))
}

paths <- c("/list/show/1.Best_Books_Ever", "/book/show/5470.1984", "/book/reviews/5470",
           "/review/list/1", "/search?q=orwell", "/work/quotes/153313")
data.frame(path = paths, allowed = unname(robots_allows("https://www.goodreads.com", paths)))

Output:

                          path allowed
1 /list/show/1.Best_Books_Ever    TRUE
2         /book/show/5470.1984    TRUE
3           /book/reviews/5470   FALSE
4               /review/list/1   FALSE
5             /search?q=orwell   FALSE
6          /work/quotes/153313    TRUE

The list and book pages this guide uses are allowed. Review listings and search are not, which is why this guide works with book-level ratings and doesn't collect individual reviews. The last row shows the longest-match rule at work: Goodreads disallows /work but allows the more specific /work/quotes.

robots.txt is a crawler convention, not a legal permission, and it isn't the only document that counts. A site's terms of use can restrict automated collection even on paths robots.txt allows. Goodreads' terms, for example, exclude data mining and automated extraction tools from the license they grant, so read them before you run these examples and ask Goodreads for permission before you collect its data for anything beyond learning.

Reviews also show why personal data needs care: each one carries a reader's name and opinions. For more habits, see these practical scraping best practices and our responsible web data principles.

Yes. rvest is part of the tidyverse, and version 1.0.5 is on CRAN as of October 2026. Version 1.0.4 added read_html_live() for JavaScript pages. For fetching, the same team now maintains httr2, which handles retries, rate limits, and parallel requests, so most modern R scrapers pair the two packages.

rvest parses HTML: it selects elements with CSS or XPath and extracts text, attributes, and tables. httr2 makes HTTP requests: it sets headers, retries failures, throttles your rate, and handles JSON APIs. You can scrape simple pages with rvest alone, but for crawls you fetch with httr2 and parse with rvest.

Plain read_html() can't, because it only sees the HTML the server sends. read_html_live() can: it runs the page in headless Chrome through chromote and gives you the rendered DOM to use with normal rvest functions. Check first for JSON inside script tags or a JSON endpoint, which is usually faster.

RSelenium is still on CRAN, but it needs Java, a Selenium server or driver, and matching browser versions, which makes setup fragile. Use it when your team already runs Selenium or needs Firefox. For most R users, read_html_live() in rvest covers rendering, clicking, and typing with far less setup.

Load the page with read_html(), select the table with html_element() and a CSS selector such as "table.data", then call html_table(). You get a tibble with one column per table column. Use header = FALSE for tables without headers, and clean numbers with readr::parse_number() afterward.

Slow down with httr2's req_throttle(), retry 429 and 503 responses with req_retry(), set an honest user agent, and cache pages so you don't refetch them. If a site still returns 403 errors or challenge pages, route the requests through a scraper API that manages proxies and rendering, and keep your rvest parser unchanged.

Put the key in your user-level .Renviron file as a line like WSA_API_KEY=your_key, restart R, and read it with Sys.getenv("WSA_API_KEY"). Never paste keys into scripts, notebooks, or URLs you share, and add any project-level .Renviron file to .gitignore so the key never reaches version control.

Technically yes: rvest's session() can submit forms and keep cookies, and httr2 can send authenticated requests. Only do it with the site owner's permission and within its terms, because logged-in areas usually hold personal or contracted data. If the site offers an official API, use that instead of scraping the logged-in pages.

Build a list of httr2 requests and send them with req_perform_parallel(), using max_active to cap how many run at once. Add req_throttle() so one host never gets more traffic than it can handle. Keep concurrency modest for any single site, because speed rarely matters as much as not getting blocked.

Scraping publicly available data is often lawful, but it depends on the site's terms, the kind of data, and local law, especially for personal data and copyrighted content. The language doesn't change that. Respect robots.txt, avoid personal data without a lawful basis, and get legal advice for commercial projects.

Your rvest parsers don't need to change when a site starts blocking you or needs JavaScript. Point the fetch at WebScrapingAPI's Scraper API and keep parsing in R. JavaScript rendering and geotargeting cost no extra credits, and failed requests aren't billed. Compare Scraper API plans, starting with a free trial of 5,000 requests.