Elixir Web Scraping: Step-by-Step Guide with Working Code

Yes, Elixir suits scrapers that fetch many pages at once. The BEAM virtual machine runs thousands of lightweight processes, so you can fetch pages in parallel, put a timeout on each one, and let a failed page crash without taking down the run. Python has more scraping libraries and more mature browser tooling.

Choose Elixir when:

• The scraper fetches hundreds or thousands of pages and you want bounded concurrency without threads or callbacks.
• It runs for a long time or on a schedule, and you want failures isolated per page.
• You already run Elixir or Phoenix and want the scraper in the same codebase.

Choose Python when:

• You're prototyping, exploring a site, or writing a one-off script. Our guide to web scraping with Python covers that path.
• You need heavy browser automation or a library with no Elixir equivalent.

Elixir web scraping needs an HTTP client and an HTML parser. Req and Floki cover both for most projects, and the standard library handles concurrency, URLs, and JSON.

Avoid hound in new code. Its latest release, 1.1.1, dates from 2020.

Setup takes four steps: install Elixir, check it, create a project, and add the dependencies.

1. Install Elixir. The code in this guide uses the built-in JSON module, so you need Elixir 1.18 or newer.
2. Check the version:

elixir --version

3. Create a new Mix project:

mix new elixir_scraper
cd elixir_scraper

4. Open mix.exs, set the Elixir requirement, and add Req, Floki, and NimbleCSV to the dependencies:

def project do
  [
    app: :elixir_scraper,
    version: "0.1.0",
    elixir: "~> 1.18",
    start_permanent: Mix.env() == :prod,
    deps: deps()
  ]
end

defp deps do
  [
    {:req, "~> 0.7.5"},
    {:floki, "~> 0.38.5"},
    {:nimble_csv, "~> 1.3"}
  ]
end

Then fetch them:

mix deps.get

~> 0.7.5 accepts bug-fix releases but not 0.8, so minor upgrades stay deliberate. Run mix hex.audit now and then. It lists retired packages and packages with known security advisories anywhere in your dependency tree.

Req sends a request in one call. Start an interactive shell inside the project with iex -S mix and try it:

iex> Req.get!("https://books.toscrape.com/").status
200

For a real scraper, build one client with Req.new/1 and reuse it for every request. Replace the contents of lib/elixir_scraper.ex with this:

defmodule ElixirScraper do
  def client do
    Req.new(
      headers: [user_agent: "elixir-scraper-tutorial/1.0 (+mailto:you@example.com)"],
      receive_timeout: 30_000,
      max_retries: 3
    )
  end

  def fetch!(req, url) do
    case Req.get(req, url: url) do
      {:ok, %Req.Response{status: 200, body: body}} -> body
      {:ok, %Req.Response{status: status}} -> raise "GET #{url} returned HTTP #{status}"
      {:error, exception} -> raise exception
    end
  end
end

What each option does:

• The User-Agent names your scraper and gives a contact address. That's good manners on a practice site. On real sites, you'll often need to set realistic request headers as well.
• receive_timeout: 30_000 gives up on a response after 30 seconds instead of waiting forever.
• max_retries: 3 caps retries (see the retry section).
• fetch!/2 pattern-matches on status 200. Any other status raises with the URL in the message, and transport errors such as timeouts are raised again. Without it, a 404 page would flow into your parser.

The Req documentation covers headers, redirects, cookies, and every other option.

Floki turns an HTML string into a document you can query with CSS selectors. You parse once with Floki.parse_document!/1, then call Floki.find/2 to get matching elements, Floki.attribute/2 to read attributes, and Floki.text/1 to read text. Save this as scripts/first_look.exs:

html = ElixirScraper.fetch!(ElixirScraper.client(), "https://books.toscrape.com/")
doc = Floki.parse_document!(html)
cards = Floki.find(doc, "article.product_pod")
IO.puts("books on this page: #{length(cards)}")

first = hd(cards)
[title] = first |> Floki.find("h3 a") |> Floki.attribute("title")
price = first |> Floki.find("p.price_color") |> Floki.text()
IO.puts("first book: #{title} (#{price})")

Run it with mix run scripts/first_look.exs:

books on this page: 20
first book: A Light in the Attic (£51.77)

Each book on the page sits in an article.product_pod card, and calling Floki.find/2 on a card searches only inside that card. Floki.attribute/2 always returns a list, because a selector can match several elements, so the [title] = match also checks that there's exactly one. Link text truncates long titles, so we read the title attribute. If you need a refresher on CSS selector syntax, our CSS selectors cheat sheet covers the patterns scrapers use most.

Since Floki 0.38, its functions no longer accept raw HTML strings. If you pass a response body straight to Floki.find/2, you get an empty list back, with no error and no warning:

html = ElixirScraper.fetch!(ElixirScraper.client(), "https://books.toscrape.com/")
IO.inspect(Floki.find(html, "article.product_pod"), label: "raw string")

IO.inspect(html |> Floki.parse_document!() |> Floki.find("article.product_pod") |> length(),
  label: "parsed"
)

raw string: []
parsed: 20

That looks exactly like a broken selector. Parse first, with Floki.parse_document!/1 for full pages or Floki.parse_fragment!/1 for snippets, and pass the parsed tree to every other function. Many older tutorials and AI answers still pass strings. The Floki changelog lists the change and the other breaking updates in the 0.38 line.

Turn each card into a map. For every book, we keep the title, the price as a number, the star rating from 1 to 5, the stock status, and an absolute URL. Add these functions inside the ElixirScraper module:

  @ratings %{"One" => 1, "Two" => 2, "Three" => 3, "Four" => 4, "Five" => 5}

  def parse_books(html, page_url) do
    doc = Floki.parse_document!(html)

    books =
      doc
      |> Floki.find("article.product_pod")
      |> Enum.map(&parse_book(&1, page_url))

    next =
      case Floki.attribute(doc, "li.next a", "href") do
        [href] -> absolute_url(page_url, href)
        [] -> nil
      end

    {books, next}
  end

  defp parse_book(card, page_url) do
    link = Floki.find(card, "h3 a")
    [title] = Floki.attribute(link, "title")
    [href] = Floki.attribute(link, "href")

    price =
      card
      |> Floki.find("p.price_color")
      |> Floki.text()
      |> String.replace("£", "")
      |> String.to_float()

    [class] = card |> Floki.find("p.star-rating") |> Floki.attribute("class")
    rating = class |> String.split() |> Enum.find_value(&Map.get(@ratings, &1))

    in_stock =
      card
      |> Floki.find("p.instock.availability")
      |> Floki.text()
      |> String.contains?("In stock")

    %{
      title: title,
      price: price,
      rating: rating,
      in_stock: in_stock,
      url: absolute_url(page_url, href)
    }
  end

  defp absolute_url(page_url, href), do: page_url |> URI.merge(href) |> URI.to_string()

Here's what the code does:

• The price becomes a float. We strip the £ sign and call String.to_float/1.
• The rating comes from a class name. The site marks ratings as class="star-rating Three", so we split the class string and look each word up in the @ratings map.
• Relative links become absolute with URI.merge/2. The href values on the page are relative, such as catalogue/a-light-in-the-attic_1000/index.html. Merging them with the page URL resolves them the way a browser would, including ../ segments on deeper pages.
• The function also returns the next page URL, or nil on the last page, for the pagination step.

Print the first two books as JSON with a short script:

url = "https://books.toscrape.com/"

{books, next} =
  ElixirScraper.client() |> ElixirScraper.fetch!(url) |> ElixirScraper.parse_books(url)

IO.puts(JSON.encode!(Enum.take(books, 2)))
IO.puts("next page: #{next}")

[{"title":"A Light in the Attic","url":"https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html","in_stock":true,"price":51.77,"rating":3},{"title":"Tipping the Velvet","url":"https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html","in_stock":true,"price":53.74,"rating":1}]
next page: https://books.toscrape.com/catalogue/page-2.html

Maps don't keep key order, which is why url shows up before price in the JSON. If a card is missing a field, the [title] = style matches raise a MatchError, which tells you right away that the markup changed.

To scrape every page, follow the "next" link until it disappears. A recursive function fetches a page, parses its books, waits half a second, and calls itself with the next URL. It stops when parse_books/2 returns nil for the next link, or when it reaches a page cap you set as a safety net. Add this to the module:

  @base_url "https://books.toscrape.com/"

  def crawl(req, start_url \\ @base_url, max_pages \\ 60) do
    do_crawl(req, start_url, max_pages, [])
  end

  defp do_crawl(_req, nil, _pages_left, acc), do: finish(acc)
  defp do_crawl(_req, _url, 0, acc), do: finish(acc)

  defp do_crawl(req, url, pages_left, acc) do
    {books, next} = req |> fetch!(url) |> parse_books(url)
    if next, do: Process.sleep(500)
    do_crawl(req, next, pages_left - 1, [books | acc])
  end

  defp finish(acc), do: acc |> Enum.reverse() |> List.flatten()

The two short do_crawl/4 clauses are the stop conditions, and pattern matching picks the right one. Each page's books are prepended to the accumulator with [books | acc], which is cheap. Appending with acc ++ books would copy the whole list on every page. finish/1 reverses the list once at the end to restore page order and flattens it into one list of maps.

Run ElixirScraper.crawl(ElixirScraper.client()) and it walks all 50 pages of the catalog and returns 1,000 books. The half-second pause spaces out requests, and the page cap guards against a "next" link that loops.

This is still scraping, not crawling: you follow one known chain of pages instead of discovering every link on a site. If you're not sure which one you need, read our explainer on scraping vs crawling.

When you already know the page URLs, fetch several at once with Task.async_stream/3. It runs a function over a list in separate processes, keeps at most max_concurrency of them running, and returns results in the original order. The catalog's page URLs follow a pattern, so we can build all 50 up front:

  def crawl_concurrent(req, pages \\ 1..50, max_concurrency \\ 4) do
    pages
    |> Enum.map(&"#{@base_url}catalogue/page-#{&1}.html")
    |> Task.async_stream(
      fn url -> req |> fetch!(url) |> parse_books(url) |> elem(0) end,
      max_concurrency: max_concurrency,
      timeout: 60_000,
      on_timeout: :kill_task
    )
    |> Enum.flat_map(fn
      {:ok, books} ->
        books

      {:exit, reason} ->
        IO.puts("page failed: #{inspect(reason)}")
        []
    end)
  end

Calling ElixirScraper.crawl_concurrent(ElixirScraper.client()) returns the same 1,000 books. Why it's built this way:

• Each page runs in its own process. If fetch!/2 raises on one page, only that process crashes. The other pages keep going.
• max_concurrency: 4 caps the load. Raising it fetches faster but puts more pressure on the server. Start low on any site you don't own.
• on_timeout: :kill_task keeps the batch alive. By default, a task that runs past timeout exits the calling process and stops the whole stream. With :kill_task, the slow page comes back as {:exit, :timeout} and the rest continue.
• Enum.flat_map/2 keeps the good pages and logs the failures.

The Req client is a plain struct, so every process can use the same req value, and Req's connection pool handles the sockets. The docs for Task.async_stream/3 list the other options, including ordered: false when order doesn't matter.

Req already retries failed requests, so you don't need to write a retry loop. By default, it retries GET and HEAD requests on 408, 429, 500, 502, 503, and 504 responses and on transport errors such as timeouts and closed connections. It waits longer after each failure, with a little random jitter, and stops after max_retries attempts (3 by default). On 429 and 503 responses, it waits for the time in the Retry-After header when the server sends one. A 404 or another 4xx status isn't retried, because asking again won't change the answer.

You can watch it work against a test endpoint that always returns 503:

req = ElixirScraper.client()
{:ok, res} = Req.get(req, url: "https://httpbin.org/status/503")
IO.puts("final status: #{res.status}")
{:ok, res} = Req.get(req, url: "https://books.toscrape.com/does-not-exist.html")
IO.puts("final status: #{res.status}")

[warning] retry: got response with status 503, will retry in 988ms, 3 attempts left
[warning] retry: got response with status 503, will retry in 1844ms, 2 attempts left
[warning] retry: got response with status 503, will retry in 3610ms, 1 attempt left
final status: 503
final status: 404

The 503 is retried three times with growing delays, and the 404 comes back at once. Your exact delays will differ because of the jitter.

One setting needs care. If you pass your own retry_delay, Req uses it for every retry and ignores the Retry-After header. Leave it unset unless you want to override what the server asks for. The docs for Req's retry step describe the retry, retry_delay, and retry_log_level options.

NimbleCSV writes CSV, and Elixir's built-in JSON module writes JSON, so both exports are a few lines. Add these functions to the module:

  def write_csv!(books, path) do
    header = ~w(title price rating in_stock url)

    rows =
      Enum.map(books, fn book ->
        Enum.map([book.title, book.price, book.rating, book.in_stock, book.url], &to_string/1)
      end)

    File.write!(path, NimbleCSV.RFC4180.dump_to_iodata([header | rows]))
  end

  def write_json!(books, path), do: File.write!(path, JSON.encode!(books))

Then wire the crawl and the exports together in scripts/scrape_books.exs:

books = ElixirScraper.crawl(ElixirScraper.client())
IO.puts("books: #{length(books)}")
ElixirScraper.write_csv!(books, "books.csv")
ElixirScraper.write_json!(books, "books.json")

Run it with mix run scripts/scrape_books.exs. The first lines of books.csv look like this:

title,price,rating,in_stock,url
A Light in the Attic,51.77,3,true,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet,53.74,1,true,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html

NimbleCSV.RFC4180 is the standard comma-separated format with quoting, so titles that contain commas or quotes stay in one column. The rows are built in a fixed field order, which matters because maps don't keep key order. to_string/1 turns numbers and booleans into text. The NimbleCSV docs cover custom separators, such as tabs.

Use Crawly when you want a framework: spiders, a request queue, middlewares for User-Agent and domain filtering, and item pipelines that validate, de-duplicate, and write items to CSV or JSON files. Stay with Req and Floki for small jobs, custom logic, or when you want full control of the dependency tree.

A Crawly project needs {:crawly, "~> 0.17.2"} and {:floki, "~> 0.38.5"} in mix.exs. Create config/config.exs with the settings:

import Config

config :crawly,
  concurrent_requests_per_domain: 4,
  closespider_itemcount: 1000,
  closespider_timeout: 1,
  middlewares: [
    Crawly.Middlewares.DomainFilter,
    Crawly.Middlewares.UniqueRequest,
    {Crawly.Middlewares.UserAgent,
     user_agents: ["elixir-scraper-tutorial/1.0 (+mailto:you@example.com)"]}
  ],
  pipelines: [
    {Crawly.Pipelines.Validate, fields: [:title, :price, :url]},
    {Crawly.Pipelines.DuplicatesFilter, item_id: :url},
    {Crawly.Pipelines.CSVEncoder, fields: [:title, :price, :url]},
    {Crawly.Pipelines.WriteToFile, extension: "csv", folder: "out", include_timestamp: false}
  ]

The spider, saved in lib/books_crawler/books_spider.ex, tells Crawly where to start and how to turn each response into items and new requests:

defmodule BooksCrawler.BooksSpider do
  use Crawly.Spider

  @impl Crawly.Spider
  def base_url, do: "https://books.toscrape.com/"

  @impl Crawly.Spider
  def init, do: [start_urls: ["https://books.toscrape.com/"]]

  @impl Crawly.Spider
  def parse_item(response) do
    {:ok, doc} = Floki.parse_document(response.body)

    items =
      doc
      |> Floki.find("article.product_pod")
      |> Enum.map(fn card ->
        link = Floki.find(card, "h3 a")
        [title] = Floki.attribute(link, "title")
        [href] = Floki.attribute(link, "href")
        price = card |> Floki.find("p.price_color") |> Floki.text()

        %{
          title: title,
          price: price,
          url: Crawly.Utils.build_absolute_url(href, response.request_url)
        }
      end)

    requests =
      doc
      |> Floki.attribute("li.next a", "href")
      |> Crawly.Utils.build_absolute_urls(response.request_url)
      |> Crawly.Utils.requests_from_urls()

    %Crawly.ParsedItem{items: items, requests: requests}
  end
end

Start it from iex -S mix with Crawly.Engine.start_spider(BooksCrawler.BooksSpider). The spider follows the "next" links, the pipelines validate and de-duplicate each book, and the run stops once it reaches 1,000 items, leaving one CSV file in out/.

Know the trade-offs before you adopt it:

• Releases have slowed. Crawly's latest release is 0.17.2, from July 2024.
• It pins an older HTTP client. Crawly requires httpoison ~> 2.2, so adding {:httpoison, "~> 3.0"} to the same project fails with "version solving failed".
• Audit the tree. That pin keeps an older HTTP stack in your project. At the time of writing, mix hex.audit flags advisories in Crawly's HTTP dependencies, so run it and read what it reports before you ship.
• The CSV has no header row. CSVEncoder writes only data rows, and WriteToFile adds a crawl ID to the file name even with include_timestamp: false.
• Encoders are pipelines. CSVEncoder or JSONEncoder must sit in pipelines before WriteToFile, not in middlewares.

If the job grows into collecting many related pages on a schedule, you can also hand the URL discovery and limits to a managed, bounded crawl.

Req only sees the HTML the server sends. If a page builds its content with JavaScript after it loads, that content isn't in the response, and your Floki selectors return nothing. Check whether that's the problem before you change tools, then pick the lightest fix: embedded data first, a headless browser second, and a managed API when rendering is only part of the problem.

Fetch the page with Req and count the elements you expect. quotes.toscrape.com/js/ shows 10 quotes in a browser. Add a counter to the module:

  def count_quotes(html) do
    html |> Floki.parse_document!() |> Floki.find("div.quote") |> length()
  end

Run it on the raw HTML:

div.quote in raw HTML: 0

Zero matches in the HTML and 10 on screen means JavaScript builds the quotes. You can confirm it by hand: View Source shows what the server sent, while the DevTools Elements panel shows the page after scripts ran.

Many JavaScript pages ship their data inside the HTML, in an inline <script> that the page's code reads. quotes.toscrape.com/js/ does this with var data = [...]. Capture that array with a regular expression and decode it with JSON.decode!/1, with no browser involved:

  def embedded_quotes(html) do
    [_, json] = Regex.run(~r/var data = (\[.*?\]);\s*for \(var i/s, html)

    json
    |> JSON.decode!()
    |> Enum.map(fn q -> %{text: q["text"], author: q["author"]["name"], tags: q["tags"]} end)
  end

Call both functions on the same response:

html = ElixirScraper.fetch!(ElixirScraper.client(), "https://quotes.toscrape.com/js/")
IO.puts("div.quote in raw HTML: #{ElixirScraper.count_quotes(html)}")
quotes = ElixirScraper.embedded_quotes(html)
IO.puts("quotes from the inline script: #{length(quotes)}")
IO.inspect(hd(quotes))

div.quote in raw HTML: 0
quotes from the inline script: 10
%{
  author: "Albert Einstein",
  text: "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”",
  tags: ["change", "deep-thoughts", "thinking", "world"]
}

That's all 10 quotes, structured, from one HTTP request. The regex anchors on the code after the array, so it stops at the right ];. Patterns like this break when the script changes, so keep them specific.

The same idea works for data that loads after the page opens. Open the DevTools Network tab, filter by Fetch/XHR, and reload. If the page calls a JSON endpoint, request it with Req directly. Req decodes JSON into maps, which beats parsing HTML.

If the data only exists after rendering, let a real browser run the scripts. Chrome's headless mode can print the rendered DOM with --dump-dom, and Elixir can run it with System.cmd/2:

  def render_with_chrome(url, chrome \\ "google-chrome") do
    {html, 0} = System.cmd(chrome, ["--headless=new", "--dump-dom", url])
    html
  end

html = ElixirScraper.render_with_chrome("https://quotes.toscrape.com/js/")
IO.puts("div.quote after rendering: #{ElixirScraper.count_quotes(html)}")

div.quote after rendering: 10

The {html, 0} match makes the call fail if Chrome exits with an error, and the rendered HTML goes through the same Floki code as before. Chrome or Chromium must be installed on the machine that runs the scraper. Pass "chromium" or a full path as the second argument if your binary has a different name. Each call starts a whole browser, so keep concurrency low. Chrome headless mode documents the other flags, such as --screenshot and --window-size.

When you need to click, scroll, or wait for a specific element, --dump-dom isn't enough. playwright_ex is an Elixir client for the Playwright Node.js driver, so it needs Node.js and Playwright installed next to your app. Wallaby drives browsers through ChromeDriver or Selenium, but it's built for testing web apps rather than scraping. Crawly can send its fetches through a browser-rendering service. This answers the "Elixir vs Selenium" question too: Selenium is a browser driver, not a language, and Elixir reaches it through WebDriver libraries like Wallaby. All of these mean running and updating browsers yourself. If you need rendering, waits, and screenshots, you can render pages without running browsers by sending the work to a hosted browser runtime.

The scraper you just built works because books.toscrape.com and quotes.toscrape.com are practice sites that welcome automated requests. Many production sites don't. The same code that returns 1,000 clean books can start failing with 403 responses, CAPTCHA pages, per-IP rate limits, browser fingerprint checks, or content that changes by country. BEAM concurrency doesn't help here, because the limits are about who's asking, not how many requests you can run.

Each DIY fix adds infrastructure to run: proxy pools, browser fleets, retry queues, and monitoring. Our guide on how to avoid getting blocked goes deeper on headers, pacing, and proxies. At some point, it makes more sense to hand the fetching to a service and keep your Elixir code for parsing and storage.

Scraper API fetches the page for you, handles proxies and retries, and can render JavaScript, while your Elixir code keeps the same Req and Floki steps. You send one GET request with your API key and the target URL, then add options such as render_js or country only when the page needs them.

This function renders the JavaScript quotes page through the API from a US location, reading the key from an environment variable:

  def scraper_api(target_url, country \\ "us") do
    api_key = System.fetch_env!("WSA_API_KEY")

    Req.get("https://api.webscrapingapi.com/v2",
      params: [api_key: api_key, url: target_url, render_js: 1, country: country],
      receive_timeout: 120_000,
      retry: false
    )
  end

Call it and check the status before you parse:

case ElixirScraper.scraper_api("https://quotes.toscrape.com/js/") do
  {:ok, %Req.Response{status: 200, body: html}} ->
    IO.puts("quotes via Scraper API: #{ElixirScraper.count_quotes(html)}")

  {:ok, %Req.Response{status: status, body: body}} ->
    IO.puts("Scraper API returned #{status}: #{inspect(body)}")

  {:error, exception} ->
    IO.puts("request failed: #{Exception.message(exception)}")
end

Save it as scripts/scraper_api.exs, set the key in your shell, and run it:

export WSA_API_KEY="your_api_key"
mix run scripts/scraper_api.exs

Why the code looks like this:

• params: encodes the target URL for you. Req builds the query string, so the ?, /, and & characters in the target URL can't break the request.
• System.fetch_env!/1 fails fast. If the variable isn't set, it raises System.EnvError before any request goes out.
• The timeout is longer than before. Rendering a page takes longer than a plain fetch, so 120 seconds leaves room for the API to load and render it.
• retry: false avoids duplicate calls. The API already retries on its side, so client retries would only repeat the same job.
• The error branch shows the body. Req decodes JSON error bodies into maps, so an invalid key prints Scraper API returned 401: %{"error" => "Invalid API key"}.

Keep the API key on the server side. Don't ship it in client code, commit it to a repository, or print it in logs.

Most Elixir scraping errors come from library changes, dependency conflicts, or pages that don't contain what you expect. Here's each symptom and its fix.

When an error isn't in the table, print the status code and the first few hundred bytes of the response body. Most scraping bugs become obvious once you see what the server sent.

Scrape in a way the site owner can live with. Read the site's robots.txt, which follows the Robots Exclusion Protocol (RFC 9309), and skip the paths it disallows for your user agent. Keep max_concurrency low, add delays, and let Req honor Retry-After when the server asks you to slow down. Identify your client with a User-Agent that includes a contact address. Follow the site's terms of service, and avoid collecting personal data unless you have a clear legal basis for it. For more on how we approach this, see our responsible web data practices.

Yes, especially for scrapers that fetch many pages at once. The BEAM runs lightweight processes, so Task.async_stream gives you bounded concurrency, timeouts, and isolated failures in a few lines. Python has more scraping libraries and browser tooling, so it's often quicker for one-off scripts and heavy browser automation.

Most projects pair Req for HTTP requests with Floki for parsing HTML through CSS selectors. Add NimbleCSV for spreadsheets, the built-in JSON module for JSON output, and Task.async_stream for concurrency. HTTPoison is a mature alternative client, and Crawly is a full framework when you need spiders and item pipelines.

Since Floki 0.38, functions no longer accept raw HTML strings, so passing a response body straight to Floki.find quietly returns nothing. Parse it first with Floki.parse_document!. If you still get an empty list, the selector may not match the markup, or the content is rendered by JavaScript and isn't in the server's HTML.

First confirm the data is missing from the raw HTML. Then look for it in an inline script or a JSON request the page makes, and decode that with JSON.decode!. If the data only exists after rendering, run headless Chrome from System.cmd, use a browser library such as playwright_ex, or call a scraping API that renders JavaScript.

Use Req and Floki when you control the whole flow: a few sites, custom logic, and your own concurrency. Use Crawly when you want a framework with spiders, request scheduling, middlewares, and item pipelines. Crawly's latest release still depends on HTTPoison 2, so check its dependency tree with mix hex.audit before adopting it.

Yes, through WebDriver libraries. Wallaby drives browsers through ChromeDriver or Selenium, though it's built for testing web apps rather than scraping. playwright_ex is an Elixir client for the Playwright Node.js driver. For simple pages, running headless Chrome with --dump-dom from System.cmd is often enough to get the rendered HTML.

Send a clear User-Agent, limit concurrency, add delays between pages, and let Req retry 429 and 503 responses, which honors Retry-After by default. Respect robots.txt and the site's terms. Sites with CAPTCHAs, fingerprinting, or per-IP limits may still block you, which is where rotating IPs or a managed scraping API helps.

You now have an Elixir scraper that fetches, parses, paginates, runs pages concurrently, retries politely, and exports clean data. A few next steps make it production-ready:

• Add a per-host rate limit, so every process shares one request budget instead of relying on sleeps and max_concurrency alone.
• Render only where you must. Keep Req for static pages and embedded JSON, and send only the pages that need JavaScript to headless Chrome.
• Hand off fetching when blocks start. Once you're spending more time on proxies and CAPTCHAs than on parsing, move the fetch step to a managed API and keep the rest of your Elixir code.

Point the same Req client at Scraper API and get the page back rendered, from the country you choose, without running browsers or proxies yourself. Start with 5,000 free requests. No extra credits for JavaScript rendering or geotargeting, and failed requests aren't billed.

See Scraper API pricing