Python Headless Browser Libraries For Web Scraping in 2026

A headless browser is a normal web browser, Chrome or Firefox or WebKit, with the GUI switched off. It still parses HTML, executes JavaScript, fires DOMContentLoaded, runs your single-page-app's React tree, and lets you click buttons, type into inputs, or grab screenshots. The difference is that nothing draws to a screen, which makes it cheap to run on a server, in a container, or inside a CI job.

It's worth contrasting three layers people lump together:

• HTTP clients like requests and httpx: fast, lightweight, but they don't run JavaScript. If the data you need is in the initial HTML, this is the right tool.
• HTML parsers like BeautifulSoup, Parsel, or lxml: take whatever HTML you fetched and let you query it. They don't fetch and they don't render.
• Headless browsers: full browser engines you drive from code. They render the page, execute JS, and expose a DOM you can query and interact with.

When you reach for a Python headless browser, you're paying for one specific capability: a real JavaScript runtime with a real DOM. Everything else, including memory cost and latency, follows from that choice. Foundational background on browser automation is a useful next stop if you're new to the category.

The honest answer most teams skip: you probably don't need a Python headless browser as often as you think. Spinning up Chromium for every request is the most expensive way to fetch a page, and a lot of "JavaScript-rendered" sites quietly expose the same data through a JSON endpoint that requests can hit directly.

Reach for a headless browser when:

• The data you need is injected after page load by client-side JavaScript (React, Vue, Svelte, Angular SPAs).
• The flow requires real interaction: clicking a "Load more" button, scrolling for infinite-scroll content, hovering to expose a menu, or completing a multi-step login.
• You need to capture screenshots, PDFs, or video recordings of the rendered page.
• The site fingerprints clients aggressively and rejects anything that isn't a real browser TLS handshake.

Skip the browser when the page is server-rendered HTML, when there's a documented or discoverable API behind it, or when you're hitting a sitemap of static pages. A request client plus a parser will be 10 to 50 times faster and cheaper, and infinitely easier to scale. Use the heaviest tool only where the page genuinely demands it.

Selenium has been around since 2004 and is still the broadest-compatibility option in the Python headless browser space. It speaks WebDriver to Chrome, Firefox, Edge, and Safari, has bindings in nearly every language, and benefits from two decades of Stack Overflow answers. If you're maintaining an existing test suite or working in a polyglot team, Selenium is usually the path of least resistance.

Selenium 4 simplified the install dramatically. Selenium Manager ships with the library and resolves the right driver binary automatically, so the days of manually downloading chromedriver.exe and matching versions are largely behind us. The official Selenium documentation is the canonical reference if you want to dig deeper. A minimal headless Chrome script looks like this:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

options = Options()
options.add_argument('--headless=new')
options.add_argument('--no-sandbox')

driver = webdriver.Chrome(options=options)
try:
    driver.get('https://example.com')
    title = driver.title
    heading = driver.find_element(By.TAG_NAME, 'h1').text
    driver.save_screenshot('example.png')
    print(title, heading)
finally:
    driver.quit()

Strengths: broad browser coverage, huge community, mature grid for distributed runs, best-known cross-language API. Weaknesses: synchronous-by-default API, no built-in auto-wait (you'll write a lot of WebDriverWait), and stock Selenium is trivial to fingerprint as automation.

For anti-bot work, the ecosystem fills the gap. selenium-stealth patches the most obvious navigator.webdriver and WebGL tells, and undetected-chromedriver is a drop-in replacement that has been the go-to for Cloudflare-protected targets for years. Neither is a silver bullet against modern fingerprint stacks, but both are still useful first lines of defense. If your project is specifically about Cloudflare, our dedicated Cloudflare-bypass walkthrough covers the practical patterns in more detail.

Playwright is the closest thing to a default recommendation for new Python headless browser projects in 2026. It's maintained by Microsoft, ships an officially supported Python binding (see the Playwright Python docs for the current install matrix), and exposes the same API across Chromium, Firefox, and WebKit. Communication runs over a persistent WebSocket connection rather than the chatty HTTP round-trips Selenium uses, which is a meaningful part of why it tends to feel snappier in benchmarks.

Install is two commands:

pip install playwright
playwright install

The first installs the Python package; the second downloads patched browser binaries to a Playwright-managed cache. A minimal async example that loads a page, waits for content, and grabs a full-page screenshot:

import asyncio
from playwright.async_api import async_playwright

async def run():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            user_agent='Mozilla/5.0 (X11; Linux x86_64) ...',
            locale='en-US',
        )
        page = await context.new_page()
        await page.goto('https://example.com', wait_until='networkidle')
        title = await page.title()
        await page.screenshot(path='example.png', full_page=True)
        print(title)
        await browser.close()

asyncio.run(run())

Three Playwright features are worth highlighting because they directly attack the things that make Selenium painful:

• Auto-wait. page.click() and page.fill() wait for the element to be attached, visible, and actionable before firing. You write less wait code, and your scripts get less flaky.
• Browser contexts. A single browser process can host many isolated contexts, each with its own cookies, storage, and proxy. This is the right primitive for parallel scraping with separate sessions.
• Trace viewer. context.tracing.start(screenshots=True, snapshots=True) records a full timeline of network requests, DOM snapshots, and console output you can scrub through later. It turns "my scrape failed yesterday in production" from a guessing game into a debugging session.

If you're starting a new Python headless browser project today, default to Playwright unless you have a specific reason not to. Our deeper Playwright web scraping guide covers selectors, locators, and routing in production detail.

Pyppeteer is an unofficial Python port of Node's Puppeteer, the original Chrome DevTools Protocol library from Google. The API is a near one-to-one mirror of Puppeteer's, which is great if you're translating snippets from Node tutorials, and the asynchronous design is genuinely efficient for short concurrent jobs. According to the original benchmarks in the source material, Pyppeteer can run roughly 30% faster than Playwright on very short scripts, though we'd treat that as indicative, not gospel.

The honest 2026 caveat: Pyppeteer's upstream maintenance has lagged. It's Chromium-only, doesn't track current Puppeteer/Chromium versions, and the GitHub issue tracker reflects a project running on community goodwill rather than active stewardship (verify the current commit cadence on the Pyppeteer repo before adopting). For new code, Playwright covers the same use cases with a more active codebase. A working snippet still looks like this:

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch(headless=True, args=['--no-sandbox'])
    page = await browser.newPage()
    await page.goto('https://example.com', {'waitUntil': 'networkidle0'})
    await page.screenshot({'path': 'example.png', 'fullPage': True})
    print(await page.title())
    await browser.close()

asyncio.run(main())

Use Pyppeteer if you have an existing Pyppeteer codebase you don't want to rewrite, or if you specifically need its CDP-flavored API. Otherwise, Playwright is the safer call. Our long-form Pyppeteer guide goes deeper on what's still worth using it for.

Splash is the odd one out: instead of running a browser inside your Python process, you run Splash itself as a Docker container that exposes an HTTP API. You send it a URL, it spins up a WebKit-based renderer, runs the JavaScript, and returns the rendered HTML, a screenshot, or whatever your Lua script computes. It pairs especially well with Scrapy via the scrapy-splash middleware.

Starting a local Splash server is one command:

docker run -p 8050:8050 scrapinghub/splash

From Python, you talk to it with plain requests:

import requests

params = {'url': 'https://example.com', 'wait': 2, 'timeout': 30}
r = requests.get('http://localhost:8050/render.html', params=params, timeout=60)
html = r.text

Strengths: process isolation (a leaky page can't crash your scraper), simple HTTP interface, and Lua scripting for custom render flows. Weaknesses: WebKit isn't always a perfect match for sites tested only against Chrome, the project moves slowly, and modern anti-bot stacks frequently flag Splash's fingerprint. Most new projects pick Playwright or a hosted API instead, but if you already have a Scrapy pipeline, Splash is still a low-friction way to bolt JavaScript rendering onto it. Our Scrapy and Splash tutorial shows the integration end to end.

At some point, running your own Python headless browser fleet stops being fun. Anti-bot vendors update their fingerprints weekly, residential proxies need rotation logic, and Chromium's memory footprint multiplies fast across containers. Hosted browser APIs solve this by exposing a remote browser you drive over HTTP or via a Playwright/Selenium-compatible WebSocket endpoint.

Conceptually they all look similar from your code's perspective. Here's a generic example that connects to a hosted service over Playwright:

from playwright.sync_api import sync_playwright

WS_ENDPOINT = 'wss://browser.example.com?token=YOUR_API_KEY'

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(WS_ENDPOINT)
    context = browser.contexts[0]
    page = context.new_page()
    page.goto('https://target.example.com')
    print(page.title())
    browser.close()

What you typically get: managed Chromium fleets, built-in residential or mobile proxies, automatic CAPTCHA handling, fingerprint randomization, and per-request session control. What you give up: a little latency on the network hop, per-request cost, and some fine-grained control over the browser binary.

The right time to switch is when one of three things happens: you start seeing consistent blocks on a high-value target, your AWS bill from running Chromium dwarfs what an API subscription would cost, or your team simply doesn't want to be in the browser-ops business. Our Browser API at WebScrapingAPI is one option in that category, and the broader hosted-browser space is mature enough now that switching providers later is mostly a credentials change.

A handful of lighter-weight tools deserve a mention even though they don't compete head-to-head with Selenium and Playwright.

• Requests-HTML. A wrapper around requests and Pyppeteer that lets you opt into JavaScript rendering only when needed. Install with pip install requests-html (Python 3.6+); the first call to .render() downloads Chromium (~150 MB) into ~/.pyppeteer/. Handy for one-off scrapes where most pages are static.
• MechanicalSoup. Not a headless browser at all, just a stateful HTTP client over BeautifulSoup that handles forms and cookies. Useful for old-school server-rendered sites, login flows without JavaScript, or filling out classic HTML forms. Install with pip install mechanicalsoup.
• nodriver. The successor to undetected-chromedriver from the same author. It drops the WebDriver layer entirely and talks straight to Chrome over CDP, which makes it harder to fingerprint as automation. It's young, but it's where a lot of the anti-detection community has moved.

None of these replace a full Python headless browser stack, but each fills a real niche.

Here's the cheat sheet. Treat the anti-bot column as a relative ranking, not an absolute score: every library can be detected by a determined fingerprinter, and every library can pass casual checks with the right plugins.

Use this table as a starting filter, then drill into the section above for the library that matches your constraints.

Take any single Python headless browser benchmark with a grain of salt: results swing on the target page, network conditions, the host machine, and whether the browser is cold-started or reused. The numbers below are reproduced from public benchmarks in the source material we built this comparison from, and we've flagged them as approximate because we haven't independently re-run them at the time of writing.

Short-script timing. In one published comparison, Playwright completed roughly 100 iterations in approximately 290 ms versus Selenium's ~536 ms over the same workload, consistent with Playwright's WebSocket transport advantage. Pyppeteer was reported to run about 30% faster than Playwright on very short scripts, presumably because it skips Playwright's auto-wait and protocol overhead.

Screenshot benchmark. A separate side-by-side run reported approximate end-to-end times of:

• Selenium: ~3.15 s
• Playwright: ~3.94 s (full-page screenshot)
• Pyppeteer: ~4.12 s
• Splash: ~4.25–6.04 s, averaging ~4.78 s

Selenium's edge there is partly because the test captured a viewport screenshot rather than a full-page render.

Practical takeaway. For steady-state pages-per-minute on a single machine, Playwright and Selenium are within the same order of magnitude; the difference rarely dominates your throughput. What dominates is concurrency strategy (browser pool versus contexts versus processes) and how much time each page spends waiting for network and JS. If you're optimizing seriously, run your own benchmark on your actual target page and your actual hardware.

If your target site uses Cloudflare, DataDome, PerimeterX, or any modern bot-management stack, the stock install of any Python headless browser will get flagged within a few requests. The fingerprintable surface is large: navigator.webdriver, missing plugins, WebGL parameters, the Chromium build's TLS/JA3 signature, even the order of HTTP/2 frames. Here's what each library actually gives you:

• Selenium. selenium-stealth patches the most obvious JavaScript tells. undetected-chromedriver and the newer nodriver go further and replace the driver layer itself. None of these change your TLS fingerprint, which is increasingly the weakest link.
• Playwright. playwright-stealth (community port of puppeteer-extra-plugin-stealth) covers the JS-side checks. Browser contexts let you rotate identities cleanly, and per-route handlers let you inject custom headers and cookies without restarting the browser.
• Pyppeteer. Limited stealth tooling, and it lags upstream Puppeteer's improvements.
• Splash. Practically no stealth story. It's WebKit, not Chrome, and modern fingerprinters catch that quickly.
• Hosted browser APIs. This is where they earn their cost. Real residential or mobile IPs, fingerprint rotation across browser builds, managed CAPTCHA solving, and TLS profiles that match retail browsers. When a target is genuinely defended, this is often the only realistic option.

Practical rule of thumb: stealth plugins and a clean residential proxy will get you past casual protection. For aggressive anti-bot stacks, you need a full Python headless browser fingerprint that matches a real Chrome on a real residential IP, and that's usually a hosted browser. Adding residential proxies for IP rotation handles the network side; the browser side is what stealth tooling and managed APIs solve.

Once you move past one page at a time, your Python headless browser strategy changes from "which library" to "which concurrency model." Three patterns cover most cases:

1. One browser, many contexts (Playwright). Cheapest and fastest. Each context is isolated cookies, storage, and proxy, but they share the browser process.
2. Multiple browser instances. More isolation, more memory. Use this when contexts leak state into each other or when you need different browser builds.
3. Multiple processes (Selenium). Selenium's sync API doesn't share well, so you typically run N driver processes behind a concurrent.futures.ProcessPoolExecutor or across machines via Selenium Grid.

A minimal Playwright fan-out using contexts and asyncio.gather:

import asyncio
from playwright.async_api import async_playwright

URLS = ['https://example.com/p/{}'.format(i) for i in range(20)]

async def fetch(browser, url):
    ctx = await browser.new_context()
    page = await ctx.new_page()
    try:
        await page.goto(url, wait_until='domcontentloaded', timeout=30000)
        return await page.title()
    finally:
        await ctx.close()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        sem = asyncio.Semaphore(5)  # cap concurrency
        async def bound(u):
            async with sem:
                return await fetch(browser, u)
        results = await asyncio.gather(*(bound(u) for u in URLS))
        await browser.close()
        print(results)

asyncio.run(main())

Cap concurrency with a semaphore, watch your memory, and recycle browsers periodically. Chromium leaks small amounts of memory per page over thousands of loads.

Local works; production is where Chromium gets opinionated. The two operational rules that save the most pain: ship a known-good base image, and pin your browser version.

A minimal Dockerfile sketch for a Playwright job:

FROM mcr.microsoft.com/playwright/python:v1.47.0-jammy
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "scrape.py"]

The official Playwright image already bundles browsers and the system fonts, codecs, and shared libraries Chromium expects. Selenium's equivalents are the selenium/standalone-chrome images.

GitHub Actions. Use the microsoft/playwright-github-action (or just pip install playwright && playwright install --with-deps) and set headless=True. Cache the browser binaries by hashing your requirements.txt to keep CI runs fast.

AWS Lambda. Full Chromium is too big for a Lambda zip. Use a container image with chromium-headless-shell, or run on Fargate/ECS where the 10 GB image limit is more forgiving. Cold-start times for full Chromium routinely exceed 2 seconds, so Lambda is best for low-volume, latency-tolerant jobs.

Always run with --no-sandbox only inside a sandboxed container, never on a host.

Skip the philosophy and follow the branches. This decision tree covers the scenarios most teams actually face when adopting a Python headless browser.

• Static or server-rendered pages, no JS needed. Don't reach for a browser at all. Use requests or httpx plus BeautifulSoup or Parsel.
• Small to medium scrape of a JS-heavy SPA, no aggressive anti-bot. Use Playwright. Modern API, official Python support, async out of the box. Verdict: Playwright.
• Existing Selenium codebase or a polyglot team where everyone knows Selenium. Stay on Selenium 4 with Selenium Manager. Don't migrate just for the sake of it. Verdict: Selenium.
• Login flows, multi-step forms, or 2FA. Either Playwright (with storage_state for session reuse) or Selenium with explicit waits. Verdict: Playwright preferred.
• Cloudflare, DataDome, or PerimeterX in the way. Stealth plugins plus residential proxies first; if that fails, move to a hosted browser API. Verdict: hosted browser.
• Existing Scrapy pipeline, light JS rendering only. Splash via scrapy-splash is still the path of least resistance. Verdict: Splash.
• Millions of pages per day, mixed targets. Hosted browser API for the protected targets, raw HTTP for the rest. Verdict: hybrid.

Most Python headless browser bugs cluster around the same handful of mistakes:

• No explicit waits. time.sleep(2) is not a wait strategy. Use Playwright's auto-wait or Selenium's WebDriverWait with explicit conditions.
• Leaked browser processes. Always close the browser in a finally block or async with. A long-running scraper that forgets to quit() will exhaust memory in hours.
• Default user agent. Headless Chrome announces itself in its UA string. Override it to a recent stable Chrome value.
• Reusing a single context. Cookies and storage from page 1 follow you to page 100. Use a fresh context per session when sessions matter.
• Missing system fonts in Docker. Pages render "correctly" but text dimensions are off and CSS-based layout detection breaks. Install fonts-liberation and fonts-noto-color-emoji in your image.

When something goes wrong, screenshot on failure, save the rendered HTML, and turn on Playwright's trace viewer. Most flaky scrapers stop being flaky within an hour of having a real timeline to look at.

• Default to Playwright for new Python headless browser code. Async API, official Python support, auto-wait, browser contexts, and the trace viewer remove the rough edges that made Selenium painful.
• Selenium is still a fine choice when you have an existing codebase, a polyglot team, or a need for the broadest browser coverage. Selenium 4 with Selenium Manager removes the install pain.
• Pyppeteer and Splash are niche, not dead. Pyppeteer for translating Puppeteer snippets, Splash for existing Scrapy pipelines. Don't pick either for greenfield work.
• Switch to a hosted browser API when anti-bot defenses, scale, or operational cost stop being worth the engineering time. The integration is mostly a credentials change.
• Benchmark on your own target. Public benchmark numbers are useful directional signals, not contracts. Cold-start, page weight, and concurrency model dominate results more than library choice.

For greenfield Python headless scraping, Playwright is usually the better default. It ships an officially supported Python binding, a native async API, auto-wait on actions, browser contexts for parallel sessions, and a trace viewer for debugging. Selenium wins on browser breadth, ecosystem maturity, and existing test infrastructure. Pick Playwright for new code, Selenium when you already have a working stack.

Pyppeteer is community-maintained and trails upstream Puppeteer and Chromium releases. It still works for short async scripts, but for new projects Playwright covers the same use cases with active maintenance, better cross-browser support, and a richer API. Keep Pyppeteer if you have a working codebase you don't want to migrate; otherwise default to Playwright.

Sometimes, with help. Stealth plugins like selenium-stealth, undetected-chromedriver, nodriver, and playwright-stealth patch the most obvious JavaScript tells, and clean residential proxies handle the IP side. Against aggressive Cloudflare or DataDome configurations, those alone often aren't enough because TLS and HTTP/2 fingerprints also leak automation. A managed browser service is the realistic fallback.

Switch when one of three things tips: you're consistently blocked on a high-value target, your infrastructure cost for Chromium fleets exceeds an API subscription, or your team doesn't want to own browser operations. Hosted services bundle residential proxies, fingerprint rotation, and CAPTCHA handling, which collapses weeks of evasion engineering into a credentials change.

Use a base image that already bundles the browser, such as mcr.microsoft.com/playwright/python or selenium/standalone-chrome. Inside the container, launch with headless=True and --no-sandbox (containers are already sandboxed). For GitHub Actions, install browsers with playwright install --with-deps and cache the binary directory keyed off your lockfile to keep CI runs fast.

Picking the right Python headless browser comes down to three honest questions: how much JavaScript does the page actually need, how aggressively is the target defended, and how much operational complexity are you willing to own. Default to Playwright for new code, stay on Selenium when you already have a working stack, treat Pyppeteer and Splash as niche specialists, and reach for a hosted browser when stealth and scale start eating your weekends. The decision tree above maps most real scenarios to a one-line verdict, and the comparison table gives you a quick filter when you need to revisit the choice.

If you reach the point where running your own Chromium fleet is no longer worth the fight, our Browser API at WebScrapingAPI gives you a managed Python-friendly headless endpoint with built-in residential proxies, fingerprint rotation, and CAPTCHA handling, so your code stays the same and the anti-bot work moves off your plate. Whatever you pick, benchmark on your real target, plan for production from day one, and don't reach for a browser when a JSON endpoint will do.