C# Web Scraping with HttpClient, AngleSharp, and Playwright

You need the .NET 10 SDK, HttpClient (built in) to download pages, and an HTML parser such as AngleSharp or HtmlAgilityPack to pull out the data. Add Microsoft.Extensions.Http.Resilience for retries, CsvHelper for CSV files, and Playwright for .NET when JavaScript builds the page. Everything else ships with .NET.

A C# web scraper downloads a page, finds the parts you care about, and saves them. HttpClient and System.Text.Json ship with .NET, and the packages below fill the gaps.

Start on .NET 10. It is the current long-term support release, with support running into late 2028. .NET 8 and .NET 9 both reach end of support on November 10, 2026, according to the .NET support policy, so tutorials pinned to them are about to age out.

C# web scraping libraries (NuGet versions as of October 2026)

Two names in the last row still show up in many C# scraping tutorials. ScrapySharp last shipped in 2018, and HtmlAgilityPack.CssSelectors.NetCore, the add-on that gives HtmlAgilityPack CSS selector support, last shipped in 2019. Neither is needed anymore: AngleSharp handles CSS selectors natively and is actively maintained, and HtmlAgilityPack itself is current. The first three rows plus Playwright cover nearly every project.

Create a console app and add the four packages the static part of this guide uses:

dotnet new console -n BookScraper
cd BookScraper
dotnet add package AngleSharp
dotnet add package HtmlAgilityPack
dotnet add package Microsoft.Extensions.Http.Resilience
dotnet add package CsvHelper

Any editor works. In Visual Studio, pick the Console App template and add the packages with Manage NuGet Packages. In VS Code, install the C# Dev Kit and run the commands in the terminal. Rider works too.

Next, save a copy of each practice page in a fixtures folder, so you can develop the parser offline. The same files become your test fixtures later:

mkdir fixtures
curl -o fixtures/books-page1.html https://books.toscrape.com/index.html
curl -o fixtures/books-page2.html https://books.toscrape.com/catalogue/page-2.html
curl -o fixtures/quotes-js.html https://quotes.toscrape.com/js/

On Windows PowerShell, use Invoke-WebRequest -OutFile fixtures/books-page1.html https://books.toscrape.com/index.html for each page. Run the app from the project folder so File.ReadAllText("fixtures/...") finds them.

One more .NET 10 feature: dotnet run app.cs runs a single .cs file with no project file, which the Playwright example uses.

Download the HTML with HttpClient, load it into AngleSharp's HtmlParser, select the repeating elements with QuerySelectorAll() and a CSS selector, then read text with TextContent and links with GetAttribute(). Convert each card into a typed record, parse prices with the invariant culture, and resolve every link against the page URL it came from.

Books to Scrape is a practice bookstore with 1,000 books across 50 catalog pages. Each book sits in an <article class="product_pod"> card with the title link, star rating, price, and stock line. Start with the shape of the data. A sealed record gives you an immutable type with value equality, so tests can compare whole rows and one SequenceEqual call can prove two parsers agree.

// Book.cs: one row of scraped data. [CsvRecord] lets CsvHelper 34 generate the CSV mapping at compile time.
using CsvHelper;

[CsvRecord]
public sealed partial record Book(string Title, decimal Price, int Rating, bool InStock, string Url);

The [CsvRecord] attribute and partial keyword are for CsvHelper 34 (see the export section). Now the parser. AngleSharp parses HTML the way a browser does, following the HTML standard, and it supports the same CSS selectors you'd type in DevTools.

// BookParser.cs: turn one Books to Scrape listing page into Book records (AngleSharp)
using System.Globalization;
using AngleSharp.Dom;
using AngleSharp.Html.Parser;

public static class BookParser
{
    static readonly HtmlParser Parser = new();

    static readonly Dictionary<string, int> RatingWords = new()
    {
        ["One"] = 1, ["Two"] = 2, ["Three"] = 3, ["Four"] = 4, ["Five"] = 5,
    };

    public static IDocument Load(string html) => Parser.ParseDocument(html);

    public static List<Book> ParseListing(IDocument page, Uri pageUrl)
    {
        var books = new List<Book>();
        foreach (var card in page.QuerySelectorAll("article.product_pod"))
        {
            var link = card.QuerySelector("h3 a");
            if (link is null) continue;                      // skip broken cards instead of crashing

            var priceText = card.QuerySelector("p.price_color")?.TextContent ?? "";
            var price = decimal.Parse(priceText.Trim().TrimStart('£'), CultureInfo.InvariantCulture);

            var ratingClass = card.QuerySelector("p.star-rating")?.ClassList
                .FirstOrDefault(c => RatingWords.ContainsKey(c));
            var stock = card.QuerySelector("p.availability")?.TextContent ?? "";

            books.Add(new Book(
                Title:   link.GetAttribute("title") ?? link.TextContent.Trim(),  // visible text is truncated
                Price:   price,
                Rating:  ratingClass is null ? 0 : RatingWords[ratingClass],
                InStock: stock.Contains("In stock", StringComparison.OrdinalIgnoreCase),
                Url:     new Uri(pageUrl, link.GetAttribute("href")).ToString()  // resolve against the page, not the site root
            ));
        }
        return books;
    }
}

Each field hides a small trap:

• The card selector. article.product_pod matches every book card and nothing else.
• The title. The visible link text is truncated ("A Light in the ..."), so read the full title from the title attribute.
• The price. The page shows £51.77. Strip the currency sign and parse it as a decimal with CultureInfo.InvariantCulture. Without the invariant culture, decimal.Parse uses the machine's regional settings, and on a German or Romanian machine "51.77" either fails or turns into 5,177.
• The rating. The stars are drawn with CSS, and the only data in the HTML is a class name such as star-rating Three. AngleSharp's ClassList exposes the classes, and a small dictionary maps the word to a number from 1 to 5. Anything unexpected becomes 0 instead of a wrong guess.
• Stock. The availability line becomes a bool.
• The URL. Links on this site are relative, and they're relative to the page's own folder, not the site root. On the home page an href starts with catalogue/, and on page 2 it doesn't. new Uri(pageUrl, href) resolves each link against the page it came from, which is what a browser does. Joining hrefs to the site root instead produces links that work on the home page and 404 on page 2 and later.

The if (link is null) continue; line keeps one odd card, such as an ad slot, from crashing the run. The tests later check the row count.

Run the parser on the saved copy first, so nothing touches the network while you iterate:

// Program.cs (parse a saved page)
var pageUrl = new Uri("https://books.toscrape.com/index.html");
var page = BookParser.Load(File.ReadAllText("fixtures/books-page1.html"));
var books = BookParser.ParseListing(page, pageUrl);
foreach (var b in books.Take(3)) Console.WriteLine(b);
Console.WriteLine($"rows: {books.Count}");

Book { Title = A Light in the Attic, Price = 51.77, Rating = 3, InStock = True, Url = https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html }
Book { Title = Tipping the Velvet, Price = 53.74, Rating = 1, InStock = True, Url = https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html }
Book { Title = Soumission, Price = 50.10, Rating = 1, InStock = True, Url = https://books.toscrape.com/catalogue/soumission_998/index.html }
rows: 20

Once the output looks right, point the same parser at the live page. The PoliteHttp.Create() helper is the configured client from the HttpClient section below. For a first try, a plain new HttpClient() works too.

// Program.cs (same parser, live page)
using var http = PoliteHttp.Create();
var pageUrl = new Uri("https://books.toscrape.com/index.html");
var books = BookParser.ParseListing(BookParser.Load(await http.GetStringAsync(pageUrl)), pageUrl);
Console.WriteLine($"live rows: {books.Count}; first: {books[0].Title} ({books[0].Price})");

live rows: 20; first: A Light in the Attic (51.77)

HtmlAgilityPack is the long-standing .NET HTML parser, and most older C# scraping tutorials use it. It's still maintained, very tolerant of broken markup, and built around XPath queries. If your team already uses it, or you think in XPath, here is the same extraction:

// BookParserXPath.cs: the same extraction with HtmlAgilityPack and XPath
using System.Globalization;
using HtmlAgilityPack;

public static class BookParserXPath
{
    public static List<Book> ParseListing(string html, Uri pageUrl)
    {
        var doc = new HtmlDocument();
        doc.LoadHtml(html);

        var cards = doc.DocumentNode.SelectNodes("//article[contains(@class,'product_pod')]");
        if (cards is null) return [];                       // SelectNodes returns null, not an empty list

        return cards.Select(card =>
        {
            var link = card.SelectSingleNode(".//h3/a")!;
            var price = card.SelectSingleNode(".//p[@class='price_color']")!.InnerText.Trim().TrimStart('£');
            var rating = card.SelectSingleNode(".//p[contains(@class,'star-rating')]")!
                .GetClasses().Last();                         // "star-rating Three" -> "Three"
            return new Book(
                Title:   HtmlEntity.DeEntitize(link.GetAttributeValue("title", "")),
                Price:   decimal.Parse(price, CultureInfo.InvariantCulture),
                Rating:  Array.IndexOf(["One", "Two", "Three", "Four", "Five"], rating) + 1,
                InStock: card.SelectSingleNode(".//p[contains(@class,'availability')]")!.InnerText.Contains("In stock"),
                Url:     new Uri(pageUrl, link.GetAttributeValue("href", "")).ToString());
        }).ToList();
    }
}

Because Book is a record, one comparison proves both parsers agree:

// Program.cs (compare with AngleSharp)
var books = BookParserXPath.ParseListing(File.ReadAllText("fixtures/books-page1.html"),
                                         new Uri("https://books.toscrape.com/index.html"));
Console.WriteLine($"rows: {books.Count}; first: {books[0]}");
var same = books.SequenceEqual(BookParser.ParseListing(
    BookParser.Load(File.ReadAllText("fixtures/books-page1.html")),
    new Uri("https://books.toscrape.com/index.html")));
Console.WriteLine($"identical to AngleSharp result: {same}");

rows: 20; first: Book { Title = A Light in the Attic, Price = 51.77, Rating = 3, InStock = True, Url = https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html }
identical to AngleSharp result: True

The XPath version uses four patterns: //tag[contains(@class,'x')] to find elements anywhere by class, .// to search inside the current node, [@class='x'] for an exact attribute match, and / for a direct child. For anything beyond that, keep an XPath syntax reference open.

Watch for one trap: SelectNodes() returns null, not an empty collection, when nothing matches. Code that loops over the result without a null check throws a NullReferenceException the first time a page changes. HtmlEntity.DeEntitize is there because HtmlAgilityPack leaves entities such as & in attribute values.

If you'd rather skip XPath entirely, HtmlAgilityPack also works with LINQ. doc.DocumentNode.Descendants("article").Where(n => n.HasClass("product_pod")) finds the same cards, and you can chain Select() from there.

Use AngleSharp if you copy selectors from browser DevTools and want standards-based parsing. Use HtmlAgilityPack if you prefer XPath or LINQ, or you're maintaining code that already uses it.

Open the page in Chrome or Firefox, right-click the data you want, and choose Inspect. Find a class or tag that repeats once per record, test it in the DevTools console with document.querySelectorAll(), then paste the same selector into AngleSharp's QuerySelectorAll(). Prefer short, stable class names over long generated paths copied from DevTools.

On Books to Scrape, right-clicking a book title and choosing Inspect lands on an <a> inside an <h3>, inside <article class="product_pod">. That article is the record boundary: one per book, twenty per page. In the console, document.querySelectorAll("article.product_pod").length returns 20.

Every selector in this guide uses one of six patterns:

DevTools' Copy selector produces long paths like #default > div > div > ... > li:nth-child(1) > article that break when the layout gains a wrapper div. A short selector on a meaningful class survives redesigns.

XPath is the other option, and it can do things CSS can't, such as selecting an element by its text or walking up to a parent. For a side-by-side comparison, see when XPath beats CSS.

Tip: If a selector finds elements in DevTools but returns zero in C#, the data probably isn't in the raw HTML. DevTools shows the page after JavaScript runs, and HttpClient sees only what the server sent. Press Ctrl+U (View Source) and search for the text. If it isn't there, jump to the JavaScript section below.

Create one HttpClient and reuse it for every request, set PooledConnectionLifetime so DNS changes are picked up, send an honest User-Agent, and add a retry pipeline that backs off on 408, 429, and 5xx responses and honors Retry-After. Microsoft.Extensions.Http.Resilience gives you that retry behavior in about fifteen lines.

There are two classic mistakes. The first is creating a new HttpClient per request: each instance opens its own connections, and over a few thousand pages that can exhaust the machine's sockets. The second is a static client that never recycles connections, so it keeps talking to an old IP address after a site's DNS changes.

Microsoft's HttpClient guidelines resolve both: use one long-lived client, and set PooledConnectionLifetime on its handler so connections are recycled every few minutes. The same guidelines show how to add resilience to a static client, which is the pattern below.

// PoliteHttp.cs: one shared HttpClient with an honest user agent, timeouts, and retries
using System.Net;
using Microsoft.Extensions.Http.Resilience;
using Polly;

public static class PoliteHttp
{
    public static HttpClient Create(TimeSpan? attemptTimeout = null)
    {
        var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
            .AddRetry(new HttpRetryStrategyOptions
            {
                MaxRetryAttempts = 3,                        // retries 408, 429, 5xx and network errors
                BackoffType = DelayBackoffType.Exponential,  // waits longer each time, honors Retry-After
                UseJitter = true,
            })
            .AddTimeout(attemptTimeout ?? TimeSpan.FromSeconds(30))
            .Build();

        var handler = new ResilienceHandler(pipeline)
        {
            InnerHandler = new SocketsHttpHandler
            {
                PooledConnectionLifetime = TimeSpan.FromMinutes(5),
                AutomaticDecompression = DecompressionMethods.All,
            },
        };

        var http = new HttpClient(handler) { Timeout = Timeout.InfiniteTimeSpan }; // the pipeline owns timeouts
        http.DefaultRequestHeaders.UserAgent.ParseAdd("csharp-scraping-tutorial/1.0 (+mailto:you@example.com)");
        return http;
    }
}

Here's what each part does:

• SocketsHttpHandler is the networking stack under HttpClient. PooledConnectionLifetime = 5 minutes recycles connections so DNS changes take effect, and AutomaticDecompression handles gzip, Brotli, and deflate.
• The retry strategy comes from Microsoft's HTTP resilience library, which builds on Polly. HttpRetryStrategyOptions already knows which failures are worth retrying: 408 Request Timeout, HTTP 429 Too Many Requests, 5xx server errors, network exceptions, and timeouts. It backs off exponentially, adds jitter so parallel workers don't retry in lockstep, and waits for the Retry-After header when the server sends one, which is the site telling you how long to pause.
• The timeout strategy gives each attempt 30 seconds by default. HttpClient.Timeout is set to infinite because HttpClient's own 100-second default would otherwise cut across retries. The pipeline owns timeouts instead. The optional attemptTimeout parameter lets slower calls, such as rendered API requests, use a longer limit.
• The User-Agent names your scraper and gives a way to contact you. Sites can reach you instead of blocking you. Replace the email with your own. For the rest of the request headers and when they matter, see which headers matter for scraping.

Three retries is enough. More rarely turns a refusal into a success, so log the failure and move on.

In ASP.NET Core or a worker service, skip the static helper and let dependency injection manage the client. Register a typed client with builder.Services.AddHttpClient<BookFetcher>() and chain .AddStandardResilienceHandler() to get retries, timeouts, and a circuit breaker with sensible defaults. IHttpClientFactory then handles connection lifetimes for you. The parser classes in this guide don't change at all. Only the place where the HttpClient comes from does.

Parse the first page, read the next-page link, resolve it against the current page URL with new Uri(pageUrl, href), and repeat until the link disappears or you reach a page limit. Reuse the same HttpClient, wait about a second between requests, and let the retry pipeline handle 429 and 503 responses.

Books to Scrape splits its catalog across 50 pages, and each page ends with a pager whose "next" item is <li class="next"><a href="page-2.html">next</a></li>. Following it until it disappears means you never guess the page count.

// Crawler.cs: follow "next" links politely, then visit detail pages with bounded concurrency
using System.Collections.Concurrent;
using AngleSharp.Dom;

public static class Crawler
{
    public static async Task<List<Book>> CrawlCatalogAsync(HttpClient http, Uri start, int maxPages = 3)
    {
        var books = new List<Book>();
        Uri? url = start;
        for (var page = 1; url is not null && page <= maxPages; page++)
        {
            var html = await http.GetStringAsync(url);
            var doc = BookParser.Load(html);
            books.AddRange(BookParser.ParseListing(doc, url));

            var next = doc.QuerySelector("li.next a")?.GetAttribute("href");
            url = next is null ? null : new Uri(url, next);   // relative to the current page
            await Task.Delay(TimeSpan.FromSeconds(1));         // about one request per second
        }
        return books;
    }

    public sealed record BookDetail(string Url, string Upc, int Available);

    public static async Task<List<BookDetail>> FetchDetailsAsync(HttpClient http, IEnumerable<string> urls)
    {
        var results = new ConcurrentBag<BookDetail>();
        var options = new ParallelOptions { MaxDegreeOfParallelism = 3 };   // a small, fixed cap per site

        await Parallel.ForEachAsync(urls, options, async (url, ct) =>
        {
            var doc = BookParser.Load(await http.GetStringAsync(url, ct));
            var table = doc.QuerySelectorAll("table.table-striped tr")
                .ToDictionary(r => r.QuerySelector("th")!.TextContent.Trim(),
                              r => r.QuerySelector("td")!.TextContent.Trim());
            var available = new string(table["Availability"].Where(char.IsDigit).ToArray());
            results.Add(new BookDetail(url, table["UPC"], int.Parse(available)));
            await Task.Delay(TimeSpan.FromMilliseconds(500), ct);
        });

        return results.OrderBy(d => d.Url).ToList();
    }
}

CrawlCatalogAsync does four things on each loop: fetch the page, parse its books with the same BookParser, look for li.next a, and wait a second. The new Uri(url, next) line matters here too. The next link on page 1 is page-2.html, relative to /catalogue/, and resolving it against the current page gives the right absolute URL every time. Each page is passed to ParseListing with its own URL, so every book link on every page resolves correctly as well.

The maxPages parameter is a safety valve. Keep it small while you develop, so a selector mistake or a loop bug costs three requests instead of three thousand. Raise it once the output looks right.

Start on catalogue/page-1.html, not the home page, so every page URL has the same shape.

// Program.cs (crawl three pages)
using var http = PoliteHttp.Create();
var books = await Crawler.CrawlCatalogAsync(http, new Uri("https://books.toscrape.com/catalogue/page-1.html"), maxPages: 3);
Console.WriteLine($"pages: 3, rows: {books.Count}");
Console.WriteLine(books[20].Url);
using var check = await http.GetAsync(books[20].Url);
Console.WriteLine($"status of a parsed detail URL: {(int)check.StatusCode}");

pages: 3, rows: 60
https://books.toscrape.com/catalogue/in-her-wake_980/index.html
status of a parsed detail URL: 200

Row 21 is the first book on page 2, and its URL returns 200, which confirms the page-relative resolution works past the first page. That's where root-relative joining usually breaks.

This next-link loop is a small crawler, and it can't visit the same page twice because each page points only forward. Once you follow links that can loop back, such as category pages, filters, or "related items", keep a HashSet<string> of visited URLs and skip anything already in it. If the difference between the two jobs is fuzzy, here's how scraping and crawling differ.

The one-second delay is self-imposed. On real sites, start slow, watch for 429s, and treat robots.txt and the terms as the ceiling.

Detail pages hold extras such as the UPC and the exact stock count. Fetching them one by one is slow, and firing them all at once is rude. Parallel.ForEachAsync sits in between: it runs an async body for each item, with at most MaxDegreeOfParallelism running at the same time.

FetchDetailsAsync in the crawler above caps that at three. Pick a small, fixed number per site rather than Environment.ProcessorCount, the value many examples reach for. Your CPU core count says nothing about what a website can handle, and scraping is almost entirely waiting on the network anyway.

Results go into a ConcurrentBag, which is safe to add to from several tasks at once, and they're sorted at the end because parallel work finishes in no particular order.

// Program.cs (detail pages, three at a time)
using var http = PoliteHttp.Create();
var page1 = new Uri("https://books.toscrape.com/catalogue/page-1.html");
var books = BookParser.ParseListing(BookParser.Load(await http.GetStringAsync(page1)), page1);
var details = await Crawler.FetchDetailsAsync(http, books.Take(6).Select(b => b.Url));
foreach (var d in details) Console.WriteLine($"{d.Upc}  {d.Available,3} available  {d.Url}");

a897fe39b1053632   22 available  https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
4165285e1663650f   20 available  https://books.toscrape.com/catalogue/sapiens-a-brief-history-of-humankind_996/index.html
e00eb4fd7b871a48   20 available  https://books.toscrape.com/catalogue/sharp-objects_997/index.html
6957f44c3847a760   20 available  https://books.toscrape.com/catalogue/soumission_998/index.html
f77dbf2323deb740   19 available  https://books.toscrape.com/catalogue/the-requiem-red_995/index.html
90fa61229261140a   20 available  https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html

Every task shares the same HttpClient, which is safe and intended: HttpClient is thread-safe for sending requests, and sharing it means the retry pipeline and connection pool apply to all workers at once.

For JSON, serialize your list of records with System.Text.Json, which ships with .NET. For CSV, use CsvHelper: mark the record with [CsvRecord], open a writer with CsvWriter.ToFile(), and call WriteRecords(). CsvHelper 34 writes numbers with the invariant culture by default, so prices keep a dot on every machine.

Both formats take a few lines:

// Export.cs: write results to CSV (CsvHelper 34) and JSON (System.Text.Json)
using System.Text.Json;
using CsvHelper;

public static class Export
{
    public static void ToCsv(IEnumerable<Book> books, string path)
    {
        using var csv = CsvWriter.ToFile(path);   // UTF-8, header row first, invariant culture by default
        csv.WriteRecords(books);
    }

    public static void ToJson(IEnumerable<Book> books, string path) =>
        File.WriteAllText(path, JsonSerializer.Serialize(books, new JsonSerializerOptions { WriteIndented = true }));
}

// Program.cs (save results)
var pageUrl = new Uri("https://books.toscrape.com/index.html");
var books = BookParser.ParseListing(BookParser.Load(File.ReadAllText("fixtures/books-page1.html")), pageUrl);
Export.ToCsv(books, "books.csv");
Export.ToJson(books, "books.json");
Console.WriteLine(string.Join(Environment.NewLine, File.ReadLines("books.csv").Take(3)));
Console.WriteLine($"wrote books.csv and books.json with {books.Count} rows");

Title,Price,Rating,InStock,Url
A Light in the Attic,51.77,3,True,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet,53.74,1,True,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html
wrote books.csv and books.json with 20 rows

The header row comes from the record's property names, and the column order follows the record's parameters. JSON keeps the same names and types, so prices stay numbers and InStock stays a boolean. Use CSV for spreadsheets and imports, JSON for programs and nested data.

CsvHelper 34 changed the API. CsvHelper 34.0.0, released October 28, 2026, is a rewrite. Code that does new CsvWriter(writer, CultureInfo.InvariantCulture), which is what most C# scraping tutorials show, no longer compiles against the current package. Either use the v34 API shown here or pin the previous version with dotnet add package CsvHelper --version 33.1.0 for older code. CsvHelper's v34 migration guide lists the rest of the changes.

This is also why Book is declared partial with a [CsvRecord] attribute. CsvHelper 34 generates the mapping between the record and its columns at compile time, and the source generator needs a partial type to add that code to.

Older versions made you pass a culture to every writer, and forgetting it meant a German machine wrote 51,77 into a comma-separated file. Version 34 defaults to the invariant culture.

Fetch the raw HTML with HttpClient and run your selector on it. If it finds nothing but the browser shows the data, JavaScript builds that content. Before starting a browser, search the page source and the DevTools Network tab: many sites ship the records as JSON that System.Text.Json can read directly.

The /js/ version of Quotes to Scrape looks identical to the regular site in a browser: ten quotes, each in a div.quote with the text, author, and tags. The raw HTML tells a different story. The server sends an empty container plus a <script> block that holds every quote as a JavaScript array, and the browser builds the div.quote elements from it after the page loads.

So check the response before you reach for a browser. If the records are in a script tag, HttpClient and System.Text.Json are enough.

// ScriptData.cs: is the data in the HTML, or built by JavaScript? Read it from the <script> tag.
using System.Text.Json;
using System.Text.RegularExpressions;

public sealed record Author(string Name);
public sealed record Quote(string Text, Author Author, string[] Tags);

public static partial class ScriptData
{
    [GeneratedRegex(@"var data = (\[.*?\]);", RegexOptions.Singleline)]
    private static partial Regex DataArray();

    public static List<Quote> ReadQuotes(string html)
    {
        var doc = BookParser.Load(html);
        var script = doc.QuerySelectorAll("script")
            .Select(s => s.TextContent)
            .First(t => t.Contains("var data"));
        var json = DataArray().Match(script).Groups[1].Value;
        return JsonSerializer.Deserialize<List<Quote>>(json,
            new JsonSerializerOptions { PropertyNameCaseInsensitive = true })!;
    }
}

// Program.cs (check, then read the script tag)
var html = File.ReadAllText("fixtures/quotes-js.html");
Console.WriteLine($"div.quote elements in raw HTML: {BookParser.Load(html).QuerySelectorAll("div.quote").Length}");
var quotes = ScriptData.ReadQuotes(html);
foreach (var q in quotes.Take(2)) Console.WriteLine($"{q.Author.Name}: {q.Text[..60]}… [{string.Join(", ", q.Tags)}]");
Console.WriteLine($"quotes parsed from the script tag: {quotes.Count}");

div.quote elements in raw HTML: 0
Albert Einstein: “The world as we have created it is a process of our thinkin… [change, deep-thoughts, thinking, world]
J.K. Rowling: “It is our choices, Harry, that show what we truly are, far … [abilities, choices]
quotes parsed from the script tag: 10

The first line is the check: zero div.quote elements in the raw HTML, even though the browser shows ten. AngleSharp then finds the script that contains var data, a source-generated regex pulls out the array, and System.Text.Json turns it into Quote records. [GeneratedRegex] compiles the pattern at build time, and PropertyNameCaseInsensitive maps lowercase keys onto the record properties.

The same idea covers two other common cases. Many product and article pages embed structured data in <script type="application/ld+json">, which you can select with AngleSharp and deserialize directly, no regex needed. And many JavaScript-heavy sites load their records from their own JSON endpoints. Open DevTools, go to Network, filter by Fetch/XHR, and reload the page. If a request returns the data you see on screen, call that URL with HttpClient.GetFromJsonAsync<T>() and skip HTML entirely.

Reading JSON is faster than rendering and breaks less often, because data formats change less often than layouts.

Use Playwright for .NET, Microsoft's browser automation library. It downloads Chromium, loads the page headlessly, waits for your selector, and lets you read elements or the rendered HTML. PuppeteerSharp and Selenium WebDriver also work. For many pages or blocked sites, render remotely with a scraper API instead of running browsers yourself.

When there's no JSON to read, or you must click or type first, use a real browser. Playwright drives Chromium, Firefox, or WebKit from C#, runs them headless by default, and has waiting built into its locators. If the idea of a browser with no window is new, here's how headless browsers work.

To keep the browser out of your main project, this is a .NET 10 file-based app that declares its own package:

#:package Microsoft.Playwright@1.64.1
#:property PublishAot=false
// render.cs: a .NET 10 file-based app. First run: dotnet run render.cs -- install
using Microsoft.Playwright;

if (args is ["install"])
    return Microsoft.Playwright.Program.Main(["install", "chromium"]);

using var playwright = await Playwright.CreateAsync();
await using var browser = await playwright.Chromium.LaunchAsync(new() { Headless = true });
var page = await browser.NewPageAsync(new() { UserAgent = "csharp-scraping-tutorial/1.0 (+mailto:you@example.com)" });

await page.GotoAsync("https://quotes.toscrape.com/js/");
var quotes = page.Locator("div.quote");
await quotes.First.WaitForAsync();                       // wait for JavaScript to build the list

foreach (var quote in (await quotes.AllAsync()).Take(3))
{
    var text = await quote.Locator("span.text").InnerTextAsync();
    var author = await quote.Locator("small.author").InnerTextAsync();
    Console.WriteLine($"{author}: {text[..50]}…");
}
Console.WriteLine($"quotes rendered: {await quotes.CountAsync()}");

// Hand the rendered HTML to the same AngleSharp code you already wrote, if you prefer
var html = await page.ContentAsync();
Console.WriteLine($"rendered HTML length: {html.Length}");
return 0;

dotnet run render.cs -- install   # once: downloads Chromium
dotnet run render.cs

Albert Einstein: “The world as we have created it is a process of o…
J.K. Rowling: “It is our choices, Harry, that show what we truly…
Albert Einstein: “There are only two ways to live your life. One is…
quotes rendered: 10
rendered HTML length: 8940

The two #: lines at the top replace a project file. #:package pulls Playwright from NuGet. #:property PublishAot=false is needed because file-based apps default to native AOT settings, and Playwright relies on reflection-based JSON serialization that AOT trims away. The first run with install downloads a matching Chromium build. On Linux, Chromium may also need system libraries, so check the Playwright for .NET docs for your operating system.

The key line is await quotes.First.WaitForAsync(). It waits until at least one div.quote exists, which means JavaScript has finished building the list. A fixed Task.Delay(3000) is too long on fast pages and too short on slow ones.

If you'd rather keep all parsing in one place, page.ContentAsync() returns the rendered HTML as a string. Pass it to BookParser.Load() and query it with AngleSharp exactly as you did for static pages.

PuppeteerSharp is a C# port of Node's Puppeteer. It focuses on Chromium, and its API maps almost line for line to Puppeteer's, which makes it a good fit when you're porting existing Node scripts to .NET.

Selenium WebDriver is the long-established option and fits teams that already run Selenium tests or a Selenium Grid. For scraping, expect to write more explicit waiting code than with Playwright, since most elements need a WebDriverWait before you can read them.

Local browsers cost memory and CPU. When you need many sessions, or interactive steps such as clicks and form fills on sites that resist automation, a hosted automated browser moves that work off your machine.

Send the target URL to the scraper API with HttpClient and parse the HTML it returns with your existing code. With WebScrapingAPI's Scraper API, call one endpoint with your api_key and the URL-encoded target, then add render_js=1 for JavaScript pages and country for location. Read the key from an environment variable, never from source code.

Fetching from your own machine is the right default. These symptoms mean it has become the bottleneck:

• 403 or 429 responses that keep coming back even at a polite rate and after retries.
• Challenge or consent pages arriving instead of the content you see in a browser.
• Content that changes by country, such as prices, availability, or search results, when you need a specific location.
• More JavaScript rendering than local Chromium instances can keep up with.

Rule out the basics first. These four habits fix many blocks, and there are other ways to reduce blocks beyond them:

• Slow down: fewer concurrent requests and a delay between them.
• Send an honest, consistent User-Agent.
• Cache pages you've already fetched instead of downloading them again.
• Fetch only the pages and fields you need.

If you run your own proxy pool, you can also route HttpClient through a proxy and keep everything else as it is.

When those don't hold up, let a scraper API take over the fetch step. WebScrapingAPI's Scraper API accepts a URL and returns the page, handling proxies, retries, and optional JavaScript rendering. From C#, it's one GET request, and a small class is the whole integration:

// WsaClient.cs: fetch pages through WebScrapingAPI's Scraper API; parsing stays the same
public sealed class WsaClient(HttpClient http)
{
    const string Endpoint = "https://api.webscrapingapi.com/v2";

    public static Uri BuildUri(string apiKey, string targetUrl, bool renderJs = false, string? country = null)
    {
        var query = new List<string>
        {
            "api_key=" + Uri.EscapeDataString(apiKey),
            "url=" + Uri.EscapeDataString(targetUrl),     // encode the target so its ? and & survive
        };
        if (renderJs) query.Add("render_js=1");
        if (country is not null) query.Add("country=" + Uri.EscapeDataString(country));
        return new Uri($"{Endpoint}?{string.Join("&", query)}");
    }

    public async Task<string> GetHtmlAsync(string targetUrl, bool renderJs = false, string? country = null,
                                           CancellationToken ct = default)
    {
        var apiKey = Environment.GetEnvironmentVariable("WSA_API_KEY");
        if (string.IsNullOrWhiteSpace(apiKey))
            throw new InvalidOperationException("Set the WSA_API_KEY environment variable first.");

        using var response = await http.GetAsync(BuildUri(apiKey, targetUrl, renderJs, country), ct);
        var body = await response.Content.ReadAsStringAsync(ct);
        if (!response.IsSuccessStatusCode)
            throw new HttpRequestException($"Scraper API returned {(int)response.StatusCode}: {body}");
        return body;
    }
}

BuildUri is public so you can see exactly what will be sent without sending anything:

// Program.cs (see what you'll send, without sending it)
var uri = WsaClient.BuildUri("YOUR_KEY", "https://quotes.toscrape.com/js/?page=1&tag=life", renderJs: true, country: "us");
Console.WriteLine(uri.AbsoluteUri);
try { await new WsaClient(new HttpClient()).GetHtmlAsync("https://books.toscrape.com/"); }
catch (InvalidOperationException ex) { Console.WriteLine($"without a key: {ex.Message}"); }

https://api.webscrapingapi.com/v2?api_key=YOUR_KEY&url=https%3A%2F%2Fquotes.toscrape.com%2Fjs%2F%3Fpage%3D1%26tag%3Dlife&render_js=1&country=us
without a key: Set the WSA_API_KEY environment variable first.

Look at the url= value. The target's own ? and & became %3F and %26. Without Uri.EscapeDataString(), the API would read tag=life as one of its own parameters and fetch the wrong page.

The key never appears in code. Set it once per shell or machine:

export WSA_API_KEY=your-key-here        # macOS and Linux
setx WSA_API_KEY "your-key-here"        # Windows (open a new terminal afterward)

In an ASP.NET Core app, dotnet user-secrets keeps the key out of your repository during development, and your host's secret store does the same in production. If the variable is missing, WsaClient stops with a clear message before sending a request, as the second output line shows. If the key is wrong, the API answers HTTP 401 with {"error":"Invalid API key"}, and WsaClient surfaces that body in an HttpRequestException, so you see the reason instead of an empty result.

Two parameters cover most needs. renderJs: true adds render_js=1, which renders the page in a browser before returning it. country: "us" adds country=us, which fetches the page from that location. JavaScript rendering and geotargeting cost no extra credits. Rendered requests take longer than plain ones, so give the client a longer per-attempt timeout with PoliteHttp.Create(TimeSpan.FromSeconds(120)).

The response is HTML, so it goes into the same BookParser you wrote at the start, unchanged:

// Program.cs (fetch through the Scraper API)
using var http = PoliteHttp.Create(attemptTimeout: TimeSpan.FromSeconds(120));
var wsa = new WsaClient(http);
var target = new Uri("https://books.toscrape.com/catalogue/page-2.html");
var books = BookParser.ParseListing(BookParser.Load(await wsa.GetHtmlAsync(target.ToString(), country: "us")), target);
Console.WriteLine($"rows via Scraper API: {books.Count}; first: {books[0].Title}");
var rendered = BookParser.Load(await wsa.GetHtmlAsync("https://quotes.toscrape.com/js/", renderJs: true));
Console.WriteLine($"div.quote after render_js: {rendered.QuerySelectorAll("div.quote").Length}");

Sample output:

rows via Scraper API: 20; first: In Her Wake
div.quote after render_js: 10

ParseListing still receives the target URL, not the API endpoint. Links resolve against the page the HTML came from, so every book URL stays correct no matter how the page was fetched. The rendered Quotes page contains the same div.quote elements Playwright saw.

Use HttpClient with AngleSharp or HtmlAgilityPack for static pages. Add the resilience pipeline and a concurrency cap when you crawl many pages. Read script-tag JSON or the site's JSON endpoint when JavaScript loads the data. Use Playwright for a few interactive pages, and a scraper API when blocks or volume make local fetching unreliable.

If you're not sure where a page fits, walk through these questions in order and stop at the first yes:

1. Is the data in View Source? Fetch with HttpClient and parse with AngleSharp or HtmlAgilityPack.
2. Do you need many pages? Add the resilience pipeline, a delay, and a small concurrency cap.
3. Is the data in a script tag or a JSON request in the Network tab? Read it with System.Text.Json.
4. Does the page need a browser, and only a few times? Use Playwright for .NET.
5. Are you blocked, do you need a specific country, or are you rendering at volume? Send the fetch through the Scraper API and keep your parser.

Save a copy of each page type as an HTML file, add an xUnit test project that parses those files offline, and assert on row counts, typed values, and absolute URLs. When the site changes its markup, a test fails in seconds instead of your scraper silently returning empty results in production.

Scrapers fail quietly: a site renames a class, and the job writes an empty file that looks like success. The pages you saved during setup are already your test fixtures.

Create a test project next to the scraper and reference it:

dotnet new xunit -n BookScraper.Tests
cd BookScraper.Tests
dotnet add reference ../BookScraper/BookScraper.csproj

Add to BookScraper.Tests.csproj so the saved pages are copied next to the test binaries:

<ItemGroup>
  <None Include="../fixtures/*.html" Link="fixtures/%(Filename)%(Extension)" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>

Then test the behavior you depend on, not the implementation:

// BookParserTests.cs: run the parser against a saved page, so tests never touch the network
public class BookParserTests
{
    static readonly Uri Page2 = new("https://books.toscrape.com/catalogue/page-2.html");

    static List<Book> ParseFixture() =>
        BookParser.ParseListing(BookParser.Load(File.ReadAllText("fixtures/books-page2.html")), Page2);

    [Fact]
    public void Finds_every_book_on_the_page() => Assert.Equal(20, ParseFixture().Count);

    [Fact]
    public void Resolves_links_against_the_page_not_the_site_root()
    {
        var first = ParseFixture()[0];
        Assert.Equal("https://books.toscrape.com/catalogue/in-her-wake_980/index.html", first.Url);
    }

    [Fact]
    public void Reads_price_and_rating_as_numbers()
    {
        var first = ParseFixture()[0];
        Assert.Equal(12.84m, first.Price);
        Assert.InRange(first.Rating, 1, 5);
    }
}

Passed!  - Failed:     0, Passed:     3, Skipped:     0, Total:     3, Duration: 147 ms - BookScraper.Tests.dll (net10.0)

The three tests guard the three things most likely to break: the row count (the card selector still matches), the URL on page 2 (links still resolve against the page), and typed values (price and rating still parse). Page 2 is the fixture on purpose, because it's where root-relative URL bugs show up.

Apply the same idea in production with a "fail loudly" rule: if a listing page returns zero cards, throw an exception instead of writing an empty file. When it fails, save a fresh copy of the page, fix the selector, and rerun. Also consider adding a ScrapedAt timestamp column to your records, so every row shows when it was collected and you can compare runs later.

Read the site's terms and robots.txt before you crawl, keep request rates low, identify your scraper with a user agent that includes contact details, and collect only the fields you need. Avoid personal data unless you have a lawful basis, and don't scrape behind a login without permission. Prefer an official API when one exists.

robots.txt is a plain text file at a site's root that tells crawlers which paths to skip, and its format is defined by the robots.txt standard (RFC 9309). Reading it takes a few lines with the client you already have:

// Robots.cs: read a site's robots.txt before you crawl (RFC 9309)
using System.Net;

public static class Robots
{
    public static async Task<List<string>> ReadRulesAsync(HttpClient http, Uri site)
    {
        using var response = await http.GetAsync(new Uri(site, "/robots.txt"));
        if (response.StatusCode == HttpStatusCode.NotFound)
            return ["No robots.txt: no crawl rules published"];
        response.EnsureSuccessStatusCode();

        var text = await response.Content.ReadAsStringAsync();
        return text.Split('\n')
            .Select(line => line.Trim())
            .Where(line => line.StartsWith("User-agent:", StringComparison.OrdinalIgnoreCase)
                        || line.StartsWith("Disallow:", StringComparison.OrdinalIgnoreCase)
                        || line.StartsWith("Allow:", StringComparison.OrdinalIgnoreCase)
                        || line.StartsWith("Crawl-delay:", StringComparison.OrdinalIgnoreCase))
            .ToList();
    }
}

// Program.cs (read robots.txt first)
using var http = PoliteHttp.Create();
foreach (var line in await Robots.ReadRulesAsync(http, new Uri("https://books.toscrape.com"))) Console.WriteLine(line);

No robots.txt: no crawl rules published

Books to Scrape publishes no robots.txt, and under the standard a missing file means no crawl rules apply. On real sites, honor the Disallow lines for your user agent and any Crawl-delay.

Keep in mind that robots.txt is a crawler convention, not legal permission. A site's terms of service, the type of data, and where you and the site operate all matter too, so read up on whether web scraping is legal before scraping anything commercial. Collect only what you need, store it securely, and stay away from personal data unless you have a clear lawful basis. For the standards we hold ourselves to, see our responsible web data principles.

There isn't one library for every job. Use HttpClient to download pages, AngleSharp or HtmlAgilityPack to parse HTML, and Playwright for .NET when JavaScript builds the content. For large or blocked jobs, keep the parser and swap the download step for a scraper API. Most production scrapers combine two or three of these.

Both are maintained and parse real-world HTML well. AngleSharp follows the HTML standard and supports CSS selectors natively, so selectors copied from browser DevTools work as is. HtmlAgilityPack is older, very tolerant of broken markup, and built around XPath and LINQ. Pick the query style your team prefers. Switching later only touches the parser.

Not with HttpClient alone, because it only receives the HTML the server sends. Playwright for .NET, PuppeteerSharp, or Selenium can run the page in a headless browser and give you the rendered content. First check the page source and network requests for JSON, which is usually faster and more reliable than rendering.

For new scrapers, Playwright for .NET is usually simpler: it installs its own browsers, waits for elements automatically, and supports Chromium, Firefox, and WebKit from one API. Selenium WebDriver still makes sense when your team already runs Selenium tests or a Selenium Grid. Both are slower and heavier than plain HTTP requests, so use them only when needed.

No .NET framework is as widely used as Scrapy is in Python. Most C# teams assemble the same pieces themselves: HttpClient with a resilience pipeline for fetching, AngleSharp or HtmlAgilityPack for parsing, Parallel.ForEachAsync or channels for concurrency, and a queue or database for URLs. That stack covers what most scraping projects need.

Slow down and retry later. Add a retry pipeline with Microsoft.Extensions.Http.Resilience, which retries 429 and 5xx responses with exponential backoff and waits for the Retry-After header when the server sends one. Lower your concurrency and add a delay between requests. If 429s continue at a polite rate, the site is asking you to stop.

Reuse one HttpClient, send an honest User-Agent, keep concurrency low, cache pages you've already fetched, and retry 429 responses only after the delay the server asks for. If a site still answers with 403 errors or challenge pages, route requests through a scraper API that manages proxies and rendering, and keep your parser unchanged.

Keep it out of source code. Read it from an environment variable such as WSA_API_KEY with Environment.GetEnvironmentVariable, use dotnet user-secrets during local ASP.NET Core development, and use your host's secret store in production. Never paste keys into URLs you share, commit them to Git, or log the full request URL.

Use Parallel.ForEachAsync with MaxDegreeOfParallelism set to a small number, such as three to five per site, and share one HttpClient across all tasks. Collect results in a thread-safe collection like ConcurrentBag. The limit should reflect what the website can handle, not how many CPU cores your machine has.

Scraping publicly available data is often lawful, but it depends on the site's terms, the kind of data, and local law, especially for personal data and copyrighted content. The programming language doesn't change that. Respect robots.txt, avoid personal data without a lawful basis, and get legal advice before scraping for a commercial project.

Your AngleSharp or HtmlAgilityPack code doesn't need to change when a site starts blocking you or needs JavaScript. Point WsaClient at WebScrapingAPI's Scraper API and keep parsing in .NET. JavaScript rendering and geotargeting cost no extra credits, failed requests aren't billed, and the free trial includes 5,000 requests.