Web Scraping with Go: Build a Reliable HTML Scraper

Start by separating page retrieval, HTML parsing, and browser execution. They are different jobs, even when a library packages several of them together. For a page that already contains the desired records in its HTML response, an HTTP client and parser are enough.

The example below combines net/http with goquery. This keeps the fetch and parse functions small enough to test separately. A change to a selector does not require a network call to verify, and an HTTP failure does not disappear inside extraction code.

Use Colly when the scheduling work becomes significant. Reach for a browser after inspecting the response and confirming that the fields are absent from the initial HTML. Running a browser does not automatically make the extracted values correct.

For web scraping with Go, this choice usually matters more than comparing language speed. Choose the least complicated tool that returns the content your parser actually needs, then verify the resulting records.

Create a directory outside any existing Go module and initialize a small project:

mkdir go-book-scraper
cd go-book-scraper

go mod init example.com/book-scraper
go get github.com/PuerkitoBio/goquery@v1.13.0

The example was tested with Go 1.26.2 and goquery 1.13.0. This pinned goquery version declares Go 1.25.0 as its minimum in its module metadata. Use a compatible toolchain rather than copying an old tutorial’s claim about the latest Go release.

Keep go.mod and go.sum with the project. Go’s dependency-management guide explains how module requirements and checksums support reproducible builds. Update dependencies deliberately and rerun the parser tests after an update.

Our target is Books to Scrape, a practice bookstore intended for scraping exercises. Its catalog contains product cards with a title, price, and link. The example assumes UTF-8 HTML and processes one catalog page per invocation. That deliberately small scope makes web scraping with Go easier to inspect before adding navigation or concurrent workers.

Save the following complete program as main.go. It extracts each book’s full title, displayed price, and absolute product URL. It keeps prices as source strings, including their currency symbol, instead of introducing numeric conversion into the first extraction step.

The selectors are specific to this practice site. Each article.product_pod is a record container, h3 a holds the book link and full title attribute, and .price_color holds the displayed price. Read fields within each card so that one book’s title cannot accidentally be paired with another book’s price.

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"net/url"
	"os"
	"strings"
	"time"

	"github.com/PuerkitoBio/goquery"
)

const maxBody = 2 << 20 // A 2 MiB limit chosen for this tutorial.

type Book struct {
	Title string `json:"title"`
	Price string `json:"price"`
	URL   string `json:"url"`
}

func parseBooks(r io.Reader, base *url.URL) ([]Book, error) {
	doc, err := goquery.NewDocumentFromReader(r)
	if err != nil {
		return nil, fmt.Errorf("parse HTML: %w", err)
	}
	cards := doc.Find("article.product_pod")
	if cards.Length() == 0 {
		return nil, fmt.Errorf("no book cards found")
	}
	books := make([]Book, 0, cards.Length())
	for _, node := range cards.Nodes {
		card := goquery.NewDocumentFromNode(node)
		link := card.Find("h3 a").First()
		title, _ := link.Attr("title")
		href, _ := link.Attr("href")
		price := strings.TrimSpace(card.Find(".price_color").Text())
		title = strings.TrimSpace(title)
		href = strings.TrimSpace(href)
		if title == "" || href == "" || price == "" {
			return nil, fmt.Errorf("book card has a missing required field")
		}
		target, err := base.Parse(href)
		if err != nil || (target.Scheme != "http" && target.Scheme != "https") {
			return nil, fmt.Errorf("book card has an invalid link")
		}
		books = append(books, Book{Title: title, Price: price, URL: target.String()})
	}
	return books, nil
}

func fetchBooks(ctx context.Context, client *http.Client, target string) ([]Book, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
	if err != nil {
		return nil, err
	}
	req.Header.Set("User-Agent", "GoBookTutorial/1.0")
	resp, err := client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("fetch page: %w", err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("unexpected HTTP status: %s", resp.Status)
	}
	body, err := io.ReadAll(io.LimitReader(resp.Body, maxBody+1))
	if err != nil {
		return nil, fmt.Errorf("read body: %w", err)
	}
	if len(body) > maxBody {
		return nil, fmt.Errorf("response exceeds 2 MiB")
	}
	return parseBooks(bytes.NewReader(body), resp.Request.URL)
}

func run(w io.Writer, target string) error {
	client := &http.Client{Timeout: 15 * time.Second}
	defer client.CloseIdleConnections()
	books, err := fetchBooks(context.Background(), client, target)
	if err != nil {
		return err
	}
	encoder := json.NewEncoder(w)
	encoder.SetIndent("", "  ")
	return encoder.Encode(books)
}

func main() {
	target := "https://books.toscrape.com/"
	if len(os.Args) > 1 {
		target = os.Args[1]
	}
	if err := run(os.Stdout, target); err != nil {
		log.Fatal(err)
	}
}

Run the program and write its JSON output only after a successful extraction:

go run . > books.tmp.json && mv books.tmp.json books.json

On the checked first page, the first record was “A Light in the Attic,” with a displayed price of “£51.77” and an absolute product URL. Inspect those three fields in your output before adapting the example. A readable JSON file alone does not prove that the selectors captured the intended values.

The temporary output file avoids replacing your last successful dataset with an empty file if the next request fails. Standard output contains the JSON document; errors go to standard error and produce a nonzero exit status.

The HTTP client has a 15-second timeout. The fetch function also limits the response to 2 MiB, a chosen tutorial bound rather than a universal page-size recommendation. It reads one extra byte so an oversized document becomes an explicit error instead of silently becoming truncated HTML.

The request’s context is passed through to the client. In a larger program, replace context.Background() at the calling boundary with the job’s context, so cancellation can stop in-flight work. The Go HTTP documentation also makes an important distinction: a response such as 404 can arrive without a transport error. That is why the status check is separate from err.

Parsing returns an error when the expected card structure or a required field disappears. This example rejects the page rather than producing a partially filled result. That is a policy choice you can change later, but do so explicitly: record rejected rows if your production pipeline accepts partial output.

Finally, product links are resolved against the final response URL. This matters when a page redirects or uses relative links. The code validates link schemes before exporting them; it does not follow those product URLs.

For web scraping with Go, these boundaries give you a useful baseline: request errors, page-size limits, selector failures, and output errors remain visible. The resulting records still need application-specific checks before you treat them as a finished dataset.

The one-page program does not crawl the entire bookstore. To inspect the next catalog page, pass its URL explicitly:

go run . https://books.toscrape.com/catalogue/page-2.html

Both catalog pages returned 20 records when checked for this tutorial. That is an observed result from the practice site, not a guarantee about future page layouts or other websites.

For automatic pagination, make discovery a separate step from record extraction. Parse the next-page link from the same downloaded document, resolve it relative to the response URL, and decide whether it belongs in the next request batch. Avoid fetching a page twice merely because extraction and navigation are separate functions.

A bounded pagination loop should do the following:

1. Begin with the approved catalog URL and a page limit.
2. Fetch and validate the current page.
3. Extract records and inspect the next-page link.
4. Resolve that link and require the expected scheme, host, and path.
5. Stop if there is no next link, the address was already visited, or the page limit is reached.
6. Wait according to the source’s operating limits before requesting the next page.

Check redirect destinations as well as discovered links if a crawl must remain on one origin. A same-origin check on the initial address alone does not constrain where an HTTP redirect sends the client.

Track visited page URLs separately from record identifiers. A repeated catalog page and a repeated book are different duplicate problems. Keep both checks understandable rather than deleting all query parameters and assuming that the addresses become equivalent.

Our GoSpider crawling guide covers a dedicated discovery workflow. When web scraping with Go grows beyond known addresses, that separation helps you tell whether missing records came from incomplete discovery or incorrect extraction.

Colly becomes useful when you are maintaining many request callbacks, navigation rules, and per-domain limits yourself. Its collector coordinates retrieval and handlers such as OnHTML, OnRequest, and OnError.

For the v2 module, use the versioned module path:

go get github.com/gocolly/colly/v2@v2.3.0

Import github.com/gocolly/colly/v2 in your Go source. The /v2 suffix matters; an older example importing the unversioned path is not selecting this module version. If you add Colly to the demonstration project later, keep the parser’s record rules consistent rather than changing the output schema at the same time.

Configure AllowedDomains to constrain destinations and use a LimitRule for the relevant domain. Check the error returned by Limit. Set a request timeout and register OnError so request failures do not disappear behind callbacks.

If you enable Async, call Wait before treating the job as finished. Protect shared slices and output writers, or hand records to one writer through a channel. A concurrent callback that appends to a shared slice is still concurrent code and needs a synchronization strategy.

Colly’s package reference documents these controls. Note that MaxDepth bounds link depth; it is not a total-page budget. A shallow page can link to many destinations.

For web scraping with Go, a framework can reduce scheduling code, but it does not choose correct selectors, validate business fields, or decide how complete a dataset must be. Keep those responsibilities explicit.

Inspect the received response before changing your extraction stack. A missing field can indicate a selector change, a different response page, or content that JavaScript has not rendered yet.

A headless browser can execute JavaScript and perform interactions. In Go, chromedp controls a browser through the Chrome DevTools Protocol. Its default behavior launches headless Chrome; browser execution still requires an appropriate browser environment.

Do not replace every HTTP request with a browser by default. First determine which content requires rendering, then wait for a relevant element or page state rather than relying only on an arbitrary sleep.

Likewise, do not treat every failed request as retryable. Set a retry budget, distinguish temporary failures from persistent rejection, and respect the source’s crawling instructions. A user-agent string identifies this tutorial client; it is not a promise of access.

Reliable web scraping with Go starts with diagnosing which stage failed. Changing selectors cannot fix a transport timeout, and increasing concurrency cannot repair a page that lacks the requested data.

The fetch and parse functions have separate inputs so you can test them without depending on a changing website. Feed parseBooks a small HTML fixture and a base URL. Check the full title attribute, original price string, and resolved product address against explicit expected values.

Then vary the fixture. Remove a price, rename the card class, omit the title attribute, and supply an invalid link. Each variation should produce the failure your pipeline expects. Include a page that returns successful HTTP status but contains no product cards.

For network behavior, use Go’s httptest package to serve controlled responses. Return 404 and 429, delay a response until the client timeout expires, and send a body over the configured size limit. These cases reveal failures that a single successful live request cannot.

The accompanying example was checked with local fixtures, go vet, and the race detector. The fixture suite covered successful JSON output and the error cases above, with 84.3% statement coverage. Two direct requests to the practice catalog separately verified the live selectors. This is functional validation, not a throughput benchmark.

Before adding parallelism, make the output path safe for multiple workers. Reuse a configured HTTP client, cap worker count, and keep a single owner for writing records. Preserve errors with enough page context to investigate them, while keeping credentials out of logs.

For web scraping with Go, increase one limit at a time and inspect valid-record counts as well as request counts. More completed requests are useful only when they produce the records your application needs.

• Separate retrieval from extraction. An HTTP client obtains a response; the parser turns its HTML into records. Keeping these functions independent lets you reproduce a selector failure from a saved fixture without making another live request.
• Define failure behavior alongside your output schema. Decide whether a missing title, empty price, or unexpected page should reject one row or the whole result. The example rejects the page, which makes a changed layout visible immediately.
• Bound navigation before automating it. Allowed destinations, visited addresses, page counts, and pacing solve different problems. Treat them as explicit controls when you extend the one-page command into a crawl.
• Add concurrency after you can trust the records. A worker limit and a single output owner make the next version easier to reason about. Monitor accepted records and rejected pages alongside network requests.

No. The HTML parser lives in the separate golang.org/x/net module. Go's standard html package and this parser have different import paths and responsibilities. goquery uses the external parser underneath its selection API. Check module requirements when adopting a tutorial rather than assuming every package maintained within the Go project ships with the toolchain.

Yes. Open the file and pass its reader to goquery.NewDocumentFromReader, checking both errors. Close the file in your calling code; the parser does not take ownership of closing it. When you need absolute links, retain the original page URL separately, because a local filename does not supply the website's URL resolution context.

Convert the input before parsing it. The golang.org/x/net/html/charset package provides NewReader, which returns a reader that converts HTML to UTF-8 using encoding detection and the declared content type. Handle conversion setup errors and keep response-size limits in your fetch layer. A fixture containing accented characters helps catch encoding problems that an ASCII-only test would miss.

Choose the representation for the consumer. If you need calculations, define the currency and decimal scale explicitly; an integer in minor units or a decimal representation may fit better than binary floating point. Preserve the original displayed value as a separate field when traceability matters. Do not remove punctuation blindly: grouping and decimal separators depend on the source's formatting conventions.

A useful first implementation of web scraping with Go has a narrow, observable contract. It requests a known page, validates the response, extracts a defined record shape, and reports failures. The bookstore example gives you that starting point with a runnable program and selectors checked against the practice catalog.

Your next step should follow the problem you actually encounter. Add pagination when records span known catalog pages. Introduce Colly when request scheduling and callbacks become substantial. Use browser rendering when the required content depends on JavaScript or interaction. In each case, keep the output schema and validation rules stable enough to compare results before and after the change.

If maintaining page retrieval becomes a separate operational burden, evaluate the Scraper API from WebScrapingAPI for that part of the workflow. Your Go application can retain responsibility for parsing, record validation, and storage. Compare the returned content against the same fixtures and field expectations you use with direct requests.

Start by running the example, inspecting its JSON, and changing one selector deliberately to see the failure. Understanding that failure path is a practical foundation for extending the scraper with confidence.