Extracting Data from APIs: A Reliable First Pipeline

Write down the row you intend to deliver before building a loop. For the fictional catalog, each accepted row needs a stable product ID, a nonempty name, a nonnegative integer price in cents, and a currency of EUR. These are exercise-specific rules, not assumptions to apply to every provider.

Separate the source contract from the destination schema. The source might represent money as decimal text, an integer, or a formatted display value. Preserve that meaning explicitly when mapping to the destination. A field named price does not tell you its unit, tax treatment, or currency by itself.

Duplicate policy deserves an explicit choice. The exercise keeps the first valid occurrence because its fixture repeats an identical product. A real changing catalog may require a comparison of timestamps or a conflict record instead. Deduplication should implement that chosen policy, not hide disagreement between observations.

When extracting data from APIs, distinguish a rejected record from a failed page. One invalid item may be quarantined while other items remain usable under your policy. An unrecognized page envelope can prevent you from trusting both the items and the continuation value, so continuing blindly would be a different risk.

Finally, define the collection scope. Record the account, filters, period, and endpoint version where relevant. An accurate record outside that scope is still the wrong output for the job.

Use three boundaries in the implementation. Retrieval obtains a response from the provider. Decoding interprets the representation. Acceptance validates and maps business records. Each boundary should return a useful result or a classified failure rather than quietly replacing a problem with an empty collection.

The retrieval layer handles the documented request, credentials, transport timeouts, and HTTP response status. It should expose enough sanitized evidence to diagnose a failure. Do not make the record validator responsible for guessing whether an HTML error page was caused by a timeout or a login requirement.

The decoding layer checks the expected format and envelope. JSON parsing establishes that the text follows JSON syntax; it does not prove the result is a product page. The JSON specification defines values and structure, leaving field meaning to the application contract.

For this exercise, a page has an items list and a next value that is either null or a nonempty string. An unexpected envelope is a page failure. Each item is then checked independently against the product rules before it reaches the accepted output.

These boundaries make extracting data from APIs easier to test. A local fixture can replace retrieval while the production decoding and acceptance logic remain exercised. You can reproduce a malformed row or repeated cursor without depending on a provider to generate that condition on demand.

When adding a real client, keep its responsibilities narrow. The Python HTTP client comparison is useful for evaluating request controls, but no client library can infer your definition of a valid product. Keep that definition in a separate, reviewable function.

Also avoid catching every exception and returning []. That pattern makes unavailable infrastructure indistinguishable from a valid empty catalog. A caller should be able to tell whether the provider returned no matching records or the extractor could not establish a valid result.

Save the following as extract_fixture.py and run it with Python 3. It uses only the standard library and makes no network requests. The fake pages include two valid products, one invalid price, and an identical duplicate. The final assertions also verify that a repeated continuation cursor fails visibly.

from copy import deepcopy

PAGES = {
    None: {"items": [
        {"id": "p1", "name": "Lamp", "price_minor": 2400,
         "currency": "EUR"},
        {"id": "bad", "name": "Chair", "price_minor": True,
         "currency": "EUR"},
    ], "next": "page-b"},
    "page-b": {"items": [
        {"id": "p1", "name": "Lamp", "price_minor": 2400,
         "currency": "EUR"},
        {"id": "p2", "name": "Book", "price_minor": 650,
         "currency": "EUR"},
    ], "next": None},
}

def valid_record(row):
    return (
        isinstance(row, dict)
        and isinstance(row.get("id"), str) and bool(row["id"].strip())
        and isinstance(row.get("name"), str) and bool(row["name"].strip())
        and type(row.get("price_minor")) is int
        and row["price_minor"] >= 0
        and row.get("currency") == "EUR"
    )

def collect(fetch_page, max_pages=5):
    cursor, visited, accepted = None, set(), {}
    rejected = duplicates = 0
    for _ in range(max_pages):
        if cursor in visited:
            raise ValueError("pagination cursor repeated")
        visited = visited | {cursor}
        page = fetch_page(cursor)
        if not isinstance(page, dict) or not isinstance(page.get("items"), list):
            raise ValueError("invalid page envelope")
        if "next" not in page or not (
            page["next"] is None
            or isinstance(page["next"], str) and bool(page["next"])
        ):
            raise ValueError("invalid continuation")
        for row in page["items"]:
            if not valid_record(row):
                rejected += 1
            elif row["id"] in accepted:
                duplicates += 1
            else:
                accepted = {**accepted, row["id"]: dict(row)}
        cursor = page["next"]
        if cursor is None:
            return list(accepted.values()), rejected, duplicates
    raise ValueError("page budget exhausted before completion")

rows, rejected, duplicates = collect(lambda cursor: deepcopy(PAGES[cursor]))
assert [row["id"] for row in rows] == ["p1", "p2"]
assert (rejected, duplicates) == (1, 1)
assert not valid_record({"id": "x", "name": "X",
                         "price_minor": True, "currency": "EUR"})
try:
    collect(lambda cursor: {"items": [], "next": "same"})
except ValueError as error:
    assert str(error) == "pagination cursor repeated"
else:
    raise AssertionError("repeated cursor was accepted")
print("2 accepted, 1 rejected, 1 duplicate; cycle check passed")

The fake transport returns a copy so extraction does not modify the fixture. The collector validates the envelope before processing rows, tracks cursors separately from product identities, and accepts only records that meet the declared rules. A repeated page and a repeated product are different observations.

Notice the numeric type check. In Python, booleans are a subclass of integers, so a broad integer-instance check would accept True as a price. This fixture deliberately excludes it with an exact type comparison. Other languages and schema tools need equivalent attention to coercion and accepted types.

The page budget is a protective bound, not a claim of completeness. If it expires before the null continuation appears, the exercise raises an error. It never returns a normal finished result just because the loop reached its configured maximum.

This is a teaching extractor, not a production storage system. It accumulates accepted records in memory and raises on a page failure rather than durably retaining partial progress. Before extracting data from APIs at scale, replace those limitations with a sink and run record that preserve already accepted work.

The next local tests should reflect your contract: a missing continuation key, a non-list items field, and conflicting duplicates. Keep expected outcomes explicit. The goal is to test decisions the real source can force, not to add a large test suite that merely repeats the implementation.

The exercise uses cursor values, but providers publish different pagination contracts. Some return links, some use offsets, and others require page numbers or continuation tokens. GitHub's pagination documentation illustrates link-based continuation for that service; it is an example, not a universal API format.

Preserve opaque continuation values exactly as the provider directs. Keep relevant filters and account context attached to the work item. A token generated for one request scope should not be treated as a free-standing instruction that applies to any subsequent query.

Validate URL-based continuation before following it. A response-provided next link should remain within the destinations your integration is authorized to contact, and redirects need the same boundary checks. Do not forward a credential to an unrelated host simply because a response included that URL.

When extracting data from APIs into durable storage, commit accepted data before advancing the checkpoint. If the checkpoint moves first and storage fails, a restart can skip records. If data is stored first and the checkpoint update fails, the next attempt may replay a page, which a deliberate duplicate policy can handle.

An atomic transaction can help when the sink and checkpoint share a storage system. If they do not, document how replay and reconciliation work. Avoid claiming exactly-once processing simply because the happy path stores one row per source identifier.

Capture the ending reason. A source-provided final page, a page budget, a deadline, and a rejected response are separate outcomes. A report containing only the accepted count cannot explain whether the job covered its intended scope.

Also test the provider's assumptions about changing data. An offset-based listing can move while you traverse it, and a cursor may expire before a later restart. Record the collection window and use a documented snapshot or incremental mechanism when the task requires stronger consistency than a live traversal offers.

A retry is another attempt at an operation, not a correction to its inputs. Repeating a request with invalid credentials or malformed parameters generally repeats the problem. Begin by classifying the failure and identifying what, if anything, could change before another attempt.

For HTTP, the method semantics in RFC 9110 help distinguish safe retrieval from operations whose repetition can have business effects. Even for reads, keep attempts bounded and respect the provider's documented behavior. Do not enable a general retry feature without checking which errors and operations it repeats.

Rate limits require attention to aggregate load. A 429 response signals a limiting condition; the counting scope is defined by the service, and the response can include delay guidance. RFC 6585 defines the status. Several workers sharing credentials may consume the same allowance.

Choose a retry owner. If the client, intermediary, and job runner all retry invisibly, one intended request can become more traffic than the collector records. Make attempt counts and elapsed job time visible enough to investigate that multiplication.

When extracting data from APIs, maintain a separate status for partial collection. If three pages were accepted and the fourth failed, preserve the accepted data according to policy and record the missing scope. Do not manufacture a final-page marker or deliver an unqualified “complete” dataset.

Protect the failure report itself. Store sanitized request identifiers, response categories, and schema errors. Avoid dumping credentials, full account records, or unrestricted bodies into a general log simply because something went wrong.

Finally, separate repeated retrieval from repeated delivery. If the downstream export fails after collection succeeded, retrying the export from stored records preserves the original observation. Fetching the source again creates a different run whose values and completeness may differ.

An accepted row should carry enough context to explain where it came from. Depending on the task, that includes source identity, collection time, request scope, endpoint version, and transformation version. Keep source modification time separate from the time your collector observed the record.

Validate the delivery boundary as well as the source response. A correct integer can become text in an export, a null can become an empty cell, or a consumer can interpret a timestamp in a different timezone. Define the output schema and check a representative file or message before publishing a whole run.

For extracting data from APIs on a schedule, add a small change monitor. Compare rejection counts, required-field presence, and completion behavior with the expectations for that source. A sudden drop in accepted records deserves investigation, but it is not automatically an extraction bug; source filters or inventory may have changed legitimately.

Keep an approved fixture for maintenance. When a provider changes an optional field, a test should show whether your mapping ignores it safely or needs an update. When a required field changes meaning, revise the contract rather than adding a broad fallback that produces plausible output.

The automated web scraping guide connects these acceptance and delivery boundaries to the wider job lifecycle. The same operational discipline applies whether the input arrives from a documented API or a parsed webpage.

Finish the first implementation with a run report someone else can use: accepted, rejected, duplicate, and failed-page counts; the ending reason; and the location of the validated output. That report turns a working script into a collection process whose limitations remain visible.

• Define field meaning, identity, duplicate policy, and completion before extracting data from APIs in a loop.
• Keep retrieval, decoding, and business validation separate so transport failures cannot become empty success results.
• Test repeated cursors, invalid rows, and page budgets locally before connecting the extractor to a live provider.
• Persist accepted data before advancing a checkpoint, and make restart behavior compatible with deliberate replay.
• Preserve partial-run evidence and validate the final delivery format as carefully as the source response.

Not necessarily. Collecting a source observation and transforming stored data are different jobs. Separating them can let you correct a mapping without making another source request. Decide how long raw observations should be retained and which transformation versions consumers need, then schedule each step according to those requirements.

Follow the compatibility policy you chose for the source. A collector that ignores unknown optional fields may continue unchanged, while one preserving a complete raw document can retain them for later review. Do not automatically expose every new field to downstream consumers without checking its meaning and access implications.

Not without an explicit mapping. CSV does not inherently retain nested objects, arrays, or distinctions such as null versus empty text. Flatten or encode those values deliberately, document the convention, and test the consumer's interpretation. Keep a richer source representation when later processing may need information the table omits.

Begin extracting data from APIs with a small contract and a failure-aware loop. The local exercise demonstrates the key decisions: accept only defined records, treat duplicate products separately from repeated cursors, and require a real ending condition. A real HTTP client adds transport concerns without replacing those rules.

Before scheduling the job, give accepted data a durable destination, make the checkpoint safe to replay, and produce a report that distinguishes complete and partial runs. Those controls let you increase scope without making missing data harder to recognize.

If some required fields are available only through webpages, evaluate WebScrapingAPI's Scraper API as an additional input. Keep the same acceptance, provenance, and delivery contract around that data so the pipeline remains consistent even when collection methods differ.