Pagination in Web Scraping: 5 Tested Python Patterns

Pagination in web scraping means moving through every relevant batch of a list rather than collecting only the first page. The control may be a numbered URL, a Next link, an API cursor, a Load more button, or an infinite-scroll trigger. Each pattern needs different navigation logic, but every reliable scraper also needs a clear stop condition, duplicate detection, and a safety limit.

This guide implements and tests all five patterns with Python. The HTTP examples use Requests and Beautiful Soup; the interactive examples use Playwright.

Quick Answer

To handle pagination in web scraping, first identify how the site requests the next batch. Iterate page numbers for predictable URLs, follow the actual Next link when the site supplies one, pass an API's returned cursor unchanged, or use a browser for Load more and infinite scroll. Stop only on a verified end signal, and always cap requests, detect loops, and deduplicate records.

Key Takeaways

• Inspect the pagination mechanism before choosing a tool. A button may simply call a JSON endpoint that does not require browser automation.
• Do not treat all pagination as page numbers. Cursor, offset, button, and scroll-based interfaces have different continuation tokens and failure modes.
• Use the site's continuation signal. Follow the supplied Next URL or cursor instead of guessing how to construct it when possible.
• A missing selector after an error is not proof that the dataset ended. Validate the response before accepting any stop condition.
• Put a hard cap on every loop. Maximum pages, clicks, scroll rounds, and elapsed time prevent bugs from creating unbounded crawls.
• Deduplicate with stable record IDs or canonical URLs. Page contents can shift while a crawl is running.
• A proxy changes the network route, not the pagination logic. Add one only when the workflow needs its location, IP type, or routing characteristics.

Tested Examples at a Glance

We ran the complete examples on September 29, 2026. Four use TestingURL.dev, a purpose-built public scraping sandbox with 60 stable demonstration products. The cursor example uses the OpenAlex pagination API and deliberately stops after a small sample.

The test environment used Python 3.11.2, Requests 2.34.2, Beautiful Soup 4.15.0, Playwright 1.63.0, and Chromium 153.0.8010.12. These results establish that the published control flows worked against the named targets on the test date. They are not a performance benchmark, and counts may change if a target changes.

What Is Pagination in Web Scraping?

Pagination divides a collection into smaller batches. A catalog with 2,000 products might return 20 items at a time so the initial page loads quickly and the application does not send the full dataset at once.

For a human, moving forward may look like clicking 2, clicking Next, pressing Load more, or scrolling. For a scraper, the important question is what network operation happens next:

The extraction logic may stay the same across batches. The navigation and termination logic does not.

How to Identify the Pagination Type

Open the page and then inspect both its links and its browser network requests.

In Chromium developer tools, open the Network panel, filter for Fetch/XHR, and perform one pagination action manually. Compare the request URL, method, parameters, cookies, and response structure. Do not assume that a button requires a browser: it may be a thin interface over a permitted JSON endpoint.

Equally, do not copy an internal endpoint blindly. It may depend on authentication, CSRF tokens, signed parameters, or terms that do not permit direct automated access. Use only data and interfaces you are authorized to collect.

Install the Python Tools

Create an isolated environment and install the packages used below:

On Windows PowerShell, activate the environment with:

The browser download is separate from the Python package installation. On a supported Linux system that reports missing browser libraries, Playwright also provides:

For a production project, pin versions after testing rather than accepting future upgrades without verification.

1. Scrape Numbered Pages With Python

Numbered pagination exposes a finite set of page URLs such as:

The safest implementation discovers the available range from the site's pagination controls instead of inventing an arbitrary last page. This complete script reads the numbered links on page 1, enforces a maximum, visits each page, and rejects duplicate product IDs.

Observed output:

This pattern is appropriate only when the visible page range is authoritative. If the site renders only a small window such as pages 1, 2, 3, ..., 100, follow the Next link instead of assuming that the visible numbers are complete.

2. Follow Next Links Instead of Guessing URLs

A Next link is often more robust than manually incrementing page or start. The server decides the next URL, so the client does not need to reproduce its pagination formula.

The practice endpoint below uses start and limit internally, but the scraper only follows the Next link. It also tracks page URLs to detect loops and refuses to leave the expected origin.

Observed output:

On a different site, the link might be identified by a[rel="next"], an accessible label, or a site-specific class. Prefer semantic attributes over brittle selectors based only on position.

3. Scrape a Cursor-Paginated API

Cursor pagination returns an opaque token representing where the next query should continue. The token might appear as next_cursor, after, continuation, or another provider-specific field.

Do not decode, edit, or increment an opaque cursor. Send back exactly what the API returned.

The tested example requests two small pages from the OpenAlex Works API. OpenAlex starts cursor traversal with * and returns the next token in meta.next_cursor. The two-page cap makes the output intentionally partial; it does not pretend to download the full dataset.

Observed output:

For a complete permitted collection, store the cursor after every successful batch. If the process stops, resume from the last committed cursor rather than starting over. A cursor can expire or become invalid, so production code must also define what happens when the API rejects a saved token.

OpenAlex explicitly recommends its snapshot rather than cursor-walking the complete dataset. The same principle applies elsewhere: use a provider's bulk export when one exists instead of turning a transactional API into an unbounded download job.

4. Handle a Load-More Button With Playwright

A Load more button commonly triggers JavaScript, updates the current DOM, and then disappears or becomes disabled at the end. Waiting a fixed number of seconds after each click is unreliable. Wait for evidence that the item count increased.

This script records the count before each click, waits for more matching elements, enforces a click cap, and verifies that product IDs remain unique.

Observed output:

Button disappearance is valid here because the target explicitly removes it when hasMore becomes false. On another site, disappearance could mean a JavaScript error, blocked request, changed selector, or modal overlay. Pair the UI state with successful network responses and item validation before declaring the crawl complete.

5. Handle Infinite Scroll With Playwright

Infinite-scroll pages usually observe the viewport or a sentinel near the bottom of the list. When the sentinel becomes visible, the page fetches another batch.

Scrolling until document.body.scrollHeight stops changing is a useful fallback, but it is not a strong end signal by itself. Ads, images, animations, and virtualized lists can change height without adding records. The practice page exposes both an item container and an explicit end message, so the scraper checks those states directly.

Observed output:

Virtualized lists need another approach. They may remove older DOM elements as new ones appear, so the number of visible cards does not represent the number collected. Capture and deduplicate each batch as it appears, or use the underlying permitted data endpoint when that is the more stable interface.

Browser Automation or the Underlying API?

Load-more and infinite-scroll interfaces often call a JSON API. If the endpoint is documented or otherwise permitted for your use, requesting it directly can reduce browser compute, transferred assets, and selector failures.

Use Requests or another HTTP client when:

• the content is present in server-rendered HTML;
• the application exposes a stable, permitted JSON request;
• navigation does not depend on browser-only state; and
• you can reproduce required authentication and session handling correctly.

Use Playwright when:

• JavaScript must run to obtain the data;
• a click, scroll, consent flow, or other browser interaction is necessary;
• the browser creates short-lived request values that should not be reimplemented; or
• you need to validate the rendered user experience, not only the response payload.

Browser automation is not automatically more reliable. It introduces browser versions, rendering, event timing, memory use, and extra network requests. Choose it because the workflow needs a browser.

Reliable Pagination Stop Conditions

A production scraper needs both a natural stop signal and a safety budget.

Do not translate every empty result into success. An empty document could be a login page, an error template, a rate-limit response, or changed markup. Check the status code, final URL, expected page identity, content type, and required fields before accepting it as the end.

Prevent Missing and Duplicate Records

Pagination can change while a scraper is running. If new items are inserted at the beginning of an offset-based list, later offsets may repeat an item already collected or skip one shifted into an earlier batch. Deleted items can cause a similar shift.

Use these controls:

1. Prefer a stable identifier. Use an employer job ID, product ID, article ID, or canonical record URL. Do not rely on a title alone.
2. Deduplicate while collecting. Maintain a set or database uniqueness constraint keyed by that identifier.
3. Store the source URL and collection time. This makes unexpected records traceable.
4. Persist checkpoints after successful writes. Store the next page URL, offset, or cursor only after the current batch is committed.
5. Define consistency requirements. A cursor often reduces shifting compared with offsets, but it does not automatically provide a database snapshot. Read the API's stated semantics.
6. Reconcile important collections. Compare expected counts, revisit a small overlap window, or run a later consistency pass when completeness matters.

If a crawl stops after page 12, the checkpoint should let it resume from a known state without duplicating pages 1 through 11.

Handle Pagination Errors Without Hiding Them

Pagination amplifies small failures because one missed continuation can silently truncate the whole dataset.

Never treat a failed middle page as if the dataset simply ended. Record the failure, preserve its continuation state, and make completeness visible in downstream data.

Where Proxies Fit Into Paginated Scraping

A pagination scraper decides what batch to request next. A proxy decides which network route and exit IP the request uses. Those are separate responsibilities:

Start with a direct connection when it satisfies the permitted workflow. A proxy may become useful when the collection requires a specific observation country, an approved outbound route, or distributed network access. Residential proxies are generally the relevant Proxidize product for global, location-specific public web data collection. Mobile proxies are more specific to US mobile-network identity and mobile-oriented testing.

Session behavior matters more than aggressive rotation. Keep a stable session while following a cursor, preserving cookies, or completing one coherent paginated flow. Changing the exit between dependent pages can invalidate server-side state or change the result set. Provider-managed rotation is better reserved for independent jobs when the workflow calls for it; our IP rotation guide explains the difference between rotation triggers and exit-selection policies.

A proxy does not repair a wrong selector, invent a missing cursor, grant access to restricted data, or make an unbounded crawl acceptable. For the broader routing setup, see web scraping with proxies and the Playwright proxy configuration guide.

Common Pagination Scraping Problems

Pagination Scraping Checklist

Before running a paginated scraper beyond a small test:

• Confirm that automated collection is allowed for the target and data in scope.
• Identify the actual pagination pattern in HTML and network traffic.
• Use a timeout on every request and navigation.
• Define maximum pages, clicks, scrolls, records, and elapsed time.
• Follow supplied URLs and cursors rather than guessing when possible.
• Validate each response before extracting or stopping.
• Deduplicate with stable record identifiers.
• Persist records and continuation checkpoints incrementally.
• Record failed pages, cursors, and reasons.
• Respect rate limits and Retry-After responses.
• Keep session and location settings consistent during dependent flows.
• Monitor request count, transferred data, valid-record rate, and completeness.

Pagination is part of a wider crawl architecture. For frontier scheduling, revisits, canonicalization, and per-host controls, read What Is Web Crawling?. Proxidize's web scraping use-case guide explains where the network layer fits into a complete data-collection system.

Final Answer

There is no universal pagination loop. Numbered pages can be enumerated after validating the range; Next links should be followed as supplied; cursor tokens should be replayed unchanged; and interactive lists should wait for verified DOM or network progress rather than fixed sleeps.

The most important implementation details are the same across all five patterns: validate each batch, preserve continuation state, reject loops and duplicates, stop on explicit evidence, and enforce a hard safety budget. Add a browser or proxy only when the actual workflow requires that layer.