n8n HTTP Request pagination: verify every record arrives

SOURCE-BASED ANALYSIS · NO HANDS-ON RATING

A successful request can still be an incomplete import

The n8n HTTP Request node has a Pagination option. It supports updating a parameter for each request or following a next-page URL supplied in the response. Before selecting a mode, inspect the destination API's documentation and an actual response. Page numbering, page-size limits, cursor fields and stopping rules are properties of that API, not universal n8n defaults.

This guide's recommendation is to define completeness before building the import. A green node and a plausible list of contacts do not establish that the full source collection arrived. Decide which source IDs you expect to receive, which filter defines the collection and where the final reconciliation will be recorded.

Worked example: 235 synthetic contacts

Assume a fictional read-only API contains exactly 235 contacts in a frozen test collection. Assume it supports a page size of 100 and a reliable next-page pointer. The expected page lengths are 100, 100 and 35. That gives three non-empty page responses and 235 unique IDs. These numbers illustrate a contract; they are not an API limit, runtime measurement or product benchmark.

Create a worksheet with response number, requested cursor or URL, returned item count, next pointer and cumulative unique IDs. The expected cumulative counts are 100, 200 and 235. If the last count is 200, the import is incomplete even if every request returned a successful status. If raw count is 235 but unique count is 220, some items repeated and the result is still unresolved.

Implementation sequence

1. Call the API once without pagination using a read-only credential and a small test collection.
2. Identify the records array and the documented next-page mechanism. Keep pagination metadata separate from contact fields.
3. Choose the matching Pagination mode in HTTP Request and map the parameter or next URL from the real response structure.
4. Set the completion rule from the API contract. Add a deliberate upper bound during development so a repeated cursor cannot run indefinitely.
5. Collect IDs across pages and compare the unique count with an independently known test total.
6. Only connect destination writes after the read-only import passes the empty, single-page and multi-page cases.

Handle changing collections deliberately

The frozen collection assumption matters. If records are created or deleted while a page-number import runs, a count mismatch may reflect source changes rather than a broken expression. Where supported, use a stable snapshot or a consistent filter boundary. Otherwise record start and end times and reconcile changes in a second pass.

Do not invent a cursor from a visible record ID unless the API explicitly defines that relationship. Do not assume an empty page is always the right termination signal. Write a short contract note with the exact provider field names and link it to the workflow's operational instructions.

Recovery and acceptance

Suppose page two fails after page one was read. If no destination writes have happened, a bounded restart of the read-only exercise is easier to inspect. If page one has already created records, recovery must account for those side effects. Use a destination key based on the source ID and reconcile existing records before replaying writes.

Accept the import when all expected synthetic IDs arrive once, the final page is handled, a repeated cursor stops safely, and an interrupted run has a documented restart procedure. Then repeat a representative check when the provider changes its API or you change filters. This proposed procedure checks completeness; it does not establish throughput or suitability for a particular account size.