Your automation only synced the first 100 records

An automation that syncs only the first batch of records is asking the API for page one and never asking for page two. That much is easy to fix. The harder failure is the one that survives correct pagination: paging by offset through a table that is being written to while you page will skip records and duplicate others, because your page boundaries shift underneath you. Both failures share the property that makes them expensive. A partial sync exits cleanly and reports success, so nothing alerts you, and the gap is found weeks later by a person who noticed a customer missing from a report.

A partial sync is not an error, which is why it survives

Runs fail loudly. Runs that do the wrong amount of work do not. Your workflow requested a list, got a valid 200 response with valid records, processed all of them, and finished. Every step is green. There is no retry to trigger, no error branch to catch it, and no signal in the execution history that distinguishes 100 of 4,000 from 4,000 of 4,000.

This puts partial syncs in a different category from the failures that break an automation silently. Those at least leave a trace somewhere. A truncated sync leaves a perfect record of a successful run. If you monitor for failures, and most operators monitor only for failures, you will never see it.

The number is usually smaller than people assume. "First 100" is the folk version. Stripe's list endpoints return 10 objects when you do not pass a limit. HubSpot's CRM search endpoints also default to 10. If you built a customer sync in a hurry and never set a page size, you did not get 100 of 4,000. You got 10.

The caps that decide how much you actually lost

Before you can reconcile anything, you need to know what each layer of your stack will hand back when you ask for everything. These are the current documented limits, checked in August 2026.

Source	Records per response if you set nothing	Hard cap
Stripe list endpoints	10	100 per page, cursor via starting_after beyond that
HubSpot CRM search	10	100 per page, and 10,000 total results per query
Airtable list records	100	100 per page, offset token beyond that
Salesforce SOQL over REST	2,000	2,000 per batch, dropping to 200 when the query selects two or more long text custom fields
Looping by Zapier	n/a	500 iterations per run

Two rows in that table deserve attention.

HubSpot's search ceiling is the one that catches teams who did everything right. Pagination is configured, the cursor advances, the loop terminates normally, and you still stop at 10,000 results because that is where the endpoint stops. It is not a page size you can raise. HubSpot's own developer community threads are full of the same answer: slice the query into date ranges, or use the export API. A correctly built loop against a 40,000 contact database returns 10,000 records and a clean exit code.

Salesforce's 200 record collapse is the one that bites intermittently. Your query runs at 2,000 per batch for months. Someone adds a second long text custom field to the SELECT list and the batch size drops by an order of magnitude. If your loop has a page cap, and it should, the cap you sized for 2,000 record batches now cuts the sync off partway through.

Correct pagination still loses records

Here is the part the setup guides skip. Turning pagination on solves the "page two was never requested" problem. It does not solve the problem that a page is a position, not a thing.

Offset paging asks for records 0 to 99, then 100 to 199, then 200 to 299. That arithmetic is only correct if the underlying list is identical at each request. It never is. Say you page a contacts table sorted by created date descending, and 12 contacts are created between your first request and your second. Every existing record shifts down 12 positions. Your request for offset 100 now returns records that were at position 88 in the list you already fetched. You get 12 duplicates.

Deletions do the reverse and are worse. Records shift up, positions slide past your cursor, and those records are never returned. Slack's engineering team documented this exact behavior when they moved their API off offset paging: with items being written during pagination, the page window becomes "unreliable, potentially skipping or returning duplicate results."

The worst combination in day-to-day automation work is offset paging sorted by last-modified descending, which is exactly how most incremental syncs are built. The sort key moves. A record you already fetched on page one gets touched by anything, a rollup, an enrichment job, a rep opening it, and it jumps back to the top of the sort. Everything below shifts down by one, and one unfetched record slides past your cursor permanently. Every write that happens during your sync window costs you a record, and you get duplicates in the same run to cover the tracks.

Three rules fix it:

Use the vendor's cursor whenever one exists. Stripe's starting_after, HubSpot's paging.next.after, Salesforce's nextRecordsUrl, Airtable's offset token. These anchor to a record, not to a position, so inserts and deletes elsewhere in the list do not move your place.

When you must use a numeric offset, sort by something immutable. Record ID or created date, ascending. Never last-modified. If the sort key cannot change, positions cannot shuffle underneath you.

Overlap your incremental window instead of butting it up against the last run. Pull records modified since the last run minus five minutes, and dedupe on ID at the destination. Overlap costs you a handful of redundant writes and buys you immunity from clock skew and mid-run edits. Handle the redundant writes the same way you would stop any automation from creating duplicates, with an upsert keyed on a stable ID rather than a create.

Four stop conditions, and the one that prevents a runaway

Most pagination loops fail at the exit, not the entry. The condition to use, in order of reliability:

Stop when the returned count is less than the page size you requested. This is the one universal signal. If you asked for 100 and got 61, that was the last page. It works on APIs that give you no cursor, no total, and no has_more flag.

Stop when the next pointer is missing or unchanged. Check has_more on Stripe, the presence of paging.next on HubSpot, done on Salesforce. Then also compare the new cursor against the previous one. A cursor that returns itself is the single most common cause of an infinite pagination loop.

Do not stop on "the response is empty." This is the default many people reach for and it is a trap, because the response is not empty. It is a wrapper object containing an empty array. Test the array at its actual path, not the body.

Always set a maximum page count. Size it at roughly three times what you expect and fail loudly when the loop hits it, rather than exiting quietly. n8n exposes $pageCount, $request, and $response inside the HTTP Request node's pagination settings, so the cap is a one-line expression. Zapier enforces one for you at 500 loop iterations, and note the cost shape there: the loop step itself is free, but every action step after it consumes one task per iteration, so a 500 iteration loop with two follow-on steps is 1,000 tasks on a single run.

Reconcile counts, not runs

Everything above reduces the chance of a partial sync. Only one thing detects one. At the end of every sync, compare the number of records you processed against the number the source says it has, and fail the run when they differ.

Most APIs hand you the total. Salesforce returns totalSize on a query response. HubSpot search returns total. Where no total exists, and Stripe is the notable case, run a cheap count query against the source or compare the destination row count between consecutive runs and alert on an implausible delta.

Then wire that comparison into an alert, because a check that writes to a log nobody reads is not a check. The same alerting you use for automations that stop running covers this, with one difference worth internalizing. Silence means failure for a scheduled job. For a sync, success means nothing at all. Only the count comparison carries information.

Tolerance is a judgment call. On a source that is actively being written to, an exact match will produce false alarms, since records get created between your fetch and your count. We usually set the threshold at one percent or five records, whichever is larger, and treat any run outside it as a failure that pages someone.

What to check this week

Pick your highest-volume sync, the one that moves customers, contacts, orders, or invoices. Find the last successful run. Count the records it processed. Then open the source system and count how many records matched the same filter. If those two numbers are not within a percent of each other, that sync has been lying to you, and the gap is as old as the workflow.

Do the same for every sync that feeds a report or a billing process, since those are the two places a missing record turns into a wrong decision or a wrong invoice. Then add the count comparison as a permanent last step so you never have to run the audit by hand again. It is fifteen minutes of work per workflow and it converts a class of failure that is invisible by default into one that alerts.

We build the reconciliation step into every sync we ship as part of a workflow automation engagement, and when a client's records are the thing being counted, the count check belongs in the data layer rather than bolted onto each individual flow. If you already suspect a sync has been dropping records and you want a second read on where the gap is coming from, tell us which two systems it moves data between.