From reading to action
A practical path through this guide.
Select a step to explore what to do and why it matters.
Read the pagination contract
Confirm cursor behavior, ordering and token expiration.
A reliable paginated sync records progress durably and can resume without skipping unfinished work. Fetching the first page successfully is only the beginning. Design for interrupted runs, changing source data, and repeated pages before increasing the dataset size.
Understand the pagination contract
Determine whether the provider uses offsets, page numbers, cursors, or continuation tokens. Check ordering guarantees and whether the token expires. Avoid assuming that an endpoint's default order remains stable while records are added or updated.
An offset-based export can shift under concurrent inserts. A cursor may be safer for some APIs, but its exact guarantees are provider-specific. For a critical migration, consider a bounded snapshot or time window and reconcile the final record count.
Commit work before progress
A safe sequence is fetch page, validate records, persist or complete the page's work, then store the next checkpoint. If you save the checkpoint first and crash before writing the records, the next run may skip data silently.
If some records fail, decide whether the page stops or individual records enter a durable exception queue. The checkpoint must not imply those failures were completed. Keep their source identifiers and error reasons available for targeted replay.
Separate identity from position
Use source record identifiers for deduplication, not page position. The same record may legitimately appear again after an overlap query or a restarted page. Updates should follow an explicit version or timestamp rule so older data does not overwrite newer destination state.
For incremental sync, define the boundary carefully. Records can share timestamps, and clocks can differ. An overlap window plus deduplication can reduce missed changes, while periodic reconciliation checks whether that strategy remains sufficient.
Verify the resume behavior
- Stop immediately after a page fetch.
- Stop midway through destination writes.
- Stop after writes but before saving the checkpoint.
- Add or change source records during the run.
- Resume using an expired or invalid continuation token.
Expected results should include the final destination set, not just the number of successful requests. Compare missing identifiers, duplicates, and changed fields. Keep a reconciliation report until the business owner accepts the sync outcome.
Further reading
The webhooks vs polling guide explains when scheduled synchronization fits. HubSpot's API documentation is a starting point for checking endpoint-specific pagination behavior.
A few practical answers
Frequently asked questions
When should a pagination checkpoint be saved?
Save progress after the page work is durably recorded or completed. Advancing before writes finish can skip records permanently when the process crashes.
Why should row position not be used as a deduplication key?
Positions can change as data is inserted or reordered. Use source record identifiers, and apply explicit update rules when a record appears again.
