Skip to main content

Paging

Many JSON APIs implement a form of data paging, where they limit the number of records returned in the response. These can be implemented in many different ways and we have built a PagingRequest function to help handle the most common scenarios.

Page Counter​

The most simple option is the Page Counter where we increment the page number until the response returns an empty response.

In the XML file this wold be implemented like the below:

<PagingRequest path="data" start="1">
  <Fetch url="{URL}/app/list/{Datasource.Name}?limit={PageSize}&amp;page={PagingRequest.Page}">
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>

Next Page Token​

Another method is the next page token, which is based on extracting the value for the next page from the data and passing it into the next request until the token is empty.

In the XML file this would be implemented like this:

<PagingRequest path="data" start="1" next-path="nextpage">
  <Fetch url="{URL}/app/list/{Datasource.Name}?limit={PageSize}&amp;page={PagingRequest.Next}">
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>

Alternatively if the Next Page token returns a URL to the next page of results then you can use it like the example below.

Where you pass the first page in the next token which sets up the initial request and the next page URL is returned from next-path token. The paging process will complete when the value of the next-page token returns null or empty string.

<PagingRequest path="data" start="1" next="{URL}/app/list/{Datasource.Name}?limit={PageSize}" next-path="next">
  <Fetch url="{PagingRequest.Next}">
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>

Next Page Token with Continuation Parameter​

info

Available from Data Sync version 6.0.3532

This is a variation of the Next Page Token for when the last page returns the current page token instead of null to signal the end. It uses another value, instead of null, to determine if it should continue or not.

The example below uses while-path and while-path-value which are used with a string entity check to decide if the loop should continue. This loop will continue all the time meta.has_more equals True.

<PagingRequest path="data" start="1" next="{URL}/app/list/{Datasource.Name}?limit={PageSize}" next-path="next" while-path="meta.has_more" while-path-value="True">
  <Fetch url="{PagingRequest.Next}">
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>

Skip & Take​

If your API is based on a Skip and Take page counter then you can use the PagingRequest like this. Where you use the internal PagingRequest.Count counter to manage the row index. start is used to set the initial counter value. The process continues until the API returns an empty response.

<PagingRequest start="0" path="data">
	<Fetch url="{URL}/app/list/{Datasource.Name}?skip={PagingRequest.Count}&amp;take={PageSize}">
		<DataTableTransform path="data" />
	</Fetch>
</PagingRequest>

Indexed Paging​

info

Available from Data Sync version 6.0.3345

Indexed paging is based on paging through rows using an Index. Each iteration of the page we move the index along according to the page size parameter value. The paging stops when the API returns an empty response or HTTP status code 204.

So you will see requests like this:

  • GET /app/list/products?startIndex=0&endIndex=10
  • GET /app/list/products?startIndex=11&endIndex=20
  • GET /app/list/products?startIndex=21&endIndex=30
  • GET /app/list/products?startIndex=31&endIndex=40

You can configure the paging to support indexing by using the below example as a guide. This uses the StartPageIndex and EndPageIndex to list the range to be returned within the Fetch URL.

<PagingRequest page-size="{PageSize}">
  <Fetch url=" http://json-test.lab.simego.com/IndexPagedApi?startIndex={PagingRequest.StartPageIndex}&endIndex={PagingRequest.EndPageIndex}"> 
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>