Deluge API Pagination to Get More Than 200 Records
- Peter
- Difficulty: Intermediate
- Estimated reading time: 10 mins
API Pagination
The api-pagination Deluge script enables you to handle pagination when calling an API. The Zoho CRM API is used here, but you can modify this to work with any API.
Core Idea
Many APIs (including the Zoho CRM API) limit the number of results you may receive per API response page, but there is still a need to retrieve additional records from the API. When calling an API you generally have the option to specify the number of results per page (though limited) and the specific page of results. For example in the query parameters of the API URL, you may specify: “?per_page=200&page=1”Deluge does not have native support for API pagination orn
whileloops, which is problematic. This script enables the ability to paginate API requests in Deluge. Configuration
Before you can use this script with Zoho CRM, you must specify the following in the script:
- Configured Zoho CRM Connection. Update the value for
connectionin theinvokeurlmethod. - Zoho CRM module you wish to query (e.g. “Contacts”, “Deals”, etc). Update the
zohoCrmModulevariable. - Number of API page results you wish to get per request. The limit for the Zoho CRM API is 200. DO NOT exceed the per page limit, or else this script will not work. Update the
perPageLimitvariable.
Zoho CRM Connection
In order to call the Zoho CRM API within a Deluge Script, you must first create a connection to it.
Connections In Zoho CRM
- Go to the Zoho CRM homepage and click setup.
- Under Developer Space, select Connections.
- Click Add Connection, then select Zoho OAuth.
- Specify a connection name (often zohocrm) and a scope, then click Create and Connect.
- Scope sets the limits of data access to the connection. It is a security best practice to give the minimum required scope to a connection.
- To retrieve records from modules select the ZohoCRM.modules.ALL scope.
- For more information on Zoho CRM API scopes, visit the Zoho CRM API documentation.
Want to learn even more?
Sign up for one of our Zoho courses!
Tutorial
Page Iteration List
Since Deluge doesn’t have a native while loop, we emulate its behavior. A for each iteration is performed over the pageIterationList variable. Within this block, we will rely on a boolean iterationComplete to determine whether we will continue to execute our API calls. The outer loop will technically continue to iterate, but will not be executing any internal code, thus allowing us to control iterations.
...
pageIterationList = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25];
...
iterationComplete = false;
for each page in pageIterationList {
if(iterationComplete == false) {
...
}
}
Note: comments removed for brevity
Important: This iteration list contains values from 1 to a number significantly greater than the expected count of result pages. If you don’t want to type this list by hand, see the Pinetools Number List Generator.
Making API Request and iteration Complete
The API request will then be executed and manipulated into the records variable which will only contain the current requested records. The records will be subsequently added to the allRecords list which will contain the accumulation of all API requests.
Since the for each process is emulating a while loop we need set our iterationComplete variable. We will set iterationComplete to true only when the the number of results on a page is less than our APIs per_page limit.
...
allRecords = List();
...
for each page in pageIterationList {
if(iterationComplete == false) {
response = invokeurl [
url: "https://www.zohoapis.com/crm/v2/" + zohoCrmModule + "?page=" + page + "&per_page=" + perPage
type: GET
connection: zohoCrmConnection
];
records = response.get("data");
allRecords.addAll(records);
if(response.get("info").get("more_records") == false) {
iterationComplete = true;
}
}
}
Click here to copy these scripts. For more Zoho-wizardry, check out our GitHub page.
Want this functionality, but don't want to do it yourself?
Click here to speak to us!
Need help with your Zoho system?
Hire a full time Zoho System Admin from our poolof highly qualified program graduates!
