top of page

Subscribe to get best practices, tutorials, and many other cool things directly to your email inbox

  • Writer's pictureAhmed Tarek

How to Recursively Call APIs to Get All Results in JavaScript

How to recursively get results per page and finally return them into one result set.


How to recursively get REST API results per page and return them into one result. JavaScript JS Promise Async Await. Best Practice Code Coding Programming Software Development Architecture Engineering
Photo by Joan Gamell on Unsplash

The Challenge


Most of the time while working on a JavaScript project, I write some code to call REST APIs. This is normal.


However, sometimes a REST API could have restrictions on the number of items it returns per request. In other words, while calling the API to get some items, these items would be divided into indexed pages/groups and you have to provide the page/group index. This would return only the items in this page/group. To get the rest, you would need to repeat the same API call but now with a different index, and so on…


 

How to recursively get REST API results per page and return them into one result. JavaScript JS Promise Async Await. Best Practice Code Coding Programming Software Development Architecture Engineering
Photo by Emily Morter on Unsplash

How To Do It?


This is what I am going to show you in this article. We are going to use a simple example of an API which returns items into pages and see how to consume this API and get all items.


We will simulate the API call with a function which returns a promise and a timeout call. The items would be retrieved from a static array of items which we would define.


Also, important to note that the logic of splitting the items of the static array into pages is based on the analysis, equations, and code provided on the article Paging/Partitioning — Learn the Main Equations to Make it Easy.


Here, in this article, I would just include the code for brevity.


 

How to recursively get REST API results per page and return them into one result. JavaScript JS Promise Async Await. Best Practice Code Coding Programming Software Development Architecture Engineering
Photo by Nubelson Fernandes on Unsplash

Let’s Write Some Code


Array Page Function


Refer to this article for details.


Array.prototype.page = function(pageSize) {
    const self = this;

    let actualPageSize = pageSize;

    if (actualPageSize <= 0) {
        actualPageSize = self.length;
    }

    let maxNumberOfPages = Math.max(1, Math.ceil(parseFloat(parseFloat(self.length) / parseFloat(actualPageSize))));

    let pagesBoundries = {};

    for (let pageZeroIndex = 0; pageZeroIndex < maxNumberOfPages; pageZeroIndex++) {
        pagesBoundries[pageZeroIndex] = {
            firstItemZeroIndex: (pageZeroIndex * actualPageSize),
            lastItemZeroIndex: Math.min((pageZeroIndex * actualPageSize) + (actualPageSize - 1), self.length - 1)
        };
    }

    return {
        actualPageSize: actualPageSize,
        numberOfPages: maxNumberOfPages,
        pagesBoundries: pagesBoundries,
        find: function(itemZeroIndex) {
            return {
                pageZeroIndex: parseInt(parseInt(itemZeroIndex) / parseInt(actualPageSize)),
                itemZeroIndexInsidePage: parseInt(parseInt(itemZeroIndex) % parseInt(actualPageSize))
            };
        }
    };
};

 

Static Items Array and Default Page-Size


const pageSize = 3;

const items = [{
    name: "Item1",
    id: 1
}, {
    name: "Item2",
    id: 2
}, {
    name: "Item3",
    id: 3
}, {
    name: "Item4",
    id: 4
}, {
    name: "Item5",
    id: 5
}, {
    name: "Item6",
    id: 6
}, {
    name: "Item7",
    id: 7
}, {
    name: "Item8",
    id: 8
}, {
    name: "Item9",
    id: 9
}, {
    name: "Item10",
    id: 10
}];

 

Execute API Call Function


const executeApiCall = function(pageZeroIndex) {
    const pagingResult = items.page(pageSize);
    const pageBoundries = pagingResult.pagesBoundries[pageZeroIndex];
    const itemsToReturn = items.slice(pageBoundries.firstItemZeroIndex, pageBoundries.lastItemZeroIndex + 1);

    const result = {
        items: itemsToReturn,
        paging: {
            pageZeroIndex: pageZeroIndex,
            totalPagesCount: pagingResult.numberOfPages
        }
    };

    return new Promise((resolve, reject) => {
        setTimeout(() => {
            resolve(result);
        }, 1000);
    });
};

What we can notice here:

  1. This is the function which simulates the API call.

  2. First it divides the static items array into pages using the static pageSize variable. Depending on the passed in pageZeroIndex parameter, it sets the itemsToReturn internal array.

  3. Then it wraps this into a final object which has a paging.pageZeroIndex and page.totalPagesCount members. These should be used by the API caller to know if there are other pages or not.

  4. Finally, this final object is resolved in a promise after one second.


 

Getting Results Using Old-fashioned Promises


const getResults = function() {
    console.log('Started getting results.');
    return getResultsRecursively(0);
};

const getResultsRecursively = function(pageIndex) {
    console.log(`Getting results of page ${pageIndex} started.`);
    let foundItems = [];

    return new Promise((resolve, reject) => {
        try {
            executeApiCall(pageIndex)
                .then((response) => {
                    if (response.items && response.items.length > 0) {
                        foundItems = foundItems.concat(response.items);
                    }

                    console.log(`Getting results of page ${pageIndex} ended.`);

                    if ((pageIndex + 1) < response.paging.totalPagesCount) {
                        getResultsRecursively(pageIndex + 1)
                            .then(nextBatch => {
                                if (nextBatch && nextBatch.length > 0) {
                                    foundItems = foundItems.concat(nextBatch);
                                }

                                resolve(foundItems);
                            })
                            .catch((error) => {
                                console.log(`Error while getting results of ${pageIndex + 1}`);
                                console.log(error);
                                reject(new Error(error));
                            });
                    } else {
                        resolve(foundItems);
                    }
                })
                .catch((error) => {
                    console.log(`Error while getting results of ${pageIndex}`);
                    console.log(error);
                    reject(new Error(error));
                });
        } catch (error) {
            console.log(error);
            reject(new Error(error));
        }
    });
};

What we can notice here:

  1. We are calling the API in recursive manner but only resolving at the end of the items.

  2. We are using page.totalPagesCount to decide if there could be another page or not.


 

Testing Calling the Old-fashioned Code


getResults()
    .then((results) => {
        console.log('results', JSON.stringify(results));
        alert('done');
    })
    .catch((error) => {
        console.log(error);
    });

And the result would be the following:


How to recursively get REST API results per page and return them into one result. JavaScript JS Promise Async Await. Best Practice Code Coding Programming Software Development Architecture Engineering

 

Getting Results Using async/await


const getResults = async function() {
    console.log('Started getting results.');

    let foundItems = [];
    let loop = true;
    let pageIndex = 0;

    do {
        console.log(`Getting results of page ${pageIndex} started.`);

        var response = await executeApiCall(pageIndex);

        console.log(`Getting results of page ${pageIndex} ended.`);

        if (response.items && response.items.length > 0) {
            foundItems = foundItems.concat(response.items);
        }

        loop = ((pageIndex + 1) < response.paging.totalPagesCount);
        pageIndex++;
    }
    while (loop);

    return foundItems;
};

What we can notice here is that we are doing the same as in the old-fashioned code but here it is simpler.


 

Testing Calling the async/await Code


const results = await getResults();
console.log('results', JSON.stringify(results));
alert('done');

And the result would be the following:


How to recursively get REST API results per page and return them into one result. JavaScript JS Promise Async Await. Best Practice Code Coding Programming Software Development Architecture Engineering

That’s it, hope you found reading this story as interesting as I found writing it.



Recent Posts

See All

Subscribe to get best practices, tutorials, and many other cool things directly to your email inbox

bottom of page
Mastodon Mastodon