Problem/Motivation
Modern REST APIs often enforce a maximum result limit per single request (e.g., 100 or 1000 items).
Currently, the Views JSON Source module performs a single HTTP request to the configured URL. If a dataset exceeds the remote API's page limit, the View can only display or aggregate the first page of results. This makes it impossible to create accurate reports (like "Total counts across the entire repository") when the total number of items exceeds the remote API's max_size.
Steps to reproduce
1. Install views_json_source module.
2. Create a View to fetch data from a remote API that has a result limit (e.g., a repository with 5,000 items where the API only returns 1,000 per request).
3. Configure the View with the API URL.
4. Observe that the View only processes the first 1,000 items.
5. Attempting to use the "Aggregation" feature in Views results in incomplete data because the module never "sees" the remaining 4,000 items.
Proposed resolution
Update the ViewsJsonQuery plugin to support an iterative fetching
loop that can "follow" pagination links.
Changes required:
1. New Configuration Options: Add settings to the View's Query options:
* Enable Pagination: Checkbox to enable multi-page fetching.
* Next Page Path: An "Apath" selector to identify the "next page" URL in the JSON response (e.g., _links/next/href or next_page).
* Maximum Pages: A safety limit (e.g., default 10) to prevent infinite loops or excessive API resource consumption.
2. Logic Update in ViewsJsonQuery::execute():
* Modify the execute method to check if pagination is enabled.
* If enabled, the query should loop: fetch a page, extract the data rows, find the "next" URL, and repeat until no "next" URL is found or the maximum page limit is reached.
* All fetched data rows should be merged into a single array before the parse() method is called, ensuring the View has access to the full dataset for sorting, filtering, and aggregation.
Issue fork views_json_source-3581987
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #3
orkutmuratyilmazCan someone review this please?