Problem/Motivation
The FileMakerApiFetcher class, which extends the HttpFetcher class from the Feeds module, lacks batching support. This limitation potentially causes timeouts when fetching large datasets from FileMaker API.
Steps to reproduce
- Create a Feeds importer using FileMakerApiFetcher.
- Configure it to fetch a large dataset from FileMaker API.
- Run the import process.
- Observe timeout or failure for large datasets.
Proposed resolution
Implement batching in FileMakerApiFetcher using '_limit' and '_offset' parameters as per the FileMaker Data API Guide, while maintaining compatibility with the parent HttpFetcher class from Feeds module.
Remaining tasks
- Implement batching logic in FileMakerApiFetcher.
- Add batch size configuration option.
- Update fetch method for batching.
- Implement progress reporting.
- Write unit tests.
- Update documentation.
- Perform thorough testing.
User interface changes
- Add batch size field in fetcher configuration form.
- Update progress indicator for batch processing.
API changes
- Modify FileMakerApiFetcher::fetch() to support batching.
- Add new protected methods for batch processing.
- Update relevant hooks/events.
Data model changes
No direct changes. Temporary data storage during import may be affected.
Issue fork feeds_filemaker-3517133
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
baikho commentedComment #5
baikho commentedComment #6
baikho commentedComment #7
baikho commented