Problem/Motivation

The FileMakerApiFetcher class, which extends the HttpFetcher class from the Feeds module, lacks batching support. This limitation potentially causes timeouts when fetching large datasets from FileMaker API.

Steps to reproduce

  1. Create a Feeds importer using FileMakerApiFetcher.
  2. Configure it to fetch a large dataset from FileMaker API.
  3. Run the import process.
  4. Observe timeout or failure for large datasets.

Proposed resolution

Implement batching in FileMakerApiFetcher using '_limit' and '_offset' parameters as per the FileMaker Data API Guide, while maintaining compatibility with the parent HttpFetcher class from Feeds module.

Remaining tasks

  1. Implement batching logic in FileMakerApiFetcher.
  2. Add batch size configuration option.
  3. Update fetch method for batching.
  4. Implement progress reporting.
  5. Write unit tests.
  6. Update documentation.
  7. Perform thorough testing.

User interface changes

  • Add batch size field in fetcher configuration form.
  • Update progress indicator for batch processing.

API changes

  • Modify FileMakerApiFetcher::fetch() to support batching.
  • Add new protected methods for batch processing.
  • Update relevant hooks/events.

Data model changes

No direct changes. Temporary data storage during import may be affected.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

baikho created an issue. See original summary.

baikho’s picture

Issue summary: View changes

  • baikho committed f85843f0 on 2.x
    Issue #3517133 by baikho: Support Batch Processing
    
baikho’s picture

Status: Active » Fixed
baikho’s picture

Assigned: baikho » Unassigned
baikho’s picture

Issue summary: View changes

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.