Problem/Motivation

When using batch_size in the source config, it creates an infinite loop.
It seems to import everything in a first batch then loops infinitely on batches containing the whole dataset.

Steps to reproduce

Here is an example migration that can reproduce the problem :

id: folio_migrate_pays
label: Pays
migration_group: folio
source:
  plugin: custom_sql_query
  key: folio_catalogue
  keys:
    - id
  track_changes: true
  batch_size: 2
  sql_query:
    #language=mysql
    'SELECT id, name FROM v_countries'
destination:
  plugin: 'entity:taxonomy_term'
  default_bundle: pays
process:
  name: name
  field_identifiant_catalogue: id
dependencies:
  enforced:
    module:
      - folio_migrate

If I call drush migrate:import folio_migrate_pays --update, it hangs forever.
If I look at what it is doing, it seems it is calling CustomSQLQuery::fetchNextBatch() again and again forever.

Proposed resolution

In the code, it looks like $batchSize is never really used.

SqlBase uses it to add a range to the query. However, in our case I am not sure that's possible, since the query is not manipulated in PHP.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

prudloff created an issue. See original summary.

  • prudloff committed b54b8344 on 8.x-1.x
    Issue #3347956: batch_size creates an infinite loop
    
prudloff’s picture

Status: Active » Fixed

I removed support for batch_size since it is broken.
We can always add it back later if we find a way to make it work.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.