Problem/Motivation

I cannot import a large number of records that are executed in multiple batches.

If the number of records to be imported exceeds 100, drush wfi command splits the task into several batches. When handling 2nd or subsequent batches, the process should skip over the records that have already been imported. However, WebformSubmissionExportImportImporter skips by line rather than by CSV record.

while ($index < $offset && !feof($handle)) {
  // => We should use fgetcsv() here.
  fgets($handle);
  $index++;
}

The problem and cause is similar to this: #3386895: Upload count is incorrect in getTotal method of WebformSubmissionExportImportImporter

Steps to reproduce

  1. Install Webform and Webform Submission Export/Import
  2. Create a webform
  3. Insert ~200 records to the form
  4. Export the submission data by drush wfx --exporter=webform_submission_export_import
  5. Import the exported files

Proposed resolution

Skip processed records by CSV record as #3386895: Upload count is incorrect in getTotal method of WebformSubmissionExportImportImporter.

Remaining tasks

Write tests

User interface changes

N/A

API changes

N/A

Data model changes

N/A

CommentFileSizeAuthor
#3 webform-submission-importer-csv-line-offset-3414928-3.patch710 bytesscambler
#2 webform-submission-importer-csv-line-offset-3414928-2.patch522 bytesscambler
webform-submission.patch534 byteskensuke-imamura

Issue fork webform-3414928

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

kensuke-imamura created an issue. See original summary.

scambler’s picture

scambler commented
StatusFileSize
new webform-submission-importer-csv-line-offset-3414928-2.patch522 bytes

Nice! I came across the same issue. I used a similar patch and it solved my problem. The only difference is when checking for an empty line, I copied the logic from L422. I'm not sure if there are situations where this will actually make a difference, but I'll upload it anyway.

scambler’s picture

scambler commented
StatusFileSize
new webform-submission-importer-csv-line-offset-3414928-3.patch710 bytes

Corrected the patch format (missing the first line) in the previous 2 patches.

cilefen’s picture

cilefen commented
Version: 6.2.2 » 6.2.x-dev
Status: Active » Needs review
jrockowitz’s picture

jrockowitz commented

Does the CVS that is being imported have empty lines?

Can you upload and example CSV that is having this issue?

jrockowitz’s picture

jrockowitz commented
Version: 6.2.x-dev » 6.3.x-dev
Status: Needs review » Postponed (maintainer needs more info)
kubrick’s picture

kubrick commented

Patch #3 solved an issue I was having with big CSV's.

jwineichen’s picture

jwineichen commented

Patch #3 also worked for me. I was importing a CSV with 1900 lines. Without the patch, the completion message said it created a little under 300 records and then updated 1600 or something, even though I added sid and uuid columns so they should have all been recognized as unique. With the patch, the the upload completed successfully and I've got 1900 submissions now.

cilefen’s picture

cilefen commented
Status: Postponed (maintainer needs more info) » Needs review
pacproduct’s picture

pacproduct commented

I think the issue is not specific to big CSVs but to big CSVs that contain entries with line breaks, as they get interpreted as new lines with fgets where they should not.

I was facing the following errors in my case because of multiline webform elements:

...
>  [warning] Line n°101: 28 values expected and only 3 found.
>  [warning] Line n°102: 28 values expected and only 5 found.
...

Using fgetcsv as suggested seems to be the way to go, and I can confirm that the patch in #3 does solve the issue as it counts lines in a similar way to \Drupal\webform_submission_export_import\WebformSubmissionExportImportImporter::getTotal, although the latter uses if (!empty($line) && !is_null(array_pop($line))) { instead of if (!empty($line) && $line !== ['']) {, I'm not sure which approach is the best.

Thanks! :)

pacproduct’s picture

pacproduct commented
Status: Needs review » Reviewed & tested by the community
liam morland’s picture

liam morland
Primary language English
Location Ontario, CA 🇨🇦
commented
Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs merge request

immaculatexavier made their first commit to this issue’s fork.

immaculatexavier opened merge request !645

immaculatexavier’s picture

immaculatexavier commented
Status: Needs work » Needs review
kensuke-imamura’s picture

kensuke-imamura commented

Patch #3 and the merge request do not seem to correctly recognize empty lines.
According to https://www.php.net/manual/en/function.fgetcsv.php, fgetcsv() returns an array containing only NULL ([NULL]) for empty lines.
In this code, it's strictly comparing with [''], so it will always fail to check empty lines.

$fp = fopen('php://memory', 'rw+');
fwrite($fp, <<<CSV
a,b,c

1,2,3
CSV);
rewind($fp);

while (($line = fgetcsv($fp))) {
  if (!empty($line) && $line !== ['']) {
    // This is always output.
    echo "not empty\n";
  } else {
    echo "empty\n";
  }
}

The first patch I submitted does not have this issue. The empty line detection method in my patch was copied from the existing condition in modules/webform_submission_export_import/src/WebformSubmissionExportImportImporter.php.

jrockowitz’s picture

jrockowitz commented
Status: Needs review » Fixed

Now that this issue is closed, please review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, please credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

liam morland’s picture

liam morland
Primary language English
Location Ontario, CA 🇨🇦
commented
Issue summary: View changes
Related issues: +#3386895: Upload count is incorrect in getTotal method of WebformSubmissionExportImportImporter