We updated from 7.2-beta3 to 7.2 beta4
drupal 7 - php 7.1

Our manual feeds import of our csv files failed, with Ajax http error Code:200 see screenshot

Some background

  • We normally do imports of +- 15000 nodes with no problems, for the last 3 years.
  • The import now had 200 lines which gave the error.
  • Nothing changed on the web server side(own server), time out value and so on.
  • Nothing changed on the db server side(own server dedicated to this site),

As a work around we needed to cut up the csv file in 8 pieces, thus 25 lines per file (4 pieces did not work, 50 lines per file).
The smaller file imports worked like normal, so all the csv settings, tamper, entity reference and so on is ok.

We also noticed for the small file the batch api is not needed and jumps from 0-> 100%, The job is small enough.
I remember that this is a feature of batch api.
So the slicing of the feeds import into small batches by batch api in feeds is not done properly.

The batch job seemed to timeout on php time out error limit. 30''
We also use batch-api and Views bulk operations on a daily basis, no time outs there.
We did a test with 500+ nodes with VBO to test the batch api , job took more than 5 minutes nicely batched per 10 on execution. no timeout.

My guess is feeds tries to do to mutch rows in one batch cycle.

CommentFileSizeAuthor
Knipsel.PNG18.38 KBikeigenwijs

Comments

ikeigenwijs created an issue. See original summary.

megachriz’s picture

megachriz
Primary language Dutch
commented

I think there were no specific changes related to the batch API between Feeds 7.x-2.0-beta3 and Feeds 7.x-2.0-beta4.

A big thing that did change was caching of HTTP downloaded content. Previously HTTP downloaded content was cached in the database. It is now cached on the file system. Also, previously HTTP downloaded content was cached statically. This was a problem as that lead to PHP needing a lot of memory if multiple large files were downloaded in one request. When a resource is now asked twice in the same request it is delivered from the cache on the file system. These files are saved in "private://feeds/cache" (or "public://feeds/cache" if no private file system is configured).

For CSV imports that cannot be completed in one run, a separate file is saved on the file system (apart from the cache). The location where this 'temporary' file is saved has changed. Previously this was in "public://feeds". Now it is in "private://feeds/in_progress" (or "public://feeds/in_progress" if no private file system is configured). The location of the 'in progress' file is configurable. See the change record.

Lastly, the base class for fetcher result has changed a lot. You can read more about that in the release notes or the change record.

I haven't tested Feeds on PHP 7.1 yet. There are a few test failures for that version, though they all are related to date fields (so that could also mean that the date module has issues on PHP 7.1).

These are the first things I can think of that might have influence on this issue.

ikeigenwijs’s picture

ikeigenwijs commented

Thanks for the fast responce, i look in to this and get back, we have some larger imports coming up.

ikeigenwijs’s picture

ikeigenwijs commented

Ok changed permissions of complete drupal install to 777 for testing purposes. --> no change still Ajax error.

Rolled back to 7.x-2.0-beta3,
Super that the instructions are included in the release notes.
Needed 2 additional clear all caches

var/www/lmgmasterdb.ugent.be/public_html/includes/cache.inc).
Error: Class 'FeedsHTTPCache' not found in _cache_get_object() (line 32 of /var/www/lmgmasterdb.ugent.be/public_html/includes/cache.inc).
Drush command terminated abnormally due to an unrecoverable error.

After rolling back the large imports purred along (php 7.1). So i can confirm beyond a reasonable doubt that beta4 is the perpetrator.

I nee dto look deeper in to the
Lastly, the base class for fetcher result has changed a lot. You can read more about that in the release notes or the change record.

bluegeek9’s picture

bluegeek9 commented
Status: Active » Closed (outdated)

Drupal 7 reached end of life and the D7 version of Feeds is no longer being developed. To keep the issue queue focused on supported versions, we’re closing older D7 issues.

If you still have questions about using Feeds on Drupal 7, feel free to ask. While we won’t fix D7 bugs anymore, we’re happy to offer guidance to help you move forward. You can do so by opening (or reopening) a D7 issue, or by reaching out in the #feeds channel on Drupal Slack.

If this issue is still relevant for Drupal 10+, please open a follow-up issue or merge request with proposed changes. Contributions are always welcome!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.