We have been importing a large website into Drupal using feeds.
All has gone well until we started importing images which needed multiple (over 90) content types (to server different destination folders) and thus 90 import filters.

After around 20 - 30 imports had been run we started to get 'fatal error: out of memory' which were initially resolved by getting out hosting company to increase our memory allocation to 1MB and now 2MB. Unfortunately we hadn't cured the problem - after 30 - 40 filters & imports had been run we got the following error while constructing a new importer

Fatal error: Out of memory (allocated 172490752) (tried to allocate 5242863 bytes) in /home2/*******/public_html/*******.co.uk/includes/cache.inc on line 450

I am suggesting that there is a limit to the number of feeds import filters that can be set up - is that possible? A few **might** be running every 30mins (our error) but most aren't. Server host is looking for the source of a memory leak.

Is feeds our possible source of this leak?

Graham

Comments

GAtherton’s picture

GAtherton commented

Update

I have now been able to disable ALL importers, none are running but the error still occurs. Server host is suggesting that there is a bug in the code causing a memory leak each time a new importer is created.

Any suggestions? How can we trace the problem?

damienmckenna’s picture

damienmckenna
Location TN, USA
commented
Title: Out of memory when using multiple feeds » Out-of-memory error when creating new importer when many importers already exist

Clarifying the title.

GAtherton’s picture

GAtherton commented

Updates from our server host

I've just had a go at creating a few feeds, without issue. I can say that creating a feed takes up to 450MB of virtual memory and 130MB of normal memory.

You don't appear to have any processes running other than the occasional page view (or me testing).

*********************

This is very interesting, I can see from your web access logs that your IP address generates 403 errors a fair bit (particularly, it seems, when adding importers).

A 403 error is normally indicative of some sort of permissions or mod_security error. However nothing seems to be logged against your website for the latter.

On looking through the web server error log I see that these messages correlate with what looks like a crash of the PHP binary.

Just out of interest, have you tried doing this with a different web browser - looks like you are using Safari? Perhaps try with Firefox, like I'm using.

***********************

NOTE: we are trying out the suggestion of a different browser (Firefox as we have been using Chrome) and will inform here of the outcome - this suggestion arises because the hosts have created multiple new importers (just empty importers with no fields/links to content type/mapping) with no error on our website - I assume that these use far less memory. We get a crash when we have run two or three imports and THEN it crashes as we initiate a new importer.

GAtherton’s picture

GAtherton commented

We have discovered the following additional info:

Once crashed we can partialy restore access in Firefox by deleting cookies (doesn't work in Chrome)

Website is OK until we try to login whereupon it crashes - this is specifically related to the presence of the session cookie. If I delete the cookie I can see the website, if it is present all I get is

Fatal error: Out of memory (allocated 174587904) (tried to allocate 4980705 bytes) in /home2/******/public_html/******/includes/cache.inc on line 450

It looks like something leaks memory and the main module we are currently using is feeds. The situation will spontaneously improve in a few hours.

It looks like I am going to have to try deleting content types and importers.

damienmckenna’s picture

damienmckenna
Location TN, USA
commented

Do you have any custom code that would be triggering, e.g. a custom parser or maybe even just some hook_node_* or hook_entity_* functions?

twistor’s picture

twistor commented
Component: Feeds Import » Code
Assigned: GAtherton » twistor

I am a bit confused by how cookies could be affecting this. I see that the failures are happening in cache.inc, by chance, how large are the files you're trying to import?

Can we get an export of one of your importers? One or two representative ones. It's not quite clear if you are setting up 20-30 importers or running 20-30 imports. I'm also not sure what you are referring to by filters.

twistor’s picture

twistor commented
Issue summary: View changes

Changed 'filter' to 'importer'

bluegeek9’s picture

bluegeek9 commented
Assigned: twistor » Unassigned
Issue summary: View changes
Status: Active » Closed (outdated)

Drupal 7 reached end of life and the D7 version of Feeds is no longer being developed. To keep the issue queue focused on supported versions, we’re closing older D7 issues.

If you still have questions about using Feeds on Drupal 7, feel free to ask. While we won’t fix D7 bugs anymore, we’re happy to offer guidance to help you move forward. You can do so by opening (or reopening) a D7 issue, or by reaching out in the #feeds channel on Drupal Slack.

If this issue is still relevant for Drupal 10+, please open a follow-up issue or merge request with proposed changes. Contributions are always welcome!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.