By vitamin-1 on

I installed Drupal 5.1. I use it with content module, I need a lot of custom nodes (approx. 44000). I wrote simple script which help me to import nodes from xml file. It parses xml, then calls node_save(). This work fine.
One day I wanted to export all my nodes to file (or delete them) and I wrote very simple script which repeatedly calls node_load (or node_delete for deletion, no matter because node_delete also calls node_load). And very soon it consumed all memory (I have php memory_limit set to 64M). I disabled content.module at all but it doesn't help. I used memory_get_usage() for debugging and I see that there is memory leaking somewhere in the Drupal core.
Here is very simple example:

$db_result = db_query("SELECT nid FROM {node} WHERE type = '%s'", 'custom_type');

while ($result = db_fetch_array($db_result)) {
$node = node_load($result['nid']);
print memory_get_usage() . "\n";
}

Little by little, it eats all memory available:

9593920
9599912
9604400
9608952
9613504
...
67678824
67682712
67686456
67690252
67694100

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 40 bytes) in /var/www/drupal/html/includes/module.inc on line 47

Is it normal behaviour? How can I fix this?

⋅ Categories: Drupal 5.x

Comments

harriska2’s picture

harriska2 commented

Tracking.

vitamin-1’s picture

vitamin-1 commented

I found that there is no memory leaking when node_load is invoked with $reset parameter set to TRUE.
However, there is still problem with node_delete and node_save (when updating existing node) because there is no way to pass $reset parameter to node_load, invoked from node_save or node_delete functions.

hickory’s picture

hickory commented

This is also common in aggregator modules which process a lot of nodes. I have a feeling it might be related to drupal_clone making a copy of the object each time, but haven't finished testing this to know for sure yet.

hickory’s picture

hickory commented

My mistake, I don't think it's drupal_clone. I believe I've fixed this now by clearing the static cache in node_load whenever it gets above 100 items.

rubymuse’s picture

rubymuse commented

I'm seeing a memory leak when using node_delete. Here is some debugging info I sent to watchdog, to document calls to memory_get_usage. The "4.1.7" entries were called immediately before a node_delete, and the 4.1.8 entries were called immediately after. DIFF shows the increase/decrease from the last call to memory_get_usage.

Each call to node_delete seems to be leaking around 7K, though it starts out at around 2K (the entries are in descending chronological order) and jumps to 7K pretty quickly.

I've got a process running as a windows service (runs continuously) and I'm wondering if there's a way I can 'unbootstrap' drupal on a regular basis to free up this memory.

debug 4.1.8 - MEMORY: 6688136; DIFF: 6928 admin
debug 4.1.7 - MEMORY: 6681208; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6681312; DIFF: 7384 admin
debug 4.1.7 - MEMORY: 6673928; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6674032; DIFF: 7032 admin
debug 4.1.7 - MEMORY: 6667000; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6667104; DIFF: 7192 admin
debug 4.1.7 - MEMORY: 6659912; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6660016; DIFF: 6968 admin
debug 4.1.7 - MEMORY: 6653048; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6653152; DIFF: 7192 admin
debug 4.1.7 - MEMORY: 6645960; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6646064; DIFF: 7088 admin
debug 4.1.7 - MEMORY: 6638976; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6639080; DIFF: 7016 admin
debug 4.1.7 - MEMORY: 6632064; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6632168; DIFF: 6944 admin
debug 4.1.7 - MEMORY: 6625224; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6625328; DIFF: 5720 admin
debug 4.1.7 - MEMORY: 6619608; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6619712; DIFF: 2032 admin
debug 4.1.7 - MEMORY: 6617680; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6617784; DIFF: 2000 admin
debug 4.1.7 - MEMORY: 6615784; DIFF: -104 admin
debug 4.1.8 - MEMORY: 6615888; DIFF: 1608 admin
debug 4.1.7 - MEMORY: 6614280; DIFF: 48 admin

johnmcgeechan’s picture

johnmcgeechan commented

Had a similar problem. I had a script that progromatically bootstrapped Drupal core and then proceeded to load and process tens of thousands of nodes. The memory leaked away until the script hit the PHP memory_limit buffers (Simply upping this in php.ini wasn't real an option, due to the amount of records I was processing). I found that the issue was due to the global $queries array that seemed to just grow and grow.

Here's the solution that worked for me -btw it's not a patch as the 'problem' was in my own bespoke script.. 

.....
$mynode = node_load($nodeId);
global $queries;
$queries = array();

A couple of things to note...

- i tried previous suggestions of clearing the node cache through various versions of node_load(FALSE, NULL, TRUE) etc , none worked for me
- You could try to 'unset' $queries, but trial this, unsetting something in the global scope from a function is not guaranteed to work, you usually have to unsetr all children first, simply setting to empty array worked for me...