investigate general php memory consumption fixes

With respect to #2, I would like to see some more detail on it.

I know the user in another thread linked to a StackOverflow question (http://stackoverflow.com/questions/1145775/why-does-this-simple-php-scri...), however, I would like to see a bug filed on php.net which confirms that this issue does indeed exist, and verify a workaround before we put in the massive effort that would be required to create pre-defined DrupalUser and DrupalNode objects.

My own Googling has turned up nothing besides that StackOverflow question to confirm that StdClass objects leak memory.

Bumping to Drupal 8 and major.

http://drupal.org/project/issues/search/drupal?version[0]=7.x&issue_tags...

I noticed that even with APC enabled, memory usage still increases when including files. See attachment - a minimal D7 install with PHP 5.3.2-1ubuntu4.7 and APC 3.1.4, total memory usage is < 10mb, drupal_load() accounts for 10% of this (and not all files are included via drupal_load() so the reality is higher).

There were two things I wanted to rule out:

1. When including .module files, we almost always execute define() several times, however define() on that request only accounts for 37,048 bytes of memory so this seems unlikely to be causing any major issues. More of that at #334024: Use const keyword to define constants instead of define().

2. The other thing we do is define all those userspace functions, looked at that a bit this evening.

To start collecting some data regarding this, I wrote a quick script bench.php (attached) just to be able to easily vary the number of functions being defined.

It writes a file containing 5,000 functions that are all basically the same - this is equivalent to 100 modules defining 50 functions each. Then takes current memory usage, then includes the file, prints both plus the difference.

Then disabled xdebug and xhprof extensions just in case, hit bench.php - with and without APC enabled:

With APC:
before: 3.41 MB after: 3.67 MB diff:273.86 KB

(and with xhprof enabled and profiling to see how much affect it has by itself: before: 4.53 MB after: 4.81 MB diff:284.12 KB)

Without APC:
before: 21.85 MB after: 24.05 MB diff:2.21 MB

Then I made bench2.php, that just defines one really, really long function, without any other way to particularly measure the difference I tweaked the numbers until file size was roughly the same:

-rw-r--r-- 1 www-data www-data 202527 2011-01-27 02:23 functions2.php
-rw-r--r-- 1 www-data www-data 203900 2011-01-27 02:18 functions.php

With APC:
before: 3.41 MB after: 3.41 MB diff:784 bytes

Without APC:
before: 21.85 MB after: 25.62 MB diff:3.78 MB

So, APC is amazing ;)

Otherwise, this does explain, to at least a certain extent, why memory usage increases just from including a file, despite common sense suggesting that APC would fix the memory from the amount of code loaded completely. There's a significant difference 250kb in this case) between one very long function and lots of little smaller ones, shame it's not the other way 'round.

I didn't do any testing with classes, but we do load some during full bootstrap, so if they're taking significantly more memory than normal functions, that might account for some of the discrepancy.

For comparison, counting user functions before and after full bootstrap you get the following:

before: 684
after: 2192

On a heavily loaded Drupal 6 site, I got the following:

before:665
after: 4156

For classes in D7 (via get_declared_classes()), not all that many really.

before: 161
after: 178

That's it for now on this...

Forgot to attach the scripts.

So my script was making functions like this:

function foo_1() {
  return 'foo';
}

function foo_2() {
  return 'foo';
}

Did quick checks with function name length and number of arguments, exactly 0 difference.

Then tried classes:

class FooBar1 {
  function bar() {
  }
  function foo() {
  }
  function baz() {
  }
}

Attached script generates 500 classes, all like that.

Now it gets a bit more fun:

before: 3.29 MB after: 4.6 MB diff:1.31 MB

File size is a lot smaller with 500 classes like this compared to 5,000 functions:
-rw-r--r-- 1 www-data www-data 44899 2011-01-28 13:21 functions.php

Also apc makes a much, much smaller difference, here's what it looks like without:

before: 19.7 MB after: 21.33 MB diff:1.63 MB

So now the majority of the memory usage is in registering the classes, whereas before it was all about the amount of code.

Here's a possible answer to that with apc:

apc.lazy_functions 0 PHP_INI_SYSTEM Available since APC 3.1.3.
apc.lazy_classes 0 PHP_INI_SYSTEM Available since APC 3.1.3.

This blog post outlines what those do:

http://tekrat.com/2009/03/10/apc-lazy-loading-initial-support/

This way there's no copying of the functions to apc from shared to local memory until they're called. However I'm not seeing any change with this enabled.

Per catch, this issue has been replaced by sub-issues tagged with "memory":

http://drupal.org/project/issues/search/drupal?text=&assigned=&submitted=&participant=&status[]=Open&issue_tags_op=or&issue_tags=memory

The individual issues are much more actionable than this general issue.