Drupal 7.x can't be installed with memory_limit=32M

This seems to come up quite a bit. Since the default limit is 8M, it would make sense to have somewhere in the .htaccess a commented line with the php_value set somewhere around 24M. Here's a quick and dirty patch to get things started. You may want to change the wording, the default limit, or the placement.

I don't think it's a good idea to have this uncommented by default since some hosts or users could already have higher limits.

Is there a way to check the current limit and set it only if it needs to be higher?

The PHP5 default memory limit is now 16M so we'll probably see less installs around with 8M. That doesn't help if we go over 16M though...

@Damien - will it install with 16M of memory with #278592: Sync 6.x extra updates with HEAD applied?

I think we might need to have .update files to avoid memory bloat when loading install files.

Also there was lot of discussion of this general issue over at #197720: Drupal requires more than 8M PHP memory_limit (although it was 8mb that time).

Just ran into this. Subscribing. Would be nice to at least warn the user.

Is there a way to check the current limit and set it only if it needs to be higher?

php ini_set() and ini_get() maybe?

Shouldn't this fail at the 'install requirements' phase? I need to test this.

Forcing memory limit as a requirement (more than the default 16MB) will likely be a pain for those who have shared hosting accounts.

we need to diagnose why this happens... but the question is how...

related to #309457: Allow profiles to specify required memory in .info file. even though it's noble to try to reduce the memory, i believe it's only stopgap, as eventually we'll cross that limit (as it seems we already have).

bumping to critical. i tried to install HEAD on a default 16M installation, and not only did it fail, it corrupted the database, giving the following on successive visits to install.php:

PDOException: SELECT menu_router.* FROM {menu_router} menu_router WHERE (path IN ()) ORDER BY fit DESC LIMIT 0, 1 - Array ( ) SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')) ORDER BY fit DESC LIMIT 0, 1' at line 4 in menu_get_item() (line 369 of /var/www/jq-7/includes/menu.inc).

See also #346073: memory exhausted on install - wrong link.

Once again ran into this problem:

1. Default install of Apache and PHP, didn't change memory_limit
2. Loaded up install.php
3. Told me that the memory limit at 16M was fine
4. Attempted to install
5. Got a very ugly big red error message
6. Increased to 32M
7. Recreated settings.php, loaded up install.php again
8. Successfully installed with a different database prefix

I think we should increase the required memory because a regular user would just give up before they even get Drupal installed. We should increase it at least until we decrease the amount of memory that Drupal requires to install.

-1 for that patch: see #2, #4 and #8

In addition, we should upgrade this from a warning to an error. Here's what happens when you install w/ default MAMP 1.7.2 (PHP 5.2.6):

1. On the warnings page, you are told about the recommended memory size, as well as the file permissions stuff. You edit php.ini and bump up your memory_limit to 32MB or so.
2. BUT! You forget to reboot Apache. Oops. Drupal's install happily proceeds along until the point that it attempts to install and you get a HTTP 500 error. That's ... helpful.
3. Then, in an extra special dose of helpfulness, clicking on the "error page" link takes you to a nice WSOD. ;)

Mostly system administrators will be modifying the php.ini file. Most users will modify the .htaccess file which doesn't require the reboot. That was the original reason I thought having a line in that distributed file would be beneficial.

Why is this premature, Damien? I hit it every time I setup a new development environment in both Drupal 6 and HEAD...

current state:
memorylimit = 16MB during install => drupal breakage (500), with no warning message when all requirements are met.

Error "Fatal error: Allowed memory size of 26214400 bytes exhausted " after installation with memory limit 26M.

But no error with memory limit of 32M. So "<32M" is correct.

I tried a fresh install with 16MB and received the, "HTTP 500 error occurred." Once I bumped it up to 32MB and started fresh no errors at all.

Attached is a simple patch to require a PHP memory limit of 32MB and upgrade from a warning to an error.

This should stay as a warning, not an error - which PHP extensions are loaded and various other factors (install profile etc.) can influence how much memory is required, so we don't want to stop someone installing with 22mb of memory if they can do it.

Ok - I can see how you don't want a hard error for it, but what about bumping up the DRUPAL_MINIMUM_PHP_MEMORY_LIMIT variable?

If I keep it at 16MB and run the Drupal (minimal) installation profile I still get a fatal error of not enough memory after clicking "Save and Continue" in the Configure Site step. If I try again with 22MB on the minimal profile it works without any errors, but if I try with the "standard" install it gives an error with 22MB, so I would say to bump up the DRUPAL_MINIMUM_PHP_MEMORY_LIMIT variable to at least 32MB.

Yes I think the minimum limit should be bumped to 32M, just not the change from warning to error.

Sorry for not making that clear in #24

Gotcha - we're on the same page then. Attached is a patch with just the minimum bumped up to 32MB, but the severity still kept as a warning

Rerolled patch in #27.

Committed to HEAD. This invariably trips up every. single. person. in #drupal who installs Drupal 7 for the first time, and we're fast-approaching alpha territory, so blatant issues like this need to be cleaned up.

We've had this issue open for almost 1.5 years, and so far have not had good luck reducing Drupal's memory footprint (although I know there's been efforts with module splitting, etc.). Maybe this will light an extra fire for us to figure this out. :)

Update: I can no longer install the default profile with 32M.

Upping my memory limit to 35M works. 34M is still too little.

(Note that this is only for the web installer. If you want to install the default profile non-interactively via a script, you need at least 41M.)

I have placed memory_get_peak_usage() calls in the installer and batch processor. Here's what happens:

Starting installation...
Running task install_select_profile... 
Running task install_select_locale... 
Running task install_load_profile... 
Running task install_verify_requirements... 
Submitting information to task install_settings_form... done.
Running task install_system_module... 
Running task install_bootstrap_full... 
Running batch process install_profile_modules... 
Current memory usage: 15.2 MB. Now installing node
New memory usage: 16.84 MB
Current memory usage: 16.84 MB. Now installing number
New memory usage: 17.22 MB
Current memory usage: 17.22 MB. Now installing field_sql_storage
New memory usage: 17.44 MB
Current memory usage: 17.44 MB. Now installing options
New memory usage: 17.56 MB
Current memory usage: 17.56 MB. Now installing text
New memory usage: 17.72 MB
Current memory usage: 17.72 MB. Now installing list
New memory usage: 17.83 MB
Current memory usage: 17.83 MB. Now installing field
New memory usage: 18.96 MB
Current memory usage: 18.96 MB. Now installing filter
New memory usage: 19.39 MB
Current memory usage: 19.39 MB. Now installing user
New memory usage: 19.52 MB
Current memory usage: 19.52 MB. Now installing block
New memory usage: 19.86 MB
Current memory usage: 19.86 MB. Now installing color
New memory usage: 20.18 MB
Current memory usage: 20.18 MB. Now installing comment
New memory usage: 20.96 MB
Current memory usage: 20.96 MB. Now installing contextual
New memory usage: 21.06 MB
Current memory usage: 21.06 MB. Now installing dashboard
New memory usage: 21.2 MB
Current memory usage: 21.2 MB. Now installing help
New memory usage: 21.24 MB
Current memory usage: 21.24 MB. Now installing image
New memory usage: 21.68 MB
Current memory usage: 21.68 MB. Now installing menu
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing path
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing taxonomy
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing dblog
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing search
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing shortcut
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing toolbar
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing overlay
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing field_ui
New memory usage: 28.99 MB
Current memory usage: 28.99 MB. Now installing file
New memory usage: 29.05 MB
Current memory usage: 29.05 MB. Now installing rdf
New memory usage: 29.24 MB
Current memory usage: 29.24 MB. Now installing default
New memory usage: 37.29 MB
Submitting information to task install_configure_form... done.
Running task install_finished... 
Using 37.6 MB. Now resetting theme and module data.
Using 37.6 MB. Now flushing all caches.
Using 39.9 MB.
Using 39.9 MB. Caching the schema now.
Using 39.9 MB. Now running cron.
Coffee's ready. Installation took 13.107 seconds.

It appears the culprit is not any single module (though menu and the default profile itself are big eaters), but rather the fact that the memory used up by installing each module is never freed up. If we managed to free memory after each installed module (particularly the menu module), the memory peak would drop back down to 20M, and could possibly even be stuffed under 16M.

Stepping into the respective install processes of menu and the default profile, it appears the major part of the footprint comes from rebuilding the menu.

Current memory usage: 29.23 MB. Now installing default
default_install: Using 29.23 MB. Adding text formats.
default_install: Using 29.23 MB. Enabling blocks.
default_install: Using 29.23 MB. Creating node types.
default_install: Using 29.23 MB. Creating taxonomy and image field.
default_install: Using 29.23 MB. Setting up roles and permissions.
default_install: Using 29.23 MB. Rebuilding menu.
default_install: Using 37.3 MB. Finished.
New memory usage: 37.3 MB

Current memory usage: 21.7 MB. Now installing menu via _install_module_batch
_install_module_batch: Using 21.7 MB. Installing menu.
menu_install: Using 21.7 MB. Listing system menus.
menu_install: Using 21.7 MB. Saving menu navigation.
menu_install: Using 21.7 MB. Saving menu management.
menu_install: Using 21.7 MB. Saving menu user-menu.
menu_install: Using 21.7 MB. Saving menu main-menu.
menu_install: Using 21.7 MB. Saving menu secondary-menu.
menu_install: Using 21.7 MB. Finished.
_install_module_batch: Using 21.78 MB. Enabling menu.
_install_module_batch: Using 29 MB. Done with menu.
New memory usage: 29 MB

It just occurred to me that I might be getting the wrong impression by looking at the peak usage. So here's another run with the current usage instead. It shows that the memory cost still rises cumulatively with every installed module, and does not go back down.

Starting installation...
Running task install_select_profile... 
Running task install_select_locale... 
Running task install_load_profile... 
Running task install_verify_requirements... 
Submitting information to task install_settings_form... done.
Running task install_system_module... 
Running task install_bootstrap_full... 
Running batch process install_profile_modules... 
Current memory usage: 14.41 MB. Now installing node via _install_module_batch
_install_module_batch: Using 14.41 MB. Installing node.
_install_module_batch: Using 15.34 MB. Enabling node.
_install_module_batch: Using 16.05 MB. Done with node.
New memory usage: 16.04 MB
Current memory usage: 16.05 MB. Now installing number via _install_module_batch
_install_module_batch: Using 16.05 MB. Installing number.
_install_module_batch: Using 16.15 MB. Enabling number.
_install_module_batch: Using 16.18 MB. Done with number.
New memory usage: 16.18 MB
Current memory usage: 16.18 MB. Now installing field_sql_storage via _install_module_batch
_install_module_batch: Using 16.18 MB. Installing field_sql_storage.
_install_module_batch: Using 16.38 MB. Enabling field_sql_storage.
_install_module_batch: Using 16.42 MB. Done with field_sql_storage.
New memory usage: 16.42 MB
Current memory usage: 16.42 MB. Now installing options via _install_module_batch
_install_module_batch: Using 16.42 MB. Installing options.
_install_module_batch: Using 16.47 MB. Enabling options.
_install_module_batch: Using 16.51 MB. Done with options.
New memory usage: 16.5 MB
Current memory usage: 16.5 MB. Now installing text via _install_module_batch
_install_module_batch: Using 16.5 MB. Installing text.
_install_module_batch: Using 16.63 MB. Enabling text.
_install_module_batch: Using 16.67 MB. Done with text.
New memory usage: 16.67 MB
Current memory usage: 16.67 MB. Now installing list via _install_module_batch
_install_module_batch: Using 16.67 MB. Installing list.
_install_module_batch: Using 16.74 MB. Enabling list.
_install_module_batch: Using 16.78 MB. Done with list.
New memory usage: 16.77 MB
Current memory usage: 16.77 MB. Now installing field via _install_module_batch
_install_module_batch: Using 16.77 MB. Installing field.
_install_module_batch: Using 17.75 MB. Enabling field.
_install_module_batch: Using 17.85 MB. Done with field.
New memory usage: 17.85 MB
Current memory usage: 17.85 MB. Now installing filter via _install_module_batch
_install_module_batch: Using 17.85 MB. Installing filter.
_install_module_batch: Using 18.23 MB. Enabling filter.
_install_module_batch: Using 18.32 MB. Done with filter.
New memory usage: 18.32 MB
Current memory usage: 18.32 MB. Now installing user via _install_module_batch
_install_module_batch: Using 18.32 MB. Installing user.
_install_module_batch: Using 18.31 MB. Enabling user.
_install_module_batch: Using 18.44 MB. Done with user.
New memory usage: 18.44 MB
Current memory usage: 18.44 MB. Now installing block via _install_module_batch
_install_module_batch: Using 18.44 MB. Installing block.
_install_module_batch: Using 18.71 MB. Enabling block.
_install_module_batch: Using 18.82 MB. Done with block.
New memory usage: 18.81 MB
Current memory usage: 18.81 MB. Now installing color via _install_module_batch
_install_module_batch: Using 18.81 MB. Installing color.
_install_module_batch: Using 19.02 MB. Enabling color.
_install_module_batch: Using 19.06 MB. Done with color.
New memory usage: 19.06 MB
Current memory usage: 19.06 MB. Now installing comment via _install_module_batch
_install_module_batch: Using 19.06 MB. Installing comment.
_install_module_batch: Using 19.77 MB. Enabling comment.
_install_module_batch: Using 19.88 MB. Done with comment.
New memory usage: 19.88 MB
Current memory usage: 19.88 MB. Now installing contextual via _install_module_batch
_install_module_batch: Using 19.88 MB. Installing contextual.
_install_module_batch: Using 19.88 MB. Enabling contextual.
_install_module_batch: Using 19.92 MB. Done with contextual.
New memory usage: 19.92 MB
Current memory usage: 19.92 MB. Now installing dashboard via _install_module_batch
_install_module_batch: Using 19.92 MB. Installing dashboard.
_install_module_batch: Using 20.02 MB. Enabling dashboard.
_install_module_batch: Using 20.06 MB. Done with dashboard.
New memory usage: 20.06 MB
Current memory usage: 20.06 MB. Now installing help via _install_module_batch
_install_module_batch: Using 20.06 MB. Installing help.
_install_module_batch: Using 20.05 MB. Enabling help.
_install_module_batch: Using 20.1 MB. Done with help.
New memory usage: 20.09 MB
Current memory usage: 20.09 MB. Now installing image via _install_module_batch
_install_module_batch: Using 20.09 MB. Installing image.
_install_module_batch: Using 20.49 MB. Enabling image.
_install_module_batch: Using 20.57 MB. Done with image.
New memory usage: 20.57 MB
Current memory usage: 20.57 MB. Now installing menu via _install_module_batch
_install_module_batch: Using 20.57 MB. Installing menu.
menu_install: Using 20.79 MB. Listing system menus.
menu_install: Using 20.79 MB. Saving menu navigation.
menu_install: Using 20.8 MB. Saving menu management.
menu_install: Using 20.8 MB. Saving menu user-menu.
menu_install: Using 20.8 MB. Saving menu main-menu.
menu_install: Using 20.8 MB. Saving menu secondary-menu.
menu_install: Using 20.8 MB. Finished.
_install_module_batch: Using 20.75 MB. Enabling menu.
_install_module_batch: Using 25.64 MB. Done with menu.
New memory usage: 25.64 MB
Current memory usage: 25.64 MB. Now installing path via _install_module_batch
_install_module_batch: Using 25.64 MB. Installing path.
_install_module_batch: Using 25.7 MB. Enabling path.
_install_module_batch: Using 25.74 MB. Done with path.
New memory usage: 25.74 MB
Current memory usage: 25.74 MB. Now installing taxonomy via _install_module_batch
_install_module_batch: Using 25.74 MB. Installing taxonomy.
_install_module_batch: Using 26.16 MB. Enabling taxonomy.
_install_module_batch: Using 26.28 MB. Done with taxonomy.
New memory usage: 26.28 MB
Current memory usage: 26.28 MB. Now installing dblog via _install_module_batch
_install_module_batch: Using 26.28 MB. Installing dblog.
_install_module_batch: Using 26.31 MB. Enabling dblog.
_install_module_batch: Using 26.38 MB. Done with dblog.
New memory usage: 26.38 MB
Current memory usage: 26.37 MB. Now installing search via _install_module_batch
_install_module_batch: Using 26.38 MB. Installing search.
_install_module_batch: Using 26.64 MB. Enabling search.
_install_module_batch: Using 26.73 MB. Done with search.
New memory usage: 26.73 MB
Current memory usage: 26.72 MB. Now installing shortcut via _install_module_batch
_install_module_batch: Using 26.73 MB. Installing shortcut.
_install_module_batch: Using 26.84 MB. Enabling shortcut.
_install_module_batch: Using 26.92 MB. Done with shortcut.
New memory usage: 26.92 MB
Current memory usage: 26.92 MB. Now installing toolbar via _install_module_batch
_install_module_batch: Using 26.92 MB. Installing toolbar.
_install_module_batch: Using 26.96 MB. Enabling toolbar.
_install_module_batch: Using 27.01 MB. Done with toolbar.
New memory usage: 27.01 MB
Current memory usage: 27.01 MB. Now installing overlay via _install_module_batch
_install_module_batch: Using 27.01 MB. Installing overlay.
_install_module_batch: Using 27.1 MB. Enabling overlay.
_install_module_batch: Using 27.16 MB. Done with overlay.
New memory usage: 27.16 MB
Current memory usage: 27.16 MB. Now installing field_ui via _install_module_batch
_install_module_batch: Using 27.16 MB. Installing field_ui.
_install_module_batch: Using 27.21 MB. Enabling field_ui.
_install_module_batch: Using 27.27 MB. Done with field_ui.
New memory usage: 27.26 MB
Current memory usage: 27.26 MB. Now installing file via _install_module_batch
_install_module_batch: Using 27.26 MB. Installing file.
_install_module_batch: Using 27.76 MB. Enabling file.
_install_module_batch: Using 27.83 MB. Done with file.
New memory usage: 27.83 MB
Current memory usage: 27.83 MB. Now installing rdf via _install_module_batch
_install_module_batch: Using 27.83 MB. Installing rdf.
_install_module_batch: Using 27.94 MB. Enabling rdf.
_install_module_batch: Using 28 MB. Done with rdf.
New memory usage: 28 MB
Current memory usage: 27.99 MB. Now installing default via _install_module_batch
_install_module_batch: Using 28 MB. Installing default.
default_install: Using 28.09 MB. Adding text formats.
default_install: Using 28.14 MB. Enabling blocks.
default_install: Using 28.23 MB. Creating node types.
default_install: Using 28.56 MB. Creating taxonomy and image field.
default_install: Using 28.7 MB. Setting up roles and permissions.
default_install: Using 28.83 MB. Rebuilding menu.
default_install: Using 33.08 MB. Finished.
_install_module_batch: Using 32.93 MB. Enabling default.
_install_module_batch: Using 32.87 MB. Done with default.
New memory usage: 32.87 MB
Submitting information to task install_configure_form... done.
Running task install_finished... 
Using 35.28 MB. Now resetting theme and module data.
Using 35.35 MB. Now flushing all caches.
Using 35.5 MB.
Using 35.58 MB. Caching the schema now.
Using 35.59 MB. Now running cron.
Using 35.66 MB.

Arancaytar: I'm curious, when you load the front page after install, what's the memory usage there? If it's still ~35MB then we probably need to go down the road of splitting up the code into multiple files. But if it's more like ~20MB, it might be worth exploring how we can make the installer periodically dump its cache as modules are installed as you suggested.

I enabled the devel performance module and looked at the memory footprint. It seems that standard page loads are only around 20-21MB, so the installer does seem to be the bottleneck. I haven't tried to generate bulk content, but I have a hunch that the registries and boot process outweighs the footprint of the content that is actually displayed.

Edit: On a front page with the full 50 nodes, the memory usage goes up to 25MB.

20-30M without APC sounds about right to me on a normal install (i had it as low as 6 or 7MB without APC).

Those memory numbers are really, really interesting.

@Arancatayar - would you be up for running xdebug on the installer, then following the same process as here http://drupal.org/node/513984#comment-2279256 to get the memory usage per-function? That should show which function or group of functions is hogging all the memory.

Can we please make sure that those instructions get put into the handbook either in http://drupal.org/node/79237 or as a sub-page there? I have no idea how anyone would find those in some totally random issue without having a catch to point them there. ;)

Added: http://drupal.org/node/659980

I am able to run xdebug and get a raw trace file, but the log analyzer is written in a syntax that is not compatible with PHP 5.2.10.

Ah, 5.3.0+. Bleh. Well, it shouldn't be that tricky to convert it to create_function() syntax.

Well, for what it's worth, here are my results. They look kind of weird in part (over 1 GB? What the?)

Showing the 100 most costly calls sorted by 'memory-own'.

                                               Inclusive        Own
function                               #calls  time     memory  time     memory
-------------------------------------------------------------------------------
get_defined_constants                    3245  3.1668 1296540192  3.1668 1296540192
preg_match_all                           3419  0.7076 50666768  0.7076 50666768
explode                                 90507  1.2286 49666696  1.2286 49666696
system_schema                             139  0.1243 42685456  0.1243 42685456
file_uri_scheme                         86454  5.0210 80916736  2.7468 35215760
array_merge                             11905  0.2190 31570560  0.2190 31570560
_menu_router_cache                        925  0.2176 27897240  0.1473 27164048
DatabaseCondition->compile               7967  1.3429 32495200  0.7161 23116144
get_defined_functions                      22  0.0425 19569624  0.0425 19569624
preg_split                              29195  0.4683 17041392  0.4683 17041392
PDOStatement->fetch                      2420  0.0840 11723896  0.0840 11723896
module_list                              2463  0.3909 12874200  0.0752 11353672
drupal_load                                87  0.1684 12056368  0.1007 10392040
SelectQuery->addField                    5877  0.1363  9523376  0.1363  9523376
InsertQuery->values                      1252  0.1392 11892720  0.0807  8363320
DatabaseCondition->condition             5236  0.2520  8047232  0.1781  8047232
_drupal_depth_first_search               4320  0.1437  9162512  0.1197  7935728
readdir                                 127618  1.9717  7886992  1.9717  7886992
PDOStatement->execute                    5684  4.9772  7149744  4.9772  7149744
DatabaseConnection->defaultOptions       5684  0.1242  6735272  0.1242  6735272
stripslashes                            58390  0.7421  6514888  0.7421  6514888
_menu_link_build                          913  0.0454  6025656  0.0454  6025656
file_get_contents                        3419  0.4445  5407304  0.4445  5407304
DatabaseCondition->mapConditionOperator  6006  0.1394  4974824  0.1392  4973912
implode                                 20809  0.3194  4958520  0.3194  4958520
DatabaseConnection->expandArguments      5684  1.0969  4890248  0.5210  4886776
DatabaseStatementBase->fetchAllAssoc       48  0.0172  4819952  0.0158  4819952
opendir                                 10260  0.2187  4629736  0.2187  4629736
install_begin_request                       1  0.0978  7376128  0.0624  4626736
_menu_router_build                          4  0.4772  7792744  0.2036  4486128
SelectQuery->addJoin                     1584  0.0530  4421368  0.0530  4421368
DatabaseStatementBase->execute           5684  5.7953 11224536  0.5457  4074792
array_values                             1603  0.0317  4016368  0.0317  4016368
variable_get                             1519  0.0392  3817776  0.0392  3817776
SelectQuery->__toString                  1853  0.6958  5324784  0.2558  3817680
menu_get_ancestors                        921  0.1656  7276560  0.1222  3774072
drupal_parse_info_format                 3245  9.3794 1370340808  2.7139  3735856
DatabaseConnection->select               1571  0.3565 12903416  0.0705  3715576
system_get_files_database                  35  0.2057  3706504  0.1021  3670280
node_schema                                36  0.0105  3571680  0.0105  3571680
t                                        1862  0.1643  4092312  0.0810  3465632
array_keys                               1826  0.0522  3321448  0.0522  3321448
func_get_args                            1294  0.0251  3216488  0.0251  3216488
module_hook                              8756  0.3293  2955480  0.2075  2955480
DatabaseCondition->__construct           4485  0.0946  2946664  0.0946  2946664
strtr                                    7384  0.2024  2709880  0.2024  2709880
drupal_parse_dependency                  1280  0.1093  3468432  0.0587  2672696
UpdateQuery->__toString                   636  0.1738  3178592  0.0653  2510928
array_intersect_assoc                     672  0.0305  2467584  0.0305  2467584
system_list                                69  0.3032  3020608  0.0862  2375568
serialize                                6794  0.1833  2284136  0.1833  2284136
SelectQuery->__construct                 1571  0.2840  9183968  0.1279  2203712
DatabaseConnection->prefixTables         6317  0.5313  4713192  0.2552  2143312
Database::getConnection                  6022  0.2054  2079480  0.2022  2073808
DatabaseSchema_mysql->getFieldTypeMap     465  0.0109  2052360  0.0109  2052360
_drupal_bootstrap_full                      1  0.0338  2027576  0.0234  1980296
_menu_link_parents_set                    829  0.0398  1923664  0.0398  1923664
module_load_include                       775  0.7756  2125664  0.0820  1788112
user_schema                                28  0.0044  1749600  0.0044  1749600
_module_build_dependencies                 32  0.4091 11733096  0.0925  1691816
rtrim                                   29204  0.3754  1688528  0.3754  1688528
system_system_info_alter                 2945  0.0557  1434352  0.0557  1434352
drupal_install_initialize_database        413  0.0240  1396312  0.0234  1373896
drupal_get_filename                      2745  0.7328  1408656  0.0712  1361416
array_slice                              2669  0.0386  1343808  0.0386  1343808
DatabaseConnection->query                5684  8.2750 30479072  0.5054  1328088
PDO->prepare                              353  0.0323  1380176  0.0141  1308792
SelectQuery->countQuery                   309  0.0804  2732128  0.0264  1268552
comment_schema                             24  0.0043  1245304  0.0043  1245304
InsertQuery_mysql->__toString             427  0.2601  5016992  0.1345  1237776
_drupal_schema_initialize                 364  0.0167  1231432  0.0167  1231432
st                                        452  0.0337  1295792  0.0241  1215400
Query->__construct                       3577  0.0848  1212720  0.0848  1212720
DatabaseConnection_mysql->mapConditionOperator  6006  0.1129  1212376  0.1129  1212376
md5                                     16388  0.2200  1180160  0.2200  1180160
_install_configure_form                     1  0.3883  2301064  0.0122  1150408
_theme_process_registry                     3  0.0493  1551408  0.0207  1130832
drupal_detect_database_types                2  0.0472  1284000  0.0289  1075160
drupal_check_profile                        1  8.3349  1295648  0.0178  1046184
_format_date_callback                    1362  0.0270  1044096  0.0268  1040672
SelectQuery->condition                   3734  0.3793  6784872  0.1473  1021480
DatabaseConnection->update                636  0.0876  2555400  0.0261   971296
UpdateQuery->__construct                  636  0.0616  1583744  0.0327   947904
preg_replace                             3939  0.1016   902592  0.1016   902592
MergeQuery_mysql->__toString              311  0.0509  1353032  0.0297   896896
taxonomy_schema                            17  0.0024   881408  0.0024   881408
DeleteQuery->__construct                  545  0.0514  1356688  0.0275   808984
DatabaseConnection->insert                461  0.0483  1402208  0.0192   800544
dashboard_system_info_alter              1671  0.0428   830720  0.0375   797248
Query->nextPlaceholder                   7601  0.1352   776912  0.1352   776912
system_menu                                 4  0.0022   737144  0.0021   731872
dirname                                  7618  0.1099   710784  0.1099   710784
require_once                               60  0.0126   752120  0.0099   705448
unserialize                               739  0.0161   695512  0.0161   695512
DatabaseSchema_mysql->processField        465  0.0723  2685648  0.0177   692984
DatabaseCondition->__clone               1236  0.0323   636928  0.0323   636928
{main}                                      1  54.9787 43629000  0.0137   634248
SelectQuery->addExpression                311  0.0091   631000  0.0091   631000
_field_sql_storage_schema                  26  0.0071   687176  0.0042   625328
DatabaseConnection->delete                545  0.0724  1960472  0.0210   603656

Also, the calls are unbelievable. Consider that get_defined_constants is called *only* in drupal_parse_info_format(). There is no way 3245 info files exist in the Drupal code-base... But I do know that all this data was aggregated from a single installation run of the default profile.

Edit: And yep, 3245 calls to the info parser for a single default installation. Every single info file in the code base is parsed 33 times, triggering a call to get_defined_constants() every time. Statically caching drupal_parse_info_file() did not decrease the peak memory usage, but it did cut the installation time by a third.

Note that these statistics are useful for finding how memory intensive individual functions are, but not so much for finding the leaks that put the cumulative usage over the 32M limit.

Another resource hog is the stream wrapper handling being called 90,000 times, which seems entirely useless when parsing code files during installation.

Damn, this is cool (insert embarrassed smiley).

This is indeed not very useful for working out why memory usage is so high at the end, and I can't think of a way to isolate a trace file to just one request like that.

file_uri_scheme() is called a bunch of times on normal requests too, but I didn't see a way out of that yet.

"Statically caching drupal_parse_info_file() did not decrease the peak memory usage, but it did cut the installation time by a third."

Can we open a separate issue for this? 300% speedup on installation, if that's really the case, would be really nice to have.

Just down by a third, not down to a third. Still, it was substantial. I split the subject off to #661420: Installation of modules is hugely inefficient.

Just cross posting this patch: http://drupal.org/node/358815 - which might be some of the same problem.

Subscribe.

I can cut the default profile's installation's peak usage by around 7MB, almost getting it under the 32M limit, by inserting the menu routers one at a time instead of building up a big query object (ie. executing the query once for each router).

The time cost caused by repetitive database queries appears to be between 0.5 and 1.5 seconds. That is not negligible, but seems worth it for 7MB - particularly since the module installation issue has some proposals that can cut execution time to 6-7 seconds anyway.

Diff:

@@ -3258,9 +3258,10 @@ function _menu_router_save($menu, $masks
       'weight' => $item['weight'],
       'file' => $item['include file'],
     ));
+    $insert->execute();
   }
   // Execute insert object.
-  $insert->execute();
   // Store the masks.
   variable_set('menu_masks', $masks);
 

Before:

Installation took 15.193 seconds and used 39.87 MB.
Installation took 15.805 seconds and used 39.87 MB.
Installation took 16.017 seconds and used 39.87 MB.

After:

Installation took 16.423 seconds and used 32.5 MB.
Installation took 16.674 seconds and used 32.5 MB.
Installation took 16.446 seconds and used 32.5 MB.

Is it possible we are facing some known PHP issues here?

Before visiting this issue I posted a comment with 5 possible PHP problems in #661420: Installation of modules is hugely inefficient. I now realize my comment on the other issue is maybe more relevant for this issue.

Best regards,

Olivier

#49 seems to be very proper use of prepared statements!

instead of using an 12X sized dataset 1 time,
we're using 12 times an 1X sized dataset.
(just an example.)

Yeah if multi-inserts are taking up that much memory, that's a real problem elsewhere - can we open a new issue for that?

7 MB would surprise me, but perhaps not on menu rebuild.

Insert statements keep the record they are going to insert as array data within the query object. If it's a multi-insert, then it's a multi-dimensional array. If you have a crap-ton of records to insert, you have a correspondingly large crap-ton of entries in that array. :-) It should scale linearly, but a menu rebuild is an edge case where you're inserting hundreds, perhaps a thousand records at once. I can see that causing memory issues.

At one point we discussed having a ->executeEveryXRecords() method on insert statements, but decided that was really the caller's responsibility to do. So we absolutely could chunk the insert up into pieces without dropping multi-insert completely here. What an optimal number is I don't know off hand, but we could start with something like 10 or 20 and test from there. Once the code is converted it's dead simple to tweak the number.

If we do that, though, we should absolutely wrap this function in a transaction. Honestly I'm surprised it isn't already. Wiping and rebuilding the menu tables really should be atomic. :-)

Here's a patch that batches the inserts in groups of 20. I also checked and the transaction really belongs a step higher in menu_rebuild(), so I added it there since we're increasing the number of queries that could potentially break the system entirely if they fail. :-)

We can haz benchmarks?

The code-style fix in this patch conflicts with core; probably fixed already.

Unfortunately, my benchmark shows a very small improvement for this patch. I'll try to find out why.

Before:
16.351s 39.87MB
17.775s 39.87MB
17.642s 39.87MB

After:
18.289s 39.23MB
16.429s 39.23MB
15.954s 39.23MB

Oops. Code error.

  $num_records = 0;

  foreach ($menu as $path => $item) {

    $query->values(...);
    ++$num_records
    if ($num_records == 20) {
       $query->execute();
    }

Of course, unless you reset $num_errors to 0, that is only going to fire for the first 20 records.

Fixed patch is here.

New benchmarks:

After (for 20):
15.753 32.95
15.338 32.95
15.733 32.95

After (for 10):
15.760 32.7
15.806 32.7
15.786 32.7

After (for 2):
16.088 32.54
15.946 32.54
15.913 32.52

30:
15.210 33.12
15.245 33.12
15.909 33.12

40:
15.222 33.54
15.615 33.54
15.787 33.54

70:
15.790 34.71
15.411 34.71
15.872 34.71

Shows that 20 really does seem to be the optimal trade-off.

From a quick look I thought the new DB layer uses transactions, when available, on ->insert already. Isn't this nesting transactions? I thought MySQL didn't support nesting. I could very well be wrong on all of this though.

Well, it's definitely working, so either MySQL is dropping the inner transaction without breaking, or there is no inner transaction going on.

Edit: The transaction wrapping is in the degenerate case, when each row requires its own database query. Obviously, MySQL supports inserting multiple rows in a single query, no transaction is needed or used. However, it would be a good idea to test this patch with an SQLite database.

There's no implicit transactions within the database layer afaik - each has been added case by case - see node_save() for example.

The DB layer handles nesting transactions in PHP-space safely. The MySQL InsertQuery implementation does not use transactions anyway, it uses multi-insert statements (which are the issue here). Insert queries also should self-guard against empty inserts, so the edge case where we are inserting an exact multiple of 20 records is already handled.

We are building a fortress around menu_rebuild() [lock API, transaction, custom batched inserts, ...]. Thats a code smell to me. I wish we would refactor that rebuild, even at this stage of the release. I'm +1 for this patch, I'm just saying ...

How should it be refactored, though? I'm not sure how the fortress is a bad thing here - it is a fairly critical system change after all.

Well truthfully I think a lock *should* include a transaction by default. We can't change that now, but for future reference... The "custom batched inserts" isn't a fortress or even custom. It's a feature of how the database API works that you can do that when necessary. Looks like here, it's necessary.

There are issues to refactor menu_rebuild(), but they involve comparing the entire database array to the entire hook_menu() + _alter() array, which doesn't bode well for bringing memory use down.

I've previously measured around 5mb taken up by menu_rebuild(), looks like we're cutting down a big percentage of that here which is great. I don't see that we can refactor it without something like menu_router_save() and kill hook_menu() - but that's not for D7 and has it's own issues I think.

OK sure, menu rebuild refactor will happen next release.

@Crell - I agree that locks should be surrounded by transactions by default. I do see that as a D7 thing in case you were on the fence and needed code reviewer :)

Locks surrounded by transactions also sounds D7-ish to me, that's not an API change which affects modules porting to any extent at all, and probably saves a lot of work when people try to convert to the locking framework later instead of variable semaphores etc.

Well, it is D8-ish if we need to change the locking API at all, since I'm not entirely sure how we'd do it currently. :-) That's not a performance question, though, so it's OT for this thread. Let's open a new one for that if you want to push it. For now, #59 needs some updating as above.

Which changes need to be made to #59?

The only explicit suggestion I can see is #64 about the edge case being implicitly handled - but that would just take us back to patch #58 without the edge case condition...

Actually #57, I think, but you're right, that's the good one.

Since the bot is ignoring us, I'm going to mark #57 RTBC. Let's hope we don't regret that. :-)

Subscribing.

After what we've seen today... let's wait for the bot. :P

Also it's not 100% clear which patch is the good one, can someone re-upload?

It may seem an stupid question (this is my first comment despite some "Suscribing") but does #57 really solve the problem? I mean, Arancaytar wrote that the peak memory usage was 32.95 MB after applying the patch, which is a remarkable improvement, but it's still over the 32M memory limit.

No, but it's one step further..

I got a bit a bit over 29MB with both (around 37MB with just the other one). Both of these changes together would put us back under the limit, though it would be a good idea to investigate further and gain a bit of a safety margin.

Could we try to see how memory usage evolves as a function of the number of include() statements?

I'm not sure whether we have a simple way to replace some occurrences with the inline code just for a try. Probably some shell script with pattern substitution could do the job?

Let's get #77 committed before it breaks again. It is a memory improvement and safety improvement. We can do more after that part is in in this and other threads. (Whether it gets us past the 32 MB magic barrier is irrelevant. Less memory usage == good.)

+++ includes/menu.inc	21 Dec 2009 10:11:59 -0000
@@ -2249,21 +2249,30 @@ function menu_rebuild() {
   return TRUE;
+
 }

Please revert.

+++ includes/menu.inc	21 Dec 2009 10:11:59 -0000
@@ -3258,8 +3269,16 @@ function _menu_router_save($menu, $masks
-  // Execute insert object.
+
+  // Execute any remaining records.

s/Execute/Insert/

The added newline seems wrong here.

Powered by Dreditor.

Removed the first newline, and changed the word (the first Execute is correct, since it refers to the query - the second one indeed should be Insert).

The second blank is debatable - I like loops to stand clear, but a counter-argument is that the "any remaining" insert is in some ways the "final iteration" of the loop and semantically a part of the "code paragraph". Removed the second newline too.

I would disagree as well and include the extra blank line there, as I see them as separate code stanzas. Frankly "execute" is accurate as well, but I'm not going to belabor the point.

#90 and #77 are identical save for stylistic non-code stuff. Drieschick, please commit one of them soon. :-)

I went with #90. We could always clean up code style en-masse in a separate patch.

As Crell mentions, menu_rebuild is pretty edge-casey in the number of queries it fires at the same time. But I do worry about this coming up elsewhere randomly in contrib as well. Hrm...

Anyway, I've committed this to HEAD as a starting point. Marking back to active since it sounded like we wanted to do further digging here.

Yay! Installation of HEAD with default profile now peaks at 32.95MB usage as opposed to 40MB. We're nearly there! :)

I've been able to install Head with only 32MB for a while now. Arancaytar can you try an install with the ini set to 32MB?

I wonder if the profile functions themselves are pushing you over now? Or maybe PHP will use more memory if it has access to it...

Revisiting #44 I am puzzled why readdir is invoked 127618 times during install.

ShutterFreak, the bench in 44 was done before #358815: drupal_get_install_files() is slow with a large tree landed. It should be much less than that now.

I've been able to install Head with only 32MB for a while now. Arancaytar can you try an install with the ini set to 32MB?

Nope, still failing.

$ drupal install CLEAN clean
Drupal Manager [Ermarian Network]
CLEAN clean
Deleting site 'd7clean_' database
Connecting to localhost as drupal:drupal
Selecting database drupal
Delete all tables in drupal.d7clean_%? Enter to confirm, Ctrl-C to abort.

  d7clean_actions...  d7clean_authmap...  d7clean_batch...  d7clean_block...  d7clean_block_custom...  d7clean_block_node_type...  d7clean_block_role...  d7clean_blocked_ips...  d7clean_cache...  d7clean_cache_block...  d7clean_cache_bootstrap...  d7clean_cache_field...  d7clean_cache_filter...  d7clean_cache_form...  d7clean_cache_image...  d7clean_cache_menu...  d7clean_cache_page...  d7clean_cache_path...  d7clean_comment...  d7clean_date_format_locale...  d7clean_date_format_type...  d7clean_date_formats...  d7clean_field_config...  d7clean_field_config_entity_type...  d7clean_field_config_instance...  d7clean_field_data_body...  d7clean_field_data_field_image...  d7clean_field_data_taxonomy_tags...  d7clean_field_data_title...  d7clean_field_revision_body...  d7clean_field_revision_field_image...  d7clean_field_revision_taxonomy_tags...  d7clean_field_revision_title...  d7clean_file...  d7clean_filter...  d7clean_filter_format...  d7clean_flood...  d7clean_history...  d7clean_image_effects...  d7clean_image_styles...  d7clean_menu_custom...  d7clean_menu_links...  d7clean_menu_router...  d7clean_node...  d7clean_node_access...  d7clean_node_comment_statistics...  d7clean_node_revision...  d7clean_node_type...  d7clean_queue...  d7clean_rdf_mapping...  d7clean_registry...  d7clean_registry_file...  d7clean_role...  d7clean_role_permission...  d7clean_search_dataset...  d7clean_search_index...  d7clean_search_node_links...  d7clean_search_total...  d7clean_semaphore...  d7clean_sequences...  d7clean_sessions...  d7clean_shortcut_set...  d7clean_shortcut_set_users...  d7clean_system...  d7clean_taxonomy_index...  d7clean_taxonomy_term_data...  d7clean_taxonomy_term_hierarchy...  d7clean_taxonomy_vocabulary...  d7clean_url_alias...  d7clean_users...  d7clean_users_roles...  d7clean_variable...  d7clean_watchdog...
Done.
Resettings settings.php
Starting installation...

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 76 bytes) in /home/arancaytar/Documents/devel/drupal/core/CLEAN/includes/menu.inc on line 3188

However, it works with 33M:

Starting installation...
Done. Installation took 8.918 seconds and used 32.86 MB.

Are you testing this with a web install? That is known to use less memory than the CLI script (see #33)...

Memory usage is going to differ depending on which PHP extensions you have loaded, APC or not. It's quite reasonable to expect some variation between machines.

So we're right at the 32 MB limit right now. That means to be safe, we still want to try and shave a couple meg off. Any idea where else we could do that? Any other big arrays we can kill?

Note: If we don't get to a safety margin of about 2-4MB (at least), we should probably increase the memory requiremen before the alpha release, see webchick's comment #31. A "fatal error, out of memory, installation partially failed" is not a good first impression, when the alternative is a helpful message telling them to increase their memory limit...

I don't think that is a alpha blocker, maybe a RC blocker...

the alpha release is only for dev

the drush installsite command is pretty nice for CLI installs of any install profile

Figured i would chime in here with more interesting data. I just enhanced Drush so it reports memory usage and watchdog entries. Here is the latest out put for installsite command.

Starting Drupal installation. This takes 30 seconds or so ... [0.35 sec, 5.84 MB]                                [ok]
WD system: user module enabled. [7.29 sec, 19.48 MB]                                                                  [info]
WD system: node module enabled. [7.9 sec, 21.06 MB]                                                                   [info]
WD system: filter module enabled. [8.04 sec, 21.62 MB]                                                                [info]
WD system: text module enabled. [8.15 sec, 21.83 MB]                                                                  [info]
WD system: options module enabled. [8.24 sec, 21.94 MB]                                                               [info]
WD system: number module enabled. [8.33 sec, 22.09 MB]                                                                [info]
WD system: list module enabled. [8.41 sec, 22.21 MB]                                                                  [info]
WD system: field_sql_storage module enabled. [8.57 sec, 22.48 MB]                                                     [info]
WD system: field module enabled. [8.99 sec, 23.68 MB]                                                                 [info]
WD system: block module enabled. [9.71 sec, 24.07 MB]                                                                 [info]
WD system: color module enabled. [9.83 sec, 24.41 MB]                                                                 [info]
WD system: comment module enabled. [10.5 sec, 25.26 MB]                                                               [info]
WD system: contextual module enabled. [10.64 sec, 25.36 MB]                                                           [info]
WD system: dashboard module enabled. [10.75 sec, 25.52 MB]                                                            [info]
WD system: help module enabled. [10.86 sec, 25.56 MB]                                                                 [info]
WD system: image module enabled. [11.29 sec, 26.06 MB]                                                                [info]
WD field: Updating field type image with module image. [11.4 sec, 26.12 MB]                                         [notice]
WD system: menu module enabled. [11.5 sec, 26.33 MB]                                                                  [info]
WD system: path module enabled. [16.09 sec, 28.24 MB]                                                                 [info]
WD system: taxonomy module enabled. [16.58 sec, 28.75 MB]                                                             [info]
WD field: Updating field type taxonomy_term_reference with module taxonomy. [16.7 sec, 28.85 MB]                    [notice]
WD system: dblog module enabled. [16.79 sec, 28.88 MB]                                                                [info]
WD system: search module enabled. [17.31 sec, 29.27 MB]                                                               [info]
WD system: shortcut module enabled. [17.76 sec, 29.47 MB]                                                             [info]
WD system: toolbar module enabled. [17.93 sec, 29.59 MB]                                                              [info]
WD system: overlay module enabled. [18.07 sec, 29.76 MB]                                                              [info]
WD system: field_ui module enabled. [18.2 sec, 29.88 MB]                                                              [info]
WD system: file module enabled. [18.34 sec, 30.5 MB]                                                                  [info]
WD field: Updating field type file with module file. [18.45 sec, 30.57 MB]                                          [notice]
WD system: rdf module enabled. [18.56 sec, 30.7 MB]                                                                   [info]
WD system: standard module enabled. [23.38 sec, 31.86 MB]                                                             [info]
WD system: update module enabled. [26.36 sec, 34.79 MB]                                                               [info]
WD user: Session opened for admin. [27.02 sec, 34.78 MB]                                                            [notice]
Command dispatch complete [31.25 sec, 35.51 MB]                                                                     [notice]
Timer 'page' is  31.04 sec. [31.25 sec, 35.5 MB]                                                                     [timer]
Peak memory usage was 36.99 MB [31.25 sec, 35.5 MB]                                                                 [memory]

Some observations:

• The huge memory jump you see before 'user module enabled' happens during drupal_install_modules(). drupal still installs a batch of modules and then enables a batch of modules. This has to change before release in order to prevent horrid dependency bugs. But apart from that, we do no watchdog() for installing so I have no insight into memory usage here.
• There is a noticeable jump after menu is enabled and before path is enabled. We can't blame this on path.install, since there is no such file. I tried same experiment with #629794: Scaling issues with batch API applied and got same results.
• Similar slowness during "standard module" (i.e. standard profile) enabling and again during update.module enabling.

drush installsite --debug --yes. this shows debug info like memory snapshots and skips confirmation which would just throw off the timer.

So... alpha's tomorrow. What's a safe value to set this minimum to? Could someone run a couple of tests?

@webchick: Installation is almost definitely safe at 40M; even in drush it seems to peak at 36M.

(Furthermore, Drupal runs comfortably within 32M after installation/while using minimal.profile, so maybe the memory requirement should be added to default.profile as default_requirements instead?)

I got over 64MB usage when visiting the Simpletest overview the first time (probably on parsing all those files) which I haven't been able to reproduce. Trying it right now with a new install uses only 27MB. I'll investigate more thoroughly tonight; but even if it turns out to be a problem, this should probably be a simpletest_requirements() change, and not drag up the memory requirement of Drupal itself.

The minimal require php memory should be different if we have others php modules ON. Like rewrite/apc/vhosts/...

So 40Mb if the peak now is 32Mb it seams ok to me.
But this only work at the instalation and with minimal profile.

So we must have at least four minimals requirements:

For:
Instalation Minimal Profile: 40MB
Instalation Standard Profile: ?? MB
One normal Drupal site with all core modules ON: ?? MB
To execute the SimpleTest: ?? MB

And say any module addition will increase the php-memory that is required..

Variable requirements has been discussed in #309457: Allow profiles to specify required memory in .info file - we can't do that here too.

It's almost tempting to remove one module from the default profile to try push this under 32mb - I vote for color.module.

Remove core modules from the default instalation to reduce php-memory isn't the solution..
Because follow that statement we will have to remove one more at D8.

Or we implement a new feature to remove the memory that we don't needed, or we increase the memory requirement.

For the purposes of alpha 1, I just committed a patch to raise this limit temporarily to 40M. But let's keep at it here so we can get this back down to 32M prior to 7.0. :)

Variable requirements has been discussed in #309457: Allow modules to specify required memory in .info file - we can't do that here too.

In individual modules during run-time, this is something else, but if we know for certain that the installation of default.profile takes a certain amount of memory (32M on the web, up to 36M in CLI, but certainly under 40M), this is a pretty fixed number. The arguments about "depending on datasets" or unforeseen circumstances don't really apply in this case.

Wouldn't a solution be to split the install process in many parts? Maybe via AJAX calls to install each module - of course, this would require JS to be enabled.

That's not possible on a command-line install. The web installer already has a fairly robust batch API, which is why it's not using as much memory...

I ran into #550124-28: Remove prepared statement caching while playing around with hook_query_alter() last night. Basically, we're caching the compiled output of every single query on the page into memory.

I'm curious if the following line in includes/database/database.inc is commented out:

$this->preparedStatements[$query] = $stmt;

...what does that do for our RAM footprint during install?

Oh, nevermind. I just saw #106, so I can do this myself. :)

HEAD:

webchicks-macbook-pro:core webchick$ drush installsite --debug --yes
*snip lots of stuff*
Peak memory usage was 19.7 MB [16.33 sec, 18.89 MB]                     [memory]

With "patch":

webchicks-macbook-pro:core webchick$ drush installsite --debug --yes
*snip lots of stuff*
Peak memory usage was 18.21 MB [18.76 sec, 17.08 MB]                    [memory]

Well, that shaves off a little. But not several MBs.

But um. Hey! It's down to < 20 MB now?! Can someone confirm those numbers?

Here's the patch if someone wants to play around with it.

Let's try that *without* the 250,000 patches that I've reviewed over the past year. ;)

#1: htaccess.patch queued for re-testing.

FYI, I just retested and webchick's patch shaves 2.5MB from a default install. Look for the MB number reported in each line below.

*** UNPATCHED ***
~/htd/prfr$ drush site-install -y -d
Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.49 sec, 4.3 MB]                                                [bootstrap]
Initialized Drupal 7.0-dev root directory at /Users/mw/htd/prfr [0.51 sec, 5.85 MB]                                         [notice]
Found command: site-install (commandfile=core) [0.56 sec, 5.84 MB]                                                       [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [0.57 sec, 5.88 MB]                                               [bootstrap]
Initialized Drupal site default at sites/default [1.24 sec, 6.11 MB]                                                        [notice]
Drush bootstrap phase : _drush_bootstrap_drupal_configuration() [1.24 sec, 6.11 MB]                                      [bootstrap]
You are about to DROP your 'installsite' database and then CREATE a new one. Do you want to continue? (y/n): y
Sites directory sites/default already exists - proceeding. [1.25 sec, 6.12 MB]                                              [notice]
Calling system(mysql  -hlocalhost -uroot information_schema -e  "DROP DATABASE IF EXISTS installsite")
Calling system(mysql  -hlocalhost -uroot information_schema -e "CREATE DATABASE installsite")
Constant DRUPAL_ROOT already defined install.php:7 [1.93 sec, 6.69 MB]                                                      [notice]
Starting Drupal installation. This takes 30 seconds or so ... [1.94 sec, 7.03 MB]                                        [ok]
WD system: user module enabled. [10.48 sec, 21.09 MB]                                                                         [info]
WD system: node module enabled. [11.34 sec, 22.74 MB]                                                                         [info]
WD system: filter module enabled. [11.59 sec, 23.32 MB]                                                                       [info]
WD system: text module enabled. [11.76 sec, 23.53 MB]                                                                         [info]
WD system: options module enabled. [11.84 sec, 23.64 MB]                                                                      [info]
WD system: number module enabled. [11.95 sec, 23.78 MB]                                                                       [info]
WD system: list module enabled. [12.02 sec, 23.9 MB]                                                                          [info]
WD system: field_sql_storage module enabled. [12.14 sec, 24.18 MB]                                                            [info]
WD system: field module enabled. [12.71 sec, 25.4 MB]                                                                         [info]
WD system: block module enabled. [13.54 sec, 25.79 MB]                                                                        [info]
WD system: color module enabled. [13.68 sec, 26.13 MB]                                                                        [info]
WD system: comment module enabled. [13.96 sec, 27.01 MB]                                                                      [info]
WD system: contextual module enabled. [14.43 sec, 27.15 MB]                                                                   [info]
WD system: dashboard module enabled. [14.51 sec, 27.3 MB]                                                                     [info]
WD system: help module enabled. [14.58 sec, 27.34 MB]                                                                         [info]
WD system: image module enabled. [14.98 sec, 27.84 MB]                                                                        [info]
WD field: Updating field type image with module image. [15.17 sec, 27.9 MB]                                                 [notice]
WD system: menu module enabled. [15.43 sec, 28.11 MB]                                                                         [info]
WD system: path module enabled. [19.24 sec, 30.03 MB]                                                                         [info]
WD system: taxonomy module enabled. [19.73 sec, 30.57 MB]                                                                     [info]
WD field: Updating field type taxonomy_term_reference with module taxonomy. [19.86 sec, 30.66 MB]                           [notice]
WD system: dblog module enabled. [19.93 sec, 30.69 MB]                                                                        [info]
WD system: search module enabled. [20.42 sec, 31.08 MB]                                                                       [info]
WD system: shortcut module enabled. [20.73 sec, 31.28 MB]                                                                     [info]
WD system: toolbar module enabled. [20.88 sec, 31.4 MB]                                                                       [info]
WD system: overlay module enabled. [20.96 sec, 31.58 MB]                                                                      [info]
WD system: field_ui module enabled. [21.04 sec, 31.69 MB]                                                                     [info]
WD system: file module enabled. [21.17 sec, 32.32 MB]                                                                         [info]
WD field: Updating field type file with module file. [21.28 sec, 32.38 MB]                                                  [notice]
WD system: rdf module enabled. [21.4 sec, 32.52 MB]                                                                           [info]
WD system: standard module enabled. [24.34 sec, 33.66 MB]                                                                     [info]
WD system: update module enabled. [25.92 sec, 35.54 MB]                                                                       [info]
WD user: Session opened for admin. [26.18 sec, 35.96 MB]                                                                    [notice]
WD actions: Action 'Publish comment' added. [28.12 sec, 36.3 MB]                                                            [notice]
WD actions: Action 'Unpublish comment' added. [28.13 sec, 36.3 MB]                                                          [notice]
WD actions: Action 'Save comment' added. [28.13 sec, 36.3 MB]                                                               [notice]
WD actions: Action 'Publish content' added. [28.14 sec, 36.3 MB]                                                            [notice]
WD actions: Action 'Unpublish content' added. [28.14 sec, 36.31 MB]                                                         [notice]
WD actions: Action 'Make content sticky' added. [28.14 sec, 36.31 MB]                                                       [notice]
WD actions: Action 'Make content unsticky' added. [28.15 sec, 36.31 MB]                                                     [notice]
WD actions: Action 'Promote content to front page' added. [28.16 sec, 36.31 MB]                                             [notice]
WD actions: Action 'Remove content from front page' added. [28.16 sec, 36.31 MB]                                            [notice]
WD actions: Action 'Save content' added. [28.17 sec, 36.31 MB]                                                              [notice]
WD actions: Action 'Ban IP address of current user' added. [28.17 sec, 36.31 MB]                                            [notice]
WD actions: Action 'Block current user' added. [28.18 sec, 36.31 MB]                                                        [notice]
WD cron: Cron run completed. [28.56 sec, 36.73 MB]                                                                          [notice]
All necessary changes to ./sites/default and ./sites/default/settings.php have been made. They have been set to read-only   [status]
for security. [28.57 sec, 36.71 MB]
Command dispatch complete [28.57 sec, 36.68 MB]                                                                             [notice]
Timer 'page' is  27.33 sec. [28.57 sec, 36.68 MB]                                                                            [timer]
Peak memory usage was 38.25 MB [28.58 sec, 36.68 MB] 

*** PATCHED ***
~/htd/prfr$ drush site-install -y -d
Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.26 sec, 4.3 MB]                                                [bootstrap]
Initialized Drupal 7.0-dev root directory at /Users/mw/htd/prfr [0.31 sec, 5.85 MB]                                         [notice]
Found command: site-install (commandfile=core) [0.55 sec, 5.84 MB]                                                       [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [0.61 sec, 5.88 MB]                                               [bootstrap]
Initialized Drupal site default at sites/default [0.63 sec, 6.11 MB]                                                        [notice]
Drush bootstrap phase : _drush_bootstrap_drupal_configuration() [0.64 sec, 6.11 MB]                                      [bootstrap]
You are about to DROP your 'installsite' database and then CREATE a new one. Do you want to continue? (y/n): y
Sites directory sites/default already exists - proceeding. [0.65 sec, 6.12 MB]                                              [notice]
Calling system(mysql  -hlocalhost -uroot information_schema -e  "DROP DATABASE IF EXISTS installsite")
Calling system(mysql  -hlocalhost -uroot information_schema -e "CREATE DATABASE installsite")
Constant DRUPAL_ROOT already defined install.php:7 [1.03 sec, 6.69 MB]                                                      [notice]
Starting Drupal installation. This takes 30 seconds or so ... [1.04 sec, 7.03 MB]                                        [ok]
WD system: user module enabled. [7.13 sec, 20.5 MB]                                                                           [info]
WD system: node module enabled. [7.62 sec, 21.96 MB]                                                                          [info]
WD system: filter module enabled. [7.72 sec, 22.48 MB]                                                                        [info]
WD system: text module enabled. [7.8 sec, 22.68 MB]                                                                           [info]
WD system: options module enabled. [7.87 sec, 22.79 MB]                                                                       [info]
WD system: number module enabled. [7.94 sec, 22.93 MB]                                                                        [info]
WD system: list module enabled. [8 sec, 23.05 MB]                                                                             [info]
WD system: field_sql_storage module enabled. [8.13 sec, 23.33 MB]                                                             [info]
WD system: field module enabled. [8.52 sec, 24.52 MB]                                                                         [info]
WD system: block module enabled. [9.49 sec, 24.87 MB]                                                                         [info]
WD system: color module enabled. [9.57 sec, 25.2 MB]                                                                          [info]
WD system: comment module enabled. [9.83 sec, 26.07 MB]                                                                       [info]
WD system: contextual module enabled. [10.18 sec, 26.17 MB]                                                                   [info]
WD system: dashboard module enabled. [10.25 sec, 26.31 MB]                                                                    [info]
WD system: help module enabled. [10.33 sec, 26.35 MB]                                                                         [info]
WD system: image module enabled. [10.73 sec, 26.84 MB]                                                                        [info]
WD field: Updating field type image with module image. [10.84 sec, 26.89 MB]                                                [notice]
WD system: menu module enabled. [10.95 sec, 27.09 MB]                                                                         [info]
WD system: path module enabled. [13.61 sec, 28.46 MB]                                                                         [info]
WD system: taxonomy module enabled. [14.08 sec, 28.97 MB]                                                                     [info]
WD field: Updating field type taxonomy_term_reference with module taxonomy. [14.16 sec, 29.04 MB]                           [notice]
WD system: dblog module enabled. [14.26 sec, 29.06 MB]                                                                        [info]
WD system: search module enabled. [14.77 sec, 29.42 MB]                                                                       [info]
WD system: shortcut module enabled. [15.04 sec, 29.61 MB]                                                                     [info]
WD system: toolbar module enabled. [15.14 sec, 29.72 MB]                                                                      [info]
WD system: overlay module enabled. [15.22 sec, 29.89 MB]                                                                      [info]
WD system: field_ui module enabled. [15.3 sec, 30 MB]                                                                         [info]
WD system: file module enabled. [15.41 sec, 30.62 MB]                                                                         [info]
WD field: Updating field type file with module file. [15.49 sec, 30.67 MB]                                                  [notice]
WD system: rdf module enabled. [15.57 sec, 30.8 MB]                                                                           [info]
WD system: standard module enabled. [19.36 sec, 31.54 MB]                                                                     [info]
WD system: update module enabled. [21.21 sec, 33.4 MB]                                                                        [info]
WD user: Session opened for admin. [21.45 sec, 33.8 MB]                                                                     [notice]
WD actions: Action 'Publish comment' added. [23.42 sec, 33.74 MB]                                                           [notice]
WD actions: Action 'Unpublish comment' added. [23.43 sec, 33.74 MB]                                                         [notice]
WD actions: Action 'Save comment' added. [23.43 sec, 33.74 MB]                                                              [notice]
WD actions: Action 'Publish content' added. [23.44 sec, 33.74 MB]                                                           [notice]
WD actions: Action 'Unpublish content' added. [23.44 sec, 33.74 MB]                                                         [notice]
WD actions: Action 'Make content sticky' added. [23.45 sec, 33.74 MB]                                                       [notice]
WD actions: Action 'Make content unsticky' added. [23.45 sec, 33.74 MB]                                                     [notice]
WD actions: Action 'Promote content to front page' added. [23.45 sec, 33.74 MB]                                             [notice]
WD actions: Action 'Remove content from front page' added. [23.46 sec, 33.74 MB]                                            [notice]
WD actions: Action 'Save content' added. [23.47 sec, 33.75 MB]                                                              [notice]
WD actions: Action 'Ban IP address of current user' added. [23.48 sec, 33.75 MB]                                            [notice]
WD actions: Action 'Block current user' added. [23.48 sec, 33.75 MB]                                                        [notice]
WD cron: Cron run completed. [23.87 sec, 34.1 MB]                                                                           [notice]
All necessary changes to ./sites/default and ./sites/default/settings.php have been made. They have been set to read-only   [status]
for security. [23.87 sec, 34.07 MB]
Command dispatch complete [23.88 sec, 34.05 MB]                                                                             [notice]
Timer 'page' is  23.24 sec. [23.88 sec, 34.04 MB]                                                                            [timer]
Peak memory usage was 36.08 MB [23.88 sec, 34.04 MB]                                                                        [memory]

BTW, I have no idea why memory consumption is twice as much as webchick's. Maybe xdebug or other php config is at play here. Would be great if webchick could repeat with latest drush and latest d7.

Numbers on one of my windows machines with latest drush and Head

Head unpatched:

Starting Drupal installation. This takes 30 seconds or so ... [0.33         [ok]
sec, 3.58 MB]
WD system: user module enabled. [6.88 sec, 11.77 MB]                      [info]
WD system: node module enabled. [11.62 sec, 12.73 MB]                     [info]
WD system: filter module enabled. [12.98 sec, 13.06 MB]                   [info]
WD system: text module enabled. [13.49 sec, 13.2 MB]                      [info]
WD system: options module enabled. [14 sec, 13.26 MB]                     [info]
WD system: number module enabled. [14.42 sec, 13.35 MB]                   [info]
WD system: list module enabled. [14.7 sec, 13.42 MB]                      [info]
WD system: field_sql_storage module enabled. [15.18 sec, 13.57 MB]        [info]
WD system: field module enabled. [15.96 sec, 14.31 MB]                    [info]
WD system: block module enabled. [17.25 sec, 14.54 MB]                    [info]
WD system: color module enabled. [17.84 sec, 14.73 MB]                    [info]
WD system: comment module enabled. [18.4 sec, 15.23 MB]                   [info]
WD system: contextual module enabled. [19.24 sec, 15.33 MB]               [info]
WD system: dashboard module enabled. [19.61 sec, 15.42 MB]                [info]
WD system: help module enabled. [19.95 sec, 15.44 MB]                     [info]
WD system: image module enabled. [20.68 sec, 15.73 MB]                    [info]
WD field: Updating field type image with module image. [21.07 sec,      [notice]
15.73 MB]
WD system: menu module enabled. [21.75 sec, 15.89 MB]                     [info]
WD system: path module enabled. [42.82 sec, 17.07 MB]                     [info]
WD system: taxonomy module enabled. [43.8 sec, 17.36 MB]                  [info]
WD field: Updating field type taxonomy_term_reference with module       [notice]
taxonomy. [44.47 sec, 17.39 MB]
WD system: dblog module enabled. [44.7 sec, 17.42 MB]                     [info]
WD system: search module enabled. [45.79 sec, 17.65 MB]                   [info]
WD system: shortcut module enabled. [46.53 sec, 17.78 MB]                 [info]
WD system: toolbar module enabled. [47.11 sec, 17.85 MB]                  [info]
WD system: overlay module enabled. [47.57 sec, 17.95 MB]                  [info]
WD system: field_ui module enabled. [47.98 sec, 18.02 MB]                 [info]
WD system: file module enabled. [48.51 sec, 18.39 MB]                     [info]
WD field: Updating field type file with module file. [48.88 sec,        [notice]
18.39 MB]
WD system: rdf module enabled. [49.08 sec, 18.51 MB]                      [info]
WD system: standard module enabled. [61.23 sec, 19.21 MB]                 [info]
WD system: update module enabled. [64.53 sec, 20.31 MB]                   [info]
WD user: Session opened for admin. [66.22 sec, 20.54 MB]                [notice]
WD actions: Action 'Publish comment' added. [74.48 sec, 20.76 MB]       [notice]
WD actions: Action 'Unpublish comment' added. [74.56 sec, 20.77 MB]     [notice]
WD actions: Action 'Save comment' added. [74.65 sec, 20.77 MB]          [notice]
WD actions: Action 'Publish content' added. [74.74 sec, 20.77 MB]       [notice]
WD actions: Action 'Unpublish content' added. [74.83 sec, 20.78 MB]     [notice]
WD actions: Action 'Make content sticky' added. [74.91 sec, 20.78 MB]   [notice]
WD actions: Action 'Make content unsticky' added. [74.99 sec, 20.78     [notice]
MB]
WD actions: Action 'Promote content to front page' added. [75.08 sec,   [notice]
20.78 MB]
WD actions: Action 'Remove content from front page' added. [75.16       [notice]
sec, 20.78 MB]
WD actions: Action 'Save content' added. [75.24 sec, 20.78 MB]          [notice]
WD actions: Action 'Ban IP address of current user' added. [75.38       [notice]
sec, 20.78 MB]
WD actions: Action 'Block current user' added. [75.46 sec, 20.78 MB]    [notice]
WD cron: Cron run completed. [78.71 sec, 21.03 MB]                      [notice]
All necessary changes to ./sites/default and                            [status]
./sites/default/settings.php have been made. They have been set to
read-only for security. [78.8 sec, 21.02 MB]
Command dispatch complete [78.8 sec, 21.01 MB]                          [notice]
Timer 'page' is  78.61 sec. [78.8 sec, 21.01 MB]                         [timer]
Peak memory usage was 21.74 MB [78.8 sec, 21.01 MB]                     [memory]

Head with no-cache-for-you #120

Starting Drupal installation. This takes 30 seconds or so ... [0.27         [ok]
sec, 3.58 MB]
WD system: user module enabled. [8.73 sec, 11.38 MB]                      [info]
WD system: node module enabled. [15.45 sec, 12.22 MB]                     [info]
WD system: filter module enabled. [16.16 sec, 12.51 MB]                   [info]
WD system: text module enabled. [16.74 sec, 12.64 MB]                     [info]
WD system: options module enabled. [17.21 sec, 12.7 MB]                   [info]
WD system: number module enabled. [17.78 sec, 12.78 MB]                   [info]
WD system: list module enabled. [18.14 sec, 12.86 MB]                     [info]
WD system: field_sql_storage module enabled. [18.81 sec, 13.01 MB]        [info]
WD system: field module enabled. [19.83 sec, 13.73 MB]                    [info]
WD system: block module enabled. [21.38 sec, 13.93 MB]                    [info]
WD system: color module enabled. [21.92 sec, 14.12 MB]                    [info]
WD system: comment module enabled. [22.72 sec, 14.6 MB]                   [info]
WD system: contextual module enabled. [23.81 sec, 14.67 MB]               [info]
WD system: dashboard module enabled. [24.18 sec, 14.76 MB]                [info]
WD system: help module enabled. [24.61 sec, 14.78 MB]                     [info]
WD system: image module enabled. [26.04 sec, 15.06 MB]                    [info]
WD field: Updating field type image with module image. [26.6 sec,       [notice]
15.06 MB]
WD system: menu module enabled. [27.36 sec, 15.21 MB]                     [info]
WD system: path module enabled. [56.46 sec, 16 MB]                        [info]
WD system: taxonomy module enabled. [57.69 sec, 16.29 MB]                 [info]
WD field: Updating field type taxonomy_term_reference with module       [notice]
taxonomy. [58.36 sec, 16.3 MB]
WD system: dblog module enabled. [58.62 sec, 16.33 MB]                    [info]
WD system: search module enabled. [59.76 sec, 16.54 MB]                   [info]
WD system: shortcut module enabled. [60.85 sec, 16.65 MB]                 [info]
WD system: toolbar module enabled. [61.63 sec, 16.71 MB]                  [info]
WD system: overlay module enabled. [62.1 sec, 16.81 MB]                   [info]
WD system: field_ui module enabled. [62.6 sec, 16.88 MB]                  [info]
WD system: file module enabled. [63.15 sec, 17.24 MB]                     [info]
WD field: Updating field type file with module file. [63.68 sec,        [notice]
17.24 MB]
WD system: rdf module enabled. [64 sec, 17.35 MB]                         [info]
WD system: standard module enabled. [80.14 sec, 17.78 MB]                 [info]
WD system: update module enabled. [83.6 sec, 18.87 MB]                    [info]
WD user: Session opened for admin. [84.55 sec, 19.08 MB]                [notice]
WD actions: Action 'Publish comment' added. [91.22 sec, 19.04 MB]       [notice]
WD actions: Action 'Unpublish comment' added. [91.28 sec, 19.04 MB]     [notice]
WD actions: Action 'Save comment' added. [91.34 sec, 19.04 MB]          [notice]
WD actions: Action 'Publish content' added. [91.4 sec, 19.04 MB]        [notice]
WD actions: Action 'Unpublish content' added. [91.45 sec, 19.04 MB]     [notice]
WD actions: Action 'Make content sticky' added. [91.51 sec, 19.04 MB]   [notice]
WD actions: Action 'Make content unsticky' added. [91.57 sec, 19.04     [notice]
MB]
WD actions: Action 'Promote content to front page' added. [91.64 sec,   [notice]
19.04 MB]
WD actions: Action 'Remove content from front page' added. [91.7 sec,   [notice]
19.04 MB]
WD actions: Action 'Save content' added. [91.75 sec, 19.04 MB]          [notice]
WD actions: Action 'Ban IP address of current user' added. [91.81       [notice]
sec, 19.04 MB]
WD actions: Action 'Block current user' added. [91.87 sec, 19.04 MB]    [notice]
WD cron: Cron run completed. [93.89 sec, 19.26 MB]                      [notice]
All necessary changes to ./sites/default and                            [status]
./sites/default/settings.php have been made. They have been set to
read-only for security. [93.95 sec, 19.25 MB]
Command dispatch complete [93.95 sec, 19.24 MB]                         [notice]
Timer 'page' is  93.78 sec. [93.96 sec, 19.23 MB]                        [timer]
Peak memory usage was 20.29 MB [93.96 sec, 19.23 MB]                    [memory]

So around 2MB less memory. Got something weird going on with those long times on this machine, but that'll be down to my setup.

Crell's benchmarks for prepared statement caching or not show no measurable difference if we remove it:

Cache no, Emulate no: 10.16 requests/second
Cache no, Emulate yes: 10.23 requests/second (same as #120)
Cache yes, Emulate no: 10.16 requests/second
Cache yes, Emulate yes: 10.25 requests/second (Current Drupal HEAD)

So I think we should do #120, however it only comments out the caching, and Crell suggested we need additional cleanup elsewhere if we do it, so CNW.

Let's remove prepared statement caching in #550124: Remove prepared statement caching, which means no patch here.

Ok, I just committed #550124: Remove prepared statement caching, so at least one major memory suck is gone.

@moshe: I cvs up -dPC drush HEAD and core, and still get similar numbers to before:

drush site-install -y -d
[snip]
Timer 'page' is  26.32 sec. [26.79 sec, 21.96 MB]
Peak memory usage was 23.04 MB [26.8 sec, 21.96 MB]

It's interesting that you ask about xdebug because I recently totally screwed MAMP and had to reinstall, and it may indeed have been around that time that I did my original tests. But tonight, I'm sure I have xdebug installed, and my memory footprint remains low.

Here's a copy of my xdebug settings:

[xdebug]
zend_extension="/Applications/MAMP/bin/php5/lib/php/extensions/no-debug-non-zts-20060613/xdebug.so"
xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_host=127.0.0.1
xdebug.remote_port=9000
xdebug.idekey=webchick

One thing I do not have is xdebug profiling on. Could that be the difference?

In any case, here's my phpinfo(), which is basically out-of-the-box MAMP: http://webchick.net/files/_temp/phpinfo.php.html

Moshe, could you post your config so we could try to get to the bottom of this memory discrepancy?

Also, if someone could run a test through the GUI and check RAM usage like Arancaytar did earlier, I think we might be able to set this down to 32 and call it good?

I don't have xdebug profiling either. I tried with APC enabled and disabled and no difference. webchick's MAMP beats my newish unibody badly. /me weeps. do you have some ramdisk or something?

Um. I don't know what ramdisk is. I'm guessing no. :) It's just on my regular ol' hard drive in the /Applications folder on my 2.5-year old MacBook Pro. I /do/ have 4GB of RAM in this machine, but I don't think that would make a difference?

@moshe weitzman "do you have some ramdisk or something?"

I haven't heard anyone use ramdisk in YEARS, they kind of died with the 486. Boy could they speed up a process though. back then they were anywhere from 100 to 1000% quicker than the slow hard drives of the time. wonder what the feasibility of putting one of those bad boys on a system today would be like?

guess in showing my age.

Putting all your mysql data into a RAMDISK or similar has dramatic speed advantages. Thats how our pifr test bots get through the whole test suite in 3 minutes. The setup for a pifr test bot is documented at http://qa.drupal.org/performance-tuning-tips-for-D7. See "Run MySQL in memory"

Thanks Moshe

I'll give it a shot on my CentOS box, I'd forgotten all about /tmpfs

One other idea, someone who's hitting the 32M limit, please try deleting every single hook_update_N() in system.install (and ideally the same in other module's .install file), then try updating and measuring the memory again. I reckon we could shave about 500MBKB or more off for non-APC hosts if we moved those out to a .update file.

I'm not, but the drush test...

HEAD:

drush site-install -y -d
[snip]
Timer 'page' is  26.6 sec. [27.35 sec, 21.96 MB]
Peak memory usage was 23.04 MB [27.35 sec, 21.96 MB] 

HEAD minus updates:

drush site-install -y -d
[snip]
Timer 'page' is  19.01 sec. [19.19 sec, 21.41 MB]
Peak memory usage was 22.49 MB [19.19 sec, 21.41 MB]

.5 MB is not all that impressive. :\ Shaving 7 seconds off run-time is nice, but that could just be a glitch.

Here's the patch I used. I just did it manually cos my brain needed a nice, brainless break from my other work, so thanks for that. :)

I think i just managed to install the bare essentials of drupal, using the minimal profile within 16MB
However, to be able to view the "Configure Site" page I needed 17 MB

I've Confirmed this by re-installing with 17MB (which brought me to the "Configure Site" page without a hitch), lowering the memory limit to 16MB, restarting the server, and refreshing (which resulted in an memory limit error).

Also, when I lowered the Memory 15MB, it installed all the tables, but wasn't able to populate the menu_router table.

Conclusion: the "Configure Site" page needs over 16MB to even get displayed. (and I think that means there is a bug somewhere.)
To be able to submit that page, and finish the installation, i needed 1MB of additional Memory (which can be expected.)

Note that this is using the minimal profile, so actual drupal performance is even worse.

(Environment: Ubuntu 9.10 off the shelf)

#1: htaccess.patch queued for re-testing.

I tried with webchick's patch and also got about 500kb less usage, peak memory for me was about 48mb though.

Killing 0.5 MB of memory consumption and speeding up by 7 seconds, by splitting updates away into $module.update.inc sounds like a very good improvement to me. Note that $module.install files are also loaded during regular site operation, whenever the cached schema is rebuilt. Hence, that change will improve performance of installing/enabling/disabling/uninstalling modules.

Splitting off update hooks makes a lot of sense, since those are even more rarely used than install hooks. It's also an easy-to-follow pattern.

Would we want to make that required, or optional? (Vis, allow modules with only one or two update hooks to leave them in the .install file.)

I think I got everything. Here is a patch to add module_load_update() and move all of the update functions, as well as the update_dependency functions to the new update.inc files. Personally I think it is a good idea to force this. After making this patch, I can honestly say it's not *that* bad.

Please review, it's my first run at this, so I'm sure I missed something.

+++ includes/module.inc
@@ -235,6 +235,16 @@ function module_load_install($module) {
 /**
+ * Load a module's update hooks.
+ */
+function module_load_update($module) {
+  // Make sure the installation API is available
+  include_once DRUPAL_ROOT . '/includes/install.inc';
+
+  module_load_include('update.inc', $module);
+}

1) Wrong arguments for http://api.drupal.org/api/function/module_load_include/7

2) I think we should additionally load $module.install files when loading $module.update.inc. It's likely that schema and installation functions are needed when needing updates.

90 critical left. Go review some!

Sun,

Thanks for the review. Here is an update to fix the argument issue, and load the .install file.

I have a bit of a bad feeling about http://api.drupal.org/api/function/drupal_get_schema_versions/7 which is called by drupal_install_modules(), which requires all .update to be loaded into memory. We might not be able to get an actual saving here, although it's possible there's a way around this.

I'd suggest to move the $module.update.inc split + discussion into a separate issue, so this issue can stay and serve as general/meta issue.

@mikejoconnor: Could you create that isue and link to it here?

Moved the .update.inc discussion to #788496: Break update functions into update.inc

I'm currently able to install the standard profile with 27mb of ram. With the minimal profile, I am able to install with 20mb of ram.

Currently, I believe this issue is resolved, unless I'm missing something.

Ok. I committed the attached patch, which moved this memory requirement back down to 32M. Let's see if we get any more bug reports about out of memory errors during installation, but it looks like just about everyone who's posted stats since we backed out that DBTNG query cache is well below that limit.

@webchick, did you ever commit that?

I came across this issue for another reason, but then read your comment that you put it back down to 32M and thought "hm, that's strange, because I'm pretty sure it's back at 40M now", and looking at CVS, it doesn't look to me like the patch was ever committed in the first place...

(I'm marking "needs review" not RTBC because I don't know if 32M is actually enough anymore.)

Oh, I did not mean for this to show up on the criticals list, oops.

alpha blocker tag no longer relevant since we're on beta now.

When I install Drupal via the command line, the script I use always prints out the maximum memory taken. With current HEAD, that comes out to around 26-27M using the Standard profile.

I just added some quick logging to the browser-based install too, and for the Standard install profile, no page request got above around 25-26M.

So, it seems safe to mark this RTBC. Other install profiles can (and will) exceed the 32M limit, but they could always add their own hook_requirements() if they need to.

Currently, in includes/bootstrap.inc:

define('DRUPAL_MINIMUM_PHP_MEMORY_LIMIT',    '40M');

Ok, let's try it again then. :P~

Committed to HEAD. For real this time. I think. :P

It's 32M now (current bootstrap.inc, at drupalcode.org).

I just installed 7.0-beta1 on Apple with MAMP (php 5.2.6 and 16M memory. Install fails on 'Install profile' step with message: "An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: http://localhost:8888/drupal-7/install.php?profile=standard&locale=en&id... StatusText: OK". Changed memory to 32 Mb and install is now succesfull.

Couldn't there be a memory check before install?

(I didn't read all the messages above. Don't know if I should start a new issue, because this concerns de beta)

@capono: There is a memory check done before install. However, I think the problem you're encountering is because when the check fails, it only triggers a warning (not an error), and there seems to be a bug in Drupal 7 where if the requirements check only results in warnings but not any errors, the warnings are never displayed.

This should be a separate issue. I created one here: #951644: Requirement warnings (e.g. for PHP memory limit) are not shown on install or update unless there is a requirement error also

@David_Rothstein: thank you.

Note that there are (some) reports coming in of 32M not being enough on particular hosting environments. See #1008362: 32M is sometimes not enough memory to install Drupal 7 for the followup issue.

I'm hearing reports that 128MB is not enough for Drupal 8. Now is a good time for folks in this issue to mobilize and start fixing up D8. You can post on #1744302: [meta] Resolve known performance regressions in Drupal 8 or start a new issue.