Make the InstallKernel progressively build the container instead of building, compiling and dumping from scratch on every module install (38x)

Awesome work! This could also speed up tests quite a lot, right?

Missing $settings['hash_salt'] in settings.php.                          [error]

Wow, this really not seems related.

About how many percent do we talk about here?

"installs locally"

With an existing settings.php (with existing db connection) or not? I recently noticed that can make a world of a difference, make sure you try it with a freshly copied settings.php.

I don't see any difference with this patch, using Ubuntu 14.10, PHP 5.6.4.

I'm installing to a tmpfs sqlite database.

Without:

Do you really want to drop all tables in the database tmp/db.sqlite? (y/n): y
Starting Drush preflight. [0 sec, 2.52 MB]                                                                                                       [preflight]
Cache HIT cid: 7.0-dev-commandfiles-0-b549b719331d46c7000915fdc6a1fe25 [0.01 sec, 2.59 MB]                                                           [debug]
Bootstrap to phase 0. [0.06 sec, 6.91 MB]                                                                                                        [bootstrap]
Bootstrap to phase 1. [0.06 sec, 6.92 MB]                                                                                                        [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.06 sec, 6.92 MB]                                                                       [bootstrap]
Initialized Drupal 8.0.0-dev root directory at /home/josh/web/develfix [0.06 sec, 7.37 MB]                                                          [notice]
Cache HIT cid: 7.0-dev-commandfiles-1-2e7968317fe3d2fd42c716898a426e3d [0.06 sec, 5.45 MB]                                                           [debug]
Found command: site-install (commandfile=site_install) [0.06 sec, 5.54 MB]                                                                       [bootstrap]
Calling hook drush_core_site_install_validate [0.11 sec, 7.61 MB]                                                                                    [debug]
Returned from hook drush_core_site_install_validate [0.11 sec, 7.62 MB]                                                                              [debug]
Calling hook drush_core_pre_site_install [0.11 sec, 7.62 MB]                                                                                         [debug]
Cache MISS cid: 7.0-dev-alias-path--0f9d5fdb5c59f362b3bbac21d82cc80e [0.11 sec, 7.73 MB]                                                             [debug]
You are about to create a /home/josh/web/develfix/sites/default/settings.php file and CREATE the 'tmp/db.sqlite' database. Do you want to continue? (y/n): y
Sites directory /home/josh/web/develfix/sites/default already exists - proceeding. [0.11 sec, 7.73 MB]                                              [notice]
Calling copy(sites/default/default.settings.php, /home/josh/web/develfix/sites/default/settings.php) [0.11 sec, 7.73 MB]                             [debug]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [0.12 sec, 8.58 MB]                                                                       [bootstrap]
Initialized Drupal site default at sites/default [0.12 sec, 8.58 MB]                                                                                [notice]
Cache HIT cid: 7.0-dev-install_profile-66ecfeb9791a023150773849f1550c5d [0.12 sec, 6.54 MB]                                                          [debug]
Cache HIT cid: 7.0-dev-commandfiles-2-479111bf70279c120e2c405fd2f78a76 [0.12 sec, 6.54 MB]                                                           [debug]
SQLITE: creating 'tmp' for creating 'tmp/db.sqlite' [0.12 sec, 6.67 MB]                                                                              [debug]
Returned from hook drush_core_pre_site_install [0.12 sec, 6.66 MB]                                                                                   [debug]
Calling hook drush_core_site_install [0.12 sec, 6.67 MB]                                                                                             [debug]
Starting Drupal installation. This takes a while. Consider using the --notify global option. [0.13 sec, 7.08 MB]                                 [ok]
Calling install_drupal(Object, Array) [0.13 sec, 7.08 MB]                                                                                            [debug]
Installation complete.  User name: josh  User password: josh [26.01 sec, 82.67 MB]                                                               [ok]
Returned from hook drush_core_site_install [26.01 sec, 82.66 MB]                                                                                     [debug]
Congratulations, you installed Drupal! [26.01 sec, 82.66 MB]                                                                                     [status]
Command dispatch complete [26.01 sec, 82.62 MB]                                                                                                     [notice]

real	0m26.120s
user	0m25.435s
sys	0m0.487s

With:

Do you really want to drop all tables in the database tmp/db.sqlite? (y/n): y
Starting Drush preflight. [0 sec, 2.52 MB]                                                                                                       [preflight]
Cache HIT cid: 7.0-dev-commandfiles-0-b549b719331d46c7000915fdc6a1fe25 [0.01 sec, 2.59 MB]                                                           [debug]
Bootstrap to phase 0. [0.05 sec, 6.91 MB]                                                                                                        [bootstrap]
Bootstrap to phase 1. [0.05 sec, 6.92 MB]                                                                                                        [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.05 sec, 6.92 MB]                                                                       [bootstrap]
Initialized Drupal 8.0.0-dev root directory at /home/josh/web/develfix [0.06 sec, 7.37 MB]                                                          [notice]
Cache HIT cid: 7.0-dev-commandfiles-1-2e7968317fe3d2fd42c716898a426e3d [0.06 sec, 5.45 MB]                                                           [debug]
Found command: site-install (commandfile=site_install) [0.06 sec, 5.54 MB]                                                                       [bootstrap]
Calling hook drush_core_site_install_validate [0.11 sec, 7.61 MB]                                                                                    [debug]
Returned from hook drush_core_site_install_validate [0.11 sec, 7.62 MB]                                                                              [debug]
Calling hook drush_core_pre_site_install [0.11 sec, 7.62 MB]                                                                                         [debug]
Cache MISS cid: 7.0-dev-alias-path--0f9d5fdb5c59f362b3bbac21d82cc80e [0.11 sec, 7.73 MB]                                                             [debug]
You are about to create a /home/josh/web/develfix/sites/default/settings.php file and CREATE the 'tmp/db.sqlite' database. Do you want to continue? (y/n): y
Sites directory /home/josh/web/develfix/sites/default already exists - proceeding. [0.11 sec, 7.73 MB]                                              [notice]
Calling copy(sites/default/default.settings.php, /home/josh/web/develfix/sites/default/settings.php) [0.11 sec, 7.73 MB]                             [debug]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [0.12 sec, 8.58 MB]                                                                       [bootstrap]
Initialized Drupal site default at sites/default [0.12 sec, 8.58 MB]                                                                                [notice]
Cache HIT cid: 7.0-dev-install_profile-66ecfeb9791a023150773849f1550c5d [0.12 sec, 6.54 MB]                                                          [debug]
Cache HIT cid: 7.0-dev-commandfiles-2-479111bf70279c120e2c405fd2f78a76 [0.12 sec, 6.54 MB]                                                           [debug]
SQLITE: creating 'tmp' for creating 'tmp/db.sqlite' [0.12 sec, 6.67 MB]                                                                              [debug]
Returned from hook drush_core_pre_site_install [0.12 sec, 6.67 MB]                                                                                   [debug]
Calling hook drush_core_site_install [0.12 sec, 6.68 MB]                                                                                             [debug]
Starting Drupal installation. This takes a while. Consider using the --notify global option. [0.12 sec, 7.08 MB]                                 [ok]
Calling install_drupal(Object, Array) [0.12 sec, 7.09 MB]                                                                                            [debug]
Installation complete.  User name: josh  User password: josh [26.29 sec, 82.72 MB]                                                               [ok]
Returned from hook drush_core_site_install [26.29 sec, 82.72 MB]                                                                                     [debug]
Congratulations, you installed Drupal! [26.29 sec, 82.72 MB]                                                                                     [status]
Command dispatch complete [26.29 sec, 82.68 MB]                                                                                                     [notice]

real	0m26.402s
user	0m25.694s
sys	0m0.511s

MySQL with:

Do you really want to drop all tables in the database d8? (y/n): y
Starting Drush preflight. [0 sec, 2.52 MB]                                                                                                       [preflight]
Cache HIT cid: 7.0-dev-commandfiles-0-b549b719331d46c7000915fdc6a1fe25 [0.01 sec, 2.59 MB]                                                           [debug]
Bootstrap to phase 0. [0.05 sec, 6.91 MB]                                                                                                        [bootstrap]
Bootstrap to phase 1. [0.05 sec, 6.92 MB]                                                                                                        [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.05 sec, 6.92 MB]                                                                       [bootstrap]
Initialized Drupal 8.0.0-dev root directory at /home/josh/web/develfix [0.06 sec, 7.37 MB]                                                          [notice]
Cache HIT cid: 7.0-dev-commandfiles-1-2e7968317fe3d2fd42c716898a426e3d [0.06 sec, 5.45 MB]                                                           [debug]
Found command: site-install (commandfile=site_install) [0.06 sec, 5.54 MB]                                                                       [bootstrap]
Calling hook drush_core_site_install_validate [0.11 sec, 7.61 MB]                                                                                    [debug]
Returned from hook drush_core_site_install_validate [0.11 sec, 7.62 MB]                                                                              [debug]
Calling hook drush_core_pre_site_install [0.11 sec, 7.62 MB]                                                                                         [debug]
Cache MISS cid: 7.0-dev-alias-path--0f9d5fdb5c59f362b3bbac21d82cc80e [0.11 sec, 7.75 MB]                                                             [debug]
sql-query: SELECT 1; [0.11 sec, 7.75 MB]                                                                                                         [status]
Executing: mysql --defaults-extra-file=/tmp/drush_pGT3e9 --database=d8 --host=localhost --silent  < /tmp/drush_VEVluR > /dev/null
You are about to create a /home/josh/web/develfix/sites/default/settings.php file and DROP all tables in your 'd8' database. Do you want to continue? (y/n): y
Sites directory /home/josh/web/develfix/sites/default already exists - proceeding. [0.12 sec, 7.75 MB]                                              [notice]
Calling copy(sites/default/default.settings.php, /home/josh/web/develfix/sites/default/settings.php) [0.12 sec, 7.75 MB]                             [debug]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [0.12 sec, 8.6 MB]                                                                        [bootstrap]
Initialized Drupal site default at sites/default [0.12 sec, 8.61 MB]                                                                                [notice]
Cache HIT cid: 7.0-dev-install_profile-66ecfeb9791a023150773849f1550c5d [0.13 sec, 6.56 MB]                                                          [debug]
Cache HIT cid: 7.0-dev-commandfiles-2-479111bf70279c120e2c405fd2f78a76 [0.13 sec, 6.57 MB]                                                           [debug]
sql-query: SELECT 1; [0.13 sec, 6.7 MB]                                                                                                          [status]
Executing: mysql --defaults-extra-file=/tmp/drush_hJCHQI --database=d8 --host=localhost --silent  < /tmp/drush_e18R2q > /dev/null
sql-query: SHOW TABLES; [0.13 sec, 6.7 MB]                                                                                                       [status]
Executing: mysql --defaults-extra-file=/tmp/drush_Rp89ti --database=d8 --host=localhost --silent  < /tmp/drush_9elqF0
Returned from hook drush_core_pre_site_install [0.13 sec, 6.69 MB]                                                                                   [debug]
Calling hook drush_core_site_install [0.14 sec, 6.7 MB]                                                                                              [debug]
Starting Drupal installation. This takes a while. Consider using the --notify global option. [0.14 sec, 7.11 MB]                                 [ok]
Calling install_drupal(Object, Array) [0.14 sec, 7.11 MB]                                                                                            [debug]
Installation complete.  User name: josh  User password: josh [31.5 sec, 82.66 MB]                                                                [ok]
Returned from hook drush_core_site_install [31.5 sec, 82.65 MB]                                                                                      [debug]
Congratulations, you installed Drupal! [31.5 sec, 82.65 MB]                                                                                      [status]
Command dispatch complete [31.5 sec, 82.61 MB]                                                                                                      [notice]

real	0m31.605s
user	0m25.757s
sys	0m0.349s

MySQL without:

Do you really want to drop all tables in the database d8? (y/n): y
Starting Drush preflight. [0 sec, 2.52 MB]                                                                                                       [preflight]
Cache HIT cid: 7.0-dev-commandfiles-0-b549b719331d46c7000915fdc6a1fe25 [0.01 sec, 2.59 MB]                                                           [debug]
Bootstrap to phase 0. [0.05 sec, 6.91 MB]                                                                                                        [bootstrap]
Bootstrap to phase 1. [0.05 sec, 6.92 MB]                                                                                                        [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.05 sec, 6.92 MB]                                                                       [bootstrap]
Initialized Drupal 8.0.0-dev root directory at /home/josh/web/develfix [0.06 sec, 7.37 MB]                                                          [notice]
Cache HIT cid: 7.0-dev-commandfiles-1-2e7968317fe3d2fd42c716898a426e3d [0.06 sec, 5.45 MB]                                                           [debug]
Found command: site-install (commandfile=site_install) [0.06 sec, 5.54 MB]                                                                       [bootstrap]
Calling hook drush_core_site_install_validate [0.11 sec, 7.61 MB]                                                                                    [debug]
Returned from hook drush_core_site_install_validate [0.11 sec, 7.62 MB]                                                                              [debug]
Calling hook drush_core_pre_site_install [0.11 sec, 7.62 MB]                                                                                         [debug]
Cache MISS cid: 7.0-dev-alias-path--0f9d5fdb5c59f362b3bbac21d82cc80e [0.11 sec, 7.75 MB]                                                             [debug]
sql-query: SELECT 1; [0.11 sec, 7.75 MB]                                                                                                         [status]
Executing: mysql --defaults-extra-file=/tmp/drush_MdMZxV --database=d8 --host=localhost --silent  < /tmp/drush_nYkmmG > /dev/null
You are about to create a /home/josh/web/develfix/sites/default/settings.php file and DROP all tables in your 'd8' database. Do you want to continue? (y/n): y
Sites directory /home/josh/web/develfix/sites/default already exists - proceeding. [0.12 sec, 7.75 MB]                                              [notice]
Calling copy(sites/default/default.settings.php, /home/josh/web/develfix/sites/default/settings.php) [0.12 sec, 7.75 MB]                             [debug]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [0.12 sec, 8.6 MB]                                                                        [bootstrap]
Initialized Drupal site default at sites/default [0.12 sec, 8.6 MB]                                                                                 [notice]
Cache HIT cid: 7.0-dev-install_profile-66ecfeb9791a023150773849f1550c5d [0.12 sec, 6.56 MB]                                                          [debug]
Cache HIT cid: 7.0-dev-commandfiles-2-479111bf70279c120e2c405fd2f78a76 [0.12 sec, 6.56 MB]                                                           [debug]
sql-query: SELECT 1; [0.12 sec, 6.69 MB]                                                                                                         [status]
Executing: mysql --defaults-extra-file=/tmp/drush_NxXu1p --database=d8 --host=localhost --silent  < /tmp/drush_JcKJMa > /dev/null
sql-query: SHOW TABLES; [0.13 sec, 6.69 MB]                                                                                                      [status]
Executing: mysql --defaults-extra-file=/tmp/drush_cqqZwU --database=d8 --host=localhost --silent  < /tmp/drush_2KCehF
Returned from hook drush_core_pre_site_install [0.13 sec, 6.69 MB]                                                                                   [debug]
Calling hook drush_core_site_install [0.13 sec, 6.7 MB]                                                                                              [debug]
Starting Drupal installation. This takes a while. Consider using the --notify global option. [0.14 sec, 7.1 MB]                                  [ok]
Calling install_drupal(Object, Array) [0.14 sec, 7.11 MB]                                                                                            [debug]
Installation complete.  User name: josh  User password: josh [31.76 sec, 82.6 MB]                                                                [ok]
Returned from hook drush_core_site_install [31.76 sec, 82.6 MB]                                                                                      [debug]
Congratulations, you installed Drupal! [31.76 sec, 82.6 MB]                                                                                      [status]
Command dispatch complete [31.76 sec, 82.56 MB]                                                                                                     [notice]

real	0m31.867s
user	0m25.769s
sys	0m0.295s

I ran dr si -d -y three times with the patch, and 3 times without. The patch made a minor difference for me (1 second). I'm determining elapsed time from the final log message reported by drush site-install. Maybe blackfire/xdebug is perturbing the measurements for larowlan?

With patch: 33,32,31 (seconds)
Without patch: 33,33,34 (seconds)

I did a couple more timings just for fun. These are without the patch

XHProf enabled: 40 seconds
XDEBUG enabled: 78 seconds (no client listening on xdebug port)

Similarly small.

1. +++ b/core/lib/Drupal/Core/DrupalKernel.php
   @@ -524,20 +524,8 @@ public function discoverServiceProviders() {
        $this->classLoaderAddMultiplePsr4($this->getModuleNamespacesPsr4($module_filenames));
   +    $class = $this->discoverModuleServiceProviders($module_filenames);
    
   

   This is wrong, because $class is never used.

2. +++ b/core/lib/Drupal/Core/DrupalKernel.php
   @@ -1070,23 +1058,7 @@ protected function compileContainer() {
        // Register application services.
        $yaml_loader = new YamlFileLoader($container);
   -    foreach ($this->serviceYamls['app'] as $filename) {
   

   The yaml_loader is only needed in the helper, so no need to have this here.

   --

   Oh, I get it, the InstallerKernel is re-using the loader, nvm. then, but should add a comment.

3. +++ b/core/lib/Drupal/Core/DrupalKernel.php
   @@ -1313,4 +1292,64 @@ public static function validateHostname(Request $request) {
   +  protected function discoverModuleServiceProviders($module_filenames) {
   ...
   +  protected function doRegisterServiceProviders($yaml_loader, $container) {
   

   +1 in general to making those helper functions.

4. +++ b/core/lib/Drupal/Core/Installer/InstallerKernel.php
   @@ -40,4 +82,123 @@ public function resetConfigStorage() {
   +    if ($this->dumpOnTerminate && $this->allowDumping) {
   +      if ($this->container instanceof ContainerBuilder) {
   +        $this->container->compile();
   +      }
   +      $this->dumpDrupalContainer($this->container, static::CONTAINER_BASE_CLASS, TRUE);
   

   Isn't this over-optimization?

   Maybe not, especially for tests.

   Else the container would be compiled for the next request.

   I personally would say its fine to never dump the container during installation.

5. +++ b/core/lib/Drupal/Core/Installer/InstallerKernel.php
   @@ -40,4 +82,123 @@ public function resetConfigStorage() {
   +    if ($this->systemInstalled) {
   +      // Once system module is installed and InstallerServiceProvider has been
   +      // deactivated, we can progressively build the container.
   +      $this->containerBuilder = $container;
   +      return;
   +    }
   

   It feels wrong to set this both on getContainerBuilder() and doCompile().

   This _should_ already be the same.

This should save time. I cannot see why it does not, but maybe it is something else.

While the idea is good, I think we will need to won't fix this:

- ContainerBuilder is before ->compile() not a real container and services will be missing from it, hence hook_install() and hook_modules_installed() would be a lie.

While that might work for core its a problem for contrib/ install profiles e.g..

Not helping core that much anyway as its only meaningful for drush as else the container needs to be build anyway via batching new requests and we just skip the ->compile() step.

In comparison to that turning off all caches and cache tags for installer duration gives a 50% performance increase.

(80s vs. 164s for a drush site-install)

So 8 seconds is not that much overall.