Problem/Motivation

This doesn't directly fall under the purview of any particular official initiative, but I'm tagging it for them anyway because it's relevant.

We all know the installer is the red headed stepchild of Drupal, even among red headed stepchildren. There are many reasons for that. Mostly it boils down to:

1) Most of Drupal expects to be running in an installed state, and so we have to hack around that for a special case.
2) Core only supports database-based sessions, so we cannot use a session to store data. That means all sorts of grotesque contortions to pass data along a multi-step form without using any sort of data storage mechanism.
3) It was implemented before any good client-side storage mechanisms existed.
4) Let's be honest, given all of the above, no one has really tried to make the installer a *good* system, just a *working* system. That was hard enough.

Proposed resolution

However!

1) We now have this great new kernel, router, and http library in core courtesy of Symfony.

2) That includes a new session implementation that includes a PHP-native backend, which we can use at all times, even in an uninstalled state. (See #335411: Switch to Symfony2-based session handling for where we're trying to use the new session code for core, too.)

3) That includes the ability to define "routes" (paths and controllers) without using the database.

4) All of our supported browsers for Drupal 8 support real client-side functionality, and arguably we can rely on Javascript, you know, existing.

That means... we can rewrite the installer to be its own Symfony-based micro-application. And in the process, greatly simplify and modernize the code.

The basic idea is this:

1) Implement a new kernel class specific to the installer, using all of the default Symfony components. No custom implementations necessary the way core will need.
2) Give the installer kernel a hard-coded set of routes. There's probably like a half-dozen here.
3) Implement each "step" in the installer as a separate controller at its own path, rather than chaining GET parameters out for a few miles. As a bonus, that makes it possible to actually test each step individually(!)
4) Use a PHP-native session or client-side sessionStorage or whatever to store data rather than throwing crap into the GET string.
5) Clean up a crapload of stuff along the way.
6) Profit!!1!

Note that we will almost certainly run into places where things are still ugly and cannot be made non-ugly just be working around them. That's fine. Anywhere that we find the installer cannot be done without ugliness is a clear sign that we need to refactor something. In a sense, we're using the kernel as a canary in the coal mine to help flush out the sort of refactoring and pulling-apart-of-things-that-are-wrongly-coupled we're going to have to do anyway.

And getting a much better and more maintainable installer out of it is good, too.

Remaining tasks

I have a preliminary stab at this in the kernel-installer branch of the WSCCI sandbox. It's barely started, but it starts to lay out the basic idea. Someone not named Crell needs to pick this up and run with it. :-) Who wants to be the hero of Drupal and make the installer not suck? (Contact me for access to the sandbox if you want.)

Once the installer is done, the same thing can and should be done to both cron.php and update.php. (Actually cron.php may not be necessary anymore, but definitely update.php should also become its own little micro-app.)

User interface changes

The installer's pseudo-paths should turn into real paths. Otherwise there should be no significant UI changes.

API changes

This *should* be possible to do without changing the API for install profiles, but that will be determined by implementation. (Although we may want to adjust the install profile API as well in light of the new installer in a follow-up patch.)

Who wants it???

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Letharion’s picture

I volunteered on IRC during the last WSCCI meeting to carry this torch. I've been a bit slow to pick things up since, but what I have done is gone over Fabiens "Create your own framework" symfony intro. I feel like having a better grasp on the symfony components would be good before I start hacking at the installer. :)

If anyone else is interested in helping out, I would of course love to co-operate on the topic.

BTMash’s picture

I need to get up to speed on everything symfony related (so I may not be able to help you out immediately, @Letharion) but I'd be interested in helping out on this as well.

Crell’s picture

Great, guys!

I recommend looking at the kernel-installer branch. It may help to piece things together for you. You can also ask in #drupal-wscci, and the folks in #symfony-dev have been very helpful as well.

cosmicdreams’s picture

So should we work within the wscci git repo or use patches. I'm guessing you're going to say create a new branch on git repo.

cweagans’s picture

@cosmicdreams, see remaining tasks section in issue summary.

sun’s picture

Custom installation profile steps are entirely missing in the steps/roadmap.

Installation profiles are able to inject 1-N entirely custom pages at certain/dedicated positions in the installer. Those not only need to be routed.

The outlined problem description is not entirely valid, too. There have been many attempts to improve and (entirely) change the architecture of Drupal's installer in the past, and merely searching the Drupal core queue for 5 minutes would have revealed a dozen of issues proposing huge architectural changes.

Before we move on with this direction, we need to make sure that the existing proposals aren't way superior to this proposal.

-enzo-’s picture

Crell

I just install the the kernel-installer branch and I'm getting this error

http://drupalbin.com/21174

LogicException: The controller must return a response (<form class="install-select-language-form" action="/core/install.php/language" method="post" id="install-select-language-form" accept-charset="UTF-8"><div><div class="form-item form-type-select form-item-langcode"> <select id="edit-langcode" name="langcode" size="1" class="form-select"><option value="en" selected="selected">English</option></select> </div> <p><a href="/core/install.php?translate=true">Learn how to install Drupal in other languages</a></p><input type="hidden" name="form_build_id" value="form-LGpFahO7ys7hhueriFfh9vscgTwH6srDhvpxcAa2Lfg" /> <input type="hidden" name="form_id" value="install_select_language_form" /> <div class="form-actions form-wrapper" id="edit-actions"><input type="submit" id="edit-submit" name="op" value="Save and continue" class="form-submit" /></div></div></form> given). in Symfony\Component\HttpKernel\HttpKernel->handleRaw() (line 146 of /Applications/MAMP/wscci/core/vendor/Symfony/Component/HttpKernel/HttpKernel.php)

I found this possible solution http://stackoverflow.com/questions/9063103/the-controller-must-return-a-...

I'm wondering if I need to create a ticket in what queue, maybe this: http://drupal.org/project/issues/1260830?categories=All

Regards,

Crell’s picture

enzo: Yes, the current code is not yet actually working. :-) The controller needs to return a response object, OR there needs to be a kernel.view listener that can turn whatever the controller returns into a response object. In the kernel branch we have such a listener that is essentially a port of drupal_deliver_html_page(). We could borrow that, or just update the controller to return a response object itself. I nominally prefer the latter.

cosmicdreams: Definitely this should be worked on in git, with periodic status patches posted here.

sun: I know that there have been other installer efforts in the past, and the D7 installer is also a considerable improvement over some previous iterations of that code. However, with the new ability to autoload classes without a database, actually use the session, and leverage HTML5 on the client-side, plus the potential to use the same workflow as core (because the Symfony components have no intrinsic database dependency), I would be rather shocked if any older proposals were able to be as clean as this.

The code right now is very preliminary, and I wasn't planning as far ahead as every detail of the current implementation, so custom pages are not yet something I thought about. Really, the kernel-installer branch right now is 2 hours of me noodling around, nothing more. :-)

One thought that came up in IRC earlier today was we could allow the entire RouteCollection to be provided by the install profile. So a given install profile could, if it wanted to, completely rearrange the entire workflow, provide its own defaults, add pages, remove pages, etc. And if we make the install profile's core a class, then we can inherit profiles from each other almost trivially (a la #555212: Install profile inheritance).

There's still plenty to figure out here, but I wanted to get people started on it because it isn't dependent on anything else (at least not seriously).

David_Rothstein’s picture

Thanks for this nice writeup.

It sounds to me like this is actually two separate proposals, and only the second depends on the first? The first is to make the installer use a session-based storage mechanism throughout (one that is available even before Drupal is installed), and the second is to put each step of the installer at its own URL.

For the first, I think you've hit the nail on the head that the lack of a single storage mechanism which persists throughout the installer is one of the biggest issues that makes the code complicated. However, I'm skeptical that client-side storage can be relied on here; we still need Drupal to be functional on older clients or with JavaScript turned off (even if it isn't pretty it should still work). File-based sessions are more promising, but we need to be careful to make sure Drupal is still installable on hosting setups that have multiple web servers. I don't think that's insurmountable (since if you're running a Drupal site on multiple web servers you're going to need something resembling a shared filesystem anyway) but it's something to keep in mind.

For the second part of the proposal, this would actually involve putting more information in the URL than the installer currently does (since only the first couple of steps of the installer currently have their own unique URLs). To the extent that the installer really is like a giant multi-step form, I'd actually think that keeping things at a smaller number of URLs (even a single URL) would be the better direction, because that's how Drupal normally deals with multi-step forms. Also, since there isn't a single defined "route" that all users take through the installer, I'd get the impression there would have to be a lot of logic to redirect people to different URLs depending on different conditions. But I might be wrong, and I guess it's hard to know without seeing it in action.

In terms of previous issues, #1226380: Generate minimal mocked or SQLite environment and use to install Drupal in full environment is probably relevant. One of the proposals there essentially involves having the installer use the database as its single storage mechanism (by pre-installing a temporary SQLite database at the beginning). The advantage of something like that compared to the proposal here is that essentially the entire Drupal API would be available right after the installer starts. The disadvantage is that SQLite would have to be a requirement for installing Drupal.

boombatower’s picture

The original hope was the Drupal would be able to run without a db by the time 8 was ready to go so the SQLite step would just be a temporary thing. I had started to rewrite the installer as a module and was using essentially the same approach described here with each page at a different url, etc.

This seems like the easier/possibly best way to get to the stage were we don't need a db, but having to do so outside of Drupal's API seems like it might almost move us back to where we are now with the installer as a separate being outside of Drupal, but if it's a cleaner one maybe that is ok.

catch’s picture

The fundamental issue with the installer IMO is this:

a. installer relies on APIs (variables, session) that hit the database by default.
b. APIs in turn depend on system_schema() to provide the schema.
c. installing system_schema() depends on the module system, which in turn depends on a.

I tried some brute force decoupling in #1153716: Decouple system module from the installer - the patch was a large, painful, rabbit hole but the issue documents a lot of the pain.

So, having session storage that defaults to PHP or file-based sessions is a good start to this, but to actually have a clean installer, I'd see the following need to happen:

- Anything that is defined in system_schema() or user_schema() that the module system itself depends on (including the {system} table itself, would need to be moved to pluggable storage backends, defaulting to NULL or file-based storage or having the storage implementation controlled by the installer, or lazy creation via #1167144: Make cache backends responsible for their own storage, or anything else which breaks the coupling. Note we have some not-quite-null implementations like the cache install backend which exists only for the cases where AJAX request populate cache tables, it might be possible now to move that to the proper NULL backend but the original bug was very, very hard to track down so not sure if it's still there or not.

Then, you'd effectively be able to 'install' Drupal by having the file system and db connection set up, and if you wanted to just do drush install {currently_required_modules} to get started, bypassing the installer as such altogether, you could.

Then the installer just does the following:

- requirements check (doesn't even need to be in the installer really, but it's currently there and update.php for convenience)
- language selection
- database connection info interface.
- install profile selection and any additional configuration wizard stuff.

But it'd be technically possible to have a working 'Drupal' install without running through the installer, even if 99.99% of sites start with install.php or drush si anyway.

Crell’s picture

David: Yeah, I'm not sure to what extent sessionStorage would be useful either; I mention it for completeness. Frankly I think there's more that we could do with batch API to make it use, say, the progress element, but that's not strictly installer related. The underlying point is that we simply have more tools at our disposal now than we've had before, which means we can do things we couldn't before to make the code better.

Also, I don't know how anyone installs Drupal directly onto a multi-head installation now, frankly, since I've never done so. I figure you folks at Acquia probably have more experience with that than anyone else right now, so any insights you have, please share. :-) We may just want to assume host fixation for the installer, or maybe even just punt on that for now.

As far as how much information is put in the URL, at that point it just becomes a question of what part of the URL, I guess. What I was figuring is that each form is its own stand-alone controller, and saves its data to the session before forwarding to "whatever the next route is". Then you could reorder, add, and remove forms however you want, and when enough forms have run then you pop to the "OK, now install all this crap" controller that reads data out of the session to decide what to do. At least that's my preliminary thinking so far, which as I said is not fully developed at all. :-)

boombatower: Well, one of the things I'd like to see this patch do is help identify places where our code depends on having an installed database where it arguably doesn't, and let us refactor that. While the kernel implementation will likely be different, and we'd have a different RouteCollection, my hope is that in the end we'd still be using most of the same code as an installed core. And where we don't, it's handled cleanly.

catch: Definitely, there's more here than just the routing. I've been toying with the idea of dropping hook_schema in favor of class-based table definitions, which would help a great deal, but I don't know when/if I'll have time/bandwidth to do so.

catch’s picture

However, I'm skeptical that client-side storage can be relied on here; we still need Drupal to be functional on older clients or with JavaScript turned off (even if it isn't pretty it should still work).

I don't think we need to support older clients or people who've disabled JavaScript in the web installer at all fwiw.

If you're in the position where you're installing Drupal, then you're also in the position where you can do one of the following:

- use a proper browser
- enable JavaScript
- use Drupal 7 for another year
- install via drush
- use a hosted solution or some other service with an alternate installer

If there's a documented group of people who somehow escape all of the above categories then we should consider them, but otherwise I'm fine with the installer making assumptions that we wouldn't make elsewhere in core if it simplifies things.

bleen’s picture

@catch IIRC the issue with installing Drupal with JS turned off was an accessibility issue because of the varied levels of js support in screen reading browsers. I havent actually looked in a long time, so this might be moot but I believe that is (was?) the underlying concern.

catch’s picture

I think that's changed a lot the past few years, for example http://www.brucelawson.co.uk/2011/javascript-and-screenreaders/ cites a survey that found over 98% of screenreader users have JavaScript enabled.

David_Rothstein’s picture

Also, I don't know how anyone installs Drupal directly onto a multi-head installation now, frankly, since I've never done so. I figure you folks at Acquia probably have more experience with that than anyone else right now, so any insights you have, please share. :-)

As of last week, I've actually moved from Acquia to Advomatic - but Advomatic does some multi-web-node hosting too so apparently this is an issue everywhere I work :) It's actually been a long time since I've tried installing Drupal (via the UI) in this kind of environment myself, but as I recall it works fine as long as you pre-populate settings.php with database credentials (and soon, with a config directory also, #1464944: Installation of configuration system fails in some cases) so that the installer doesn't have to try to write settings.php itself.

Since we can probably assume the public files directory (and/or the config directory within it) is in a location shared between webnodes, I think if a file-based session were stored in that directory too it would work fine. If it were stored in the default PHP location of /tmp instead, I know that could cause things to explode since that directory is not necessarily shared. I guess the only question is if the public files directory is easy to detect (and use) right at the very beginning of the installer? For the most part it should be, unless the directory isn't yet writable by the web server.

Note we have some not-quite-null implementations like the cache install backend which exists only for the cases where AJAX request populate cache tables, it might be possible now to move that to the proper NULL backend but the original bug was very, very hard to track down so not sure if it's still there or not.

The bug has actually reappeared in Drupal 8, both its symptoms (#1536262: Entering a site name when installing Drupal 8 has no effect until caches are cleared) and its root cause (#1394648: The installer's cache backend no longer overrides all cache-clearing methods, which can lead to fatal errors). Although strangely the two don't seem related this time, so who knows, maybe it actually can be moved to the NULL backend.

--

Regarding JavaScript, most people will have it enabled, but a few won't (and text-based browsers run from a command line or a script won't either, that kind of thing). I would think the installer should follow whatever standard we follow for the admin UI (not necessarily for end users) in this regard, if at all possible?

cosmicdreams’s picture

Tried to get start with helping this tonight, but it seems that I have quite a bit of reading to do before I should write any code.

Right now I see how the current routes are created and where the database comes into play. One thing that could help my understanding is to accomplish something small.

@Crell : It seems like the best place to start is to get a second page rendering properly after language is selected. Do you agree? Or is there something simpler to address?

Crell’s picture

That seems like a good next step, yes.

Paul Simard’s picture

A couple of thoughts arose as I reviewed everything that had come before:

PHP is upgrading releases 5.4.x -- Among other things, there is session support for the progress component directly supported.

Client side storage pre-DB -- As a quick work-around a small dedicated DB using mongoDB for installation/update stuff only might be a consideration.

If not, and I'm just generating noise here, please disregard.

Paul

Letharion’s picture

@#19, while I would like to require 5.4, I believe that's out of question for D8.
sqlite has been considered for a "pre-db" db, but if symfony can allow us to remain db-less, then avoiding a new requirement is nice.

Letharion’s picture

As suggested in 17/18, I have gotten the second page to render, something at least. Code in branch kernel-installer-letharion

One that I can't figure out is how the regular installer supresses form redirection. In the Kernel when the language form is submitted, everything fails because Drupal tries to lookup a path alias.
I just hacked drupal_redirect_form for now.

Second problem, which is probably very simple and me being slow, is that the language code that I write to the session doesn't carry over to the next form.

sun’s picture

I already suggested in #1357912-19: Convert path language code schema to langcode that drupal_lookup_path() should not attempt to deal with URL aliases in the installer and update script (i.e., when MAINTENANCE_MODE is defined).

I'd love to see a separate issue to fix that. The change makes sense on its own, regardless of whether we'll have a kernel-based installer or not.

Everett Zufelt’s picture

I have no problem, from an accessibility perspective, with having Javascript required for the installer, provided that the Javascript used is accessible.

Letharion’s picture

Thank you Sun. I opened #1541128: Do not look up any URL aliases in MAINTENANCE_MODE to get that maintenance mode fix in.

Gaelan’s picture

How about this: ask for MySQL root login info and create the database and user automatically!

marcingy’s picture

@Gaelan that comment is off topic for this issue

sun’s picture

One more note, important: Whatever we come up with here, it needs to be testable.

I'm currently working hard to allow us to test the Drupal installer automatically, and also use the regular Drupal installer to set up the so called child site for running tests. The latter requires to invoke the Drupal installer from within an already installed and working Drupal site, as part of a regular page request.

Thus, I hope that we can resolve most things by leveraging the new dependency injection and class loader. Which in turn would mean that the testing framework would merely have to fake a temporary DI container to be able to invoke the non-interactive installer. Hopefully, that is.

Damien Tournoud’s picture

Title: Implement modern Kernel-based installer » Implement the installer as a proper kernel

Let's focus this issue on re-implementing the installer as a proper kernel. This should be a super easy task, as there is only basically one route in the installer.

This is only a code refactoring task. None of the other things discussed in this issue are going to be solved with *just* this. The installer UI is trivial. The complex part of the installer is under the hood.

Ideally, we would manage to:

  • Have a trivial mini-installer, (ie. just language selection, profile selection, database configuration, installing the system module), in which almost nothing of Drupal is actually booted up (we basically need the form system, a mock of the language system, a mock of the session system)
  • Do the rest of the installation process in a fully-bootstrapped Drupal

In a nutshell, I would like the first part of the installation process to look less like Drupal that it looks today and the second part of it to be a full-featured Drupal.

rocketeerbkw’s picture

I'd also like to help on this. I've been working through the Symfony docs this week and I'll begin familiarizing myself with how the current installer works over the weekend. I've helped with contrib modules and am familiar with patches and git but I've never done core work so I will be asking for directions.

If @Letharion is still spearheading the issue, maybe we can get on IRC and quickly get up to speed on what's next.

Crell’s picture

Letharion is probably the best contact person right now. I'll be back in the US next week so hopefully can be around to help if needed. Welcome aboard! :-)

Letharion’s picture

@rocketeerbkw I'm always online on IRC, though obviously not always available. I'm still working on the installer, and would love to have a second set of eyes on it. I recently lost quite a bit of work to a broken harddrive which has set me back a bit.

moshe weitzman’s picture

Any new thoughts or code here?

Letharion’s picture

I didn't make any progress for quite some time here, but had a good talk with Larry during DrupalCon, as well as recruited two colleagues as contributer to this.

As quite a lot has changed in core since the original work was started, we started up a new sandbox and will try to reboot this issue there.

After the talk with Larry, I believe one difference between the current "Proposed resolution" and what we agreed upon, is that it's a goal to reduce the number of (hardcoded) steps in the installer to the minimum, so that we can hand over to the profile ASAP, as suggested by Damien in #28.

Letharion’s picture

After some minor updates today, I've pushed the latest code I have to my sandbox.

There's probably a number of changes from the beginning of may up until now that I need to ensure are merged into the controllers. Anyone else who's interested in helping out could start by taking a look at this changes, and help pushing them into the controllers.

kika’s picture

Can not see the latest push...

Letharion’s picture

I must be really good at abusing git. I had a merge-conflict that I couldn't work out. Pushed the code to a new branch in the mean time.

cosmicdreams’s picture

What is the name of the branch we should be working on?

Letharion’s picture

I exercised a great deal of imagination and named it kernel-installer_.

cosmicdreams’s picture

Thanks for the clarification. I thought it was the reboot branch.

Anonymous’s picture

can we get a patch against HEAD in here? thanks.

cordoval’s picture

FileSize
29.59 KB

here it is @beejeebus :)

Crell’s picture

Status: Active » Needs review

What's this a patch of? :-)

Status: Needs review » Needs work

The last submitted patch, 1530756-41.patch, failed testing.

cordoval’s picture

forcefully trying to help :'( going to bed now

cordoval’s picture

I am doing commits here just in case i will come back and push to the sandbox once i am done

https://github.com/cordoval/drupal-installer/blob/feature/cleanup/

patcon’s picture

Also, for anyone looking to roll a patch at any point, github has it covered :)
https://github.com/cordoval/drupal-installer/compare/master...feature/cl...

cordoval’s picture

I haver run into a blockade

https://github.com/cordoval/drupal-installer/blob/97025aa1e3bf4efe4038b6...

I ran into problems for finding drupal_html_ something functions and stuff

is there a more standalone i can use the form component in drupal? I am very unfamiliar with that component and doubt we want to load all of that since it would depend on configuration and stuff.

@Crell any ideas?

Thanks,

cordoval

Crell’s picture

This is where Letharion ran into trouble, too. The Form API is not really component-ized. I suggest simply loading the code file (form.inc) manually in the InstallKernel boot process. That's fugly we won't be able to remove for a while yet, so let's just keep the fugly parts localized as much as possible.

fubhy’s picture

Just as an update: Letharion, cordoval and me have teamed up to move forward with this. Work is happening in @letharion's sandbox. We are making good progress and are already discussing further steps.

Letharion’s picture

Update: I worked on the installer during the weekend.
I was hoping I'd get the installer working, so I could post a patch, but I have some issues, I believe in the hand-over from the installer-specific DIC to the "real" DIC, so currently the install fails when trying to run the profiles .install file.

I'll keep at it, and will now instead hope to have a patch ready during the coming week.

A small tasks for anyone who wants to help out would be to review #1807870: Don't assume $form_ids are strings (function names) when building forms. as without solving that, we'll have lots of errors for every form the installer uses.

Letharion’s picture

FileSize
50.64 KB

Uploading a patch.

I still can't get the final steps to work, uploading the patch in hopes of getting some feedback.

The InstallerKernel currently doesn't have any knowledge about which modules are enabled, but needs to know for steps like "Configure your site" to work. Currently the installation will fail as a lot of function calls will be directed at modules that aren't loaded, even though they are enabled.

Letharion’s picture

Minor update.

cordoval’s picture

Letharion, are you around, I want to help so i will start working and try to commit something if i understand it.

cordoval’s picture

I am sorry but can i ask some questions?

```php
$routes->add('basesystem', new Route('/basesystem', array(
'_controller' => array(new BaseSystem($generator, $request), 'interactive'),
)));

$routes->add('bootstrapfull', new Route('/bootstrapfull', array(
'_controller' => array(new BootstrapFull($generator, $request), 'interactive'),
)));

$routes->add('profile_modules', new Route('/profile_modules', array(
'_controller' => array(new ProfileModules($generator, $request), 'interactive'),
```

why did you add those extra steps? I am not sure they were there on the regular installer.

Also I think rather than building the stuff too generic, I think we should approach the installer in a different way, just making the default load of a bootstrapped setup already and then send the user to the dashboard of the actual install to customize.

Please correct me if I am wrong.

cordoval’s picture

@Letharion please do not post patches since this is a git tracked effort, with a patch I cannot see what your train of thought is.
I couldn't find a fresher branch of your work on your sandbox.

Letharion’s picture

@Cordoval:
53: Sorry, I wasn't, but maybe we can co-operate during the week instead? :)
54: All three steps come from the original install. You can see the full list in install_tasks() in install.core.inc.
55: Patches is, for better or worse, how the Drupal community is used to do this kind of thing. I've pushed a _3 branch now. I needed to rebase on 8.x, which again caused me to need a new branch. (I should talk to initiative owners about how to do this in a nicer way, cause creating new branches like this sucks)

patcon’s picture

Github has automatic mirroring set up for the drupal project, for what it's worth... I'm sure some people will disagree that this is a good approach, but it's there if those doing the work choose to use it
https://github.com/drupal/drupal

Crell’s picture

Posting periodic status patches to the issue is the current best practice, since that makes them Dreditor-available for review purposes. Even if it will eventually go in via a merge, that's the preferred way to do it.

For the original kernel branch, I merged from 8.x periodically before posting a new patch and just accepted the messier history. For the routing branch, I periodically rebased against 8.x, deleted the remote branch, and pushed with a -u to recreate it. I could get away with that because it was really just me working on it, whereas the original kernel issue had a dozen people involved. Please don't leave a bunch of _1, _2, _3 branches lying about, as that just confuses people.

Or we could bribe sdboyer to enable force pushes on sandboxes. :-)

Letharion’s picture

Status: Needs work » Needs review

I suppose merging might just be the way to do it. Yes, I do know that the new _X-branch for every rebase approach is severly lacking. (I have opened a feature request against the infra project #1785628: Add project specific setting to allow non fast forwards.).

Actually, I just realized that the patch _does_ allow an installation to happen. It unfortunately currently fails with an ugly fatal error, but that only prevents the final configuration form from appearing. After manually updating user 1 with a name and password, the site works.

Based on that, I'm setting the patch to needs review.

I wanna see how many tests break, and the patch really could use a good review. :)

Status: Needs review » Needs work

The last submitted patch, 1530756-52-symfony_installer.patch, failed testing.

Berdir’s picture

+++ b/core/includes/install.core.incundefined
@@ -47,6 +66,139 @@
+  require_once DRUPAL_ROOT . '/core/includes/install.inc';
+  require_once DRUPAL_ROOT . '/core/includes/form.inc';
+  require_once DRUPAL_ROOT . '/core/includes/common.inc';
+  require_once DRUPAL_ROOT . '/core/includes/unicode.inc';
+  require_once DRUPAL_ROOT . '/core/includes/ajax.inc';
+  include_once DRUPAL_ROOT . '/core/includes/module.inc';
+  require_once DRUPAL_ROOT . '/core/includes/cache.inc';
+  $module_list['system']['filename'] = 'core/modules/system/system.module';
+  $module_list['entity']['filename'] = 'core/modules/entity/entity.module';
+  $module_list['user']['filename']   = 'core/modules/user/user.module';
+  module_list('module_enabled', $module_list);
+  drupal_load('module', 'system');
+  drupal_load('module', 'entity');

Note: entity.module is no more, it's includes/entity.inc again. That might blow things up, if nothing else :)

Also, quite a list to get rid of, some parts will hopefully happen, but I don't think everything.

- install.inc: Possibly, as this issue should rewrite a fair bit of that already anyway...
- form.inc: very unlikely. But the only thing that queries the database is the cache (k/v soon for form_state) right now I think, and we can already plug that.
- common.inc: I guess you just need some specific parts from this one?
- module.inc: katbailey is working on the ExtensionHandler issue, which also includes an install-time implementation.
- cache.inc: I've been working on that.
- not sure which parts you need from system and user, will hopefully be possible to deal with that.

+++ b/core/includes/install.core.incundefined
@@ -47,6 +66,139 @@
+
+  // Check existing settings.php.
+  $install_state['settings_verified'] = install_verify_database_settings();
+
+  if ($install_state['settings_verified']) {
+    // Initialize the database system. Note that the connection
+    // won't be initialized until it is actually requested.
+    require_once DRUPAL_ROOT . '/core/includes/database.inc';
+
+    // Verify the last completed task in the database, if there is one.
+    $task = install_verify_completed_task();
+

I also have an issue that moves the database in DIC, but it's a bit problematic as we create so many containers...

Letharion’s picture

Me and cweagans had a skype chat, and discussed this issue, the kernel approach, vs the one boombatower started earlier #1226380: Generate minimal mocked or SQLite environment and use to install Drupal in full environment, which I will refer to as the sqlite approach. There are my thoughts after our talk.

I like the idea in the sqlite approach, while I of course also have a certain attachment to the code I've written. We agreed that cweagans would look more into the feasability of a hard sqlite dependency and/or shipping Drupal with sqlite, while I would take a look at the proposed code for the sqlite approach.

Since some parts of the installer won't go away in either approach, such as picking language and profile, it's possible that we can re-use those parts of the of the patch in this issue, even while we're going with a sql mini installation.

One of the advantages of the sqlite approach, as stated by cweagans, is that the new "real" installer would be much smaller, since most of the code would turn into regular module. This would make working on the installer itself less scary, and creating new profiles would be just like creating any other module. This certainly has it's merits.

I believe there is a very similar advantage of the Kernel approach. It will also make the installer a lot less scary, because we will:

  1. Get rid of a lot of global state, $install_state, more specifically.
  2. The drupalism "Install tasks", will turn into "Controllers", a concept that a large number of developers will be familiar and comfortable with.
  3. Since we will also re-use the core patterns of a Symfony based kernel, and a DIC, Drupal and the installer will be a lot more a like, compared to today.

Summary, from my point of view:
Sqlite approach:
Advantages: The installer code becomes very small. Profiles becomes modules.
Drawbacks: Possibly a hard dependency on sqlite, although there is also a suggestion for writing a simple hardcoded and array based dbtng backend. While a lot smaller, the remaining installer will remain a "drupalism".
Potential obstacles: There's no way currently of settings the database in a "read only mode". Fixing this could possibly lead to us moving some of the "if ($in_installer) {} else {}"-oddities from where they are to dbtng.

Kernel approach:
Advantages: High re-use of coding patterns between core and the installer. No extra/new external dependencies.
Drawbacks: The installer will mostly remain as big as it is today.

sun’s picture

@Letharion: I'm afraid, #62 is off-topic for this issue. I'd suggest to copy your comment over to #1226380: Generate minimal mocked or SQLite environment and use to install Drupal in full environment and empty out your comment here.

cweagans’s picture

Title: Implement the installer as a proper kernel » Rewrite the installer. Again.

Or we could just change the title

sun’s picture

Title: Rewrite the installer. Again. » Use a proper kernel for the installer
Crell’s picture

Installer-as-module and shipping with an SQLite DB are a totally different model. The goal of this issue is to make the installer a stand-alone micro-app that needs no existing data-storage systems. That makes it more stable, and also for those parts that can and should be shared it serves as an impetus to make those systems able to work in both a normal and an installer environment. That in turn forces good architectural cleanup, and means doing the same to the update system later becomes fairly straightforward. Let's stay focused on that.

sun’s picture

Hi there :)

I looked through the existing patch/code being proposed in this issue, and I have a couple of concerns regarding whether we are accounting for the actual requirements for these micro-apps, and thus, whether we're implementing the proper architecture here. In no particular order:

I expected to see a more sophisticated, generic "Ordered-Task/Checklist/Step Manager Service" or "Guide Service" in this code. The "ordered" aspect is crucial. The install_tasks*() and install_run*() functions are relatively mature implementations of an algorithm which I'll call "Stepper" for now (as I don't know what the proper name for this is). What they do is to set up a container and registry for prioritized application steps that need to be performed. Each step is a separate handler that gets registered with the Stepper, and all steps are eventually sorted into a fixed order. The Stepper is the step controller in charge, and on invocation, it checks which steps have been executed — by invoking a particular method, or when available, reading step completion information from persistent storage. Each step handler is able to return whether it has completed, whether it needs to re-run for all steps, and a step can also request that it is to be skipped since its preconditions are no longer met or its goal became irrelevant. Upon invocation, a step possibly returns a Response that forms the main page content.

So I really expected to see the current install tasks manager functionality ported into such a generic Stepper service here. The Stepper functionality is very generic — update.php apparently should use it, too, and thinking even beyond that, even modules could have a use-case for that (e.g., think of a store checkout process).

Closely related to that, implementing the installer step pages on individual/separate routes doesn't look right to me, as the route names are in no apparent order, and the step[per] controller has to be re-specified for every route again. A query parameter indicating the page to show seems more appropriate, and would ensure that the Stepper is the controller for all steps.

What is *really* important is that extensions (primarily installation profiles, but also modules) are able to

  1. Add new steps to the list, at any position.
  2. Remove steps from the list (or force them to be skipped).
  3. Replace and take over a predefined step.

1) should be clear, that's the most basic installation profile functionality. 2) and 3) are a bit more advanced, but still possible currently; a concrete example for that is the Demo install profile, which essentially places itself as the first possible step in the list and allows you to re-install a site from a snapshot/backup - after completion of this step, all steps are effectively done, since the site has been replaced with a working installation.

In all of this, I think it is important to differentiate between these Step controller steps, and an individual step that invokes a batch to perform operations. There can be multiple steps that may trigger a batch run, each, independently from each other. Even if it is only for the purpose of this discussion, we should call them precisely; i.e., steps vs. [batch] operations.

Each step can be a plain HTML main page content response, a form, a directly executed action (immediately moving forward to the next step upon completion), or a batch with operations.

What I think we should definitely aim for is to detach a step's actually performed low-level/backend action from its frontend/UI part. With that I specifically refer to the form and batch steps, which are currently executed via programmed form submissions (eeeek) when the installer is invoked non-interactively. Instead of that, I think we want a clear API/UI separation, and each step should define an ::execute() method (or similar).

In general, a Step controller should be able to eliminate most parts of the global $install_state, but I think we will still need some kind of $settings/$params "state" of values that defines execution parameters and is passed to all steps.

In these micro-apps, we are typically facing the situation that we have ~3 different environments to cater for:

  1. Pre-execution: There is nothing at all yet (install.php), or we have to enforce-fake that there is nothing, since everything is not ready yet (update.php). This is also where configuration is set up and where data potentially needs to persist across requests without a storage.
  2. Execution: The micro-app's primary operations are performed, possibly in a batch. In this env, extensions slowly happen to exist (or are made compatible), and each extension that "appears", needs to be able to participate in the subsequent operations. Technically, this environment is the same as a regular, full-blown Drupal environment; the only difference is an enforced, limited list of extensions in each operation performed.
  3. Post-execution: If something went wrong, this environment is identical to the Pre-execution environment; there's nothing we can rely on. However, if all steps and all operations have been executed, then we are in a full-blown Drupal environment, which is identical to index.php.

At least for the installer use-case, I think there is no way around injecting the service container as a dependency into the steps, since the entire purpose of almost all of the steps is essentially to set up and configure services.

I'd love to help out there, but I don't have access to that sandbox, and I also don't want to step on anyone's toes in trying to "slightly" revamp this code...

What do you think?

fubhy’s picture

Yes. I have recently started hacking away on the UpdateKernel which I implementing in a similar way as to what you are suggesting here (except for the obvious differences that it has compared to the Installer). I have all day tomorrow to work on that so if you are interested to talk about it I would definitely be available for that. What about you @letharion?

Letharion’s picture

#67:

I expected to see a more sophisticated, generic "Ordered-Task/Checklist/Step Manager Service" or "Guide Service" in this code. The "ordered" aspect is crucial. The install_tasks*() and install_run*()

I haven't missed that this is a requirement, I just have neither taken the time to really consider how it would work, nor attempted to write it. My initial thought would have been to have an equivalent of install_tasks() responsible for generating the current list of controllers, and then, pass the generated list by reference on to the profile as soon as it's loaded. This still leaves the entire stepping mechanism out, yes, I know.

I certainly like your idea for making the "Stepper" generic enough that it could be re-used by other parts of Drupal, but that haven't struck me before at all, so at this point I have no other comments about that.

A query parameter indicating the page to show seems more appropriate, and would ensure that the Stepper is the controller for all steps.

Just a ?step=[machine_name_of_step] would be ideal then, no? A machine name instead of a number would be more robust, in case the order changes, and we'd store the minimum amount of info in $_GET. (As opposed to other more persistant data, such as profile choice).

the step[per] controller has to be re-specified for every route again

I don't really understand what you are saying here.

In general, a Step controller should be able to eliminate most parts of the global $install_state, but I think we will still need some kind of $settings/$params "state" of values that defines execution parameters and is passed to all steps.

While I don't see how the controller itself would eliminate globals (though it might, maybe I'm just slow), I agree that some state needs to be passed between the Steps. Fubhy suggested replacing the current array with an object so we can have {g,s}etters on it. A crazy idea that came to mind right now is, could we use CMI to manage this state, removing the need for an installer specific class?

What I think we should definitely aim for is to detach a step's actually performed low-level/backend action from its frontend/UI part

Yes, this has been in the back off my mind as well, though it hasn't been written yet. I've been thinking in terms of "This part should be easily accessible to something like drush", while other parts, such as UI related code, should be separated from that, yes.

I'd love to help out there, but I don't have access to that sandbox, and I also don't want to step on anyone's toes in trying to "slightly" revamp this code...

You now have full access :) If building a better installer/updater/etc requires a bit of stepping on my toes, then so be it. I'd still like to be involved though :)

What do you think?

I really glad you took the time to do such an extensive review, thank you!

#68:
I'm up for talking tomorrow, yes. :)

Letharion’s picture

Crell’s picture

I don't think it makes sense to bring in a whole bundle for the installer, but we absolutely can look at them and steal ideas/concepts. Controller wizards sounds like something we'll want to have in Drupal anyway, although a solid central system for them may not make it into D8 core.

sun’s picture

@Letharion: Heya, I had to work on the installer for a couple of other issues, and I think that the changes I'm proposing in #1798732-42: Convert install_task, install_time and install_current_batch to use the state system (and following) should also be a huge help with the kernel conversion. What do you think?

Letharion’s picture

I'll read up on it, thanks :)

Letharion’s picture

I had entirely missed that we had a new state system, so finding that was great. At first glance I agree with what Sun said, using the state system would improve the installer as well as help improving it further. I will see what I can do to help with #1798732: Convert install_task, install_time and install_current_batch to use the state system and it's blocker #1835144: pifr_drupal.client.inc makes too many assumptions about Drupal installer internals before I continue here.

catch’s picture

Category: feature » task
tim.plunkett’s picture

Version: 8.x-dev » 9.x-dev
moshe weitzman’s picture

sun’s picture

Version: 9.x-dev » 8.x-dev
Assigned: Unassigned » sun
Status: Needs work » Postponed
Issue tags: -html5

This is going to happen for D8, but we first need to move all of the post-base-system stuff out of the installer, because it does not actually run in the crippled installer environment. In other words, >60% of the installer code will no longer live in the installer.

I don't have a concrete issue to point you for that yet, because it is blocked on some other issues, sorry.

sun’s picture

#2192649: Remove drupal_set_title() from installation and update process laid some first ground work, to consistently return regular responses (render arrays or Response objects) from all installer steps/tasks.

Next, a major baby step:

#2213357: Use a proper kernel in early installer

Untangling that spaghetti code was a very painful job and after doing it, I now understand why no one felt brave enough to attack that.

That issue will unblock a full range of other issues: (1) Converting all installer forms into FormInterface, (2) Converting all installer steps/tasks into controllers, (3) Using proper routes in the installer, and (4) Leaving the install.php environment, as soon as the base system is ready to operate.

Please note that I'm going to create further child/spin-off issues to clean up all peripheral code that does not pertain to the actual installer functionality itself. The goal is to come back to this issue when the following state is reached: install.core.inc + install.inc contains the actual installer application code only.

In other words: There's too much spaghetti. We will fix the easy/trivial stuff first, so as to end up with a clear picture of what constitutes the actual installer.

kika’s picture

Now when #2213357: Use a proper kernel in early installer is done, makes it sense to re-activate this?

sun’s picture

Title: Use a proper kernel for the installer » [meta] Use a proper kernel for the installer

#2245249: Convert installer forms to FormInterface just converted all installer forms into proper classes.

Next step in line would be to convert the controller-alike code, but before doing so, we need to figure out what "kind of" of controllers are actually needed and how they are invoked, especially in light of the step/task/flow architecture. To do so, I created an initial proposal:

#2234315: Dependency graph resolved plugins for installer tasks

mgifford’s picture

Status: Postponed » Active
Crell’s picture

Version: 8.0.x-dev » 8.1.x-dev

Let's try for this in 8.1. If it would be too much of an API change there, it should be a priority for 9.0. *sigh*

mgifford’s picture

Assigned: sun » Unassigned

Just unassigning issues that haven't been developed for a bit in the D8 queue.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

andypost’s picture

There was an issue to add sqlite db for installer, so proper kernel and pre build container could be loaded from there

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

voleger’s picture

Why we can't use drupal kernel instance? Why we can't use $environment property to handle kernel behavior?

voleger’s picture

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

voleger’s picture

voleger’s picture

Version: 8.9.x-dev » 9.1.x-dev
xjm’s picture

andypost’s picture

Looking on https://symfony.com/blog/new-in-symfony-5-1-improved-microkernel probably better to keep different kernels as of now
But we can simplify them and keep install/update out of environment which supposed to be used for development purposes or ccould be helpful to run drupal_migrate env or milti-site to solve issues like #2985199: Extensions in Multisite Directories Not Registered When Rebuilding Cache

Ref https://symfony.com/doc/current/configuration/micro_kernel_trait.html

andypost’s picture

Issue tags: +Symfony 5

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

andypost’s picture

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.