Install system overview

I am writing this to communicate the general design, and project plan of my proposed install system. Unfortunately, to accomplish that I need to explain a fair amount of the background of how the project originated, and how it has evolved since then. This install system has been a goal I have been working towards for a while, and the plans I set into motion more than a year ago are busy reaching fruition now, and as such it is time for me to decloak the project, and follow the example of other big projects we have going on in the community at the moment by seeking more community assistance, and clearly defining the goals and requirements.

This post only aims to give a general overview of the install system. Each component will be discussed in depth in their own threads.

Project history

I first became interested in Drupal's install process in the run up to the Drupal 4.4.0 release. By this time I had been maintaining the PostgreSQL port for a few weeks, and I was thinking about a way to extend and manage the PostgreSQL support in Drupal, but not only in core, but contrib as well.

I spoke to Dries about it, and he was sufficiently interested that I started work on it. The first part of the problem I approached was the installation and upgrading of database schema's. This part of the system I christened the install API. It is an extension of the Drupal API to allow for the use of supporting files I called install files (they are like modules, but end with .install). The functionality of each of these files is essentially analogous to database.mysql, database.pgsql and updates.inc in Drupal core, but generalized into an API and made available to Drupal core and contributed modules alike. One of the other base requirements was that these files be external to the modules themselves, and not be required for normal operation.

I started coding, and at Dries' request, used Drupal itself as the first example of the system. I know now that was a mistake, as the Drupal system itself exists as a special case. The limitation I ran into, was that Drupal is structured to execute without any checks for such things as database availability. Meaning that all of Drupal's API's were unavailable in your typical installation scenario. This in turn meant that my fledgeling install.php file ended up exploding to several hundred / thousand lines of code copied verbatim from Drupal itself to be able to accomplish what I had set out to do, an extension to the Drupal API. Thus was exposed the single largest obstacle to getting the install system in core, and in an effort to continue development on the install API. I ended up hacking core, until it was in a state to at least bootstrap enough of the system to allow the install script to function.

After a lot of refinement, and several rewrites of my install script, I realized I was writing a piece of general functionality that deserved to exist on it's own, but be useable by the drupal install system. This system was the wizard API. This API's job was to generate a set of pages with questions on them, that had to be traversed in order, until all the validation was successfully passed. It was general purpose enough that it could be re-used for such things as node forms and surveys, but was powerful enough to drive the install script.

This is the point at which I stopped actively developing the install system and switched my focus to the new template system, and the theme that would become the PHPTemplate engine. This doesn't mean the project died however. Since that time, most of the code written has been running in production, and has been through several refinements. My goals however were very specific, and not aimed at allowing Drupal to become installer enabled.

My major contribution to the CivicSpace project was the initial development of their installation wizard using a completely rewritten version of the wizard API I had designed. It has been used successfully in their project for many months now, and it's kind of funny that every time I mention I am writing an install system that I should just use CivicSpace's. =) In CivicSpace, the limitation that I mentioned earlier has had the effect of requiring them to have several hundred (900 or so) lines of code copied directly from Drupal, in order to allow it to function. Once again, code like this will never be allowed in Drupal, and nor would we want it to.

As part of the hosting infrastructure I have written for Bryght, I further developed my install API to allow us to easily deploy, manage and upgrade Drupal sites. The system has been running in production there for several months, and has even been contributed to Drupal.org in patch form. I have not had time to follow up and get it into core yet however, but I hope that this post will offer the required incentives.

Being able to install a single module is all fine and well, but Drupal still lacked an essential understanding of module versioning or module dependencies, and this is the largest part of the install headache for many new users. There was a definite need for a dependency subsystem to be able to flag required packages, and not allow packages to be installed that would not operate. A prime example of this is modules which are installed on the wrong version of Drupal.

At the Drupal Conference in Antwerp, Vlado approached me about a dependency system he had been working on to allow him to quickly deploy and generate custom sites along with some fairly advanced distribution and configuration plans. Since then me and Vlado have been coordinating on his dependency system, and very recently his pet dependency system has reached a level where we will be integrating all our code and moving the project even farther ahead.

Install system components

Install API

• Install and maintain database schemas.
• Same mechanism used for maintaining core, through being ties to the system module.
• Allow for automatic database prefixing.
• Allow for easier maintenance and tracking of postgres port.

Wizard API

• Used to generate the 2 installation wizards. Pre-configure and post-configure.
• Allows for any modules to define their own wizards to use.
• Allows for other method of specifying variables, that can be validated and executed on (external CLI tools).

Dependency API

• Introduce packages.
• Introduce meta-information for all packages (dependencies, versioning, etc.).
• Used for installation, and for admin/modules.
• Virtual packages. Example :
  • E-commerce package, and other bundles.
  • Install profiles (DrupalEDU, etc.), virtual packages can still have wizards and settings.

Packages

A package is a anything that can be distributed. Such as:

• Modules
• Themes
• Styles
• Translations
• Theme-engines

Package manager

This component is not part of the core requirements of the install system but I am bringing it up because it will undoubtedly be mentioned by someone. It is planned, but there are several serious reprecussions to it.

• App-casting of all Drupal packages, down to each site that runs Drupal. Will be useful to broadcast new releases, especially in the case of vulnerabilities
• Allow for administrator user to automatically download and install / upgrade all packages.
• Use dependency system to find packages that are needed, including flagging when an upgrade/install is not possible.
• Allowing any external code to run on your system is a major security vulnerability, and will require write access to your install directory.
• Requires serious rethink of Drupal release procedures, possibly including such avenues as digitally signed packages.
• Essentially a web based apt-get for any running Drupal distribution.

Project plan

This is a short list, which I will extend in the near future to a full status list, much like the CCK status page. I am not the only person working on this, but we have reached a point where we can start integrating all the bits and pieces. This is the order in which I see the project completed, however it is not the order in which development has been taking place, as many of the components were designed to be as general as possible, and as such function completely on their own, we have a good amount of tested and production ready code already.

Phase 1: implementation of the install API

Goal: creation of install.inc and allow for upgrade and install of any contributed module that has a .install file. Minimal integration with admin/modules page, to allow for the least invasive basic functionality.
Status: PATCH

Phase 2: implementation of Dependency API

Goal: creation of dependency.inc and integration into Drupal administration system. Introduce the concept of packages, and meta-information. Build on phase one to allow for automatic install and upgrade of required components (no automatic download).
Status: IN PROGRESS

Phase 3: implementation of Wizard API

Goal: integration of Wizard API into system. It might need some refactoring, and I would like to see some features added, but it is pretty complete, and has been powering the CivicSpace installer for a very long time.
Status: IN PROGRESS

Phase 4: allow for 'safe' start up of Drupal, to allow for maximum re-usabilty of core.

Goal: the big issue that has been holding up the install system for over a year. This requires an extensive face-lift and re-think of the entire Drupal initialization process, and possibly reworking of part of Drupal's theme system (it needs a presentable fail-over theme in the absence of a database to tell it what theme to use, for instance). Recently, everyone's hero chx has independently started on a path of development with his variable bootstrapping patch, that might indeed form the core of the solution to this issue.
Status: IN PROGRESS

Phase 5: core install wizard

Goal: now that Drupal can initialize without a database, and it has all these API's that are designed specifically to facilitate installation of modules, we implement a clean, cohesive pre-install wizard for the system package, which allows for creation of config files. This should be triggered through an install.php file which is multisite aware, and capable of creating more than one site (I have a very good mechanism for this I will relate later.). If an already installed site is accessed, it would require the administrator to log in (not using user.module, to minimize dependencies on code that might get out of date) and update the site through an interface. It's actually a lo simpler than it sounds, and we have both my original installer code and the civicspace installer to base the install wizard on.
Status: IN PROGRESS

At this point, what I feel are the core requirements for the install system have been reached. What follows is how I feel future functionality should develop, using the groundwork we have built.

Phase 6: re-work release process

Goal:taking advantage of all the meta-information for packages and themes, rebuild the project module to show dependencies and all other information. Build RSS streams of all the available packages for each available version and allow user sites to check their modules against what have been installed and notify the administrator packages are available. For successfull upgrade of sites into the install system, it will likely mean we have to switch over contrib on a certain date, and all modules should be up to date and using the install api on that day. Modules that installed without it will cause problems, as they have no version info to inspect.

Phase 7: creation of install profiles

Goal: to allow new users to more easily install Drupal, and power-users to easily clone sites, we should probably develop several install profiles. DrupalED, DrupalBLOG and DrupalCOM are examples of what should be install profiles. Install profiles are nothing more than virtual packages that provide a custom wizard, and depend on a set of modules. Install profiles are essentially any package that 'provides: install-profile'. The post-configuration wizard (right after the database has been installed and the site can start up normally), will allow you to select one of the install profiles it found on the site. This is one of the places where the dependency system shows it's real power.

Phase 8: package manager

Goal: this is the big one. I absolutely love the idea, but the security minded among us have cold chills just thinking about it. Firstly, you will need write access to more than just your files directory. Secondly, it could irretrievably destroy your site. Thirdly, it opens up all sites with it enabled to a wide plethora of attacks (the recent firefox exploit should be proof enough). Furthermore, to do this responsibly, the release process on Drupal.org is going to have to include digitally signed packages of all developers, and it's going to require security auditing of anything we beam down to the sites.

All that said, I think this is the ultimate goal of the install system, and it is the piece of functionality that makes it all worth it. So, I will probably build the functionality, as carefully and as safely as I can, and then only allow someone to use it after they have A) downloaded it from contrib themselves, B) read and agreed to the terms of agreement, C) specifically enabled a variable in their settings.php to enable them to use it. Wordpress already has something like this. Although I'd hate to see what happens when we have a profile-Wordpress install profile package. *coff* *coff*

Closing words

I'm exhausted after typing that out, and I am quite busy this week, but I will try and type out / draw out the design document for each of the components. If all this seems like overkill, and way too complex, I assure you.. it isn't. You are looking at the very big picture of almost 2 years worth of development on several internally cohesive components. There's enough reason to include each of these components seperately into Drupal core, and they were designed to function as completely additional extensions to Drupal, so apart from the bootstrap re-shuffling, they are minimally invasive.

The order I have designated of the phases was assigned because I feel that each part builds on top of the other, and has enough merit. I chose to tackle contrib first, as an installation framework for core first, would mean phases 1 through 6 would have to be accomplished before patches could be accepted into core. Feel free to discuss additional requirements and thoughts, I will try to answer you timely, but I have a lot of backed up work along with some pretty cool theme system trickery I have up my sleeve.