INSTALL.txt documentation fixes / changes / improvements

Nice work!

I'll reflect this in the g.d.o docs as well.

Steve

+ adduser --home /var/aegir aegir 

you want --system here.

+ /var/aegir/drush/drush.php dl --destination=/var/aegir/drupal-6.*/profiles/hostmaster/modules hosting install_profile_api

do we really need the full path for the destination? or can't we just put "." here?

same here:

+ /var/aegir/drush/drush.php dl --destination=/var/aegir/drupal-6.*/profiles/hostmaster/modules admin_menu
+ /var/aegir/drush/drush.php dl --destination=/var/aegir/drupal-6.*/profiles/hostmaster/themes eldir

this won't work if the settings.php is in site.example.com:

+Now point your browser to http://aegir.example.com/ , assuming
+DNS is properly configured to point to your webserver, otherwise you
+may want to try http://localhost/ .

Should be made clearer.

Thanks for the hard work!

This mean that the recommended setting IS to install the AEgir hosting into /site/Aegir.xxxx.com and NOT as the default site ?

I thought is was not the case anymore following issues and some docs, and more importantly it does not work without a patch because some drush commands are ran without a --uri="Aegir.xxxx.com" parameter. Theres is an open issue about this and a pending patch. just FYI.

Thanks for the work, I'll try to finish my Aegir install and review the new doc ASAP

@annakan: the recommend setting IS to install AEgir in sites/aegir.example.com and NOT the default site. It has never been the case before, it's a new feature of 0.3 that this actually works. The other patch will be committed shortly (i assume you are talking about #553202: Hosting 'Initial Setup' drush command requires --uri parameter or else looks up sites/default/settings.php, testing appreciated...).

I've reworked it a bit taking as a starting point mig5 patch.

Quick and dirty changelog:
* moved some general info (no windows, unix like) from "Web Server" to previous main "Aegir" section. Added a note indicating example instructions for debian-like systems are provided.
* updated all references to Aegir 0.2 to 0.3.
* changed all "sudo apt-get" to just apt-get.
* added a Aegir system user section. Some sentences literally copied from hostmaster profile.
* dropped all CVS instructions in favor of drush dl. Both CVS and drush instructions given.
* removed "other requirements" section.
* install_profile_api instructions still present. Just drop it when situation is cleared up.
* changed drupal-6.* to drupal-6.x and put a message warning to make the replacement to drupal version just downloaded.

New changes:

1) Forced to 72 chars length when possible. more rst style.
2) Added two environment variables to use in the install instructions:
- export DRUPAL=drupal-6.13
- export AEGIR_TAG=DRUPAL-6--0-3-RC3

This way all version specific instructions are isolated to those two lines and the drush 2.0 section.

3) Use another export for drush: export DRUSH=/var/aegir/drush/drush.php

4) Be more verbose in each component installation. Just a little paragraph with some info for drush, provision, drupal, hostmaster, hosting,..

5) Install profile API instructions still present. Forced to download version 2.1.

6) creation of config and backup folders moved up to Aegir user section.

7) Added a DNS Configuration section instructing about etc/hosts

8) Use relative paths almost all the time. Minimal usage of `cd`

9) Added /var/aegir/platforms as recommended location for extra platforms

Finally, some instructions are duplicated in INSTALL.txt and hostmater.profile and should be dropped from hostmaster.profile. They are:

* first page of the wizard (complete). " A system capable of running drupal. UNIX. No windows".
* second page of the wizard. Almost complete (create user account, add to web server group) but sudo instructions. Sudo line can be moved to INSTALL.txt. Perhaps Aegir's user home dorectory can be known via php ¿? and used as a base for next paths (drush, backup, config)
* third page:
- drush and provision are already installed. Just ask for the drush path. It can be prepopulated with location recommended in INSTALL.txt (/var/aegir/drush/drush.php). Currently it shows an example on /home/aegir...
- config, backup: verbose can be moved to install.txt and simply ask for the paths (possibly prepopulated from Aegir's user home directory path)
- apache config is already done. Related issue for conflicting apache configuration: #553416: Redundant/oudated instruction between install.txt, install.sh and the install wizard
* Forth page: database: ok
* Fifth page: optional hosting features: ok

New changes include more verbosity:

New sites database names and users will have the form `site_$nid`, where
$nid is the node id of the site being created (Aegir create sites as
nodes).

That virtual host is a special one that includes a NameVirtualHost
directive for *:80. Debian uses to provide such a directive in
/etc/apache2/ports.conf. Other distributions may include it directly in
httpd.conf. If your apache server already include such a directive, you
may comment it out in Aegir's main virtualhost.

New patch with validated rest syntax. Attached is the generated html with `rest2web`.

I've uploaded it to http://faita.net/~jonhattan/INSTALL.html for if you wanna quickly put an eye on it.

Overall: the patch should be generated with cvs diff -u (unified format).

That really belongs to the frontend documentation, not install.txt:

New sites database names and users will have the form `site_$nid`, where
$nid is the node id of the site being created (Aegir create sites as
nodes).

Heck, we could build a manual out of stuff like this and it won't fit in INSTALL.txt. INSTALL needs to be quick and to the point, an easy howto for people to install without having to clikety their way all over the wizard. ... Besides, that's a bit misleading: Aegir can not only create site_N databases, but manage *any* database on the server provided it's associated with a drupal. So it can backup and remove other databases.

That belongs to the configuration file itself, it should be a comment in it:

That virtual host is a special one that includes a NameVirtualHost
directive for *:80. Debian uses to provide such a directive in
/etc/apache2/ports.conf. Other distributions may include it directly in
httpd.conf. If your apache server already include such a directive, you
may comment it out in Aegir's main virtualhost.

People are expected to edit the template anyways (that should be very clear in INSTALL), and they can notice the comment and fix it if required. Also, the duplicate NameVirtualHost are harmless and may in fact help diagnose configuration problems (for example if the server is already configured for NameVirtualHost 1.2.3.4:80, aegir may break).

chown -R aegir /var/aegir/*

That should be chown -R aegir /var/aegir, no?

Checking out Aegir's code from CVS allows you to easily update to future newer versions.

That's not actually true: UPGRADE.txt is what makes upgrades easier and you should *never* use CVS to upgrade hosting or hostmaster.

su -c "su -s /bin/bash aegir"

That's just weird, why the double su?

At the time this document is being written current drush stable version is 2.0. Perhaps a newer version does exist and this document is not updated yet.

Add: "You should use the latest stable version of the 2.x branch."

mkdir .drush

belongs to provision, below, so we avoid duplication.

9) Added /var/aegir/platforms as recommended location for extra platforms

I don't see that anywhere on http://faita.net/~jonhattan/INSTALL.html... I would remove 'Installing new platforms' altogether, it's documented in the "create platform" page and will eventually be automated anyways.

Regarding duplication:

* first page of the wizard (complete). " A system capable of running drupal. UNIX. No windows".

I don't mind that duplication.

* second page of the wizard. Almost complete (create user account, add to web server group) but sudo instructions. Sudo line can be moved to INSTALL.txt. Perhaps Aegir's user home dorectory can be known via php ¿? and used as a base for next paths (drush, backup, config)

I think we can keep this duplication too... As vertice said, when we verify, we can point people to errors and also how to fix it in the wizard...

* third page:
- drush and provision are already installed. Just ask for the drush path. It can be prepopulated with location recommended in INSTALL.txt (/var/aegir/drush/drush.php). Currently it shows an example on /home/aegir...
- config, backup: verbose can be moved to install.txt and simply ask for the paths (possibly prepopulated from Aegir's user home directory path)
- apache config is already done. Related issue for conflicting apache configuration: #553416: Redundant instruction to Include /path/to/config/vhost.d in install wizard?

1. not sure what you mean in drush
2. i would keep the config/backup instructions
3. let's treat the apache config situation separate

In general, about the duplication, let's deal with it when we get there. Just reroll (sorry :) the patch with the recommendations here and we'll deal with the wizard later.

Also, a catch by mig5:

09:52:27 <mig5> cvs co -d profiles/hostmaster/modules/hosting -r$AEGIR_TAG contributions/themes/eldir
09:52:31 <mig5> that doesn't look right
09:53:11 <mig5> cvs co -d profiles/hostmaster/themes/eldir -r$AEGIR_TAG contributions/themes/eldir

Great work, thanks!

1. site_$nid paragraph removed
2. vhost paragraph removed (should I fill an issue against hosting for that?)
3. chown -R aegir /var/aegir/* It is ok because adduser will create /var/aegir owned by aegir.
4. removed "checking out from cvs is easy to update". It was just a try to give a difference between using cvs and drush.
5. double su: coppied out from /usr/share/doc/postgresql-8.3/README.Debian.gz (we suppose at that momento we are not root)
6. added 2.x drush branch message
7. moved mkdir .drush below to provision.
8. /var/aegir/platforms was removed in one of last changes. All additional platforms info removed now.
9. eldir path fixed
10. unified patch attached :)

alright, committed. a few issues:

* you included an unrelated chunk in the patch (hostmaster.forms.inc), don't do that
* please do mark as "needs review" when you upload a new patch... for review. :)

otherwise: thanks!

Some new changes:

* reverted to rst System commands:: as sugested by anarcat. It was not working before because of a missing blank line. Each time INSTALL.txt is changed. It should be parsed to verify. I use to parse to html with rst2html INSTALL.txt > INSTALL.html.
* changed adduser --system --home /var/aegir --ingroup www-data aegir to

 adduser --system --home /var/aegir aegir
 adduser aegir www-data

because "explicit is better than implicit", to be more consistent with hostmaster.profile and perhaps os-wide.

Even more:

* changing eldir and admin_menu instructions to use ./profile instead of $DRUSH_DIR/profile to be consistent with above instructions
* introduced $AEGIR_DOMAIN to allow for easy replacing `aegir.example.com`
* added an advice to not exit the aegir shell during installation. Perform root commands in other terminal

Aditionally two changes to install.sh.txt:
* changed version to RC4
* added install_profile_api installation

Credits line:
mig5

Ough spects it is the last for today:

A bug was introduced in #16 when removing --in-group from adduser. Now permissions for www-data must be set: chgrp www-data sites/$AEGIR_DOMAIN/{settings.php,files}. This is the only difference with previous patch.

Quick review.

 adduser aegir www-data

That's explicit only to some people. I, for a long time, didn't know what that meant, maybe because of BSD-land, I don't know. I find --ingroup much more explicit. But then that's nitpicking.

+ /var/aegir/$DRUPAL_DIR

Isn't DRUPAL_DIR /var/aegir/drupal6.x?

Regarding changes to install.sh: great, but we have a systemic problem here: the release process needs to be aware of all the places where we need to change that tag when releasing, not sure install.sh or even install.txt is documented there.

The patch otherwise has my go.

The redundancy issue is another issue.

Changes:

1. revert Checkpoint to literals. removing variables (/var/aegir/$DRUPAL_DIR thing)
2. added --group aegir to adduser -> adduser --system --group aegir --home /var/aegir aegir because --system put the user in `nogroup`.
3. added a comment to -> adduser aegir www-data #make aegir a user of group www-data

For completeness those could be defined: $AEGIR_USER, $AEGIR_GROUP, $AEGIR_HOME

I wonder if there's a way to automatically propagate the current cvs tag to install.txt and install.sh as Drupal does with $Id$. Or it is even possible in git land.

patch runs fine, except last line in aegir_vhost () function, where the insert of "Include /var/aegir/config/vhost.d/*" line is not executed, also the /ect/apache2/httpd.conf remains empty. Symptom is when I start Aegir (http://aegir.local), the default index.html of /var/www appears.

Quick-and-dirty solution is to manually include the line into httpd.conf.

I use a fresh Ubuntu 9.04 server.

Here is my output:

SECTION 4 : Creating Virtual host configuration file
Choose a domain for your hosting site. It should be already pointed to this computer:  
aegir.local
Creating /var/aegir/config/vhost.d to store configuration files
apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
[Fri Aug 21 17:43:39 2009] [warn] NameVirtualHost *:80 has no VirtualHosts

I set it back to active, see #22.

doka: this issue is mainly about INSTALL.txt (at present the first source of information for installing Aegir). In my last patch I added two little changes to install.sh.txt to update it to current RC4.

install.sh.txt is a script to ease the installation on a debian-like system and is not supposed to be used for end users (it also installs git-core and other things). It is not in sync with INSTALL.txt and is not a priority at all to sync them. So please use INSTALL.txt as a reference. I think this is why you are having this issue: #556548: Misleading instructions at "Configure your database server" page.

That said, install.sh.txt do the inclusion of "Include /var/aegir/config/vhost.d/" to file apache2.conf not httpd.conf. The command to find the file where to do the inclusion is:

$ apache2ctl -V  |grep SERVER_CONFIG
 -D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf"

Thanks for the clarification, my issue has been around install.sh.txt.

You are right, after running install.sh, there is a line at the end of /etc/apache2/apache2.conf

Include /var/aegir/config/vhost.d

But the closing / is missing there. And even I put the slash to the end, the aegir site is broken in Ubuntu (no clear URL in Drupal, i.e.).

What helped me is to put

Include /var/aegir/config/vhost.d/*

into httpd.conf

doka. The line has always worked for me is

Include /var/aegir/config/vhost.d

and at least this instruction is the same across all aegir's install instruction docs :)

As an example this is what debian (and for sure ubuntu) do at the end of apache2.conf:

aegir@jengibre:~$ tail /etc/apache2/apache2.conf 
# Include generic snippets of statements
Include /etc/apache2/conf.d/
# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/

----

returning to INSTALL.txt issues. Attached is a patch with better and simplified instructions for "becoming aegir user". This ensure the aegir user has the correct environment (set $HOME,...) and `/var/aegir/.drush` is visible to drush.

You are right, the /etc/apache2/apache2.conf looks exactly like in my case also as you described.

My issue is, that in spite of the correct content of /etc/apache2/apache2.conf, if I start aegir in the browser, I get the content from Debians /var/www/index.html, and not the Aegir start page. And if I put Include /var/aegir/config/vhost.d/* into the httpd.conf, than I have the Aegir start page.

doka: it happens to you that apache default vhost (sites-available/default) is taking precedence over aegir's vhosts.

If fact apache2.conf do this:

Include /etc/apache2/httpd.conf
Include /etc/apache2/conf.d/
Include /etc/apache2/sites-enabled/

So If you put the line in httpd.conf it will take the highest precedence. INSTALL.txt puts the line in `/etc/apache2/conf.d/aegir`. And if you put the line at the end of apache2.conf it will get the lowest precedence. Other thing you can do is disable default vhost (a2dissite default). Anyway, `install.sh.txt`is to be used internally, in the "you know what your doing" case :)

Currently there is an ongoing effort to unify documentation and, as I've already said, INSTALL.txt is the only recommended source at present (wizard must be updated, wiki page must be updated and install.sh.txt is not for end-users usage --it is not mentioned anywhere.. it's just an undocummented file in the cvs).

#534302: installation profile requires writable settings.php for one step further than expected/advised is finally fixed. Extra advice about that bug is removed in latest patch.

committed. thanks.

Hello, again :)

(nobody expects the Spanish inquisition... right ? :)

redoing the install with the new install.txt

I was able to redo a full install from CVS (mostly RC4 I think) following the INSTALL.TXT from end to end.
No frill.
No added value.
It went mostly fine and the instruction were much more clear and efficient.
99% copy and paste with a little bit of brain added there and there.

Only one think stopped me to have a first install verify check that passed :

Initializing drush commandfile: provision_mysql
Undefined index: db_url
Including /var/aegir/.drush/provision/web_server/verify.provision.inc
Including /var/aegir/.drush/provision/platform/verify.provision.inc
Including /var/aegir/.drush/provision/db_server/verify.provision.inc
Could not select the mysql database.
An error occurred at function : drush_provision_mysql_provision_verify_validate
Command dispatch complete
Removing task from hosting queue
An error occurred at function : drush_hosting_hosting_task
Changes for drush_hosting_hosting_task module have been rolled back.
Command dispatch complete

It seems the INSTALL.TXT doc is still missing the "MySQL-ism" bit

after

CREATE DATABASE aegir;
 GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, \
  CREATE TEMPORARY TABLES, LOCK TABLES ON aegir.* TO \
  'aegir'@'localhost' IDENTIFIED BY 'XXXXXXXX';

we should do

CREATE DATABASE aegir;
 GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, \
  CREATE TEMPORARY TABLES, LOCK TABLES ON

aegir.%

 TO \
  'aegir'@'localhost' IDENTIFIED BY 'XXXXXXXX';

the % thing ...
source : http://drupal.org/node/553348#comment-1942496

After I did it the verify step passed cleanly.

Cron and *BSD problem

As for FreeBSD-ism I have one nagging problem : the cron does not run.
To be more specific it does run but does nothing, it is a typical case of the command not being runnable by the Cron "shell".
Cron "task" run under a specific environment that is even more reduced in the BSD than in the linux world..
I am quite sure I just need to figure the env vars I need to pass for the PhP script to run ... still searching though.

One sure thing is that it is widely recommend to use FULL PATHS in cron jobs, so even under linux the line should be based on the output of whereis php and not just "PhP".

If anybody already encountered the problem of PhP scripts not running in Cron and knows what's missing I am all ears :)
I'll keep you posted if I find anything ...

small details maybe worth mentioning

One small comment : maybe the "hostname" part "aegir.example.com" should be abstracted in an environment variable and the command use that.
That way once you have set all you env var with the export things the rest is really only a simple replay of the procedure...

And last, maybe we should mention that is is better to have the rewrite rule in the xxxx.conf file and document and support that, and so be able to lock the .htaccess down, .. and it prevents "brutal" drupal-xxx untarring to overwrite or impose a wrongly configured .htaccess ..
Previously the .htaccess always was the 'thing to remember" to put aside while doing an upgrade, getting rid of it (don't caring about it any more) would be a plus.

Finally, I have several questions about the file layout and the expected use of AEgir.

Maybe someone could answer me to help me better express my "critics".

What I feel bad about in the current layout :

1. I don't like neither the fact that we mix the special AEgir hosting install with the rest of the side, nor the fact that it is not even the "default" site and is drowned into the sea of other sites...
2. I don't like either the fact that if I use a different platform, It will be tucked in another subdir (/aegir/platforms) while it is conceptually the same as the platform that is hosting the AEgir profile AND many "regular/hosted" sites potentially ...
3. I don't like the fact that the "web published" part is not cleanly separated from the rest, putting me at the risk of a simple configuration mistake that would publish too much datas, or a bug one day in AEgir httpd.conf rewriting doing the same... Plus I can't have my apache documentROOT set to a "safe" directory, it is either /var/aegir (too high, I have conf data, back up, drush and .drush and so on under that) or /var/aegir/drupal-xx : too low it won't cover a /platform and it will change each time drupal changes version. It has a bad smell to me...
4. This "problem" is the same for off-line backups, and versioning, you have a special case for the hosting platform and the hosted sites that happens to run under the same codebase, and one other case for the other platforms .....

To better form my judgment could one respond to these questions :
how are we expected to upgrade the hosting platform ? :

• backup; unpack the tar and be done with it (hum :) )
• reinstall an Aegir under the new platform and migrate the sites ? (how do we do that? )

If we had the hosting platform separated from the other platforms we would have a cleaner upgrade path :

• upgrade (even brute force) the AEgir platform...
• if it goes wrong, none of your customer website on your "production" platform would be impacted
• if it goes ok you can copy/un-tar the new platform under the platform directory and migrate your sites as needed
• no chances that one "hosted site" module would impact the migration and impact the AEgir platform, or that a failed AEgir migration would impact the whole hosted sites ...

The only price is a small code duplication of the AEgir hosting platform if it is used as a hosted sites platform too , yes I hate that too but is seems like a good trade off to me ...

Am I way off ? or is it something worth investigating and me redoing an install with a bit different layout that goes into that direction ?
I could then submit it to criticism and we'll see if it is worth including into the install.txt document ?

I am open to all comments ....

Thanks again to the AEgir team ...

Wow, too much information here. Please open a new issue if there is a real problem to be solved here. I'll try to tackle some of the stuff you raised...

GRANT ... ON aegir.% TO ...

uh? aegir.%? I've been using aegir.* for ages, am I missing something?

For the cronjob issue, you may want to contribute to #323732: Improve the crontab entry to enhance portability.

Regarding the .htaccess, it simply is ignored in aegir: the generated apache configuration contains all the goods (which doubles as a performance enhancement).

I don't like neither the fact that we mix the special AEgir hosting install with the rest of the side, nor the fact that it is not even the "default" site and is drowned into the sea of other sites...

Having the site as 'default' is confusing and error-prone, and it makes processing of existing sites harder (because we ignore default sites (see #453540: consider the 'default' site like a regular site, with exceptions).

I don't like either the fact that if I use a different platform, It will be tucked in another subdir (/aegir/platforms) while it is conceptually the same as the platform that is hosting the AEgir profile AND many "regular/hosted" sites potentially ...

I actually agree with that.

I don't like the fact that the "web published" part is not cleanly separated from the rest...

Good point, open a separate issue please.

how are we expected to upgrade the hosting platform ? :

See UPGRADE.txt.

if it goes wrong, none of your customer website on your "production" platform would be impacted

The UPGRADE.txt process doesn't (need to) impact your production platform.

All in all, before adding more stuff to the issue, please consider if it is related to upgrade.txt instead (in which case you should open a separate issue), if it's something else than the recent changes that happened in INSTALL.txt (in which case you should open a separate issue) or if it's , well, a separate issue (in which case... )

All in all, please try to keep things issues isolated, you have a tendency of trying to do everything at once and while it's nice to have feedback, it makes it very hard to followup on your work because it's all over the place. ;) You could have opened at least 4 separate issues with the things you raised here.

keep it simple, but no simpler.

Ok thanks for the feedback, I am bit new at "collaborating" on an open source project (more like ranting sometime for me, sorry about that)

I am often reluctant to open new issues because I am not sure my judgment is sound but I am wrong about that. Opening an issue focused on a problem that can be closed quickly is better than opening a discussion here.

I'll try do to better.

@annakan - absolutely no harm done here, you are doing this pretty well if you are a beginning in open source. you are documenting properly and extensive in what you write, that's great. it's just a matter of getting used to the various policies in each project that's tricky. :)

here I rather have separate issues opened for each issues as we like to fix stuff, it's gratifying and it's clearer.

don't hesitate to open issues if you feel there is a problem and you have read the bug reporting guidelines and the FAQ. if you are unsure, categorize it as a "Support request" so it doesn't show up as a bug to us (we can always change it back later if it's really a bug).

thank you for you contribution.

I'm working on updates to INSTALL.txt to roll in...

http://drupal.org/node/556548#comment-2043552
http://drupal.org/node/558498

and draw on some of the comments here:
http://drupal.org/node/553416

As well as some more descriptive text from the g.d.o docs.

I've reopened this ticket so I can also bear in mind previous discussions while I work.