Setting $options['alias-path'] in one drushrc.php file should not overwrite default values, or values set in other config files

Didn't even read all of that... just getting as far as this:

That is, setting $options['alias-path'] does not add to the list of places to look for aliases

It is true that if you assign to $options['alias-path'], it is going to overwrite whatever is already there; this is standard php behavior. However, since drushrc.php files are executable php code, you could use array_merge to combine lists if you want or need to.

Once you have this fixed, you should find that the various *.aliases.drushrc.php files do "stack". n.b. that *.alias.drushrc.php will only be examined for the one alias whose name matches the first portion of the filename (the "*" above), whereas *.aliases.drushrc.php will be searched for all aliases.

Namespaces are not supported, though; you cannot distinguish between 'live' in @client1 and 'live' in @client2. You need to name them something like @client1live and @client2live. Namespaces would be nifty, though.

Okay, I read the rest of your post.

If you want to solve the "passwords in version control" problem, then you could have your settings.php read your $db_url from another file.

Alias files do stack. You just can't have two aliases with the same name.

If you don't want to modify $options['alias-path'] with php code, then you could pre-define all of the locations your users should be allowed to store their alias files. It isn't terrible to put your personal alias files in a pre-defined location is it?

If you think that 'alias-path' should be an array so that you can say, for example $options['alias-path']['my-unnecessary-key'] = '/my/location/for/aliases';... well, I don't think that feature is terribly necessary, since there are workarounds (per #1) that I think are fine. But we could always put it up for discussion if you provide a patch.

Edit: I was at least partially insane when I wrote the above. $options['alias-path'][] = '/my/location/for/aliases;' would be a reasonable way to handle it.

As Greg says, they already do stack. Multiple are checked in the same request. Still, I'm open to making that option be an array or a delimited list of paths.

As for passwords, thats a problem well handled in core drush. See sql-sync for example. It calls out to sql-conf in a background process in order to learn the password for a remote site. Your custom commands could do the same. Search for $values = drush_do_site_command($alias_record, "sql-conf", array(), array('db-url' => TRUE));

... then just add the standard locations to $options['alias-path'] in your drushrc.php?

I would not be opposed to a patch that pre-populated $options['alias-path'] to an array containing the default locations. Then, with the addition suggested in #3, anyone could decide whether to replace or extend the alias-path. Existing drushrc.php files would continue to work as they do right now.

I might do #7 for drush-4; assigning to myself for consideration. In the meantime, patches are still welcome.

In #1104438: alias search path doesn't respect normal include paths (e.g. --include), i argue that there's no reason why there should be a different search path for aliases than for commandfiles or drushrc files.

Right now, those are three totally different search paths. One is defined in drush_sitealias_alias_path, another in _drush_config_file and the last one is in _drush_find_commandfiles.

All three have interesting properties:

* commandfiles search path can be augmented with -i on the commandline, and is bootstrap-sensitive. it also looks in the system folder (SHARE_PREFIX, defaults to /usr/./share/drush/commands) as it's nice to packaging systems. finally, it looks into modules directories.
* we look for drushrc files in the drupal site root, but not the others. we also add -c to the list of drushrc files
* the alias algorithm could be considered the poorest, as it's hard to extend as described here

I could concede that the commandfiles search algorithm may be different (code not configuration, let's not go there), so maybe we should just put that one aside for now.

I believe, however, that we should unify the alias and config file search paths algorithms. That would resolve the issue described as we could just add -i for extra search paths. Or at least we would solve the issue in one shot for everything.

Edit: for context - I am working on Aegir's ACL support to allow multiple teams to work on different projects and not access each others' stuff, so i guess there's a similar requirement for me here. My issue is that I want to be able to include "those aliases over there" regardless of my $HOME...

I re-read Crell's posts and I don't get what his issue is. I think he needs to put global alias files in /etc/drush and project specific ones in the drupal root for each project. they stack perfectly. I don't think he needs alias-path.

yes, it would be good to unify the $search_path variants a bit.

Grayside wrote an interesting blog post, Configure Drush in Your Git Repository that makes this feature (append-able alias paths) more interesting.

Here is a patch that allows you to add $options['alias-path'][] = '...'; to as many drushrc.php files as you need/want to, without interfering with --alias-path on the cli or the default alias search paths.

Rewriting the code to make 'include' and 'alias-path' the same thing would have been messy. They don't work quite the same way. It was easy, however, to add all of the commandfile paths (including the 'include' option) to the alias-path context. You now have the option of putting your drush alias files in your 'include' path. n.b. $options['include'] can only be set in the early drushrc.php files. Also, I made no attempt to load drushrc.php files from the 'include' path. This seems like a reasonable compromise between complexity and utility, though.

Continuing the theme of this issue, this patch enhances #11 so that alias files in different locations can both define an alias with the same name. If this is done, the two alias records are now merged together. Previously, an arbitrary one of them would 'win', and the other would be ignored.

Advantage of the old way: If you had @site1.dev and @site2.dev, you could use @dev, and it would reference an arbitrary one of the two sites. Now, @dev would merge the two records together, giving undefined results. Since the old way was not terribly useful, this is probably not a bad change.

Advantage of the new way: If you have some alias records defined in a location checked in under version control and shared among multiple developers, those alias records can now be overridden with directives in ~/.drush/myoverrides.aliases.drushrc.php. (n.b. you can define $aliases['site1.dev'] in your myoverrides file, and it will override @site1.dev, which was defined as $aliases['dev'] in site1.aliases.drushrc.php.)

I think it is unlikely that anyone is using aliases in a way that is incompatible with the change in #12 (that is, if you have @site1.dev and @site2.dev, it is probably unlikely that you are using @dev to select an arbitrary site named 'dev'). Any comments on this, though? Anyone want to review before I commit?

All tests pass if you apply #12; just ignore the rejected parts, already committed in #1263016: Identify the Drupal site to be used earlier in drush's bootstrap for drush quick-drupal.

Agreed. Looks good to me.

Could you add a comment explaining whats going on in the completeTest change? I have seen that failing for me so am curious.

On my drive in to work yesterday I realized I could be just a bit more ambitious here. New patch coming shortly.

Regarding the 'complete' unit tests, there are a bunch of them that run some code to generate a list of completions, and then the code tests to see if the first and the last are correct. The problem here is that complete does not sort the eligible completions, so sometimes when the underlying code changes, a complete unit test can fail only because the order of the results change. When I encounter these situations, I just change the test so that it passes again, but I think a more permanent fix (sorting) would be an even better idea.

Drush site aliases, now with extra coolness.

I used to store my site aliases in files called 'live.aliases.drushrc.php', 'dev.aliases.drushrc.php' and so on, with one file for each kind of sites. However, as the number of sites that I managed grew, I started to instead group them by site, in files called 'site1.aliases.drushrc.php' and so on. This later scheme was more convenient for most uses (drush sa @site1 to list all of the different instances of site1 being one benefit), but I also missed being able to do things like drush @live status "Drupal Version"

This patch allows the best of both worlds by defining site alias lists for @dev, @live, etc. for every alias defined in each alias group. For example:

# File: site1.aliases.drushrc.php

$aliases['dev'] = array(
    'root' => '/path/to/drupal',
    'uri' => 'dev.mydrupalsite.com',
  );
  $aliases['live'] = array(
    'root' => '/other/path/to/drupal',
    'uri' => 'mydrupalsite.com',
 );

File: site2.aliases.drushrc.php

$aliases['dev'] = array(
   'root' => '/path/to/drupal2',
   'uri' => 'dev.mydrupalsite2.com',
 );
 $aliases['live'] = array(
   'root' => '/other/path/to/drupal2',
   'uri' => 'mydrupalsite2.com',
 );

Then the following special aliases are defined:

• @site1 An alias named after the groupname
  may be used to reference all of the
  aliases in the group (e.g. drush @mydrupalsite status)
• @site1.dev The @dev alias from site1.aliases.drushrc.php
• @site1.live The @live alias from site1.aliases.drushrc.php
• @live A reference to all of the aliases named "live"
  (@site1.live and @site2.live).

It is also possible to merge user-specific options from one alias
file into the site alias definition from another file. This is useful
if, for example, a workgroup distributes base alias definitions in
a project under version control; individual team members may then
add additional options to the site aliases as shown below:

# File: ~/.drush/overrides.aliases.drushrc.php

 $aliases['site1.live'] = array(
   'my-option' => 'my-value',
 );

In this case, the alias @site1.live will contain all of the records defined
in the primary alias record merged with the options from the override file.
It is equally valid to define an alias named 'live' in a file called
~/.drush/site1.aliases.drushrc.php.

That sounds pretty good. It does give a lot of importance to the name of the alias. I'd love for msonnabaum to share what he is doing in drush deploy with site aliases. He is using custom filtering where he assigns an arbitrary property like 'role' => 'webserver' and then he takes actions on a site list that's further filtered by that role. He could describe it better.

I was thinking about filtering on properties, but I felt like adding xpath-like syntax to drush site aliases might be a bit... odd? Maybe %tag could look for 'role' => 'webserver' easily enough. Compatibility with drush deploy would be good.

My current thinking is to add %group as a new form of site alias specification; e.g. drush %group status.

You would be able to define groups in your drushrc.php file:

$options['alias-group']['live'] = array( '%role' => 'live');

In this instance, drush %live status would run core-status on any site whose site alias contained the attribute '%role' => 'live'. (The % in role is optional, but if you leave it out, then drush will pass --role=live to any drush command where that site alias is used; maybe not always desirable.)

If there is no group definition for a given tag, then by default, %tag could default to array('%group' => 'tag') and/or array('%tag' => TRUE). It might also be worth introducing a shorthand form drush %role=live status as a shortcut for the alias group option given above, although truthfully I would prefer to not use this form on the cli too often.

Finally, I think it might also be worthwhile to introduce single-argument versions of drush rsync and drush sql-sync. drush rsync %tag could be a synonym for "push code to the @peer alias of every site alias in the group %tag", and drush sql-sync %tag could go the other way, and indicate "pull the database from the @peer alias of every site alias in the group %tag".

I think these features would be helpful for people who work on groups of Drupal sites pulled in from different projects pulled from vcs.

Here is a start. Does still need some work and docs, but I'm looking for some feedback.

I decided that it was better to define collections in alias files, not drushrc.php files.

If an alias does not already have %role, then it is given '%name' = '1', where '%name' is the last part of the alias name (e.g. %live or %dev). If %role => x exists, then '%x' => 1 is defined, where '%x' is the value of '%role'.

If there is no alias collection for %foo defined, then '%foo' => '1' is assumed.

These defaults mean that you may use drush %live status much in the same way that drush @live status was used in #16. You may also use drush %role=live status to select all aliases with '%role' => 'live'.

Alias overrides must now be defined as $overrides['aliasname'] = array(.... The old code from #16 that merges together any two aliases with the same name is still in this patch, but it should go -- pending only a decision on what should be done when two aliases have the exact same name. Not sure about that at the moment.

I'm not clear on why these collection aliases need to start with a % instead of a @. Would be nice not to introduce a new convention.

I personally like this query feature but am a little worried about layering on more complexity to sitealias.inc.

I pinged Mark about this issue.

I actually ended up not doing the filtering by alias attribute, although I may reconsider it if it was supported upstream. My general approach with deploy is to just group aliases in files when you need them grouped, use the existing @ alias to refer to all of them, and use "parent" when overrides are necessary. A little clunky, but it works.

I'm having a hard time seeing the use case for calling an alias name over many group files. Like the example of @live being a group alias for @site1.live and @site2.live. That's pretty confusing to me, and I can't imagine there would be a command I'd want to run over a particular environment on ALL my group site aliases that had that env defined. Maybe I'm missing something there.

And I'm also with moshe on the % syntax. I think its safe to say that 99% of our users don't even know about the existing group aliases, so keeping this stuff simple is very important IMO. I'm not seeing the need for new syntax.

There were two reasons for using the % syntax: 1.) avoid namespace conflicts between collections and site aliases, and 2.) avoid the overhead of processing alias files for collections when they are not being used. I can work around 1 by only checking for collections if there is no explicitly-named site alias matching the request. I also thought of a way to recode the alias loading code to avoid the extra overhead of 2 by processing both kinds of aliases on the same pass. I'll role a new patch that uses @ instead of % for collections.

Common use cases for operating on collection of sites include:

• drush @live status "Drupal version" (Are all of my sites up-to-date?)
• drush sql-sync @dev (Pull the database from live for all of my dev sites -- more code needed to support this in sql-sync and rsync)
• drush @dev en devel (Turn on devel on all of my dev sites)

etc.

All of these can be done by maintaining site lists, of course, but the point of allowing 'alias-path' to be extended rather than replaced is that it allows you to pull in your alias files from multiple sources (e.g. one alias file per site, with each site checked out from a shared vcs). Once you start doing that, it is also useful to be able to refer to classes of sites without having to maintain the site lists independently.

We should extend the new 'grayside drushrc example' such that it sets alias-path as well, thus picking up alias files in the drush subdir of git repo. See #1255544: Read drush configuration file from a 'drush' folder at root of git repository.

--include should be processed in the same way as --alias-path; see #1306112: additional directories option doesn't support paths with spaces in.

This patch makes handling of --alias-path, --include and --config more uniform and less surprising.

* Any of these options can be set on the command line using a list of files/directories separated by PATH_SEPARTOR.

* Any of these options can be set in a drushrc.php file; if they are, the values set in one location will merge together with the values set in other locations.

* Backend invoke will correctly forward these settings values to invoke calls on the local machine, but will not automatically include them on remote calls. If --alias-path is specified on the command line, only the values provided on the command line will be sent; the default locations will not be sent.

* To allow backend propagation of cli values to be done correctly, long/short options (e.g. '--include' vs. '-i') are now reconciled to the long form at context-initialization time. The was done for all options with long and short forms for consistency, so there is no longer any need to call drush_get_option(array('r', 'root')); drush_get_option('root') will now suffice -- a nice code cleanup.

* 'alias-path' and 'include' are now unified, in that both the alias path and the include path will be considered when aliases are searched for. So, --include will specify both aliases and command file search paths, whereas --alias-path will only add to the alias search path. This unifies the two in a way that I think is acceptable.

The extensions to alias groups #15 - #24 have all been removed; we can pick these up on another issue.

I got rid of the distinction between $options['alias-path'][] = '...' and $options['alias-path'] = array(...), as I thought this was confusing. I think it actually is rare to want to turn off the default locations; the main use case for doing that is in the functional tests, and they already get around this by setting environment variables. Allowing the replace made it difficult to handle alias-path / include values specified on the command line, which would be added into the context first, then overwritten if set in any configuration file (whereas cli options should usually take precedence over configuration files). Always merging made this work better.

Good idea to ditch those recent enhancements for now.

'alias-path' and 'include' are now unified, in that both the alias path and the include path will be considered when aliases are searched for. So, --include will specify both aliases and command file search paths, whereas --alias-path will only add to the alias search path. This unifies the two in a way that I think is acceptable.

This strikes me as confusing still. Maybe we should just have an option for each (alias-path, config, include) and not try to multiplex any of them. Should we rename 'include' to 'command-path'? Anyway, the description for --include still only mentions commands.

Adding the tag 'Needs change notification'. That indicates that there is an API change here that needs to be documented. drupal core and all contrib projects now have a great system for doing that. See . We should ideally create change notification records for our all our API changes in drush5. I think it is convenient to communicate new features and API additions this way too, but that's debateable.

Update: more about change notifications

+++ b/examples/example.drushrc.php
@@ -87,9 +87,9 @@
+// *.aliases.drushrc.php files.  Overrides previous definitions.

Do alias-path definitions override or accumulate? Conflicts with your description of the patch.

+++ b/includes/command.inc
@@ -448,7 +448,12 @@ function drush_redispatch_get_options() {
+  // There are only a limited number of short options anyway; drush reserves
+  // all for use by drush core.

Hah. Nice land grab :)

+++ b/includes/context.inc
@@ -202,18 +206,51 @@ function drush_set_config_special_contexts(&$options) {
+          unset($options[$info['short-form']]);          ¶

trailing spaces. you have a couple more instances of leading or trailing spaces. one way for you to catch these (after the fact) is to use dreditor greasemonkey script. see http://drupal.org/project/dreditor

I added a change record for this issue here: http://drupal.org/node/1313574

Change records are cool. I see that there is a link to Drush's list of change records under the "Development" heading on the right sidebar on the drush project page.

Here is a new patch; --include and --alias-path are now separate again. I'm not really in favor of renaming --include or --config, but I suppose we could. I updated the grayside git example and the docs on --include and --alias path in the example drushrc.php file.

Protocol question: do we remove the "Needs change notification" tag from an issue after the change notification has been created, or do we keep it on until the issue is committed and we believe the change notification is 'final'?

Code looks good. RTBC. I think we implicitly test these options already but it would be ideal to document any gaps in our test coverage.

Perhaps you could expand the description of the command-specific and site-aliases global options. I think those are not meant to be passed on the CLI but it would help to know how Drush uses those internally.

I would say that we remove the tag once we feel that the change notification record is complete. In practice, you can't really say that before the patch has been committed. I would expect the CNR to trail the commit by a few days or so.

In the CNR, we should tell folks if they should convert their config and alias files to use long form option names. If they don't need to change anything, let's state that too.

Aliases and scripts can continue to use short options without change. I'll update the CNR to be explicit about that.

I did realize one thing that will break in contrib drush commands: drush_set_option('r', $root); must be changed to drush_set_option('root', $root);. I could add code that would allow the former to keep working if you want; otherwise I'll just document it.

Committed #29. Added a comment in CNR about short form options in drush_set_option.

I'm fine with not allowing short form to be used in code as in drush_set_option('root', $root);.

Remove tag