Better configuration management of solr server settings [#1145126]

My use case is while staging several sites on the same server which includes one multi core solr instance, but this is a problem also when pushing a site from staging to production when your default solr server url differs.

I like to think of the solr uri as a setting very specific to the server I'm on, much like the db credentials in settings.php. At the moment this url is store in the apachesolr_server table, and there is no way to alter these urls in apachesolr_load_all_servers() either. If these urls were in the variables table at least, settings.php would have a chance to set them.

Example: I have 2 instances of my site on the same server (testing.example.com and staging.example.com) and I want to use the same multicore solr server. If I deploy a snapshot of the database in each of these sites, they will both get the same solr server url settings (whatever is stored in the db). I would have to log into each site and set http://localhost:8983/solr/testing and http://localhost:8983/solr/staging as url settings respectively.

One work around is to store all the potential solr server settings in the db, and switch the default server id in settings.php, but that's not ideal.

Comment	File	Size	Author
#11	apachesolr-1145126.diff	1.23 KB	johnnycastrup
#11

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

pwolanin CreditAttribution: pwolanin commented 3 May 2011 at 00:39

Agreed - that seems less than ideal. However, I'm not sure what a good approach here would be. I guess we could store all the servers in the Drupal variables table instead of in a separate table, but that seems like a bit of a regression.

Maybe we should think of some way to rewrite the URLs based on $conf?

Comment #2

scor CreditAttribution: scor commented 3 May 2011 at 01:27

Title:	Strategy for deploying solr sites on multi core servers	» Better configuration management of solr server settings
Category:	support	» feature

Maybe we should think of some way to rewrite the URLs based on $conf?

What do you mean? override the value of the apachesolr_server table if they exist in $conf? that would be similar to having them in the variable table. I'm not sure to understand what you mean...

Comment #3

pwolanin CreditAttribution: pwolanin commented 3 May 2011 at 01:40

No, I mean essentially define a variable or $conf that's input to strtr or preg_replace for the URLs inside function apachesolr_load_all_environments()

e.g. it could look like: array('@/solr/(.+)@', '/solr/dev-$1')

Comment #4

scor CreditAttribution: scor commented 3 May 2011 at 02:09

if you're going to rewrite the url based on $conf, you might as well just use the url or settings specified in $conf and save a regexp on each page load. What's the advantage of a rewrite in comparison to an explicit override in settings.php?

$conf['apachesolr_server_custom'] = array(
  'solr' => array(
    'url' => 'http://localhost:8983/solr/mycustomcoreonthisserver',
  ),
);

edit: tweaked snippet.

Comment #5

scor CreditAttribution: scor commented 3 May 2011 at 02:17

ok, this data is cached, so ignore my regexp performance comment, but please explain the advantage of strtr as opposed to having the whole server settings defined in settings.php

Comment #6

pwolanin CreditAttribution: pwolanin commented 3 May 2011 at 02:20

If you try to define the whole thing, then you need to potentially track every setting that's added or changed?

I would suggest you skip the URL regex unless the variable is non-empty, or perhaps you could have such a pattern per env_id?

Comment #7

scor CreditAttribution: scor commented 3 May 2011 at 02:31

I'm thinking something like this while loading the servers settings

      if (!empty($GLOBALS['conf']['apachesolr_server_custom'][$id])) {
        $server = $GLOBALS['conf']['apachesolr_server_custom'][$id] + $server;
      }

If you try to define the whole thing, then you need to potentially track every setting that's added or changed?

whatever is set in settings.php takes precedence, you can set a url, or the facets. though the use case is url here, making it generic is probably easiest.

Comment #8

pwolanin CreditAttribution: pwolanin commented 3 May 2011 at 03:02

Does PHP actually do a recursive array merge with the + operator?

Comment #9

scor CreditAttribution: scor commented 3 May 2011 at 04:24

no, it does not, but that's fine for the use case of the url, though it would not work if you only wanted to override one element of [conf][apachesolr_enabled_facets], but not sure whether this is really necessary to be supported.

Comment #10

scor CreditAttribution: scor commented 6 May 2011 at 15:22

I wonder how other modules do similar things. The difference I guess is that most of the variables which are usually overidden by settings.php are flat arrays or strings. Here the server settings variable is a nested array. I don't think array_merge_recursive() will work because it creates an array if 2 strings are conflicting. maybe that's what drupal_array_merge_deep_array() is for?

Comment #11

johnnycastrup CreditAttribution: johnnycastrup commented 22 June 2011 at 18:52

File	Size
apachesolr-1145126.diff	1.23 KB

I use a system of applying different settings files per host environment (via a VirtualHost SetEnv directive), enabling local development, testing and production settings. This is reasonably functional for environment specific stuff that can go into $conf (and be read from there).

As far as I can tell from the apache_get_solr() method (apachesolr.module:1305), it could be pretty simple to merge stuff, since each of the host, port and path variables are handled separately.
I've attached a diff that demonstrates a possible and very (too?) simple implementation, and which works with the following included from settings.php:

     $conf['apachesolr_host']    = 'user:pwd@solr.example.com';
     $conf['apachesolr_port']    = '8983';
     $conf['apachesolr_path']    = '/solr/example';

It does have some issues, in particular with regard to transparency; The config settings will be loaded into the form on the solr admin page, and if you press save, they'll be applied to the database, which might be confusing.

But maybe this could be a way to go about it?