Solr 3.4 Grouping and Field Collapsing

Are you sure about solr 3.5? This is a branch that is not even released yet? Would it be possible to use solr 3.4 and be willing to create a patch that would allow grouped results in solr to be shown in drupal correctly?

Thanks!

Could you also post the code here for other people that might be interested in this feature? A bit more explanation would be appreciated! :-)

Well, nothing is really fixed until it appears in the module ;-)

Would you like to be our guinnea pig and make a patch for this functionality (for our D7 version) so people can easily apply and review your changes? Say for example that you add a configuration option to a custom search page where you mention the results have to be grouped (so you add the group parameter to it) and you also make those changes to the library.

It might not be that easy to actually commit this, but at least we make the life of a developer a bit easier!

#927542: Field Collapsing now in Solr 4.0 dev has interesting information about this.

Hi,
I just started to use this mechanism. What I just did, is modifying apachesolr_do_query() so it reforms the result from using group = true, without group.main to the default result form. Additionally to make this useful, I add the groupId and groupValue to the $results. And of course in a custom module I add the needed search parameters. This results in having the groupId + groupValue available in the search-result.php template, so I can do magic via Javascript.
Now, is this an approach worth generalizing?

The attached screenshot shows, what is working at the customer now. As mentioned above, this required me to modify apachesolr.module.php apachesolr_do_query(). Using group.main=true to receive results in the usual form, seems to prevent identifying more then result inside a group.
I would like to work on this further, but I don't know, if this approach is worth to follow on.

First of all, please go for a modular approach first and see if you can make it work using a custom module. (sandbox?) Once that works we can see if we can incorporate this change back in to the main branch. Would you be able to do this?

Other that that it looks like you are on the right track, please show us some code in a sandbox ;-)

How about implementing a hook to preprocess the response before apachesolr_search_process_response() does it's thing. That way we can handle $response->grouped which has all the results just not in the structure expected by apachesolr_search_process_response().

function apachesolr_search_process_response($response, DrupalSolrQueryInterface $query) {
  
  // Hook to allow modifications of the response
  foreach (module_implements('apachesolr_preprocess_response') as $module) {
    $function = $module . '_apachesolr_preprocess_response';
    $function($response, $query);
  }

Funny, I was about to propose the same thing, after rewriting my current code accordingly :)

However: apachesolr_search_process_response() is too late, because the response gets already cached in apachesolr_do_query() and all the blocks and external modules will use the response from their.
I recommend implementing the hook_apachesolr_preprocess_response() in apachesolr_do_query() right before the apachesolr_static_response_cache() call.

I am still planing to do this in a proposal, probably later this week/next week.

Having re-read the README there is the option to provide your own query class via the variable apachesolr_query_class. I'm investigating that route for manipulating the response in the query object itself, by extending SolrBaseQuery and implementing the search function.

FYI, this is how I'm using grouping by implementing my own query class that extends SolrBaseQuery.
Bear in mind this is a very bespoke solution that needs all sorts of manipulation with hook_apachesolr_query_prepare() & hook_apachesolr_process_results(). You probably don't want your query class to do this but the point is to show an easy place to manipulate the response without patching the apachesolr modules.

// Tell apachesolr to use my class
  variable_set(
    'apachesolr_query_class', 
    array(
      'file' => 'Mymodule_Search_Solr_Base_Query',
      'module' => 'mymodule',
      'class' => 'MymoduleSearchSolrBaseQuery'
    )
  ); 

contents of Mymodule_Search_Solr_Base_Query.php

class MymoduleSearchSolrBaseQuery extends SolrBaseQuery implements DrupalSolrQueryInterface {

  public function search($keys = NULL) {
    if ($this->abort_search) {
      return NULL;
    }
    $response = $this->solr->search($keys, $this->getSolrParams());
    
    if ($this->getParam('group')) {
      $group_field = $this->getParam('group.field');

      // Structure things how apachesolr_search_process_response() expects them
      // I'm only interested in one group, though multiple are possible
      $response->response->numFound = count($response->grouped->{$group_field[0]}->groups);
    
      foreach ($response->grouped->{$group_field[0]}->groups as $i => $obj) {
        foreach ($obj->doclist->docs as $doc) {
          $response->response->docs[] = $doc;
        }
      }
      
    }
    
    return $response;
  }
}

Very good. Works like a charm without touching apachesolr core code. Really nice work.

Where do you set the variable?

I set the variable on module install/update.

Hey,

took me a while, but here is a first shot:

http://drupal.org/sandbox/ChristianAdamski/1623610

It should already work, based on a new install with only 3 articles / ApacheSolr 3.6
I'm also adding the current state as an attachment.
Just download the module and enable it, should work. You can set the field to group by in the admin area.

This is my first Drupal module. Should there be interest & feedback, I will try to apply it.

Let's continue this in the apachesolr sort module

You can install my grouping module and the sort module parallel to each other. They don't conflict, but they also don't do much in combination. It seems to me the grouping module might be best fit as a separate module?

I've send my sandbox project through Coder and PHP_CodeSniffer and adjusted the code accordingly.

Should I apply to publish it as a module on its own or move the other way and join it into apachesolr_sort code?

I would be sorry to just abandon the code.

I'd love to join this in this module, so if you can start a patch perhaps?
We have facilitated this issue #1700472: Make 'apachesolr_search_process_response_callback' a search environment variable so you can now hook in to the process function to massage the results coming from solr.

This should make it much easier to have the grouping per environment.

Now without custom query class :)

I'm still not able to have pagination with the current patch, but grouping works already + first page is grouped. Custom theming is necessary though.
Thanks to ChristianAdamski for giving us the first start. I did tweak it and I am still tweaking it so it is a hard work in progress.

The following patch adds perfect pagination + some configurable values for the grouping

How to use :

Apply Patch
Enable the module
Go to your environment edits page
Select : Enable grouping
Select how many results per group you want + the field you want to group on.

Now with valid patch

Cleaner patch :)

Last one, I guess I am committing this patch and hope for a follow-up.

Feel free to improve the javascript bit.

@ChristianAdamski : I ripped out most of the accordeon code, because it was too project specific. I am adding the group name above each group now.

Committed.

If someone want to make a backport, feel free.

So just for understanding: does this include my patch grouping stuff and this is done?

This includes a lot of your initial patch. If you want to make follow-ups, please create a new issue so we can improve that step by step.

Thanks again for getting the ball rolling on this one.

Well, what do you want improved?