Autocomplete in search box powered by Solr

If you have totally public content, there is an easier way to do auto-complete. See: http://drupal.org/node/375341

A good start, but I'm not sure that is suitable for general use since AFAIK, you can never have access controls work with this.

The Solr path being hard-coded in the PHP is a problem - why don't you jsut use the current path and append '/autocomplete'?

Also, why do you search for 'solr'?

see also:

http://www.mattweber.org/2009/05/02/solr-autosuggest-with-termscomponent...

http://wiki.apache.org/solr/TermsComponent

Use drupal_json instead.

+function apachesolr_autocomplete_callback($query = '') {
+  $suggestions = apachesolr_autocomplete_get_suggestions($query);
+  print drupal_json($suggestions);
+}

I'd get rid of the define and hardcode the path into the form. It doesn't make it any easier to change if it's a define, and it makes it harder to read.

We don't need a variable for this
+ // Set parameters for partial term query
+ $autocomplete_field_name = 'spell';

You can just make it
+ $matches = $response->facet_counts->facet_fields->spell

We can get rid of $html
+ $html = theme('apachesolr_autocomplete_highlight', $keys, $suggestion, $count);
+ $suggestions[$suggestion] = $html;

I'm not sure what I think of the hook. What would you do with the node title? Search for the whole title?

+ * Sample implementation of hook_apachesolr_autocomplete_suggestions().
+ * This function gets nodetitle suggestions from the partial query.
+ * @param keys The user-entered query
+ */
+function _nodetitles_apachesolr_autocomplete_suggestions($keys, $suggestions_to_return = 5) {

Why is the function _nodetitles_apachesolr? I admit I'm not a fan of _function notation.

Also (and I know this is really nit-picky), please don't keep copying the code comments on these lines. In fact... it'd be nice if we could improve them so they make more sense =)

+  // Cache the built query. Since all the built queries go through
+  // this process, all the hook_invocations will happen later
+  apachesolr_current_query($query);

Probably also want to handle form ID "search_theme_form"

Could you explain this limitation to me?

+  // Don't handle more than one word right now (creates false expectations)
+  #if ($first_part != "") {
+  #  return array();
+  #}
+  if ($last_part == "") {
+    return array();
+  }

Thanks!

Is this part needed?

+    $form["basic"]["inline"]['keys']["#size"] = 60;

I guess it's because we want extra space? But won't it mess with people's themes?

Really nice feature. Congrats.

I think it's fair to put some things like that into the $_SESSION object. That's what I'm doing to make autocompletes work right in a different setting on a client project. Every time you submit the form save the state of that checkbox, then it's there for the autocomplete to call on.

The menu handler doesn't need a title:

+  $items['apachesolr_autocomplete/autocomplete'] = array(
+    'title' => t('Apache Solr Autocomplete'),

Nor does it need page arguments:

+    'page arguments' => array(),

drupal_json does the printing for you:

+  print drupal_json($suggestions);
// becomes
+  drupal_json($suggestions);

I'm still skeptical of the hook: + $results = module_invoke_all('apachesolr_autocomplete_suggestions', $query);

We could reduce these three functions to one:

+/**
+ * Callback for url apachesolr_autocomplete/autocomplete
+ * @param query The user-entered query
+ */
+function apachesolr_autocomplete_callback($query = '') {
+  $suggestions = apachesolr_autocomplete_get_suggestions($query);
+  print drupal_json($suggestions);
+}
+
+function apachesolr_autocomplete_get_suggestions($query) {
+  $results = array();
+  $results = module_invoke_all('apachesolr_autocomplete_suggestions', $query);
+  if (!is_array($results)) {
+    return false;
+  }
+  return $results;
+}
+/** 
+ * Implementation of hook_apachesolr_autocomplete_suggestions().
+ *
+ * @param $keys 
+ *   The user-entered query.
+ */
+function apachesolr_autocomplete_apachesolr_autocomplete_suggestions($keys) {
+  $results = array();
+  $results = array_merge($results, _apachesolr_autocomplete_suggest_word_completion($keys, 5));
+  $results = array_merge($results, _apachesolr_autocomplete_suggest_additional_term($keys, 5));
+  return $results;
+}

becomes

+/**
+ * Callback for url apachesolr_autocomplete/autocomplete
+ * @param query The user-entered query
+ */
+function apachesolr_autocomplete_callback($query = '') {
+  $results = array();
+  $results = array_merge($results, _apachesolr_autocomplete_suggest_word_completion($keys, 5));
+  $results = array_merge($results, _apachesolr_autocomplete_suggest_additional_term($keys, 5));
+  drupal_json($results);
+}

Or if you think that getting the suggestions is somehow a useful API function for others, then two functions: one to get the suggestions, and the callback to drupal_json them. But the hook seems like overkill to me (unless you can explain some compelling use cases).

Here are my comments on this patch as it stands

- Remove inline styles from theme functions
- static cache copy and paste error:

// Cache the built query. Since all the built queries go through
+  // this process, all the hook_invocations will happen later
+  apachesolr_current_query($query);

AND

$query = apachesolr_drupal_query($keys, '', '', '');

Reading this patch (I haven't ran it locally yet) seems that the autocomplete doesn't work with phrases. Meaning it only auto completes for a given word.

I would suggest instead, leveraging TermsCompent* http://wiki.apache.org/solr/TermsComponent

O, and I would suggest looking for a Javascript only solution. There are solid JS to Solr examples. It avoids bootstrapping Drupal which is a huge boon. You don't need Drupal expect for the apachesolr_nodeaccess. So it would be possible to put the data you need for the node access* into JS variables and that would allow you to query Solr from JS and avoiding Drupal.

So if I was writing this patch, I would look to leverage the TermsComponent handler and I would work hard for a JS only solution, avoid Drupal bootstrap.

*Edit: Ok i get why your not using TermsComponent. Node access. But your not returning nodes, your returning term matches. So not sure node_access is important here. It would suck to have the autocomplete suggest a term and it no match anything you have access too.

I get why we would want to bypass Drupal but I don't think it's realistic. The assumption of the module so far is that Solr is a protected resource and Drupal is its gatekeeper. I'd not like to introduce the exception to that here.

I'm working on this patch and have made serious changes (and it's looking good!) but I'm tired and going to bed. I'll post my updates tomorrow.

We can change mm for this case. I know how to do that, I'll put it in the merged patch tomorrow.

Some changes I've made:

- I'm either sending it to term completion (if the last keystroke was a character), or to further suggested terms (if the last keystroke was a space). I like this much better than always merging the two. When you're typing a word you want word completion suggestions, and when you type a space you want guidance for the next word.
- I've overhauled the algorithm for word suggestions introducing a naive scoring and some stop word detection.

I've merged our two patches, more or less. We've both started looking at scoring algorithms to find the most "interesting" of the word suggestions. I also renamed some functions. The most important differences I'll post here for you to think about. My version of the patch isn't really done yet as I'm not happy with the results.

Here you can see that I stopped combining 5 suggestions with 5 terms, and made it an either/or thing based on the last character. I like this much better and suggest we both adopt it going forward.

/**
 * Callback for url apachesolr_autocomplete/autocomplete
 * @param query The user-entered query
 */
function apachesolr_autocomplete_callback($keys = '') {
  $results = array();
  $keys = strtolower($keys);
  $last_char = substr($keys, strlen($keys) - 1);
  if ($last_char == ' ') {
    $results = array_merge($results, apachesolr_autocomplete_additional_term($keys, 10));
  }
  else {
    $results = array_merge($results, apachesolr_autocomplete_word_completion($keys, 10));
  }
  drupal_json($results);
}

The other major changes I've been playing with are in the term suggestions. I'll post here for your inspiration, but I'm not happy with the results yet.

function apachesolr_autocomplete_additional_term($keys, $suggestions_to_return = 10) {
  $keys = trim($keys);
  if ($keys == "") {
    return array();
  }
  // We need the keys array to make sure we don't suggest words that are already
  // in the search terms.
  $keys_array = explode(' ', $keys);
  $keys_array = array_filter($keys_array);

  $query = apachesolr_drupal_query($keys, '', '', '');

  // Set parameters for partial term query
  $params = array(
    'facet' => 'true',
    'facet.field' => array('spell'),
    'facet.limit' => $suggestions_to_return,
    'facet.mincount' => 1,
    'start' => 0,
    'rows' => 0,
  );
  
  // @janusman - this line adds lots of other facets.
  //apachesolr_search_add_facet_params($params, $query);
  apachesolr_current_query($query);

  // Allow other modules to modify query.
  apachesolr_modify_query($query, $params, 'apachesolr_autocomplete');
  if (!$query) {
    return array();
  }

  // Try to contact Solr.
  try {
    $solr = apachesolr_get_solr();
  }
  catch (Exception $e) {
    watchdog('Apache Solr', $e->getMessage(), NULL, WATCHDOG_ERROR);
    return array();
  }

  // Query Solr
  $response = $solr->search($keys, $params['start'], $params['rows'], $params);
  $suggestions = array();
  foreach ($params['facet.field'] as $field) {
    foreach ($response->facet_counts->facet_fields->{$field} as $terms => $count) {
      // THIS DOESN"T SPLIT!!!!
      $term = preg_replace('/[_-]+/', ' ', $term);
      foreach (explode(' ', $terms)  as $term) {
        if ($term = trim(preg_replace('/['. PREG_CLASS_SEARCH_EXCLUDE .']+/u', '', $term))) {
          $matches[$term] += $count;
        }
      }
    }
  }
  $matches_clone = $matches;
  $stopwords = array('and', 'are', 'but', 'for', 'into', 'not', 'such', 'that', 'the', 'their', 'then', 'there', 'these', 'they', 'this', 'was', 'will', 'with');
  foreach ($matches_clone as $term => $count) {
    if ((strlen($term) > 3) && !in_array($term, $stopwords) && !array_search($term, $keys_array)) {
      
      // @janusman this is another way to perhaps fish out the interesting terms. See how I sort later
      // Longer strings get higher ratings.
      $matches_clone[$term] += strlen($term);
    }
    else {
      unset($matches_clone[$term]);
      unset($matches[$term]);
    }
  }
  // The $count in this array is actually a score. We want the highest ones first.
  arsort($matches_clone);
  // Shorten the array to the right ones.
  $matches_clone = array_slice($matches_clone, 0, $suggestions_to_return, TRUE);

  // One more array for the final results
  $matches_final = array();
  foreach ($matches_clone as $term => $count) {
    // Get the real counts back
    $matches_final[$term] = $matches[$term];
  }
  // Sort the results by the real counts
  arsort($matches_final);

  // Build array of suggestions
  foreach ($matches_final as $term => $count) {
    $suggestion = $keys . ' ' . $term;
    $suggestions[$suggestion] = theme('apachesolr_autocomplete_highlight', $keys, $suggestion, $count);
  }
  return $suggestions;
}

Also note that I removed "results" from the count display due to real-estate concerns. I think it takes up too much space.

function theme_apachesolr_autocomplete_highlight($keys, $suggestion, $count = 0) {
  $html = '';
  $html .= '<div class="apachesolr_autocomplete suggestion"><strong>' . substr($suggestion, 0, strlen($keys)) . '</strong>' . substr($suggestion, strlen($keys)) . '</div>';
  if ($count) {
    $html .= "<div class='apachesolr_autocomplete message' style='float:right'>$count</div><br style='clear:both'>";
  }
  return $html;
}

Some stuff to look at for inspiration http://dzineblog.com/2009/07/20-ready-to-use-auto-completion-scripts.html

errr... If I intended to, I didn't, and I've wiped the install since then :( Fortunately it looks like I recorded the salient bits in the comment.

For mm, this is sample code that I have in modify_query in a custom module:

  // xor can be "any" or "all", as in results must have _any_ or the keywords, or _all_ of the keywords. ie, OR/AND
  if (isset($values['xor'])) {
    $params['mm'] = ($values['xor'] == 'any') ? '1' : '100%';
  }

The xor flag is a switch between AND and OR. If mm = 1 than it is an OR query. If it equals 100% is is an AND query.

I really have a need to have this autocomplete feature for a project. I'm new to Solr, but I'm willing to help this along as best I can.

I've downloaded 6.x-2.x-dev, but I don't see this code in there, and it looks like there has been a lot of discussion since the last posted patch in this thread. Is there somewhere I can get the most current code for this?

Thanks.

Wow. Nice feature. Subscribing.

I just did the same thing and have been testing it over the past hour. It rocks! This is exactly what we needed. I'll have a bunch of time to work on this next week if needed, so let me know what you think needs to be improved.

To answer your question, go ahead and close this one, and I'll grab what you have from the new project page.

janusman, yeah, I'll close it for now. Your new module will be the center of activity on this feature now. Great work!

FIXED: See the dedicated module for this feature: http://drupal.org/project/apachesolr_autocomplete