Support location based search

Changing status. Is there any reason that this shouldn't be committed?

i'm working on integrating geocluster with search_api_solr: #1800850: Use a solr plugin with search_api_solr for clustering

therefore i extended the patch a little, see attachment

I included the above patch in #1509380-8: Adopt common solrconfig.xml and schema.xml files for Apache Solr Search Integration and Search API Solr, where we also get the necessary schema fields for this to work.

The Search API Location module contains alter hooks, too, though, which we should just integrate into the Solr service class. The attached patch is a start for this, at least implementing the normal filtering. It is based on this comment in the Location project issue queue: #1798168-9: Major module modifications.
Also, you'll need to apply the above-mentioned patch for adopting the new config files before applying this one.

The patch from #3 was just committed. Thanks!

Here is a new patch including complete support. Only facets are still missing, but will probably be added tomorrow.
To be used with the new module version of search_api_location, published in this sandbox.

Patch #6 as it stands works for me. It clearly covers geofilt and I think geodist just fine.

You may stop here. The rest is me thinking aloud:-

I was wondering about how you could support bbox as well, and after looking into the future https://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4 and thinking about what other potential engines might implement. My idea was you could require the filter arguments to passed in Solr4 standard [which is basically common: lat,lon; lon lat (plus similar for boxes); WKT; and then a custom Circle() type]. Or more simply require WKT plus Circle().

For solr3 implementation then the arguments could be circle or approximated to the polygons bbox; or when Solr4 comes the polygons can be sent as is and the circle approximated. I'm assuming geoPHP here for getting centroid/bounds/...

What I don't know is if it makes for a confusing UX and a whole different set of filters would be needed to make it obvious the difference between the circle and approximated box Solr3 and polygon Solr4 implementations.
But then are lots of different filters confusing, each appearing dependent on your backend?

+++ b/service.incundefined
@@ -567,6 +567,63 @@ class SearchApiSolrService extends SearchApiAbstractService {
+        // Prepare the filter settings.
+        if (isset($spatial['radius'])) {
+          $radius = (float) $spatial['radius'];
+          $spatial_method = 'geofilt';
+          if (isset($spatial['method']) && in_array($spatial['method'], array('geofilt', 'bbox'))) {
+            $spatial_method = $spatial['method'];
+          }
+          $fq[] = "{!$spatial_method pt=$point sfield=$field d=$radius}";

ekes: note that while its a little hidden, the patch already seems to support bbox filter

still, i think it could use an api-doc like comment that explains the possible parameters being passed through $query->getOption('search_api_location').

patch works, waiting for facets :)

ekes: note that while its a little hidden, the patch already seems to support bbox filter

Embarrassing. I totally overlooked that last night.

Then in the immediate term this can be added to the filter, and embedded filter, as an option.

I'm still not convinced how this then works when it generalizes to other searches though. It's solr3 specific, and then solr3 specific options need to be added to the filters.

ekes, in #7 you have linked an article on extended search capabilities of solr 4. it also states

The existing spatial support introduced in Solr 3 is still present and is still the default used in Solr's example schema via the named "location" field type.

#1676224: Support Apache Solr 4.0 isn't really there yet and even then, i understand that the solr 3 specific implementation will still be functional. so i don't see any problem in that regards.

The lat/lon Pair index with Search API doesn't work , when i check the apache solr index i see in front of the latlonfield : Not indexed
May i change something in the schema ?
Any solution please

i just stumbled across what ekes mentioned in #7: solr 3 doesn't support a real bounding box filter. the patch in #6 correctly implements this, just in case somebody is wondering why a radius is needed for bbox :)
http://stackoverflow.com/questions/12653144/geospatial-search-within-a-b...

I have used solr 3 and in my view I used WKT to display the latlon pair , however when I add the 2 conditional filter (POINT and RADIUS ) nothing change , i can't expose the field for select the searching range.
Maybe i forgot something , may i have some help please ?

TY

for geocluster i have implemented a very simple the bbox filter, see my previous comment in #13:

the code basically is just a custom range query implemented in the preQuery method which can be implemented likewise using hook_search_api_solr_query_alter:

  list($left, $bottom, $right, $top) = explode(',', $_GET['bbox']);
  $fq = &$call_args['params']['fq'];
  $query = $field . ':[' . $bottom . ',' . $left . ' TO ' . $top . ',' . $right . ']';
  $fq[] = $query;

http://drupalcode.org/project/geocluster.git/blob/refs/heads/7.x-1.x:/mo...

Hello drunken monkey,

You mentioned this at #6 "Only facets are still missing, but will probably be added tomorrow." I don't see them in the commit logs. Are facets still on the plate?

Finally, here is the patch with facets. Sorry it took a while.

I ran into the problem that there doesn't seem a way to use Solr's geodist() function with a field name that needs escaping. Found out just now that it seems field names that aren't valid Java identifiers are discouraged. Not good, and sadly a bit late.
However, I was able to work around this using geofilt, it just means that we can't have facets like "in a distance of X to Y km", but only "nearer than X km". Doesn't really matter, though, I think. If a filter with a lower bound is passed, we silently ignore it for now. (Using fq=-{!geofilt …} doesn't work.) Maybe we'll find a solution some time later. (If someone knows of one, please shout!)

Everything else should work the same, so it should be enough if someone could test/review the facets.

Oh, and the “Add facet for missing values” isn't working for location fields, but I don't think it makes much sense there anyways.

Dear drunken monkey,

Patch applies fine for me.

I have not yet figured out how to get results. I tried:
1. Adding a Geofield LatLng pair exposed filter
2. Setting filter to operator to "="
3. Inputting lat long value like: "47.268048,19.154663"

The view returns as empty.
I tested this point separately on solr admin and it returned results, yet when doing via Search API Views filter it shows empty results.
(for example mysolrip.com/solr/select/?q=&fl=*&fq={!geofilt%20sfield=locm_field_item_all_locations:field_r_geofield:latlon%20pt=47.268048,19.154663%20d=1000})
Could you share how you use this at the moment?

PS: Ouch, I just realized I need to enable Search API Location Views module... Awesomeness so far. Still testing.

I think $spatial_method has to be set always and not only if a radius was set.
This because later $spatial_method is used, independent from radius, to create $facet_params['facet.query'].

@ giorgio79: Oh, sorry I didn't make that clear enough. I'm glad you found it out and it seems to work for you.

@ das-peter: Ah, you're right, thanks for catching that! Is everything else alright with the patch, is it RTBC in your opinion?

@drunken monkey Frankly speaking I implemented it without doing detailed review ;) Visually it looks good and besides the already mentioned thing I couldn't find any functional issues while using it.
However, I'm unsure about the static values here:

+            $step = (isset($radius) ? $radius : 100) / 5;
+            for ($k = 1; $k < 5; $k++) {

I think it would be nice if the number of steps, the step-size and the "non-radius-fallback" value would be configurable.

Hm, makes sense. I guess the number of steps we can easily (and should) get from the facet's limit setting. That's what it's there for, after all. See attached patch for that. (It uses one item less than the current one, though, because the radius itself isn't included. I don't really know what would make the most sense in that regard. We also have no easy way of returning an active facet again (unless we want to return the no-op active radius in other circumstances, too – and use an additional facet query for it).)

Using a hard-coded value as the default upper bound is, of course, also far from ideal. However, I can't really tell how we could let the user configure that (other than with a hidden variable, of course). Where should the configuration be, and how would we get it there from the Solr module?

Oops, attached the half-finished patch by accident.

+++ b/service.inc
@@ -567,6 +567,83 @@ class SearchApiSolrService extends SearchApiAbstractService {
+            $step = (isset($radius) ? $radius : 100) / $steps;
+            for ($k = $steps - 1; $k > 0; --$k) {

I don't think $steps - 1 is necessary / right. This will exclude the last step. E.g. if I have a radius of 1000km and 50 facets, the facets will only cover up to 980km and not the full range of 1000km.

I don't think $steps - 1 is necessary / right. This will exclude the last step. E.g. if I have a radius of 1000km and 50 facets, the facets will only cover up to 980km and not the full range of 1000km.

I know. But what good would providing a filter for "1000 km" be, if you're already filtering for that radius? See also my comment in #23:

See attached patch for that. (It uses one item less than the current one, though, because the radius itself isn't included. I don't really know what would make the most sense in that regard. We also have no easy way of returning an active facet again (unless we want to return the no-op active radius in other circumstances, too – and use an additional facet query for it).)

Ah sorry, I over-read this. However, it's less the logic that bothers me it's about the feel. Somehow it's odd not to have that facet - but maybe that's just me :P

About the count queries, here I've again an odd feeling.
As far as I get it the "higher" facets also contain the lower result count as well.
Means 1 result for 75km, 1 result for 150km will lead to facets like:
<50km (0)
<100km (1)
<200km (2)

I think it would be really nice to have (additionally) something like the module provided before:
0 - 50km (0)
50 - 100km (1)
100 - 200km (1)

I think the later facet approach is the one also defined here: http://wiki.apache.org/solr/SpatialSearch#How_to_facet_by_distance

It looks like we could use a dedicated facet api "display widget" for spatial facets. Using one of the default display widgets will use sensitive defaults - but the dedicated widget will allow you to configure more details.
I know it's not ideal but the most generic approach I can think of atm.

I think it would be really nice to have (additionally) something like the module provided before:

…

I think the later facet approach is the one also defined here: http://wiki.apache.org/solr/SpatialSearch#How_to_facet_by_distance

Yes, it is, but we can't use that style of Solr query anyways (see #17). We could of course easily convert the facet ranges to that style on the Drupal side, not even specific to Solr (e.g., even in a dedicated Facet API plugin, as you mention).
When we get a way to leave this option to the user, it's of course a win. But in my opinion, this has far less use than the first style, implemented at the moment. Usually I want to find things inside a certain area around a point, and not things that are not too close and not too far. At least from what I can think of.

A dedicated display widget, or query type, could definitely be a good idea, yes. At the least we could append " km" to the values then. Or give the users the option how the ranges should be stacked, as you mention above.

A while ago I tried getting a distance facet module going, but did not get very far. I made you drunken monkey and das-peter comaintainers. I am happy to transfer the entire project :) : http://drupal.org/sandbox/giorgio79/1889026

Thanks, giorgio79! Good to know there's existing code for that. And if the widget works for other search modules, too, all the better.
In that case, it would probably really make sense to leave this in a separate project, and maybe link to it from this one once it's un-sandboxed.

The maintainer list of the project is a bit frightening, though. If one or two out of the list are really interested in implementing this, probably they should take over. I myself would prefer it if someone else would take that project – I have too little time already for my existing ones.

Yep, the list is definitely big. I offered the module so far to everyone that matters today in Drupal geo :D I am happy for the new maintainer to trim the list, noone has taken it so far.

Attached is a re-roll, according to #1846254: Remove the SolrPhpClient dependency.
Anyone want to test/review still, or provide additional comments, or should I just commit?

Let it roll :)

I just gave it a rough visual review and have nothing to nag about ;)
And from my point of view it's more important to have at least the basic functionality than figuring out all the little details that could pop up.
E.g. how the facet should work and so on is a discussion for UX people and shouldn't stop this.

OK, thanks for your repeated feedback!
Committed.

How about:

Filtering by an arbitrary rectangle

Sometimes the spatial search requirement calls for finding everything in a rectangular area, such as the area covered by a map the user is looking at. For this case, geofilt and bbox won't cut it. This is somewhat of a trick, but you can use Solr's range query syntax for this by supplying the lower-left corner as the start of the range and the upper-right corner as the end of the range. Here's an example: &q=*:*&fq=store:[45,-94 TO 46,-93]. LatLonType does not support rectangles that cross the dateline, but RPT does. If you are using RPT with non-geospatial coordinates (geo="false") then you must quote the points due to the space, e.g. "x y".