Search API integration

It is certainly a possibility. However, honestly, search integration is pretty much the last thing on my list right now (geocoding, mapping, & geoprocessing all take priority for me).

If you would like to jump in and build this I'm certainly willing to review and commit it, but don't expect independent movement on this for a while.

Let's be realistic here...

definitely tracking that :).

For proximity search with solr you need to use localsolr http://www.gissearch.com/localsolr

The http://drupal.org/project/localsolr is certainly a good starting point. The solr part is very easy and consist simply to something like:

hook_add_to_index(&$doc,$node){
  $doc->lat = $node->geofield['latitude'];
  $doc->lng = $node->geofield['longitude'];
}

I might be interested to working on that and I already did it for geo+apachesolr but sadly the code was to "special" to make a contribution. I need to understand more Search API but it look very nice and well made. I hop it can be as a extension of Search API. Shadlington you will be interested of working on it with me? We can speak about it on IRC...

geofield + solr with localsolr + search API + views + geocode + OpenLayers = my dream from years!

Yeah, I think it should be pretty easy to get a solr LatLngType working from geofield. I think the latest version of solr also supports storing polygons and lines, which would be nice to integrate, but might be something to work on once we have Lat/Lng support

+1 here offering help. I'm no programmer- but maybe I could test? Please let me know how I can help.

Subscribing.

Subscribing

This is the first step #1153480: geofield_field_info should set a property_type

@phayes I would be happy to assist here. I had created my own geo-tagging field module and integrated it with Search API already. I haven't released the code as there are now many geo field modules for d7 and I didn't want to add another.

Anyway, if you want I would be happy to work on this. If so please add me as a co-maintainer and I can start submitting some patches and checking them in.

Cheers,

David

Ah, if David has already done this, then maybe my expertise isn't that necessary anymore. How did you solve it, in a Solr-specific way?

Important to know, in any case, would be what use cases this should in the end support? Once there is Entity API integration for the field, the data could in principle be indexed. As you noted, there is sadly no general support for adding new field types (and I really can't think of any way to do it). But doing it in a Solr-specific way shouldn't be too hard. Is there good documentation on Solr's location fields?
Then, should this already support proximity search? This would then probably need a processor, and/or maybe a data alteration to add this as a separate field.

However, without knowing the internals of this module and how Solr works, I can't really say how to do this exaclty. However, maybe David can help better anyways.

The documentation for Solr spatial search http://wiki.apache.org/solr/SpatialSearch

david, I've added you as a maintainer

@phayes Great, thanks for that. I will start working on that this evening. What is your preferred method of me pushing up these initial changes? A new branch or just as patches for now? Then when you are happy I can commit them?

Please let me know what works best.

Cheers,

David

A branch would be best. Thanks for asking!

Just letting everyone know that I have created a new branch and pushed up my changes. The branch is: 7.x-1.x-entity-integration. I have tested this with lat / lon values only. Will have to add some converters to take other values and turn them into lat / lon for solr ingestion.

This now exposes Geofield to the Entity API. In the case of Search API you can now select Geofield as a field type and it will index with your nodes.

There is more work to do with use-cases and the like, but it is a good start.

Cheers,

David

Subscribing

geofield.module line #383: $property['property info'] = field_geo_field_data_property_info();

not finding the "field_geo_field_data_property_info()" function in the changes you pushed up, however according to: http://drupal.org/node/1156554 if i replace that function call with: geofield_field_info() all works fine, and the field shows up as selectable in an index for the search api. Haven't tested past this, will comment back with results.

@camidoo - thanks for finding that error. Sorry for the problem, I was doing testing with another module that had that code so I didn't catch the error.

In the next hour or so I will update the code and push it back into the branch.

Cheers,

David

@camidoo - Fixed: http://drupalcode.org/project/geofield.git/commit/4854e3f

Thanks again.

Great work guys! Does this work with search api's built in database indexing?

@mattsmith3 Yes, since this is on the entity level search_api will index it and it will be useable by any search_api backend. Sorry for taking so long to reply, I am on holidays so not manning the computer as much as normal :)

Cheers,

David

Thanks for the work on this. With this branch I can see my Geofield fields in the Search API Index Field configuration interface but have no way to store the data as a LatLonType in Solr. Does anyone know if there's any work being done on that front? It's my understanding that the location data needs to be stored in a LatLonType field in order to do any geographic querying of the data.

So exactly what format do you need it to be stored within solr? Can you point to an API doc or blog post or something? i would be happy to chase up from there how it can be done.

Cheers,

David

From the docs about LatLonType: "Values for this type are of the form latitude,longitude, although behind the scenes, the latitude and longitude are indexed as separate numbers. "

More details here: http://wiki.apache.org/solr/SpatialSearch#LatLonType

You must edit your schema.xml to add support for this type. I haven't dug into it deeply yet so I don't know how possible this is but we'll also need to add a new field type to the Search API (called something like "geo point"?).

I haven't dug into it deeply yet so I don't know how possible this is but we'll also need to add a new field type to the Search API (called something like "geo point"?).

I can assure you that this is, sadly, quite impossible. The data types used in the Search API have to be hard-coded (at least I couldn't think of any other way to do this) as all service classes (= backend plugins) have to be aware of all available data types, so they can store them properly.
You'll have to use a Solr-specific solution, I'm afraid.

drunken monkey,

Thanks for you response. Couldn't we do something cheaty like map "geo-point" to LatLon in solr, but to a text-field in all other backends?

No, I don't think so. You'll really have to give it the type "string" and then somehow detect in the Solr backend that this is really a lat/lon field.
Or, rather, in indexing you maybe don't need to do anything at all – just explicitly define the field, e.g., "ss_field_location", and give it the correct LatLon type. Then you'll just have to take this into account when searching (at which point the service class has to know what fields are geolocations, admittedly).

Back when designing the Search API, allowing custom field types with default fallbacks would have been a possible solution, though. It's just too late for that now, I'm afraid.

Thanks for the definitive response, drunken monkey. Clearly I'm going to need to think this through more carefully.

Interesting... subscribing :)

Subscribe.

There is now a Search API issue for adding custom data types after all: #1260834: Add a way to define custom data types.

Also, mollux managed to get spatial searches working with Search API Solr—don't know whether he used this module, though. He'll post the code soon, though. In any case, it will provide a guideline on how to do this—the initial results look great.

I'd love to see mollux's solution. In the interim I'm considering writing a module that provides a data alteration callback or a processor along with an alternative schema.xml to allow you to map geofields to solr LatLonType fields.

hello,
I'm still cleaning up the code, I'm almost finished, and hope to put it on line in a couple of days.
You can see a demo on http://spatial.mollux.be, you can login with demo/demo to add your own nodes with an address.

greetings,
Mollux

@mollux: the demo looks awesome!
@drunken monkey & davidseth: I'm currently using the Locations module for Lat/Lon of nodes & Feeds to import my data from a CSV. Would it be relatively straightforward to migrate to Geofield?
Also, I have several custom Select boxes in my current search. Is there an ability in your module to accommodate custom search form elements?
Thanks.

Also, I have several custom Select boxes in my current search. Is there an ability in your module to accommodate custom search form elements?

How it's done depends on the type of search you use, but in principle yes.

@mollux very nice! I don't hide my impatience to download the code and try on my server ;).

What mollux posted for a demo is exactly what I need.....Sub.

I've got a solution, too, but it requires hacking search_api_solr because the framework for adding conditions to queries isn't flexible enough to accommodate spatial queries. The other alternative is to implement hook_search_api_solr_query_alter but then you're designing to very specific use cases rather than properly leveraging the framework.

Unless I'm missing something the only real solution to this will come when #1260834: Add a way to define custom data types is resolved, and that's not a trivial task.

I just commited the code :)
you can find here: http://drupal.org/sandbox/mollux/1269950

Thanks for the code, mollux. Looks pretty good and in many ways remarkably similar to what I built. You were probably smart to implement hook_search_api_solr_query_alter() rather than hacking the search_api_solr module.

Anyone trying to use this should note that they'll need to mark their geofield fields as type "location" either by implementing hook_search_api_index_load() or by hacking their search_api module to add "location" as a field type. You'll also need to edit your schema.xml to add support for LatLonType fields.

With all that said, I haven't quite gotten your module to work yet, mollux. I'm very close but I'm trying to use it in an openlayers view and I think the js in your module is giving me trouble.

Regardless, my own code works great for my needs (I'm solely implementing a views filter) so I think I'll go ahead and polish it up and stick it in a sandbox on here in case it helps someone.

@mollux - Great stuff. Thanks, cloning now.

@azinck - Looking forward to seeing your repo to compare / contrast. Thanks.

Ok, my code is up here: http://drupal.org/sandbox/azinck/1288258

@mollux: I switched to your approach of using hook_search_api_solr_query_alter() rather than my previous approach of hacking the search_api_solr module. Much improved.

I've provided a schema.xml with the necessary changes and README.txt contains some instructions on how to implement the module. I hope it helps someone.

@azinck - would it help if I added 'LatLonType' as the datatype for geofield? I have repo access and created the '7.x-1.x-entity-integration' branch. Or how can we do this so that people can just use the '7.x-1.x-entity-integration' branch and your module without any other glue?

Cheers,

David

@davidseth -- I don't think it's something Geofield itself should be responsible for.

If we end up developing a module out of this we could have that module detect which fields are geofield fields and automatically change their Search API field types to LatLonType in hook_search_api_index_load() but it'll throw a PHP notice when viewing the Search API fields interface (and possibly elsewhere). That problem will persist until #1260834: Add a way to define custom data types is resolved because the data types that Search API supports are hard-coded in the Search API module.

Probably the best thing to do in the short term is provide an interface (possibly leveraging Search API's data alteration callback infrastructure) to choose which fields to make LatLonType so that we don't force that type on all Geofields for all Search API indexes. That would be pretty trivial to write. You'd still get the PHP notices but at least you wouldn't have to dig into code to wire things up.

Ok, I've updated the module to do what I described in my previous comment. You can now choose which Geofields you want to store in Solr as a LatLonType in your Search API Index's Data Alteration Callback section. I'm kind of cheating by putting it here since it's not really a data alteration but it's a convenient place to hang index-specific configuration in the GUI. Maybe it would be possible to move it to the Index's settings page? I haven't looked into it.

Note that you will get PHP notices thrown in your Fields configuration screen after enabling the Data Alteration Callback.

You can find the module here:
http://drupal.org/sandbox/azinck/1288258

As I'm eager to release a stable as soon as possible, I'm pretty sure #1260834: Add a way to define custom data types will soon (read: in the next week) be fixed. It's actually not such a big change, I think.
We could then move the whole Solr-specific code into the Solr service class itself, and also provide the modified schema.xml as an alternative in the module itself. (We can't make it the only schema.xml, as it won't work with older Solr versions.)

@ davidseth: No, you can't just use a new type in the Entity API integration, as Entity API won't recognize it. You'll have to use "text" or "token" for the property.

I know David create a branch with entity integration on #18.

But this is not only linked with Search API, perhaps we can review that on #1285176: Property Info callbacks -- Add field metadata (there is an other patch also) and commit in the main branch.

@azinck
What version of Solr do you use?
I had 1.4 which gave me an exception that the class LatLonType needed is missing.
I installed 3.4 but I cannot get the Geofield to get indexed as location type field. It always says "Field Type: ignored".
So the views filter doesn't really do anything.... :(

(And I am pretty sure I have done everything else right, dev for search api, geofield entity-integration, Geocode for drupal 7, your version of geofield proximity filter.)

@matrixlord: I'm using 3.3 and haven't tested on 3.4. You're using my version of the schema, right?

@azinck. Still no luck with 3.3. I am using your schema.xml.
I am going to see the details once again.
I see in the system output of solr that it passes the correct point and radius. So it must be an indexing problem.

One last question. When you select the GeoField field at the search api on the fields page, you choose string, right? (After that, you enable from the workflow to index it as Solr LatLonType)

Thanks for your time

When you go to the workflow page and activate "Index Geofields as Solr LatLonType" are you then scrolling down to the callback settings and choosing your geofield in the list? I did this to allow you the option have multiple geofields in your index and not have to index them all as LatLonType.

It shouldn't matter what type you select for the geofield on the fields page as correctly configuring the callback on the workflow page will result in overriding that setting anyway.

I had the same issue, I 1st selected the geofield to be indexed, then hit save, i then went to the workflow page and selected "Index Geofields as Solr LatLonType" and hit save. If "Index Geofields as Solr LatLonType" is already selected then hit save anyway. Then on the search api fields page, there should be no drop down on the geofield field. Dont hit save on this page else it will default back to string. I was then able to use the proximity filter in views. Selecting anything from the drop down seems to void out the prox. filter, and you just get the standard options.

However I dont know what to enter in for the address....I wish there was a way to edit this and make it a zip code. It still doesnt seem to filter anything by prox. no matter what I enter in for the "address" in the exposed filter.

Hello,
@azinck, yes, I select the geofield that I am using on "Index Geofields as Solr LatLonType". And pretty much I follow the steps bigsyke uses.
I also uploaded a photo to illustrate what bigsyke means about fields page on search api.
Maybe Entity API version matters? (I am just saying...). I am using the latest dev(28/09).

P.S. What versions of "Search API Solr search" and search "solr-php-client" do you use. I have dev 09/09 and r60.

Hmm, I'll have to take a look at the latest dev versions. Some of the behavior you describe is a little different from what I'm seeing. It's possible that the changes made in the last week on #1260834: Add a way to define custom data types are affecting this.

@bigsyke: it sounds like maybe you don't have the geocode module installed. I'm just calling google as a geocoder so it can handle almost any address format, including zip codes.

I'm using Search API Solr 1.0-beta4, and r60 of the php client.

Entity API version should have nothing to do with it.

This should be obvious, but after you configure the callback on the workflow page you're rebuilding your Solr index, right?

Actually I have tried everything! Rebuild, disable and enable again, new index, new view. I even tried molux's solution and "tried" to hack search api to add new field type "location". But I still see "Field Type: ignored" on Solr Schema Browser :(

As you may have guessed by now I am a total noob padawan when it comes to drupal that's why I asked about Entity...

Out of curiosity. Why you didn't use this: http://drupal.org/project/geocoder ?

You might be looking at the wrong field in your Solr Schema Browser. The field where the data is stored in Solr is now named differently. If your field is field_location in Drupal, then you should now see the following fields in your schema browser:

• locs_field_location
• locs_field_location_0___tdouble
• locs_field_location_1___tdouble

If you click on either of the *_tdouble fields you should see a document count and some of your top data.

If you don't see that data then I'd question if you have any geofield data stored in your nodes.

The Geocoder project was created less than a week ago, after I built this module. It looks like the version of Geocode I used (which was from a sandbox project phayes set up) became the Geocoder module, so I suspect it'll work just the same, I just need to change the dependency list in my module's .info file.

Sorry! I didn't notice the dates. I just tried Geocoder and it seems to work. I just changed from your module->geofield_proximity_filter_handler.inc@line 74-75
this:

  	  $handler = geocode_get_handler('google');//calling this just to include the necessary files to use geocoding
  	  if($geometry = geocode_google($address)){

to this:

  	  $handler = geocoder_get_handler('google');//calling this just to include the necessary files to use geocoding
  	  if($geometry = geocoder_google($address)){

...just in case someone else wants to use it.

Anyway, in my case
I went to search api->includes->index_entity.inc@line 451 and added

	echo "<pre>";
	print_r($data);
	exit;

And I see:

....
           [field_geofield] => Array
                (
                    [name] => Location
                    [type] => LatLonType
                    [boost] => 1.0
                    [value] => 38.3565,23.3347
                    [original_type] => string
                )
....

So I guess everything is working to this point. I index user profiles, not nodes, if this has anything to do.

• locs_field_location
• locs_field_location_0___tdouble
• locs_field_location_1___tdouble

If you click on either of the *_tdouble fields you should see a document count and some of your top data.

I guess if could make Solr to index it properly this is what I would see. In my case it is only "s_field_geofield".

Thanks again

Aha! I forgot I left a check in there that limited this to only working with node indexes. Sorry!

I've just checked in an update that now makes use of the geocoder module and fixes the issue you're seeing with indexing not working when indexing non-nodes.

If you don't want to grab the update you can just edit geofield_proximity_filter.module and delete everything on line 44 and line 50 to allow it to work with user indexes.

WooHoo!!! It works! I can't believe it was so simple and I couldn't see it...
(And you changed to Geocoder too!)

Thanks yet again!!!

Any chance that somebody involved with this issue can create a patch to be applied to the main branch?

This is very specific to Search API Solr integration so it either belongs in that module or in its own module. I believe mollux is going to turn his version of the module into a full-fledged project sometime soon so be on the lookout for that.

Azinck and me agreed that we will continue with one module, the search_api_spatial one. So please use and test that one :)
I added the project to the queue to be approved, so hopefully it won't take too long.

I'm working on the integration with #1260834: Add a way to define custom data types , and will commit it when it is implemented

@mollux
I would be more than happy to test it!
Could you please help us on some things?

• What hacks do I need to make to search api? Until #1260834: Add a way to define custom data types is resolved. Shall I use the latest patch at #10?
• What do I need to alter on Solr schema.xml?
• What other modules do I need?

If you don't have the time to explain, I understand. I 'll just wait(very eagerly).

Azinck and me agreed that we will continue with one module, the search_api_spatial one. So please use and test that one :)
I added the project to the queue to be approved, so hopefully it won't take too long.

I'm working on the integration with #1260834: Add a way to define custom data types, and will commit it when it is implemented

How do you plan to implement the Solr part? Will you continue using hooks (and/or maybe a service subclass), or do you plan a patch to the Solr module? With the „official“ hook way of defining custom data types I think the latter would be the better option, we could just include support for the new data type in the Solr service class. We'd just need a different schema for Solr 3.x then.
But in the end it's up to you, of course. Maintenance might be easier with a separate solution.

I just committed my code. I'm still waiting to get it approved as a full project. :)

@matrixlord : use the patch at #15 in #1260834: Add a way to define custom data types, use the schema.xml from the search_api_spatial module, and apply the patch (search_api_solr-compatibility-with-search_api_spatial.patch) to the search_api_solr module. Than it should work :)

@drunken monkey : I think it is best to patch the search_api_solr module and not with a . Maybe we can make a new branch, for apache solr 3.1+, so we can include a schema.xml thats supports latlon fields. I will discuss this further in the search_api_solr issue queue

@mollux
Thanks for your answer! I downloaded your sandbox module and it does not contain schema.xml or the patch.

Switching this issue over to the new module's issue queue.

@matrixlord : please check if you cloned the 7.x-1.x branch (git clone --branch 7.x-1.x mollux@git.drupal.org:sandbox/mollux/1269950.git) and not the master branch. All the things you need are in there :)

@mollux
Oops! Noobness :)
I downloaded it finally. I tried yesterday all day and I couldn't get Solr to index properly. It was always indexing them as strings and not locs,coordinates.
Let me try it another day and I will get back with a proper report.

Ok! I found something.
I have made the patches on search_api_solr and search_api. I use geofield with the latest patch from #1285176: Property Info callbacks -- Add field metadata. I have also replaced schema.xml on solr.
I have created a geofield field and I am selecting from search_api@fields tab to index it as "latitude/longitude".
Then I go to search_api@workflow tab and there is no "Spatial search" processor.

So, I go to processor_spatial.inc@line 21 and add:

echo $value['type'];

And it says the type is "string". I believe it should be "location".

I wish I could help more but I don't know where else I have to look.

@mollux
Now I found something else,

In the search_api_spatial.module@search_api_spatial_search_api_data_type_info() function there is no "conversion callback" as suggested in the search_api@hook_search_api_data_type_info() so it must use the fallback which is type "string".

But if it gets indexed as string, it doesn't show up later in spatial processor. Because, there it checks if it is "location" ($value['type'] == 'location').

As you understand I am not sure if what I am saying is correct or even actually matters.

P.S. I use it to index profile entities, not nodes(just in case)

@matrixlord : the search_api_spatial module hasn't implemented all the changes in #1260834: Add a way to define custom data types comment 17 . I will try to look into it tonight and will let you know if it is fixed.

In the search_api_spatial.module@search_api_spatial_search_api_data_type_info() function there is no "conversion callback" as suggested in the search_api@hook_search_api_data_type_info() so it must use the fallback which is type "string".

Having a conversion callback is no requirement, and hasn't got anything to do with the fallback. The fallback is there for the case where the service class can't handle the real type.

Thanks guys! One of these days I may have to learn to read...Sorry, my impatience to make it work got to me.

Having a conversion callback is no requirement, and hasn't got anything to do with the fallback. The fallback is there for the case where the service class can't handle the real type.

Then it should be working...I will try again today!

@matrixlord : I fixed the problem. Please test with the latest 7.x-1.x branch version.
If you have other issues, can you make a new issue for it? so It is easier to maintain :)

I will close this issue, as the basic integration between geofield en search_api is done with the search_api_spatial module. If you have other issues, feature request,... please add them to the queue.

Just updated '7.x-1.x-entity-integration' branch with latest changes. See http://drupal.org/node/1285176#comment-5172768