Integrating with search_api

After looking at the __awesome__ screencast prepared by Thomas, I am now totally convinced to start working on this :-)

This is my start version(aka needs work, search missing, but indexing already works and with tons of debug :-p), not sure about what to store in the "data" for the xapian document, I suppose that is going to be clear when I have implemented SearchApiServiceInterface::search().

Now, basic search is working!

So, as the last post, lots of things to care about(debugging messages, catch exceptions, get values instead of hard-coding them, support facets, complete xapian server support), but this is working!

Thomas, I think I am not getting the right way to set values on the server. Should I have to store manually somewhere my settings using SearchApiServiceInterface::configurationFormSubmit()?

Upps.. ignore the last question, seems like it's simple form api stuff, blame me :-p

Thomas, thanks for taking the time to notify me about that patch, like you mentions, seems not really related, but I need to try this patch on the new code, by now, I have a new(cleaner) version.

In this new patch debug messages are out, support for remote xapian databases should be working.

So I still need to:

• catch exceptions on every xapian call
• support facets: I need to try to do this, like in other places, I think the actual implementations can help.

Questions:

• get values instead of hard-coding them: this is really fixed all around, but I need help to get range values(number/position of values to retrieve) to use at SearchApiServiceInterface::search().
• on SearchApiServiceInterface::search(), I am using $query->getOriginalKeys() to get the original string. I really do not know if it is a better idea to make a method to process the value returned from $query->getKeys(), or if it is better to define a query class which implements SearchApiQueryInterface for xapian. A reference to take a decision here could be $extra variable logic, at xapian_query().

Thanks for taking the time to review this!

I got a little problem: #936956: Argument 1 passed to search_api_admin_server_view() must be an instance of SearchApiServer

About _hard-coded_ values, I mainly was pointing my hard-coded values, but I get rid of most of them in the last patch :-p, but I end up with one more; which are these lines in the search() method:

+++ search_api_xapian/service.inc
@@ -0,0 +1,442 @@
+      //FIXME stop hardcoding this
+      $start = 0;
+      $length = 10;

+++ search_api_xapian/service.inc
@@ -0,0 +1,442 @@
+      $query_parser->set_stemming_strategy(XapianQueryParser::STEM_SOME); //TODO this should be an option

Yep, like you mentioned stemming should be an option, and that's planned ;-)

About using getKeys(), ok, I going to follow the method you mentioned soon.

About languages, yep, I should auto-map values, thanks for pointing that (aka /me use to make lazy first implementations :-p)

Powered by Dreditor.

This still needs some work about the details mentioned.

But for now, attaching a better patch with:

• no more hard-coding values
• stop depending on xapian.lib.inc from xapian module
• catch xapian exceptions everywhere
• add install file to check xapian bindings as requirement.

Ok, here it is a patch that use ->getKeys() at search borrowing solr flattenKeys() method as suggested.

In the other side, it seems like parseKeys() have some problems to identify quoted strings(it first implode with spaces), so, taking in account that xapian have it own query parse class(see specially the flags parameter on parse_query() method), I think it would be great to have a way to overwrite the query class, but I am not sure how to do that.

Two more big TODOs for this issue(or maybe two more issues?):
- support views integration
- support facets integration

After looking a little more(SearchApiQuery class) on the api, it expects to pass parsed keys as array.

Xapian try to do the search stuff on C++, as optimally as it can, so, all the parsing done with XapianQueryParser class is actually done internally, and the result of the parsing is a XapianQuery object, which is different from the expected array. So I think I would stay with the last patch approach of flattenKeys().

In the other side, it would be great to have a way to use the power of xapian to configure the query(as we can now configure the server options), for example the xapian parser flags to define how to do the parsing.

BTW, I end up finding one thing I think it's a little bug somewhere: #938982: Not all SearchApiQuery options are passed

Just another update, the following patch include:

• api update: configurationForm* server methods changed its signature
• api update: use new machine_name to identify servers
• use the indicated default search_api query conjuntion as the default xapian operator at xapian parse_query
• avoid xapian query FLAG_DEFAULT to support versions less than 1.0.11
• implement hook_uninstall
• better description for server service
• default way to index unknown field types
• minor: right default values at xapian server service configuration form

I think this is almost ready for a first commit, I see views integration is working, but filters are not working at view, and I suppose the same for sorting, I need to review my search() method to include filter and sort idea that I have seen in the other implementations.

In the other side, I think facet support is going to be an independent feature request, since it implies propose a way to use facets with xapian search engine.

Thomas, thanks for notifying the api chages by mail.

Well, I still need to implement correctly filters to support views, but now views depends on ctools :-o, so I going to review that soon.

By now, here is a patch that:

• a better name for search api implementation module
• avoid namespace problems if both core integration and search_api are enabled
• see if search_api is enable at hook_uninstall before deleting rows
• api change: #939482-25: Fix exportables machine_name is now the recommended identificator.

To be able to really work with this patch first we need to rely on this bugs to be closed ;-)

• At search_api: #958378: "Clear index" is broken
• At entity: #957684: duplicated function definition of entity_ui_validate_machine_name()

Ok, soon was not too soon :-/, but I am updating this to point to #962582: Cross-entity searches, that is becoming really interesting and now have a patch! :-)

BTW moving status to reflect the last work to do here implement correctly filters to support view

API changes pending to review/incorporate: #991632-6: Incorporate newly available entity hooks.

Related open issue(bug?): #1002210: Can not use search indexes

Ok, I just finally committed the patch at #16 after taking a look to API changes from Search API module.

I have also added three new issues related with this one:

• #1042684: Make sure views integration throught Search API is working correctly
• #1042688: Implement a facet to focus search
• #1042694: Implement more like this search api feature with xapian expand terms

@pillarsdotnet: Thanks for reporting the compatibility problem, but I am moving that to other issue #1042704: Fix compatibility of xapian and search_api_xapian hook_requirements() implementations, since it's almost offtopic here.

yeap, it got in some time ago http://drupal.org/commitlog/commit/6108/0fa8cdaeb86607c9178d7994129e89b1...

@pillarsdotnet: Please stop hijacking this issue :-/, open another one instead. I am using master now for 7.x