How to get Search API Solr work with i18n content translation

Hi dronken monkey,

Thanks so much for your extensive answers, I really appreciate it! Since I was (am) new to the Search API and Search API Solr modules and Solr itself, I really had a hard time getting to know Solr, it's XML schema, configuration options, filters, etc., but also defining which functionality is provided by Solr or by the Search API (Solr) module.

Since your 1st comment I tried many things and eventually got i18n content translation (opposed to field translation) working with correct stemming and spellcheck in English (EN) and Dutch (NL) using 1 Solr core, but 2 Search API indexes.

I will describe how I managed to do this in detail, hopefully others can benefit from this. Also see the attached custom module (which can be used as a base for one's own implementation) and the Solr schema and config files.

Important: see last paragraph on assumptions, caveats and code modifications!

Stemming and field name mapping

When querying for "service", stemming makes it possible to also find pages containing "services" and "servicing". In order to get stemming to work for different languages, we have to define separate Solr field types and Solr fields with language prefixes.

The language in the default setup is English and works out of the box, but to make it easier to adjust and maintain settings we define new field types for both NL and EN, and also copy stopwords.txt, synonyms.txt and protwords.txt and paste them as lang/stopwords_en.txt etc. If the Dutch files are not present in the lang folder, copy them from the English versions.

Field name mapping

The Search API Solr module uses dynamic fields to handle certain Solr fields in the same way, based on their name's prefix. Example: the Drupal's body field has a value, safe value, etc. Search API generates a string identifier like body:value, Search API Solr maps this to tm_body$value. The prefix tm_ and the dynamic field settings make Solar treat this field as a full text (t) multi valued (m) field.

The hook hook_search_api_solr_field_mapping_alter() allows us to alter the field name mapping, so we can localize tm_body$value into ie. tm_nl_body$value to apply Dutch stemming. See attached module.

Separate Search API indexes per language

Search API Solr maps field names per Search API index and not per node. This means all indexed nodes in a Search API index use the exact same mapping, which makes it impossible to index nodes from different languages in a single Search API index. So we have to create two separate Search API indexes.

To make hook_search_api_solr_field_mapping_alter() aware of which language prefix to apply, we use the 'Language control' data alteration option at the workflow / filters tab of the concurring Search API index. Enable this option and adjust the callback settings below. We use 'Language field: Language' and select only one language (and optionally Language undefined). Our implementation of the hook_search_api_solr_field_mapping_alter() hook requires that only a single language (and optionally Language undefined) is selected for localized mapping!

schema_extra_types.xml

From schema.xml we copy the text and textSpell field types and paste it in schema_extra_types.xml. Once for EN and once for NL by adding _en and _nl suffixes to their names. Further we localize settings by changing the paths to imported text files (from stopwords.txt to lang/stopwords_en.txt) and changing the language attribute of the SnowballPorterFilterFactory for stemming.

schema_extra_fields.xml

Here we copy + paste the commented German example code for both EN and NL and replace all _de occurrences with their localized counterparts.

Before doing this, modify the example code in the following way:

1: The type of field spell_de was text_de, but should have been textSpell_de;

2: Add copyField parameters for dynamic fields:

<copyField source="ts_de_*" dest="spell_de"/>
<copyField source="tm_de_*" dest="spell_de"/>

so all localized full text field values will be automatically added to the right localized spell field.

Solr spellcheck

When querying for "internt" (mind the missing -e), Solr spellcheck offers a "Did you mean 'internet'" link. The Search API Spellcheck module extends the Search API module for this purpose.

The Search API Solr module's default configuration uses a field (opposed to ie. a file) for it's spellcheck functionality. When querying a word, spellcheck searches the indexed values of this field for similar words. The default configuration uses the field spell which values are a concatenation of all full text field values of a node.

We created localized spell fields in our schema so we can use separate dictionaries for each language.

solrconfig_extra.xml

To build these dictionaries, we have to define them in solrconfig_extra.xml. All we have to do is copy + paste the default dictionary twice (EN + NL) and change the names of the new dictionaries from default to spellchecker_en / spellchecker_nl and append localized suffixes _en and _nl to the other string values.

NB: In case of multiple dictionaries, Solr requires a dictionary named default (or with no name defined at all), otherwise Solr will not start.

We are almost there!

We defined new Solr field types, fields and dictionaries. From within Drupal we created two Search API indexes and altered the field name mapping. After reloading Solr and indexing the content from both indexes, all data is (hopefully) successfully stored using our custom field mapping.

All that there is left to do is some localization on query time using hooks (see attached module):

1. hook_search_api_solr_query_alter() alters 1) the names of fields that have highlighted search keys and 2) the name of the spellcheck dictionary;
2. hook_search_api_solr_search_results_alter() creates custom excerpts bases on localized spell fields.

Important:

* We create 2 completely separated localized search pages using 1 Solr core, 2 Search API indexes and 2 Views;

* From the Search API index' workflow / filters tab we select 1 language and optionally Language undefined. If you select Language undefined for both languages, all Language undefined content gets indexed twice (no harm done, but gives a little extra overhead);

* All Solr fields of type textSpell need to be stemmed as well if you use exerpts / (highlighted) snippets, otherwise a query for "service" does find a node containing "services", but Solr does not return any snippets;

* In schema.xml I commented out the following copyField parameters for dynamic fields to avoid both spell and spell_nl fields from being created (we only want the localized field);

<copyField source="ts_*" dest="spell"/>
<copyField source="tm_*" dest="spell"/>

* Is the use of dynamic Solr fields like tm_ and tm_nl_ a smart thing? The first applies field type text, the last applies field type text_nl. Does the latter overwrite the first or are they both applied? If they are both applied, are both stemming factories being used?

* Always reload Solr after editing Solr schema and configuration files. This can be done from the Tomcat control panel at localhost:8080/manager/ (assuming you use Tomcat and at port 8080) or the Tomcat GUI ( TOMCAT_FOLDER/bin/Tomcat7w.exe on my Windows 7 workstation);

* In solrconfig_extra.xml we set that dictionaries are being built when the index is being optimized. This is automatically done by cron once a day. When testing make sure you optimize the Solr core, otherwise spellcheck does not offer any query suggestions. See the control panel at localhost:8080/solr/NAME_OF_CORE/ or by visiting localhost:8080/solr/NAME_OF_CORE/update?optimize=true.

Thank you for the clarification. Based on your answers above and at #2147957: Hard coded field key ("spell") in SearchApiSolrService::getExcerpt() I updated comment #4 and the provided custom module.

@lazzyvn: See to this issue attached files, best of luck!