Optionally index fulltext language fallback fileds as langauge undefined (und)

Devel is enabled on our development environments, what do you want us to check ?
Can you indicate what you mean by the language fallback processor ? We do use a language with fallback field , but previously our assumption is that it does whatever it needs to do at query time. I didnt see a processor of this name in the Processors page. The sm_fallback_language field is populated with all of our supported languages if thats what you mean.

Can you explain how this is intended to work ? Say we enable en , ja-jp and fr-fr and we translate some but not all of our content. So a ja-jp user would expect to see some Japanese and some English results. If you send an en node for indexing , the text in say title is getting also copied as a ja-jp field ? Why ? Previously our fallback was based at runtime on the query - the ja-jp user would see results in en or ja-jp ( but not fr-fr) - however at index time the en document would have only en fields and ja-jp would only have ja-jp

One of our issues is that because we can have localized English (not necessarily a translation but regulatory content) we have 150 odd languages which makes the problem severe for us but even if i limit to the above 3 languages , i cant see the reasoning behind it.

From Solr Devel , I have attached the same node indexed on 4.2.8 v/s 4.2.9 of search_api_solr

I think I see what you are trying to do but again it looks wrong for precisely the reason you specify (
The reason is that both field types might be configured differently.) - If you have a fr-fr field type or a ja-jp field type with different settings (As they must be for different languages) , copying english text into those fields doesnt work right ? You can see why this is wrong if you put japanese characters into english fields right?

Also I would like to understand how you expect the query to work - there is a checkbox that states limit results to only this language or not . For those that have it as not , they would anyway already be querying the "en" fields and the "fr-fr" fields . We use fall back for query time
So en document ==> all en fields populated
fr-fr document ==> all fr-fr fields populated
At query time we can say give all documents matching the "en", "fr-fr" ,"en-fr" fields

It looks like you expect
en document ==> all en , en-fr, fr-fr (supported fields populated
fr-fr document ==> only fr-fr populated
At query time you expect to specify only fr-fr or only en-fr or only en right ?

>Feel free to provide patches for a better implementation
>But in the other cases, if you don't provide the other fields a filter for fr will lead to zero results because the en_* fallback won't be used.
Im still trying to wrap my head around what works better so Im far away from a patch. In our case we have a million records which are in English and which will always be in English (Scientific data, Drupal custom entities) - but we also have the traditional content which will be translated. For us the replication of the fields into languages killed both disk and performance and indexing speed. But if I exclude such scenarios

>Again, in your case that fallback feature is not appropriate and should be turned off.
Im trying to understand this feedback - We do have a fallback since not everything is translated. Our current design uses sm_language_fallback.
Node1 (say en only) - sm_language_fallback =>en, fr-fr, en-fr i.e. this record is a candidate for everyone since nothing else exists
Node2_en => sm_language_fallback => en i.e. this record is a candidate only explicitly for en
Node2_en_fr => sm_language_fallback => en_fr,fr_fr i.e. this record is a candidate for both en-fr and fr-fr since no fr-fr exists
Node3_en => sm_language_fallback => en i.e. this record is a candidate only explicitly for en
Node3_en_fr => sm_language_fallback => en_fr i.e. this record is a candidate for only en-fr only
Node3_fr_fr => sm_language_fallback => fr_fr i.e. this record is a candidate for fr-fr only
i.e. in our case , we populate the fallback field with what the content applies to. The index however only has the field populated

when we query for a French user we will search the _en, _en-fr, _fr-fr fields (hooked into alterQuery) with a fq (facet) value of sm_language_falbback:en_fr if say the user had selected en-fr as their language(or fr-fr if the user had selected french). So we do not miss any search records , and we show the right precedence. So in the example above
a en-fr user sees Node1, Node2_en_FR and Node3_en_fr and (fq=sm_language_fallback:en-fr)
a fr-fr user sees Node1, Node2_en_FR and Node3_fr_fr and (fq=sm_language_fallback:fr-fr)

We cant turn off the fallback , nor can we restrict the languages (en_fr and fr-fr are both valid and so both would have to be checked right)

@mkalkbrenner
Not yet , will try to do it this week