We see, e.g., in #2573457: Make stopwords translatable that our current access to the item language is a bit too low-level. The Solr backend, e.g., will probably want to index the fulltext fields in separate Solr fields according to the item's language, with proper analysis based on the language. For that, they'd need to determine the item's language at search time (which might not properly work for non-entities), and it would also be good (for performance reasons at least) to determine the possible languages affected in a search query.
Therefore, just having an "Item language" field might be a bit little for language support, we might want to have the language more on the same level as, e.g., the datasource of an item – available explicitly both on the item object and in the index. And, even more so, also in a more prominent place on the query, so it becomes easier to determine the languages of results a search will return (which, I think, isn't really necessary for the datasource).
So, my suggestion here would be to do almost the opposite of our previous direction: re-add something like the languages option to queries again, remove the "Language" processor and the "Item language" field but make the language a first-class citizen of items (with support from the datasource to determine an item's language) and define a magic field, similar to search_api_datasource, to filter on the language (instead of having it explicitly as a field).
The great danger of having both a languages option and a magic field for the query would of course be that, similar to what happened in D7, people would just use addCondition() and ignore the option, which would render all the possible benefits of the option void. However, for things like combined conditions (e.g., only include the und language for a certain datasource), having it available as a field is absolutely necessary, too, so I think we should just document that, usually, the option is the preferred way, and it should in any case cover all the possible languages returned, and then conditions can be used where necessary to filter that down for specific special cases. If the option is left untouched, of course, items should be returned regardless of language, so even if someone doesn't use this properly we don't lose much.
(Also, I write "languages option", but I actually think it should be a proper getter and setter. Also, this time we should leave it to users to add und if they want it.)
Estimated Value and Story Points
Value and Story points are in the scale of fibonacci. Our minimum is 1, our maximum is 21. The higher the number, the more value or work a certain issue has.
Value: 8
Story Points: 8
| Comment | File | Size | Author |
|---|---|---|---|
| #23 | 2641392-23--language_support.patch | 61.93 KB | drunken monkey |
Comments
Comment #2
drunken monkeyComment #3
drunken monkeyChange of heart.
Comment #4
borisson_Fixed a small typo in the IS. We should probably write a clear todo for this issue as well.
Comment #5
drunken monkeyThe "todo" is first to decide what we want to do, which use cases necessitate which functionality, whether it's really justified to add such a special case for the item language, etc. Some input of people like Markus Kalkbrenner, who have a good idea about (and work a lot with) multilingual sites and their searches, would probably be good here. (Gonna contact him to ask him for his comments.)
The further path forward would then of course depend on the decision.
Comment #6
mkalkbrennerIn general apachesolr_multilingual requires a "language" field.
For full text fields there's always a language specific field to which a query could be explicitly limited. But if you query non full text fields you have to limit the query to a specific language.
The same is required for CLIR (cross language information retrieval) where multiple translations are indexed as one entry. If the language field is not present you can't distinguish between the content (language) written by a human and the machine translation added to this search index entry.
If we decide to remove the "language" field in search_api, we 'll re-add it dynamically in apachesolr_multilingual.
But I recommend to keep something like the current approach of search_api:
Comment #7
drunken monkeyThanks for your insight here, Markus! Your rationale of course makes sense, something like the language field would indeed be required.
However, I think my second suggestion would also cover this pretty well, and even help in some way. Currently, I guess you need to always just use the current page language for filtering, or search through the query whether there is a language filter anywhere? With my suggestion, you'd just have to use
$query->getLanguages()and would know right away which language(s) should be returned.The
search_api_languagefield would be gone as such, but backends would still be required to index the language, even more explicitly than currently, because they'd be required to be able to filter on it later on.So, I think, from what you describe, this would actually be a superior solution.
Comment #8
drunken monkeyHere's a first draft of what I had in mind. "Needs review" because I'm interested which tests fail horribly with this, but in any case we'd at least need to add additional tests for this functionality (and probably some more polish in various places).
Comment #11
drunken monkeyThis should at least fix the tests.
Comment #12
drunken monkeyAnd here are the tests (plus a fix for a completely unrelated bug I just discovered through those tests, having been there for at least 16 months).
Comment #13
drunken monkeyA bit of clean-up.
(Interesting, and a bit worrying, that the DB search defaults module's tests didn't notice that we had an invalid field contained in the index. Might be fixed, or at least mitigated, by #2612104: Cascade properly when processors providing new properties are disabled, though.)
Comment #14
mkalkbrennerI did a first quick review of the patch. Just to ensure that I got it right, can you update the issue summary to only match your current approach?
In general I think that the patch seems to be OK and is an improvement!
But we will need some special handling when we implement that language condition group within search_api_solr.
For the CLIR feature search_api_solr_multilingual might need to add more languages to the filter. Currently that happens after the Solr query has been built. With the new API introduced by this patch we might change this.
But during writing this comment I start thinking about the CLIR feature in general. Maybe it's a good idea to move it from search_api_solr_multilingual to a separate module called search_api_clir and make it available to all backends.
Comment #15
borisson_There are a bunch of unrelated changes in the ProcessorIntegrationTest, I think. But they're all good changes, so let's keep those in. Overall this is looking great and it looks like @mkalkbrenner agrees.
Of course I did find a couple of smaller things, I think this could get in as-is if you don't agree with those points.
I don't see why these changes are needed, can we revert that?
Can we split this up to make it more readable? Is there a reason the operator check is lowercase for
INand uppercase forAND?Comment #16
drunken monkey@ mkalkbrenner: Thanks for the review! I updated the issue summary. If this sounds OK to you, I think we can commit this.
@ borisson_: Fixed both your concerns, so it should now be RTBC for both of us.
They definitely don't belong here, that's right. Since they are just style changes, I just committed them without any issue. (I know the previous was fine, too, regarding coding standards, but I think it looks better this way – I dislike one-line associative arrays, and this way is also more consistent with the other mappings.)
You're right, this is unreadable. I introduced a new helper method on the query plugin class for getting the group conjunction (or "type", as Views calls it), I think that's enough (by getting rid of the nested conditions, at least). I also tried re-writing this more, to not use a statement with three boolean expressions, but it ended up not really being cleaner, just more verbose.
The case difference is very easy to explain:
Comment #17
borisson_Comment #18
mkalkbrennerI would say go ahead and commit it. But wait ...
PLEASE, commit #2709951: Add a generic base for backend tests first!
I will help to adjust the patch in #16 to the new test base class then!
But this way round it's much less work!
Comment #19
mkalkbrennerThe advantage will be that we immediately will notice what potentially breaks in the Solr backend.
Comment #20
borisson_Comment #21
drunken monkeyRe-rolled and slightly re-formatted (test code).
Comment #22
borisson_I like the reformat there, that looks great! I went trough the entire patch again and I found 1 thing I have a question about, but overall this looks great!
Why did we make this public?
Comment #23
drunken monkeyGood point, thanks for catching that!
The property comes from Views' query plugin, where the property is (of course) public, so apparently I assumed some filters would access it directly. However, I tested it locally and this doesn't seem to be the case after all.
Attaching the adapted (and re-rolled) patch again, but if the test bot agrees I'll commit it right away.
Comment #27
drunken monkeyExcellent, committed.
Thanks again, everyone!
Comment #28
mkalkbrennerThere should be no new release until #2719843: Call to getValues() on non object is fixed.
Comment #29
mkalkbrennerComment #30
mkalkbrennerThe basic Solr tests are passing again:
https://travis-ci.org/mkalkbrenner/search_api_solr/builds/128484297
So #2719843: Call to getValues() on non object doesn't block a release anymore. Thanks.