I have Arabic texts indexed in Solr atm and searches work just as they're expected to.
They return the correct results and show excerpts on where the matches are.
However, since the latest Drupal release, I decided to also update this module, and now highlights aren't triggered for non-English words.
Note: I am not using the Search API Multilingual Solr Search, since production will only use Arabic searches, not English.
The expected behavior worked before, but now it seems to struggle with the different encoding.
Solr still handles it perfectly and hooking on hook_search_api_solr_search_results_alter() shows that the Solarium $result does in fact contain the correct highlighting tags under data -> response -> highlighting.
I added small screenshot to support this.
The results in the view's excerpts don't contain any HTML tags whatsover, it's a pure string.
<ul>
<li>
… أن يكون خطرات لأفراد. # ونهضتنا الحاضرة، التى انبثقت فى جميع … للمؤلف وقد كتبه بخطه، قال: # ذكر نبذة من ترجمة مؤلف هذا التاريخ … المصرية، ثم كافل المملكة الشامية. سألته عن مولده فقال: # مولدى …
</li>
</ul>
So Solr processes this normally, but I suppose it's a small issue which perhaps has something to do with not using multibyte string functions.
Any ideas?
| Comment | File | Size | Author |
|---|---|---|---|
| #10 | scrot_4.png | 43.66 KB | darm |
| #10 | scrot_3.png | 30.11 KB | darm |
| #10 | scrot_2.png | 69.82 KB | darm |
| #10 | scrot_1.png | 46.44 KB | darm |
| highlight_scrot.png | 12.86 KB | darm |
Comments
Comment #2
drunken monkeyI think the Search API Multilingual Solr Search module is also meant to be used for single-language non-English sites – as the project description page also states.
Have you tried switching to the 2.x branch of the Solr module? That already includes the code from the Multilingual module, so might work better in general.
Since I don't speak (or even read) Arabic, or any language with a non-Latin script, this is rather hard for me to reproduce. We theoretically took great care to keep all our code multi-byte-compatible – but of course we could have made a mistake somewhere.
If you already have a
hook_search_api_solr_search_results_alter()implementation, could you include something like this (or look at the values directly in a debugger)?This should show whether the highlighted data is correctly transferred from the Solr response to the results data. Otherwise, it's probably a bug in the Solr backend, not in the Search API itself.
Oh, also I now remember that the Solr backend recently switched to cooperating more with the Search API "Highlight" processor. So, using the latest version, you might need to enable and configure that processor for things to work correctly. (But I'm not sure – I'm not working much with Solr in D8.) This only makes sense if the highlighted fields data is present in the items as seen above – otherwise, you'll just get our own, crappy (compared with Solr) highlighting implementation, which should only be a last ressort when using Solr.
Comment #3
darm commentedOh sorry, I meant to open this ticket in the Search API Solr module, I'll reopen this ticket in the correct module.
Just for completeness:
dpm($item->getAllExtraData());does not contain any highlighting information.It contains one field but nothing related to highlighting:
The thing is, I use the the highlights in the Solarium result object, to calculate other things as well and this still works, so Solr does its part, but they're incorrectly processed.
Thanks for your input though, I'll coin this to the Solr module.
Comment #4
drunken monkeyPlease don't do this. In almost all cases, it's better to just move an existing issue to the correct project.
Doing that now, let's see whether Markus knows more.
Comment #5
mkalkbrennerThat requires the 8.x-2.x branch of search_api_solr. It won't exist in 8.x-1.x.
But as mentioned in the issue description, solarium returns highlighted snippets.
However, is it possible to switch to 8.x-2.0-rc1 before we proceed? We had some issues with highlighting in general and took a different approach in 8.x-2.x.
Comment #6
darm commentedThanks for the response.
I just updated to version 8.x-2.0-rc1 using drush, but it fails when I clear my cache or try to uninstall.
Following are the steps I took, is there something forgot to do?
Comment #7
mkalkbrennerWe only support the installation via composer.
Composer also verifies more requirements, for example PHP 7. Which PHP version are you using?
And we rely on the class loader built by composer.
And if you need to migrate your setup from 1.x you should read https://cgit.drupalcode.org/search_api_solr/tree/UPGRADE_FROM_MULTILINGU...
Comment #8
darm commentedAh, well, we're still working with PHP 5.6 unfortunately, and enforcing a PHP 7 upgrade just like that, isn't something I can do.
Nevertheless, not quite sure how this feature stopped working.
The multilingual feature seems like a great extension to use, but it wasn't necessary before.
The query still gets transported to Solr, Solr still processes it correctly and returns a correct result with the correct [HIGHLIGHT] tags.
So all of this still works like it did before the update, only problem is that in my view's excerpts now only English queries get a visual highlight.
If you need more information from me, I'm happy to provide it, but sadly I can't simply upgrade everything to the latest version.
Comment #9
mkalkbrennerSuch result highlighting in Views never worked with 8.x-1.x well without custom code.
At least you need to turn on 'Retrieve result data from Solr' and 'Highlight retrieved data'!
But as soon as Search API requires to load the original entity, the (highlighted) results get overwritten.
This will for example happen if you don't disable the access checks in Views.
That's why we rewrote it for 8.x-2.x.
Comment #10
darm commentedWell I never had an issue with it, but the access rules for the view is set to None.
Which settings are these specifically?


What I did to make this work, was the following:
1. activating the highlight processor on the index
2. adding the Search: Excerpt field to the view.
This results in English being tagged, but Arabic results are left untouched.


Comment #11
mkalkbrennerThe configuration you applied only works with 8.x-2.x! The highlighting you see is a PHP post processing that doesn't apply Solr's return values.
As 8.x-1.x is not in integrated with the Highlight Processor,
Comment #12
mkalkbrennerComment #13
darm commentedI turned off the Highlights processor like you said, but those server settings were already enabled.
The only thing that happens now, is the correctly matched entities are shown, but the excerpts in my view no longer contain anything.
When picking pure field values in the view like so
Content datasource: Source Text (indexed field),these fields are just getting rendered in their entirety without any highlighting, which is not what I'm looking for.
Is there any additional information I can provide you to give insights in how my setup looks?
Comment #14
mkalkbrennerDon't use entity rendering in views. This kills the 8.x-1.x highlighting, just like the access check does.
The highlighting is a pain in 8.x-1.x and was written long before Search API itself was stable.
If highlighting is that important I really encourage you to upgrade to PHP7 and 8.x-2.x where highlighting should work out of the box.
BTW According to your screenshots you never used Solr's built-in highlighter in the past but the simple PHP implementation provided by Search API via the processor. So it might be an option to debug the regular expressions there in order to stay with your old setup. Maybe it really an encoding issue.
Comment #15
darm commentedAlright, I'll poke around and see if I can intercept where it goes wrong.
Pretty sure it's just a simple encoding issue somewhere which got through :)
If I find anything, I'll let you know.
Also, just to be complete, I'm using Fields as format in the views, not rendered entities.
Comment #16
mkalkbrennerOK. I'm waiting for feedback now.
Comment #17
mkalkbrennerI assume the patch in #3001030: The excerpt should leverage the backend highlighter with stemmer & accents will solve the issue.
Comment #18
mkalkbrennerfixed by #3003524: Excerpts don't work with stemmed keys or diacritical marks