Empty excerpts with Views

This works for me, thansk!

Although, I notice that this patch (and most of the highlighting/excerpt) options only exist when searching a single index. It looks like the searchMultiple method needs to be updated to include the latest highlighting/excerpt code for multi-index searching.

Hi - i'm also experiencing this issue, however not all searches return an empty excerpt - only the items for which the 'spell' field is missing, which is consistent with the above. (Incidentally i've no idea what the conditions are that cause 'spell' to be present or not, despite looking for patterns in my searches)

The patch above seems to fix the issue for me. I have one question though - is there a reason why $this->options['highlight_data'] needs be empty in order for the 'spell' to be injected? I had to uncheck this on my solr service settings for the patch to work, but i imagine some people might want to keep this enabled.

Should be fixed with #1992602: Wrong/Incompatible field used for spellcheck index creation.

Hi,

I've tried updating to the latest version of search_api_solr (7.x-1.0), copying over all the new solr config files, restarting solr, AND re-indexing, but unfortunately i still get the bug.

For the time being i've re-rolled the patch here

I'm sorry, I can't reproduce this error.
Doesn't your Solr query already contain something like &hl.fl=spell? Please check your Solr logs for the query, and execute it to see whether highlighting is returned by Solr. Then the error would have to be in the result processing of the Solr module, or in Views.

I've got this issue. search_api_page works fine returning excerpts, but the views component of Search API doesn't. Excerpts are just empty when I add the field in views.

This is what I get on a dpm from getExcerpt for highlights:

highlighting (Object) stdClass
- mycs-91 (Object) stdClass
-- tm_search_api_viewed (Array, 1 element)
-- tm_title (Array, 1 element)

... and the right stuff is in there. (eta: hmm, ok, I see there's no 'spell' component in there. Should there be?)

eta: hmm, ok, I see there's no 'spell' component in there. Should there be?

Yes, the spell field is the base used for creating excerpts. The others will just add highlighting to the respective fields.
Have you activated highlighting in the Search API server options?
Also, do you have an exposed "Search: Fulltext search" filter in the view, and are excerpts also empty when you enter something there?

I'm not getting excerpts either, patch above doesn't work. I suspect I have something misconfigured

My error was between the keyboard and the chair. There weren't viable excerpts coincidentally for the specific things I was using to test-search for. So, that. ;)

(So, the patch IS working for me. I can't tell you if what wasn't working before was as a result of not having the patch, though. But with the patch, and good content, excerpts are showing.)

I'm searching for text that is in title and body fields; those should have viable excerpts, right?

Yes, those definitely should. To troubleshoot this I queried Solr directly to see what the difference was between what Solr thought and what Views was showing me.

Yea, #1992602 doesn't fix it for me either. Thankfully, pfournier and rich.3po's patch works great.

@ cgove: Could you please post the Solr query (can be found in the Solr log) without applying the patch?

With patch:

params={facet=true&sort=score+desc&facet.limit=10&hl.mergeContiguous=true&f.ss_og_group_ref:title.facet.limit=50&hl.simple.pre=[HIGHLIGHT]&hl=true&f.ss_oa_section_ref:title.facet.limit=50&fl=item_id,score&f.ss_type.facet.limit=50&hl.simple.post=[/HIGHLIGHT]&facet.field=ss_oa_section_ref:title&facet.field=ss_og_group_ref:title&facet.field=ss_type&facet.field=ds_created&fq=bs_status:"true"&fq=index_id:node_index&f.ds_created.facet.limit=-1&hl.fragsize=70&facet.missing=false&facet.mincount=1&qf=tm_title^8.0&qf=tm_attachments_field_oa_media^1.0&qf=tm_attachments_field_basic_file_file^1.0&qf=tm_search_api_viewed^1.0&qf=tm_body:value^1.0&qf=tm_body:summary^1.0&qf=spell^0.00000001&hl.fl=spell&json.nl=map&wt=json&rows=15&hl.snippets=3&facet.sort=count&start=0&q="angles"} hits=4 status=0 QTime=75

Without patch:

params={facet=true&sort=score+desc&facet.limit=10&hl.mergeContiguous=true&f.ss_og_group_ref:title.facet.limit=50&hl.simple.pre=[HIGHLIGHT]&hl=true&f.ss_oa_section_ref:title.facet.limit=50&fl=item_id,score&f.ss_type.facet.limit=50&hl.simple.post=[/HIGHLIGHT]&facet.field=ss_oa_section_ref:title&facet.field=ss_og_group_ref:title&facet.field=ss_type&facet.field=ds_created&fq=bs_status:"true"&fq=index_id:node_index&f.ds_created.facet.limit=-1&hl.fragsize=70&facet.missing=false&facet.mincount=1&qf=tm_title^8.0&qf=tm_attachments_field_oa_media^1.0&qf=tm_attachments_field_basic_file_file^1.0&qf=tm_search_api_viewed^1.0&qf=tm_body:value^1.0&qf=tm_body:summary^1.0&hl.fl=spell&json.nl=map&wt=json&rows=15&hl.snippets=3&facet.sort=count&start=0&q="angles"} hits=4 status=0 QTime=52

I did some more testing. The issue isn't happening to all queries, only a few of them. Applying this patch makes everything work in almost all cases. Even with the patch, one of the results can still have no excerpt. It's hard to explain without an example, so here is a couple test cases. The no have no excerpt and the yes returned one properly.

No patch:

Query A:

no
no
no
no

Query B:

yes
no
yes
yes
yes
no
no
yes
no
yes
no
yes
no

After patch:

Query A:

yes
yes
yes
no

Query B:

yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
no
yes
yes

This is weird. I copied your exact query into a query to my local Solr instance (just changing the index ID and the keyword) and it returned excerpts exactly like expected. And in Views they also still work, so I don't see any errors in Drupal either.

Could you input the query without patch manually into Solr and see whether it returns any excerpts (recognizable by a "highlighting" element in the result)? What if you append &hl.requireFieldMatch=false to it?

Also, what Solr version are you (and others here) using? Maybe something's wrong with the config files for a specific version … (Are you using the latest version of the config files?)

Regarding the patch, you can all of course use that ifn it fixes your problems. However, be aware that it will make it impossible to restrict searching to a subset of your indexed fulltext fields. If you are always searching through all available fulltext fields, this is of course no problem. It just means that I cannot commit this patch to the module – we'll have to find a different solution for that (and, therefore, we'll first have to find the problem).

I am using the config files for 4.x from the latest dev version. My Solr version is 4.4.0. The problem persists directly in the solr admin interface. Highlights are blank for my problem queries with no qf=spell^0.00000001. hl.requireFieldMatch=false doesn't help either. I'm happy to try anything else too.

Thanks for helping track this down!

Thanks for giving me access to your Solr server!
However, I'm sorry to say I still have no idea what's happening there. I'm using the exact same version as you locally and it works fine, but on your server the problem is there as you say.
I see you have changed the provided configuration files slightly. While I don't see how that could influence this (and I've manually tried re-setting the relevant parameters in the URL), maybe you can try to replace them with the exact files provided with this module?

Otherwise, I'm sorry, but for now I can just advise anyone encountering this problem to apply the patch in #5 (or add a hook_search_api_solr_query_alter() implementation in a custom module that does the same).
Maybe someone with more knowledge about Solr internals will be able to solve this.

Thanks for checking it out! I haven't touched the conf files. The ones on the server now are a direct copy of the files in 2013-Sep-02 dev "search_api_solr/solr-conf/4.x" excluding mapping-ISOLatin1Accent.txt as the read me suggests. I originally used the files from search_api that Open Atrium 2 b4 uses by default and it exhibited the same behavior.

I'll keep using the patch for now. Thanks again!

I have exactly the same problem. Without patch most of my exceprts in views were empty. When I applied the patch, much more are now filled, but still ther are some wchich are empty. Are there any other solution to this?

I found out the following and I think this is because I am developing French site. The empty excerpts are those which do not match with the word fully. So if I search "pomme" I have the correct result (including pommes), but the nodes which have plural of this word have empty excerpts -> "pommes". Can I solve this somehow?

Edit: I just want to add, that I don't have this problem with Apachesolr and Apachesolr views modules, which is weird. There is always filled excerpt (snippet in Apachesolr case) and highlighting also works as expected.

Hm, weird. Have you disabled all processors (except "HTML filter", possibly) on the search index's "Filters" tab?

Otherwise, please try to get the request sent to Solr and Solr's response to see what might be wrong there. I've just written a short guide in the handbook, hopefully that will be useful. (I also distinctly remember having written almost exactly the same at some other time, but I can't for the life of me find it anymore.)

Just wondering if the above patch has been included in the latest version of Search API? I see that it is about 9 months old.

I am having a similar problem and I am a little lost on how to resolve it. If perhaps the issue

No, the patch wasn't added, and it won't be unless it's really the only solution (and I can even reproduce the bug, or understand its origin).

If you want to help find a solution:

please try to get the request sent to Solr and Solr's response to see what might be wrong there. I've just written a short guide in the handbook, hopefully that will be useful. (I also distinctly remember having written almost exactly the same at some other time, but I can't for the life of me find it anymore.)

After finishing one project with Apachesolr module I am here again with the same problem. On a fresh new project I decided to try this module once again, since I believe it has better potential than apachesolr. However I am facing the same problem (still it's possible to be risen by my limited knowledge with solr). So here we go. Currently I have one article created, which has only body textarea field and is indexed in solr (version 4.9.0) like this (every filter is turned off including html one):

{
        "id": "cblat5-default_node_index-21",
        "index_id": "default_node_index",
        "item_id": "21",
        "hash": "cblat5",
        "site": "http://foods/",
        "is_author": 1,
        "tm_body$value": [
          "<p>this is a testing body of a cool recipe with apples</p>\n"
        ],
        "spell": [
          "<p>this is a testing body of a cool recipe with apples</p>\n",
          "test"
        ],
        "ds_changed": "2014-08-19T15:12:16Z",
        "is_comment_count": 0,
        "ds_created": "2014-08-19T15:12:16Z",
        "bs_promote": true,
        "ss_search_api_language": "und",
        "bs_sticky": false,
        "tm_title": [
          "test"
        ],
        "ss_type": "article",
        "_version_": 1476879814724944000,
        "timestamp": "2014-08-19T15:32:27.721Z"
      }

when I search for "cool" I got nice excerpt like this:
this is a testing body of a cool recipe with apples …

However when searching for 'apple' the excerpt is empty even with the patch above (with 'apples' it's ok only with the patch)

The request sent to Solr for apple:

http://192.168.56.104:8080/solr/select?fl=item_id%2Cscore&qf=tm_body%24value%5E1.0&qf=tm_steps%5E1.0&qf=tm_title%5E5.0&qf=spell%5E0.00000001&fq=index_id%3A%22default_node_index%22&fq=hash%3Acblat5&start=0&rows=10&hl=true&hl.fl=spell&hl.simple.pre=%5BHIGHLIGHT%5D&hl.simple.post=%5B%2FHIGHLIGHT%5D&hl.snippets=3&hl.fragsize=70&hl.mergeContiguous=true&wt=json&json.nl=map&q=%22apple%22

Solr response:

stdClass Object
(
    [response] => stdClass Object
        (
            [numFound] => 1
            [start] => 0
            [maxScore] => 0.0047079227
            [docs] => Array
                (
                    [0] => stdClass Object
                        (
                            [item_id] => 21
                            [score] => 0.0047079227
                        )

                )

        )

    [highlighting] => stdClass Object
        (
            [cblat5-default_node_index-21] => stdClass Object
                (
                )

        )

)

When searching "cool" the highliting is ok:

    [highlighting] => stdClass Object
        (
            [cblat5-default_node_index-21] => stdClass Object
                (
                    [spell] => Array
                        (
                            [0] => 
this is a testing body of a [HIGHLIGHT]cool[/HIGHLIGHT] recipe with apples



                        )

                )

        )

I thought it could be helpful to post request url for 'apples' with and without patch. So here they are:

With the patch:

http://192.168.56.104:8080/solr/select?fl=item_id%2Cscore&qf=tm_body%24value%5E1.0&qf=tm_steps%5E1.0&qf=tm_title%5E5.0&qf=spell%5E0.00000001&fq=index_id%3A%22default_node_index%22&fq=hash%3Acblat5&start=0&rows=10&hl=true&hl.fl=spell&hl.simple.pre=%5BHIGHLIGHT%5D&hl.simple.post=%5B%2FHIGHLIGHT%5D&hl.snippets=3&hl.fragsize=70&hl.mergeContiguous=true&wt=json&json.nl=map&q=%22apples%22

Without the patch:

http://192.168.56.104:8080/solr/select?fl=item_id%2Cscore&qf=tm_body%24value%5E1.0&qf=tm_steps%5E1.0&qf=tm_title%5E5.0&fq=index_id%3A%22default_node_index%22&fq=hash%3Acblat5&start=0&rows=10&hl=true&hl.fl=spell&hl.simple.pre=%5BHIGHLIGHT%5D&hl.simple.post=%5B%2FHIGHLIGHT%5D&hl.snippets=3&hl.fragsize=70&hl.mergeContiguous=true&wt=json&json.nl=map&q=%22apples%22

.. where apparently missing "qf=spell^0.00000001" part in the query.

However don't get confused with "apple" and "apples" case, since for apple I got empty excerpt even if qf=spell^0.00000001 is presented.

However don't get confused with "apple" and "apples" case, since for apple I got empty excerpt even if qf=spell^0.00000001 is presented.

At least that part I can explain: since the spell field was primarily introduced for spellchecking (as the name suggests), no stemming is done on this field. That means, while "apple" will match "apples" in the (normally stemmed) tm_body$value field, it won't match in spell, resulting in no highlighting.
Come to think of it, a configuration difference between spell and all other text fields could also be responsible for "apples" not matching – since that's immediately followed by an HTML tag (</p>) in your example, maybe the StandardTokenizer the spell field uses doesn't recognize that as an individual word, instead tokenizing it as a single word "apples

".
To verify, go to the Solr admin UI's "Schema browser" and look at the term information for the spell field.

However, I still have not the slightest idea why adding the additional qf parameter would work.
However, I do think that just using a different field for the excerpt should work. (POC) Patch attached, please test (requires re-indexing)!

Oh, this is mighty useful. I was banging my head on the desk trying to figure this out.
Anyway, I can confirm the patch from #28 works miracles for my search results.

Ah, thanks for testing, great to hear it works for you!
Anyone else want to test? With 16 followers, at least one or two others could test as well before I commit …

Also, I spotted one little bug in the patch, fixed version attached.
(I don't remember anymore why the patch was only POC – seems to work fine enough. I just need to remember to mention this in the release notes, so people will know they need to re-index to properly use excerpts. Or does anyone else spot something that's missing in this patch?)

The patch made it even worse. Without patch I get excerpts for "materalien" and none for "material".
With patch I get no excerpts at all.

Before patch:
params={hl.fragsize=70&spellcheck=true&facet.missing=false&f.ss_field_search_group.facet.limit=50&facet=true&sort=sort_field_search_group+desc,+score+desc&facet.mincount=1&hl.mergeContiguous=true&facet.limit=10&qf=tm_body$value^3.0&qf=tm_field_article_text$value^3.0&qf=tm_field_certificate_text$value^3.0&qf=tm_field_microsite_text$value^3.0&qf=tm_field_page_text$value^3.0&qf=tm_field_product_text_intro$value^3.0&qf=tm_title^8.0&hl.simple.pre=[HIGHLIGHT]&json.nl=map&hl.fl=spell&wt=json&hl=true&rows=10&fl=item_id,score&hl.snippets=3&start=0&facet.sort=count&q="materialien"&hl.simple.post=[/HIGHLIGHT]&facet.field={!ex%3Dfacet:field_search_group}ss_field_search_group&fq=index_id:"default_node_index"} hits=2 status=0 QTime=6

After patch:
params={hl.fragsize=70&spellcheck=true&facet.missing=false&f.ss_field_search_group.facet.limit=50&facet=true&sort=sort_field_search_group+desc,+score+desc&facet.mincount=1&hl.mergeContiguous=true&facet.limit=10&qf=tm_body$value^3.0&qf=tm_field_article_text$value^3.0&qf=tm_field_certificate_text$value^3.0&qf=tm_field_microsite_text$value^3.0&qf=tm_field_page_text$value^3.0&qf=tm_field_product_text_intro$value^3.0&qf=tm_title^8.0&hl.simple.pre=[HIGHLIGHT]&json.nl=map&hl.fl=content&wt=json&hl=true&rows=10&fl=item_id,score&hl.snippets=3&start=0&facet.sort=count&q="materialien"&hl.simple.post=[/HIGHLIGHT]&facet.field={!ex%3Dfacet:field_search_group}ss_field_search_group&fq=index_id:"default_node_index"} hits=2 status=0 QTime=4

Seems like &hl.fl=spell is missing after the patch in my case.

Did you re-index after applying the patch? As said, that's necessary.

Dooh! *facepalm*
Now it's working like a charme, thank you!

Good to hear.
Does anyone else want to test?

OK, then I guess it works.
Committed.

Hi - i regret to report that i'm still experiencing this issue. I'm working on a completely different project to when i commented in #2, so it seems something is still awry here

I'm currently using:
- search_api 7.x-1.13
- search_api_solr dev release (2014-Nov-13)

Reindexed, updated schema config etc

I have a list of results coming out of a view with a search-api-solr back end; some results have excerpts and others dont. I'm not sure what information i need to provide to make it easier to investigate. However, i have put some debug into the SearchApiSolrService::getExcerpt() function, and every row returns an excerpt -- ie no problem. However this excerpt does not seemed to be used by the view at all - the excerpt is slightly different there, and if i return an empty string this has no effect on the view. Am i barking up the wrong tree here?

Also if it helps, i have also noticed that searching with plural keywords effects the empty excerpt. ie if i search 'dogs', then results are returned with either 'dog' or 'dogs' in them, *however* the results that only contain 'dog' and *not* 'dogs' have an empty excerpt. Maybe this provides a clue? In these tests, the excerpt debug mentioned above still works correctly - ie excerpts containing 'dog' are generated

Thanks for your help

This sounds to me like you are using the "Highlighting" processor in your index?
If you are, disable it and this problem should be fixed. Solr does its own highlighting, you don't need the Search API processor for that.

I did indeed have highlighting enabled - disabling it appears to have fixed the issue, so thanks for that!

I think it should be made clearer on the settings page that these two features can conflict - see attached patch which may help users in future.

cheers

Hi

I've added this this patch but am still having no success with seeing an excerpt for the fields I wish.

I've probably overlooked something obvious; see as you mention that 'highlighting' should not be selected in '> configuration > Search API > my search > ‘Filters’ ' Then should highlighting be selected somewhere else; or is showing the field 'Search: Excerpt' enough to display this.

Many thanks

@DeejayM

Not sure if you did this already, but you need to disable highlighting on the index settings (ie 'Filters' page), and enable it instead at server level.

To do this, edit the Solr server settings (still within Search API settings), scroll down and expand the 'Advanced' fieldset, and there it is

cheers

Thanks @rch.3po #42

I knew they'd be something simple missing. So to confirm for anyone else . goto

> admin > configuration > 'Your Server' > edit

> under 'Advanced' > tick 'Return an excerpt for all results.'

> 'Save settings'

I think it should be made clearer on the settings page that these two features can conflict - see attached patch which may help users in future.

We already have this note in the Solr service class's description:
"Will use internal Solr preprocessors, so Search API preprocessors should for the most part be deactivated."
The same is also written in the README.txt. I'm aware that almost nobody actually reads either, but I don't think the solution is writing it in more places.
For Drupal 8, we are already thinking about how to better guide users to the right decisions there, so hopefully this will be less of a problem in the future.

I've probably overlooked something obvious; see as you mention that 'highlighting' should not be selected in '> configuration > Search API > my search > ‘Filters’ ' Then should highlighting be selected somewhere else; or is showing the field 'Search: Excerpt' enough to display this.

Enabling that filter instead of Solr's highlighting should also work, but Solr usually handles it better.

I used the Solr Highlighting option. I showed my results in views using index. But the Title field is not Highlighting in the view. I can see the excerpts highlighting but not title and taxonomy terms in the views for searched keyword

Hello,

i send you this message because i'm a little dissapointed. I have installed for the first time the Solr Apache module on my site, but finally I'm getting a empty view (see image). The number of results seems to be good, but i don't see the datas. Maybe my problem joins the #1 post.

Can you help me to clarify the settings of this module and resolve my problem?

1- I'm newby with this module.Have I forgotten something? (fields misconfigured...)
2- I have a problem on my site? (all are green except in views).
3- The distant server has a problem?

Here is the differents steps, i have executed in the admin of my site:

1- I have installed & activated Apache Solr module, search api & search api solr. (Do i need to install theses 3 modules ?).
2- I have configured the distant server (solr) in Apache solr module: it's green (see image). The server is located at this url: http://api.archives-ouvertes.fr/search/index, and I want index title_s, docid, uri_s...fields.
3- I have configurated the distant server in search api : it's green (see image)
4- In fields I have added the fields of the server (see image)
5- In the workflow of search api, i have aggegated 2 fields (docid & title_s) to test the system, and i get my fields the the tab fields (see image)
6- I have reindexed the items and i get the good number of results (see image).
7- Then i have created my view with the index I have precedently created. In this view I get the good number of results (995... items) but the datas aren't visibles. If i make a search with a filter i get no datas et several errors (see image)

Can you help me to resolve this problem? Your support would be really appreciated because I am not familiar with theses modules. I can pay maybe 1 or 2 hours if this is enough to solve the problem.

Thanks for your replies.

Your problem seems rather different from the previous one in this issue. Please just create a new issue for such a question next time!

1- I have installed & activated Apache Solr module, search api & search api solr. (Do i need to install theses 3 modules ?).

No, the first works on its own – either install the first, or the latter two.

That also causes most of your problems, as far as I can see.
However, you also seem to have severely mis-configured the Search API. For the index's item type, you seem to have selected "Search index" – but that will just index the search configuration you are creating, not the content on your site (which is what you probably want to index). Select "Content" instead. That and disabling the "Apache Solr Search Integration" module should get you on the right track.
If you still have problems, please create a new issue.

Can you help me to resolve this problem? Your support would be really appreciated because I am not familiar with theses modules. I can pay maybe 1 or 2 hours if this is enough to solve the problem.

Support in the issue queue is free, no need to pay me.
If you need quicker support (though usually it doesn't take this long in the queue – sorry for that!) and are prepared to pay for it, please contact me via my contact form.

Title field is not Highlighting in the view. I can see the excerpts highlighting but not title

I've also noticed this...

Patch 30 worked for me ... just remember to reindex content to see the changes.

I've also noticed this...

Have you enabled both "Retrieve results data from Solr" and "Highlight retrieved results data" in the Search API server settings (under "Advanced")?

@ mkinnan: You don't need to patch anymore, just use a recent dev version (or the next stable release which should come out soon).

My bad, I wasn't using dev version

OK, then I'll close here again.
Please only re-open if you're sure you're having exactly this problem and it's not solved in the latest dev version! Otherwise, open a new issue.

Hello there. I see this is creating "content" field which was not a case in versions before.
$doc->setField('content', implode("\n\n", array_filter($text_content)));

My problem with this is, that now I have duplicated content in "spell" field, since the content is a sum of all text fields and then it is appended to spell, but the spell already contains all text fields as well (due to configuration in solr schema.xml)

<copyField source="content" dest="spell"/>

<copyField source="ts_*" dest="spell"/>
<copyField source="tm_*" dest="spell"/>

Is this intended? Somehow I get this really weird. Can somebody explain this? So should I just unset content field in solr_documents_alter if I don't want the spell field get duplicated?

Real example:

        "tm_title": [
          "TOP recipes"
        ],
	"tm_collection_text$value": [
          "fsfasafs"
        ],
	"content": "fsfasafs\n\nTOP recipes",
        "spell": [
          "fsfasafs",
          "TOP recipes",
          "fsfasafs\n\nTOP recipes"
        ],

i've got an interesting use case:
- have an index of item type - file
- displaying file name and description columns in a view for the index
- need a full text search which searches both the name and description columns
- need the name column to be sortable
- need highlighting for both columns

what i've got:
- full text search which searches both name and description successfully
- name column is sortable (using an aggregate field so that it can be sortable and searchable)
- highlighting for description column (via my solr server's advanced settings)

issues:
- the highlighting isn't displaying on the name column, i'm guessing because it's actually displaying the string version of the name, while solr is highlighting the fulltext version (again, because i'm using an aggregate field for the name)
- the highlighted results come back as bold in the description column, as opposed to a more classic yellow highlight (is this a solr issue and not a search api issue?)

let me know what you think!