in solrconfig.xml:

10000

This is a max number of words in a field. How many words does one have in a long issue thread with comments? A Word count on this full issue suggests it has > 30 k on the first page: http://drupal.org/node/29706

solr will truncate longer texts, making the truncated part invisible to searching.

CommentFileSizeAuthor
#7 348832-maxfieldlength-7.patch1.64 KBpwolanin

Comments

JacobSingh’s picture

JacobSingh commented

maxFieldLength has serious impacts on RAM usage for Solr and hence performance.

I'm not sure what is the reasonable default here, but this is a consideration we can't ignore.

robertdouglass’s picture

robertdouglass commented

So maybe we eventually need to index comments separately? The disadvantage of that would be the overall relevancy of a node should be calculated on the basis of its entire text and all the comments. On the other hand, it might be beneficial to searchers to be able to link directly to the exact comment that matches the search.

Of course, for nodes with comments with pagination, this would still lead to the "can't link to a comment on page >2" problem.

JacobSingh’s picture

JacobSingh commented
Status: Active » Closed (fixed)

I don't see a course of action here. Please re-open if it becomes relevant again.

pwolanin’s picture

pwolanin commented
Version: 6.x-1.x-dev » 7.x-1.x-dev
Status: Closed (fixed) » Active

let's reconsider, at least for 7

pwolanin’s picture

pwolanin commented

This is also quite relevant w.r.t. attachment indexing.

pwolanin’s picture

pwolanin commented
Title: how long should maxFieldLength be? » increae maxFieldLength
Version: 7.x-1.x-dev » 6.x-1.x-dev

Discussed with Solr experts in IRC, and reading in the bookLucene in Action, there seems to be no real performance hit from increasing this. Obviosuly if you have many docs with huge fields, it will take more memory to index them. Lucene in Action suggests that you usually want to use unlimited for field length. A quick scan doesn't show me how to do that for Solr, but doubling to 20k is certainly feasible.

pwolanin’s picture

pwolanin commented
Title: increae maxFieldLength » increase maxFieldLength
Status: Active » Needs review
StatusFileSize
new 348832-maxfieldlength-7.patch1.64 KB

Patch to increase that value - please bump the version number to something in sync w. the release number

pwolanin’s picture

pwolanin commented
Status: Needs review » Fixed

applied this patch to 6.x-1.x and 5.x-2.x. Also just changed maxfield length in 6.x-2.x and HEAD without adjusting the version names yet, since we should likely wait for 6.x-2.0 for that?

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

sjeandroz’s picture

sjeandroz commented

hello,

I had this problem, my nodes who are too long were truncated.
but when I increased this parameters, my nodes where indexed but for the part who was truncated before, the snippet's mecanism don't works...

My nodes appears in the search result but snippets were not built (only for the part who whas truncated before I increase this param)...

If anyone could help me please... (and sorry for my bad level in english...)

Thanks

sjeandroz’s picture

sjeandroz commented

Ok I solved my problem.
I fix the "hl.maxAnalyzedChars" to a very hight number and it's Ok.

Thanks