document preparation runs words together

cspitzlay is right. And I found an additional problem.

For Example body will be indexed in Solr using a Whitespace Tokenizer. So two strings seperated by   won't become two tokens and could therefor not be found when searching because the token looks like this: Abc Def

I attached an improved version of this patch which also deals with html entities and removes some redundancy regarding stripping of control chars.

I'm not sure this order of operations is right - we use $text later so we need to strip control characters.

Any reason not to move to solr.HTMLStripWhitespaceTokenizerFactory?

@pwolanin: You're right. These html tags that get individually indexed should also not contain control characters. I adjusted the patch.

BTW The document that gets posted to solr should not contain any control characters at all. Maybe there's a better place to strip these before sending the document instead of doing so on some fields.

@DamZ - answered that in IRC (breaks the ISO char mapping and also does not strip tags form the stored content, only the tokens, so search sniipets could have html including unmatched tags).

@mkalkbrenner - likely we cold propose this as an upstream enhancement in the Solr PHP class.

please ignore this post

Sorry, I attached the wrong file at previous post :-(

I created a patch to move stripping of control chars to Service.php:
http://code.google.com/p/solr-php-client/issues/detail?id=5

Attached you'll find an adjusted version of the clean text patch that relies on the patched version of Service.php.

+  $text = html_entity_decode($text, ENT_COMPAT, 'UTF-8');
+  return strip_tags(apachesolr_strip_ctl_chars($text));

I don't think we want to decode.

Here's a minimal version that uses the more efficient str_replace()

Actually - that's not really robust - like this is safer.

commiting #10 - setting back to active for consideration of   issues

What kind of problem do you see with html_entity_decode?

An example:

The quick brown fox jumps over the lazy dog.

Without converting   into a blank a search for "fox" has no result.

Theoretically this should also cause problems, but I haven't verified it yet:

German Umlauts like in Hängebrücke

Without converting ä and ü into "ä" and "ü" a search for "Hängebrücke" should have no result.

From my point of view solr should index plain text and not html (we already do strip_tags but that's not enough). If you want to have html entities when displaying the search result you have to add them again at this point.

Anybody who can second this issue?

(cspitzlay told me that he could reproduce the   problem but not German Umlauts)

I hope to find some time soon to do some deeper investigations on this. Meanwhile I use my patch in our project.

Well, we should perhaps decode but then run check_plain on it or some such. Basically - we don't want < > to be converted into tags that would be striped by strip_tags().

For example - if the node content includes the output of a code filter.

"we don't want < > to be converted into tags that would be striped by strip_tags()"

Good Argument. What about simply changing the order of commands:

$text = strip_tags($text);
$text = html_entity_decode($text, ENT_COMPAT, 'UTF-8');

check_plain could than be used to show the search result.

No, we can't check_plain() the search result since we have tags for highlighting key words generated by Solr.

You're right. Simply using check_plain() will break highlighting.

So please have a look at the attached patch. It solves the   problem by decoding html entities, but should not break highlighting or the representation of text entered using code filter.

this looks like excessive work - let's just run check_plain() or htmlspecialchars() after the decode

see: #447622: Encoding of CCK facets