PREG_CLASS_CJK doesn't include all CJK characters.

We need to check and see if this is an issue in Drupal 7, fix there if so, and then (either way) check on/fix Drupal 6.

Another issue lists some other character ranges that need to be added to PREG_CLASS_CJK -- I have marked that issue as a duplicate, but please check that issue for additional character ranges:
#115843: add CJK ranges above U+FFFF to PREG_CLASS_CJK

Here's what we have now for CJK in Drupal 7:

\x{3041}-\x{30ff}\x{31f0}-\x{31ff}\x{3400}-\x{4db5}\x{4e00}-\x{9fbb}\x{f900}-\x{fad9}

http://www.unicode.org/charts/index.html has a bunch of information about CJK and Hangul character ranges... they go beyond what is suggested above.

And from that other issue, I can confirm from the unicode.org site:
http://unicode.org/reports/tr11/tr11-11.html - "U+20000 to U+2FFFD and U+30000 to U+3FFFD are intended for CJK ideographs"

So it does look like we want to expand this. But I'm not sure about the exact proposal above. I think that proposal is not complete enough.

I checked out the charts from the above sites. Things that I think should be part of the CJK regular expression:

http://unicode.org/reports/tr11/tr11-11.html : 20000 - 2FFFD, 30000 3FFFD

Additionally from http://www.unicode.org/charts/index.html :
- CJK Unified : 4E00 - 9FCF
- CJK Extension A : 3400 - 4DBF
- CKJ Compat : F900 - FAFF
- Hangul Jamo : 1100 - 11FF
- H. Extension A : A960 - A97F
- H. Extension B : D7B0 - D7FF
- H. Combatibilty : 3130 - 318F
- H. Half-width : FF00 - FFEF
- H. Syllables: AC00 - D7AF
- Hiragana: 3040 - 309F
- Katakana: 30A0 - 30FF
- K. extension : 31F0 - 31FF
- K. half-width : FF00 - FFEF

OK, I wanted to save that before my browser died or something. Putting this all together, it looks like we need, in order:

1100 - 11FF
3040 - 309F
30A0 - 30FF
3130 - 318F
31F0 - 31FF
3400 - 4DBF
4E00 - 9FCF
A960 - A97F
AC00 - D7AF
D7B0 - D7FF
F900 - FAFF
FF00 - FFEF
20000 - 2FFFD
30000 - 3FFFD

As compared to what we have now:
\x{3041}-\x{30ff}\x{31f0}-\x{31ff}\x{3400}-\x{4db5}\x{4e00}-\x{9fbb}\x{f900}-\x{fad9}
At least my list is a super-set... :)

I found some more on http://www.unicode.org/charts/index.html
Bomofo : 3100 - 312F
Bomofo ext : 31A0 - 31BF
Kanbun : 3190 - 319F
Lisu : A4D0 - A4FF
Yi : A000 - A48F
Yi radicals : A490 - A4CF
Kangxi radicals : 2F00 - 2FDF
CJK radicals : 2E80 - 2EFF
CJK strokes : 31C0 - 31EF
Ideographic descriptions : 2FF0 - 2FFF

Making the list:
1100 - 11FF
2E80 - 2EFF
2F00 - 2FDF
2FF0 - 2FFF
3040 - 309F
30A0 - 30FF
3100 - 312F
3130 - 318F
3190 - 319F
31A0 - 31BF
31C0 - 31EF
31F0 - 31FF
3400 - 4DBF
4E00 - 9FCF
A000 - A48F
A490 - A4CF
A4D0 - A4FF
A960 - A97F
AC00 - D7AF
D7B0 - D7FF
F900 - FAFF
FF00 - FFEF
20000 - 2FFFD
30000 - 3FFFD

I think it's time for a patch... (some of the above can be consolidated by the way)

I'm working on this, in case you couldn't tell...

Well.
Just a note that I needed to be a bit more careful when looking at these charts, because some of the characters are excluded. Sigh.

Here's a patch:
- Adds new ranges to the CJK preg class. Note that punctuation, symbols, and numbers are excluded, since they're handled elsewhere in search_simplify().
- Adds lots of documentation/comments.
- Changes a variable name within the search_expand_cjk() function from $l to $length for clarity
- Adds a test for the CJK tokenizer.

+++ modules/search/search.module	7 May 2010 20:25:07 -0000
@@ -103,12 +103,25 @@
+ * Matches CJK (Chinese, Japanes, Korean) letter-like characters.
...
+ * Splits CJK (Chinese, Japanes, Korean) text into tokens.

s/Japanes/Japanese

Doh!

Ah, overlap_cjk -- that wasn't documented anywhere so I'm glad to see it somewhat documented now. Are there modules that set this variable, or do we expect people to somehow find out? Ideally, Drupal would set it automatically based on the configured languages but that is for another issue. This patch looks RTBC to me.

Committed to CVS HEAD. Thanks.

Dries: regarding that variable, it is a configuration option on the search module settings page in the UI. See:

http://api.drupal.org/api/function/search_admin_settings/7

I don't think it needs further doc, since it's in the UI already.