Can't add job from entity written by non-latin language (like cjk).

I see.
We are talking about support for language plugins and understanding more about a language.
#1891842: Introduce language plugins

The idea would be to introduce a controller that can handle language specific things.
In your case, the source language controller can implement, how to check for non-emptyness.
Languages that have similar behaviours can have a common controller / parent and there can be a default language controller.

I guess we should just not rely on wordcounts currently and switch back to strlen to check non-emptyness only.

Can you provide a patch?

That's why we have drupal_strlen
http://api.drupal.org/api/drupal/includes%21unicode.inc/function/drupal_...

Note also, my word was just about detecting non-emptyness of a string. Not counting words.
The wordcount will be wrong as long as we don't have language plugins.

Finally i guess, we can rely on mb extensions for non-ASCII languages. At least i would just define it as a requirement or we will end up doing ugly code and workarounds till we die. ;-)

Yay, done! Please review the patch. It contains a replacement for str_word_count function that works correctly with non-latin languages (as well as with latin). It doesn't rely on any dependencies. Unit test is included.
@barami It returns 1 for '가' ;)

Weird, http://api.drupal.org/api/drupal/modules!system!system.api.php/function/hook_install/7 says that all module's functions are available during setup. =/
Here you are a new patch, that deals with it.

DrupalUnitTestCase doesn't flush all caches after setUp - it only resets static, what's a pity.
One more attempt.

+class TMGMTWordCountUnitTestCase extends DrupalUnitTestCase {

We have TMGMTHelperTestCase for testing helper functions, I would probably move it there.

Otherwise looks good.

+ $text = trim(preg_replace('/ {2,}/', ' ', $text));

Also I do not really agree with this. If a sentence contains recurring words, translation services will still count it.

If the remove of duplicate makes sense, please provide test coverage. With translation, customers don't pay per unique words (in languages i know of)...

Please move the tests into our TMGMTHelperTestCase.

Beside this, ready to commit. So looking forward to an updated patch.

Here you are updated patch (interdiff is attached).
1) Test moved to mentioned testcase.
2) Please note comment changing: we are removing "duplicate spaces" not just any word duplicates, sorry for misspelling.
3) I've also added a test case for it: "repeat repeat" are 2 words, not one.
Please review once more.

Note that this also overlaps with #1902636: Add character counts to translation jobs. While an improved word count is useful, the other issue would allow to rely on the character count for being not-empty.

About strlen/mbstring, that is a non-issue. That's why the drupal_*() wrapper functions exist which use the mbstring extension if available and that extension is available in most cases nowadays.

Yes, looks great as a starting point. Thanks!

I only commit this to make sure, non-latin languages work. However i don't want to have additional language specific things in TMGMT core as long as we don't have language plugins.
So i have removed your non-ascii character. Being a few words wrong isn't a problem.

Committed, pushed.

Crosspost!

Note that we can't use a drupal unicode wrapper, as there's nothing like drupal_word_count.
http://api.drupal.org/api/drupal/includes%21unicode.inc/7

However i guess we're fine with this intermediate situation and still can add the character based calculation on top of that.

Ooops :-)