Use POST instead of GET

I attached a patch to change from GET to POST request.

data is in x-www-form-urlencoded format and existing options added to the doRequest function are overwritten with drupal_http_request options that enable a POST request.

Limit is now effectively 5000 chars, as set by the api itself:
https://developers.google.com/translate/v2/faq

I had trouble applying the last patch, although the changes seem to have worked for me.

I recreated the changes, cleaned up some of the syntax formatting and then recreated the patch. I would also like to patch the tmgmt module to show the total character count, in addition to the total word count. The word count can be misleading, since the google translate API has a max character limit instead of a word limit.

Testbot bump, this might need changes in the tests.

I had spotty luck getting POST to work, and it only increases the length from 1400 to 5000 characters (which is still too small for many pages I want to translate). Attached is a patch which removes the length limit entirely, by breaking up large fields. It does not change the method from GET to POST, but it fulfills the spirit of this feature by removing length limits on translated data.

The new code works as follows.

If a field is too big (> $maxCharacters after URL encoding), it looks for HTML paragraph tags and tries to split it into small enough chunks. If a paragraph is still too big it tries to split it on sentence boundaries (delimited by "." "!" or "?"). If a single sentence is still too big it will split it on white space boundaries. If a single word is too big it leaves it untranslated. It makes a reasonable attempt to put as much stuff in each call to google as possible, to minimize latency.

I've been using this on my sites for a couple of weeks now, and it appears to work well on nodes with large bodies.

I think this patch is still useful even if the module is modified to use POST, as it will remove the 5000 character limit. To change the limit for splitting large text, simply change $maxCharacters. It is set to 1400 right now because that seems to be close to the maximum size google translate will accept.

Hope this helps.

Thanks for working on this.

+++ b/tmgmt_google.plugin.incundefined
@@ -34,7 +34,16 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+  protected $translationREs = array('/(<\/[pP][^>]*>)/',   // Paragraphs
+                                    '/([.!?])/',           // Sentences
+                                    '/((?:\s|&nbsp;)+)/'); // white space

tmgmt_word_count() has a list of characters it considers as punctuation, not all of them apply here but maybe some do?

In general, splitting text up is quite tricky, and will require a good amount of tests, also in non-english languages.

Additionally, it's a thing that many translators need, so we should try to extract this into common functions/methods so that e.g. the microsoft translator can use it. That would then live in the core module.

Might also require some thinking to come up with a good API to reduce code duplication.

+++ b/tmgmt_google.plugin.incundefined
@@ -100,18 +101,67 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+      // put in as many fields as we can...

Comments should start with an uppercase character, end with a ".", < 80 characters and consist of complete, actual english sentences.

+++ b/tmgmt_google.plugin.incundefined
@@ -100,18 +101,67 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+      // FIXX SEE IF IT WORKS LIKE THIS BEFORE MAKING MORE CHANGES

Comments like this should be removed :)

+++ b/tmgmt_google.plugin.incundefined
@@ -100,18 +101,67 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+      $qLength = 0;

Only class propertiy should use camel case, normal variables should be $q_length/$query_length (variable names are usually not shortened)

+++ b/tmgmt_google.plugin.incundefined
@@ -100,18 +101,67 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+        $fieldLength = strlen(urlencode($field)) + 3;

Drupal has wrappers for this: drupal_strlen(). That uses the mbstring extension when available.

The question, how does Google count/treat multibyte characters?

+++ b/tmgmt_google.plugin.incundefined
@@ -100,18 +101,67 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+      // FIXX this goes away
       // Split $q into chunks of self::qChunkSize.
-      foreach (array_chunk($q, $this->qChunkSize) as $_q) {
+      /* foreach (array_chunk($q, $this->qChunkSize) as $_q) {
 
         // Get translation from Google.
         $result = $this->googleRequestTranslation($job, $_q);
 
         // Collect translated texts with use of initial keys.
-        foreach ($result['data']['translations'] as $translated) {
-          $translation[$keys_sequence[$i]]['#text'] = $translated['translatedText'];
+        foreach ($result as $translated) {
+          $translation[$keys_sequence[$i]]['#text'] = $translated;
           $i++;
         }
       }

Then do it away :)

+++ b/tmgmt_google.plugin.incundefined
@@ -128,6 +178,78 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+   * Helper method to split long text into chunks and translate it.

I don't like Helper method prefixes, just state what it does, starting with a verb: "Splits long text into chunks and translates it."

+++ b/tmgmt_google.plugin.incundefined
@@ -128,6 +178,78 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+  function translateChunks($text, $index, TMGMTJob $job)

Opening { should be on the same line as the method.

+++ b/tmgmt_google.plugin.incundefined
@@ -128,6 +178,78 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+    if (preg_match('/^((<\/?[pP][^>]*>)|(<[bB][rR][^>]*\/?\s*>)|\n|\s|&nbsp;)+$/', $text)) {
+      return $text;
...
+    if ($index >= count($this->translationREs)) {
+      // Too big, and no way to split it - return text untranslated
+      return $text;

This is when it can't be splitted? We can't just return the source text then?

+++ b/tmgmt_google.plugin.incundefined
@@ -128,6 +178,78 @@ class TMGMTGoogleTranslatorPluginController extends TMGMTDefaultTranslatorPlugin
+      // print_r($match, false);
+      $nextPos = $match[1] + strlen($match[0]);
...
+        // print("continuing because this is inside a tag '" . substr($text, $translatedUpTo, $nextPos - $previousPos) . "'\n");

Debug code should be removed.

This seems to be a pretty major issue with the translator. I would suggest breaking up nodes into short sentences, then posting them one by one. Breaking at a full-stop doesn't work well (I tried it), as the full-stop is used for many, many purposes - not least as a decimal point.

This problem is very heavy and stops the module from working for many cases. How can we proceed here? It's really terrible.

I tested around a lot and was NOT successful using the POST method, anyway I think there must be a way.

The path from #6 works good, but also seems to run into limits in some cases. So my suggestion would be to clean it (#6) up like suggested and then commit it to the next dev release? I think there are many people having problems with the GETs limitations.

The patch from #6 worked for me, and I agree with @Anybody's comments in #10 and #11.