Overcoming arbitrary line lengths of .po translations

...have you ever received a merge request for source containing changed .po-files?

Proposal

Option which makes translation export seek to keep translations single-lined instead of splitting them in parts of maximum 70 chars.

Limitation

Converted .po-files must be importable by core.
Thus lines cannot be longer than 10 * 1024 minus room for gettext context (e.g. msgid_plural);
10 * 1024 maximum defined in _locale_import_read_po().

For full reasoning, and a drush script which converts existing .po files

See potx issue drush script to ease .po revision diffing.

CommentFileSizeAuthor
#1 drupal-locale-export-single-lined-2509634-1.patch1.18 KBjacobfriis
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jacobfriis’s picture

jacobfriis CreditAttribution: jacobfriis commented
FileSize
drupal-locale-export-single-lined-2509634-1.patch1.18 KB

Patch for D7.
And then:
drush vset locale_export_prefer_single_lined 1

jacobfriis’s picture

jacobfriis CreditAttribution: jacobfriis commented
Assigned: jacobfriis » Unassigned
Status: Active » Needs review
cilefen’s picture

cilefen CreditAttribution: cilefen commented

I do not know much about translations, but could this be an improvement to Drupal 8 as well?

jacobfriis’s picture

jacobfriis CreditAttribution: jacobfriis commented

Yes, it certainly would be an improvement to D8 too.

But I cannot figure out where the max. line length (if any) of translation import is defined in D8. Kind of get lost in the import bulk operation ;-)
Setting the max. for export is 'easy' and would have to be done here:

Drupal\Component\Gettext\PoItem::formatString() {
  ...
  $parts = explode("\n", wordwrap(str_replace('\n', "\\n\n", $string), 70, " \n"));
  ...
}

Without knowledge of the max. line length for import (if any) it would be hazardous to increase the max. for export.

So I would like to see if there's any interest in the issue, before venturing into more research. But if I stumble upon the solution (import max.?) then I will for sure make a D8 patch too ;-)

cilefen’s picture

cilefen CreditAttribution: cilefen commented

Actually, if this is a D8 issue, according to the backport policy, it must be fixed in 8.0.x-dev (or 8.1.x-dev if it is disruptive) first. If that is the case, the version on this issue should be changed.

jacobfriis’s picture

jacobfriis CreditAttribution: jacobfriis commented

So I should wait untill someone cares and figures out how to fix it in D7?
Look, we've got some production sites out there in the real world, which need this fix, and which are D7 based.

Currently there only exists a patch for D7, and none for D8. So it makes perfectly sense to keep the issue as D7 until someone figures out how to fix in D8. And BTW you cannot run an automated test of a D7 patch against D7 codebase if the issue is set on D8.
Not having read those policies, I'm still sure that it's perfectly alright to find and fix D7 issues as such - even if the same issue exists in D8.

If you have a fix: change the issue to D8 status and upload the patch ;-)

jacobfriis’s picture

jacobfriis CreditAttribution: jacobfriis commented

Sorry, really didn't mean to be rude. But the thing is, if we turn it into a D8 issue now, we'll have to hide the D7 patch - otherwise the D7 patch will auto-magically get tested against D8, and fail.
And if we hide the D7 patch, then folks can't see that there - at least - exists a D7 patch.
So it's a bit catch-22'ish ;-)
I will look into the matter for D8 within a week or so. And if I find a solution, I'll naturally switch the issue to D8.
But currently it wouldn't be constructive, as far as I see it.

cilefen’s picture

cilefen CreditAttribution: cilefen commented

The availability of a patch is not a consideration as to which branch an issue should be in. I am trying to help. This issue will not be committed to Drupal 7 first if it exists in Drupal 8. From the policy:

All proposed changes to Drupal core should be posted against the current development version (currently Drupal 8). This includes issues that were originally opened against previous versions - even if the original target was Drupal 6 or Drupal 7, once 8.x development starts, this is the target version for every change.

Note that even if you disagree with this, and think that posting patches against the current stable release will get them fixed faster, this will have the opposite effect

I know practically nothing about translations in general so I have nothing to say about it but it looks like Drupal 8 continues with .po files so this could be an issue with it.

jacobfriis’s picture

jacobfriis CreditAttribution: jacobfriis commented

Aye, I get your point. It shan't expect the D7 patch to merged into the D7.
And yes, as it turns out, I actually committed an infringement against policies. I really didn't know that.
So I won't do that again. Next time I have a fix to an issue (which isn't a security matter etc.) of D7 but don't have a fix to D8 I'll just have to sit on it, or release off-drupal.org.
But I cannot undo things now.

cilefen’s picture

cilefen CreditAttribution: cilefen commented

Posting the patch is good! It will certainly help.

jacobfriis’s picture

jacobfriis CreditAttribution: jacobfriis commented
Version: 7.x-dev » 8.0.x-dev
Priority: Normal » Minor
Status: Needs review » Postponed

Patch for D7 at #1.
Solution for D8 not found yet due to uncertainty about max. line length for D8 translation import (#4).

jacobfriis’s picture

jacobfriis CreditAttribution: jacobfriis commented
Status: Postponed » Active

There seem to be no max. line length for translation import

Drupal\Component\Gettext\PoStreamReader::readLine() {
  ...
  $line = fgets($this->_fd);

So a quick-n-dirty would be:

Drupal\Component\Gettext\PoItem::formatString() {
  ...
  if (\Drupal::config('locale.settings')->get('translation.export_single_lined')) {
    $split_length = (10 * 1024) - 20;
  }
  else {
    $split_length = 70;
  }
  ...
  $parts = explode("\n", wordwrap(str_replace('\n', "\\n\n", $string), $split_length, " \n"));

But that's for sure not the correct way to do it. No component class uses configuration like that.
Instead you'ld probably extend PoStreamReader and override readLine(). But I have no idea how you tell we should the overriding class ;-)

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.