Problem/Motivation

PHP wordwrap() function handles UTF-8 incorrectly. It trims Unicode texts twice shorter. There are no mb_ version either.

I compilated a drupal_wordwrap() function from php.net examples, and supplied it with $cut parameter. Tested on Russian, works perfect.

I guess, the phpversion() checking is redundant, but I have no idea how to tweak this part, so I'm ready to listen any suggestion.

Please review, the patch.

Steps to reproduce

Proposed resolution

Remaining tasks

First, write a test for MailFormatHelper using UTF-8.

User interface changes

N/A

API changes

N/A

Data model changes

N/A

Release notes snippet

CommentFileSizeAuthor
#40 unicode_safe_wordwrap-540228-40.patch7.11 KBsuperspring
#36 patch_d7_wordwrap.patch3.09 KBmaximn
#35 patch_d7_wordwrap.patch3.09 KBmaximn
#33 540228_Implement_UTF8-safe_wordwrap-33.patch2.73 KBhaza
#8 drupal_wordwrap_0.patch2.43 KBneochief
#6 drupal_wordwrap_6_13.patch2.86 KBOnkelTem
#4 drupal_wordwrap.patch2.66 KBneochief
#1 drupal_wordwrap-D6.patch2.7 KBneochief
drupal_wordwrap.patch3.49 KBneochief

Comments

neochief’s picture

neochief commented
StatusFileSize
new drupal_wordwrap-D6.patch2.7 KB

Patch for 6.x, just in case.

lilou’s picture

lilou commented
Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch failed testing.

neochief’s picture

neochief commented
Status: Needs work » Needs review
StatusFileSize
new drupal_wordwrap.patch2.66 KB

Update for TB.

Status: Needs review » Needs work

The last submitted patch failed testing.

OnkelTem’s picture

OnkelTem commented
StatusFileSize
new drupal_wordwrap_6_13.patch2.86 KB

Updating patch for D6.13

OnkelTem’s picture

OnkelTem commented

There is an error in the patch in drupal_wordwrap function.
Add 'u' modifier to this code:

    if (version_compare(phpversion(), '5.2.3') > 0) {
      $str = preg_split('#\h+#', $str);
    }
    else {
      $str = preg_split('# +#', $str);
    }

to get:

    if (version_compare(phpversion(), '5.2.3') > 0) {
      $str = preg_split('#\h+#u', $str);
    }
    else {
      $str = preg_split('# +#u', $str);
    }

Not tested much, but this is fixing breaking my line:

РАЗВИТИЕ КОРПОРАТИВНОГО САЙТА => � АЗВИТИЕ КО� ПО� АТИВНОГО САЙТА

neochief’s picture

neochief commented
Status: Needs work » Needs review
StatusFileSize
new drupal_wordwrap_0.patch2.43 KB

Thanks for the addition, updating patch for 7.x (let's port it to earlier version, when 7.x will be ready).

Status: Needs review » Needs work

The last submitted patch failed testing.

sreynen’s picture

sreynen commented
Status: Needs work » Needs review

Automated tests failed to run, setting back to needs review to try again.

sreynen’s picture

sreynen commented

Automated tests failed to run, setting back to needs review to try again.

Status: Needs review » Needs work

The last submitted patch failed testing.

neochief’s picture

neochief commented

Something wrong with the test. Patch isn't fail, it's just fail to test.

Status: Needs work » Needs review

Re-test of drupal_wordwrap_0.patch from comment #8 was requested by mcarbone.

Status: Needs review » Needs work

The last submitted patch, drupal_wordwrap_0.patch, failed testing.

kaakuu’s picture

kaakuu commented

Subscribed.

good_man’s picture

good_man commented

Subscribed.

good_man’s picture

good_man commented

I see that the last failed test gave the following message:
Ensure the patch only contains unix-style line endings.

I reviewed it and all end of line characters are Unix-style. maybe you resubmit it to see if it'll pass this time.

grendzy’s picture

grendzy commented
Priority: Critical » Normal
Antton’s picture

Antton commented
Version: 7.x-dev » 6.15

As a result of using the patch - at the end of each line by two spaces and transfer to the next line.
As a result, when reading the letter e-mail client displays it as a message in one line.

grendzy’s picture

grendzy commented
Version: 6.15 » 7.x-dev
Issue tags: +Needs backport to D6
Antton’s picture

Antton commented
Version: 7.x-dev » 6.16

The patch fixes the problstrong, but partly. At the end of each row are whitespace. A problstrong with the transfer lines in the mail client Mozilla Thunderbird. Mozilla Thunderbird prolongs string if the end of the line is whitespace.

Original letter:

username,__

Thank you for registering at site. You may now log in to login uri_
using the following username and password:__

username: username__
password: password__

You may also log in by clicking on this link or copying and pasting it_
in your browser:__

login url__

This is a one-time login, so it can be used only once.__

After logging in, you will be redirected to edit uri so you can_
change your password.__

-- site team__

Shows Mozilla Thunderbird:

username,
Thank you for registering at site. You may now log in to login uriusing the following username and password:
username: username password: password
You may also log in by clicking on this link or copying and pasting itin your browser:
login url
This is a one-time login, so it can be used only once.
After logging in, you will be redirected to edit uri so you canchange your password.
-- site team

How to fix a patch to the end of the line, where we need a new line, - no whitespace? Thanks.

grendzy’s picture

grendzy commented
Version: 6.16 » 7.x-dev

Please do not change the version. Bugs must be fixed in the latest development version before backporting.
http://drupal.org/node/10262

Antton’s picture

Antton commented
Version: 7.x-dev » 6.16
Status: Needs work » Closed (fixed)

Solved the problem

Added the line:
$ return = trim ($ return);

Before:
return $ return;

It removes the trailing spaces in text strings __
Now Mozilla Thunderbird normally displays the letter.

sreynen’s picture

sreynen commented
Version: 6.16 » 7.x-dev
Status: Closed (fixed) » Needs work

Changing status and version back; doesn't sound like this is fixed.

damien tournoud’s picture

damien tournoud commented
Title: wordwrap() & UTF-8 » Implement a UTF8-safe wordwrap()
Version: 7.x-dev » 8.x-dev
Issue tags: +Needs backport to D7
pillarsdotnet’s picture

pillarsdotnet commented

There is a UTF-8 compatible wordwrap included with #299138: Improve \Drupal\Core\Utility\Mail::htmlToText()

Andreyy’s picture

Andreyy commented
Status: Needs work » Needs review

drupal_wordwrap.patch queued for re-testing.

good_man’s picture

good_man commented
Issue tags: +D8MI

Tagging for D8 Multilingual Initiative.

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented
Issue tags: +Needs tests

Invalid patch format, since the patch does nto apply anymore. Also needs tests.

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented
Status: Needs review » Needs work
gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented
Issue tags: +language-base

Adding base language system tag.

haza’s picture

haza
Primary language French
commented
Status: Needs work » Needs review
StatusFileSize
new 540228_Implement_UTF8-safe_wordwrap-33.patch2.73 KB

Implemented that over D8. Let's see what the bot thinks about that...

(no tests added for now)

Status: Needs review » Needs work

The last submitted patch, 540228_Implement_UTF8-safe_wordwrap-33.patch, failed testing.

maximn’s picture

maximn commented
StatusFileSize
new patch_d7_wordwrap.patch3.09 KB
maximn’s picture

maximn commented
Version: 8.x-dev » 7.x-dev
Status: Needs work » Needs review
StatusFileSize
new patch_d7_wordwrap.patch3.09 KB

Patch for for D7

Status: Needs review » Needs work

The last submitted patch, patch_d7_wordwrap.patch, failed testing.

marcingy’s picture

marcingy commented
Version: 7.x-dev » 8.x-dev

Please keep version at d8.

jair’s picture

jair commented
Issue tags: +Needs reroll

Needs reroll

superspring’s picture

superspring commented
Status: Needs work » Needs review
StatusFileSize
new unicode_safe_wordwrap-540228-40.patch7.11 KB

This patch is a tidied up version of some of the previous patches.
- Code has been rearranged (into the Unicode class)
- More wordwrap uses have been found and updated
- Code has been commented far more.

Status: Needs review » Needs work

The last submitted patch, unicode_safe_wordwrap-540228-40.patch, failed testing.

ekl1773’s picture

ekl1773
she/her
Primary language English
commented
Issue tags: -Needs reroll

Does not need reroll! Applies to Head at 902b741

oriol_e9g’s picture

oriol_e9g
Primary language Catalan
Location Barcelona
commented
Status: Needs work » Needs review
Issue tags: -Needs backport to D6, -Needs tests, -Needs backport to D7, -D8MI, -language-base

#40: unicode_safe_wordwrap-540228-40.patch queued for re-testing.

Status: Needs review » Needs work

The last submitted patch, unicode_safe_wordwrap-540228-40.patch, failed testing.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

quietone’s picture

quietone commented
Version: 8.9.x-dev » 9.4.x-dev
Issue summary: View changes
Issue tags: +Bug Smash Initiative

@neochief, thanks for the report and the initial patch.

Closed #2754669: Chinese/Japanese text wrapping as a duplicate.

This still seems to be valid. I didn't see a test of UTF8 for MailFormatHelper. 

$ grep -ri wordwrap core | grep .php
core/tests/Drupal/Tests/Core/Mail/MailFormatHelperTest.php:    $this->assertEquals(wordwrap($body, 77, " \n"), $processed_body, 'Body text is wrapped.');
core/lib/Drupal/Component/Gettext/PoItem.php:    $parts = explode("\n", wordwrap(str_replace('\n', "\\n\n", $string), 70, " \n"));
core/lib/Drupal/Core/Mail/MailFormatHelper.php:      $line = wordwrap($line, 77 - $values['length'], $values['soft'] ? " \n" : "\n");
core/lib/Drupal/Core/Mail/MailFormatHelper.php:    $line = wordwrap($line, 996 - $values['length'], $values['soft'] ? " \n" : "\n", TRUE);

Talked to darvanen in #bugsmash and we weren't able to figure out what the stringwrappers are doing.

As a first step, let's get a test of MailFormatHelper with a UTF-8 source.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

chi’s picture

chi commented

Still relevant in Drupal 11.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.