Problem/Motivation

Refactor PhpTransliterationTest

Follow #3000630: Transliteration causes 2 capital letters at the beginning of a word we need tests:

  • Do not strip spaces from the beginning and end of a string.
  • Do not reduce multiple spaces between words.

Proposed resolution

  • Add the necessary tests.
  • For better readability let's join testTransliterationUnknownCharacter, testTransliterationWithMaxLength and testPhpTransliteration. They have the same parameters.

Remaining tasks

None

Comments

Krzysztof Domański created an issue. See original summary.

krzysztof domański’s picture

Assigned: krzysztof domański » Unassigned
Status: Active » Needs review
Parent issue: » #3000630: Transliteration causes 2 capital letters at the beginning of a word
StatusFileSize
new3.41 KB
krzysztof domański’s picture

-      // Max length.
-      ['de', $two_byte, 'Ae Oe', '?', 5],

Why removed? Because the same is added (irrelevant difference 5 => 17):

$two_byte = 'Ä Ö Ü Å Ø äöüåøhello';
+  public function providerTestTransliterationWithMaxLength() {
+    return [
+      // Each test case is (language code, input, output, unknown character, max
+      // length).
+      // It should never split up the transliteration of a single character.
+      ['de', 'Ä Ö Ü Å Ø äöüåøhello', 'Ae Oe Ue A O aeoe', '?', 17],
+      ['de', 'Ä Ö Ü Å Ø äöüåøhello', 'Ae Oe Ue A O aeoe', '?', 18]
krzysztof domański’s picture

Issue summary: View changes
Issue tags: +Quick fix
StatusFileSize
new5.1 KB

Update issue summary.

Anonymous’s picture

Status: Needs review » Reviewed & tested by the community
alexpott’s picture

Status: Reviewed & tested by the community » Needs work

Nice work!

There's one more thing to do in this refactor... we can make the provider use array keys instead of comments - this makes the PHPUnit descriptions much better when running the tests.

krzysztof domański’s picture

Status: Needs work » Needs review
StatusFileSize
new9.15 KB
new6.94 KB

Adds array keys instead of comments.

alexpott’s picture

Here's the new return... easier to review than a diff...

    return [
      'Test ASCII in English.' => [
        'en', $random, $random
      ],
      'Test ASCII in some other language with no overrides.' => [
        'fr', $random, $random
      ],
      // Test 3 and 4-bytes characters in a language without overrides.
      // Note: if the data tables change, these will need to change too! They
      // are set up to test that data table loading works, so values come
      // directly from the data files.
      'Test 3 bytes characters.' => [
        'fr', $three_byte, 'c'
      ],
      'Test 4 bytes characters.' => [
        'fr', $four_byte, 'wii'
      ],
      'Test 5 bytes characters.' => [
        'en', $five_byte, '??'
      ],
      'Test a language with no overrides.' => [
        'en', $two_byte, 'A O U A O aouaohello'
      ],

      // Test language overrides provided by core.
      ['de', $two_byte, 'Ae Oe Ue A O aeoeueaohello'],
      ['de', $random, $random],
      ['da', $two_byte, 'A O U Aa Oe aouaaoehello'],
      ['da', $random, $random],
      ['kg', $three_byte, 'ts'],

      'Test strings in some other languages.' => [
        'tr', 'Abayı serdiler bize. Söyleyeceğim yüzlerine. Sanırım hepimiz aynı şeyi düşünüyoruz.', 'Abayi serdiler bize. Soyleyecegim yuzlerine. Sanirim hepimiz ayni seyi dusunuyoruz.'
      ],

      // Do not split up the transliteration of a single character.
      ['de', 'Ä Ö Ü Å Ø äöüåøhello', 'Ae Oe Ue A O aeoe', '?', 17],
      ['de', 'Ä Ö Ü Å Ø äöüåøhello', 'Ae Oe Ue A O aeoe', '?', 18],

      // Not affecting spacing.
      'Do not strip spaces from the beginning and end of a string.' => [
        'en', ' Hello world! ', ' Hello world! '
      ],
      'Do not strip spaces from the beginning of a string.' => [
        'pl', ' Drupal Kraków Community', ' Drupal Krakow', '?', 14
      ],
      'Do not strip spaces from the end of a string.' => [
        'pl', 'Drupal Kraków Community', 'Drupal Krakow ', '?', 14
      ],
      'Do not reduce multiple spaces between words.' => [
        'en', 'Too    much    spaces between words !', 'Too    much    spaces between words !',
      ],

      // Illegal/unknown unicode.
      'Illegal/unknown unicode.' => [
        'en', chr(0xF8) . chr(0x80) . chr(0x80) . chr(0x80) . chr(0x80), '?????'
      ],
      'Illegal/unknown unicode with non default replacement.' => [
        'en', chr(0xF8) . chr(0x80) . chr(0x80) . chr(0x80) . chr(0x80), '-----', '-'
      ],
      'Illegal/unknown unicode inside the word.' => [
        'en', 'Hel' . chr(0x80) . 'o World', 'Hel?o World'
      ],
      'Illegal/unknown unicode at the end of the word.' => [
        'en', 'Hell' . chr(0x80) . ' World', 'Hell? World'
      ],
      'Non default replacement.' => [
        'en', chr(0x80) . 'ello World', '_ello World', '_'
      ],
      'Non-US-ASCII replacement in English.' => [
        'en', chr(0x80) . 'ello World?', 'Oello World?', 'Ö'
      ],
      'Non-US-ASCII replacement in some other language.' => [
        'pl', chr(0x80) . 'óóść', 'ooosc', 'ó'
      ],
      'Empty replacement.' => [
        'en', chr(0x80) . 'ello World', 'ello World', ''
      ],

      // Keep the original question marks.
      'Keep the original question marks.' => [
        'en', chr(0xF8) . '?' . chr(0x80), '???'
      ],
      'Keep the original question marks when non default replacement.' => [
        'en', chr(0x80) . 'ello ? World?', '_ello ? World?', '_'
      ],
      'Keep the original question marks in some other language.' => [
        'pl', 'aąeę' . chr(0x80) . 'oółżźz ?', 'aaee?oolzzz ?'
      ],
      'Ensure question marks are replaced when max length used.' => [
        'en', chr(0x80) . 'ello ? World?', '_ello ?', '_', 7
      ],
    ];

This shows not all tests have keys... I think we should fix that here. Also if the comments are duplicates then imo they are not useful.

alexpott’s picture

FILE: ...rupal/Tests/Component/Transliteration/PhpTransliterationTest.php
----------------------------------------------------------------------
FOUND 0 ERRORS AND 22 WARNINGS AFFECTING 22 LINES
----------------------------------------------------------------------
 122 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: $random
 125 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: $random
 132 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'c'
 135 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'wii'
 138 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: '??'
 141 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'A O U A O aouaohello'
 152 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'Abayi serdiler bize. Soyleyecegim
     |         |     yuzlerine. Sanirim hepimiz ayni seyi
     |         |     dusunuyoruz.'
 161 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: ' Hello world! '
 164 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 14
 167 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 14
 175 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: '?????'
 178 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: '-'
 181 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'Hel?o World'
 184 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'Hell? World'
 187 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: '_'
 190 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'Ö'
 193 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'ó'
 196 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: ''
 201 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: '???'
 204 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: '_'
 207 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 'aaee?oolzzz ?'
 210 | WARNING | [x] A comma should follow the last multiline array
     |         |     item. Found: 7
----------------------------------------------------------------------
PHPCBF CAN FIX THE 22 MARKED SNIFF VIOLATIONS AUTOMATICALLY
----------------------------------------------------------------------

Also need to fix the coding standards... you can run these locally doing something like $ composer run phpcs -- core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php

krzysztof domański’s picture

Issue tags: -Quick fix
StatusFileSize
new9.75 KB

@alexpott thanks for the feedback.

The coding standards have been fixed and unnecessary comments removed. All tests have keys now.

krzysztof domański’s picture

StatusFileSize
new1.19 KB
new9.6 KB
new9.6 KB

The patch won't apply in previous dev version. That is because #2895315: Danish characters are not translated correctly with transliteration was pushed only to 8.7.x. Both versions have a slight but significant difference:

--- a/core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php
+++ b/core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php
@@ -136,8 +136,8 @@ public function providerTestPhpTransliteration() {
-      ['dk', $two_byte, 'A O U Aa Oe aouaaoehello'],
-      ['dk', $random, $random],
+      ['da', $two_byte, 'A O U Aa Oe aouaaoehello'],
+      ['da', $random, $random],

It requires here an additional patch for 8.6.x-dev.

Anonymous’s picture

Status: Needs review » Reviewed & tested by the community
alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php
    @@ -104,7 +104,7 @@ public function providerTestPhpTransliteration() {
    -    // Note that the 3-byte character is overridden by the 'kg' language.
    +    // Note that the 3-bytes character is overridden by the 'kg' language.
    

    The original is in my opinion correct here - it is a 3-byte character

  2. +++ b/core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php
    @@ -118,101 +118,93 @@ public function providerTestPhpTransliteration() {
    +      'Test 3 bytes characters from data table in a language without overrides.' => [
    

    3-byte

  3. +++ b/core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php
    @@ -118,101 +118,93 @@ public function providerTestPhpTransliteration() {
    +      'Test 4 bytes characters from data table in a language without overrides.' => [
    

    4-byte

  4. +++ b/core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php
    @@ -118,101 +118,93 @@ public function providerTestPhpTransliteration() {
    +      'Test 5 bytes characters not existing in the data table.' => [
    

    5-byte

  5. +++ b/core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php
    @@ -118,101 +118,93 @@ public function providerTestPhpTransliteration() {
    +      'Test 2 bytes characters in a language without overrides.' => [
    

    2-byte

  6. English plurals are awful and the language follows very few rules - http://www.wargs.com/misc/poem.html
krzysztof domański’s picture

Status: Needs work » Reviewed & tested by the community
StatusFileSize
new2.16 KB
new9.04 KB
new9.04 KB

Only grammatical correction so back to RTBC. Another patch for 8.6.x-dev has also been added.

larowlan’s picture

Status: Reviewed & tested by the community » Needs review
+++ b/core/tests/Drupal/Tests/Component/Transliteration/PhpTransliterationTest.php
@@ -88,8 +88,8 @@ public function providerTestPhpTransliterationRemoveDiacritics() {
-    $transliterator_class = new PhpTransliteration();
-    $actual = $transliterator_class->transliterate($original, $langcode, $unknown_character, $max_length);
+    $transliteration = new PhpTransliteration();
+    $actual = $transliteration->transliterate($original, $langcode, $unknown_character, $max_length);

out of scope?

Can you upload a patch without the changes to the indentation and wrapping as its very hard to see what has actually changed here?

krzysztof domański’s picture

Can you upload a patch without the changes to the indentation and wrapping

I have no idea how to do it easy.

krzysztof domański’s picture

Title: Refactor PhpTransliterationTest » Not affecting spacing in PhpTransliterationTest
Issue tags: +Quick fix
StatusFileSize
new4.26 KB

Can you upload a patch without the changes to the indentation and wrapping as its very hard to see what has actually changed here?

New patch only with significant changes needed by #3000630: Transliteration causes 2 capital letters at the beginning of a word.

  • Not affecting spacing.
  • Join unnecessary additional functions.

Notice: This patch (in contrast to the previous ones) will be applied without problem in 8.6.x-dev.

Anonymous’s picture

Status: Needs review » Reviewed & tested by the community

Looks good!

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed a09ce7750a to 8.7.x and 94bfb9e5db to 8.6.x. Thanks!

Can we get a follow-up to add the array keys in again now it will be simpler to review. They will be helpful.

  • alexpott committed a09ce77 on 8.7.x
    Issue #3015992 by Krzysztof Domański, alexpott, larowlan: Not affecting...

  • alexpott committed 94bfb9e on 8.6.x
    Issue #3015992 by Krzysztof Domański, alexpott, larowlan: Not affecting...
krzysztof domański’s picture

Title: Not affecting spacing in PhpTransliterationTest » Refactor PhpTransliterationTest
Status: Fixed » Needs review
StatusFileSize
new6.98 KB
new6.98 KB

All tests have keys now. No change in asserts and they have the same order.

It requires here an additional patch for 8.6.x-dev. Why? #11

alexpott’s picture

Title: Refactor PhpTransliterationTest » Not affecting spacing in PhpTransliterationTest
Status: Needs review » Fixed

@Krzysztof Domański please file a new issue. We have one commit per issue. I should have been more explicit that "Can we get a follow-up" means create a new issue. Sorry.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.