UPDATE: Editing the data bank is probably too risky, and a solution is best found in the contrib module Pathauto, see #1811856: Add transliteration options for special characters such as æ, ø, å, ä, ü, etc..
When generating URL path aliases from nodes with Danish special characters in the title, the characters 'å' and 'ø' gets transliterated incorrectly to 'a' and 'o' instead of 'aa' and 'oe' when node language is English. Danish language nodes work as expected.
A fair share of Drupal installation may be installed with only English as language, but the content is in reality Danish, Norwegian, Icelandic, Faroese or Swedish. In such cases, æ will be transliterated correctly, but not ø or å, which is inconsistent. Also, this can cause slight differences in URL aliases for multilingual sites, where you prefer them to be streamlined, see #3157288: German letter "ä", "ü" in translation and Drupal 8 pathauto and Greek characters.
Example:
A node with the title "1æ 2ø 3å 4Æ 5Ø 6Å" gets these paths, first in a node with language Danish, and then English:
/da/1ae-2oe-3aa-4ae-5oe-6aa
/en/1ae-2o-3a-4ae-5o-6a
The English language node should get this path (identical with the Danish node, except for en bit):
/en/1ae-2oe-3aa-4ae-5oe-6aa
Æ and æ
The Danish alphabet contains the special characters æ, ø, and å, but Æ and æ are missing in the core/lib/Drupal/Component/Transliteration/data/da.php file, probably because they are already covered in the Generic transliteration data file x00.php:
The standard English transliteration defines the transliteration for Æ and æ correctly on this line, at position #7, as 'AE' (0xC6):
0xC0 => 'A', 'A', 'A', 'A', 'A', 'A', 'AE', 'C', 'E', 'E', 'E', 'E', 'I', 'I', 'I', 'I',
... and æ at position #7, as 'ae' (0xE6):
0xE0 => 'a', 'a', 'a', 'a', 'a', 'a', 'ae', 'c', 'e', 'e', 'e', 'e', 'i', 'i', 'i', 'i',
The core/lib/Drupal/Component/Transliteration/data/da.php file:
<?php
/**
* @file
* Danish transliteration data for the PhpTransliteration class.
*/
$overrides['da'] = [
0xC5 => 'Aa',
0xD8 => 'Oe',
0xE5 => 'aa',
0xF8 => 'oe',
];
Possibly include umlaut letters from German language as well? See #3157288: German letter "ä", "ü" in translation, Conversion table for diacritics (e.g. "ü" → "ue"), and How to handle URLs with diacritic characters.
If the characters æ, ø and å were defined in the generic transliteration data file, it would help Drupal sites where the content is in Danish, Norwegian, Icelandic, Faroese, and Swedish, but "housed" in an English language Drupal installation, because they then won't have to create a custom module.
It will also streamline URL aliases between languages for multilingual Drupal installations.
Set up transliteration in Drupal, so that these substitutions take place in all languages, English, Danish, Spanish, Norwegian, Swedish, etc.:
æ > ae
Æ > ae
Ø > oe
ø > oe
Å > aa
å > aa
Possibly include umlaut and special letters from German language as well?
ä > ae
ö > oe
ü > ue
Ä > Ae
Ö > Oe
Ü > Ue
ß > ss
... or should it be handled in another issue?
Until this gets fixed, I found the transliteration overrides function, which does the trick.
Note, that if you are using this with a migration, you need to remove the nodes completely with a rollback. The aliases seem to stay, even with the --delete parameter :)
/**
* Implements hook_transliteration_overrides_alter().
*/
function mymodule_transliteration_overrides_alter(&$overrides, $langcode) {
// Transliterate Danish letters æ, å, and ø for English-language nodes.
if ($langcode == 'en') {
$overrides = [
0xC5 => 'Aa',
0xD8 => 'Oe',
0xE5 => 'aa',
0xF8 => 'oe',
0xC6 => 'Ae',
0xE6 => 'ae',
];
}
}From https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Language%...
x00.php file| Comment | File | Size | Author |
|---|
Show commandsStart within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes, plain diff MR !12593
Comments
Comment #2
ressaComment #3
ressaAdding workaround.
Comment #4
xjmI'm confused by the title --
åandøappear to be in the override file whileæis not. Shouldn't the title mentionæinstead, if that's what the issue is about?However, isn't the underlying difference though that
åandøare pronounced very differently from the English letters they most closely resemble, whereas the standard transliteration foræin most languages including English itself is alreadyae? That's why the override file was needed for the first two but not the third.In
drupal/core/lib/Drupal/Component/Transliteration/data/x00.php, the standard transliterations ofAEandaefor0xC6and0xE6already appear at the expected positions, at least as far as I understand how it works.Can you provide manual testing steps to more clearly illustrate the bug, or a test case?
Comment #6
ressaThanks for looking at this @xjm. I have previously opened the transliteration mapping file
core/lib/Drupal/Component/Transliteration/data/x00.phpbut didn't get it ... But I now understand the structure after checking in https://byte-tools.com/en/ascii/ -- that0xc0is the first value,0xc1is number two, etc.Like you write, for some reason, the standard English transliteration defines the transliteration for Æ and æ correctly on this line, at position #7, as
'AE'(0xC6):0xC0 => 'A', 'A', 'A', 'A', 'A', 'A', 'AE', 'C', 'E', 'E', 'E', 'E', 'I', 'I', 'I', 'I',... and æ at position #7, as
'ae'(0xE6):0xE0 => 'a', 'a', 'a', 'a', 'a', 'a', 'ae', 'c', 'e', 'e', 'e', 'e', 'i', 'i', 'i', 'i',I am not sure what the thought behind transliterating æ, å and ø differently is ... But the problem is, quite often a Drupal installation may be installed with only English language, but the content is in reality Danish, Norwegian, Icelandic, Faroese or Swedish. In such cases, æ will be transliterated correctly, but not ø or å, which is inconsistent. Also, this can cause slight differences in URL aliases for multilingual sites, where you prefer them to be streamlined, as can be seen in for example #3157288: German letter "ä", "ü" in translation.
But if the characters æ, ø and å were defined in the generic transliteration data file, it would help Danish, Norwegian, Icelandic, Faroese and Swedish language web sites in an English Drupal installation to not have to create a custom module, as well as streamline URL aliases between languages, and I have created an MR.
I tried removing the
da.phpfile but got weird test results, so I restored it.Maybe there should also be mapping added for Aa, Ae and Oe? I am not sure in which of the data files this needs to be done, though.
Comment #7
ressaAdding some more context in the Issue Summary, in an attempt to make it clearer what the challenge is.
Comment #8
ressaAdding
x00.phplines with æ in Issue Summary.Comment #9
xjmComment #10
xjmThanks @ressa for the excellent IS update. (Just adding some punctuation; disregard my small changes.)
Comment #11
xjmThis is one of those things best reviewed with
git diff --color-words, so that's what I did. Most of the changes look correct but there are a couple I don't understand.The changes to the unicode table look correct and reflect the summary:
Several tests change transliterations in the expected ways
But this one seems to be transliterating
åtoainstead ofaa. Kan du hjelpe meg å forstå hvorfor? (See what I did there? Forgive any grammatical errors; I only had one semester of Norwegian.)And then this I do not get at all in
PhpTransliterationTest:We seem to be adding
åandøinstead of removing them?I will try to understand better, but given the scope has expanded to correcting the base table, let's ask for subsystem review here too.
Thanks a bunch!
Comment #12
xjmComment #13
ressaThanks for a fast response and thorough review @xjm, I really appreciate it!
Great idea using coloured diff's, they really help spot the changes. Og godt norsk efter kun et semester, jeg forstår det! (jeg håber du kan læse dansk)
About item #3 in your list (
modules/search/tests/src/Functional/SearchNodeDiacriticsTest.php), I could not figure out why the test broke ... it is looking for the string "påŔťıçȉpǎǹţș" which seems present in the page. I figured that since the test tests "diacritics in the search phrase." that I could simply change "å" to "a", because I mistakenly assumed "å" isn't a diacritic.But I now realize that "å" is indeed also a diacritic, so I reverted the Diacritic test in SearchNodeDiacriticsTest.php (#3 in your list) back to its original state, as well as the "Test all characters in the Unicode" bit in PhpTransliterationTest.php (#4 in your list), so these tests will now fail.
It is interesting that Æ is included in x00.php, but not Ø, since it is not a diacritic, as I understand it.
Great idea to get some more eyes on this, by requesting a subsystem maintainer review, since I don't know if the approach in the current MR is the correct method, or even if it's a good idea to start with, maybe it has unintended consequences ...?
I have added some more things to consider in the Issue Summary, under "Remaining tasks". Thanks so far!
Comment #14
xjmJa! Jeg vet du har den poteten i halsen, men jeg kan lese dansk greit.
Test fail is:
So, yeah, I think we'll want @amateescu's guidance here on whether to change the test expected output, add additional test coverage, change code, or what. 🤷♀️
Edit: I haven't had a chance to review the updated MR yet but will return to it when I can.
Comment #15
ressaHa, ja det synes svenskerne og nordmændene jo vi har :)
Yes, it's probably best to wait and see what @amateescu thinks before proceeding. Reviewing the updated MR can probably wait as well, since the current approach could be breaking a well planned structure.
Comment #16
ressaAdding question if German letters should be included, and links to related issues.
Comment #17
ghost of drupal pastI would urge extreme caution in changing core/lib/Drupal/Component/Transliteration/data/x00.php or anything else in the data bank. The problem is that without a language code you have no idea what ø actually is. It might be the Danish character -- or it might not be.
Changing language specific overrides are much more safe.
Yes, this is not ideal. Transliteration never is :(
Comment #18
ressaThanks for sharing your insight, and clarifying that editing the data bank is probably too risky ... It's great to get sorted out, since we can then move on to find another solution.
It would then seem like this is best handled in Pathauto, by adding something like the drop-downs under Punctuation > Double quotation marks (") like suggested in #1811856: Add transliteration options for special characters such as æ, ø, å, ä, ü, etc., for æ, ø, and å.
So closing this issue, and hope it can happen in that other issue :)