Problem/Motivation
Drupal prevents loading translation strings that contain possibly dangerous tags. When you import translations or translate in the UI, each string is run through locale_string_is_safe(), and if that function returns FALSE, the string is rejected and not stored in the database.
However, both core and contributed projects can create translatable strings with these tags in them. If they do, and translators leave the tags in (as they generally should when translating strings with HTML in them), the translations will never get imported into sites.
Proposed resolution
Use a runtime assert() statement to test all translatable strings using code identical to locale_string_is_safe().
This will ensure that Drupal Core and contributed modules will not have any of these strings in their code, as long as there is at least one automated test that displays the string and all the tests pass. Which means that going forward, Core and any other projects that enforce passing automated tests before committing code, plus any projects that go back and fix tests that suddenly start failing, will be free of these untranslatable strings.
The patch will also need to fix the broken strings in Drupal core.
Remaining tasks
1. Make a patch that adds this assert to the TranslatableMarkup class, and fixes any Core strings that fail the test. [There was one such string in Core, plus another one in a test class. Both have been fixed in the patch.]
2. Review.
3. Commit.
User interface changes
Strings that are translated will not contain any "unsafe" tags that would make them not importable or not translatable in the translation UI.
API changes
A runtime assert in the TranslationMarkup constructor will verify that any translatable string has no unsafe tags, if runtime asserts are being checked (i.e., during tests only -- generally these asserts do not run on production sites).
Data model changes
No.
Release notes snippet
A runtime assert has been added to the TranslationMarkup constructor, which verifies that any translatable string has no unsafe tags. Unsafe tags are any HTML tags that are not listed in the locale_string_is_safe() function. Strings that don't pass this test are not importable when a site imports translations from localize.drupal.org, so it is important that Drupal Core and contributed projects not contain these unsafe strings. This assert will only be checked if runtime asserts are live, which is generally only true when tests are being run. Contributed project tests will see their existing tests fail after this change, if they have any unsafe strings in their code base, assuming that they have test coverage that triggers those strings to be displayed.
| Comment | File | Size | Author |
|---|---|---|---|
| #38 | 3157707-37.patch | 8.11 KB | siemen_hermans |
| #36 | 3157707-36.patch | 8.36 KB | ranjith_kumar_k_u |
| #35 | 3157707-nr-bot.txt | 144 bytes | needs-review-queue-bot |
| #26 | 3157707-26.patch | 8.31 KB | jhodgdon |
| #26 | interdiff24-26.txt | 1.24 KB | jhodgdon |
Issue fork drupal-3157707
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
gregglesMaking the html tags show up in the issue summary and adjusting title. It's for XSS and other reasons like defacement and phishing.
Comment #3
jhodgdonOK... I think you are saying that there is a list of HTML tags that, effectively, cannot or should not be used in any translatable strings, because if they are translated on localize.drupal.org, then those translations are imported into a site, they will be skipped. Meaning those strings are effectively not translatable with our tools. Right?
Then should we generate errors in POTX when such strings are read from projects, and make issues in projects to remove them? It would be good do document this somewhere too?
Comment #4
jhodgdonAnd/or do we need a Coder rule to detect this?
Comment #5
jhodgdonComment #6
joachim namysloThat's a brilliant summary of the Issue. Jennifer. A method/module to filter and list affected strings is needed, because currently 114+ languages are affected of this issue in several modules and themes. Detect them manually would take at least a decade or so.
Comment #7
jhodgdonAdding issue summary.
Comment #8
joachim namysloComment #9
tobiasbI believe div/code are not the problems. Attributes inside a div (
<div class='alert alert-warning alert-sm'><strong>WARNUNG:</strong>)or invalid html like<front>.Comment #10
tobiasbAll html-elements which are not part of the list in locale_string_is_safe() should be detected. perhaps a fancy combination of potx + coder.
Comment #12
joachim namysloComment #13
joachim namyslo#10 I'd like it if that would be true. But take this example here:
https://localize.drupal.org/translate/languages/de/translate?sid=2
You can translate it, aprove the translation and install the varbase distribution Just to verify that this String will get filtered.
When it comes to technology there is no way to believe something but there are many different ways to prove it.
I did this with the string above in particular so I know that presumption is correct but doesn't apply to that issue in all cases.
Comment #14
jhodgdonHere's a thought about how we could enforce this in Drupal core at least.
When anything is run through t() and the other translation functions in Drupal, the actual string translation ends up being run through the 'string_translation' service.
So, we could instrument all Functional tests (the ones that visit UI pages) with their own string translation service class, which could override the doTranslate() method so that it runs all the strings that come in through locale_string_is_safe(), and flags an error if they fail that test. This could be run even if the locale module wasn't turned on.
Comment #15
jhodgdonHere's a patch that implements #14, more or less. What it does: Unless a test using BrowserTestBase overrides the default member variable
$strictStrings = TRUE, every translatable string that is rendered during the test is checked to see that it is locale-safe. I added this to the existing FunctionalTestSetupTrait, which is also where the config strict checking is done. As the patch is now, it would also affect any contrib modules that run BrowserTestBase tests.I verified locally that I could make a test fail by editing the code so that there was a string rendered with a
<div>tag in it. Then when I removed that line from my code, my test passed. (Eventually, this patch probably needs to have a unit or functional test in it that verifies that the test is actually working right, by introducing a string that isn't locale safe and verifying an exception is thrown, but let's see if this is even a viable approach first.)So... Let's see what happens when we run the Core tests. Are there locale-unsafe strings in Core? If so, they should fail the tests and we should fix them. I expect there might be some extraneous fails too if there are some tests in Core that are using locale-unsafe strings to test something.
Comment #16
jhodgdonFix code sniff.
Comment #18
jhodgdonTwo tests in the System module apparently have non-locale-safe strings being used in the tests. So, for those tests, disable checking. Also fix up some docs.
Comment #19
jhodgdonOK, tests pass now... To verify that this test addition would have caught non-locale-safe strings, I made up a should-fail patch by adding a div tag in some HTML output for a page that hopefully is displayed in a test. The real patch still probably needs a test for the test?
Comment #20
jhodgdonThe test bar says "Build successful", but if you click through to the results on that last patch, you'll see that about 15 tests failed due to the div tag in the string. So... Any thoughts about this approach?
Comment #21
jhodgdonI discussed this in Slack today with @alexpott and he suggested putting an assert into the TranslatableMarkup constructor instead of the current approach of an exception in a custom translator. I'm working on a patch for that.
Comment #22
jhodgdonHere's a patch with this new approach. The tests pass locally for me, with an update to the Breadcrumb test module so that it doesn't use an non-locale-safe string.
Interdiff is not useful -- this is a completely new approach and every line of the patch is different from the previous patch.
Comment #24
jhodgdonOK! This is good. The new test found an actual error -- a typo in a string in the node module where it had
<br/>instead of<br />. Here's a fix.Comment #26
jhodgdonLooks like 1 test had the bad string from node module hard-wired in it.
Comment #27
jhodgdonUpdating issue summary. Also made a draft change record. https://www.drupal.org/node/3199803
I believe this is ready for review.
Comment #28
jhodgdonUpdating title too.
Comment #30
joachim namysloComment #34
joachim namysloComment #35
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #36
ranjith_kumar_k_u commentedRe-rolled #26
Comment #38
siemen_hermans commentedPatch compatible with version 11.3.2
Comment #42
jurgenhaas