Problem/Motivation

Drupal prevents loading translation strings that contain possibly dangerous tags. When you import translations or translate in the UI, each string is run through locale_string_is_safe(), and if that function returns FALSE, the string is rejected and not stored in the database.

However, both core and contributed projects can create translatable strings with these tags in them. If they do, and translators leave the tags in (as they generally should when translating strings with HTML in them), the translations will never get imported into sites.

Proposed resolution

Use a runtime assert() statement to test all translatable strings using code identical to locale_string_is_safe().

This will ensure that Drupal Core and contributed modules will not have any of these strings in their code, as long as there is at least one automated test that displays the string and all the tests pass. Which means that going forward, Core and any other projects that enforce passing automated tests before committing code, plus any projects that go back and fix tests that suddenly start failing, will be free of these untranslatable strings.

The patch will also need to fix the broken strings in Drupal core.

Remaining tasks

1. Make a patch that adds this assert to the TranslatableMarkup class, and fixes any Core strings that fail the test. [There was one such string in Core, plus another one in a test class. Both have been fixed in the patch.]

2. Review.

3. Commit.

User interface changes

Strings that are translated will not contain any "unsafe" tags that would make them not importable or not translatable in the translation UI.

API changes

A runtime assert in the TranslationMarkup constructor will verify that any translatable string has no unsafe tags, if runtime asserts are being checked (i.e., during tests only -- generally these asserts do not run on production sites).

Data model changes

No.

Release notes snippet

A runtime assert has been added to the TranslationMarkup constructor, which verifies that any translatable string has no unsafe tags. Unsafe tags are any HTML tags that are not listed in the locale_string_is_safe() function. Strings that don't pass this test are not importable when a site imports translations from localize.drupal.org, so it is important that Drupal Core and contributed projects not contain these unsafe strings. This assert will only be checked if runtime asserts are live, which is generally only true when tests are being run. Contributed project tests will see their existing tests fail after this change, if they have any unsafe strings in their code base, assuming that they have test coverage that triggers those strings to be displayed.

Issue fork drupal-3157707

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Joachim Namyslo created an issue. See original summary.

greggles’s picture

Title: XXS filter prevents translation import when <div> and <button> are used » Text filter prevents translation import when <div> and <button> are used
Issue summary: View changes

Making the html tags show up in the issue summary and adjusting title. It's for XSS and other reasons like defacement and phishing.

jhodgdon’s picture

OK... I think you are saying that there is a list of HTML tags that, effectively, cannot or should not be used in any translatable strings, because if they are translated on localize.drupal.org, then those translations are imported into a site, they will be skipped. Meaning those strings are effectively not translatable with our tools. Right?

Then should we generate errors in POTX when such strings are read from projects, and make issues in projects to remove them? It would be good do document this somewhere too?

jhodgdon’s picture

And/or do we need a Coder rule to detect this?

jhodgdon’s picture

Issue tags: +i18n
joachim namyslo’s picture

That's a brilliant summary of the Issue. Jennifer. A method/module to filter and list affected strings is needed, because currently 114+ languages are affected of this issue in several modules and themes. Detect them manually would take at least a decade or so.

jhodgdon’s picture

Issue summary: View changes

Adding issue summary.

joachim namyslo’s picture

Issue tags: +Global2020
tobiasb’s picture

I believe div/code are not the problems. Attributes inside a div (<div class='alert alert-warning alert-sm'><strong>WARNUNG:</strong>)or invalid html like <front>.

tobiasb’s picture

All html-elements which are not part of the list in locale_string_is_safe() should be detected. perhaps a fancy combination of potx + coder.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

joachim namyslo’s picture

joachim namyslo’s picture

#10 I'd like it if that would be true. But take this example here:

https://localize.drupal.org/translate/languages/de/translate?sid=2

You can translate it, aprove the translation and install the varbase distribution Just to verify that this String will get filtered.

When it comes to technology there is no way to believe something but there are many different ways to prove it.

I did this with the string above in particular so I know that presumption is correct but doesn't apply to that issue in all cases.

jhodgdon’s picture

Here's a thought about how we could enforce this in Drupal core at least.

When anything is run through t() and the other translation functions in Drupal, the actual string translation ends up being run through the 'string_translation' service.

So, we could instrument all Functional tests (the ones that visit UI pages) with their own string translation service class, which could override the doTranslate() method so that it runs all the strings that come in through locale_string_is_safe(), and flags an error if they fail that test. This could be run even if the locale module wasn't turned on.

jhodgdon’s picture

Status: Active » Needs review
StatusFileSize
new5.15 KB

Here's a patch that implements #14, more or less. What it does: Unless a test using BrowserTestBase overrides the default member variable $strictStrings = TRUE, every translatable string that is rendered during the test is checked to see that it is locale-safe. I added this to the existing FunctionalTestSetupTrait, which is also where the config strict checking is done. As the patch is now, it would also affect any contrib modules that run BrowserTestBase tests.

I verified locally that I could make a test fail by editing the code so that there was a string rendered with a <div> tag in it. Then when I removed that line from my code, my test passed. (Eventually, this patch probably needs to have a unit or functional test in it that verifies that the test is actually working right, by introducing a string that isn't locale safe and verifying an exception is thrown, but let's see if this is even a viable approach first.)

So... Let's see what happens when we run the Core tests. Are there locale-unsafe strings in Core? If so, they should fail the tests and we should fix them. I expect there might be some extraneous fails too if there are some tests in Core that are using locale-unsafe strings to test something.

jhodgdon’s picture

StatusFileSize
new5.14 KB
new612 bytes

Fix code sniff.

Status: Needs review » Needs work

The last submitted patch, 16: 3157707-16.patch, failed testing. View results

jhodgdon’s picture

Status: Needs work » Needs review
StatusFileSize
new6.36 KB
new2.31 KB

Two tests in the System module apparently have non-locale-safe strings being used in the tests. So, for those tests, disable checking. Also fix up some docs.

jhodgdon’s picture

StatusFileSize
new7.25 KB
new910 bytes

OK, tests pass now... To verify that this test addition would have caught non-locale-safe strings, I made up a should-fail patch by adding a div tag in some HTML output for a page that hopefully is displayed in a test. The real patch still probably needs a test for the test?

jhodgdon’s picture

The test bar says "Build successful", but if you click through to the results on that last patch, you'll see that about 15 tests failed due to the div tag in the string. So... Any thoughts about this approach?

jhodgdon’s picture

Status: Needs review » Needs work

I discussed this in Slack today with @alexpott and he suggested putting an assert into the TranslatableMarkup constructor instead of the current approach of an exception in a custom translator. I'm working on a patch for that.

jhodgdon’s picture

Status: Needs work » Needs review
StatusFileSize
new6.29 KB

Here's a patch with this new approach. The tests pass locally for me, with an update to the Breadcrumb test module so that it doesn't use an non-locale-safe string.

Interdiff is not useful -- this is a completely new approach and every line of the patch is different from the previous patch.

Status: Needs review » Needs work

The last submitted patch, 22: 3157707-22.patch, failed testing. View results

jhodgdon’s picture

Status: Needs work » Needs review
StatusFileSize
new7.16 KB
new890 bytes

OK! This is good. The new test found an actual error -- a typo in a string in the node module where it had <br/> instead of <br />. Here's a fix.

Status: Needs review » Needs work

The last submitted patch, 24: 3157707-24.patch, failed testing. View results

jhodgdon’s picture

Status: Needs work » Needs review
StatusFileSize
new8.31 KB
new1.24 KB

Looks like 1 test had the bad string from node module hard-wired in it.

jhodgdon’s picture

Issue summary: View changes

Updating issue summary. Also made a draft change record. https://www.drupal.org/node/3199803

I believe this is ready for review.

jhodgdon’s picture

Title: Text filter prevents translation import when <div> and <button> are used » Prevent strings unsafe for translation in code

Updating title too.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

joachim namyslo’s picture

Related issues: +#2936699: Open Social

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

joachim namyslo’s picture

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new144 bytes

The Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

ranjith_kumar_k_u’s picture

StatusFileSize
new8.36 KB

Re-rolled #26

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

siemen_hermans’s picture

StatusFileSize
new8.11 KB

Patch compatible with version 11.3.2

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

meeni_dhobale made their first commit to this issue’s fork.

jurgenhaas’s picture