Problem/Motivation

Internationalizing Domain Names in Applications (IDNA) is a mechanism that has been around for a few years now and is based on multiple RFCs to allow the usage of domain names containing non-ASCII characters.

Relevant RFCs are the following (taken from https://uasg.tech):
Internationalized Domain Names for Applications (IDNA):Definitions and Document Framework
https://tools.ietf.org/html/rfc5890
Internationalized Domain Names in Applications (IDNA): Protocol
https://tools.ietf.org/html/rfc5891
The Unicode Code Points andInternationalized Domain Names for Applications (IDNA)
https://tools.ietf.org/html/rfc5892
Right-to-Left Scripts forInternationalized Domain Names for Applications (IDNA)
https://tools.ietf.org/html/rfc5893
Internationalized Domain Names for Applications (IDNA): Background, Explanation, and Rationale
https://tools.ietf.org/html/rfc5894
Mapping Characters forInternationalized Domain Names in Applications (IDNA) 2008
https://tools.ietf.org/html/rfc5895

Drupal 7 && Drupal 8/9
The current implementation of Drupal's URL syntax verifier valid_url() / UrlHelper::isValid() does only support 5 out of 19 examples for Internationalized Domain Names defined by the Universal Acceptance Steering Group (https://uasg.tech/).

Drupal 9
Although we are already shipping with the Symfony Url constraint which supports all these cases, we are still using a custom implementation that fails in many valid cases.

Example internationalized domain names and the validity in D7/D8:

Structure Status D7 Status D8 Example
Ascii.new ok ok https://ua-test.link
Ascii.long ok ok https://ua-test.technology
Idn.ascii fail fail https://普遍接受-测试.top
Ascii.idn fail fail https://ua-test.世界
Idn.idn fail fail https://普遍接受-测试.世界
Idn-open dot-idn fail fail https://普遍接受-测试。世界
Ascii.punycode ok ok https://ua-test.xn--rhqv96g
Punycode.ascii ok ok https://xn----f38am99bqvcd5liy1cxsg.top
Punycode.punycode ok ok https://xn----f38am99bqvcd5liy1cxsg.xn--rhqv96g
RTL.ascii fail fail https://اختبار-القبولالعالمي.top
RTL.RTL fail fail اختبار-القبولالعالمي.شبكةhttps://
Ascii.new/Unicode fail fail https://ua-test.link/我的页面
Ascii.long/Unicode fail fail https://ua-test.technology/我的页面
Idn.ascii/Unicode fail fail https://普遍接受-测试.top/我的页面
Ascii.idn/Unicode fail fail https://ua-test.世界/我的页面
Idn.idn/Unicode fail fail https://普遍接受-测试.世界/我的页面
Idn-opendot-idn/unicode fail fail https://普遍接受-测试。世界/我的页面
RTL.ascii/Unicode fail fail https://ختبار-القبولالعالمي.top/我的页面
RTL.RTL/Unicode fail fail https://اختبار-القبولالعالمي.شبكة/我的页面

References:
https://tools.ietf.org/html/rfc3986
https://uasg.tech/
https://en.wikipedia.org/wiki/Internationalized_domain_name
https://www.icann.org/resources/pages/idn-2012-02-25-en
https://bgp.he.net/report/dns#_alltlds

Proposed resolution

Drupal 9
* Use Symfony's Url Constraint for validation.
Drupal 7
@todo

Remaining tasks

Drupal 9

  • Patch Drupal core to use Symfony's Url Constraint
  • Review
  • Create release notes snippet

Drupal 7
@todo

User interface changes

none

API changes

none

Data model changes

none

Release notes snippet

Original report by [username]

CommentFileSizeAuthor
#20 3045484-nr-bot.txt146 bytesneeds-review-queue-bot
#13 3045484-12.patch7.97 KBquietone
#13 interdiff-11-12.txt5.27 KBquietone
#12 3045484-12.patch8.22 KBquietone
#12 interdiff-11-12.txt455 bytesquietone
#11 3045484-11.patch8.28 KBquietone
#11 interdiff-8-11.txt1.19 KBquietone
#8 3045484-8.patch8.12 KBquietone
#8 3045484-8-fail.patch6.22 KBquietone
#8 diff-4-8.txt5.07 KBquietone
#2 internationalized-domain-names-3045484-1.patch3.85 KBsanduhrs

Comments

sanduhrs created an issue. See original summary.

sanduhrs’s picture

sanduhrs
he/him
Primary language German
Location 🇪🇺 Heidelberg, Germany, Europe
commented
StatusFileSize
new internationalized-domain-names-3045484-1.patch3.85 KB

Patch attached, please review.

sanduhrs’s picture

sanduhrs
he/him
Primary language German
Location 🇪🇺 Heidelberg, Germany, Europe
commented
Issue summary: View changes
sanduhrs’s picture

sanduhrs
he/him
Primary language German
Location 🇪🇺 Heidelberg, Germany, Europe
commented
Related issues: +#1657886: Filter "Convert URLs into links" doesn't support multilingual web addresses, +#389278: Create IDN encoding and decoding functions, +#295021: Expand/augment UrlHelper::isValid() to improve correctness and handle more URL types, +#739066: IDN Support

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

quietone’s picture

quietone commented
Issue summary: View changes
Status: Active » Needs review
StatusFileSize
new diff-4-8.txt5.07 KB
new 3045484-8-fail.patch6.22 KB
new 3045484-8.patch8.12 KB

Ah, nice an issue that does use Symfony for Url validation.

Rerolled for 9.2.x and modified the test so that :// wasn't always prepended to the Url being tested. The list of Urls needs some work to organize with cSpell disable/enable.

The last submitted patch, 8: 3045484-8-fail.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 8: 3045484-8.patch, failed testing. View results

quietone’s picture

quietone commented
Status: Needs work » Needs review
StatusFileSize
new interdiff-8-11.txt1.19 KB
new 3045484-11.patch8.28 KB

Let's see if adding a NotBlank validator to the Url validation fixes the tests.

quietone’s picture

quietone commented
StatusFileSize
new interdiff-11-12.txt455 bytes
new 3045484-12.patch8.22 KB

Yes, still forget to run commit-code-check.

quietone’s picture

quietone commented
StatusFileSize
new interdiff-11-12.txt5.27 KB
new 3045484-12.patch7.97 KB

Clean up of the cSpell: disable/enable and modify UrlHelper to use the $allowedProtocols.

quietone’s picture

quietone commented
Category: Bug report » Feature request

I could be wrong but this doesn't fit the criteria for a bug, so changing to feature request.

quietone’s picture

quietone commented
Title: Internationalized Domain Names (IDN) » Add support for Internationalized Domain Names (IDN)

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

needs-review-queue-bot’s picture

needs-review-queue-bot commented
Status: Needs review » Needs work
StatusFileSize
new 3045484-nr-bot.txt146 bytes

The Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.