Problem/Motivation

As detected by #2484693: Telephone Link field formatter InvalidArgumentException with 5 digits or fewer in the number, PHP does not guarantee the parse_url function results for URIs.
The problem is that PHP's parse_url is 'designed' for url's not for URIs.
Because by documentation parse_url:

This function is intended specifically for the purpose of parsing URLs and not URIs.

That can lead to many issues as this function is widely used in the Core as shown in the following table:

The ones which work:

File/Class Function/Method Result
core/includes/batch.inc _batch_finished OK
core/includes/install.core.inc install_check_translations OK
core/includes/install.core.inc install_retrieve_file OK
\Drupal\Component\Utility\UrlHelper externalIsLocal OK
\Drupal\Component\Utility\UrlHelper parse OK
\Drupal\Core\DrupalKernel initializeRequestGlobals OK
\Drupal\Core\Database\Database convertDbUrlToConnectionInfo OK
\Drupal\Core\Utility\UnroutedUrlAssembler assemble OK
\Drupal\language\Form\NegotiationUrlForm validateForm OK
\Drupal\language\Plugin \LanguageNegotiation\LanguageNegotiationUrl getLangcode OK
\Drupal\language\Tests \LanguageUILanguageNegotiationTest testLanguageDomain OK
\Drupal\language\Tests\LanguageUrlRewritingTest testDomainNameNegotiationPort OK
\Drupal\link\Plugin\Field\FieldWidget\LinkWidget getUriAsDisplayableString OK
\Drupal\menu_link_content\Entity\MenuLinkContent preSave OK
\Drupal\node\Tests\PagePreviewTest testPagePreview OK
\Drupal\search\Tests\SearchLanguageTest testLanguages OK
\Drupal\simpletest\BrowserTestBase getAbsoluteUrl OK
\Drupal\simpletest\BrowserTestBase setUp OK
\Drupal\simpletest\WebTestBase getAbsoluteUrl OK
\Drupal\system\Tests\Form\RebuildTest testPreserveFormActionAfterAJAX OK
\Drupal\system\Tests\Pager\PagerTest testActiveClass OK
\Drupal\system\Tests\Pager\PagerTest testPagerQueryParametersAndCacheContext OK
core/modules/update/update.manager.inc update_manager_file_get OK
\Drupal\views\Plugin\views\display\PathPluginBase validatePath OK
\Drupal\views\Plugin\views\field\FieldPluginBase renderAsLink OK
\Drupal\views_ui\ViewEditForm getDisplayDetails OK

Not OK:

\Drupal\Core\Url fromUri Not OK (see the issue summary)
\Drupal\Core\Validation\Plugin\Validation \Constraint\PrimitiveTypeConstraintValidator validate Not OK
\Drupal\link\Plugin\Field\FieldWidget\LinkWidget getUserEnteredStringAsUri Not OK (it transforms tel:xxx to internal:tel:xxx)
\Drupal\link\Plugin\Field\FieldWidget\LinkWidget validateUriElement Not OK (partly because of getUserEnteredStringAsUri)
\Drupal\link\Plugin\Validation \Constraint\LinkExternalProtocolsConstraintValidator validate Not OK (for both tel:911 [error], tel:65536 [error] and tel:0123456789 [failure])
\Drupal\menu_link_content \Plugin\migrate\process\d6\InternalUri transform Not OK (converts the tel:xxx URL to internal:/tel:xxx)
\Drupal\shortcut\Controller \ShortcutSetController addShortcutLinkInline Not OK (converts the tel:xxx URL to internal:/tel:xxx but it's not likely to happens in shortcuts)
core/modules/system/system.module system_retrieve_file Can throw notices but not likely to be used with a tel URI

Proposed resolution

We have Drupal\Component\Utility\UrlHelper (URL). We should create a similar UrIHelper (URI) class which is a true RFC 3986 parser.

Note: An external parser can't be used with our current Drupal requirements (PHP 5.6+ or php-intl requirements, both which Drupal doesn't have now)

Wrap parse_url in a drupal_parse_uri function so we can manage all specific cases and unit test it.

Remaining tasks

Discuss, Fix PHP, Fix Drupal, Enjoy

User interface changes

None.

API changes

An added URIHelper class.

Data model changes

None.

CommentFileSizeAuthor
#58 interdiff-51-58.txt813 bytesndf
#58 2575577-58.patch2.08 KBndf
#58 2575577-58-test-only.patch1.37 KBndf
#57 2575577-57-test-only.patch1.32 KBndf
#57 2575577-57.patch2.03 KBndf
#57 interdiff-51-57.txt652 bytesndf
#56 interdiff-51-54.txt397 bytesndf
#54 2575577-54.patch2.04 KBndf
#54 2575577-54-test-only.patch1.33 KBndf
#51 2575577-51.patch1.68 KBacbramley
#51 2575577-51-test-only.patch994 bytesacbramley
#47 2575577-47-test-only.patch763 bytesravi.shankar
#45 2575577-44.patch1.45 KBarthur.baghdasar
#42 2575577-testFromUriTelephoneNumber.patch763 byteseleonel
#39 interdiff_38-39.txt457 bytespooja saraah
#39 2575577-39.patch720 bytespooja saraah
#38 2575577-38.patch725 bytesarthur.baghdasar
#19 interdiff.txt3.77 KBmallezie
#19 uri-validator-2575577-19.patch14.47 KBmallezie
#17 uri-validator-2575577-17.patch10.7 KBmallezie
#11 2575577-11.patch969 bytesAnonymous (not verified)
#2 2575577-getUserEnteredStringAsUri-911.png9.4 KBDuaelFr
#2 2575577-getUserEnteredStringAsUri-65536.png13.06 KBDuaelFr
#2 PrimitiveTypeConstraintValidator_tel_failure-2575577-1.patch1.03 KBDuaelFr
#2 LinkExternalProtocolsConstraintValidator_tel_failure-2575577-1.patch1.56 KBDuaelFr

Issue fork drupal-2575577

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

DuaelFr created an issue. See original summary.

DuaelFr’s picture

DuaelFr’s picture

Status: Active » Needs review

Changing status to Needs review so the tests are going to show the failures of the tel URIs handling.

Status: Needs review » Needs work

The last submitted patch, 2: PrimitiveTypeConstraintValidator_tel_failure-2575577-1.patch, failed testing.

The last submitted patch, 2: PrimitiveTypeConstraintValidator_tel_failure-2575577-1.patch, failed testing.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dawehner’s picture

I'm wondering wether \Drupal\Component\Utility\UrlHelper::parse already does what we need here? Maybe we need a UriHelper though.

Anonymous’s picture

Thank you @DuaelFr for the excellent drawn issue!

+1 for UriHelper, or vendor URI RFC 3986 parser instead php parse_url in Url::fromUri. Because by documentation parse_url:

This function is intended specifically for the purpose of parsing URLs and not URIs.

But now we have fatal error wherever used TelephoneLinkFormatter with 5 digits or less in the number. Maybe as temp solution we can apply any simple patch? Example this patch works for me.

Anonymous’s picture

and my patch needs security review, of course. Maybe add rawurlencode or mask like '/^tel:\d{1,5}/'

mallezie’s picture

I think an external compliant parser could probably be easiest.
We create an UriHelper based on the external parser, switch the needed methods were needed with the new parser. And as a followup we could do the same for UrlHelper, and mark that even deprecated in the end.

Looking at external parsers, think https://github.com/thephpleague/uri is most promising, and it's successor for php 5.6+ (in the future https://github.com/thephpleague/uri-parser)
Other option is https://github.com/Riimu/Kit-UrlParser

Give me some days to work on a starter patch.

Anonymous’s picture

@mallezie, it will be very cool if you will have the opportunity to do it! But note, all you mentioned parsers have system requirements intl extension. But Drupal 8 not requires this extension by default. And this can be a problem.

mallezie’s picture

Thanks, for the heads up. I missed the intl requirement. Then i think the only option is a custom UriHelper. I'll start with that.

mallezie’s picture

Issue summary: View changes

Reformatted IS (mostly to remove wrapping under sidebar), and resuggested possible solution.

mallezie’s picture

Version: 8.2.x-dev » 8.3.x-dev
Status: Needs work » Needs review
FileSize
10.7 KB

This adds an URIHelper class. And fixes the primitive type validation. The ExternalLink still fails (i've incorparated the tests from DualFr).
The regex-parsing comes from https://gist.github.com/kpobococ/92f120c6c4a9a52b84e3 Not sure if this is allowed to copy paste this / how to give attribution for this in code.
This will fail, but settings for needs review to test if the first tests at least pass.

This needs first discussing of this is the right approach. Needs tests and fixes for the other not OK uses mentioned in IS.

Status: Needs review » Needs work

The last submitted patch, 17: uri-validator-2575577-17.patch, failed testing.

mallezie’s picture

Added more (failing) tests for the other cases. (So we have a base to fix ;-) ).
Not sure about two 'wrong' parse_url usages.

addShortcutLinkInline is only used to add a page inline to the default shortcut set (through the star on a page). So abusing this from an external url (tel uri) doesn't seem possible.

system_retrieve_file is used to download content, so it's up to the developer IMO not to use it to download a URI, but only a URL. (In core only used for translations and updates).

Setting to needs review to demonstrate more failing tests.

Status: Needs review » Needs work

The last submitted patch, 19: uri-validator-2575577-19.patch, failed testing.

effulgentsia’s picture

Not commenting on the patch, just on the problem space that this issue is surfacing...

From the issue summary:

The problem is that PHP's parse_url is 'designed' for url's not for URIs.

And:

\Drupal\link\Plugin\Field\FieldWidget\LinkWidget getUserEnteredStringAsUri Not OK (it transforms tel:xxx to internal:tel:xxx)

We're in an interesting situation here. The job of LinkWidget (and most other "not ok" places identified in the issue summary) is to populate the value of an href attribute. According to https://www.w3.org/TR/html5/links.html#links-created-by-a-and-area-elements, this is defined as holding URLs, not URIs. https://www.w3.org/TR/html5/references.html#refsURL then proceeds to link to both https://tools.ietf.org/html/rfc3986 and to https://url.spec.whatwg.org/. From what I can tell, tel URIs are not URLs, per the distinction of RFC3986. Which is partly why PHP's parse_url doesn't parse them correctly. However, they are allowed into href values, which is a potential contradiction of the HTML5 spec defining href values as being for URLs. Unless the resolution to that contradiction is https://url.spec.whatwg.org/#goals, which seeks to remove the distinction between URL and URI. But that document isn't an officially adopted standard yet, which makes it odd to me that HTML5 would redefine URL to mean something based on that broader, but not yet standardized, definition.

So, we may be in a situation where HTML5 defines URL differently than PHP does. Which sucks if it means we can't use PHP's URL functions when dealing with HTML5 URLs.

ndf’s picture

#13
++ for and external vendor library. This seems to be a typical library functionality that won't exists soon php-core but will be be shared/needed in a lot of php-projects.

xjm’s picture

Title: Use the parse_url function more carefuly » UrlHelper does not support tel:
Component: base system » routing system
Issue tags: +Triaged core major

The framework and release managers discussed this issue awhile back and determined that the scope in the summary is primarily about tel:. Since this is a sanitization API that is not behaving as developers expect, we agreed the issue is a major bug.

Anonymous’s picture

tel: is only one of many uri schemes. We should not expect processing URI by url_parse. We need URI RFC 3986 parser (@mallezie worked on this in #17, #19, but I'm not smart enough to do this review).

Also it would be great to handle the //site.com, because it is a good practice to use // instead of http and https protocols. But may be it other issue.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

J-Lee’s picture

Found this issue while searching for

InvalidArgumentException: "The internal path component 'tel:%2B4912345-0' is external. You are not allowed to specify an external URL together with internal:/." at /core/lib/Drupal/Core/Url.php line 419'

which is triggered via the TelephoneLinkFormatter in Drupal 8.9.

acbramley’s picture

This can be triggered easily with a link field. We found this via a client trying to create a link to tel:000.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

arthur.baghdasar’s picture

Rerolled the patch from #11 to work with 9.4.x

pooja saraah’s picture

Fixed failed commands on #38
Attached interdiff

pooja saraah’s picture

Status: Needs work » Needs review
smustgrave’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests, +Bug Smash Initiative

Will need a test case.

eleonel’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests
FileSize
763 bytes

Added test case for #39

$ cd core
$ ../vendor/bin/phpunit --filter testFromUriTelephoneNumber tests/Drupal/Tests/Core/UrlTest.php
PHPUnit 8.5.26 #StandWithUkraine

Testing Drupal\Tests\Core\UrlTest
.                                                                   1 / 1 (100%)

Time: 132 ms, Memory: 8.00 MB

OK (1 test, 2 assertions)

arthur.baghdasar’s picture

Added merge request, test looks good to me.

arthur.baghdasar’s picture

Here is a complete patch with tests.

smustgrave’s picture

Status: Needs review » Needs work

Wow quick turn around!

Putting into NW because the test patch seemed to pass

You want the tests only patch to fail and the full patch to pass to signify the problem is solved

ravi.shankar’s picture

Added test only patch as per comment #46.

smustgrave’s picture

#47 is the same as #42 so please disregard

But for test only patches it helps to append test-only.patch

smustgrave’s picture

So was the problem was that is wasn't marking tel:1234 as external? Appears to be doing that now without the patch?

Sure I'm missing something though.

ameymudras’s picture

As mentioned in #49 the I tested with URL tel:123-4567-8901 and without using the patch the URL is external. I think the tests needs to be extensive to cover the 'Not Ok ' scenarios mentioned in the description

acbramley’s picture

Status: Needs work » Needs review
FileSize
994 bytes
1.68 KB

This should be red/green.

The last submitted patch, 51: 2575577-51-test-only.patch, failed testing. View results

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Took a look at the code and all seems clean to me. Glad to see the test case got worked out!

ndf’s picture

Status: Reviewed & tested by the community » Needs review
FileSize
1.33 KB
2.04 KB

Can we add a couple of more test cases? See this file from laravel: https://github.com/laravel-validation-rules/phone/blob/master/tests/Vali...

I have added the E164 cases.

ndf’s picture

ndf’s picture

FileSize
397 bytes

The interdiff for #54 - #51

ndf’s picture

ndf’s picture

The last submitted patch, 58: 2575577-58-test-only.patch, failed testing. View results

ameymudras’s picture

Status: Needs review » Reviewed & tested by the community

Tested on Drupal 9.5.x and php 8.1

- The issue summary is clear
- The patch applies cleanly and tests pass
- Used small snippet of code to test the issue and it works as expected.

Marking this as RTBC

The last submitted patch, 58: 2575577-58-test-only.patch, failed testing. View results

The last submitted patch, 58: 2575577-58-test-only.patch, failed testing. View results

catch’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs issue summary update
  1. +++ b/core/lib/Drupal/Core/Url.php
    @@ -281,7 +281,14 @@ public static function fromUri($uri, $options = []) {
    +    if (preg_match('/^tel:/', $uri)) {
    

    This could just be str_starts_with() in Drupal 10.

  2. +++ b/core/lib/Drupal/Core/Url.php
    @@ -281,7 +281,14 @@ public static function fromUri($uri, $options = []) {
    +    if (preg_match('/^tel:/', $uri)) {
    +      $parts = explode(':', $uri);
    +      $uri_parts['scheme'] = $parts[0];
    

    Should this be explode(':', $uri, 2), or alternatively, should it fail if there's a second colon?

Also neither the title nor issue summary match the proposed solution - why are we only changing Url::fromUri() here?

smustgrave’s picture

After further review it appears the patch in #11 was meant to be a temporary solution while a UriHelper class was written.

#17 and #19 attempted to solve the problem to match the issue summary.

Seems around #38 the temporary patch was rerolled and carried forward as the solution (including by myself)

So are we abandoning the UriHelper class all together?

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.