Problem/Motivation
As detected by #2484693: Telephone Link field formatter InvalidArgumentException with 5 digits or fewer in the number, PHP does not guarantee the parse_url function results for URIs.
The problem is that PHP's parse_url is 'designed' for url's not for URIs.
Because by documentation parse_url:
This function is intended specifically for the purpose of parsing URLs and not URIs.
That can lead to many issues as this function is widely used in the Core as shown in the following table:
The ones which work:
| File/Class | Function/Method | Result |
|---|---|---|
| core/includes/batch.inc | _batch_finished | OK |
| core/includes/install.core.inc | install_check_translations | OK |
| core/includes/install.core.inc | install_retrieve_file | OK |
| \Drupal\Component\Utility\UrlHelper | externalIsLocal | OK |
| \Drupal\Component\Utility\UrlHelper | parse | OK |
| \Drupal\Core\DrupalKernel | initializeRequestGlobals | OK |
| \Drupal\Core\Database\Database | convertDbUrlToConnectionInfo | OK |
| \Drupal\Core\Utility\UnroutedUrlAssembler | assemble | OK |
| \Drupal\language\Form\NegotiationUrlForm | validateForm | OK |
| \Drupal\language\Plugin \LanguageNegotiation\LanguageNegotiationUrl | getLangcode | OK |
| \Drupal\language\Tests \LanguageUILanguageNegotiationTest | testLanguageDomain | OK |
| \Drupal\language\Tests\LanguageUrlRewritingTest | testDomainNameNegotiationPort | OK |
| \Drupal\link\Plugin\Field\FieldWidget\LinkWidget | getUriAsDisplayableString | OK |
| \Drupal\menu_link_content\Entity\MenuLinkContent | preSave | OK |
| \Drupal\node\Tests\PagePreviewTest | testPagePreview | OK |
| \Drupal\search\Tests\SearchLanguageTest | testLanguages | OK |
| \Drupal\simpletest\BrowserTestBase | getAbsoluteUrl | OK |
| \Drupal\simpletest\BrowserTestBase | setUp | OK |
| \Drupal\simpletest\WebTestBase | getAbsoluteUrl | OK |
| \Drupal\system\Tests\Form\RebuildTest | testPreserveFormActionAfterAJAX | OK |
| \Drupal\system\Tests\Pager\PagerTest | testActiveClass | OK |
| \Drupal\system\Tests\Pager\PagerTest | testPagerQueryParametersAndCacheContext | OK |
| core/modules/update/update.manager.inc | update_manager_file_get | OK |
| \Drupal\views\Plugin\views\display\PathPluginBase | validatePath | OK |
| \Drupal\views\Plugin\views\field\FieldPluginBase | renderAsLink | OK |
| \Drupal\views_ui\ViewEditForm | getDisplayDetails | OK |
Not OK:
| \Drupal\Core\Url | fromUri | Not OK (see the issue summary) |
| \Drupal\Core\Validation\Plugin\Validation \Constraint\PrimitiveTypeConstraintValidator | validate | Not OK |
| \Drupal\link\Plugin\Field\FieldWidget\LinkWidget | getUserEnteredStringAsUri | Not OK (it transforms tel:xxx to internal:tel:xxx) |
| \Drupal\link\Plugin\Field\FieldWidget\LinkWidget | validateUriElement | Not OK (partly because of getUserEnteredStringAsUri) |
| \Drupal\link\Plugin\Validation \Constraint\LinkExternalProtocolsConstraintValidator | validate | Not OK (for both tel:911 [error], tel:65536 [error] and tel:0123456789 [failure]) |
| \Drupal\menu_link_content \Plugin\migrate\process\d6\InternalUri | transform | Not OK (converts the tel:xxx URL to internal:/tel:xxx) |
| \Drupal\shortcut\Controller \ShortcutSetController | addShortcutLinkInline | Not OK (converts the tel:xxx URL to internal:/tel:xxx but it's not likely to happens in shortcuts) |
| core/modules/system/system.module | system_retrieve_file | Can throw notices but not likely to be used with a tel URI |
Proposed resolution
We have Drupal\Component\Utility\UrlHelper (URL). We should create a similar UrIHelper (URI) class which is a true RFC 3986 parser.
Note: An external parser can't be used with our current Drupal requirements (PHP 5.6+ or php-intl requirements, both which Drupal doesn't have now)
Wrap parse_url in a drupal_parse_uri function so we can manage all specific cases and unit test it.
Remaining tasks
Discuss, Fix PHP, Fix Drupal, Enjoy
User interface changes
None.
API changes
An added URIHelper class.
Data model changes
None.
| Comment | File | Size | Author |
|---|---|---|---|
| #58 | interdiff-51-58.txt | 813 bytes | ndf |
| #58 | 2575577-58.patch | 2.08 KB | ndf |
| #58 | 2575577-58-test-only.patch | 1.37 KB | ndf |
| #57 | 2575577-57-test-only.patch | 1.32 KB | ndf |
| #57 | 2575577-57.patch | 2.03 KB | ndf |
Issue fork drupal-2575577
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
duaelfrAdded some files that illustrate some of the failures shown in the IS.
+ made that table more readable
Comment #3
duaelfrChanging status to Needs review so the tests are going to show the failures of the tel URIs handling.
Comment #10
dawehnerI'm wondering wether
\Drupal\Component\Utility\UrlHelper::parsealready does what we need here? Maybe we need aUriHelperthough.Comment #11
Anonymous (not verified) commentedThank you @DuaelFr for the excellent drawn issue!
+1 for UriHelper, or vendor URI RFC 3986 parser instead php
parse_urlinUrl::fromUri. Because by documentation parse_url:But now we have fatal error wherever used
TelephoneLinkFormatterwith 5 digits or less in the number. Maybe as temp solution we can apply any simple patch? Example this patch works for me.Comment #12
Anonymous (not verified) commentedand my patch needs security review, of course. Maybe add
rawurlencodeor mask like'/^tel:\d{1,5}/'Comment #13
mallezieI think an external compliant parser could probably be easiest.
We create an UriHelper based on the external parser, switch the needed methods were needed with the new parser. And as a followup we could do the same for UrlHelper, and mark that even deprecated in the end.
Looking at external parsers, think https://github.com/thephpleague/uri is most promising, and it's successor for php 5.6+ (in the future https://github.com/thephpleague/uri-parser)
Other option is https://github.com/Riimu/Kit-UrlParser
Give me some days to work on a starter patch.
Comment #14
Anonymous (not verified) commented@mallezie, it will be very cool if you will have the opportunity to do it! But note, all you mentioned parsers have system requirements intl extension. But Drupal 8 not requires this extension by default. And this can be a problem.
Comment #15
mallezieThanks, for the heads up. I missed the intl requirement. Then i think the only option is a custom UriHelper. I'll start with that.
Comment #16
mallezieReformatted IS (mostly to remove wrapping under sidebar), and resuggested possible solution.
Comment #17
mallezieThis adds an URIHelper class. And fixes the primitive type validation. The ExternalLink still fails (i've incorparated the tests from DualFr).
The regex-parsing comes from https://gist.github.com/kpobococ/92f120c6c4a9a52b84e3 Not sure if this is allowed to copy paste this / how to give attribution for this in code.
This will fail, but settings for needs review to test if the first tests at least pass.
This needs first discussing of this is the right approach. Needs tests and fixes for the other not OK uses mentioned in IS.
Comment #19
mallezieAdded more (failing) tests for the other cases. (So we have a base to fix ;-) ).
Not sure about two 'wrong' parse_url usages.
addShortcutLinkInline is only used to add a page inline to the default shortcut set (through the star on a page). So abusing this from an external url (tel uri) doesn't seem possible.
system_retrieve_file is used to download content, so it's up to the developer IMO not to use it to download a URI, but only a URL. (In core only used for translations and updates).
Setting to needs review to demonstrate more failing tests.
Comment #21
effulgentsia commentedNot commenting on the patch, just on the problem space that this issue is surfacing...
From the issue summary:
And:
We're in an interesting situation here. The job of
LinkWidget(and most other "not ok" places identified in the issue summary) is to populate the value of anhrefattribute. According to https://www.w3.org/TR/html5/links.html#links-created-by-a-and-area-elements, this is defined as holding URLs, not URIs. https://www.w3.org/TR/html5/references.html#refsURL then proceeds to link to both https://tools.ietf.org/html/rfc3986 and to https://url.spec.whatwg.org/. From what I can tell,telURIs are not URLs, per the distinction of RFC3986. Which is partly why PHP'sparse_urldoesn't parse them correctly. However, they are allowed intohrefvalues, which is a potential contradiction of the HTML5 spec defining href values as being for URLs. Unless the resolution to that contradiction is https://url.spec.whatwg.org/#goals, which seeks to remove the distinction between URL and URI. But that document isn't an officially adopted standard yet, which makes it odd to me that HTML5 would redefine URL to mean something based on that broader, but not yet standardized, definition.So, we may be in a situation where HTML5 defines URL differently than PHP does. Which sucks if it means we can't use PHP's URL functions when dealing with HTML5 URLs.
Comment #22
ndf commented#13
++ for and external vendor library. This seems to be a typical library functionality that won't exists soon php-core but will be be shared/needed in a lot of php-projects.
Comment #23
xjmThe framework and release managers discussed this issue awhile back and determined that the scope in the summary is primarily about
tel:. Since this is a sanitization API that is not behaving as developers expect, we agreed the issue is a major bug.Comment #24
Anonymous (not verified) commentedtel:is only one of many uri schemes. We should not expect processing URI by url_parse. We need URI RFC 3986 parser (@mallezie worked on this in #17, #19, but I'm not smart enough to do this review).Also it would be great to handle the
//site.com, because it is a good practice to use//instead ofhttpandhttpsprotocols. But may be it other issue.Comment #32
j-leeFound this issue while searching for
which is triggered via the TelephoneLinkFormatter in Drupal 8.9.
Comment #33
acbramley commentedThis can be triggered easily with a
linkfield. We found this via a client trying to create a link to tel:000.Comment #38
arthur.baghdasar commentedRerolled the patch from #11 to work with 9.4.x
Comment #39
pooja saraah commentedFixed failed commands on #38
Attached interdiff
Comment #40
pooja saraah commentedComment #41
smustgrave commentedWill need a test case.
Comment #42
eleonelAdded test case for #39
Comment #44
arthur.baghdasar commentedAdded merge request, test looks good to me.
Comment #45
arthur.baghdasar commentedHere is a complete patch with tests.
Comment #46
smustgrave commentedWow quick turn around!
Putting into NW because the test patch seemed to pass
You want the tests only patch to fail and the full patch to pass to signify the problem is solved
Comment #47
ravi.shankar commentedAdded test only patch as per comment #46.
Comment #48
smustgrave commented#47 is the same as #42 so please disregard
But for test only patches it helps to append test-only.patch
Comment #49
smustgrave commentedSo was the problem was that is wasn't marking tel:1234 as external? Appears to be doing that now without the patch?
Sure I'm missing something though.
Comment #50
ameymudras commentedAs mentioned in #49 the I tested with URL tel:123-4567-8901 and without using the patch the URL is external. I think the tests needs to be extensive to cover the 'Not Ok ' scenarios mentioned in the description
Comment #51
acbramley commentedThis should be red/green.
Comment #53
smustgrave commentedTook a look at the code and all seems clean to me. Glad to see the test case got worked out!
Comment #54
ndf commentedCan we add a couple of more test cases? See this file from laravel: https://github.com/laravel-validation-rules/phone/blob/master/tests/Vali...
I have added the E164 cases.
Comment #55
ndf commentedComment #56
ndf commentedThe interdiff for #54 - #51
Comment #57
ndf commentedPatch did not apply, hereby another try
Comment #58
ndf commentedspellcheck :/
another retry
Comment #60
ameymudras commentedTested on Drupal 9.5.x and php 8.1
- The issue summary is clear
- The patch applies cleanly and tests pass
- Used small snippet of code to test the issue and it works as expected.
Marking this as RTBC
Comment #63
catchThis could just be str_starts_with() in Drupal 10.
Should this be explode(':', $uri, 2), or alternatively, should it fail if there's a second colon?
Also neither the title nor issue summary match the proposed solution - why are we only changing Url::fromUri() here?
Comment #64
smustgrave commentedAfter further review it appears the patch in #11 was meant to be a temporary solution while a UriHelper class was written.
#17 and #19 attempted to solve the problem to match the issue summary.
Seems around #38 the temporary patch was rerolled and carried forward as the solution (including by myself)
So are we abandoning the UriHelper class all together?
Comment #67
marc.bauI see you added
['tel:+1 (555) 666-7777']what is nice as I'm wondering if spaces and brackets should be url encoded in TelephoneLinkFormatter.RFC https://datatracker.ietf.org/doc/html/rfc3966#section-5.1 shows an example where
<a href="tel:+1-201-555-0111">(201) 555-0111</a>spaces are removed and brakets become hyphens. This seems not addressed here. But maybe this is a follow up issue for TelephoneLinkFormatter?Comment #68
nigelcunningham commentedWhen implementing this, please don't forget use cases outside of the USA. In Australia, we have some shorter six digit numbers (132 500, for example) and there are also the emergency service numbers (not necessarily 911 - alternatives I know include 000, 111 and 112.
Comment #69
darvanen#64: The documentation as such indicates that tel: should be supported and #11 is correct that PHP's native parse_url is not fit for purpose here. I think we need to either revive the custom approach or do another scan to see if there's a library out there now which supports this requirement.
Comment #70
dpiThere shouldnt be any validation of anything after `tel:` to be honest. Theres simply to many localized variants, take a look a look a https://github.com/giggsey/libphonenumber-for-php which is implements the PHP version of the unofficially standard phone number validation and formatting library libphonenumber.
I think, non-empty-string is the only thing we should be validating against.
Re this issue as a whole, things would be a hell of a lot easier if we don't consider tel:
Comment #71
bradjones1Related/potential duplicate that addresses this more broadly: #3186651: Not all URIs are URLs, but UriWidget won't accept non-URL URIs
Comment #72
nigelcunningham commentedI started work on updating the patch in earlier commits for merging, per SMustgrave's comment in #64, but have been asked to move on to other things, so I'm pushing what I've gotten done so far in the hope that maybe it won't go to waste.
Comment #74
mikemccaffreyIt would be amazing if we could try to get some version of the fix committed into core. A client want to have links to 988 for the National Suicide and Crisis Helpline, so supporting short telephone numbers could literally stop someone from killing themselves.
Comment #75
dwwAs of PHP 8.5, we could use this:
https://www.php.net/releases/8.5/en.php#new-uri-extension
However, it'd be nice to be able to backport fixes to released branches. Maybe the php85 polyfill provides this? I haven't checked.
Comment #76
dwwDrat, nope:
https://github.com/symfony/polyfill-php85
However, this looks promising:
https://uri.thephpleague.com/polyfill/7.0/