Problem/Motivation
As detected by #2484693: Telephone Link field formatter InvalidArgumentException with 5 digits or fewer in the number, PHP does not guarantee the parse_url
function results for URIs.
The problem is that PHP's parse_url is 'designed' for url's not for URIs.
Because by documentation parse_url:
This function is intended specifically for the purpose of parsing URLs and not URIs.
That can lead to many issues as this function is widely used in the Core as shown in the following table:
The ones which work:
File/Class | Function/Method | Result |
---|---|---|
core/includes/batch.inc | _batch_finished | OK |
core/includes/install.core.inc | install_check_translations | OK |
core/includes/install.core.inc | install_retrieve_file | OK |
\Drupal\Component\Utility\UrlHelper | externalIsLocal | OK |
\Drupal\Component\Utility\UrlHelper | parse | OK |
\Drupal\Core\DrupalKernel | initializeRequestGlobals | OK |
\Drupal\Core\Database\Database | convertDbUrlToConnectionInfo | OK |
\Drupal\Core\Utility\UnroutedUrlAssembler | assemble | OK |
\Drupal\language\Form\NegotiationUrlForm | validateForm | OK |
\Drupal\language\Plugin \LanguageNegotiation\LanguageNegotiationUrl | getLangcode | OK |
\Drupal\language\Tests \LanguageUILanguageNegotiationTest | testLanguageDomain | OK |
\Drupal\language\Tests\LanguageUrlRewritingTest | testDomainNameNegotiationPort | OK |
\Drupal\link\Plugin\Field\FieldWidget\LinkWidget | getUriAsDisplayableString | OK |
\Drupal\menu_link_content\Entity\MenuLinkContent | preSave | OK |
\Drupal\node\Tests\PagePreviewTest | testPagePreview | OK |
\Drupal\search\Tests\SearchLanguageTest | testLanguages | OK |
\Drupal\simpletest\BrowserTestBase | getAbsoluteUrl | OK |
\Drupal\simpletest\BrowserTestBase | setUp | OK |
\Drupal\simpletest\WebTestBase | getAbsoluteUrl | OK |
\Drupal\system\Tests\Form\RebuildTest | testPreserveFormActionAfterAJAX | OK |
\Drupal\system\Tests\Pager\PagerTest | testActiveClass | OK |
\Drupal\system\Tests\Pager\PagerTest | testPagerQueryParametersAndCacheContext | OK |
core/modules/update/update.manager.inc | update_manager_file_get | OK |
\Drupal\views\Plugin\views\display\PathPluginBase | validatePath | OK |
\Drupal\views\Plugin\views\field\FieldPluginBase | renderAsLink | OK |
\Drupal\views_ui\ViewEditForm | getDisplayDetails | OK |
Not OK:
\Drupal\Core\Url | fromUri | Not OK (see the issue summary) |
\Drupal\Core\Validation\Plugin\Validation \Constraint\PrimitiveTypeConstraintValidator | validate | Not OK |
\Drupal\link\Plugin\Field\FieldWidget\LinkWidget | getUserEnteredStringAsUri | Not OK (it transforms tel:xxx to internal:tel:xxx) |
\Drupal\link\Plugin\Field\FieldWidget\LinkWidget | validateUriElement | Not OK (partly because of getUserEnteredStringAsUri) |
\Drupal\link\Plugin\Validation \Constraint\LinkExternalProtocolsConstraintValidator | validate | Not OK (for both tel:911 [error], tel:65536 [error] and tel:0123456789 [failure]) |
\Drupal\menu_link_content \Plugin\migrate\process\d6\InternalUri | transform | Not OK (converts the tel:xxx URL to internal:/tel:xxx) |
\Drupal\shortcut\Controller \ShortcutSetController | addShortcutLinkInline | Not OK (converts the tel:xxx URL to internal:/tel:xxx but it's not likely to happens in shortcuts) |
core/modules/system/system.module | system_retrieve_file | Can throw notices but not likely to be used with a tel URI |
Proposed resolution
We have Drupal\Component\Utility\UrlHelper (URL). We should create a similar UrIHelper (URI) class which is a true RFC 3986 parser.
Note: An external parser can't be used with our current Drupal requirements (PHP 5.6+ or php-intl requirements, both which Drupal doesn't have now)
Wrap parse_url
in a drupal_parse_uri
function so we can manage all specific cases and unit test it.
Remaining tasks
Discuss, Fix PHP, Fix Drupal, Enjoy
User interface changes
None.
API changes
An added URIHelper class.
Data model changes
None.
Comment | File | Size | Author |
---|---|---|---|
#58 | interdiff-51-58.txt | 813 bytes | ndf |
#58 | 2575577-58.patch | 2.08 KB | ndf |
#58 | 2575577-58-test-only.patch | 1.37 KB | ndf |
#57 | 2575577-57-test-only.patch | 1.32 KB | ndf |
#57 | 2575577-57.patch | 2.03 KB | ndf |
Issue fork drupal-2575577
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
DuaelFrAdded some files that illustrate some of the failures shown in the IS.
+ made that table more readable
Comment #3
DuaelFrChanging status to Needs review so the tests are going to show the failures of the tel URIs handling.
Comment #10
dawehnerI'm wondering wether
\Drupal\Component\Utility\UrlHelper::parse
already does what we need here? Maybe we need aUriHelper
though.Comment #11
Anonymous (not verified) CreditAttribution: Anonymous commentedThank you @DuaelFr for the excellent drawn issue!
+1 for UriHelper, or vendor URI RFC 3986 parser instead php
parse_url
inUrl::fromUri
. Because by documentation parse_url:But now we have fatal error wherever used
TelephoneLinkFormatter
with 5 digits or less in the number. Maybe as temp solution we can apply any simple patch? Example this patch works for me.Comment #12
Anonymous (not verified) CreditAttribution: Anonymous commentedand my patch needs security review, of course. Maybe add
rawurlencode
or mask like'/^tel:\d{1,5}/'
Comment #13
mallezieI think an external compliant parser could probably be easiest.
We create an UriHelper based on the external parser, switch the needed methods were needed with the new parser. And as a followup we could do the same for UrlHelper, and mark that even deprecated in the end.
Looking at external parsers, think https://github.com/thephpleague/uri is most promising, and it's successor for php 5.6+ (in the future https://github.com/thephpleague/uri-parser)
Other option is https://github.com/Riimu/Kit-UrlParser
Give me some days to work on a starter patch.
Comment #14
Anonymous (not verified) CreditAttribution: Anonymous commented@mallezie, it will be very cool if you will have the opportunity to do it! But note, all you mentioned parsers have system requirements intl extension. But Drupal 8 not requires this extension by default. And this can be a problem.
Comment #15
mallezieThanks, for the heads up. I missed the intl requirement. Then i think the only option is a custom UriHelper. I'll start with that.
Comment #16
mallezieReformatted IS (mostly to remove wrapping under sidebar), and resuggested possible solution.
Comment #17
mallezieThis adds an URIHelper class. And fixes the primitive type validation. The ExternalLink still fails (i've incorparated the tests from DualFr).
The regex-parsing comes from https://gist.github.com/kpobococ/92f120c6c4a9a52b84e3 Not sure if this is allowed to copy paste this / how to give attribution for this in code.
This will fail, but settings for needs review to test if the first tests at least pass.
This needs first discussing of this is the right approach. Needs tests and fixes for the other not OK uses mentioned in IS.
Comment #19
mallezieAdded more (failing) tests for the other cases. (So we have a base to fix ;-) ).
Not sure about two 'wrong' parse_url usages.
addShortcutLinkInline is only used to add a page inline to the default shortcut set (through the star on a page). So abusing this from an external url (tel uri) doesn't seem possible.
system_retrieve_file is used to download content, so it's up to the developer IMO not to use it to download a URI, but only a URL. (In core only used for translations and updates).
Setting to needs review to demonstrate more failing tests.
Comment #21
effulgentsia CreditAttribution: effulgentsia at Acquia commentedNot commenting on the patch, just on the problem space that this issue is surfacing...
From the issue summary:
And:
We're in an interesting situation here. The job of
LinkWidget
(and most other "not ok" places identified in the issue summary) is to populate the value of anhref
attribute. According to https://www.w3.org/TR/html5/links.html#links-created-by-a-and-area-elements, this is defined as holding URLs, not URIs. https://www.w3.org/TR/html5/references.html#refsURL then proceeds to link to both https://tools.ietf.org/html/rfc3986 and to https://url.spec.whatwg.org/. From what I can tell,tel
URIs are not URLs, per the distinction of RFC3986. Which is partly why PHP'sparse_url
doesn't parse them correctly. However, they are allowed intohref
values, which is a potential contradiction of the HTML5 spec defining href values as being for URLs. Unless the resolution to that contradiction is https://url.spec.whatwg.org/#goals, which seeks to remove the distinction between URL and URI. But that document isn't an officially adopted standard yet, which makes it odd to me that HTML5 would redefine URL to mean something based on that broader, but not yet standardized, definition.So, we may be in a situation where HTML5 defines URL differently than PHP does. Which sucks if it means we can't use PHP's URL functions when dealing with HTML5 URLs.
Comment #22
ndf CreditAttribution: ndf at Dx Experts for Triquanta commented#13
++ for and external vendor library. This seems to be a typical library functionality that won't exists soon php-core but will be be shared/needed in a lot of php-projects.
Comment #23
xjmThe framework and release managers discussed this issue awhile back and determined that the scope in the summary is primarily about
tel:
. Since this is a sanitization API that is not behaving as developers expect, we agreed the issue is a major bug.Comment #24
Anonymous (not verified) CreditAttribution: Anonymous commentedtel:
is only one of many uri schemes. We should not expect processing URI by url_parse. We need URI RFC 3986 parser (@mallezie worked on this in #17, #19, but I'm not smart enough to do this review).Also it would be great to handle the
//site.com
, because it is a good practice to use//
instead ofhttp
andhttps
protocols. But may be it other issue.Comment #32
J-LeeFound this issue while searching for
which is triggered via the TelephoneLinkFormatter in Drupal 8.9.
Comment #33
acbramley CreditAttribution: acbramley at PreviousNext for Service NSW commentedThis can be triggered easily with a
link
field. We found this via a client trying to create a link to tel:000.Comment #38
arthur.baghdasar CreditAttribution: arthur.baghdasar commentedRerolled the patch from #11 to work with 9.4.x
Comment #39
pooja saraah CreditAttribution: pooja saraah at Srijan | A Material+ Company for Drupal India Association commentedFixed failed commands on #38
Attached interdiff
Comment #40
pooja saraah CreditAttribution: pooja saraah at Srijan | A Material+ Company for Drupal India Association commentedComment #41
smustgrave CreditAttribution: smustgrave at Mobomo commentedWill need a test case.
Comment #42
eleonelAdded test case for #39
Comment #44
arthur.baghdasar CreditAttribution: arthur.baghdasar commentedAdded merge request, test looks good to me.
Comment #45
arthur.baghdasar CreditAttribution: arthur.baghdasar commentedHere is a complete patch with tests.
Comment #46
smustgrave CreditAttribution: smustgrave at Mobomo commentedWow quick turn around!
Putting into NW because the test patch seemed to pass
You want the tests only patch to fail and the full patch to pass to signify the problem is solved
Comment #47
ravi.shankar CreditAttribution: ravi.shankar at OpenSense Labs commentedAdded test only patch as per comment #46.
Comment #48
smustgrave CreditAttribution: smustgrave at Mobomo commented#47 is the same as #42 so please disregard
But for test only patches it helps to append test-only.patch
Comment #49
smustgrave CreditAttribution: smustgrave at Mobomo commentedSo was the problem was that is wasn't marking tel:1234 as external? Appears to be doing that now without the patch?
Sure I'm missing something though.
Comment #50
ameymudras CreditAttribution: ameymudras at Salsa Digital commentedAs mentioned in #49 the I tested with URL tel:123-4567-8901 and without using the patch the URL is external. I think the tests needs to be extensive to cover the 'Not Ok ' scenarios mentioned in the description
Comment #51
acbramley CreditAttribution: acbramley at PreviousNext for Service NSW commentedThis should be red/green.
Comment #53
smustgrave CreditAttribution: smustgrave at Mobomo commentedTook a look at the code and all seems clean to me. Glad to see the test case got worked out!
Comment #54
ndf CreditAttribution: ndf for Triquanta commentedCan we add a couple of more test cases? See this file from laravel: https://github.com/laravel-validation-rules/phone/blob/master/tests/Vali...
I have added the E164 cases.
Comment #55
ndf CreditAttribution: ndf for Triquanta commentedComment #56
ndf CreditAttribution: ndf for Triquanta commentedThe interdiff for #54 - #51
Comment #57
ndf CreditAttribution: ndf for Triquanta commentedPatch did not apply, hereby another try
Comment #58
ndf CreditAttribution: ndf for Triquanta commentedspellcheck :/
another retry
Comment #60
ameymudras CreditAttribution: ameymudras at Salsa Digital commentedTested on Drupal 9.5.x and php 8.1
- The issue summary is clear
- The patch applies cleanly and tests pass
- Used small snippet of code to test the issue and it works as expected.
Marking this as RTBC
Comment #63
catchThis could just be str_starts_with() in Drupal 10.
Should this be explode(':', $uri, 2), or alternatively, should it fail if there's a second colon?
Also neither the title nor issue summary match the proposed solution - why are we only changing Url::fromUri() here?
Comment #64
smustgrave CreditAttribution: smustgrave at Mobomo commentedAfter further review it appears the patch in #11 was meant to be a temporary solution while a UriHelper class was written.
#17 and #19 attempted to solve the problem to match the issue summary.
Seems around #38 the temporary patch was rerolled and carried forward as the solution (including by myself)
So are we abandoning the UriHelper class all together?