This task has been created as a central place to develop the IDN/Punycode encode and decode functions.

http://www.phpclasses.org/browse/package/1509.html (LGPL) might be interesting as a starting point.

related issues:
#308138: Make valid_email_address() support IDNs
#295021: Expand/augment UrlHelper::isValid() to improve correctness and handle more URL types
#368472: valid_url() marks correct IDN domains as invalid

Comments

hass’s picture

hass commented

Subscribe

Reg’s picture

Reg commented

subscribe

dropcube’s picture

dropcube commented

Subscribe

mattyoung’s picture

mattyoung commented

subscribe

Breakerandi’s picture

Breakerandi commented

subscribe

chx’s picture

chx commented
Version: 7.x-dev » 8.x-dev
Category: task » feature
Priority: Critical » Normal
mfer’s picture

mfer commented

Subscribe

BurakD’s picture

BurakD commented

subscribe

chx’s picture

chx commented

I am so happy that we have seven subscribes! So who is starting a contrib project to create the code, let it mature and then get into D8.

jcfiala’s picture

jcfiala commented

I might, chx, as I've been dealing with IDN-related stuff as part of the link module.

That said, I want to convert Link over to D7 first.

I welcome people providing useful references and suggestions on how to handle this problem.

hass’s picture

hass commented

While looking around the net how we can solve the validation issue I found an idea to convert into ASCII and validate than. So, converting every URL with http://www.php.net/manual/en/function.idn-to-ascii.php may be an easy solution. After the conversion, run the old fashioned valid_url(). It sound too easy to me, but it may work!?

alexanderpas’s picture

alexanderpas commented

do note that this solution is only availble for PHP5.3+

hass’s picture

hass commented

Yeah, but better than having no solution :-)

mfer’s picture

mfer commented

Also note that a lot of environments have idn_to_ascii compiled out. For example, MAMP. Many drupal developers use MAMP. Note that there is a ticket in MAMP to correct this.

Konstantin Komelin’s picture

Konstantin Komelin commented

Year ago I had an idea to create utility module based on Matthias Sommerfeld's library. But if I understand correctly it's not allowed to include LGPL code into the Drupal project.

I still use that library for one of my projects and it works well for Russian domains like www.с-днем-рождения.рф

So, we can ask Matthias to relicense it to GPL or Dual.
If Matthias accepts I will create contrib project with two easy wrappers, smth like drupal_idna_encode($domain) /drupal_idna_decode($punycode)

Does it make sense, guys?

pancho’s picture

pancho
Location UTC+2 🇪🇺 EU
commented
Status: Active » Closed (won't fix)

idn_to_ascii() is available in PHP 5.3, so we can close this for D8.
Also, we most probably don't want to add a library just to backport this to D7.

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented
Issue summary: View changes
Status: Closed (won't fix) » Active
Related issues: +#308138: Make valid_email_address() support IDNs, +#295021: Expand/augment UrlHelper::isValid() to improve correctness and handle more URL types, +#368472: valid_url() marks correct IDN domains as invalid, +#739066: IDN Support

idn_to_ascii() isn't 'available', its only an option if you have the intl php extension loaded. (see comment here: https://drupal.org/node/1427516#comment-8756179)

Additionally, it has a bug in it where it will fail silently and return an empty string if the string passed into it is > 68 characters.
(https://bugs.php.net/bug.php?id=67084)

In order to have this in drupal 8, I recommend that we vendor a git repo like the following:
https://github.com/true/php-punycode (looks like MIT license)
Or something like this:
http://phlymail.com/en/downloads/idna-convert.html (GPL) (LGPL)
Or this if we want a js solution for whatever front end needs we might have.
https://github.com/bestiejs/punycode.js (MIT License)

That way we dont have to worry about environmental configurations or bugs and we can use those encode/decode libraries on other issues.

Konstantin Komelin’s picture

Konstantin Komelin commented

http://phlymail.com/en/downloads/idna-convert.html (GPL)

LGPL! See #15

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented

Whoops. I misread the page- but the point being that there are plenty of punycode implemetations out there, and core should support Internationalized Domain Names. There are issues all over core and contrib that relate to this, and if we'd like for Drupal to have any adoption in countries that support Internationalized Domain Names, then this is pretty critical to that objective. Im not sure if this is something that could be achieved in contrib.. but maybe?

Konstantin Komelin’s picture

Konstantin Komelin commented

Thank you @Mixologic for your opinion.

I tend to think that idn functionality should be in core. I'm not sure if it's possible now because of unusual "code freeze" (or feature freeze, can't characterize it) stage of the D8.
I also think that we can implement the API as a contrib for D7. That was my original idea.

And thanks for the links to Punycode libs. They can be useful for my projects.

Best,
Konstantin

Anonymous’s picture

Anonymous (not verified) commented

Subscribe

Anonymous’s picture

Anonymous (not verified) commented

I think that this issue should be corrected in Drupal 7, also, as many sites will be using 7 for years to come.

maggie_s’s picture

maggie_s commented

In Drupal 7.

I used idn_to_ascii in order to encode my URL's and the valid_url($url, TRUE) for validation.
Validation is not correct.
"http://akademie-für-gestaltung-regensburg.de/", encoded into "xn--http://akademie-fr-gestaltung-regensburg-0fe.de/" isn't validated.
So solution #11 doesn't work doesn't work for me.

I installed idna_convert (konstantin.komelin conversion proposal), and "http://akademie-für-gestaltung-regensburg.de/", encoded into "http://xn--akademie-fr-gestaltung-regensburg-uxb87k.de/", validates fine.
However, using valid_url, "http://www.", "http://." and "http://www" are considered valid, when they aren't.

If I user filter_var, "http://www." and "http://." are invalid, but "http://www" is valid.

Are there any other solution proposals?

Konstantin Komelin’s picture

Konstantin Komelin commented

@maggie_drupal write your own regular expression ;)

Konstantin Komelin’s picture

Konstantin Komelin commented

And please don't check protocols, check domain names

Konstantin Komelin’s picture

Konstantin Komelin commented

Alright, I've created IDNA Convert module that provides the conversion functions for D7. It works perfectly.
Please note it includes LGPL library, which use has not been totally approved yet.
Feel free to port it to D8 with a couple of lines of code if you wish.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

smustgrave’s picture

smustgrave commented
Status: Active » Postponed (maintainer needs more info)
Issue tags: +stale-issue-cleanup

Thank you for sharing your idea for improving Drupal.

We are working to decide if this proposal meets the Criteria for evaluating proposed changes. There hasn't been any discussion here for over 8 years which suggests that this has either been implemented or there is no community support. Your thoughts on this will allow a decision to be made.

Since we need more information to move forward with this issue, the status is now Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.

Thanks!

smustgrave’s picture

smustgrave commented

Wanted to bump 1 more time before closing.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.