Drupal does not use ISO language codes, iso.inc is misleading

The includes/iso.inc file contains two things: a list of languages and a list of countries. The list of languages is certainly not based on ISO, it uses language tags as defined by the W3C (or locale identifiers if you don't like W3C's terminology). The code comments for country codes do not suggest it is related to ISO country codes, is it?

Given this information, iso.inc is pretty misleading and should be renamed. Given that it has the country list which IS used on stock Drupal sites by system module and the language list which is only used if you have Locale module enabled (but might by used by contrib in a way without locale module, although that sounds very unlikely), we are even looking at a file that has a dual purpose.

One of the possibilities is that we move the language list code under locale module's tree, and the country list stays on as countries.inc. Reasonable? Better ideas?

@sun: is it just me or those language code lists include en-gb, pt-br, zh-hans, xx-lolcat and the other languages we have and going to have in there with these patterns? A big number of our language codes do match the ISO 639-1 identifiers, however, even the help text on the locale UI has this to say:

<a href="@rfc4646">RFC 4646</a> compliant language identifier. Language codes typically use a country code, and optionally, a script or regional variant name. <em>Examples: "en", "en-US" and "zh-Hant".</em>

So it asks for RFC4646 language tags (but it does avoid calling them language tags, yeah!), which now that I'm looking were actually obsoleted by RFC5646 language tags 2 years ago (see http://www.w3.org/International/articles/language-tags/). Select ISO 639 language codes are the first part of a language tag, but language tags obviously much broader, richer. They are also defined by the IETF.

I'm not happy about naming our include files based on standards bodies, but in this case it would be iso-ietf.inc? Not sure that makes sense.

BTW the fact that we are using language tags in Drupal not language codes (even if we attempt to call them language codes at places) was discussed multiple times in #1216094: We call too many things 'language', clean that up especially deeply by @plach.

Yes, they are *derived* from ISO language standards, obviously making up totally new things was not a good idea. However, they "forked" the ISO standards for use in language tags and are not referencing the original ISO standards for meaning but have their own registry which is *derived* from the original ISO data. From the wikipedia article:

Although subtags are often derived from ISO standards, they do not follow these standards absolutely, as this could lead to the meaning of language tags changing over time. In particular, a subtag derived from a code assigned by ISO 639, ISO 15924, ISO 3166 (or UN M.49 only for supranational geographical regions) remains a valid (though deprecated) subtag even if the code is withdrawn from the corresponding ISO standard. If the ISO standard later assigns a new meaning to the withdrawn code, the corresponding subtag will still retain its old meaning. This stability was introduced in the (now obsolete) RFC 4646 (and confirmed in its successor). Before RFC 4646, changes in the meaning of ISO codes could cause changes in the meaning of language tags.

Per the explanation, language tags inherit from ISO data but are forked off of ISO. The authoritative list for these values is in fact maintained by the IANA (http://www.iana.org/assignments/language-subtag-registry). As explained by the W3C at http://www.w3.org/International/articles/language-tags/#registry:

As mentioned above, you used to find subtags by consulting the lists of codes in various ISO standards, but now you can find all subtags in one place. The IANA registry looks a little complicated at first, compared to the ISO code lists, but it is easy enough to use once you understand its structure.

(Emphasis mine). This IANA registry search tool verifies if the language identifiers we use now match the standard. Some examples:

http://rishida.net/utils/subtags/index.php?check=pt-br&submit=Check
http://rishida.net/utils/subtags/index.php?check=en-gb&submit=Check
http://rishida.net/utils/subtags/index.php?check=zh-hant&submit=Check
http://rishida.net/utils/subtags/index.php?check=xx-lolspeak&submit=Check (xx not in the registry, they do have tlh for Klingon though, ha!)
http://rishida.net/utils/subtags/index.php?check=gsw-berne&submit=Check (gsw in itself is Swiss German, berne unknown)

Looks like we are not doing bad at all there generally.

includes/used-to-be-iso-ietf-iana-w3c.inc? :)

A little history:

- HTML 4 defined RFC 1766 as reference for language codes (http://www.w3.org/TR/1999/REC-html401-19991224/struct/dirlang.html#h-8.1.1)
- XHTML 1.0 (that Drupal 7 genearates) refers to RFC 3066 as reference for language codes, which obsoleted 1766 at the time (http://www.w3.org/TR/xhtml1/)
- HTML 5 (that Drupal 8 will generate) defines BCP 47 (the permanent name for W3C's language tags) as reference for language codes, which as discussed above is at the moment defined by RFC 5646 (obsoleted 4646 which previously obsoleted 3066) (http://dev.w3.org/html5/spec/elements.html#the-lang-and-xml:lang-attributes)

So to put it in another way, Drupal 7 is a bit ahead of its time by refering to RFC 4646 (the standard it uses defines RFC 3066 to be used). However, for Drupal 8 to be an HTML 5 suporting CMS, we should follow their lead to use BCP 47 (currently defined by RFC 5646). It is not an ISO thing once again :)

Sorry, cross post. I get how you think iso.inc is the best place for a country and language list. Ok. My motivation with this issue are to

(a) bring all that belongs to locale under one roof (I also have other issues that move locale.inc stuff under locale module, fix function names, etc).

(b) clear up this confusion that Drupal is an ancient CMS that is bound to ISO standards that the web moved over from years ago; the W3C says we should think of them as obsolete (http://www.w3.org/WAI/ER/IG/ert/iso639.htm), only sites like the hardly accurate w3schools site say HTML uses them (http://www.w3schools.com/tags/ref_language_codes.asp)

(c) get rid of magic names like iso.inc, that can be anything and not good for DX; ISO defined so many things, they have 18500 standards in their catalog (ref: http://www.iso.org/iso/store.htm)

@sun: yeah, see... The main problem with saying these are ISO codes is that if you search for ISO language codes in relation the web, all you find is outdated crap. The web moved on. ISO never had standards for the combination of their codes as en-gb and pt-br uses it and if people want to abide the standards and use ISO codes for their languages, they will not be compatible with the rest of the web. Google indexing, browser preferences, etc. all use the BCP 47 language tags. To be able to interface with the web, we need to embrace the change that happened a long time ago.

Last mention I found of ISO regards to HTML languages is at http://www.w3.org/International/O-help-lang.html (before HTML 4, don't know when was the page published) when they wrote "it's a good idea", it was not part of a standard yet. By the time they released HTML 4 (1999, http://www.w3.org/TR/html4/types.html#type-langcode) they refer to RFC 1766 which defines a combination of ISO codes (not ISO codes per say) as language codes (http://www.ietf.org/rfc/rfc1766.txt).

12 years passed since the W3C stopped recommending we use ISO language codes and they forked to refer to derivative standards.

Here is a quick patch that just updated the code comments for the predefined language list. I was about to post this to underline the irony that "langauge codes" as understood by Drupal are defined by "W3C language tags" of which the predefined ones are in "iso.inc" in a function name "_locale_get_predefined_list". Language code, language tag, ISO and locale all in there in concert. Yeah!

Anyway, I like the proposal to rename the file, so will include that in the next edition of the patch as well.

Here is one that renames iso.inc to standard.inc and does the suggested renames +

- updates code comments on both the country list and the language list
- removes the useless grouping on the language list (discussed with @jhodgdon and @sun)
- updates the language instructions as well to refer to up to date stuff like code comments in standard.inc

Tagging.

Yes, this was the idea as discussed above.

Tagging Usability for the simplified explanation of what we consider language codes.

The usability portion of this is:

-      '#description' => t('<a href="@rfc4646">RFC 4646</a> compliant language identifier. Language codes typically use a country code, and optionally, a script or regional variant name. <em>Examples: "en", "en-US" and "zh-Hant".</em>', array('@rfc4646' => 'http://www.ietf.org/rfc/rfc4646.txt')),
+      '#description' => t('Use language codes as defined by <a href="@w3ctags">W3C language tags</a> for interoperability. Language codes typically have a language and optionally, a script or regional variant name. <em>Examples: "en", "en-gb" and "zh-hant".</em>', array('@w3ctags' => 'http://www.w3.org/International/articles/language-tags/')),

With thinking that RFC 4646 is as alien to a user as it can be. Leading a description up with that is turning them off most probably. W3C language tags is less jargon, W3C is maybe still a very geeky thing, but less so than RC 4646 :D Also, fixes the description that language codes are typically country based with examples of "en" and "zh", which are hardly countries. This part of the patch will be candidate for backport once committed.

Well, yeah, well, unfortunately W3C calls them language *tags*, and part of language *tags* are language *codes*, which are you know followed by script or regional variants optionally :) So we sacrifice accuracy here if we refer to W3C language *codes*, when that is in fact misleading for people who actually try to understand the trail of consequences here. However, if we refer to "language *tags*" (the complete language identifier as named by W3C), then we need a bit of explanation for this, since these things in our terminology are language codes. Alternatively of course, we can spread this and name it language *tag* everywhere in Drupal, but that is easily not as straightforward terminology, is it? BTW this was discussed to hell in #1216094: We call too many things 'language', clean that up (language codes vs. tags vs. identifiers).

Ok, updated patch with exactly that, should be back to RTBC then? :)

A little piece of the affected code got moved to modules/locale/locale.bulk.inc in the meantime. Here is a full reroll with the same changes.

Tagging for HTML5 given we align our language code support with the spec here.

@webchick, @Dave Reid: I think you might agree with the unofficial framework initiative plan to name functions/APIs prefixed with the file name (not just module name they are in). So standard.inc functions became standard_*(). My original proposal above was to move the language list to locale and leave countries in includes as countries.inc => countries_*(). Sun argued that (1) the language list might be used outside of locale module (2) we should have a file with multiple lists of standard stuff put altogether, not for individual things like countries and languages therefore (3) he suggested standard.inc which nobody disputed so far.

We cannot really name it languages.inc because (1) its not only languages (2) there is also a language.inc with language API functions and this can quickly get confusing. We cannot really call it locale-support.inc because (1) I think its as vague as standard.inc, locale has LOTs of functionality, and it has supporting code in bootstrap.inc, languages.inc, locale.inc, etc. (2) the country list is used by system even if no locale module exists, so its really not a locale support file.

Any better suggestions for code organization or file naming?

@johnv: as I've explained above, the country list is definitely not "locale specific" (ATM at least, not in terms of how Drupal has "locale module" == "UI translation"), system module uses it even when locale module is not enabled. It also makes lots of sense to have that without locale module, since for an international shop for example with all English UI, recording country information is still vital. Also, @sun argued above that even the language list should not be confined to locale module, that might be used by other modules (though no direct example in core).

It is clearly an option that we rename locale module wholesale to something else and then we'll free up "locale" as the industry accepted concept of collections of data to use depending on locale, so we could move there then. However, as it is now, "locale" in core means the language list and UI translation and mixing it with other understandings of locale sounds like a mistake.

Sorry for that, added http://drupal.org/node/1276626.

Tagging for base language system. Lowering priority since the original issue was definitely not critical.

@sir_squall: RFC 4646 has long been obsoleted by BCP 47. See http://www.w3.org/International/articles/language-tags/ for more information.

The entries in the registry follow certain conventions with regard to upper and lower letter-casing. For example, language tags are lower case, alphabetic region subtags are upper case, and script tags begin with an initial capital. This is only a convention! When you use these subtags you are free to do as you like, unless you are constrained by the rules of the system you are working with. For HTML and XML language markup, the case should not matter.

(Emphasis from me). I'm not sure there is anything to fix here.

Feel free to open an issue for this that would let us discuss in detail and link it back here. Thanks!