America/Montreal and other linked timezones missing from Default time zone setting

Drupal uses timezone_identifiers_list() to enumerate the timezones.

I think this may be a PHP bug:

<?php
print_r(timezone_identifiers_list());

The output does not contain Montreal for me on PHP 5.6.10.

See also https://core.trac.wordpress.org/ticket/26656
https://answers.launchpad.net/ubuntu/+source/tzdata/+question/239234

If you could, please open a PHP bug ticket on timezone_identifiers_list() and link it here.

Now that is another thing altogether. Drupal 8 is probably the same so we have to fix it there first.

With #5:

The patch #5 works well for me on the latest 8.0.0-dev git cloned version (system and user timezone).

#5 works well for me. Finally I can see Australia/Canberra now, thanks for it.

So... following the links on https://bugs.php.net/bug.php?id=66102 and looking at https://github.com/eggert/tz/commit/242980fc79ef5fb676383f91505f9cbce6ff...

zone.tab is not a complete list of zones; it's merely a list for use by tzselect and similar programs, i.e., it's for selecting TZ values appropriate for users' needs. It omits redundant entries that would complicate the task of TZ selection. America/Montreal is redundant and so it is omitted. (Earlier, we thought it wasn't redundant, but that was an error.)

Why doesn't that apply to Drupal?

I think this is OK, but it may need sign-off from someone (who?)

Note the patch is not hardcoded to one country, despite the name. What we're doing is calling timezone_identifiers_list(), which is an alias of DateTimeZone::listIdentifiers(), with the 'ALL_WITH_BC' constant rather than the default 'ALL', where 'BC' = backwards compatible.

- Can reproduce original problem - America/Montreal missing from timezone list.
- Fixed by patch (applies cleanly to 8.3.x)
- Single line patch with no coding issues.
- I assume this doesn't merit a change record as it's merely adding extra options, not altering any existing settings
- Note the patch now increases the list from 417 to 474 options - i.e. 57 changes, so either it's been pruned or the ref to 160 in #2 was a typo. [Edit - see comment #19 - the exact length depends on PHP version and DB used, and the 100 or so missing is because ignore lots of abbreviations etc. that aren't in the Continent/City format]
- To the best of my knowledge, this would, in core, affect: the site regional setting (on install and in config), individual user timezones and the Date field widget ("time zone override")
- Issue summary updated

On the 2013e release of the tz database, America/Montreal was one of the zones removed from zone.tab, but it is still available in the northamerica file. It was because the group maintaining the database "incorrectly thought that Montreal differed from Toronto in the early 1970s".[1] The database primarily define zones that differ after 1970 (Unix time epoch), yet people might want to obtain Montreal's pre-1970 time changes, and linking America/Montreal to America/Toronto is not completely accurate. Thus, the decision was to keep this entry, pending a mechanism to winnow out zones like this to a new category.[1]

From the wikipedia article. We need to know the pros and cons of using a list with BC. The pro has been stated as UX but what about interoperability with systems that don't use a BC list?

Yep, in PHP you can load the backward compatible ones regardless:

echo strtotime('now America/Montreal');
echo strtotime('now Australia/Canberra');

and it looks to me like they're giving correct timestamps (i.e. matching Toronto and Sydney respectively.)
If you supply a nonsense timezone it returns false, as does a partial match - e.g. America/M - just to rule out any pattern matching.

Likewise Unix seems to support the extended list:

$ TZ=America/Toronto date; TZ=America/Montreal date

Mon Feb 13 16:02:39 EST 2017
Mon Feb 13 16:02:39 EST 2017

$ TZ=Australia/Canberra date; TZ=Australia/Sydney date
Tue Feb 14 08:02:52 AEDT 2017
Tue Feb 14 08:02:52 AEDT 2017

That thing I mentioned about 57 versus 160 more timezones? It's because we have a hardcoded regexp matching the continents and UTC:

if (preg_match('!^((Africa|America|Antarctica|Arctic|Asia|Atlantic|Australia|Europe|Indian|Pacific)/|UTC$)!

and the full contains loads of stuff like:

• Etc/GMT+0
• Etc/GMT+1
• MST
• MET
• WET
• Canada/East
• Canada/Central
• Canada/West

Also my test servers have marginally different versions of PHP 7 - one of which has a few more options that the other.

So #19 suggests that the BC list is different between different versions of PHP - what does this mean for database transportability? What happens if a user that selects a timezone that exists and then the PHP version changes?

Some further info (I'm not proposing a solution btw, nowhere near enough experience):

PHP uses the "Olson" timezone DB (the IANA one). Supposedly, you can add this PECL extension if you're unable to upgrade PHP itself.
There's a function timezone_version_get() to see which DB version you have (its on phpinfo() as well).
Versions numbering is year.release - e.g. version_compare('2016.1','2016.10'); would tell you which was more recent.
I've not investigated whether timezones are ever removed from the DB / how they maintain backward compatibility.

I'm neutral to negative on the current solution.

Have we surveyed other systems as to whether they use the BC list or not?

I am also concerned about the scope of this issue. The bug report is that 'America/Montreal' isn't listed because of bug in the IANA tzdata, but the solve is to add in all of the BC zones.

I am not sure if adding BC is proper. What are we being BC with? This shouldn't be about convenience, but true backwards compatibility.

On one hand, people shouldn't be using BC for a new install; there is nothing to be BC with. On the other hand, BC may be beneficial when the TZ identifier is used to communicate with an outside system (but I think that the TZ offset for a particular date/time is the more common case, eg ISO string).

I think we should consider two options.

1. Maintain a list of "extra" timezones that we merge into `\DateTimeZone::listIdentifiers(\DateTimeZone::ALL)` to account for bugs in tzdata. And yes, I would really like to get rid of that global function. For this I would propose creating a DateTimeZonePlus class that extends \DateTimeZone to parallel DateTimePlus that has this functionality. And then, replace timezone_identifiers_list() with `DateTimeZonePlus::listIdentifiers()` (this idea is still a bit half-baked, not at peak caffeine load yet).

2. Have a checkbox on install and/or a setting that would enable using \DateTimeZone::ALL_WITH_BC.

They may not be mutually exclusive.

Of those, I prefer #1.

@mpdonadio:

Neutral questions/observations:

- do we want another list to maintain ourselves? Arguably there are benefits for our list of countries, where we've shortened some of the more clunky ISO names, is same applicable to timezones? e.g. how many "bugs in tzdata" are there likely to be? would we have to monitor all changes to daylight savings dates as well?
- I like the (apparent) simplicity of #2, but, as you say, they need not be mutually exclusive.
- any setting should clearly be off by default, however people have to specify the site's timezone on installation, before they get a chance to adjust the config setting
- how many sites does this actually affect anyway - is it worth the effort (i.e. is it an inconvenience "I wish my city was on this list rather than [some other place]", or a bug "this missing timezone causes a problem with my client's site because...")
- For #1, are you suggesting deprecating system_time_zones() and removing in 9.0.0, or just changing what it wraps around?
- consensus here seems to be BC could be risky, imho doesn't seem a bad thing to give people a choice

From https://github.com/eggert/tz/blob/master/northamerica#L1416-L1430

# From Paul Eggert (2015-03-24):
# See America/Toronto for most of Quebec, including Montreal.
#
# Matthews and Vincent (1998) also write that Quebec east of the -63
# meridian is supposed to observe AST, but residents as far east as
# Natashquan use EST/EDT, and residents east of Natashquan use AST.
# The Quebec department of justice writes in
# "The situation in Minganie and Basse-Côte-Nord"
# http://www.justice.gouv.qc.ca/english/publications/generale/temps-minganie-a.htm
# that the coastal strip from just east of Natashquan to Blanc-Sablon
# observes Atlantic standard time all year round.
# http://www.assnat.qc.ca/Media/Process.aspx?MediaId=ANQ.Vigie.Bll.DocumentGenerique_8845en
# says this common practice was codified into law as of 2007.
# For lack of better info, guess this practice began around 1970, contra to
# Shanks & Pottenger who have this region observing AST/ADT.

America/Montreal is not in the latest version of zone.tab. It is also not different on http://www.nrc-cnrc.gc.ca/eng/services/time/time_zones.html or http://www.nrc-cnrc.gc.ca/eng/services/time/faq/index.html#Q5 (my RASC handbook lists these as the definitive references for Canada). I would follow those sources over the current Wikipedia article.

Based on this, I am not convinced we have an actual problem to solve.

Re rolled the patch and applied, Please review.

Hello

Tested #25 america_montreal_and-2530306-25.patch

1) Created a new Drupal 8.3.x site.
2) Selected America/Montreal as a timezone.
3) Clicked Save
4) Site successfully installs with no error messages & No errors in log
5) I visit /admin/config/regional/settings and see that America/Montreal is the timezone.

This appears to pass

Screens shots below showing testing

Thanks for the work, but everyone needs to re-read the issue rather than just re-rolling and approving the patch – we don't have agreement on whether (a) there is actually a problem or (b) that the patch is a suitable solution.

I am still not convinced that we have an actual problem. I am also 100% against adding the whole BC list, even if we continue to filter against the PREG above.

America/Montreal is the same as America/Toronto post-1970. The IANA list has America/Toronto in northamerica.

The BC list will do two things:

1. It will bring in the backzone data, which is a partial reconstruction of pre-1970 data. It is incomplete and changes.
2. It will bring in the backward data, which are the links to allow old names to be used in legacy systems (when location names change, when compound locations get introduced, etc).

The problem is that \DateTimeZone::listIdentifiers(\DateTimeZone::ALL_WITH_BC) does not distinguish between the two.

#1 may be a good idea for people who need pre-1970 behavior, but #2 will add duplicate entries when compound locations are in zoneinfo proper (eg, America/Indiana/Indianapolis and America/Indianapolis). The other problem is that the full zoneinfo database can very per system, depending on what version of PHP they are using (see https://3v4l.org/3sEMa) or if they have the PECL extension. There is no feasible way to maintain this in our code; we currently have a critical issue because of broken HEAD directly due to a change in zoneinfo.

You can also do a config override:

$ drush -y cset system.date timezone.default 'America/Montreal'

just worked for me.

This isn't about adding more cities to make things easier for people. We have issues to improve the UX of this

- #2847651: Improve timezones selector with optgroups
- #2853386: Interactive time-zone picker

The second is really worth exploring. I don't like the map idea, but would love it if we could come up with something similar to what tzselect does.

If someone thinks we have a problem, then make an Issue Summary update that describes what the problem is, and what a solution could be. Otherwise, I think this needs to be Closed (Won't Fix).

@bohemier, do other systems you use have "America/Montreal" as a TZ option (the IANA name), not just "Montreal, Canada?" (the nearest city)?

Another possible solve here is to allow contrib solutions by introducing `hook_system_time_zones_alter(&$zones)`, but that wouldn't necessarily help at the install stage.

diff --git a/core/modules/system/system.module b/core/modules/system/system.module
index a1a6b71..5849dcf 100644
--- a/core/modules/system/system.module
+++ b/core/modules/system/system.module
@@ -1360,6 +1360,7 @@ function system_mail($key, &$message, $params) {
  */
 function system_time_zones($blank = NULL) {
   $zonelist = timezone_identifiers_list();
+  \Drupal::moduleHandler()->alter('system_time_zones', $zonelist);
   $zones = $blank ? ['' => t('- None selected -')] : [];
   foreach ($zonelist as $zone) {
     // Because many time zones exist in PHP only for backward compatibility

or

diff --git a/core/modules/system/system.module b/core/modules/system/system.module
index a1a6b71..573ab03 100644
--- a/core/modules/system/system.module
+++ b/core/modules/system/system.module
@@ -1369,6 +1369,7 @@ function system_time_zones($blank = NULL) {
       $zones[$zone] = t('@zone', ['@zone' => t(str_replace('_', ' ', $zone))]);
     }
   }
+  \Drupal::moduleHandler()->alter('system_time_zones', $zones);
   // Sort the translated time zones alphabetically.
   asort($zones);
   return $zones;

I have read the Issue Summary and most of the comments and this seems like a task and not a bug.

The issue was opened to resolve usability problems with the timezone (#30), specifically that some timezones where no longer in the list. A patch was developed by later reviews raised concerns about making the change. Some are:

• about interoperability with systems that don't use a BC list, #17
• Different versions of PHP use different BC list. #19 and #20
• A new install should not be using a BC list. #22
• Not being convinced there is a problem. #29

At the end a solution of adding a hook was agreed to but there has been no action on that in 5 years. The patch that is here add the BC list and has been rerolled a few times (even though a decision to use that approach was not yet agreed to).

Given the opposition to making this change this could be closed as works as designed. I'll ask in #bugsmash.