Problem/Motivation
When building queries that make use of timestamp field by granularity (e.g. \Drupal\views\Plugin\views\sort\Date::query in core and anything custom/contrib that follows that pattern), time zones that have varying offsets through the year (e.g. Europe/London which is +00:00 over winter and +01:00 over summer) have broken results.
This is because this method works by getting a timezone local date by forcing GMT (+00:00) in \Drupal\views\Plugin\views\query\Sql::setupTimezone and then for the expression generating a date by getting unix timestamp 0 and adding the relevant number of seconds, followed by the required timezone offset (
DATE_ADD('19800101',
INTERVAL $field SECOND) + INTERVAL $offset_seconds SECOND).
This breaks in timezones with varying offsets, as it uses the offset at the time of the query being run, not the relevant offset of the timestamp we are manipulating. This results in unpredictable results for anything between 23:00 and 01:00 in the other part of the year (so summer times when querying in winter).
Proposed resolution
In my specific case, I have worked round it by overriding \Drupal\views\Plugin\views\query\Sql and preventing the explicit setting of timezone (so it uses SYSTEM) and simply using the built in FROM_UNIXTIME function.
However, I think this only works because mysql is on a server on Europe/London, as is php and my end use. When looking for a general solution, the best I could come up with was using CONVERT_TZ with mysql's timezone tables populated, but this doesn't seem like a great interopable solution.
Another approach:
I think (but I am definitely not certain of all use cases) \Drupal\views\Plugin\views\query\MysqlDateSql::setTimezoneOffset always setting a timezone rather than an offset should solve this. It then begs the question of whether there is any benefit to manually building the date in \Drupal\views\Plugin\views\query\MysqlDateSql::getDateField rather than using FROM_UNIXTIME.
Remaining tasks
When we think we have a solution, we should test it with MySQL, PostreSQL, and SQLite.
API changes
??
| Comment | File | Size | Author |
|---|---|---|---|
| #14 | 2923131-13-test-only.patch | 9.92 KB | benjifisher |
Comments
Comment #4
dubcanada commentedAlso getting this issue.
For example if you have an exposed date and you have a timezone with saving time (for example America/Halifax which is -3 or -4 depending on time) is always INTERVAL -14400 regardless of time of year.
Comment #5
dubcanada commentedThis seems to be an issue because getTimezoneOffset in the QueryPluginBase uses the current time to get the offset, rather than the requested time via the exposed date.
Comment #6
blacklabel_tom commentedHi,
Bumping into this one too.
Would it make sense to add a timezone override in the Advanced => Query options so we can set timezones for specific views?
This would cover my use case where most views are fine, but a couple are to do with bookings saved in UTC which the view should use.
Cheers
Tom
Comment #9
pameeela commentedThis is normal based on the documented priority definitions, especially since the IS mentions a workaround.
Comment #10
pameeela commentedComment #11
joachim commented> I have worked round it by overriding \Drupal\views\Plugin\views\query\Sql
A workaround that involves creating your own custom Views query plugin is a pretty hardcore solution that's going to be way out of the reach of many developers.
I don't think that's what the issue priority definitions mean by a workaround, so I'd say this should be a major at least.
Comment #12
pameeela commentedFair enough! Updated.
Comment #14
benjifisherI just closed #3057780: Views QueryPluginBase::getTimezoneOffset doesn't take into account DST for dates other then now as a duplicate of this issue. What do you think of keeping the title from that issue?
This bug led to #3207086: [HEAD BROKEN] Consistent failure in MonthDatePluginTest. I started looking into that and found my way to
getTimezoneOffset(). Then I searched the issues for mentions of that function, which is how I found this issue and #3057780.I am attaching a test-only patch that proves the bug. I am setting the status to NR to trigger a test. Unless the testbot is mad at me, it should set the status back to NW. The failing test is
Drupal\Tests\views\Functional\Plugin\MonthDatePluginTest::testMonthDatePlugin.Comment #15
benjifisherI found another duplicate: #3181657: Views date timezone handling is broken for zones with DST. That issue also has a failing test, and a different proposed resolution. I am copying it into the issue summary here.
I already added a note to the issue summary that we should test with MySQL, PostgreSQL, and SQLite when we think we have a fix. I do not think that we have to exercise all three on the test-only patch.
Comment #22
snowee@swis.nl commentedJust wanted to chime in here with the situation where I ran into issues relating to this. Whilst the originally reported issue is not quite the same, the other tickets that mention the root cause I ran into were closed as duplicates of this issue; so I'm placing this here.
In my case the root cause was due to \Drupal\views\Plugin\views\query\QueryPluginBase::getTimezoneOffset() as also reported in #3057780 and #3181657
Let me describe the confluence of circumstances that triggered the issue for me:
- The datetime entered in the CMS is in the user's timezone (Europe/Amsterdam) in this case
- The datetime is for an all day event, which means the entered time is set to 00:00:00 (this is date_recur's handling of all day events)
- The datetime is properly stored in the database as UTC
- It is currently winter time
- The date we are filtering the view on is in summer time
Due to the timezone offset being calculated being incorrect (only 3600 seconds (1 hour) interval is added to the database values in the query) all day events would show a day early when in winter time and looking at summertime events.
Our solution was to create a custom filter that extends the datetime Date filter, do our own timezone offset calculation (based upon the filter value) and disable the original calculation.
Since it was mentioned that creating a plugin might be out of reach for many developers, I'll share our solution here so it can be used as a starting point for others:
and then overriding the specific field's filter via the module file: