Problem/Motivation

When building queries that make use of timestamp field by granularity (e.g. \Drupal\views\Plugin\views\sort\Date::query in core and anything custom/contrib that follows that pattern), time zones that have varying offsets through the year (e.g. Europe/London which is +00:00 over winter and +01:00 over summer) have broken results.

This is because this method works by getting a timezone local date by forcing GMT (+00:00) in \Drupal\views\Plugin\views\query\Sql::setupTimezone and then for the expression generating a date by getting unix timestamp 0 and adding the relevant number of seconds, followed by the required timezone offset (

DATE_ADD('19800101',
 INTERVAL $field SECOND) + INTERVAL $offset_seconds SECOND

).

This breaks in timezones with varying offsets, as it uses the offset at the time of the query being run, not the relevant offset of the timestamp we are manipulating. This results in unpredictable results for anything between 23:00 and 01:00 in the other part of the year (so summer times when querying in winter).

Proposed resolution

In my specific case, I have worked round it by overriding \Drupal\views\Plugin\views\query\Sql and preventing the explicit setting of timezone (so it uses SYSTEM) and simply using the built in FROM_UNIXTIME function.

However, I think this only works because mysql is on a server on Europe/London, as is php and my end use. When looking for a general solution, the best I could come up with was using CONVERT_TZ with mysql's timezone tables populated, but this doesn't seem like a great interopable solution.

Another approach:

I think (but I am definitely not certain of all use cases) \Drupal\views\Plugin\views\query\MysqlDateSql::setTimezoneOffset always setting a timezone rather than an offset should solve this. It then begs the question of whether there is any benefit to manually building the date in \Drupal\views\Plugin\views\query\MysqlDateSql::getDateField rather than using FROM_UNIXTIME.

Remaining tasks

When we think we have a solution, we should test it with MySQL, PostreSQL, and SQLite.

API changes

??

CommentFileSizeAuthor
#14 2923131-13-test-only.patch9.92 KBbenjifisher

Comments

andrewbelcher created an issue. See original summary.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dubcanada’s picture

Priority: Major » Critical

Also getting this issue.

For example if you have an exposed date and you have a timezone with saving time (for example America/Halifax which is -3 or -4 depending on time) is always INTERVAL -14400 regardless of time of year.

dubcanada’s picture

This seems to be an issue because getTimezoneOffset in the QueryPluginBase uses the current time to get the offset, rather than the requested time via the exposed date.

blacklabel_tom’s picture

Hi,

Bumping into this one too.

Would it make sense to add a timezone override in the Advanced => Query options so we can set timezones for specific views?

This would cover my use case where most views are fine, but a couple are to do with bookings saved in UTC which the view should use.

Cheers

Tom

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

pameeela’s picture

Priority: Critical » Normal

This is normal based on the documented priority definitions, especially since the IS mentions a workaround.

pameeela’s picture

Issue tags: +Bug Smash Initiative
joachim’s picture

> I have worked round it by overriding \Drupal\views\Plugin\views\query\Sql

A workaround that involves creating your own custom Views query plugin is a pretty hardcore solution that's going to be way out of the reach of many developers.

I don't think that's what the issue priority definitions mean by a workaround, so I'd say this should be a major at least.

pameeela’s picture

Priority: Normal » Major

Fair enough! Updated.

benjifisher’s picture

Version: 8.9.x-dev » 9.2.x-dev
Issue summary: View changes
Status: Active » Needs review
StatusFileSize
new9.92 KB

I just closed #3057780: Views QueryPluginBase::getTimezoneOffset doesn't take into account DST for dates other then now as a duplicate of this issue. What do you think of keeping the title from that issue?

This bug led to #3207086: [HEAD BROKEN] Consistent failure in MonthDatePluginTest. I started looking into that and found my way to getTimezoneOffset(). Then I searched the issues for mentions of that function, which is how I found this issue and #3057780.

I am attaching a test-only patch that proves the bug. I am setting the status to NR to trigger a test. Unless the testbot is mad at me, it should set the status back to NW. The failing test is Drupal\Tests\views\Functional\Plugin\MonthDatePluginTest::testMonthDatePlugin.

benjifisher’s picture

Issue summary: View changes

I found another duplicate: #3181657: Views date timezone handling is broken for zones with DST. That issue also has a failing test, and a different proposed resolution. I am copying it into the issue summary here.

I already added a note to the issue summary that we should test with MySQL, PostgreSQL, and SQLite when we think we have a fix. I do not think that we have to exercise all three on the test-only patch.

Status: Needs review » Needs work

The last submitted patch, 14: 2923131-13-test-only.patch, failed testing. View results

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

snowee@swis.nl’s picture

Just wanted to chime in here with the situation where I ran into issues relating to this. Whilst the originally reported issue is not quite the same, the other tickets that mention the root cause I ran into were closed as duplicates of this issue; so I'm placing this here.

In my case the root cause was due to \Drupal\views\Plugin\views\query\QueryPluginBase::getTimezoneOffset() as also reported in #3057780 and #3181657

Let me describe the confluence of circumstances that triggered the issue for me:
- The datetime entered in the CMS is in the user's timezone (Europe/Amsterdam) in this case
- The datetime is for an all day event, which means the entered time is set to 00:00:00 (this is date_recur's handling of all day events)
- The datetime is properly stored in the database as UTC
- It is currently winter time
- The date we are filtering the view on is in summer time

Due to the timezone offset being calculated being incorrect (only 3600 seconds (1 hour) interval is added to the database values in the query) all day events would show a day early when in winter time and looking at summertime events.

Our solution was to create a custom filter that extends the datetime Date filter, do our own timezone offset calculation (based upon the filter value) and disable the original calculation.

Since it was mentioned that creating a plugin might be out of reach for many developers, I'll share our solution here so it can be used as a starting point for others:


namespace Drupal\custom_event\Plugin\views\filter;

use Drupal\datetime\Plugin\views\filter\Date;
use Drupal\views\Attribute\ViewsFilter;
use Drupal\views\Plugin\views\query\Sql;

#[ViewsFilter("custom_event_schedule")]
final class EventScheduleFilter extends Date {

  /**
   * @see \Drupal\views\Plugin\views\query\QueryPluginBase::getTimezoneOffset()
   * @link https://www.drupal.org/project/drupal/issues/2923131
   * @link https://www.drupal.org/project/drupal/issues/3057780
   * @link https://www.drupal.org/project/drupal/issues/3181657
   */
  protected function opSimple($field): void {
    $query = $this->query;
    assert($query instanceof Sql);

    $date_value = $this->value['value'];
    if ($this->options['granularity'] == 'year') {
      $date_value = preg_replace('/^(\d{4})$/', '$1-01-01', $date_value);
    }

    $dtz = new \DateTimeZone(date_default_timezone_get());
    $dt = new \DateTime($date_value, $dtz);
    $offset = $dtz->getOffset($dt);

    // Replace default offset calculation.
    // The default calculation is based on 'now' instead of input value.
    $this->calculateOffset = FALSE;
    $query->setFieldTimezoneOffset($field, $offset);

    parent::opSimple($field);
  }

}

and then overriding the specific field's filter via the module file:

/**
 * Implements hook_views_data_alter().
 */
function custom_event_views_data_alter(array &$data): void {
  assert(isset($data['date_recur__node__event__schedule']['event__schedule_value']['filter']['id']));
  $data['date_recur__node__event__schedule']['event__schedule_value']['filter']['id'] = 'custom_event_schedule';
}

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.