It would be useful to provide a migrate process plugin that allows migrating data from the Date module on D7.

Issue fork smart_date-3060042

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

DamienMcKenna created an issue. See original summary.

DamienMcKenna’s picture

Project: Smart Trim » Smart Date
matt.mendonca’s picture

While working on our migration, I had to write a plugin to convert D7 date, conditional and time fields (business rules) to D8 smart date. This plugin isn't general purpose, but maybe it can help serve as an example / guide for other people.

Some notes / caveats:
- It was meant for internal purposes, so it could probably be refactored / cleaned up (right now it's good enough for our needs)
- It takes source values from 3 different fields, not just 1 to 1 from date field
- There are some short cuts taken in the interest of getting our migrations done quickly (source mapping not passed in config, but looked up at run time in the plugin)
- I had originally intended to just process the top level smart date field with the transform method, but ended up having to process the "sub fields" (start, end, duration) individually, hence why the transform method does the all work for all the sub fields
- The NISTBaseProcessPlugin class just provides the log and throwPluginException utility methods used in the validate methods

Anyway, here is what the yaml mapping looks like:

field_event_date/value:
  plugin: date_time_to_smart_date
field_event_date/end_value:
  plugin: date_time_to_smart_date
field_event_date/duration:
  plugin: date_time_to_smart_date

and here is the plugin code:


namespace Drupal\nist_migrate_process_plugin\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\Row;

/**
 * Transforms the D7 date field and time field values into a smart date value.
 *
 * @MigrateProcessPlugin(
 *   id = "date_time_to_smart_date"
 * )
 */
class DateTimeToSmartDate extends NISTBaseProcessPlugin {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    // Since the smart date field is a multi property field, this
    // gets only the sub property - i.e. end_value, duration, etc.
    $subProperty = explode('/', $destination_property);
    $subProperty = $subProperty[1];

    $dateFieldValue = $this->getFieldValue($row, 'field_date');
    $timeFieldValue = $this->getFieldValue($row, 'field_time');

    // This field is reversed for whatever reason. True = *not* full day event
    // False = *is* full day event. To avoid confusion, flip it.
    $fullDayEventValue = (bool) $this->getFieldValue($row, 'field_full_day_event');
    $isFullDayEvent = !$fullDayEventValue;

    $startTimestamp = 0;
    $endTimestamp = 0;

    /*
     * Since some of the fields are derived from others and the
     * calculations aren't heavy duty, go ahead and always
     * calculate all the fields and just return what we need.
     */
    if ($dateFieldValue) {
      if (!empty($dateFieldValue['value'])) {
        $startTimestamp = $this->getTimestampFromDate($dateFieldValue['value']);

        // Default the end date to match the start date
        // since worst case scenario we calculate it if set.
        $endTimestamp = $startTimestamp;
      }

      if (!empty($dateFieldValue['value2']) && ($dateFieldValue['value2'] !== $dateFieldValue['value'])) {
        $endTimestamp = $this->getTimestampFromDate($dateFieldValue['value2']);
      }
    }

    if ($timeFieldValue) {
      // Since the startTimestamp and endTimestamp timestamps are based on
      // midnight, and the D7 timeField value is the number of
      // seconds since midnight, we should be able to just add the
      // timeFieldValue to the corresponding timestamps.
      if (!empty($timeFieldValue['value'])) {
        $startTimestamp += $timeFieldValue['value'];
      }

      if (!empty($timeFieldValue['value2'])) {
        $endTimestamp += $timeFieldValue['value2'];
      }
      elseif (!empty($timeFieldValue['value'])) {
        $endTimestamp += $timeFieldValue['value'];
      }
    }

    if ($isFullDayEvent) {
      $endTimestamp = $this->convertTimeStampToFullDay($endTimestamp);
    }

    // Calculate the duration by diffing the start and end date
    // and convert to minutes (timestamp is in seconds)
    $duration = ($endTimestamp - $startTimestamp) / 60;

    $smartDateFieldValues = [
      'value' => $startTimestamp,
      'end_value' => $endTimestamp,
      'duration' => $duration,
    ];

    $fieldValue = $this->validate($subProperty, $smartDateFieldValues[$subProperty]);

    return $fieldValue;
  }

  /**
   * Helper to get the specified field value from the source data.
   *
   * Also this will potentially clean up the value
   * (extract the value from nested arrays, etc.).
   *
   * @param Drupal\migrate\Row $row
   *   Migrate row.
   * @param string $fieldName
   *   Fieldname.
   *
   * @return any
   *   Field value.
   */
  private function getFieldValue(Row $row, string $fieldName) {
    $fieldValue = NULL;
    $sourceFieldValue = $row->getSource()[$fieldName];

    if (!empty($sourceFieldValue)) {
      $fieldValue = $sourceFieldValue;
    }

    if (is_array($fieldValue) && (count($fieldValue) === 1)) {
      $fieldValue = array_values($fieldValue)[0];
    }

    if (is_array($fieldValue) && (count($fieldValue) === 1) && isset($fieldValue['value'])) {
      $fieldValue = $fieldValue['value'];
    }

    return $fieldValue;
  }

  /**
   * Helper function to get the unix timestamp value from a given date string.
   *
   * @param string $date
   *   Date.
   *
   * @return int
   *   Timestamp.
   */
  private function getTimestampFromDate(string $date): int {
    // Since the time is stored in a separate field in D7,
    // ignore any time information from the date field.
    $timestamp = explode(' ', $date);
    $timestamp = strtotime($timestamp[0]);

    return $timestamp;
  }

  /**
   * Helper to convert a timestamp to the smart date all day format.
   *
   * @param int $timestamp
   *   Timestamp.
   *
   * @return int
   *   All day Timestamp.
   */
  private function convertTimeStampToFullDay(int $timestamp): int {
    // If this is a full day event, we need to set the ending timestamp
    // to be at 11:59:00pm on the last day.
    $fullDayEventEndTime = "11:59:00pm";
    $date = date('Y-m-d', $timestamp);
    $date .= " {$fullDayEventEndTime}";
    $timestamp = strtotime($date);

    return $timestamp;
  }

  /**
   * Helper function to do some basic validation on the generated data.
   *
   * @param string $propertyName
   *   Property name.
   * @param int $propertyValue
   *   Property value.
   *
   * @return any
   *   Property value if validated or NULL.
   */
  private function validate(string $propertyName, int $propertyValue) {
    if ($propertyValue < 0) {
      $errorMessage = "DateTimeToSmartDate->validate: {$propertyName} shouldn't be less than 0 (is {$propertyValue}).";
      $this->log($errorMessage);
      $this->throwPluginException($errorMessage);
      return NULL;
    }

    return $propertyValue;
  }

}

DamienMcKenna’s picture

Any reason to not just import the start & end values as-is, then have a simpler function for filling in the "duration" value?

mandclu’s picture

@DamienMcKenna it doesn't seem like there's a lot of logic currently in there to set the duration. What's your interest in doing that separately?

trackleft2’s picture

FYI when adding recurring event data from d7 into smart_date d8/9 You have to:
1. Destructure/Parse the repeat rule
2. Create a new 'smart_date_rule' from each rule

INSERT INTO `smart_date_rule` (`rid`, `uuid`, `rule`, `freq`, `limit`, `parameters`, `unlimited`, `entity_type`, `bundle`, `field_name`, `start`, `end`, `instances`)
VALUES
	(1, 'c50427e4-ad51-4966-8d62-bcfa249e6f5e', 'RRULE:FREQ=YEARLY;BYMONTHDAY=14;BYMONTH=3', 'YEARLY', NULL, 'BYMONTHDAY=14;BYMONTH=3', 0, 'node', 'bundle_name', 'field_name', 1615680000, 1615680000, X'613A303A7B7D');

The instances field is generated on the fly by smart_date, but you can trigger like this

        $rrule = SmartDateRule::load($value);
        $before = NULL;
        // Retrieve all instances for this rule, with overrides applied.
        if ($rrule->limit->isEmpty()) {
          $before = strtotime('+ 24 months');
        }
        $instances = $rrule->getRuleInstances($before);
        $rrule->set('instances', ['data' => $instances]);

        $rrule->save();

Once you have instances you can save via multi-value event field

        $first_instance = FALSE;
        $instancesAgain = $rrule->getRuleInstances($before);

        foreach ($instancesAgain as $rrule_index => $instance) {
          // Apply instance values to our template, and add to the field values.
          $first_instance['value'] = $instance['value'];
          $first_instance['end_value'] = $instance['end_value'];
          // Calculate the duration, since it isn't returned.
          $first_instance['duration'] = ($instance['end_value'] - $instance['value']) / 60;
          $first_instance['rrule_index'] = $rrule_index;
          $first_instance['rrule'] = $rrule->id();
          $first_instance['timezone'] = 'America/Phoenix';
          $values[] = $first_instance;
        }
        return $values;

Hope this helps.

BenStallings’s picture

In case it helps anyone, here is the custom process plugin I wrote (in a custom module called sg_migrate):


namespace Drupal\sg_migrate\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Drupal\smart_date_recur\Entity\SmartDateRule;

/**
 * Parses Drupal 7 date_repeat values into smart_date_recur values.
 *
 * @MigrateProcessPlugin(
 *   id = "smart_date_recur"
 * )
 */
class SmartDateRecur extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    static $rrule;
    if (!isset($rrule) || $rrule->getRule() != $value['rrule']) {
      // Create a new rrule.
      preg_match('/FREQ=([^;]+).*((COUNT|UNTIL)=[^;]+)/', $value['rrule'], $m);
      $rrule = SmartDateRule::create([
        'start' => strtotime($value['value']),
        'end' => strtotime($value['value2']),
        'rule' => $value['rrule'],
        'freq' => $m[1],
        'limit' => $m[2],
        'entity_type' => $this->configuration['entity_type'],
        'bundle' => $this->configuration['bundle'],
        'field_name' => $this->configuration['field_name'],
        'timezone' => $value['timezone'],
      ]);
      $rrule->save();
    }
    $parsed = [
      'value' => strtotime($value['value']),
      'end_value' => strtotime($value['value2']),
      'duration' => strtotime($value['value2']) - strtotime($value['value']),
      'rrule' => $rrule->id(),
    ];
    return $parsed;
  }

}

Example use:

process:
  field_dates:
    plugin: smart_date_recur
    source: field_when
    entity_type: node
    bundle: opportunity
    field_name: field_dates
BenStallings’s picture

Update: the plugin code I provided above is problematic because it doesn't populate the rule instances, which effectively prevents cron from running on the site. I can see that I should have generated the list of instances programmatically using the SmartDateRule object, but I can't make out which of that object's methods, if any, allow you to set the instances of a rule to a list of values.

BenStallings’s picture

After studying the conversation at https://www.drupal.org/project/smart_date/issues/3165402 , it sounds like maybe I should not be saving the field values at all as I did above, but instead turning them into instances of the rule and letting the module save the values to the field. Is that correct?

If so, is it just necessary to invoke SmartDateRule::create() on an array that defines 'instances' as an array of start & end values, and then ->save() the resulting rule object?

Thanks for the feedback.

mandclu’s picture

Unfortunately, it isn't quite that easy. When migrating to a recurring event, you will want to create the rule, but then use the rule object to generate the instances, then add those instances as Smart Date field values, but with the rule id set as the rrid. Does that help?

BenStallings’s picture

That does help, except that I'm not sure how to handle any exceptions to the rules that may have been reflected in the source field values. (That's why I had been importing the values rather than generating them.) Maybe I'll just ignore any exceptions and tell the users to check their imported data for accuracy. Thanks for the feedback.

BenStallings’s picture

OK, here is what I wound up doing instead of the process plugin shown above. This is a custom source plugin, and it's got some hardcoded values specific to my use case, so please take it as a starting point.


namespace Drupal\sg_migrate\Plugin\migrate\source;

use Drupal\migrate\Row;
use Drupal\node\Plugin\migrate\source\d7\Node;
use Drupal\smart_date_recur\Entity\SmartDateRule;

/**
 * Volunteer opportunities.
 *
 * @MigrateSource(
 *   id = "sg_opportunity",
 * )
 */
class Opportunity extends Node {

  /**
   * Save an rrule for recurring dates.
   */
  public function prepareRow(Row $row) {
    parent::prepareRow($row);

    $when = $row->getSourceProperty('field_when');
    $timezone = \Drupal::config('system.date')->get('timezone')['default'];
    $dates = $rdata = [];
    $counter = 0;
    foreach ($when as $k => $d) {
      if (strtotime($d['value']) > 2147483647 || strtotime($d['value2']) > 2147483647) {
        // This value is out of range due to the Year 2038 bug.
        $counter = $k+1;
        continue;
      }
      if ($d['rrule'] > '') {
        if (isset($rdata['rule']) && $d['rrule'] != $rdata['rule']) {
          // Save the old rule before continuing.
          $rrule = SmartDateRule::create([$rdata]);
          foreach($rdata as $f => $v) {
            $rrule->set($f, $v);
          }
          $rrule->save();
          // Now that we have a rule ID, process the field values that belong to that rule.
          $ri = 1;
          for ($i = $counter; $i < $k; $i++) {
            $dates[$i] = [
              'value' => strtotime($when[$i]['value']),
              'end_value' => strtotime($when[$i]['value2']),
              'duration' => strtotime($when[$i]['value2']) - strtotime($when[$i]['value']),
              'rrule' => $rrule->id(),
              'rrule_index' => $ri,
              'timezone' => $timezone,
            ];
          }
          $counter = $k;
        }
        if (!isset($rdata['rule']) || $d['rrule'] != $rdata['rule']) {
          // Create a new rule with this as the first instance.
          preg_match('/FREQ=([^;]+).*((COUNT|UNTIL)=[^;]+)/', $d['rrule'], $m);
          $rdata = [
            'start' => strtotime($d['value']),
            'end' => strtotime($d['value2']),
            'rule' => $d['rrule'],
            'freq' => $m[1],
            'limit' => $m[2],
            'instances' => [
              'data' => [
                1 => [
                  'value' => strtotime($d['value']), 
                  'end_value' => strtotime($d['value2']),
                ],
              ],
            ],
            'entity_type' => 'node',
            'bundle' => 'opportunity',
            'field_name' => 'field_dates',
          ];
        }
        else {
          // Add this instance to an existing rule.
          $rdata['instances']['data'][] = [
            'value' => strtotime($d['value']), 
            'end_value' => strtotime($d['value2']),
          ];
        }

      }
      else {
        // This date doesn't repeat.
        $dates[$k] = [
          'value' => strtotime($d['value']),
          'end_value' => strtotime($d['value2']),
          'duration' => strtotime($d['value2']) - strtotime($d['value']),
          'timezone' => $timezone,
        ];
        $rdata = [];
        $counter = $k+1;
      }
    }
    if (isset($rdata['rule'])) {
      // Save the rule.
      $rrule = SmartDateRule::create([$rdata]);
      foreach($rdata as $f => $v) {
        $rrule->set($f, $v);
      }
      $rrule->save();
      // Process field values belonging to this rule.
      $ri = 1;
      for ($i = $counter; $i <= $k; $i++) {
        $dates[$i] = [
          'value' => strtotime($when[$i]['value']),
          'end_value' => strtotime($when[$i]['value2']),
          'duration' => strtotime($when[$i]['value2']) - strtotime($when[$i]['value']),
          'rrule' => $rrule->id(),
          'rrule_index' => $ri,
          'timezone' => $timezone,
        ];
        $ri++;
      }
    }
    // Save the field values.
    $row->setSourceProperty('dates', $dates);
  }

}

The migration YAML file then looks like:

source:
  plugin: sg_opportunity
  node_type: opportunity
process:
  field_dates: dates

because the values for field_dates have already been changed to the correct format by the source plugin.

BenStallings’s picture

Status: Active » Needs review

OK, I think I just created a merge request at https://git.drupalcode.org/project/smart_date/-/merge_requests/24 . I couldn't make a patch because this is a new file. The file is a process plugin that can handle not just Drupal 7 Date fields from a database source, but also from a Views data export. This was not a trivial effort because Views converts the rrules into a human-readable format, and I had to convert them back to rrules again.

I was scratching my own itch here and will not be offended if someone wants to make substantial changes. For example, it might make more sense to split this into two plugins, one for parsing the Views output and one for converting Drupal 7 Date data to Smart Date data. Then people who want to use both can just chain them together. However, I have no need to separate them, so I'm not doing that at this time.

trackleft2’s picture

FYI, you can make a patch from your MR by adding .patch to the end of the URL
https://git.drupalcode.org/project/smart_date/-/merge_requests/24.patch

Or diff like this
https://git.drupalcode.org/project/smart_date/-/merge_requests/24.diff

I wouldn't link to these merge request patches within composer.json on a production site, though since they update every time you commit to them, but they are useful for making patch files to upload to drupal.org.

BenStallings’s picture

Thanks for the tip, @trackleft2!

  • mandclu committed b4de6b2 on 3.6.x authored by BenStallings
    Issue #3060042: Provide migrate support to upgrade from Date module on...
mandclu’s picture

Version: 8.x-1.x-dev » 3.6.x-dev
Status: Needs review » Fixed

@DamienMcKenna @BenStallings Thanks very much for your work on this. Would have preferred to get a little more community input into how this works, but since the work here is really contained in its own class, if it still needs a little work we can figure that when any issues are identified.

Merged in, will include in a new release shortly.

danflanagan8’s picture

Thanks for the plugin, @BenStallings!

It seems to have almost worked for my case, which was migrating from a D7 site using date_recur to a D9 site using smart_date. I had to make two changes to the code.

1. I had to update the annotation to include handle_multiples = TRUE
2. I had to do a strict inequality on line 148:

// Now that we've sorted out the start & end values, we're finally ready to work with them.
if ((int)$d['value'] !== $d['value']) {

After making those changes, I got data to migrate and it was mostly correct. I ended up having trouble with timezones, which are different in my source data (UTC) and the target site (LA). Timezones always confuse me though, so I'm not sure if it was a bug in the process plugin or a problem somewhere else (like me!).

BenStallings’s picture

Status: Fixed » Needs work

Thanks for the feedback, @danflanagan8. Could you please supply your changes in a patch? I'm reopening the ticket until then.

danflanagan8’s picture

Status: Needs work » Needs review
FileSize
6.72 KB
7.63 KB
934 bytes

Here's a patch with the start of a test we can work against. It very basic and not at all DRY, but it's way better than nothing. Also the small fixes I proposed. Turns out the !== thing doesn't appear to be a problem in php8+, just in php7.

Though this project is apparently not configured to run tests on d.o.

Locally, both tests fail without the handle_multiples line. The second test fails because the 2nd and 3rd fields instances that are expected are not created.

This is perhaps my writing poor assertions if those are generated on cron or something like that. I'm new to the module!

BenStallings’s picture

Thanks again, @danflanagan8. I had trouble with the strict inequality because UNIX timestamps were being misidentified as strings, and I had to change it back to !=. Would something like this work for your data?

if ((int)$d['value'] == $d['value']) {
  // Make sure UNIX timestamps are integers and not strings.
  $d['value'] = (int)$d['value'];
  $d['value2'] = (int)$d['value2'];
}
else {
  // Convert other string values to UNIX timestamps for consistent processing.
  $d['value'] = strtotime($d['value']);
  $d['value2'] = strtotime($d['value2']);
}
BenStallings’s picture

Here's an updated patch with my fixes.

BenStallings’s picture

Dang it. Fixing a typo that was in #25.

BenStallings’s picture

Another bug bites the dust. I was trying to unset a loop variable. :facepalm:

BenStallings’s picture

And another one gone, and another one gone. Convert all-day dates from midnight-to-midnight to midnight-to-23:59:00.

BenStallings’s picture

Doing math on timestamps worked great right up until somebody spanned daylight savings time. :(

Doing all the math with PHP DateTime objects instead resolves the problem.

BenStallings’s picture

Only do date methods on a variable if it actually contains a date object..!

BenStallings’s picture

Handle some additional edge cases for Views' English output of rrules.
Remove parsing of Views output by commas, since its English output of rrules can contain commas. :facepalm:

nightlife2008’s picture

Hey everyone,

Thanks for providing the Process plugin code! It helped me to get things started, but instead of recreating the source's values in the destination drupal 8/9, I took another route to generate the actual date instances of a repeated date value...

In my setup, it became clear to me that some repetition items were missing in the source data values, which in turn prohibited me from creating override rules for some of those items.

Rather than breaking my head over this situation, I stepped back and had a fresh look onto things, leading me to this conclusion:

When looping over the source field values in the process plugin, I should just take each first item of a repeating sequence of values, or the actual value if not repeating.

In case of a repeating date value, I just ignore all other values of that same rule, and use SmartDateRule's code to generate the actual repetition-instances. That seemed to be the solution to make sure everything adds up in my destination drupal site...

below my custom process plugin based on the ParseDates process plugin from the patch:


namespace Drupal\aok_migrate\Plugin\migrate\process;

use Drupal\Core\Database\Database;
use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Drupal\smart_date_recur\Entity\SmartDateOverride;
use Drupal\smart_date_recur\Entity\SmartDateRule;

/**
 * Given Drupal 7 date ranges and/or repeating dates, or serialized values from Views, parse for a Smart Date field.
 *
 * Three formats of incoming data are accepted:
 * - Drupal 7 Date values of the form `['value' => START, 'value2' => END, 'rrule' => STRING]` (where rrule is optional);
 * - Views' "Date and Time" format, which you might use in an XML data export
 *   `<span class="date-display-start" ...>START</span> to <span class="date-display-end" ...>END</span>`
 *   with an optional `<div class="date-repeat-rule">STRING</div>` if you enable that option in the field of the View;
 * - or plain text serialized `START to END`.
 * The date values themselves can be in any format that works with PHP's strtotime() function.
 *
 * Example:
 *
 * @code
 * process:
 *  field_smart_date:
 *    plugin: activity_dates
 *    source: field_date
 *    entity_type: node
 *    bundle: opportunity
 * @endcode
 *
 * The entity_type and bundle are only necessary for repeating dates.
 *
 * @MigrateProcessPlugin(
 *   id = "activity_dates",
 *   handle_multiples = TRUE
 * )
 */

class ActivityDates extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    if (!is_array($value)) {
      // Check for multiple serialized values.
      foreach(['|', ';'] as $delimiter) {
        if (strpos($value, $delimiter)) {
          $value = explode($delimiter, $value);
          break;
        }
      }
      $value = (array) $value;
    }

    if (isset($value['value'])) {
      // Only one value for the field.
      $value = [$value];
    }

    $timezone = new \DateTimeZone('UTC');

    foreach ($value as $k => $d) {
      // Now that we've sorted out the start & end values,
      // we're finally ready to work with them.
      if ((int)$d['value'] == $d['value']) {
        // Make sure UNIX timestamps are integers and not strings.
        $d['value'] = (int)$d['value'];
        $d['value2'] = (int)$d['value2'];
      }
      else {
        // Convert other string values to UNIX timestamps for consistent processing.
        $d['value'] = (int)strtotime($d['value']);
        $d['value2'] = (int)strtotime($d['value2']);
      }

      $start = date_create_from_format('U', $d['value']);
      $start->setTimezone($timezone);
      $end = date_create_from_format('U', $d['value2']);
      $end->setTimezone($timezone);

      if ($end->format('H:i:s') == '00:00:00') {
        // If the end time is midnight, make it 23:59 instead.
        $end->setTime(23, 59);
        $d['value2'] = $end->format('U');
      }

      if ($d['value2'] > 2147483647) {
        // This value is out of range due to the Year 2038 Bug. https://en.wikipedia.org/wiki/Year_2038_problem
        // Although PHP is unfazed by the bug, MySQL and MariaDB are afflicted, so we can't store these values.
        unset($value[$k]);
        continue;
      }
    }

    // Now that the data is cleaned up, convert it to SmartDate format.
    $dates = $rdata = [];

    for ($k = 0; $k < count($value); $k++) {
      $d = $value[$k];

      // Skip any values that got unset above.
      if (!is_array($d)) continue;

      // Value with rrule
      // Value without rrule
      if (isset($d['rrule']) && !empty($d['rrule'])) {
        if (!isset($rdata['rule']) || $d['rrule'] != $rdata['rule']) {
          // Delete any EXDATEs in the past to reduce the rule length.
          if (preg_match('/EXDATE:[^;]+/', $d['rrule'], $match_exdate)) {
            preg_match_all('/(\d{8}T\d{6}Z+)/', $match_exdate[0], $matches_exdate_dates);

            foreach ($matches_exdate_dates[0] as $exdate_item) {
              if (substr($exdate_item, 0, 8) < date('Ymd')) {
                $d['rrule'] = str_replace($exdate_item, '', $d['rrule']);
              }
            }
          }

          $newrule = $d['rrule'];
          // Check the rrule for Year 2038 Bug compliance.
          preg_match('/FREQ=([^;]+).*((COUNT|UNTIL)=([^;]+)+)/', $newrule, $m);
          $freq = [
            'DAILY' => 24*60*60,
            'WEEKLY' => 7*24*60*60,
            'MONTHLY' => 31*24*60*60,
            'YEARLY' => 366*24*60*60
          ];
          $limit = $m[2];
          $limit_date_only = NULL;

          if ($m[3] == 'COUNT') {
            $enddate = (int)$d['value2'] + $m[4] * $freq[$m[1]];

            if ($enddate > 2147483647) {
              $limit = 'COUNT=' . (int)((2147483647 - (int)$d['value2'])/$freq[$m[1]]);
              $newrule = str_replace($m[2], $limit, $newrule);
            }
          }
          elseif ($m[3] == 'UNTIL') {
            // Decrement with one day for our rrule limit.
            $enddate = strtotime($m[4]);

            if ($enddate > 2147483647) {
              $limit = 'UNTIL=' . date('Y-m-d\THis', 2147483647);
              $limit_date_only = 'UNTIL=' . date('Y-m-d', 2147483647);
            }
            else {
              $limit = 'UNTIL=' . date('Y-m-d\THis', $enddate);
              $limit_date_only = 'UNTIL=' . date('Y-m-d', $enddate);
            }

            $newrule = str_replace($m[2], $limit, $newrule);
          }
          elseif (!isset($m[2])) {
            $limit = 'UNTIL=' . date('Y-m-d\THis', 2147483647);
            $limit_date_only = 'UNTIL=' . date('Y-m-d', 2147483647);

            $newrule .= ';' . $limit;
          }

          // Extract parameters from RRule.
          $parameters = [];
          preg_match_all('/(BYDAY|BYHOUR|BYMINUTE|BYMONTHDAY)=([^;]+)+/', $newrule, $matches_by);
          if (!empty($matches_by[0])) {
            foreach ($matches_by[0] as $index => $occurence) {
              // Remove the +.
              $replacement = str_replace('+', '', $occurence);
              $newrule = str_replace($occurence, $replacement, $newrule);

              $parameters[] = $replacement;
            }
          }

          preg_match_all('/(BYSETPOS|INTERVAL)=([^;]+)+/', $newrule, $matches_params);
          if (!empty($matches_params[0])) {
            foreach ($matches_params[0] as $index => $occurence) {
              $parameters[] = $occurence;
            }
          }

          $exdates = [];
          $rdates = [];
          if (preg_match('/EXDATE:[^;]+/', $newrule, $match_exdate)) {
            preg_match_all('/(\d{8}T\d{6}Z+)/', $match_exdate[0], $match_exdate_dates);
            if (!empty($match_exdate_dates[0])) {
              $exdates = $match_exdate_dates[0];
            }

            $newrule = str_replace($match_exdate[0], '', $newrule);
          }

          if (preg_match('/RDATE:[^;]+/', $newrule, $match_rdate)) {
            preg_match_all('/(\d{8}T\d{6}Z+)/', $match_rdate[0], $match_rdate_dates);

            if (!empty($match_rdate_dates[0])) {
              $rdates = $match_rdate_dates[0];
            }

            $newrule = str_replace($match_rdate[0], '', $newrule);
          }

          // Change all instances of the noncompliant rrule to the newrule so we can reuse it.
          if ($newrule != $d['rrule']) {
            foreach ($value as &$item) {
              if ($item['rrule'] == $d['rrule']) {
                $item['rrule'] = $newrule;
              }
            }

            $d['rrule'] = $newrule;
          }

          // Create a new rule to generate our instances.
          $rdata = [
            'start' => $d['value'],
            'end' => $d['value2'],
            'rule' => $newrule,
            'freq' => $m[1],
            'limit' => $limit_date_only ?? $limit,
            'parameters' => implode(';', $parameters),
            'unlimited' => $d['value2'] ? 0 : 1,
            'entity_type' => $this->configuration['entity_type'],
            'bundle' => $this->configuration['bundle'],
            'field_name' => $destination_property,
          ];

          $smartdate_rule = SmartDateRule::create($rdata);

          // Get default limit if none provided.
          if ($smartdate_rule->limit->isEmpty()) {
            $month_limit = SmartDateRule::getMonthsLimit($smartdate_rule);
            $before = strtotime('+' . (int) $month_limit . ' months');
          }
          else {
            $before = NULL;
          }

          // Generate instances.
          $gen_instances = $smartdate_rule->makeRuleInstances($before)->toArray();
          $instances = [];
          foreach ($gen_instances as $gen_instance) {
            $gen_index = $gen_instance->getIndex();
            $instances[$gen_index] = [
              'value' => $gen_instance->getStart()->getTimestamp(),
              'end_value' => $gen_instance->getEnd()->getTimestamp(),
            ];
          }

          // Set the instances and save our rule.
          $smartdate_rule->set('instances', $instances);
          $smartdate_rule->save();

          // Convert our instances to date items.
          $ri = 1;
          foreach ($instances as $instance) {
            $dates[] = [
              'value' => $instance['value'],
              'end_value' => $instance['end_value'],
              'duration' => (int) round(($instance['end_value'] - $instance['value']) / 60),
              'rrule' => $smartdate_rule->id(),
              'rrule_index' => $ri,
              'timezone' => '',
            ];

            $ri++;
          }

          // Add RDATEs as extra date items.
          if (!empty($rdates)) {
            $start_time = date('His', $rdata['start']);
            $end_time = date('His', $rdata['end']);

            foreach ($rdates as $rdate) {
              $rdate_datetime = date_create_from_format('Ymd\THis\Z', $rdate);
              $rdate_datetime->setTimezone(new \DateTimeZone('Europe/Brussels'));
              $rdate_date = $rdate_datetime->format('Ymd');

              $dates[] = [
                'value' => strtotime($rdate_date . 'T' . $start_time),
                'end_value' => strtotime($rdate_date . 'T' . $end_time),
                'duration' => 0,
                'timezone' => '',
              ];
            }
          }

          // Add EXDATEs as override rules.
          if (!empty($exdates)) {
            foreach ($exdates as $exdate) {
              $ri = NULL;
              foreach ($instances as $instance_index => $instance) {
                if (date('Y-m-d', ($instance['value'] + 7200)) == date('Y-m-d', (strtotime($exdate) + 7200))) {
                  $ri = $instance_index;
                  break;
                }
              }

              if (!empty($ri)) {
                $exdate_rule_values = [
                  'rrule' => $smartdate_rule->id(),
                  'rrule_index' => $ri,
                ];
                // Save the rule.
                $ex_rrule = SmartDateOverride::create($exdate_rule_values);
                $ex_rrule->save();
              }
            }
          }
        }
      }
      else {
        // This date doesn't repeat.
        $dates[] = [
          'value' => $d['value'],
          'end_value' => $d['value2'],
          'duration' => (int) round(($d['value2'] - $d['value']) / 60),
          'timezone' => '',
        ];

        // Reset rule data.
        $rdata = [];
      }
    }

    return $dates;
  }
}

Greets,
Kim

BenStallings’s picture

Thanks for your alternate take, @nightlife2008! Here's another patch on my previous plugin, tweaking the date format of the limit field to use Y-m-d\THis instead of Ymd\THis\Z.

choster’s picture

This might be more of a support question, but perhaps it is a use case not yet considered. I am migrating a D7 Date module field to D9 Smart Date, but it is stored as "ISO Date." MariaDB/MySQL does not support this natively, so D7 Date creates it as a varchar like "2022-12-21T19:30:00" with no timezone stored; all times are normalized to the site timezone, which is America/New_York.

When I migrate, these strings get converted to timestamps as if they were originally UTC and not America/New_York. For example, 2022-12-21T09:30:00 is saved as 1671615000 instead of the actual time, 1671633000. If it were as easy as adding five hours to all the times I'd have done it, except that the offset varies during daylight saving time.

Did I miss a step in the migration? Or is there an easy way to recalculate the timestamps accounting for daylight saving?

mandclu’s picture

@choster, definitely a tangent :)

Actually, the reality is a little different from what you describe. Drupal core date fields store values as ISO strings without timezones, as you describe, though there is no issue with storing these in MariaDB/MySQL. Many would have preferred a native database datetime storage, but not all supported database engines support that (I'm looking at you, Postgres).

Smart Date stores values as timestamps (currently INT values, soon to be BIGINT to solve the 2038 problem), and can optionally store timezones too.

mandclu’s picture

Sorry, re-reading your comment I think I misinterpreted it at first. AFAIK D7 Date DID support native datetime storage, but site builders had a choice and it sounds like for the site you're working on, someone chose ISO date. If all values were normalized to the site timezone of America/New_York then in your migration you should still be able to use the ISO date string and timezone to create a DrupalDateTime object and have that convert to a timestamp that will be correct based on DST.

BenStallings’s picture

The time zones were not getting converted right - D7 stores its dates in UTC.

BenStallings’s picture

FileSize
11.57 KB

Truncate rrules longer than 255 characters by removing their EXDATE or RDATE values as necessary.

jeffschuler’s picture

Version: 3.6.x-dev » 4.1.x-dev
FileSize
11.92 KB
1.24 KB

Using the patch in #38 the "limit" end date of a recurrence is being specified with time in the string, and therefore not showing up in the UI – where only a date is allowed.

It looks, from smart date data that's been input manually (non-migrated), like the rule DB column does include HHMMSS ('\THis') in the UNTIL string but the limit column does not.

i.e.

Should be:
rule: RRULE:FREQ=DAILY;INTERVAL=1;UNTIL=2017-01-08T235900
limit: UNTIL=2017-01-08

But actually is migrated as:
rule: RRULE:FREQ=DAILY;INTERVAL=1;UNTIL=2017-01-08T235900
limit: UNTIL=2017-01-08T235900

Here's an update of the patch in #38.

Looks like @nightlife2008 had this issue too, with a similar solution in #32.

jeffschuler’s picture

FileSize
12.27 KB
583 bytes

Missed the one that matters most. Fix and simplification.

jeffschuler’s picture

FileSize
11.92 KB

One more try. Sorry for the spam.

jeffschuler’s picture

FileSize
12.1 KB
1.13 KB

There's another issue in ParseDates with the break; after generating new repeat values. It breaks out of the loop completely, meaning that additional values don't get processed at all.

This for me resulted in a bunch of " [warning] A non-numeric value encountered ParseDates.php:341" errors.

Updated patch to fix this.