Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
It would be useful to provide a migrate process plugin that allows migrating data from the Date module on D7.
Comment | File | Size | Author |
---|---|---|---|
#42 | smart_date-3060042-42.patch | 12.1 KB | jeffschuler |
#38 | smart_date-3060042-38.patch | 11.57 KB | BenStallings |
#37 | smart-date-process-3060042-37.patch | 11.21 KB | BenStallings |
#33 | smart-date-process-3060042-33.patch | 10.25 KB | BenStallings |
#31 | smart_date-process-3060042-31.patch | 10.25 KB | BenStallings |
Issue fork smart_date-3060042
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3.5.x changes, plain diff MR !24
- 3060042-provide-migrate-support changes, plain diff MR !23
Comments
Comment #2
DamienMcKennaComment #3
matt.mendonca CreditAttribution: matt.mendonca commentedWhile working on our migration, I had to write a plugin to convert D7 date, conditional and time fields (business rules) to D8 smart date. This plugin isn't general purpose, but maybe it can help serve as an example / guide for other people.
Some notes / caveats:
- It was meant for internal purposes, so it could probably be refactored / cleaned up (right now it's good enough for our needs)
- It takes source values from 3 different fields, not just 1 to 1 from date field
- There are some short cuts taken in the interest of getting our migrations done quickly (source mapping not passed in config, but looked up at run time in the plugin)
- I had originally intended to just process the top level smart date field with the transform method, but ended up having to process the "sub fields" (start, end, duration) individually, hence why the transform method does the all work for all the sub fields
- The NISTBaseProcessPlugin class just provides the log and throwPluginException utility methods used in the validate methods
Anyway, here is what the yaml mapping looks like:
and here is the plugin code:
Comment #4
DamienMcKennaAny reason to not just import the start & end values as-is, then have a simpler function for filling in the "duration" value?
Comment #5
mandclu CreditAttribution: mandclu at Northern Commerce commented@DamienMcKenna it doesn't seem like there's a lot of logic currently in there to set the duration. What's your interest in doing that separately?
Comment #6
trackleft2 CreditAttribution: trackleft2 as a volunteer and at The University of Arizona commentedFYI when adding recurring event data from d7 into smart_date d8/9 You have to:
1. Destructure/Parse the repeat rule
2. Create a new 'smart_date_rule' from each rule
The instances field is generated on the fly by smart_date, but you can trigger like this
Once you have instances you can save via multi-value event field
Hope this helps.
Comment #7
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedIn case it helps anyone, here is the custom process plugin I wrote (in a custom module called sg_migrate):
Example use:
Comment #8
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedUpdate: the plugin code I provided above is problematic because it doesn't populate the rule instances, which effectively prevents cron from running on the site. I can see that I should have generated the list of instances programmatically using the SmartDateRule object, but I can't make out which of that object's methods, if any, allow you to set the instances of a rule to a list of values.
Comment #9
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedAfter studying the conversation at https://www.drupal.org/project/smart_date/issues/3165402 , it sounds like maybe I should not be saving the field values at all as I did above, but instead turning them into instances of the rule and letting the module save the values to the field. Is that correct?
If so, is it just necessary to invoke SmartDateRule::create() on an array that defines 'instances' as an array of start & end values, and then ->save() the resulting rule object?
Thanks for the feedback.
Comment #10
mandclu CreditAttribution: mandclu commentedUnfortunately, it isn't quite that easy. When migrating to a recurring event, you will want to create the rule, but then use the rule object to generate the instances, then add those instances as Smart Date field values, but with the rule id set as the rrid. Does that help?
Comment #11
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedThat does help, except that I'm not sure how to handle any exceptions to the rules that may have been reflected in the source field values. (That's why I had been importing the values rather than generating them.) Maybe I'll just ignore any exceptions and tell the users to check their imported data for accuracy. Thanks for the feedback.
Comment #12
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedOK, here is what I wound up doing instead of the process plugin shown above. This is a custom source plugin, and it's got some hardcoded values specific to my use case, so please take it as a starting point.
The migration YAML file then looks like:
because the values for field_dates have already been changed to the correct format by the source plugin.
Comment #16
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedOK, I think I just created a merge request at https://git.drupalcode.org/project/smart_date/-/merge_requests/24 . I couldn't make a patch because this is a new file. The file is a process plugin that can handle not just Drupal 7 Date fields from a database source, but also from a Views data export. This was not a trivial effort because Views converts the rrules into a human-readable format, and I had to convert them back to rrules again.
I was scratching my own itch here and will not be offended if someone wants to make substantial changes. For example, it might make more sense to split this into two plugins, one for parsing the Views output and one for converting Drupal 7 Date data to Smart Date data. Then people who want to use both can just chain them together. However, I have no need to separate them, so I'm not doing that at this time.
Comment #17
trackleft2 CreditAttribution: trackleft2 as a volunteer and at The University of Arizona commentedFYI, you can make a patch from your MR by adding .patch to the end of the URL
https://git.drupalcode.org/project/smart_date/-/merge_requests/24.patch
Or diff like this
https://git.drupalcode.org/project/smart_date/-/merge_requests/24.diff
I wouldn't link to these merge request patches within composer.json on a production site, though since they update every time you commit to them, but they are useful for making patch files to upload to drupal.org.
Comment #18
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedThanks for the tip, @trackleft2!
Comment #20
mandclu CreditAttribution: mandclu commented@DamienMcKenna @BenStallings Thanks very much for your work on this. Would have preferred to get a little more community input into how this works, but since the work here is really contained in its own class, if it still needs a little work we can figure that when any issues are identified.
Merged in, will include in a new release shortly.
Comment #21
danflanagan8Thanks for the plugin, @BenStallings!
It seems to have almost worked for my case, which was migrating from a D7 site using date_recur to a D9 site using smart_date. I had to make two changes to the code.
1. I had to update the annotation to include
handle_multiples = TRUE
2. I had to do a strict inequality on line 148:
After making those changes, I got data to migrate and it was mostly correct. I ended up having trouble with timezones, which are different in my source data (UTC) and the target site (LA). Timezones always confuse me though, so I'm not sure if it was a bug in the process plugin or a problem somewhere else (like me!).
Comment #22
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedThanks for the feedback, @danflanagan8. Could you please supply your changes in a patch? I'm reopening the ticket until then.
Comment #23
danflanagan8Here's a patch with the start of a test we can work against. It very basic and not at all DRY, but it's way better than nothing. Also the small fixes I proposed. Turns out the
!==
thing doesn't appear to be a problem in php8+, just in php7.Though this project is apparently not configured to run tests on d.o.
Locally, both tests fail without the handle_multiples line. The second test fails because the 2nd and 3rd fields instances that are expected are not created.
This is perhaps my writing poor assertions if those are generated on cron or something like that. I'm new to the module!
Comment #24
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedThanks again, @danflanagan8. I had trouble with the strict inequality because UNIX timestamps were being misidentified as strings, and I had to change it back to !=. Would something like this work for your data?
Comment #25
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedHere's an updated patch with my fixes.
Comment #26
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedDang it. Fixing a typo that was in #25.
Comment #27
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedAnother bug bites the dust. I was trying to unset a loop variable. :facepalm:
Comment #28
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedAnd another one gone, and another one gone. Convert all-day dates from midnight-to-midnight to midnight-to-23:59:00.
Comment #29
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedDoing math on timestamps worked great right up until somebody spanned daylight savings time. :(
Doing all the math with PHP DateTime objects instead resolves the problem.
Comment #30
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedOnly do date methods on a variable if it actually contains a date object..!
Comment #31
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedHandle some additional edge cases for Views' English output of rrules.
Remove parsing of Views output by commas, since its English output of rrules can contain commas. :facepalm:
Comment #32
nightlife2008 CreditAttribution: nightlife2008 commentedHey everyone,
Thanks for providing the Process plugin code! It helped me to get things started, but instead of recreating the source's values in the destination drupal 8/9, I took another route to generate the actual date instances of a repeated date value...
In my setup, it became clear to me that some repetition items were missing in the source data values, which in turn prohibited me from creating override rules for some of those items.
Rather than breaking my head over this situation, I stepped back and had a fresh look onto things, leading me to this conclusion:
When looping over the source field values in the process plugin, I should just take each first item of a repeating sequence of values, or the actual value if not repeating.
In case of a repeating date value, I just ignore all other values of that same rule, and use SmartDateRule's code to generate the actual repetition-instances. That seemed to be the solution to make sure everything adds up in my destination drupal site...
below my custom process plugin based on the ParseDates process plugin from the patch:
Greets,
Kim
Comment #33
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedThanks for your alternate take, @nightlife2008! Here's another patch on my previous plugin, tweaking the date format of the limit field to use Y-m-d\THis instead of Ymd\THis\Z.
Comment #34
choster CreditAttribution: choster as a volunteer commentedThis might be more of a support question, but perhaps it is a use case not yet considered. I am migrating a D7 Date module field to D9 Smart Date, but it is stored as "ISO Date." MariaDB/MySQL does not support this natively, so D7 Date creates it as a varchar like "2022-12-21T19:30:00" with no timezone stored; all times are normalized to the site timezone, which is America/New_York.
When I migrate, these strings get converted to timestamps as if they were originally UTC and not America/New_York. For example, 2022-12-21T09:30:00 is saved as 1671615000 instead of the actual time, 1671633000. If it were as easy as adding five hours to all the times I'd have done it, except that the offset varies during daylight saving time.
Did I miss a step in the migration? Or is there an easy way to recalculate the timestamps accounting for daylight saving?
Comment #35
mandclu CreditAttribution: mandclu commented@choster, definitely a tangent :)
Actually, the reality is a little different from what you describe. Drupal core date fields store values as ISO strings without timezones, as you describe, though there is no issue with storing these in MariaDB/MySQL. Many would have preferred a native database datetime storage, but not all supported database engines support that (I'm looking at you, Postgres).
Smart Date stores values as timestamps (currently INT values, soon to be BIGINT to solve the 2038 problem), and can optionally store timezones too.
Comment #36
mandclu CreditAttribution: mandclu commentedSorry, re-reading your comment I think I misinterpreted it at first. AFAIK D7 Date DID support native datetime storage, but site builders had a choice and it sounds like for the site you're working on, someone chose ISO date. If all values were normalized to the site timezone of America/New_York then in your migration you should still be able to use the ISO date string and timezone to create a DrupalDateTime object and have that convert to a timestamp that will be correct based on DST.
Comment #37
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedThe time zones were not getting converted right - D7 stores its dates in UTC.
Comment #38
BenStallings CreditAttribution: BenStallings at Interdependent Web LLC for Share Good USA commentedTruncate rrules longer than 255 characters by removing their EXDATE or RDATE values as necessary.
Comment #39
jeffschulerUsing the patch in #38 the "limit" end date of a recurrence is being specified with time in the string, and therefore not showing up in the UI – where only a date is allowed.
It looks, from smart date data that's been input manually (non-migrated), like the rule DB column does include HHMMSS (
'\THis'
) in the UNTIL string but the limit column does not.i.e.
Should be:
rule:
RRULE:FREQ=DAILY;INTERVAL=1;UNTIL=2017-01-08T235900
limit:
UNTIL=2017-01-08
But actually is migrated as:
rule:
RRULE:FREQ=DAILY;INTERVAL=1;UNTIL=2017-01-08T235900
limit:
UNTIL=2017-01-08T235900
Here's an update of the patch in #38.
Looks like @nightlife2008 had this issue too, with a similar solution in #32.
Comment #40
jeffschulerMissed the one that matters most. Fix and simplification.
Comment #41
jeffschulerOne more try. Sorry for the spam.
Comment #42
jeffschulerThere's another issue in ParseDates with the
break;
after generating new repeat values. It breaks out of the loop completely, meaning that additional values don't get processed at all.This for me resulted in a bunch of " [warning] A non-numeric value encountered ParseDates.php:341" errors.
Updated patch to fix this.