Join forces and reduce duplicate effort?

Hi Derek,

Generally speaking, I am all in favour of working together instead of duplicating effort. I would also love to see any/all of this in core instead of in a contrib module, as I built this because I believe it's the admin experience most authors would want.

That said, there is still the question of storage. I don't believe that storing dates as strings is a good idea. It's portable and workable, but but with even a medium-sized dataset, it becomes slow. I've encountered a number of other devs who have had the same experience, and had to make their own custom workarounds.

Also, while Smart Date provides a "custom" format by default, it provides for numerous formats, and the ability to define more formats.

Smart does goes further than core in making the formatting more granular, for example to allow for more sophisticated formatting of time or date ranges.

All that being said, I have also considered if there are ways to make the pieces that Smart Date provides more granular: storage format, admin input, and formatter, and make them each an extension of the core datetime field. I think my biggest hesitation is the work that would impose on a site builder to install these disparate pieces and make them work together. With Smart Date, there is a sophisticated and performance experience available as soon as you install the module and add the field to a content type.

All that to say... Yes, let's keep this conversation going. We should all be working together to solve common problems, especially if we agree on the ways in which those problems would best be solved.

This is definitely an enlightening conversation, and I wish I had heard about datetime_extras before I started working on Smart Date.

Some further comments:

I don't disagree about the limitations of timestamps, or the benefits on native RDBMS datetime fields, but the core datetime and datetime range fields don't use either, as per #2366213: Date field doesn't use database-native date storage. They store date values as string, which can lead to very serious performance issues, especially when used on enterprise-scale site, which seems to be the use case for which Drupal is increasingly designed. If datetime_extras could provide provide support for alternate data storage (to be defined when creating the field as was possible in previous versions of Drupal), I'd be more inclined to throw my support behind this approach. Ideally this would include support for the native datetime db storage (which is a heavy lift, considering none of this is in core) or at least timestamp ranges (which can leverage classes and methods already in core, as Smart Date does).

I do see a lot of duplication between Smart Date and some of the alternates you cite. Using a separate field for duration has a degree of logic, but seems like it would be problematic, for example if there are multiple datetime range values, you would need to generate a matching number of duration fields, and make sure they know how to find their equivalents. Strictly speaking, I think you could make the duration something that isn't stored, and only generated/used as part of the admin interface (e.g. a JS-generated element).

Based on my work so far on reducing duplication in date/time output, I would suggest that you need to anticipate more potential use cases. For example, I currently use the following for testing:
Aug 14th 2019, all day (single day, all day)
Aug 14th to 15th 2019, all day (multiple days in the same month, all day)
Aug 14th to Sep 14th 2019, all day (multiple days in the same year, all day)
Dec 30th 2019 to Jan 15th 2020, all day (multiple days that span more than one year, all day)
Aug 14th 2019, 10am (a time with no duration)
Aug 14th 2019, 9 to 10am (a time range within the same am/pm span)
Aug 14th 2019, 11:30am to 12:30pm (a time range within the same day)
Aug 14th 2019, 11:30pm to Aug 15th 2019, 12:30am (a time range spanning multiple days)

For that last one, I have also seen some sources that suggest a format like:
Aug 14th/15th 2019, 11:30pm to 12:30am (when a time span flows into the next day)

Smart Date already supports all of the above, except this last potential format.

I believe that Smart Date also implements a lot of what you suggest related to the Compact Format, but tries to do so with less manually specified/configured by the site builder.

I will also say that long term I hope to allow the smart date formats (similar to what you have implemented as date_range_format) to be fully translatable, so that, for example, default settings can support the preferred date and time formats for each language. It is quite broad spectrum, as you can see at https://help.talend.com/reader/3zI67zZ9kaoTVCjNoXuEyw/YHc8JcQYJ7mWCehcQR...

If we can get aligned on some of these approaches and datetime_extras is better supported/aligned with core, I would consider moving whatever incremental elements that are of benefit here over to that module, and shifting my focus there. As it is, I will update the project page to mention datetime_extras as an alternate to consider.

I just tried to install datetime_extras on my local to try out the datetime range with duration. When I tried to change over my widget I got this fatal error: Class 'Drupal\duration_field\Plugin\Validation\Constraint\GraularityStringInterface' not found

That said, I can see already some differences. In Smart Date, I wanted to provide some optional controls over what duration intervals can be used. For example, supposing you are making a website for a conference and you know all sessions will be either 45 min or 90 min. In Smart Date, in the field definition you can set these as allowed durations, and by not adding the "custom" duration value, all events will have to use one of the provided options, and the end date/time becomes read only.

Generally speaking, I am all in favour of working together instead of duplicating effort. I would also love to see any/all of this in core instead of in a contrib module, as I built this because I believe it's the admin experience most authors would want.

The goal of datetime_extras was/is to be a playground to support real world examples w/o the restrictions of the strict core gates, with the hope of getting some of them into core. That said, the balancing act is do-it-all and do-a-lot fields/widgets/formatters are really hard to get good test coverage on.

@dww I was using 8.x-2.0-rc2 already. The error I'm getting is what's described in #3053506: Error when adding duration field and also mentioned in this comment.

As for the duration enforced values, I don't think you get my meaning. I'm not talking about granularity, I'm talking about the ability for a site builder/admin to only allow specific intervals (e.g. 45 minutes or 90 minutes) at which point no other values would be allowed (30 minutes, 60 minutes, 120 minutes, etc.).

In creating the Smart Date interface, I studied popular calendar applications like Apple Calendar and Google Calendar, to try and make something that would be similar enough to feel intuitive right away.

It's weird, those are the steps I followed, except my content type isn't named Event, it's Date Range Test. Still no luck getting datetime_extras to work, or at least with your duration widget. What version of core do you usually use it with?

As for the duration discussion, I still don't think they're equivalent. You solution will allow for the use case of durations of 45 min and 90 minutes, but would also allow for 135 min or 180 min, which may not be desired. We may have to agree to disagree on this, as we both seem to be very fond of the interfaces we've built. ;-)

I understand your meaning about not trying to make yet another "all day" solution. What is planned for datetime_extras to handle this? I feel like at a cursory glance date_all_day has most/all of the potential concerns of Smart Date (a separate field type, its own storage and formatter mechanism, etc.) but does so in service of a narrower solution (e.g. solving only the "all day" issue, but not duration, reducing duplication in output, output formatting, etc). Sadly the core issue seems to be mired in disagreements about how to interpret "all day". For Smart Date I tried to take a very pragmatic approach, so it interprets "all day" as the full span of the day, meaning no special storage is required, and it's easy to build queries/views that will return all day events the same as any other.

As for compact output (or as I prefer to call it, deduplication), I'd personally prefer to stay away from having site builders supply completely separate date strings for each use case. I worry that it will too easily lead to unexpected inconsistencies. For Smart Date I would have preferred to stay away from allowing a special time format for times that fall on the hour for the same reason, but I couldn't see a way to handle different language-specific norms otherwise (e.g. "3pm" in English, "15h" in French, etc).

In terms of the way Smart Date works today, are there particular pieces you'd like to see in datetime_extras?

@dww I will confess that I have been continuing to think about this thread. I've also thought about adding some broader thoughts to #3146014: Add an event listing to Umami (to show off core's datetime handling) but I've seen too often the damage done when an actionable thread goes off track.

I want to add some thoughts, at a higher level, about what I've observed in my time working in Drupal date/time space. Maybe this thread isn't the best place for it, but sharing here and happy to continue elsewhere if that would be helpful.

When I started working on Smart Date, the key idea I had was really a widget: a date range, with interface-level understanding of duration, so that the process of managing dates and times could be easier, and more like the calendar applications that are so pervasive.

Around the same time, I realized that it was increasingly common on my projects for client to want better output formatting for dates, especially for deduplication of the date.

And around the same time I was getting a site ready to launch, and noticed that the slowest page (by a significant margin) in a test crawl I did was for an events archive page. It was puzzling because it seemed like a pretty simple page, a list with a low number of filters and a basic sort. I documented the query that views generated in #3048072: Date Range field creates very slow queries in Views.

So ultimately I ended up building something cohesive that tries to deliver on all three, but I do sometimes wonder: could I make the widget that I built work with core datetime ranges? Could the formatter do its work on datetime ranges?

One of the things that was amazing when I started to work on Smart Date was being able to leverage and extend the existing objects and methods in Drupal core. It made my work much, much easier. I will confess that I briefly considered trying to make Smart Date work with RDBMS-native date fields the way Drupal 7's Date module did, but doing so would have required writing all of that myself. For Smart Date, I was able to essentially extend the core Datetime Range fields, and use methods already defined in core to convert the values to/from timestamps, which core also uses widely. That freed me up to focus my time on the user-facing parts I was really passionate about.

That said, it has been an ongoing struggle to get other modules to support Smart Date. Understandably, other maintainers are reluctant to start supporting other contrib modules, because it has the potential to become a slippery slope of maintaining support for a larger set of modules, with an unknown degree of longevity or supportedness. At least one maintainer created a plugin system, but there have been challenges because the system hasn't been stable, even between minor releases.

The larger point is that in building Smart Date I had the privilege of standing on the shoulders of giants, by leveraging significant functionality that was already in Drupal core. If there are parts of Smart Date that could be contributed to core I would be more than happy to do so.

One thing I've been thinking a lot about lately is the "type" of field. One of the issues in the date/time space is getting module maintainers to support new field types, but should they have to? I was able to leverage of lot of core's work in Datetime Range by converting my preferred storage method (timestamps) to Drupal's Datetime objects. If we could establish some kind of universal method by which module maintainers could expect to retrieve a field's Datetime objects, they wouldn't have to worry about the underlying storage method. Similarly, there could be a standardized way to determine if a field was single value or a range, and if the latter, the end of the range.

If we could abstract away these universal methods from the underlying storage, it would be much easier to maintain support for a core's timestamp field, datetime and datetime range fields, Smart Date, Datetime Extras, and a large number of other modules trying out their own approaches.

Those are the thoughts I wanted to share, and happy to help in whatever way would move the conversation forward.

P.S. I've also seen how much work you put into the Drupal community in general and wanted to say how grateful we are for all the work you do.

@manclu: Thanks for coming back and sharing your latest thoughts on this. I can appreciate how you started down this path, and how you were able to produce something that helps.

I heard about this module when it first launched, saw that is was using its own storage, and then dropped it. That's because I couldn't, in good conscience, recommend that my clients use a date field with its own storage, when they'd eventually have to invest significant effort migrating their data to core fields (however they evolve).

I just looked it up again to see if there was any movement in getting this functionality into core (as I saw a new release in the newsletter), and that's how I discovered this thread.

What I'd like to see come out of this discussion is the following (at least for the front-end bits):

1. Integrate useful components from here into Datetime Extras.
2. Get everything playing nicely together in that module, making it core-worthy as described by @dww.
3. Move all of that into core, either piece by piece or altogether.

The performance problem can be tackled in parallel via either #3064640: Provide alternate storage backends for native DB date fields or #2657888: Add Date function support in DBTNG; I'm not sure which one of those makes more sense.

@dww Thanks for pushing on this; it's still quite relevant IMHO.

If we can get all of this done in the not-too-distant future, hopefully we won't have too much of a divergence, which would prevent a complicated migration.

In truth, the storage is probably the least custom part of Smart Date. It extends core's Datetime Range widgets to map to core's timestamp fields. The only thing custom is that the timestamps are a range. And IIRC correctly I've seen at least two other modules that do something similar. The one I came across before I started working on Smart Date I reached out to, but never heard back. I've heard from a number of people that storing dates as strings doesn't work at scale, which is what led me down this path.

All that to say that there's an appetite for other storage methods. I will stand by my earlier statement that I think it would be awesome (and way simpler for project authors) if we could decouple "how it's stored" from "how many values does it have". Smart Date follows methods used in core for both those questions, but the specific combination is not currently supported by core. Right now "type" of field is a combination of both, and may have stifled innovation in how dates and times are handled.

I would definitely be willing help move whatever parts of Smart Date can be agreed on as worthy of consideration for core into Datetime Extras, or whatever other method would make the Drupal experience for managing dates better aligned with what content authors are used to.

Based on the earlier discussions it sounds like there were divergent opinions on what route to take on a number of fronts. If we can find some common ground would happily use this as an avenue to move forward.

@colan thanks for pointing me at those additional issues also. As usual, lots of great discussions going on in the Drupal world.

I apologize for my ignorance on some of the items I mentioned above.

@mandclu: Given that you're the one that understands the ins & outs of your own module, how would you feel about turning this issue into a roadmap? I'm thinking of lists of issues in the following sections, which we can add to the issue summary (IS) here and fill in with subtasks (other issues):

1. Core or Datetime Extras prerequisites
   • #99996: ...
   • #99997: ...
2. Components/features of Smart Date that should be moved over
   • #99998: ...
   • #99999: ...

Obviously, I'm not expecting you to do all of this work yourself, just come up with a sane plan (as you're the most qualified). Once we have a plan in place, other can jump in and help with targeting specific issues.

As far as dealing with conflicts earlier in this thread, I don't see that as a blocker. Let's simply turn each of those into "Figure out the best way to handle XXXXX"-type issues, and we can add them to the lists above.

Once we get this issue set up in a way that can be used to divide and conquer, we'll have a goal that we can all work towards.

OK I can start working on that. At a high level, here's what I would describe as the logical parts of Smart Date currently:

Storage: timestamp ranges
- extends core Datetime range and timestamp fields
- proposal: consider #3064640: Provide alternate storage backends for native DB date fields the place where the conversation about bringing alternative storage into core will take place

Default Widget: core Datetime Range + duration, allday
- currently the configuration for allowed duration values is set at the field config level, but if moving this into core as more of a widget, it may may more sense for this to be widget settings instead. Down side is that you can't do the same validation for data being created via other interfaces e.g. REST, migration, etc.
- Smart Date works with a granularity of minutes, so "all day" is considered 12am to 11:59pm. There has been a fair bit of discussion about this, but from my perspective this turns out to be a very practical way to handle events. For example, you can build an events view to show events whose end date is in the future, and all day events will show as expected
- Would this be considered a submodule of Datetime Extras?

Timezone Widget: allow an author to specify a timezone override per entry
- I know there are existing issues (e.g #2632040: [PP-1] Add ability to select a timezone for datetime field) to solve this in core, but I feel like I would have a hard time dropping support for this in the contrib space if this hadn't been solve in core. Would probably require a schema update to Datetime Extras/core Datetime fields, at a minimum

Default Formatter: deduplication and other "natural language" presentation of date ranges
- quite a bit of work has gone into adding logic to the way dates ranges are presented, to make the output closer to how people would write a similar range. Given the amount of adoption Smart Date has seen not only in North America but also in places like Germany and the Netherlands, would seem to support that this is working well across languages and culturally-specific date norms. To be able to achieve this, however, Smart Date uses a more granular date format. Instead of just requiring a single PHP date format string, it uses separate strings for day, time, time on the hour, all day label, range separator, date/time join, and whether the date or time should be shown first. I'm thinking this formatter and the code for these entities should be in a submodule, since not everyone will want/need this functionality

Duration Formatter: There was also an appetite for a format that wouldn't output the range, but instead the start date/time and duration. This was pretty simple to implement, and since it relies on the granular formats described above it could go into the formatter submodule.

Boolean field: For the Smart Date formats I wanted some simple checkboxes but couldn't find boolean fields in core, so I made a simple class to provide them. It wouldn't surprise me if there's already an issue for this, or maybe the support is there and I missed it.

Recurring Dates: Smart Date takes an opposite approach to date_recur in that the recurring instances are directly populated into the node/entity as field deltas, and the rule is stored as a separate, related entity. This means you can easily do things like build a view that contains a mixture of recurring and non-recurring dates, and no special joins are required. This probably makes sense to stay in contrib, but might need some kind of hook/plugin to maintain the current functionality.

Tokens support; Could be submitted as a patch to Token

Fullcalendar View Support: a lot of work has gone into this, to allow for not only display but drag and drop editing of dates and even instances of recurring dates. This would probably make sense staying in contrib

If all of the above sounds reasonable, I can start on a more formal roadmap. I will confess that I still have some doubts all the current functionality will work as seamlessly as a loosely (at best) series of components, but I'm ready to give it my best effort.

@mandclu: #18 certainly looks like a good outline. Please proceed!

To answer your one question:

Would this be considered a submodule of Datetime Extras?

Yes, I don't see why not. Create an issue there, and we can link to it from here (like all of the others). Do the same for anything else that doesn't have an issue yet (either in Core or DE, whichever makes more sense).

Edit: Sorry, I responded too quickly. I think that decision can be discussed in the issue, once it exists. We can point folks to it to solicit feeback.

I think there are similarities, but also some important differences in terms of how Smart Date's default widget works. To be honest I was thinking I would make a new widget, e.g. "Meeting widget", since that's a typical use case that the widget was really optimized to handle, based on best-of-breed solutions from companies like Google, Apple, etc.

> I heard about this module when it first launched, saw that is was using its own storage, and then dropped it.

Me too.

The work you've done here @mandclu is so impressive, but it feels too risky to cut myself from the Drupal continent and live on the smart date island.

One thing I've been thinking a lot about lately is the "type" of field.
If we could establish some kind of universal method by which module maintainers could expect to retrieve a field's Datetime objects, they wouldn't have to worry about the underlying storage method.

I suggest this is where to concentrate. If you can make smart date work with core field types, then you grow your community massively. This brings in a lot of energy to work on merging smart date features into non smart date work on similar features.

What is needed to make smart date formatters and widgets work with core fields?

@jonathanshaw Thanks for the feedback.

To be honest, the next step is to take the rough outline proposed in #18 and create a plan on Datetime Extras, along with child issues for the specific deliverables.

I suspect I'll probably start with the widget, then try a formatter, and then assess next steps from there.

While we're having some high level discussions, one thing that occurred to me is that I'll have to figure out a new naming convention. Until now I could name things like "Smart Date formats" and not have to worry about conflicts. The formats Smart Date uses to more effectively format time and date strings are a lot more granular, so they'll still need to be separate from the core "Date and time formats". Maybe something like "Date and time format builders"?

One other note related to this thread. There's a good chance I'll be working on issues related to this thread during DrupalCon contribution day. If you'd be interested in helping out, join the Datetime group at https://contrib2020.getopensocial.net/group/datetime/about . I have also set up a BoF about Datetime in Drupal, for anyone interested in sharing their perspective but not able to contribute, at https://events.drupal.org/global2020/bofs/dates-and-times-drupal

I decided that a good step towards the work above would be to first add support for core Datetime and Datetime Range fields within Smart Date. I've created #3158515: Support core datefield range fields with the Smart Date Widget (any feedback is welcome) and will open a similar issue for the standard formatter.

@dww I was also finally able to get datetime extras working and can see what you mean that your duration widget has definite similarities to Smart Date's widget. I suppose the main difference between the two is that the duration widget asks an editor to choose between working with a duration vs working with an end date, where Smart Date's widget uses JS to use both, so changes to one dynamically update the other.

Been reading this thread with interest, in particular comment #13:

When I started working on Smart Date, the key idea I had was really a widget: a date range, with interface-level understanding of duration, so that the process of managing dates and times could be easier, and more like the calendar applications that are so pervasive.

and

If we could abstract away these universal methods from the underlying storage, it would be much easier to maintain support for a core's timestamp field, datetime and datetime range fields, Smart Date, Datetime Extras, and a large number of other modules trying out their own approaches.

What if we were to produce a simple form element for date ranges, unrelated to fields?

You still need a field widget, but it becomes a much smaller wrapper around the form element, taking care of translating the values to and from the field item.

In much the same way that we have a TextfieldWidget that doesn't do much more than output a TextField form element.

produce a simple form element for date ranges

Smells right to me.

@erik.erskine interested to hear more. How would what you're proposing differ from the current Datetime Range?

I've now made #3158616: Support core datefield range fields with the Smart Date Formatters to make Smart Date's formatters with with core Datetime Range fields as well. Again, based on what I've read there are probably some distinct similarities to the work being done in #2834016: Add 'Compact' datetime range formatter. I haven't had a chance to test any of the patches in there, or the daterange_compact module.

Unless I get some feedback about a serious issue for this latest patch or the one in #3158515: Support core datefield range fields with the Smart Date Widget I'll probably roll a 3.0.0-alpha1 release soon of Smart Date, allowing the widget and formatters to work with core datetime range fields.

Could we get a status update on this now that v4 is out? I'm wondering what's changed with respect to this issue.

Thanks for surfacing this issue again.

I am definitely still open to collaboration, and in fact my work on the Date Augmenter API was inspired by the discussion in this thread, in the spirit of developing date formatting capabilities that are agnostic of the formatter being used.

Almost three years later, I think it's worth re-examining if Datetime Extras is the best place to collaborate. There hasn't been a release for that module since this thread went quiet, or even an issue resolved.

Would it make more sense to open one or more core issues about upgraded datetime elements, e.g. widgets and formatters? There might still be time to get these into core as experimental in D10, with an eye to getting them stable for D11.

@mpdonadio was the person to ask back then, but not sure he's still working on this stuff. He's on this thread though so maybe he'll respond. (If not, there's always his contact form.)