Problem/Motivation

https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html

> Values for milliseconds-since-the-epoch and seconds-since-the-epoch must be non-negative. Use a formatted date to represent dates before 1970.

So the module fails to index any entities with a date field / timestamp before 1970.

Steps to reproduce

Create a date field on a content-type, add it to the index, create a new node with 1950.01.01. in the date field.

Proposed resolution

Use `formatted_date` instead of the epoch.

Data model changes

As I know, it won't change the data model, just how we store the date in ES, it's still a date for Elasticsearch.

CommentFileSizeAuthor
#10 3189051-pre-epoch-dates-10.patch1.87 KBaron novak
#8 3189051-pre-epoch-dates-8.patch1.69 KBaron novak
#4 3189051-4.patch1.59 KBsokru
#3 3189051-epoch-to-formatted-date-2.patch547 bytesaron novak
#2 3189051-epoch-to-formatted-date-1.patch563 bytesaron novak

Comments

Aron Novak created an issue. See original summary.

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
StatusFileSize
new 3189051-epoch-to-formatted-date-1.patch563 bytes
aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
StatusFileSize
new 3189051-epoch-to-formatted-date-2.patch547 bytes

Actually the previous patch worked, but caused an Elasticsearch warning, let's see this one.

sokru’s picture

sokru commented
StatusFileSize
new 3189051-4.patch1.59 KB

If you look at Search API's Date type plugin , you see it will return a timestamp with default date field. There might be many services depending on date field as it is currently. I'd prefer adding new data type for date field.

Attached a patch with "Full date" type plugin that is able to handle pre-epoch dates.
Edit: Maybe this new plugin should be added on Search API module instead.

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented

@sokru, https://www.drupal.org/files/issues/2021-03-18/3189051-4.patch seems to be the right direction, but I receive this during indexing:

Location 	https://foobar.ddev.site:4443/batch?_format=json&id=7&op=do&op=do_nojs
Referrer 	
Message 	failed to parse field [created] of type [date] in document with id 'entity:node/7:en'. Preview of field's value: '1616075733'. failed to parse date field [1616075733] with format [strict_date_optional_time] for id: entity:node/7:en

So it seems in my case, it tried to index the data as a timestamp simply.

sokru’s picture

sokru commented

Did you change your field type to full_date and cleared the index?
Works on my machine: https://github.com/KalleVuorjoki/drupal-elasticsearch/tree/3189051-pre-e...

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented

So the difference is that in my failing case, I try to index "created", you index a date field.
Checking how to resolve this.

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
StatusFileSize
new 3189051-pre-epoch-dates-8.patch1.69 KB

I could tackle it easily in \Drupal\elasticsearch_connector\Plugin\search_api\data_type\FullDateDataType::getValue

sokru’s picture

sokru commented

I think we need to rephrase the description from "Represents points in time" to something like, "Formatted date. Use for pre-epoch dates (dates before 1st of January 1970)".

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
StatusFileSize
new 3189051-pre-epoch-dates-10.patch1.87 KB
sokru’s picture

sokru commented
Status: Needs review » Reviewed & tested by the community

Looks good to me.

  • sokru committed b4f22a17 on 8.x-7.x authored by Aron Novak 
    Issue #3189051 by Aron Novak, sokru: Handle pre-epoch dates
sokru’s picture

sokru commented
Status: Reviewed & tested by the community » Fixed

Changed the deprecated DATE_ISO8601 to DATE_ATOM. That makes dates a bit different:

DATE_ISO8601:
2023-01-06T09:14:34+0000
DATE_ATOM:
2023-01-06T09:14:34+00:00

I hope that's fine.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

baysaa’s picture

baysaa commented

Does this affect 8.0.x?

I'm getting

Drupal\Component\Plugin\Exception\PluginNotFoundException: The "full_date" plugin does not exist. Valid plugin IDs for Drupal\search_api\DataType\DataTypePluginManager are: elasticsearch_connector_date_range, elasticsearch_connector_rank_feature, object, elasticsearch_connector_search_as_you_type, decimal, date, string, text, integer, boolean in Drupal\Core\Plugin\DefaultPluginManager->doGetDefinition() (line 53 of core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php).

sokru’s picture

sokru commented

The upgrade path didn't caught this, should create a new issue for providing the upgrade path.
Using the "string" plugin provides following date

1919-01-10T10:10:10

Which I hope is sufficient.