Problem/Motivation

When I have a field with field type date_range and I add it to the index with SearchAPI type "Date range" then the content indexing fails with the following error:

[1:109] failed to parse field [node_field_date_range_single_range] of type [date_range] in document with id 'entity:node/2:en'. Preview of field's value: '1760042673'. [1:109] error parsing field [node_field_date_range_single_range], expected an object but got null for index: entity:node/2:en

When I add the start (value) and end (end_value) parts of the date range field one by one as separated index fields then the content indexing works. There is a valid document in the Elasticsearch index. Something like this:

{
  "node_field_date_range_single_end": [
    1760046350
  ],
  "node_field_date_range_single_start": [
    1760042673
  ],
  "search_api_id": [
    "entity:node/2:en"
  ],
  "search_api_datasource": [
    "entity:node"
  ],
  "search_api_language": [
    "en"
  ]
}

But the following views filter operators do not work:

<  Is less than
<= Is less than or equal to
=  Is equal to
!= Is not equal to
>= Is greater than or equal to
>  Is greater than

The use case. I have an "Event" content type with a date_range field, but I am not able to list all the "Upcoming events" or the "Past events".

Steps to reproduce

  1. Follow the docs to Setup for working on an issue. Log in as an administrator.
  2. Go to /admin/modules, and enable Core's Datetime Range (datetime_range module) and its dependencies.
  3. Go to /admin/structure/types/add, and add an Event content type (machine name event).
  4. Go to /admin/structure/types/manage/event/fields/add-field, click Date and time, enter Label = Event date (machine name field_event_date), Choose a field type = Date range, click the Continue button, then click the Save button to accept the defaults on the second page.
  5. Go to /node/add/event, set Title = Workshops, Start date = 2026-04-07 09:00:00, End date = 2026-04-08 17:00:00, then click the Save button.
  6. Go to /node/add/event, set Title = Sessions, Start date = 2026-04-09 09:30:00, End date = 2026-04-09 16:30:00, then click the Save button.
  7. Go to /node/add/event, set Title = Sprints, Start date = 2026-04-10 09:45:00, End date = 2026-04-11 16:15:00, then click the Save button.
  8. Go to /admin/config/search/search-api/add-index, set Index name = Events, Datasources = Content, Server = elasticsearch_server, then click the Save button.
  9. Go to /admin/config/search/search-api/index/events/fields/add/nojs; and click the Add button in the Title (title) row and the Event date (field_event_date) row. Click the Expand button in the Event date (field_event_date) row, then click the Add button in the End date value (field_event_date:end_value) field. Then click the Done button. On the Manage fields for search index Events, in both the field_event_date row and field_event_date:end_value rows, set Type = Date. Click the Save changes button.
  10. Go to /admin/config/search/search-api/index/events. Under Start indexing now, click the Index now button. Wait for the batch job to complete. You should see the Status message Successfully indexed 3 items.
  11. Go to /admin/structure/views/add. Set View name = Event search (machine name event_search). Under View settings, Show Index Events sorted by: Unsorted. Under Page settings, check Create a page, and make sure that Path is set to event-search. Click the Save and edit button.
  12. You are taken to the view/edit page at /admin/structure/views/view/event_search. Under Fields, remove all existing fields, then click the Add button, and check Title (indexed field) in the Content data-source category. Click the Add and configure fields button, then click the Apply button to accept the defaults on the second page. Under Filter criteria, click the Add button, and check Event date in the Content datasource category. Click the Add and configure filter criteria button, then check Expose this filter to visitors, to allow them to change it. Also check Expose operator. Click the Apply button. Don't forget to click Save on the view itself.
  13. Go to /event-search. You see the Event search view you created. You see 3 results: Workshops, Sessions, and Sprints.
  14. Set Event Date = Is less than 2026-04-09 09:30:00, then click the Apply button.
    Expected result (i.e.: with patch applied): You see 1 result: Workshops.
    Actual result (i.e.: before patch applied): You see 0 results.
  15. Set Event Date = Is less than or equal to 2026-04-09 09:30:00, then click the Apply button.
    Expected result (i.e.: with patch applied): You see 2 results: Workshops, and Sessions.
    Actual result (i.e.: before patch applied): You see 0 results.
  16. Note: Ignore the Is equal to option for now: it will be addressed in #3584265: Fix "= Is equal to" and "!= Is not equal to" operators on date fields. See #7 and #14 for details.
  17. Set Event Date = Is not equal to 2026-04-09 09:30:00, then click the Apply button.
    Expected result (i.e.: with patch applied): You see 0 results and no Error message.
    Actual result (i.e.: before patch applied): You see 0 results, and the Error message Undefined operator "!=" for field "field_event_date" in filter condition..
    Note: Ignore the unexpected result (0 results) will be addressed in #3584265: Fix "= Is equal to" and "!= Is not equal to" operators on date fields. See #7 and #14 for details.
  18. Set Event Date = Is greater than or equal to 2026-04-09 09:30:00, then click the Apply button.
    Expected result (i.e.: with patch applied): You see 2 results: Sessions, and Sprints.
    Actual result (i.e.: before patch applied): You see 3 results: Workshops, Sessions, and Sprints.
  19. Set Event Date = Is greater than 2026-04-09 09:30:00, then click the Apply button.
    Expected result (i.e.: with patch applied): You see 1 result: Sprints.
    Actual result (i.e.: before patch applied): You see 3 results: Workshops, Sessions, and Sprints.
  20. Set Event Date = Is between, Min = 2026-04-09 09:30:00, Max = 2026-04-10 10:00:00, then click the Apply button.
    Expected result (i.e.: with patch applied): You see 2 results: Sessions, Sprints.
    Actual result (i.e.: before patch applied): You see 3 results: Workshops, Sessions, and Sprints.
  21. Set Event Date = Is not between, Min = 2026-04-09 09:30:00, Max = 2026-04-10 10:00:00, then click the Apply button.
    Expected result (i.e.: with patch applied): You see 1 result: Workshops.
    Actual result (i.e.: before patch applied): You see 3 results: Workshops, Sessions, and Sprints.
  22. Note: Ignore the Negated regular expression option for now, it is out of scope: see #7 and #3577065: Drop support for not_regular_expression operator for search_api_date views filter.
  23. Set Event Date = Is empty (NULL), then click the Apply button. You see 0 results.
    Note that a warning is logged, this will be addressed in #3584210: Irrelevant warnings when filter value is null.
  24. Set Event Date = Is not empty (NOT NULL), then click the Apply button. You see 3 results: Workshops, Sessions, and Sprints.
    Note that a warning is logged, this will be addressed in #3584210: Irrelevant warnings when filter value is null.

Proposed resolution

Fix how we detect UNIX timestamps (Elasticsearch calls this epoch_second format) by checking if the value itself is an integer instead of assuming it must be an array containing integers.

Restore the tests mistakenly deleted in #3535733: Range Searches; make sure that we test BOTH ISO-8601 strings and UNIX integer timestamps.

Handle both variants of the "not equal" operator (Views uses != internally but Search API uses <> internally)

Remaining tasks

  1. Write a patch - done by @sweetchuck in #2
  2. Add tests - done by @mparker17 in #7
  3. Community review and testing, move to RTBC if passes - done by @fathershawn in #18
  4. Maintainer review and testing - @mparker17 is happy with it as of #16
  5. Merge to 9.0.x - done by @mparker17 in #20
  6. Merge to 8.0.x - done by @mparker17 in #22
  7. Release 9.0.x - released in 9.0.0-alpha3 by @mparker17
  8. Release 8.0.x - released in 8.0.0-alpha7 by @mparker17

User interface changes

None.

Introduced terminology

None.

API changes

None.

Data model changes

None.

Release notes snippet

Fixed an issue where the "Is less than", "Is less than or equal to", "Is greater than or equal to", "Is greater than", "Is between", and "Is not between" filters did not work for Date fields.

Note that Date field issues with "Is equal to" and "Is not equal to" will be addressed in a future issue.

Note that the warning mistakenly output with "Is empty (NULL)", and "Is not empty (NOT NULL)" will be addressed in a future issue.

Note that the "Negated regular expression" option for date fields is an issue with Search API, see #3577065: Drop support for not_regular_expression operator for search_api_date views filter.

Note that mapping Drupal core's Datetime Range fields to Elasticsearch Date Range date_range fields is blocked by Search API, see #3583991: Support core's date_range field type's end_value: as a work-around, index the start and end date as separate date fields.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

sweetchuck created an issue. See original summary.

dewalt’s picture

Status: Active » Needs review
StatusFileSize
new988 bytes

Attaching the MR diff as a patch to safely use with composer

dewalt’s picture

StatusFileSize
new25.91 KB

I confirm that the patch works for my case:

"Upcoming only" exposed views filter stopped working. With this patch it works again.

Filter configuration

dewalt’s picture

@sweetchuck could you please try to include previously deleted test in the MR, would it pass? https://git.drupalcode.org/project/elasticsearch_connector/-/commit/a4fd...

mparker17’s picture

Note #3535733-18: Range Searches... TL;DR if I recall, we deleted the function that @dewalt is referencing because it was an override for an upstream test that was passing, but omitted the range tests we re-added in that issue.

That being said, we could override the function again; but in the override, call the parent method (i.e.: so the upstream tests are run as-is), and then add some additional tests for timestamps after the upstream tests.

mparker17’s picture

I apologize for the delay in replying.

I've put back the tests that were deleted in #3535733: Range Searches and confirmed that the change to src/SearchAPI/Query/FilterBuilder.php in the merge request fixes them.

I noticed some of the tests would break if dates were being interpreted at different instants, (i.e.: if PHP's date.timezone setting was not UTC), so I made the test data more specific (this may also be related to an upstream issue in search_api, #3137821: Date views filters don't always function correctly).

I also updated the documentation with the tests to make them easier to understand why we were testing those values and why we were expecting the results we expected.

I noticed that a test for the ">=" range operator was missing, so I added one.

I tried testing the "= Is equal to" operator, but it didn't seem to do anything; I'm not sure if that is because I'm specifying the wrong instant, or something else.

I tried testing the "!= Is not equal to" operator, but I got the error "Undefined operator "!=" for field "created" in filter condition.". Debugging, it seems like Views uses != internally but Search API uses <> internally, so I've had to update our logic to handle both variants. See also \Drupal\search_api\Plugin\views\query\SearchApiQuery::sanitizeOperator(). Once I had fixed that, then it seems like I'm getting a similar issue to the "= Is equal to" operator, i.e.: it doesn't seem to do anything.

Also, while debugging this, I noticed a potential issue upstream in Search API, so I added #3577065: Drop support for not_regular_expression operator for search_api_date views filter to the search_api queue. I also updated the test module to set the "created" dates we use while importing data.

It seems like I'm being pulled away to work on other things right now, so I'm going to file a separate issue to solve the issues with the "!= Is not equal to" operator and the "= Is equal to" operator.

Let's see if the tests pass. If they do, then I would greatly appreciate it if @sweetchuck and/or @dewalt could test the latest changes and report back here if the latest changes fix the issues they were seeing.

mparker17’s picture

The tests do pass — @sweetchuck and/or @dewalt could you test the latest changes and report back here if the latest changes fix the issues you were experiencing?

mparker17’s picture

I've rebased the original merge request onto the latest 8.0.x and created a 9.0.x branch, and both seem to pass tests.

I'll update the issue summary shortly, and I've added a backport tag to remind myself to merge to both branches when it's ready.

I would really appreciate it if someone could try out this change and report back if it fixes their issue and/or if it causes any regressions.

Thank you very much in advance!

mparker17’s picture

Issue summary: View changes
Issue tags:

Updated the issue summary, part 1.

mparker17’s picture

While writing the steps to reproduce for this, I noticed an issue with Search API, and created #3583991: Support core's date_range field type's end_value.

Essentially the "Date range" option that appears in the select lists in the "Type" column when managing fields for an index (i.e.: at /admin/config/search/search-api/index/YOUR_INDEX/fields) is a Search API Data Type plugin that defined in the Elasticsearch Connector module.

But until #3583991: Support core's date_range field type's end_value is resolved (or we port the search_api_opensearch custom Date Range processor plugin), we cannot really fix the error message initially reported in the issue summary ([1:109] failed to parse field [node_field_date_range_single_range] of type [date_range] in document with id 'entity:node/2:en'. Preview of field's value: '1760042673'. [1:109] error parsing field [node_field_date_range_single_range], expected an object but got null for index: entity:node/2:en) because the end date isn't available at the right time, so we can't build the JSON object that Elasticsearch expects for a date_range field.

(EDIT: I created #3584267: Fix elasticsearch_connector_date_range Search API Data Type plugin so it indexes without errors as a follow-up for when upstream #3583991 is resolved).

So the work-around would be to do what @sweetchuck did, i.e.: add the start (value) and end (end_value) parts of the date range field one by one as separated index fields.

I will write the test case in the issue summary to include @sweetchuck's work-around.

sweetchuck’s picture

Thank you @mparker17. I hope that I will have time on this weekend to check the merge request.

mparker17’s picture

From #7...

I tried testing the "= Is equal to" operator, but it didn't seem to do anything; I'm not sure if that is because I'm specifying the wrong instant, or something else.

I tried testing the "!= Is not equal to" operator, [...] it seems like I'm getting a similar issue to the "= Is equal to" operator, i.e.: it doesn't seem to do anything.

... I figured out why these operators don't seem to be working...

Drupal converts the dates we enter in the Views filter (e.g.: the string "2026-04-09 09:30:00") to UNIX timestamps (e.g.: the integer 1775741400 in my timezone), and \Drupal\elasticsearch_connector\SearchAPI\Query\FilterBuilder::buildFilterTerm() converts the filter to Elasticsearch Query DSL (JSON format) as...

  1. Equals: {"query": {"bool": {"filter": {"term": {"field_event_date": 1775741400}}}}}
  2. Not equals: {"query":{"bool":{"filter":{"bool":{"must_not":{"term":{"field_event_date":1775741400}}}}}}}

... i.e.: it sends the UNIX timestamp in the JSON as an integer.

But through manual testing, I found that if I send the UNIX timestamp as a string (i.e.: "1775741400") then the queries work as-expected.

I suppose a quick way to fix this would be to modify FilterBuilder::buildFilterTerm() so it always casts $condition->getValue() to a string... but I worry that might break other field types (Decimal, Integer, Latitude/Longitude, etc.), so I'd like to be more selective about when I cast it to a string.

As I said in #7, I (still!) need to file a separate issue to fix Equals and Not equals, but now I have a better idea of what I need to do.

mparker17’s picture

Issue summary: View changes

Added steps to reproduce.

mparker17’s picture

Issue summary: View changes
Issue tags: -Needs issue summary update

Added a release notes snippet

fathershawn’s picture

Reviewed the code - this looks like a good change

mparker17’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

I haven't heard back from @sweetchuck but I've re-tested this and so has @fathershawn so I'm going to merge this.

  • mparker17 committed 227808a1 on 9.0.x
    fix: #3551591 Filter does not work with date range fields
    
    By:...
mparker17’s picture

Issue summary: View changes
Status: Reviewed & tested by the community » Patch (to be ported)
Issue tags: -needs backport to 8.0.x

Merging to 8.0.x next.

mparker17’s picture

Issue summary: View changes
Status: Patch (to be ported) » Fixed

Merged to 8.0.x.

I will update this issue when this fix is released.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

mparker17’s picture

Issue summary: View changes

The changes in this issue were released in elasticsearch_connector-9.0.0-alpha3, and elasticsearch_connector-8.0.0-alpha7

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.