Problem/Motivation

In #3551591-14: Filter does not work with date range fields we discovered that the "= Is equal to" and "!= Is not equal to" operators were not working for dates (i.e.: did not match when they should match), because Elasticsearch needs the UNIX timestamps to be encoded as strings in the query.

At the time, @mparker17 did not want to blindly cast $condition->getValue() to a string in \Drupal\elasticsearch_connector\SearchAPI\Query\FilterBuilder::buildFilterTerm() because he worried that solution might break other field types (Decimal, Integer, Latitude/Longitude, etc.). He wanted to be more-selective about when $condition->getValue() was casted to a string.

Steps to reproduce

  1. Follow the docs to Setup for working on an issue. Log in as an administrator.
  2. Go to /admin/modules, and enable Core's Datetime Range (datetime_range module) and its dependencies.
  3. Go to /admin/structure/types/add, and add an Event content type (machine name event).
  4. Go to /admin/structure/types/manage/event/fields/add-field, click Date and time, enter Label = Event date (machine name field_event_date), Choose a field type = Date range, click the Continue button, then click the Save button to accept the defaults on the second page.
  5. Go to /node/add/event, set Title = Workshops, Start date = 2026-04-07 09:00:00, End date = 2026-04-08 17:00:00, then click the Save button.
  6. Go to /node/add/event, set Title = Sessions, Start date = 2026-04-09 09:30:00, End date = 2026-04-09 16:30:00, then click the Save button.
  7. Go to /node/add/event, set Title = Sprints, Start date = 2026-04-10 09:45:00, End date = 2026-04-11 16:15:00, then click the Save button.
  8. Go to /admin/config/search/search-api/add-index, set Index name = Events, Datasources = Content, Server = elasticsearch_server, then click the Save button.
  9. Go to /admin/config/search/search-api/index/events/fields/add/nojs; and click the Add button in the Title (title) row and the Event date (field_event_date) row. Click the Expand button in the Event date (field_event_date) row, then click the Add button in the End date value (field_event_date:end_value) field. Then click the Done button. On the Manage fields for search index Events, in both the field_event_date row and field_event_date:end_value rows, set Type = Date. Click the Save changes button.
  10. Go to /admin/config/search/search-api/index/events. Under Start indexing now, click the Index now button. Wait for the batch job to complete. You should see the Status message Successfully indexed 3 items.
  11. Go to /admin/structure/views/add. Set View name = Event search (machine name event_search). Under View settings, Show Index Events sorted by: Unsorted. Under Page settings, check Create a page, and make sure that Path is set to event-search. Click the Save and edit button.
  12. You are taken to the view/edit page at /admin/structure/views/view/event_search. Under Fields, remove all existing fields, then click the Add button, and check Title (indexed field) in the Content data-source category. Click the Add and configure fields button, then click the Apply button to accept the defaults on the second page. Under Filter criteria, click the Add button, and check Event date in the Content datasource category. Click the Add and configure filter criteria button, then check Expose this filter to visitors, to allow them to change it. Also check Expose operator. Click the Apply button. Don't forget to click Save on the view itself.
  13. Go to /event-search. You see the Event search view you created. You see 3 results: Workshops, Sessions, and Sprints.
  14. Set Event Date = Is equal to 2026-04-09 09:30:00, then click the Apply button.
    Expected result: You see 1 result, Sessions.
    Actual result: You see 0 results
  15. Set Event Date = Is not equal to 2026-04-09 09:30:00, then click the Apply button.
    Expected result: You see 2 results, Workshops, and Sprints.
    Actual result: You see 0 results

Proposed resolution

Determine if casting $condition->getValue() to a string will break Decimal, Integer, Latitude/Longitude, etc. results.

Determine how easy it would be to determine the type of $condition->getValue() and cast it when it is a UNIX timestamp.

Remaining tasks

  1. 9.0.x merge request - done by @mparker17 in #4
  2. 8.0.x merge request - done by @mparker17 in #5
  3. Maintainer review and testing - can skip because the code and tests were written by @mparker17, a maintainer
  4. Community review and testing, move to RTBC if passes - skipped by @mparker17 in #7
  5. Merge to 9.0.x - merged by @mparker17 in #8
  6. Merge to 8.0.x - merged by @mparker17 in #10
  7. Release 9.0.x - released in 9.0.0-alpha3 by @mparker17
  8. Release 8.0.x - released in 8.0.0-alpha7 by @mparker17

User interface changes

None.

Introduced terminology

None.

API changes

None.

Data model changes

None.

Release notes snippet

Fixed an issue where the "Is equal", and "Is not equal" filters did not work for Date fields.

Issue fork elasticsearch_connector-3584265

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

mparker17 created an issue. See original summary.

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes

Determine if casting $condition->getValue() to a string will break Decimal, Integer, Latitude/Longitude, etc. results.

Manual testing both string and non-string query values for integer and float fields on the data that gets inserted by \Drupal\elasticsearch_connector_test\Drush\Commands\ElasticsearchConnectorTestCommands::addSearchApiTestContent() did not change the search results, which is promising.

Looking at the JSON specification and the Elasticsearch 9 documentation...

  1. JSON only supports as double-quoted strings, numbers, true, false, null, objects, and arrays as values. (i.e.: so Latitude/Longitude should already be encoded as one of those types)
  2. The JSON serializer takes care of quoting strings.
  3. Elastic allows number values to be passed as strings.
  4. Elastic allows boolean values (true, false) to be passed as strings.
  5. Elasticsearch cannot search for null values.
  6. This leaves objects and arrays to figure out.
mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented

This leaves objects and arrays to figure out.

Re-reading the Elasticsearch documentation for term queries, the value in a term query must be a string, i.e.: it cannot be an object or array.

Given that \Drupal\elasticsearch_connector\SearchAPI\Query\FilterBuilder::buildFilterTerm builds term queries for Equals and not-Equals...

$filter = match ($condition->getOperator()) {
    '=' => [
        'term' => [$condition->getField() => $condition->getValue()],
    ],
    // ...
    '<>', '!=' => [
        'bool' => ['must_not' => ['term' => [$condition->getField() => $condition->getValue()]]],
    ],
    // ...
};

... then I'm satisfied we can always cast $condition->getValue() to a string in this function.

mparker17 opened merge request !200

mparker17 opened merge request !201

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes
Status: Active » Needs review

Created merge request !200 for 9.0.x

Created merge request !201 for 8.0.x

Leaving "needs backport to 8.0.x" tag to remind me to merge both MRs

Moving to Needs review: @sweetchuck, @dewalt, @fathershawn, if you have time for a review, I would greatly appreciate it! If it works, please change the issue status to "Reviewed & tested by the community"!

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

I haven't heard back from anyone in 7 days, but I want to make a release with this issue included, so I'm going to merge this change. Thanks everyone.

  • mparker17 committed 0172e85c on 9.0.x 
    fix: #3584265 Fix "= Is equal to" and "!= Is not equal to" operators on...
mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Version: 9.0.x-dev » 8.0.x-dev
Issue summary: View changes
Status: Reviewed & tested by the community » Patch (to be ported)
Issue tags: -needs backport to 8.0.x

Merged to 9.0.x; merging to 8.0.x next.

  • mparker17 committed 4602c1a3 on 8.0.x 
    fix: #3584265 Fix "= Is equal to" and "!= Is not equal to" operators on...
mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes
Status: Patch (to be ported) » Fixed

Merged 8.0.x now too. I will update this issue when I make a release.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

mparker17’s picture

mparker17
he/him
Primary language English
Location UTC-5
commented
Issue summary: View changes

The changes in this issue were released in elasticsearch_connector-9.0.0-alpha3, and elasticsearch_connector-8.0.0-alpha7

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.