Setup

  • Solr version: 7.7.3
  • Drupal Core version: 9.4.8
  • Search API version: 8.x-1.26
  • Search API Solr version: 4.2.9
  • Configured Solr Connector: Standard / Acquia Search

Issue

Since upgrading from Search API Solr 4.2.8 to 4.2.9 we have an error on search related to boost configuration.

The error from Solr is this:

can not use FieldCache on multivalued field: dm_field_sort_date

We use Acquia Solr but I am also able to replicate this locally on a Lando setup.

In the search index processors, we have "Boost more recent dates" enabled, and the error goes away when I disable this.

I've regenerated the solr configuration after updating to 4.2.9.

So it seems that something has changed between 4.2.8 and 4.2.9 that is breaking the boost more recent dates.

Any suggestions for how to debug this?

I'm thinking it might be some solr config that needs tweaking, so looking there initially, but any thoughts from people who know more would be much appreciated.

Thank you!

CommentFileSizeAuthor
#17 3326515-17.patch1.28 KBgertlor
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Finn Lewis created an issue. See original summary.

finn lewis’s picture

In the 4.2.9. there's a new method getBoostableSolrField

https://git.drupalcode.org/project/search_api_solr/-/blob/4.2.9/src/Util...

Called from here:

https://git.drupalcode.org/project/search_api_solr/-/blob/4.2.9/src/Plug...

This does not do the same 'hackish' thing that the getSortableSolrField method does, which renames the field type from (in my example) from a dm_ prefix to a ds_prefix to pretend the field is no multivalued.

The comment reads as follows:

// For other multi-valued fields (which aren't sortable by nature) we
// use the same hackish workaround like the DB backend: just copy the
// first value in a single value field for sorting.

The merge request above adds this same approach to the getBoostableSolrField method which fixes the error for me.

Not sure if this is the ideal approach, but it works for me so far!

Happy to rework this if there's a better way.

Many thanks,

Finn

mkalkbrenner’s picture

Before we continue here, are you sure that you are using the scheme this module provides? Or could it be that you're using Solr's default managed schéma?

cbrody’s picture

The schema used is the one provided by this module. The issue is also present when using Acquia-provided config (Latest) Drupal 8 / 9 - Search API Solr 4.2.0 - Solr 7 - [drupal-4.2.0-solr-7.x-0] - v1.0.

cbrody’s picture

This resolves the issue we were having with the boost processor – can we get the MR approved?

mkalkbrenner’s picture

The minimum required schema version currently is 4.2.8, not 4.2.0.

mkalkbrenner’s picture

Status: Active » Needs work

Regardless if the patch avoids that error or not, you have an issue in your entity model.
It makes no sense to boost on most recent dates if you randomly pick one out of multiple.

For example, if you store 1980 and 2023 in these fields, the entity will be shown at the end of the results if 1980 is select or on top of the list when 2023 is selected.

You need to implement that selection by yourself.
You can define an additional sine valued filed on the entity and store the most recent date in it on entity save.

Or you can. Add a dummy field to the index and do that calculation in a Search API Solr pre index event.

Or you can implement a Search API processor to do it.

But I know that we have that stupid hack for sort fields to compatible to the database backend, just to avoid errors in views regardless that the result might be wrong.
But the database backend can't boost on dates. So it is a question if we really should repeat the same erroneous approach here or if the exception is correct to inform you about issues in your data model.

At least the patch should be extended to log an error if someone tries to boost on a multi valued field.

finn lewis’s picture

Thanks for your feedback @mkalkbrenner.

As to the data model, I will let @cbrody comment on that as he is more familiar with the data model and how and why it is how it is.

Even so, the data model did not change but changes in the module between 4.2.8 and 4.2.9 did break things, so I do think it would be good to get the patch committed if possible to maintain the behaviour that we were relying on, even if less than ideal.

I will look at adding a log message with something appropriate.

cbrody’s picture

Thanks for the input @mkalkbrenner. In our case the date boost works by using search_api's aggregated field mechanism to select the last encountered value of either created or field_date_published (a field that is present only on certain specific entity types, so if present the aggregate field takes its value). The resulting dm_field_sort_date is therefore a single-valued field, and the data model is sound for our purposes. The underlying issue seems to be that fields are given a dm_ prefix, regardless of whether they are actually multi-valued or not. A solution might be for the aggregated field tool to name and type fields according to their cardinality.

mkalkbrenner’s picture

Thanks @cbrody for that explanation. So your data model seems to be fine.

But the bug fix should be something else. We have a detection for that cardinality. Maybe it doesn't work correctly for that use-case and needs to be fixed.

But could you please verify the 'Fallback to multiValued field types' setting in the advanced options of the Search API Server Backend config?
Maybe it is turned on in your installation.

cbrody’s picture

We've tested with 'Fallback to multiValued field types' enabled and disabled and the error still appears with the unpatched module.

mkalkbrenner’s picture

Thanks for checking this.

So the error must be here in SearchApiSolrBackend::formatSolrFieldNames():

if ($field->getDataDefinition()->isList() || $this->isHierarchicalField($field)) {
  $pref .= 'm';
}
else {
   try {
     // Returns the correct list of field definitions including
     // processor-added properties.
     $index_properties = $index->getPropertyDefinitions($field->getDatasourceId());
     $pref .= $this->getPropertyPathCardinality($field->getPropertyPath(), $index_properties) != 1 ? 'm' : 's';
  }

Either that aggregated field is erroneously declared as list or as hierarchical field or getPropertyPathCardinality() returns a wrong result.
@Finn Lewis could you debug that?

finn lewis’s picture

It looks like aggregated fields are always declared as a list:

https://git.drupalcode.org/project/search_api/-/blob/8.x-1.28/src/Plugin...

  public function getPropertyDefinitions(DatasourceInterface $datasource = NULL) {
    $properties = [];

    if (!$datasource) {
      $definition = [
        'label' => $this->t('Aggregated field'),
        'description' => $this->t('An aggregation of multiple other fields.'),
        'type' => 'string',
        'processor_id' => $this->getPluginId(),
        // Most aggregation types are single-valued, but "Union" isn't, and we
        // can't know which will be picked, so err on the side of caution here.
        'is_list' => TRUE,
      ];
      $properties['aggregated_field'] = new AggregatedFieldProperty($definition);
    }

    return $properties;
  }

Can we check if the aggregated field is single or multivalued rather than just relying on the is_list property?

What happens if we just remove the check for $field->getDataDefinition()->isList() to fall back to the getPropertyPathCardinality method?


              else {
                if ($this->isHierarchicalField($field)) {
                  $pref .= 'm';
                }
                else {
                  try {
                    // Returns the correct list of field definitions including
                    // processor-added properties.
                    $index_properties = $index->getPropertyDefinitions($field->getDatasourceId());
                    $pref .= $this->getPropertyPathCardinality($field->getPropertyPath(), $index_properties) != 1 ? 'm' : 's';
                  }
                  catch (SearchApiException $e) {
                    // Thrown by $field->getDatasource(). As all conditions for
                    // multiple values are not met, it seems to be a single
                    // value field. Note: If the assumption is wrong, Solr will
                    // throw exceptions when indexing this field. In this case
                    // you should add an explicit 'isList' => TRUE to your
                    // property or data definition! Or activate
                    // fallback_multiple in the advanced server settings.
                    $pref .= empty($this->configuration['fallback_multiple']) ? 's' : 'm';
                  }
                }
              }
            }
mkalkbrenner’s picture

Can we check if the aggregated field is single or multivalued rather than just relying on the is_list property?

The perfect solution would be to fix that in Search API, for example by providing a dedicated aggregated field for union.
In Search API Solr we can't skip the check for lists because it is also required for other fields.
So what we can do in Search API Solr is to add a check before we check for lists. If it is an aggregated field we could read its config and check for single value aggregations.

finn lewis’s picture

The perfect solution would be to fix that in Search API, for example by providing a dedicated aggregated field for union.

Thanks @mkalkbrenner - started an issue in search_api to explore that: https://www.drupal.org/project/search_api/issues/3340305

Meanwhile, I will take a look at adding a check before to delve into the aggregated field and see if we can check for single value aggregations.

gertlor’s picture

StatusFileSize
new1.28 KB

This patch works in combination with https://www.drupal.org/project/search_api/issues/3340305#comment-14940876 to ensure the field prefix for aggregated fields works correctly.

finn lewis’s picture

That looks great @Gertlor!! I will test.

finn lewis’s picture

The patch in #17 works for me, in combination with the patch for search_api in comment #4 of https://www.drupal.org/project/search_api/issues/3340305#comment-14940876

Thank you @Gertlor !!!

Assuming this approach is sound, what's the best way to get this committed and released @mkalkbrenner ?

finn lewis’s picture

Status: Needs work » Reviewed & tested by the community
mkalkbrenner’s picture

Version: 4.2.9 » 4.x-dev
Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs tests

The patch looks good. But ideally we can extend the existing test for field names.

That change will require to deploy a new config-set to Solr and to re-index. Therefore the required minimum schema version needs to be raised.

mkalkbrenner’s picture

Status: Needs work » Reviewed & tested by the community
Issue tags: -Needs tests

Search API 1.29 contains the required counterpart. I'll raise the minimum version requirement for Search API and the schema version for the next release and include this patch.

mkalkbrenner’s picture

Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.