Currently getPropertyDefinitions() unsets fields that have custom storage set. Computed fields are treated as custom storage, since it calls hasCustomStorage() in ContentEntity::getPropertyDefinitions() (/docroot/modules/search_api/src/Plugin/search_api/datasource/ContentEntity.php):

// Exclude properties with custom storage, since we can't extract them
    // currently, due to a shortcoming of Core's Typed Data API. See #2695527.
    foreach ($properties as $key => $property) {
      if ($property->getFieldStorageDefinition()->hasCustomStorage()) {
        unset($properties[$key]);
      }
    }

Comments

kducharm created an issue. See original summary.

kducharm’s picture

kducharm’s picture

8.x-1.0-beta2 patch version 8.x-1.x version worked, this is unnecessary

kducharm’s picture

Issue summary: View changes
Status: Active » Needs review
drunken monkey’s picture

Component: General code » Plugins

Thanks for reporting this issue!
This seems a bit of an unclean solution, though, accessing the definition directly. I'd just add && !$property->isComputed() to the condition instead – much cleaner using the interface provided.

However, the larger problem is that, according to a quick test, computed fields cannot be retrieved automatically any more than fields with custom storage can. So showing these properties as options would be just as confusing and pointless as for normal custom storage properties.
Or what's your rationale there?

drunken monkey’s picture

Status: Needs review » Postponed (maintainer needs more info)
kducharm’s picture

My use case for having Computed Fields available for search indexing is for adding data to the search index that I want to fill in during hook_entity_storage_load(), and want that data available from Views built on the Search API index.

For example, a Person entity (DB table) that doesn't contain in itself any data about what Shirt Size(s) they wear. A ShirtSize entity (DB table) has a unique ID, Shirt Size, and a Person ID - but I want the Person index to include their Shirt Size for creating facets. I can add a computed field of "Shirt Sizes" as an entity reference in the Person entity, and fill in a hook_entity_storage_load() that populates the $Person->setShirtSizes($shirt_sizes_ids). Then when I go to configure fields for Person index, I have full access to the Person Shirt Sizes entity all in one spot.

This comes about where you can't alter your DB table/entity to include a column with the entity reference already populated to the destination, and the need to build a View on a Search Index. On a regular view, I would just add to ViewsData to make a new field with the relationship but I can't do that with Search API (as far as I know) since it's not a DB query.

I'm open to other suggestions too, I know there are various ways to get data directly into the search index, but seem to lose the whole entity reference benefits in Views.

drunken monkey’s picture

It's clear that being able to index computed fields (or other fields with custom storage) would be useful in a lot of cases, that's not the issue. The issue, as said, is that it's just not possible to do generically with the current Drupal Core Entity API (as far as I know). Unless that's possible, making the fields available to be added to the index won't get us anything (except end user confusion).

For your use case, you can create a custom processor that just adds your property to the datasource (see \Drupal\search_api\Plugin\search_api\processor\AddURL for an example). That custom property can also be an entity reference, that's no problem – and then you should be able to use it in Search API Views just like any other indexed field (at least once #2789431: Allow the display of processor-generated fields/properties in Views is fixed).

If you have any general solution for this problem, please explain it.
Otherwise, I'd say we change the issue to a "Support request" and mark it "Fixed".

cyberwolf’s picture

I stumbled upon this issue while experimenting with computed fields. Let me describe my use case.

I want to expose the year of publication of nodes as facets on a search delivered by search_api, on an index in Elasticsearch with the elasticsearch_connector module, and (obviously) the facets module. I thought I could accomplish this by indexing the date and then there would be a convenient option in facets to do this, but there doesn't seem a way (yet) to make a facet from a part of the indexed date, or I just couldn't find it :)

So, being pretty new to D8 but quite experienced with D7, I remembered the possibility to add 'extra' fields without any storage requirements in D7, searched for something similar in Drupal 8, and ended up with the computed fields as described on https://www.drupal.org/node/2112677. Although the guidelines on that page did not seem to be entirely correct, with some trial and error I managed to create a created_year property on nodes (derived from the value of the 'created' property), which gets successfully indexed by search_api when the patch from this ticket is applied. The biggest pitfall for me was the example BixProfileCurrentCompany class on the documentation page. search_api's Utility\FieldsHelper::extractFieldValues() expects the $data to be 'iterable' (it loops over it with a foreach) when it's a list, but the example BixProfileCurrentCompany does not properly initialize a list on construct, although it extends FieldItemList, so eventually no values were extracted because there was nothing to iterate through.

The essential pieces to create the computed field for my use case follow below. They still need polishing, and they currently live in a custom module named 'ocmwle_common', please don't pay attention to that.

src/Plugin/YearCreated.php:

<?php

namespace Drupal\ocmwle_common\Plugin;

use Drupal\Core\Datetime\DateFormatterInterface;
use Drupal\Core\Field\FieldItemList;
use Drupal\Core\TypedData\TypedDataInterface;
use Drupal\node\Entity\Node;

class YearCreated extends FieldItemList {

  public function __construct($definition, $name = NULL, TypedDataInterface $parent = NULL) {
    parent::__construct($definition, $name, $parent);

    $this->extractYear();
  }

  public function getValue($include_computed = FALSE) {
    $this->extractYear();

    return parent::getValue($include_computed);
  }

  private function extractYear() {
    /** @var Node $node */
    $node = $this->getEntity();
    $created = $node->getCreatedTime();

    /** @var DateFormatterInterface $dateFormatter */
    $dateFormatter = \Drupal::service('date.formatter');
    $year = $dateFormatter->format($created, 'custom', 'Y');

    $offset = 0;
    $this->list[$offset] = $this->createItem($offset, $year);
  }
}

ocmwle_common.module:

<?php

use Drupal\Core\Entity\EntityTypeInterface;
use Drupal\Core\Field\BaseFieldDefinition;
use Drupal\ocmwle_common\Plugin\YearCreated;

/**
 * Implements hook_entity_base_field_info().
 */
function ocmwle_common_entity_base_field_info(EntityTypeInterface $entity_type) {
  if ($entity_type->id() === 'node') {
    $fields = [];

    $fields['created_year'] = BaseFieldDefinition::create('integer')
      ->setLabel(t('Created year'))
      ->setDescription('The year of the created date (computed)')
      ->setComputed(TRUE)
      ->setClass(YearCreated::class);

    return $fields;
  }
}

I am attaching an updated patch for search_api, which follows the advice of drunken monkey in #5.

Regarding the last comments in #8, I am too new to D8 to judge wether it is possible or not to do this generically with the current Drupal Core Entity API. In my specific case, it eventually seemed to work. If you have particular cases you know that do not work correctly, please share and I might contribute some time to look at them.

drunken monkey’s picture

Status: Postponed (maintainer needs more info) » Needs review
StatusFileSize
new1.12 KB
new1.1 KB

Huh, you're right, this does look like it should actually work. I had merely happened to test with the "Path" field (the only computed field in Core, as far as I can see), and that just seems to be not implemented properly (for this use). But it might actually work with most other computed fields – not really a way to be sure.

We should, however, exclude the "Path" field, so it doesn't cause confusion. I don't like to do this in general, but it's in Core after all, and otherwise we're bound to get some bug reports for that.
Also, I reformatted a bit to get rid of the overlong line.

Revised patch attached. Please review whether it works correctly for you and I can commit.

Status: Needs review » Needs work

The last submitted patch, 10: 2814925-10--index_computed_properties.patch, failed testing.

kducharm’s picture

I didn't actually use the patch #10 but pasted in the changes to my project, I can at least confirm this still allowed all of the computed fields I defined to show up as choices when adding fields to the search index. Would love to see it merged, good find.

drunken monkey’s picture

Status: Needs work » Needs review

Thanks for reporting back with this!
Anyone else want to test the patch?

borisson_’s picture

Status: Needs review » Reviewed & tested by the community

I don't like that its two nested if statements but I understand you did this because of readability so RTBC imho.

drunken monkey’s picture

Status: Reviewed & tested by the community » Fixed

Good to hear, thanks for reviewing!
I'm not sure I like it, either, but I guess we can go with it. (Even though I wonder why I didn't have the two ifs in the other order …)
Committed.
Thanks again, everyone!

sukanya.ramakrishnan’s picture

I cant get this patch to work with views? I cannot add a computed field to a search view successfully. Does this work for anyone else?

Thanks
Sukanya

drunken monkey’s picture

This should work, as far as I can see. Maybe the data type in question is just not supported?
Please create a new issue with your problem description and a bit more details.
(Would be good to hear from others here, though, whether this works for them. I don't have a computed field (other than "Path") available for testing.)

sukanya.ramakrishnan’s picture

Thanks for the response.

This is the error I got

Notice: Undefined index: in Drupal\views\Plugin\views\field\Field->getFieldStorageDefinition() (line 332 of core/modules/views/src/Plugin/views/field/Field.php).
Drupal\views\Plugin\views\field\Field->getFieldStorageDefinition() (Line: 346)
Drupal\views\Plugin\views\field\Field->defineOptions() (Line: 99)
Drupal\search_api\Plugin\views\field\SearchApiEntityField->defineOptions() (Line: 17)
Drupal\views_field_metadata\Plugin\views\field\SearchApiEntityFieldWithMetadata->defineOptions() (Line: 138)
Drupal\views\Plugin\views\PluginBase->init(Object, Object, Array) (Line: 102)
Drupal\views\Plugin\views\HandlerBase->init(Object, Object, Array) (Line: 115)
Drupal\views\Plugin\views\field\FieldPluginBase->init(Object, Object, Array) (Line: 177)

when i check the Drupal\views\Plugin\views\field\Field->getFieldStorageDefinition() method, there is a specific condition to get the definitions for only non-computed fields which is why the view is failing,

Thanks,
Sukanya

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

pwaterz’s picture

I would like this issue to be reopened. The approach taken in patch 10 is making too big of assumptions about not being able to index fields that have custom_storage set to TRUE. Here is some example code that does NOT work with the current patch because the field can't be selected:

Execute this php script:

$content_type = NodeType::create(['type' => 'some_new_content_type']);

// Create field storage
$field_storage = [];
$field_storage['id'] = 'node.some_field_with_custom_storage' ;
$field_storage['field_name'] = 'some_field_with_custom_storage';
$field_storage['type'] = 'text_long';
$field_storage['entity_type'] = 'node';
$field_storage['custom_storage'] = TRUE;
$field_storage = FieldStorageConfig::create($field_storage);
$field_storage->save();

// Create a field with the custom storage
$drupal_field = FieldConfig::create([
 'field_storage' => $field_storage,
 'bundle' => 'some_new_content_type',
 'label' => 'Some field label',
 'settings' => array('display_summary' => TRUE),
 ]);
$drupal_field->save();

In some_module.module

function some_module_entity_storage_load(array $entities, $entity_type) {
  if ($entity_type == 'node') {
    foreach ($entities as $entity) {
      if ($entity->type == 'some_new_content_type') {
        $entity->set('some_field_with_custom_storage', 'some random text', FALSE);
      }
    }
  }
}

I've attached my purposed solution as a patch. Now you can add 'some_field_with_custom_storage' field to the index, and if you index the nodes, it will index properly. Thoughts?

The issue really is in Drupal core in that you can create fields where you can not retrieve their values via typeddate api, but we can not rely on checking if something has custom storage. I think we need to just exclude known fields until core api allows us to effectively check if it can be indexed.

Perhaps I should open a new issue for this?

cyberwolf’s picture

@pwaterz: The assumption that fields that have custom storage can not be indexed, was there already. I think it's better to open a new ticket to change something regarding the other assumption.

pwaterz’s picture

Will do

anshu raj’s picture

Here is the patch for Drupal 9

anshu raj’s picture

StatusFileSize
new1.13 KB

Here the patch for index_computed_properties