Currently, all entity properties up to a certain level of indirection will be available as field handlers for Search API index base tables.
While a nice work-around for me not needing to dive into the code of another Views subsystem, this is quite restricted, doesn't abide to the normal way this works in Views and, most importantly, can lead to serious performance issues for more complex site.
Therefore, I think there is little choice than to finally do this the right, not the easy way, and use relationships. It might be possible that a generic Entity Relationship Handler in the Entity API could help us there – but probably I'd have to write that one myself as well. Right now, I have little to no conception of how relationships work in Views (from a developer's perspective), so … let's see, what might be best there.

In any case, this would probably lead to some data loss on updating, unless there's a feasible way to write an update function. Therefore, this has definitely to happen before a 1.0 release – tagging accordingly.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jziwenchen’s picture

thank your work.

how is going on this feature ?

mh86’s picture

subscribing

Berdir’s picture

Subscribing

BeaPower’s picture

sub... this would mean I could add fivestar to search results right?

drunken monkey’s picture

No, I don't think so (if it isn't possibly now in the way you mean it). This wouldn't actually add (much) functionality, but rather fix the way current functionality is provided.

BeaPower’s picture

Ok, is there a patch anywhere to include fivestar ratings? I'm creating a reviews site, Thanks.

Shadlington’s picture

You can display fivestar in results already if you use the entity row style rather than fields.
However, you cannot sort or filter by fivestar as it does not implement the entity api.

If you want to go into further detail on this its really more appropriate in its own issue - and would need an issue in the fivestar queue to get anywhere useful with regards to sorting/filtering/etc.

EDIT: And now that I've looked at the queue I can see you've already done that #1256572: Integrate with Entity API

drunken monkey’s picture

See #1266036: Add generic Views entity tables with fields and relationships for 80% of the work for this issue.
Then, I'll just have to throw out field handlers for related fields and add the relationships.

drunken monkey’s picture

Status: Active » Needs work
FileSize
33.15 KB

I'm now starting to implement this in parallel to my work in #1266036: Add generic Views entity tables with fields and relationships. Attached is a version adapted to the current state of the Entity API views_integration branch. It doesn't work as such, has one nasty but probably small bug (no field handlers are added) and surely tons of others. Particularly, I have yet to figure out how to support non-entities without extending all the field handlers. Got an idea, though.

Anyways, this is not really suitable for testing, but already shows where this is going (or supposed to be going).

Pedro Lozano’s picture

Subscribing. I've been trying to build today a search view that needs this. I'll do some testing as soon as it's working.

Thanks.

drunken monkey’s picture

Current status. Relationships still not working in any way, but the rest is looking more and more promising.

jakonore’s picture

subscribing

jsacksick’s picture

subscribing

tnightingale’s picture

subscribing

drunken monkey’s picture

Latest status. As written in #1266036-12: Add generic Views entity tables with fields and relationships, almost everything already seems to work (as far as my bit of testing goes). Relationships are missing, though.

drunken monkey’s picture

Little correction to avoid breaking the entity row plugin (which still uses the magic $row->entity property).

drunken monkey’s picture

Status: Needs work » Needs review

But seems to be done, as far as I can tell. (Some bugs will surely spring up, though.)

marcoka’s picture

i applied the newest patch with git apply -v patchname.patch somehow git applys not all....testing with pact p1 again.

drunken monkey’s picture

Did you apply the patch in #1266036-14: Add generic Views entity tables with fields and relationships as well? The patch here is just an application of the changes in Entity API – without those, this has to fail.

Pedro Lozano’s picture

Works like a charm for a very simple setup, I'm trying now to use it on an existing site but everything breaks when the patch are applied.

Is a upgrade path needed for any of the two patches (entity, search_api)?

drunken monkey’s picture

Yes, as the field handlers for the Search API tables have changed (as have, consequently, some of their options) and you also have to use relationships to get some fields that were previously included by default, some existing views might break.
Creating update code would probably be possible, but rather tedious as you'd have to hack deep into the views' data structures. Regretable as it is, I think it would be easier if people just updated their existing views by hand. Usually, only a few field handlers should be affected.

Or do you mean something else by „everything breaks“? A little elaboration would be nice here.

Pedro Lozano’s picture

I got several PDO sql exceptions but I didn't saved the text to paste it here.

I'm trying another approach now. Uninstalling search_api, upgrading entity, upgrading search_api, enable search_api again and build the view form scratch. I'll report later how things are going.

Pedro Lozano’s picture

I got several PDO sql exceptions but I didn't saved the text to paste it here.

I'm trying another approach now. Uninstalling search_api, upgrading entity, upgrading search_api, enable search_api again and build the view form scratch. I'll report later how things are going.

drunken monkey’s picture

Revised version adding support for click-sorting. Use together with #1266036-19: Add generic Views entity tables with fields and relationships.

@ Pedro: It's weird, everything just kept on working for me. Did you make sure to properly clear the caches, etc.?

das-peter’s picture

subscribing

Pedro Lozano’s picture

After this patch I would expect that any view that can be build without using a search index as a base table can also be build using the search index as a base, since you can fetch the referenced entity.

It is not true for the view I'm trying to build. It involves entityreference fields. Looks like the relationships support of entityreference fields is not working when I build the view using the search index.

I got the following data structure.

* Entity type: THINGS
* THINGS have an entityreference field pointing to content type A
* There is another content type B that has an entityreference field pointing to entities of type THINGS.

Without using search api, I can build a view that:

* Lists THINGS
* Prints the node title and fields from the referred node (type A) (Using entityreference views relationship support)
* Prints the node title and fields of nodes of type B that are referrencing each THING. (Using entityreference *reverse reference* views support).

When building the same view using the search index I cannot build such a view since the relationships options that Entityreference provides are not available in the views relationships interface.

So when building the view without search api I see the following options in the relationship dialog:

Entity Reference: Referenced Entity (A bridge to the Content entity that is referenced via field_referred_node)
Entity Reference: Referencing entity (A bridge to the Things entity that is referencing Content via field_referred_node)
Entity Reference: Referenced Entity (A bridge to the Things entity that is referenced via field_referred_thing)
Entity Reference: Referencing entity (A bridge to the Content entity that is referencing Things via field_reffered_thing)

None of these options are available when building the same view using the search index.

Let me know if you need more info.

drunken monkey’s picture

None of these options are available when building the same view using the search index.

None of these? That's weird, I thought that the first of these options should show. Do you get the option of indexing the reference (field_referred_node) on the index's „Fields“ tab? Otherwise, the issue might just be that Entity Reference doesn't provide Entity API integration yet, which is nothing we could fix.
Reverse entity references are not supported right now. I'd have to look into how these work / could be made to work here.

Anyways, the problem here is not with the Search API part but with the Entity API part, so this should probably better be discussed in #1266036: Add generic Views entity tables with fields and relationships.

tnightingale’s picture

Here is #24 rerolled to work with 7.x-1.x

drunken monkey’s picture

Slight adjustment regarding sanitizing of field identifiers.

drunken monkey’s picture

Adding preloading of result entities.

das-peter’s picture

Am I right that the patch contains a "forecast" of this #1260812: Move "Database search" into its own project?
The patch deletes all the contrib module search_api_db.
Next guess, the removed module can be found here soon: http://drupal.org/project/search_api_db

das-peter’s picture

Just applied latest patch from here and synced to the latest entity views branch.
I discovered that if the search index is out of sync I get this notice:

Notice: Undefined offset: 163 in entity_plugin_row_entity_view->render() (Line 106 of modules/entity/views/plugins/entity_plugin_row_entity_view.inc).

This further leads to this nice exception - an thus to a broken site:

EntityMalformedException: Missing bundle property on entity of type node. in entity_extract_ids() (Line 7414 of drupal/includes/common.inc).

It looks like this could be fixed in the function render() of entity/views/plugins/entity_plugin_row_entity_view.inc by changing this condition:
if (isset($id)) {
to this:
if (isset($id) && isset($this->entities[$id])) {

But I'm not sure if this is the (only|right) place where this kind of sanitization should take place. Has the search_api a good way to check wheter an entity id is still valid?

All other stuff looks very good - after reindexing :)
Thank you very much!

drunken monkey’s picture

Am I right that the patch contains a "forecast" of this #1260812: Move "Database search" into its own project?

Ah, you're right, sorry!
Attached is a patch without these unrelated changes.

It looks like this could be fixed in the function render() of entity/views/plugins/entity_plugin_row_entity_view.inc by changing this condition:

So the problem occurs when the Search API returns an entity ID of an already deleted entity?
a) This shouldn't happen at all, as we immediately remove deleted entities from indexes.
b) I guess this will, as you say, have to be fixed in the Entity API. You should create an issue there (if you haven't done so already).

All other stuff looks very good - after reindexing :)

Good to hear, thanks for testing!

mh86’s picture

Hey Thomas,
one of my views still throws some notices which I think are caused by this patch:
Undefined property: SearchApiViewsQuery::$group_operator in SearchApiViewsQuery->build() (Zeile 157 von search_api/contrib/search_api_views/includes/query.inc

drunken monkey’s picture

That's (probably) known and unrelated: see and review #1305736: Notice: Undefined property: SearchApiViewsQuery::$group_operator.

mh86’s picture

Status: Needs review » Needs work

from http://drupal.org/node/1266036#comment-5106500 and according to Thomas, related to this issue

There is one use case I still couldn't get working:
I have a Search API index with field collections. Additionally I have my own entity property from a field collection to a profile2. Adding this relation in Views basically works, but when using a field that is based on the views_handler_field_entity handler (e.g. Rules Link in combination with the patch in #1264258: Update to latest views changes), errors are thrown and no values are displayed. Following code in pre_render
list($this->entity_type, $this->entities) = $this->query->get_result_entities($values, !empty($this->relationship) ? $this->relationship : NULL, $this->field_alias);

seems to ignore the relationship (although it is correctly set in $this->relationship) and returns the field collections instead of the profiles.

drunken monkey’s picture

Status: Needs work » Needs review
FileSize
37.34 KB

Sorry for overlooking this one. Please see if the attached patch fixes the problem.

One difficulty I encountered was that get_result_entities() doesn't define what to do if a relationship maps to multiple entities for a row. (To be more precise, it doesn't define anything at all, of course. But for that specific case, even looking at the Views default implementation doesn't help.) I now just return the first entity, as returning an array might screw things up considerably more than just returning too few entities.

mh86’s picture

Status: Needs review » Needs work

hm, no, it still doesn't work.
From what I can see $this->query->get_result_entities() just returns the profiles (now correctly), but misses the entity type.

drunken monkey’s picture

Oops, sorry!
Should be fixed now, also fixed the error in help texts for some properties (e.g., body text).

drunken monkey’s picture

Status: Needs work » Needs review
mh86’s picture

Status: Needs review » Reviewed & tested by the community

Well done, Thomas. My use cases (which are quite complex) seem to work now. Once the Entity API Views stuff is committed, I would say this is ready as well.

fago’s picture

Status: Reviewed & tested by the community » Needs work

from the entity api issue:

>Of course, since they are from different base tables. Nothing we can do about it, I'm pretty sure – >Views just isn't really suited for our use cases (although I consider this behaviour rather confusing in >normal Views, too – in my opinion, the group shouldn't list the base tables, but also directly the >relationships, as I did it in my previous Views integration).

Ouch. I had another look and indeed each field only works with one of the base tables. That's a real UX nightmare though, so we need to find a solution here somehow. Changing how Views does it is probably out of scope and I think it basically makes sense as it keeps the amounts of fields to select lower.

So maybe we should just fix the names of the search api index fields to be named differently? E.g. "Indexed node: Title"?
Related but a minor thing is, that the base-table name is just named like the index. If you choose "node" as index name choosing base tables is going to be confusing. I'd suggest using "Search API: index name" there. So maybe "Search API Index name: Field name" would be a good naming for fields too then? Well, that would be belong into the search api issue then.

-> We need to solve this UX problem.

drunken monkey’s picture

Status: Needs work » Needs review
FileSize
40.21 KB

Little we can do about this, I think.
I admit it could be confusing for users, but the same goes for the other alternatives. Also, this can be easily changed later, too.
Therefore, let's just leave it like it is and mayb change it if the masses begin to revolt.

Attached is a patch adapting to the latest changes in #1266036-90: Add generic Views entity tables with fields and relationships.

drunken monkey’s picture

Updated.

davidseth’s picture

#44 does not apply cleanly for me with the latest dev code.

mh86’s picture

Status: Needs review » Needs work

In order to get the multi load functionality for base entities working, entity_load() must be called before EntityFieldHandlerHelper::extract_property_multiple() is used in get_result_wrappers().

drunken monkey’s picture

Status: Needs work » Needs review
FileSize
38.03 KB

#44 does not apply cleanly for me with the latest dev code.

Then you're doing something wrong.

In order to get the multi load functionality for base entities working, entity_load() must be called before EntityFieldHandlerHelper::extract_property_multiple() is used in get_result_wrappers().

Ah, yes, thanks. Should be fixed in the attached patch. Also enabled multi-loading for entities with no static cache.

Any other objections?

mh86’s picture

Status: Needs review » Needs work

Something goes wrong in #47. It seems to multi load the entities, but doesn't display them any more (in my view only one row out of 3 is shown, in another view all rows are display, but the values of a related field are missing).
Adapting patch from #43 by moving the entity_load() call up in get_result_wrappers() works as expected.

drunken monkey’s picture

Status: Needs work » Needs review
FileSize
36.99 KB

Like this?

mh86’s picture

Status: Needs review » Needs work

yep, like this. but I think we can remove the check for the static cache, as done in #47.

drunken monkey’s picture

Status: Needs work » Needs review
Issue tags: -D7 stable release blocker
FileSize
1.58 KB

yep, like this. but I think we can remove the check for the static cache, as done in #47.

No, we can't (or shouldn't). In #47 I used the entities in the wrappers, thus making this sensible for non-statically-cached entities, too. Since that didn't seem to work, we are now back to the old way, so it now finally works at least …

Committed this now. Any cosmetical or performance-enhancing changes can wait (in theory even until 1.1), we can't wait forever for RC1.

For making it also work with non-statically-cached entities, please see if the attached patch works for you. It's almost the same as the previous one (#47), though, and I don't really know what might go wrong there for you. It works at least for me.

jaymallison’s picture

So I'm curious about this issue and whether it's related to a something that's "missing" in my available fields in my View.

I've built out a view against a content index.

This view contains a relationship for a Drupal Commerce product reference against the listed node.

After adding this relationship, I can add fields for all of the Drupal Commerce fields except the "Add to cart" form. Under a normal Content view, this field is available and works great through a product reference relationship. It's only when the source is for the view is a Search index that this problem arises.

I'm using all of the latest -dev versions of Search Api and related modules. And I've applied that patch in #51 since it's not included in the current -dev.

I would have created a new issue, but it seems this must be related to this issue. Considering after I read this entire issue I saw someone mention that the bottom line with this patch was that you should be able to recreate any view from the search index that you can build with a normal view - if it's working correctly.

Let me know if there is any technical data I can provide.

Let me just say that this project is epic, and I am "this" close to basically developing a full e-commerce site that rivals Newegg in searching and filtering options contextually for each section thanks to this module. This is the last little piece...lol. Thank you for all that you do, it's amazing.

mh86’s picture

Status: Needs review » Reviewed & tested by the community

Great to have this finally committed :) Patch from #51 works as well.

Concerning #52, I guess Drupal Commerce doesn't yet use the generic field_entity handler with the view entity tables (see #1270890: provide a way to provide entity-related views fields for more info).

jaymallison’s picture

I was afraid it might be an issue with Commerce, although I had hoped that since it works fine under a normal view that it would work with the Search index.

I'm not trying to pull the add to cart form off a search indexed field. I've added a new relationship to the view for the Content: Product and it is off of that relationship that I should be able to add the add to cart field.

I'm sure you already knew that though and I'm just going to have to wait for this to be resolved another way.

drunken monkey’s picture

Status: Reviewed & tested by the community » Fixed

@ mh86: Great, committed. Thanks for testing!

@ jaymallison: As Matthias says, this isn't something that should work. For that, the Commerce module would have to provide that Views field in a query plugin-independent way, as introduced in #1270890: provide a way to provide entity-related views fields. Or you could manually add that field to the corresponding entity_X Views table. We use those tables for our relationships, not the normal ones provided for Views. Maybe the first two posts in #1266036: Add generic Views entity tables with fields and relationships might help understand the principle.

jaymallison’s picture

Ok, thanks for responding and giving me some clues on where to go with this. Otherwise, this system is working really nicely. Great work!

jaymallison’s picture

I hate to post in this issue again, but I'm not sure if this is related or not...

If you add fields to the index via the "Add related fields" dialog in the admin and then have them indexed, you of course can access those fields as normal fields in views... What I'm not seeing though, is a way to allow those fields to be a part of a views Sort. They simply just don't show up in the Sort section of views, even though I've set them as string or integer and they are single value.

drunken monkey’s picture

Does work for me. Are you sure both the field and the relation are single-valued? In the HTML code of the index's „Fields“ tab, what are the „Type“ values you can select for the field?

Also, please start creating new issues for such questions.

jaymallison’s picture

I'll create a new issue, sorry.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

checker’s picture

Have you created a new issue? I can't find it and i have exactly the same problem like #57.

checker’s picture

OK, the relation was not single-valued and this is not support i guess.