First, thanks for creating this module! I am using it to power our search service on Travel Massive.

I am having trouble setting up a conditional filter on fulltext index where I want to exclude any null/empty fields. The condition works for String indexes, but not for Fulltext. I'd like to use a fulltext so I can can improve the matching. However as the fulltext index prevents me using the "is not null" condition, I can't implement the desired search behavior I need.

ie:
$query_filter_has->condition('user:field_link_twitter:url', null, '<>');

Behaviour:

If the field "user:field_link_twitter:url" is indexed as "String", the condition is applied successfully and filters out all null results.
If the field "user:field_link_twitter:url" is indexed as "Fulltext", the condition is not applied and results with null values are returned by the search query.

It looks like in service.inc line 1682, this logic gets caught by this code when a '<> null' condition is parsed:

$keys = $this->prepareKeys($f[1]);
if (!isset($keys)) {
continue;
}

In this scenario, $f[1] is null, so no filter query is applied. There's no other code that handles things, so the query condition is not correctly generated.

Hope this information helps and that there is a solution to support what I'm trying to do without having to move to a different search service. I like search_api_db :)

Thanks,
Ian

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

travelvc created an issue. See original summary.

drunken monkey’s picture

Thanks for reporting this problem!

While the documentation isn't completely clear about whether checking fulltext fields for NULL values is actually supported (and it kinda sounds like it isn't), Solr already seems to support it and I can't really see any reason why we shouldn't.
The attached patch should implement this, please test/review!

Also, we should make the documentation clearer in this regard.

Finally, the D8 version probably has exactly the same problem (at least definitely the documentation).

The last submitted patch, 2: 2745655-2--fulltext_field_null_condition--tests_only.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 2: 2745655-2--fulltext_field_null_condition.patch, failed testing.

drunken monkey’s picture

Oh, forgot the "minimum chars" setting has been changed at that point in the test code.
But testing before that would be even worse (since we'd then not even have an item without title), so just changing the assertion. Should be fine now. (Locally, I get over 30 fails with the clean checkout, so can't really tell.)

The last submitted patch, 5: 2745655-5--fulltext_field_null_condition--tests_only.patch, failed testing.

travelvc’s picture

Thanks for taking time to create the patch, however unfortunately it doesn't seem to work for me.

My steps were:

1. Applied patch in comment #5
2. Changed index type from string to fulltext
3. Reindexed search db

The results are the same as my original comment.

drunken monkey’s picture

Are you sure you applied the correct patch – i.e., the last one, with the green tests?

  • drunken monkey committed 4e30fb2 on 7.x-1.x
    Issue #2745655 by drunken monkey: Fixed "not empty" condition on...
drunken monkey’s picture

Project: Search API Database Search » Search API
Version: 7.x-1.x-dev » 8.x-1.x-dev
Component: Code » Database backend
Status: Needs review » Patch (to be ported)
Issue tags: -needs port to Drupal 8

Since the test demonstrates that this indeed fixes an existing problem, I've now committed this regardless.
If you still have problems with the latest dev version (after this commit), please create a new issue.

travelvc’s picture

Sorry for the delay to respond. The patch required a small modification to make it work.

In your patch you added the following three lines to createFilterCondition():

// If the field is in its own table, we have to check for NULL values in
// a special way (i.e., check for missing entries in that table).
if ($f[1] === NULL && $field['column'] === 'value') {
  $query = $this->connection->select($field['table'], 't')
    ->fields('t', array('item_id'));
+  if (search_api_is_text_type($field['type'])) {
+    $query->condition('t.field_name', $f[0]);
+  }
  $cond->condition('t.item_id', $query, $not_equals ? 'IN' : 'NOT IN');
  continue;
}

I had to modify the enclosing if statement as follows:

if (($f[1] === NULL && $field['column'] === 'value') or ($f[1] === NULL && $field['type'] === 'text')) {

This is what works for me in the end:

// If the field is in its own table, we have to check for NULL values in
// a special way (i.e., check for missing entries in that table).
if (($f[1] === NULL && $field['column'] === 'value') or ($f[1] === NULL && $field['type'] === 'text')) {
 $query = $this->connection->select($field['table'], 't')
   ->fields('t', array('item_id'));
 if (search_api_is_text_type($field['type'])) {
   $query->condition('t.field_name', $f[0]);
 }
 $cond->condition('t.item_id', $query, $not_equals ? 'IN' : 'NOT IN');
 continue;
}

Thanks again for steering me in the right direction, and hope this feedback is helpful.

Ian

drunken monkey’s picture

Version: 8.x-1.x-dev » 7.x-1.x-dev
Status: Patch (to be ported) » Needs work
Issue tags: +Release blocker

Hm, that's curious. From what I see in the code, $field['column'] === 'value' should always be TRUE for text types. Can you find out what $field['column'] is for you in this case? (And, if possible, whether that value is also stored in the server options, or added somewhere else in the code.)

In any case, though, it seems like it would be a good idea to set 'column' to 'word' right in getFieldInfo() for all fulltext fields. Currently, it's a bit confusing (and prone to cause errors) that the value is just wrong for text fields.

(The "Release blocker" tag is for the D8 port.)

travelvc’s picture

I put a print_r($field) inside if (search_api_is_text_type($field['type'])) { and here's what came out:

Array ( [table] => search_api_db_tm_search_multi_index_text [column] => user_field_link_twitter_url [type] => text [boost] => 1.0 )
Array ( [table] => search_api_db_tm_search_multi_index_text [column] => node_field_link_twitter_url [type] => text [boost] => 1.0 )

My query filters are set up as so:

$query_filter_has->condition('user:field_link_twitter:url', null, '<>');
$query_filter_has->condition('node:field_link_twitter:url', null, '<>');

In the configuration under "Select fields to index" I have these fields set as "Fulltext".

The field is a standard Drupal "Link" type - this is what the underlying table looks like:

mysql> describe field_data_field_link_twitter;
+-------------------------------+------------------+------+-----+---------+-------+
| Field                         | Type             | Null | Key | Default | Extra |
+-------------------------------+------------------+------+-----+---------+-------+
| entity_type                   | varchar(128)     | NO   | PRI |         |       |
| bundle                        | varchar(128)     | NO   | MUL |         |       |
| deleted                       | tinyint(4)       | NO   | PRI | 0       |       |
| entity_id                     | int(10) unsigned | NO   | PRI | NULL    |       |
| revision_id                   | int(10) unsigned | YES  | MUL | NULL    |       |
| language                      | varchar(32)      | NO   | PRI |         |       |
| delta                         | int(10) unsigned | NO   | PRI | NULL    |       |
| field_link_twitter_url        | varchar(2048)    | YES  |     | NULL    |       |
| field_link_twitter_title      | varchar(255)     | YES  |     | NULL    |       |
| field_link_twitter_attributes | mediumtext       | YES  |     | NULL    |       |
+-------------------------------+------------------+------+-----+---------+-------+
10 rows in set (0.00 sec)

Hopefully this additional information helps!

drunken monkey’s picture

I think I see what's going wrong there. I've opened #2795245: Fix changing of a field's type to "Fulltext" – could this be the problem? Did you change the types of those two fields from "String" (or anything else) to "Fulltext" at some point?
If so, please check out the patch in that issue and see whether it resolves your problem. To test, please just change the types of those fields again to "String" and then back to "Fulltext" – the change should now be handled correctly by the DB backend class and subsequent "is (not) NULL" filters work correctly.

travelvc’s picture

Success! I applied the patch (2795245-5--switch_to_fulltext_type.patch) and followed your instructions (changed the fulltext to string, then back to fulltext).

Note that I also required the patch in this issue to be applied (2745655-5--fulltext_field_null_condition.patch).

My next question is - how best to apply these fixes in production (given I've got a working implementation via the patch in this issue and my own modifications). Should I hold out until a combined patch or new release is made?

Thanks for your attention to this issue - and let me know if I can test anything else for you.

drunken monkey’s picture

Version: 7.x-1.x-dev » 8.x-1.x-dev
Status: Needs work » Patch (to be ported)

OK, great to hear, thanks for testing!
Then I'm moving this back to D8 and we can work on fixing this problem in the other issue. (Maybe also leave a comment there that the patch worked for you, so I remember. Or wait until I post a new patch that fixes the test fail and help me test that one.)

drunken monkey’s picture

It's already working fine in D8 due to other changes, so we just need to add the test, too.

borisson_’s picture

Status: Needs review » Reviewed & tested by the community

Just a test? Sure!

  • drunken monkey committed 7e086b0 on 8.x-1.x
    Issue #2745655 by drunken monkey: Added test for (NOT) NULL conditions...
drunken monkey’s picture

Status: Reviewed & tested by the community » Fixed

Good, thanks for reviewing!
Committed.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

AaronBauman’s picture

Does regressionTest2745655 cover views behavior as well?
I'm working on a custom search api argument, and seems like conditions involving fulltext NULL / NOT NULL are not working.

Not sure whether that merits a new issue, or if there's one open already

Marios Anagnostopoulos’s picture

I stumbled upon this as well. In my case I am trying to add a non empty filter for the default body field of nodes.
My field gets indexed as an empty string (like it should). No custom code applies to this process so far.

I am using elasticsearch and my indexed item looks like

{
  ...settings
  "_source": {
    "_language": "en",
    "rendered_item": [
      ...some markup
    ],
    "body": [
      ""
    ],
    "id": [
      1
    ],
    ... other source fields
  },
  ...
}

Are the two issues relevant?? IDK.

Also @AaronBauman, if you find a related issue please link it here, thanks.