Problem/Motivation

The MySQL function CONCAT_WS() takes a separator as first parameter. The current code in Drupal\views\Plugin\views\filter\Combine Combine->query() adds separators between the fields following too. This is redundant as the first parameter already defined the separator.

$count = count($fields);
$separated_fields = array();
foreach ($fields as $key => $field) {
  $separated_fields[] = $field;
  if ($key < $count-1) {
    $separated_fields[] = "' '";
  }
}
$expression = implode(', ', $separated_fields);
$expression = "CONCAT_WS(' ', $expression)";

Proposed resolution

This can be greatly simplified:

$expression = implode(', ', $fields);
$expression = "CONCAT_WS(' ', $expression)";

CONCAT_WS() compatibility for other database drivers than MySQL is a related, but unrelated issue ;)

Remaining tasks

User interface changes

API changes

Original report by @MrHaroldA

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue category Task because it simplifies existing code
Issue priority Normal, because it removes maintenance costs
Disruption Not disruptive, because its just internal changes
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

MrHaroldA’s picture

Status: Active » Needs review
FileSize
791 bytes
MrHaroldA’s picture

Issue summary: View changes
dawehner’s picture

Category: Bug report » Task
Issue summary: View changes

Looks really nice, especially given that we have dedicated test coverage for this filter.

Added an issue summary and a beta evaluation. I think though that this is more of a task than a bug fix, don't you think so?

MrHaroldA’s picture

Category: Task » Bug report
Status: Needs review » Needs work

I think though that this is more of a task than a bug fix, don't you think so?

I thought so at first too, but now I'm sure this is a bug, and it even needs work ;)

If you perform a phrase search using double quotes (") you'd have to add three spaces in between instead of one: "word word2". This is a bug, and a fairly big one too, which isn't covered in the test cases.

My patch solves this, but the tests have to be fixed first to show it.

MrHaroldA’s picture

Status: Needs work » Needs review
FileSize
1.6 KB
2.37 KB

Here's a test to help identify this bug.

MrHaroldA’s picture

Yeah, the summary must be changed too as this isn't a task anymore ...

The last submitted patch, 5: views-concat_ws-2405907-5-D8-test_only.patch, failed testing.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

Nice, though with no longer adding the quotes, are we maybe introducing/having existing security problems?
Given that fields are just the fields which appears in the query:

        $fields[] = "$field->tableAlias.$field->realField";

I would say no.

MrHaroldA’s picture

I'm not removing quotes, I'm removing space strings. Speaking about vague terms ;)

dawehner’s picture

Oh right.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/modules/views/src/Tests/Handler/FilterCombineTest.php
@@ -81,6 +81,46 @@ public function testFilterCombineContains() {
 
+  public function testFilterCombineContainsPhrase() {

We're missing a method doc block saying what this is testing.

MrHaroldA’s picture

We're missing a method doc block saying what this is testing.

I copied that from public function testFilterCombineContains() { which also doesn't have a method doc block.

alexpott’s picture

@MrHaroldA - well that's not obeying our standards. Not everything in core does :( - But we shouldn't add new violations if we spot them in new code.

MrHaroldA’s picture

That's exacly what I told a collegue of mine ;)

I'm brainstorming over what the doc should read. Is something like "Test if we can filter results using a phrase over two combined fields." enough information?

alexpott’s picture

Sounds good - should be "Tests..." though... maybe Tests filtering result using a phrase that matches combined fields. is better

MrHaroldA’s picture

MrHaroldA’s picture

Status: Needs work » Needs review

The last submitted patch, 16: views-concat_ws-2405907-16-D8-test_only.patch, failed testing.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

Adressed from alex got adressed

alexpott’s picture

Project: Drupal core » Views (for Drupal 7)
Version: 8.0.x-dev » 7.x-3.x-dev
Component: views.module » Code
Status: Reviewed & tested by the community » Patch (to be ported)

Committed 8c30ce9 and pushed to 8.0.x. Thanks!

Thanks for adding the beta evaluation to the issue summary.

alexpott’s picture

Project: Views (for Drupal 7) » Drupal core
Version: 7.x-3.x-dev » 8.0.x-dev
Component: Code » views.module
Status: Patch (to be ported) » Needs work

I've rolled this back due to it causing #2411703: Random fail in Drupal\user\Tests\UserAdminTest

We need to be able to configure whether to search across concatenated fields or search using an OR.

MrHaroldA’s picture

Well, I found this already: in real life, you'd have to enclose the search string with double quotes to search across fields, but in the test that doesn't work as it finds nothing.

So I guess the test method is failing, not my patch. Somewhere the test case inserts double quotes? Or doesn't cut the string up?

Have to look into it more though ...

  • alexpott committed 8c30ce9 on 8.1.x
    Issue #2405907 by MrHaroldA: Views combined filters add redundant...
  • alexpott committed ba831b2 on 8.1.x
    Revert "Issue #2405907 by MrHaroldA: Views combined filters add...

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

  • alexpott committed 8c30ce9 on 8.3.x
    Issue #2405907 by MrHaroldA: Views combined filters add redundant...
  • alexpott committed ba831b2 on 8.3.x
    Revert "Issue #2405907 by MrHaroldA: Views combined filters add...

  • alexpott committed 8c30ce9 on 8.3.x
    Issue #2405907 by MrHaroldA: Views combined filters add redundant...
  • alexpott committed ba831b2 on 8.3.x
    Revert "Issue #2405907 by MrHaroldA: Views combined filters add...
Lendude’s picture

I always figured that the 3 spaces were a delimiter between fields and there for a purpose.

+++ b/core/modules/views/src/Tests/Handler/FilterCombineTest.php
@@ -82,6 +82,49 @@ public function testFilterCombineContains() {
+        'value' => 'ohn Si',

I would actually expect this to return no results (as the current HEAD does), because none of my fields contain this value, it is only valid as an byproduct of the way we search across multiple fields.

I always thought the spaces were a bit of a weird choice as delimiters, but I would be against removing them.

Lendude’s picture

Status: Needs work » Closed (works as designed)
Issue tags: -Needs backport to Views 7.x-3.x

Actually, thinking about this more, spaces make perfect sense as delimiters because they are the only characters that won't give false positive results when using the 'word' operators that explode on spaces.

The spaces need to stay in.

MrHaroldA’s picture

Status: Closed (works as designed) » Needs work

The spaces need to stay in.

That is absolutely true, but the current code injects 2 spaces instead of one.

$separated_fields[] = "' '";

And

$expression = "CONCAT_WS(' ', $expression)";

This will generate a query containing double spaces, which prevents you from searching across multiple fields, unless the end user also adds double spaces in between.

Lendude’s picture

This will generate a query containing double spaces

It will create a query with triple spaces actually, because the inserted space will also be concatenated with the space separators.

which prevents you from searching across multiple fields

Which I feel is exactly the point when using the 'contains' operator. You don't want to find any results that only exist because we happen to search across multiple fields using a trick that concatenates the values (even if you did want to enable this kind of multi-field search, it would only work for fields that just happen to be concatenated in sequence, a sequence unknown to any end-user).

When you want to search for multiple terms across multiple fields, you use the 'contains any word'/'contains all words' operators, not the 'contains' operator. Of course that doesn't currently work #2664530: Views Combined fields filter with "Contains any word" or "Contains all words" operator results in an incorrect SQL query and a fatal error, but that's not the point :)

MrHaroldA’s picture

If you store name parts in separate fields (first name, last name, etc), searching on multiple fields in a predefined order is very nice.

For example: "old A" will match my full name ;)

Lendude’s picture

I'm not saying the end result will always be bad, but like the example in the test 'value' => 'ohn Si' this will mostly lead to 'what the ??' results for end users. Finding rows where none of the fields actually match the searched for data.

If you happen to have a View that puts your first name and current place of residence in sequence and you live in Amsterdam, then your search will also return true, but it's not a result you were actually looking for or expecting.

The current implementation gives clear results, 'the search string needs to be in one field', going with one space would lead to unpredictable results. Your use-case can also be accomplished using the other operators (once we get them to work....., which reminds me, I should look at that other issue again :)

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

  • alexpott committed 8c30ce9 on 8.4.x
    Issue #2405907 by MrHaroldA: Views combined filters add redundant...
  • alexpott committed ba831b2 on 8.4.x
    Revert "Issue #2405907 by MrHaroldA: Views combined filters add...

  • alexpott committed 8c30ce9 on 8.4.x
    Issue #2405907 by MrHaroldA: Views combined filters add redundant...
  • alexpott committed ba831b2 on 8.4.x
    Revert "Issue #2405907 by MrHaroldA: Views combined filters add...

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Lendude’s picture

Status: Needs work » Closed (works as designed)

Closing this now, this works as designed.

amaisano’s picture

I'm extending this Combine class to work with integers and other fields (like datetime). Having spaces in between each field on a CONCAT_WS fails horribly as the query only checks the first field before an errant space separator.

CONCAT_WS(' ', field_a, field_b) -> works as expected and is a properly formed MYSQL CONCAT_WS function call
CONCAT_WS(' ', field_a, ' ', field_b) -> does not work (even if matches are found in field_b) due to bad function call (presence of blank fields)

I assume however that there must be some magic the empty fields (space fields), performs when searching against string-based field values and that is why this patch was rejected?

MrHaroldA’s picture

I assume however that there must be some magic the empty fields (space fields), performs when searching against string-based field values and that is why this patch was rejected?

I'm still in doubt why this got rejected. The three spaces are obviously a bug, but it hides unwanted behavior: matching part of strings from sequential fields.

If the user used three spaces in his search string, the partial match across sequential fields is back.

So the current implementation is wrong, but convenient.

Lendude’s picture

If the user used three spaces in his search string, the partial match across sequential fields is back.

Nope, because multiple sequential spaces get stripped from the search string. It is impossible to search across fields now, the triple space separator is the only way to do this, because it is impossible to get into your search string.

suchdavid’s picture

@Lendude Actually for me the filter gives results with three spaces. I mistakenly used the combined filter like other people to combine first_name last_name field then I found this issue. So tested with three spaces and got result with it. I used it as simple exposed filter.