Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Problem/Motivation
The MySQL function CONCAT_WS() takes a separator as first parameter. The current code in Drupal\views\Plugin\views\filter\Combine Combine->query() adds separators between the fields following too. This is redundant as the first parameter already defined the separator.
$count = count($fields);
$separated_fields = array();
foreach ($fields as $key => $field) {
$separated_fields[] = $field;
if ($key < $count-1) {
$separated_fields[] = "' '";
}
}
$expression = implode(', ', $separated_fields);
$expression = "CONCAT_WS(' ', $expression)";
Proposed resolution
This can be greatly simplified:
$expression = implode(', ', $fields);
$expression = "CONCAT_WS(' ', $expression)";
CONCAT_WS() compatibility for other database drivers than MySQL is a related, but unrelated issue ;)
Remaining tasks
User interface changes
API changes
Original report by @MrHaroldA
Beta phase evaluation
Issue category | Task because it simplifies existing code |
---|---|
Issue priority | Normal, because it removes maintenance costs |
Disruption | Not disruptive, because its just internal changes |
Comment | File | Size | Author |
---|---|---|---|
#16 | interdiff-5-16.txt | 828 bytes | MrHaroldA |
#16 | views-concat_ws-2405907-16-D8.patch | 2.42 KB | MrHaroldA |
#16 | views-concat_ws-2405907-16-D8-test_only.patch | 1.65 KB | MrHaroldA |
Comments
Comment #1
MrHaroldA CreditAttribution: MrHaroldA commentedComment #2
MrHaroldA CreditAttribution: MrHaroldA commentedComment #3
dawehnerLooks really nice, especially given that we have dedicated test coverage for this filter.
Added an issue summary and a beta evaluation. I think though that this is more of a task than a bug fix, don't you think so?
Comment #4
MrHaroldA CreditAttribution: MrHaroldA commentedI thought so at first too, but now I'm sure this is a bug, and it even needs work ;)
If you perform a phrase search using double quotes (") you'd have to add three spaces in between instead of one: "
word word2
". This is a bug, and a fairly big one too, which isn't covered in the test cases.My patch solves this, but the tests have to be fixed first to show it.
Comment #5
MrHaroldA CreditAttribution: MrHaroldA commentedHere's a test to help identify this bug.
Comment #6
MrHaroldA CreditAttribution: MrHaroldA commentedYeah, the summary must be changed too as this isn't a task anymore ...
Comment #8
dawehnerNice, though with no longer adding the quotes, are we maybe introducing/having existing security problems?
Given that fields are just the fields which appears in the query:
I would say no.
Comment #9
MrHaroldA CreditAttribution: MrHaroldA commentedI'm not removing quotes, I'm removing space strings. Speaking about vague terms ;)
Comment #10
dawehnerOh right.
Comment #11
alexpottWe're missing a method doc block saying what this is testing.
Comment #12
MrHaroldA CreditAttribution: MrHaroldA commentedI copied that from
public function testFilterCombineContains() {
which also doesn't have a method doc block.Comment #13
alexpott@MrHaroldA - well that's not obeying our standards. Not everything in core does :( - But we shouldn't add new violations if we spot them in new code.
Comment #14
MrHaroldA CreditAttribution: MrHaroldA commentedThat's exacly what I told a collegue of mine ;)
I'm brainstorming over what the doc should read. Is something like "Test if we can filter results using a phrase over two combined fields." enough information?
Comment #15
alexpottSounds good - should be "Tests..." though... maybe
Tests filtering result using a phrase that matches combined fields.
is betterComment #16
MrHaroldA CreditAttribution: MrHaroldA commentedHere ya go!
Comment #17
MrHaroldA CreditAttribution: MrHaroldA commentedComment #19
dawehnerAdressed from alex got adressed
Comment #20
alexpottCommitted 8c30ce9 and pushed to 8.0.x. Thanks!
Thanks for adding the beta evaluation to the issue summary.
Comment #21
alexpottI've rolled this back due to it causing #2411703: Random fail in Drupal\user\Tests\UserAdminTest
We need to be able to configure whether to search across concatenated fields or search using an OR.
Comment #22
MrHaroldA CreditAttribution: MrHaroldA commentedWell, I found this already: in real life, you'd have to enclose the search string with double quotes to search across fields, but in the test that doesn't work as it finds nothing.
So I guess the test method is failing, not my patch. Somewhere the test case inserts double quotes? Or doesn't cut the string up?
Have to look into it more though ...
Comment #27
LendudeI always figured that the 3 spaces were a delimiter between fields and there for a purpose.
I would actually expect this to return no results (as the current HEAD does), because none of my fields contain this value, it is only valid as an byproduct of the way we search across multiple fields.
I always thought the spaces were a bit of a weird choice as delimiters, but I would be against removing them.
Comment #28
LendudeActually, thinking about this more, spaces make perfect sense as delimiters because they are the only characters that won't give false positive results when using the 'word' operators that explode on spaces.
The spaces need to stay in.
Comment #29
MrHaroldA CreditAttribution: MrHaroldA as a volunteer and commentedThat is absolutely true, but the current code injects 2 spaces instead of one.
$separated_fields[] = "' '";
And
$expression = "CONCAT_WS(' ', $expression)";
This will generate a query containing double spaces, which prevents you from searching across multiple fields, unless the end user also adds double spaces in between.
Comment #30
LendudeIt will create a query with triple spaces actually, because the inserted space will also be concatenated with the space separators.
Which I feel is exactly the point when using the 'contains' operator. You don't want to find any results that only exist because we happen to search across multiple fields using a trick that concatenates the values (even if you did want to enable this kind of multi-field search, it would only work for fields that just happen to be concatenated in sequence, a sequence unknown to any end-user).
When you want to search for multiple terms across multiple fields, you use the 'contains any word'/'contains all words' operators, not the 'contains' operator. Of course that doesn't currently work #2664530: Views Combined fields filter with "Contains any word" or "Contains all words" operator results in an incorrect SQL query and a fatal error, but that's not the point :)
Comment #31
MrHaroldA CreditAttribution: MrHaroldA as a volunteer and commentedIf you store name parts in separate fields (first name, last name, etc), searching on multiple fields in a predefined order is very nice.
For example: "old A" will match my full name ;)
Comment #32
LendudeI'm not saying the end result will always be bad, but like the example in the test
'value' => 'ohn Si'
this will mostly lead to 'what the ??' results for end users. Finding rows where none of the fields actually match the searched for data.If you happen to have a View that puts your first name and current place of residence in sequence and you live in Amsterdam, then your search will also return true, but it's not a result you were actually looking for or expecting.
The current implementation gives clear results, 'the search string needs to be in one field', going with one space would lead to unpredictable results. Your use-case can also be accomplished using the other operators (once we get them to work....., which reminds me, I should look at that other issue again :)
Comment #38
LendudeClosing this now, this works as designed.
Comment #39
amaisano CreditAttribution: amaisano commentedI'm extending this Combine class to work with integers and other fields (like datetime). Having spaces in between each field on a CONCAT_WS fails horribly as the query only checks the first field before an errant space separator.
CONCAT_WS(' ', field_a, field_b) -> works as expected and is a properly formed MYSQL CONCAT_WS function call
CONCAT_WS(' ', field_a, ' ', field_b) -> does not work (even if matches are found in field_b) due to bad function call (presence of blank fields)
I assume however that there must be some magic the empty fields (space fields), performs when searching against string-based field values and that is why this patch was rejected?
Comment #40
MrHaroldA CreditAttribution: MrHaroldA as a volunteer and commentedI'm still in doubt why this got rejected. The three spaces are obviously a bug, but it hides unwanted behavior: matching part of strings from sequential fields.
If the user used three spaces in his search string, the partial match across sequential fields is back.
So the current implementation is wrong, but convenient.
Comment #41
LendudeNope, because multiple sequential spaces get stripped from the search string. It is impossible to search across fields now, the triple space separator is the only way to do this, because it is impossible to get into your search string.
Comment #42
suchdavid CreditAttribution: suchdavid at Integral Vision Ltd commented@Lendude Actually for me the filter gives results with three spaces. I mistakenly used the combined filter like other people to combine first_name last_name field then I found this issue. So tested with three spaces and got result with it. I used it as simple exposed filter.