Hi guys!
Here is my use case:
I have a textfield in my content type, which is called field_group. I've created a view_1 with exposed filter for that field (operator "contains") and rewriting output of that field as a link to view_2 with contextual filtering by field_group.
I need to remove from filter results all duplicates of field_group value.

Here's an example:
We have 10 nodes with field_group value "ws589e3", 10 nodes with value "ws473u4" and 10 nodes with value "as589s4"
User enters "ws" in exposed filter and gets 2 nodes (1 of "ws589e3" and 1 of "ws473u4").
Then he clicks on field_group value and gets redirected to other view where he see all nodes with the same value.

Any solutions are welcome, thx in advance!

Files: 
CommentFileSizeAuthor
#17 1677692-argument.patch2.39 KBsqeph
#13 1677692-13.patch2.3 KBdamiankloip
Test request sent.
Previous result: PASSED: [[SimpleTest]]: [MySQL] 1,555 pass(es).
[ View ]
#12 1677692-12.patch2.31 KBdamiankloip
PASSED: [[SimpleTest]]: [MySQL] 1,555 pass(es).
[ View ]
#11 1677692-11.patch1.77 KBdamiankloip
PASSED: [[SimpleTest]]: [MySQL] 1,555 pass(es).
[ View ]
#5 views-search-duplicates-1677692-5.patch723 byteskid_icarus
PASSED: [[SimpleTest]]: [MySQL] 1,555 pass(es).
[ View ]
#2 view--test.txt3.4 KBptrl
#2 Screen Shot 2012-10-05 at 11.02.07.png117.08 KBptrl
#2 Screen Shot 2012-10-05 at 11.02.27.png24.38 KBptrl
#2 Screen Shot 2012-10-05 at 11.02.36.png82.02 KBptrl

Comments

chebureque’s picture

any1?

ptrl’s picture

Version:7.x-3.3» 7.x-3.5
Category:support» bug
StatusFileSize
new82.02 KB
new24.38 KB
new117.08 KB
new3.4 KB

Think I have the same problem. Clean install with Views 3.5 (and latest dev).

Have a basic page with the title "Welcome to…". And create a simple view that list all content (only have that one node) that shows node title, sort by date and in the filters i have Search Terms and exposes that to the user.

When I filter on "Welcome" in the exposed search term field the welcome node is shown. When I search for "Welcome to" the node gets duplicated.

Welcome-query:

SELECT node.title AS node_title, node.nid AS nid, node.created AS node_created, SUM(search_index.score * search_total.count) AS score
FROM
{node} node
LEFT JOIN {search_index} search_index ON node.nid = search_index.sid
LEFT JOIN {search_total} search_total ON search_index.word = search_total.word
WHERE (( (node.status = '1') AND( (search_index.type = 'node') AND( (search_index.word = 'welcome') ))))
GROUP BY search_index.sid, score, node_title, nid, node_created
HAVING (( (COUNT(*) >= '1') ))
ORDER BY node_created DESC
LIMIT 10 OFFSET 0

Welcome to-query:

SELECT node.title AS node_title, node.nid AS nid, node.created AS node_created, SUM(search_index.score * search_total.count) AS score
FROM
{node} node
LEFT JOIN {search_index} search_index ON node.nid = search_index.sid
LEFT JOIN {search_total} search_total ON search_index.word = search_total.word
INNER JOIN {search_dataset} search_dataset ON search_index.sid = search_dataset.sid AND (search_index.type = search_dataset.type)
WHERE (( (node.status = '1') AND( (search_index.type = 'node') AND (search_dataset.data LIKE '% welcome %' ESCAPE '\\') AND (search_dataset.data LIKE '% to %' ESCAPE '\\') )))
GROUP BY search_index.sid, score, node_title, nid, node_created
HAVING (( (COUNT(*) >= '1') ))
ORDER BY node_created DESC
LIMIT 10 OFFSET 0
chebureque’s picture

Status:Active» Needs review
<?php
/**
 * Implements hook_views_pre_execute
 *
 * Alter query, before its's been executed
 */
function custom_module_views_pre_execute(&$view) {
  if (
$view->name == '<your_view_machine_name>') {
   
$view->build_info['query']->groupBy('field_data_<your_field_machine_name>.<your_field_machine_name>_value');
  }
}
?>

Create a custom module, insert this code (just remember to set your field and view machine names, e.g. field_group & search_page respectively, in my case) and turn it on. Works like a charm for me!

ptrl’s picture

Status:Needs review» Active

Doesn't seem to help me. Looking into the query I see that is groups by "GROUP BY search_index.sid, score, node_title, nid, node_created". Problem with GROUP BY score is that the score isn't the same for the keyword and that created duplicated content. When I manually remove it I get the expected results, no duplicated content. Is there anyway to remove score from getting added in the first place?

kid_icarus’s picture

StatusFileSize
new723 bytes
PASSED: [[SimpleTest]]: [MySQL] 1,555 pass(es).
[ View ]

So this fixed it for me, but I'm sure this line of code is in there for some reason... Essentially I'm removing the score HAVING (( (COUNT(*) >= :search_index_keys) ))) from the GROUP BY clause. Feedback greatly appreciated, hopefully this issue can move forward a bit :)

kid_icarus’s picture

Status:Active» Needs review
kid_icarus’s picture

Title:Remove duplicates from exposed filter results» Remove duplicates from exposed search filter results
ptrl’s picture

Status:Needs review» Needs work

Awesome!

This fixes duplicate content for me, but breaks the pager. I still get the pager even thou the result is only one node.

ptrl’s picture

Notice more errors with removing that line. Certain content doesn't show up at all.

ptrl’s picture

Status:Needs work» Active

Maybe I'm on to something.

I removed words from the search that where shorter than "Minimum word length to index" and no more duplicate content. At least what I have notice for the search terms I have tried.

Not sure how to alter the search query so that it removes short words, so for now I'll have add a validation function to the forms that exposes the search key field.

The validation function looks like this:

<?php
  $keys
= explode(" ", $form_state['values']['keys']);
  foreach (
$keys as $id => $key) {
    if (
strlen($key) < variable_get('minimum_word_size', 0)) {
      unset(
$keys[$id]);
    }
  }
 
$form_state['values']['keys'] = implode(' ' , $keys);
?>

A problem with this that I see is that it will also filter keys like OR. You could do a simple case sensitive filter but i'm sure this should be fixes in the views search somewhere.

damiankloip’s picture

Status:Active» Needs review
StatusFileSize
new1.77 KB
PASSED: [[SimpleTest]]: [MySQL] 1,555 pass(es).
[ View ]

Me and dawehner were looking into this issue today for a bit, as we were seeing similar problems. We think this is to do with the search score being added to the query all the time.

damiankloip’s picture

StatusFileSize
new2.31 KB
PASSED: [[SimpleTest]]: [MySQL] 1,555 pass(es).
[ View ]

Maybe this? Added the join to the conditional and some (,maybe) better description text...

damiankloip’s picture

StatusFileSize
new2.3 KB
Test request sent.
Previous result: PASSED: [[SimpleTest]]: [MySQL] 1,555 pass(es).
[ View ]

Proper comment on options_form.

dawehner’s picture

Project:Views» Drupal core
Version:7.x-3.5» 8.x-dev
Component:exposed filters» search.module
Status:Needs review» Active

Perfect. Let's commit it and forward-port later.

cbrasfield’s picture

I had this same exact issue. The search score unfortunately didn't fix it when I searched directly from the search_block_form, but whenever I would remove the unindexed words (such as of), I would get the correct results. I override the search submit and noticed that the keys were not being passed correctly. I updated the function theme_form_submit to include a redirect with the keys:
$form_state['redirect'] = array('search/results', array('query'=>array('keys'=>trim($form_state['values'][$form_id]))));

jimik42’s picture

Version:8.x-dev» 7.21
Component:search.module» views.module
Category:bug» support

Code looks great and I can see it is included in the latest version of views, however I have one question.

Where do you find the checkbox that is used to 'Remove search score'

Can't see it anywhere.

Thank you

sqeph’s picture

Issue summary:View changes
StatusFileSize
new2.39 KB

D7/Views: #13 solved the problem for me as well, only I needed the 'Remove search score' option for the argument handler not the filter.

dcam’s picture

Version:7.21» 8.x-dev
Component:views.module» search.module
Category:Support request» Bug report

Resetting the issue categories so this will get forward-ported to 8.x per #14.

jhodgdon’s picture

Can someone please write up an issue summary, so we know what needs to be ported? And which patch was committed to what module?

dawehner’s picture

jhodgdon’s picture

Thanks! We'll have to see whether this applies to Drupal 8.x. Is there a test for this at all? Under what conditions does having search_score produce duplicates? Really could use an issue summary explaining the conditions that makes this happen....

And... there are a couple of typographical errors in this patch. Like:

+      '#description' => t('Check this box to remove the search score from the query. This can help reduce help reduce duplicate search results when using this filter.'),

... help reduce help reduce duplicate...

+        // Create a new join to relate the 'serach_total' table to our current 'search_index' table.

typo in serach_total

jhodgdon’s picture

Project:Drupal core» Views
Version:8.x-dev» 7.x-3.x-dev
Component:search.module» Code
Status:Active» Fixed

OK... So I went up to the first few comments in this issue, and it looks like the issue that was solved by this (hack?) patch was not anything to do with the issue summary, but instead comment #2. Which is that if you use the Search Keywords filter, and type in two keywords, you get duplicates in your results.

Assuming that is the problem... I just tested this in Drupal 8 on simplytest.me, and I am unable to duplicate the problem. So I think I am going to set this back to being a Views 7.x issue. I think we fixed the problem in the core Search module in 8 a few months ago.

If I have misunderstood the problem, or if someone can verify that it is still a problem in 8, please please please make an issue summary that describes the problem so we can address it in Drupal 8.

rahullamkhade queued 13: 1677692-13.patch for re-testing.

Status:Fixed» Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

aasarava’s picture

Sorry to reopen this issue, but I believe it's still a problem in D7. The solution that was committed (patch from #13 above) has the unfortunate side effect of not allowing sorting by search score, since it removes the join on the search score field. Typically you want search results to be sorted by score. So the patch is only a limited fix.

I suspect the problem has to do with the way Drupal 7 constructs its GROUP BY statements using all the fields. If the GROUP BY were only limited to search_score.sid, then it's possible the results would not have duplicates.

I've come up with the following workaround: First, edit the query settings for any views that use the search term field and add "search_fix" as a query tag. Then add the following code to a custom module:

function MYMODULE_query_alter($query) {
  if ($query->hasTag('search_fix')) {
    $groups =& $query->getGroupBy();
    $having =& $query->havingConditions();

    // Views adds all fields to the GroupBy clause when
    // the search filter handler tries to add search_index.sid
    // to the GroupBy clause. We don't want all the fields,
    // so we unset them, and the Having clause, and then re-add
    // search_index.sid again.
    if (isset($groups['search_index.sid'])) {
      foreach ($groups as $key => $val) {
        unset($groups[$key]);
      }
      unset($having[0]);

      $query->groupBy("search_index.sid");
    }
  }
}

Note that this unsets all the GroupBys and Having clauses on the query, so if you set any others (beyond what the search filter handler sets), then your query may get messed up.

Also, I'm not sure if the resulting SQL works for Postgres. If I understand correctly, the whole reason that Views adds all the fields to the GroupBy clause in the first place is to remain compatible with an old Postgres requirement. Is that still valid? If not, then there might be a way to turn the code above into a patch for Views.

aasarava’s picture

Status:Closed (fixed)» Needs work
Barryvdh’s picture

I'm still having this issue. Where can I set this option?

I'm searching for 'IT manager' for example, but that gives me 5 times the same result. Preferably, I'd still like to find matches for 'it'..

jhodgdon’s picture

A somewhat related issue is #1615438: Search terms: nodes dont appear when using multiple arguments.

On #1615438-30: Search terms: nodes dont appear when using multiple arguments (comment #30 on that issue), there is a patch that fixes both that other issue, and at least in my testing, this issue as well. Please test it out.