Problem/Motivation
Aggregation is not properly working in views. It is not possible currently to have a single field return it's distinct values using aggregates.
Steps to reproduce
- Use Devel Generate and create content for, at least, 2 node-types ('page' and 'article' comming with Standard profile should be fine).
- Add the "Content Type" field to the View. It should be the only field.
- Enable aggregation.
- Set the "Content Type" fileds aggregation settings "Group results together" (is should be the default setting).
We are expecting to see only a distinct list of Content Type names and only for the types that have nodes.
Proposed resolution
Add an additional condition that lets GROUP BY to be added to the query even no aggregation function has been set.
Remaining tasks
- Review
- RTBC
- Commit
User interface changes
None.
API changes
None.
Original report by @david_garcia
Beta phase evaluation
| Issue category | Bug |
|---|---|
| Unfrozen changes | Unfrozen because it only fixes a bug in Views aggregation |
Reporting 2 issues here:
[1] Aggregation is not properly working in views. It is not possible currently to have a single field return it's distinct values using aggregates.
[2] Incompatiblity with SQLServer because having constant values in an aggregate clausule makes no sense (oh yeah MySQL will swallow).
Explained in detail:
- Taking a very simple test case, I want to get a distinct list of all different content types in my system:
0. Create a view for the Node entity.
1. Add the "Content Type" field to the View.
2. Add contextual filter for user ID.
3. Enable agreggation on the query properties.
Result: No GROUP BY is added to the query, results are repeated (as long as you have some content in the system).
After looking through code and debugging, it looks like views is asumming that for a view to have aggregates one of these 2 things must happen:
(a) count($this->having) should be > 0
(b) in compile_fields, at least one of the fields needs to have !empty($field['function'])
\views\modules\field\views_handler_field_field.inc line 1316, method query()
$this->has_aggregate = FALSE;
$non_aggregates = array();
list($non_aggregates) = $this->compile_fields($fields_array, $query);
if (count($this->having)) {
$this->has_aggregate = TRUE;
}The solution is to add an aditional condition:
(c) $this->view->display_handler->options['group_by'] is set to 1
thus, the above code will turn into:
$this->has_aggregate = FALSE;
$non_aggregates = array();
list($non_aggregates) = $this->compile_fields($fields_array, $query);
if (count($this->having)) {
$this->has_aggregate = TRUE;
}elseif($this->has_aggregate == FALSE){
$this->has_aggregate = $this->view->display_handler->options['group_by'];
}A non-related issue I came accross, but is just a few lines below, has to do with the fact that on SQL SERVER you CANNOT USE CONSTANT VALUES IN A GROUP BY CLAUSULE:
http://www.sql-server-helper.com/error-messages/msg-164.aspx
The thrown exception, for google:
Each GROUP BY expression must contain at least one column that is not an outer reference.
Postgre and MySQL will swallow this.
Actually, having a constant value in a GROUP BY makes no sense as it will not alter the results, so I think we can safely prevent any constant column being added to the group by:
Turning this:
\views\modules\field\views_handler_field_field.inc line 1328, method query()
foreach ($groupby as $field) {
$query->groupBy($field);
}
into this:
foreach ($groupby as $field) {
if($fields_array[$field]['table'] == NULL){
continue;
}
$query->groupBy($field);
}
About this change, I am not sure how $fields_array[$field]['table'] behaves on other scenarios such as computed columns.
| Comment | File | Size | Author |
|---|---|---|---|
| #87 | interdiff.txt | 618 bytes | claudiu.cristea |
| #87 | aggregation_not_working-2159347-87.patch | 6.05 KB | claudiu.cristea |
| #63 | aggregation_not_working-2159347-63.patch | 3.08 KB | claudiu.cristea |
Comments
Comment #1
david_garcia commentedComment #2
infojunkiePlease follow these guidelines to make a patch.
Comment #3
david_garcia commentedComment #4
david_garcia commentedComment #6
david_garcia commentedComment #7
xl_cheese commentedI'm currently using the patch in #6 with no issues at velorangutan.com
Comment #8
sachbearbeiter commentedusing the patch with no issues ...
Comment #9
geek-merlinSo we have #6 and #7 as positive indication so i think it's time for maintainers to take a look on this.
Comment #10
vidorado commented#6 works for me
Comment #11
Anonymous (not verified) commentedTesting the functionality in combination with views selective filter (in sandbox) https://drupal.org/sandbox/david_garcia_garcia/2162097. No issues found.
Comment #12
david_garcia commentedREVIEW POSSIBLE ISSUE WHEN HAVING AGGREGATE AND DISTINCT OR PURE DISCTINCT AT THE SAME TIME IN A VIEW. (DOES THAT MAKE SENSE?¿)
Comment #13
antpre commentedWorks for me also.
Comment #14
rooby commentedSorry to do this but any maintainer that see this will say the same thing so better to fix it before they review it.
You won't get a patch committed until it adheres to the drupal coding standards.
In this case, no using tabs for indenting, uniform 2 space indenting, elseif on new line, adding missing spaces in if statements.
Comment #15
david_garcia commented@rooby, very true...
This was one of the first patches I ever made for Drupal, it's been a long way since then.
Reviewd and re-rolled against latest dev.
Comment #16
rooby commentedThanks.
Setting back to previous status seeing as the reason for needs work has been addressed.
Comment #17
roball commentedJust wanted to mention here that committing this patch is required to run the Views Selective Filters module. Any changes to get it committed?
Comment #18
mandreato commentedMigrated from Views Hacks to Views Selective Exposed Filters and needed this patch. +1 RTBC
Comment #19
trumanru commentedI need it so much too!
P.S: Hands off the Andromeda Galaxy!
P.P.S: I can't figure out what else should I write here to get this patch into Views module.
Comment #20
aufumy commentedMaybe that you will buy the maintainers a drink?
Comment #21
roball commentedIncreasing the priority since a lot of users are requesting this feature.
Comment #22
nicolas bouteille commentedI just installed Views Selective Filters module and it is awesome. The module needs this patch which I have applied manually. No errors at all. Since there is very few code in the patch I really hope it'll get committed before next release version! Or my client site will break :(
Comment #23
rooby commented~ Off topic ~
@Nicolas Bouteille:
It is not uncommon to need to use patched versions of modules.
Part of a good module updating procedure is handing patched modules.
If you keep the patch files you use with your site, when it comes to updating you check anything you are going to update for patch files, if there is a patch file you review the issue on drupal.org (all patches should have the node id of the issue in the file name - if not I recommend you add it to the file name when you add it to your site for this reason) and see if the patch has been committed in the new version (or if there is a newer version of the patch).
Then you update then module and reapply the patch if it still hasn't been committed.
For more info on this see http://drupal.stackexchange.com/a/59104/10729
Comment #24
nicolas bouteille commentedI like to auto update modules. I think it works very well. So instead of manually checking for patch files before updating, I prefer blocking the possibility to auto-update and get a warning that tells me that I cannot auto-update but have to do it manually instead and check if patch has been committed before:
So yes indeed my site won't break if the patch is not committed in next release but I was just saying that so that I can be committed faster ;)
Comment #25
spazfoxAfter applying the patch in #15, many of my reports generated by Commerce Reporting are now broken. Specifically, sales reports now show only overall totals and can no longer be broken down to show daily, weekly, monthly, or yearly sales.
Comment #26
dawehnerSuch fundamental patches seriously need a test. Ensure that we also have a test for the other cases where it used to work, sorry.
Comment #28
massiws commentedI migrated from Views Hacks to Views Selective Exposed Filters.
No problem after applying this patch.
Comment #29
david_garcia commentedAdded a test to the patch. Hope it helps to push forward the patch. This is critical, I cannot believe that with views we cannot simply get a distinct list of values over a single column.
Comment #30
david_garcia commented@spazfox: An underlying bug like this, that has been wondering around for such long has, for sure, made other modules to incorrectly implement views.
Comment #31
david_garcia commentedComment #32
asb commentedComing here to get 'Views Selective Filters' working. The patch from #29 applies cleanly:
So far, no side effects encountered. However, I don't see any 'Views Selective Filters' in Views UI.
Comment #33
david_garcia commented@asb: Make sure to clear off all your caches!
Comment #34
asb commented@david_garcia: I did (
drush cc all). So I can not tell if it's 'Views Selective Exposed Filters', or if it's the patch that does not work as advertised.Comment #35
david_garcia commented@asb: The patch has nothing to do with the Views Selective filters showing in the filter list. You should open the issue in that project and add some screen captures to describe what you are seeing and what you are expecting to see.
Comment #36
timwoodPatch in #29 works for me.
Comment #37
alcroito commentedThe patch extracts the group by option directly from the display handler, which is not correct, if the setting was set in the default display, and was not overridden.
To fix that the line
should be replaced with
so it correctly tries to fallback to the option set on the default display as well.
Comment #38
david_garcia commentedTotally true, fixed.
Comment #39
david_garcia commentedPatched for D8.
Comment #41
david_garcia commentedComment #43
david_garcia commentedComment #44
david_garcia commentedAs this is a straight port to D8 and the D7 version is tested on over 600 sites, I think I can safely mark as RTBC.
Hope it can get some attention in D8, then backport. Looks like any Core or Views D7 interventions are freezed but for backports.
Comment #45
catchI'd rather see the comments expanded than the link back to the issue, or both.
typo: orther.
typo: bussines. Could use a more descriptive comment explaining why this needs to be removed.
Trailing whitespace here and elsewhere.
What's a $fieldField?
Also these aren't object properties so should be $field_table etc.
Since we check if $fieldTable == NULL, we could do that first, and only then figure out the field count, no need to do it all up front.
Comment #46
david_garcia commentedLet's try again, I tried to fix all issues.
Comment #47
Anonymous (not verified) commentedFor D7 #38 worked for me without issues.
Using it with Views Selectieve Filters https://www.drupal.org/project/views_selective_filters.
Thanks.
Comment #48
jhedstromStill trailing whitespace.
I think the remainder of the issues from #45 have been addressed though.
Comment #49
david_garcia commentedComment #51
claudiu.cristeaThe patch doesn't apply.
Comment #52
claudiu.cristeaCorrect interdiff.txt here.
Comment #53
claudiu.cristeaComment #54
claudiu.cristeaHere's also the D7 patch but tests are backported from #51.
Again, I removed the SQL Server logic. That should be handled in its own issue node.
Comment #55
david_garcia commented@claudiu.cristea Thank you for the effort. Makes me sad though to see the SQL Server part removed. Because of the nature of that bug and the small SQL Server community around Drupal it is prompt to become one of those one-line-no-brain-melter-never-commited issues.
Comment #56
claudiu.cristea@david_garcia, I can see the problem but with that inside this patch will never pass the core committers. And it's correct that every problem/issue should have its own ticket.
Comment #57
david_garcia commentedDone: #2375407: Views should not use constant values or expressions in aggregates
Will have to repatch, probaly wait to see if this first issue ever gets commited.
Comment #58
claudiu.cristeaBlocks #2375407: Views should not use constant values or expressions in aggregates.
Comment #59
jhedstromCould we add a code comment here?
Comment #60
claudiu.cristeaVoilà!
Comment #61
jhedstrom#60 looks good. This fixes a major bug in Views, and is therefore allowed under the beta phase (I've added this to the issue summary too).
Comment #62
tstoecklerTo have an optional argument being followed by a non optional argument is very strange. Let's remove the default here. It's fine to support passing NULL explicitly, but we do not have to provide a default.
Comment #63
claudiu.cristeaOK. @tstoeckler can you RTBC back?
Comment #64
claudiu.cristeaComment #65
jhedstrom#63 addresses the concerns in #62. Back to RTBC.
Comment #66
tstoecklerThanks for that, looks much better IMO.
Back to RTBC.
Edit: X-post
Comment #67
alexpottThis issue addresses a major bug and is allowed per https://www.drupal.org/core/beta-changes. Committed 23b70df and pushed to 8.0.x. Thanks!
Comment #69
david_garcia commentedThen back to 7.x...
Comment #70
roball commentedGreat this finally found the way into D8 - congrats david_garcia !
Hopefully we will see this in D7, too.
Comment #71
david_garcia commentedComment #72
david_garcia commentedWrong patch in previous post....
Comment #75
david_garcia commentedWrong project, moved back to views.
Comment #78
mallezieRetesting patch
Comment #79
mallezieComment #82
claudiu.cristeaThis has been committed to D8, the backport is trivial. @dawehner?
Comment #83
dawehnerDid someone managed to manually test the patch? Just curious.
Comment #84
claudiu.cristea@dawehner, I'm currently using the patch from #63 (https://www.drupal.org/files/issues/aggregation_not_working-2159347-60-d...) against Views 7.x-3.8 (patched with make file).
I haven't tried the latest rerolls.
Comment #85
claudiu.cristeaOK, this patch is tested against 7.x-3.x-dev. I tested manually and then I ran
ViewsQueryGroupByTesttest locally. All clean.Interdif against #79.
Comment #86
dawehnerDid someone considered to use !$this->has_aggregate?
Comment #87
claudiu.cristea:)
Comment #89
dawehnerAlright, let's get it in.
Comment #90
roball commentedGreat!
Comment #91
david_garcia commentedSpecial thanks to @claudiu.cristea for helping make this happen.
Comment #92
claudiu.cristeaComment #94
roball commentedAs of the release of Views 7.x-3.11, we officially have this feature in Views, so the Views Selective Filters can now be used without relying on a dev snapshot of Views. Good news!