Problem/Motivation

When working on aggregated Views for Commerce Reports, I noticed some weird behaviors with our Price field. Output would be fine when the normal "group", but once I changed to SUM or AVG it would render as a numeric/integer field and lose all formatting. I did some digging and found out the Display plugin overrides the handler based on information in \Drupal\views\Plugin\views\query\Sql::getAggregationInfo.

The problem is that \Drupal\views\Plugin\views\field\FieldPluginBase::addAdditionalFields applies that grouping to each additional field. In the case of an entity field that means it tries to SUM/AVG the bundle, entity ID, or any other field columns. In our case, it kills the currency code.

Upon reviewing the code I see:


    $group_params = [];
    if ($this->options['group_type'] != 'group') {
      $group_params = [
        'function' => $this->options['group_type'],
      ];
    }
    if (!empty($fields) && is_array($fields)) {
....
          $params += $group_params;
          $this->aliases[$identifier] = $this->query->addField($table_alias, $info['field'], NULL, $params);

As you can see in this screenshot, the langcode, bundle, and currency_code become "0" since it performs SUM() on a string.

Proposed resolution

Views should not apply the group/aggregate function on all additional fields.

I don't know if this is considered BC or not. None of the additional fields are rendered or ever managed (or ever had valid data.)

Remaining tasks

Create patch
Discuss if BC
Hopefully fix?

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#3 2975149-3.patch1.24 KBmglaman
#2 aggregate-bundle-sum.png123 KBmglaman

Issue fork drupal-2975149

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mglaman created an issue. See original summary.

mglaman’s picture

Issue summary: View changes
StatusFileSize
new123 KB

Adding screenshot.

mglaman’s picture

Status: Active » Needs review
StatusFileSize
new1.24 KB

Proposed fix.

lendude’s picture

As far as I know, all this grouping is done to make valid SQL. MySQL doesn't care, but apparently PostGres does. So I queued for postgres too, to see if that currently breaks anything there.

Since the fields get added to the query, SQL wants a grouping, so if I'm reading this right, for any additional fields we currently assume the same grouping as the base field. It's a guess, sure, but it's a fair guess.

Again, if I'm reading this right, the current proposed fix completely removes the grouping, which, I think, would lead to an invalid SQL query.

mglaman’s picture

Lendude, thanks for the reply.

This was the output of our SQL query from Views. Hopefully PostGres passes

SELECT commerce_order_report__mail.delta AS commerce_order_report__mail_delta, commerce_order_report__mail.langcode AS commerce_order_report__mail_langcode, commerce_order_report__mail.bundle AS commerce_order_report__mail_bundle, commerce_order_report__mail.mail_value AS commerce_order_report__mail_mail_value_1, commerce_order_report__amount.delta AS commerce_order_report__amount_delta, commerce_order_report__amount.langcode AS commerce_order_report__amount_langcode, commerce_order_report__amount.bundle AS commerce_order_report__amount_bundle, commerce_order_report__amount.amount_number AS commerce_order_report__amount_amount_number_1, commerce_order_report__amount.amount_currency_code AS commerce_order_report__amount_amount_currency_code, DATE_FORMAT(FROM_UNIXTIME(commerce_order_report.created), '%Y-%m') AS commerce_order_report_created, COUNT(commerce_order_report.order_id) AS commerce_order_report_order_id, COUNT(DISTINCT commerce_order_report__mail.mail_value) AS commerce_order_report__mail_mail_value, SUM(commerce_order_report__amount.amount_number) AS commerce_order_report__amount_amount_number, DATE_FORMAT((DATE_ADD('19700101', INTERVAL commerce_order_report.created SECOND) + INTERVAL -21600 SECOND), '%Y%m') AS commerce_order_report_created_month, MIN(commerce_order_report.report_id) AS report_id
FROM 
{commerce_order_report} commerce_order_report
LEFT JOIN {commerce_order_report__mail} commerce_order_report__mail ON commerce_order_report.report_id = commerce_order_report__mail.entity_id AND commerce_order_report__mail.deleted = '0'
LEFT JOIN {commerce_order_report__amount} commerce_order_report__amount ON commerce_order_report.report_id = commerce_order_report__amount.entity_id AND commerce_order_report__amount.deleted = '0'
WHERE commerce_order_report.type IN ('order_report')
GROUP BY commerce_order_report_created, commerce_order_report__mail_delta, commerce_order_report__mail_langcode, commerce_order_report__mail_bundle, commerce_order_report__mail_mail_value_1, commerce_order_report__amount_delta, commerce_order_report__amount_langcode, commerce_order_report__amount_bundle, commerce_order_report__amount_amount_number_1, commerce_order_report__amount_amount_currency_code, commerce_order_report_created_month
ORDER BY commerce_order_report_created_month DESC
LIMIT 11 OFFSET 0
lendude’s picture

GROUP BY commerce_order_report_created, commerce_order_report__mail_delta, commerce_order_report__mail_langcode, commerce_order_report__mail_bundle, commerce_order_report__mail_mail_value_1, commerce_order_report__amount_delta, commerce_order_report__amount_langcode, commerce_order_report__amount_bundle, commerce_order_report__amount_amount_number_1, commerce_order_report__amount_amount_currency_code, commerce_order_report_created_month
Ah good, so the additional fields still get added to the 'group by' clause, which was my main concern.

So now it comes down to a lack of options to steer the grouping type, I guess. There is no way, currently, to specify the type of grouping for 'additional' fields. So in the current situation, the guess is that it should be the same as the base field and with the fix the guess is that it should not use a grouping type.

A guess is still a guess, and we can (nay, we will) guess wrong.

None of the additional fields are rendered or ever managed (or ever had valid data.)

But they are available as tokens right? So they can be printed? If that is so, I think we do need to consider BC.

The only BC way forward that springs to mind is adding an option to the grouping settings that allow you to steer this: 'Use grouping type on additional fields', or something along those lines. Any other ideas?

mglaman’s picture

But they are available as tokens right? So they can be printed? If that is so, I think we do need to consider BC.

I am not sure, I could check.

The only BC way forward that springs to mind is adding an option to the grouping settings that allow you to steer this: 'Use grouping type on additional fields', or something along those lines. Any other ideas?

I suppose "Use grouping type on additional fields" as a default makes sense. Then it's just a document step in Commerce Reports "select SUM or AVG, and uncheck this checkbox"

I'd love to have the same options from the EntityField handler, but Views overrides it to be a generic Numeric handler. Where you can specify additional fields to include (whereas with Numeric I think they're all automatically included?)

This is the only "quick" fix I can think of since there is no way to control the field handler when it is aggregated, from an implementation standpoint.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

smustgrave’s picture

Status: Needs review » Needs work
Issue tags: +Needs Review Queue Initiative, +Needs tests

This issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue [2700+ issues] to around 400 (1 month or less), following Review a patch or merge request as a guide.

Think next step would be adding a test case to show this issue.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

pebosi’s picture

Could you add a re-rolled patch for 10.1 and 10.2 ?

ankithashetty made their first commit to this issue’s fork.

ankithashetty’s picture

Patch in #3 still gets applied on 10.1, 10.2 and 11.x.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.