I'm helping a company migrate from an old ecommerce platform to Ubercart. They're old database has 15,000+ Attributes. Now granted, that's insane, I agree. I'm working with them to consolidate them, because most of them are "Color" or "Colors". But I figured let's give it a try, and pull them all in (with the Migrate API: http://www.drupal.org/project/migrate), in case we need to wait until after the launch to reorganize them.

However, after doing so, I was unable to load the Attributes admin list page (/admin/store/attributes). It would just hang indefinitely.

I took a look at the page callback, uc_attribute_admin(), and was able to isolate the problem. Line 26 of uc_attribute/uc_attribute.admin.inc:

  $result = pager_query("SELECT a.aid, a.name, a.label, a.required, a.ordering, a.display, COUNT(ao.oid) AS options FROM {uc_attributes} AS a LEFT JOIN {uc_attribute_options} AS ao ON a.aid = ao.aid GROUP BY a.aid, a.name, a.label, a.ordering, a.required, a.display". tablesort_sql($header), 30, 0, "SELECT COUNT(aid) FROM {uc_attributes}");

What it's doing is querying the database for a list of attributes, and it's using the pager_query() function to return that list in chunks so it can be displayed 30 at a time, with a pager at the bottom for browsing through them. The problem is, it's also trying to squeeze in an Option-count lookup via a LEFT JOIN in the same query, so that it can populate the column on the page that tells you how many Options are in each Attribute. This JOIN is very costly, because it is performed to the entire table BEFORE the results are limited by the pager.

In most circumstances, the time it takes to perform this JOIN is negligible, but if you have 15,000+ products, you're in for a wait.

So, here's a patch for anyone that has way too many Attributes. It basically just moves the Option counting out of the original query, and performs it in the while() loop AFTER the results have been limited down to 30 by pager_query().

Probably makes sense to put this into core (minus the extraneous comments I made) just so all sites can be as scalable as possible.

Comments

tr’s picture

Version: 6.x-2.4 » 6.x-2.x-dev
Status: Needs review » Needs work

Kicking the bot ...

tr’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, uc_attribute.admin_.inc_.attribute_page.patch, failed testing.

m.stenta’s picture

Status: Needs work » Needs review
StatusFileSize
new1.25 KB

Rerolled the patch against the latest 6.x-2.x branch, using the git diff > patch method so it can be automatically tested.

Also removed my unnecessary comments.

Status: Needs review » Needs work

The last submitted patch, attribute_pager_limit-992904-4.patch, failed testing.

tr’s picture

Status: Needs work » Needs review
longwave’s picture

I wonder if the original query is so slow due to #841976: No index on aid field in uc_attribute_options table

m.stenta’s picture

That could very well be part of it. Even so, I still think this modification would provide an additional performance benefit. Combining this with an index of the "aid" column would make a HUGE difference.

Again, though, this query currently only affects people who have an outrageous number of attributes. But why not let end-users be outrageous if they want to? It's better than waiting 5 minutes for a page to load.

longwave’s picture

Looking quickly at the patch, do we need the GROUP BY clause any more? As we're only selecting one row per attribute, there isn't actually any grouping going on is there?

longwave’s picture

And if we don't need the GROUP BY we probably don't need the separate count query either.

m.stenta’s picture

StatusFileSize
new1.15 KB

I think you're right. Updated patch attached. (double check to make sure that's what you meant)

It doesn't make a difference without the GROUP BY and count query on my test site, so it looks good to me.

I wish I still had the outrageously large attribute list from the site I'm working on to test this again with. We have since decided to write a script to consolidate them, so now we're down to a reasonable number.

longwave’s picture

Version: 6.x-2.x-dev » 7.x-3.x-dev
Status: Needs review » Patch (to be ported)

Fixed an issue with the count query and simplified the main query further to remove the table alias. Commit: http://drupalcode.org/project/ubercart.git/commitdiff/fa81bf4

longwave’s picture

Status: Patch (to be ported) » Fixed

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.