I'm helping a company migrate from an old ecommerce platform to Ubercart. They're old database has 15,000+ Attributes. Now granted, that's insane, I agree. I'm working with them to consolidate them, because most of them are "Color" or "Colors". But I figured let's give it a try, and pull them all in (with the Migrate API: http://www.drupal.org/project/migrate), in case we need to wait until after the launch to reorganize them.
However, after doing so, I was unable to load the Attributes admin list page (/admin/store/attributes). It would just hang indefinitely.
I took a look at the page callback, uc_attribute_admin(), and was able to isolate the problem. Line 26 of uc_attribute/uc_attribute.admin.inc:
$result = pager_query("SELECT a.aid, a.name, a.label, a.required, a.ordering, a.display, COUNT(ao.oid) AS options FROM {uc_attributes} AS a LEFT JOIN {uc_attribute_options} AS ao ON a.aid = ao.aid GROUP BY a.aid, a.name, a.label, a.ordering, a.required, a.display". tablesort_sql($header), 30, 0, "SELECT COUNT(aid) FROM {uc_attributes}");
What it's doing is querying the database for a list of attributes, and it's using the pager_query() function to return that list in chunks so it can be displayed 30 at a time, with a pager at the bottom for browsing through them. The problem is, it's also trying to squeeze in an Option-count lookup via a LEFT JOIN in the same query, so that it can populate the column on the page that tells you how many Options are in each Attribute. This JOIN is very costly, because it is performed to the entire table BEFORE the results are limited by the pager.
In most circumstances, the time it takes to perform this JOIN is negligible, but if you have 15,000+ products, you're in for a wait.
So, here's a patch for anyone that has way too many Attributes. It basically just moves the Option counting out of the original query, and performs it in the while() loop AFTER the results have been limited down to 30 by pager_query().
Probably makes sense to put this into core (minus the extraneous comments I made) just so all sites can be as scalable as possible.
| Comment | File | Size | Author |
|---|---|---|---|
| #11 | attribute_pager_limit-992904-11.patch | 1.15 KB | m.stenta |
| #4 | attribute_pager_limit-992904-4.patch | 1.25 KB | m.stenta |
| uc_attribute.admin_.inc_.attribute_page.patch | 1.9 KB | m.stenta |
Comments
Comment #1
tr commentedKicking the bot ...
Comment #2
tr commentedComment #4
m.stentaRerolled the patch against the latest 6.x-2.x branch, using the git diff > patch method so it can be automatically tested.
Also removed my unnecessary comments.
Comment #6
tr commented#4: attribute_pager_limit-992904-4.patch queued for re-testing.
Comment #7
longwaveI wonder if the original query is so slow due to #841976: No index on aid field in uc_attribute_options table
Comment #8
m.stentaThat could very well be part of it. Even so, I still think this modification would provide an additional performance benefit. Combining this with an index of the "aid" column would make a HUGE difference.
Again, though, this query currently only affects people who have an outrageous number of attributes. But why not let end-users be outrageous if they want to? It's better than waiting 5 minutes for a page to load.
Comment #9
longwaveLooking quickly at the patch, do we need the GROUP BY clause any more? As we're only selecting one row per attribute, there isn't actually any grouping going on is there?
Comment #10
longwaveAnd if we don't need the GROUP BY we probably don't need the separate count query either.
Comment #11
m.stentaI think you're right. Updated patch attached. (double check to make sure that's what you meant)
It doesn't make a difference without the GROUP BY and count query on my test site, so it looks good to me.
I wish I still had the outrageously large attribute list from the site I'm working on to test this again with. We have since decided to write a script to consolidate them, so now we're down to a reasonable number.
Comment #12
longwaveFixed an issue with the count query and simplified the main query further to remove the table alias. Commit: http://drupalcode.org/project/ubercart.git/commitdiff/fa81bf4
Comment #13
longwavePorted and committed to 7.x: http://drupalcode.org/project/ubercart.git/commitdiff/7718179