Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Problem/Motivation
Excess load times failed to return all lists.
With a bit of debugging, I noticed a number of things going astray.
#1 They have more than 500 lists, so not all data was returned
This is used everywhere: array('count' => 500)
#2 Timeout issues appeared to be the norm with this many lists.
Locally running, I managed to get the first 500 to eventually load by hard-coding increased timeouts into the code
Digging deeper, each of the 500 lists has two separate calls back to get data from MailChimp
https://___.api.mailchimp.com/3.0/lists once
https://___.api.mailchimp.com/3.0/lists/___/interest-categories 500 times
https://___.api.mailchimp.com/3.0/lists/___/merge-fields 500 times
I didn't benchmark it, but commenting out interest groups appeared to significantly decrease load times, but that could have just been my local connection fluctuating.
#3 High DB load on cache
And as an aside, the blob size for 500 lists is 5.4 MB, a fairly chunky load each time.
Proposed resolution
#1
Followup on #2708603: Maximum limit of 10 lists regarding the limit
#2
A few ideas, none easy :/
- decouple the main lookup and the interest-groups / merge fields, using on-demand lookups instead
- permanent caching with dirty flag to re-check lists on cron
- batch cron lookups
Potentially, the second could be the most promising.
Simple timestamp flag added to the lists in the cache, and loop through doing a smaller number of lookups on the older records on cron.
#3
__links doesn't appear used.
drupal_set_message('Before: ' . strlen(serialize($lists)));
foreach ($lists as $list) {
if (isset($list->_links)) {
unset($list->_links);
}
if (isset($list->mergevars)) {
foreach ($list->mergevars as $mergevar) {
if (isset($mergevar->_links)) {
unset($mergevar->_links);
}
}
}
}
drupal_set_message('After: ' . strlen(serialize($lists)));
Before: 5,376,623
After: 1,285,326
~80% wasted memory here.
Remaining tasks
User interface changes
On demand lookups could require this.
i.e. Sign-up forms, using an AJAX refresh onChange with the select MailChimp Lists values to refresh the merge fields with a lookup if required.
API changes
Unlikely.
Data model changes
Unlikely.
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | drushBulkCreateCommand.patch | 4.58 KB | rjacobsen0 |
| #7 | 2870632-mailchimp-many-lists-6.patch | 2.22 KB | aspilicious |
| #6 | 2870632-mailchimp-many-lists.patch | 1.66 KB | aspilicious |
| |||
Comments
Comment #2
Alan D. CreditAttribution: Alan D. commentedNote that for #3, I had already removed the interest groups, so the actual storage / memory was significantly higher with these, and these probably have a __links property too that could be unset too.
Comment #3
Greg BoggsHi Alan,
Thanks for contributing here! We're interested in expanding the module to support more than 500 lists, and we will discuss it at our next check in as a team. Feel free to keep us posted on your thoughts and progress. If you have a solution that doesn't impact other users, we'll certainly work with you on it.
~Greg
Comment #4
Alan D. CreditAttribution: Alan D. commentedRelated, #2839768: Performance problems with mailchimp_get_lists()
I was indirectly doing this commenting out the interest groups when doing the bench-marking for #3 :)
Comment #5
michiellucas CreditAttribution: michiellucas commentedEvery x time the cache gets cleared we have to wait 2 minutes for the cache to be rebuild.
Why do you need to cache ALL the lists? Why not just cache the lists that the website is using?
Function mailchimp_get_lists -> $result = $mc_lists->getLists(array('count' => 500));
On a mailchimp account with 200 lists this takes to much time.
Comment #6
aspilicious CreditAttribution: aspilicious commentedI made a quick fix for my client. This fixes the issue in the frontend. Only the selected lists will be loaded.
Comment #7
aspilicious CreditAttribution: aspilicious commentedComment #8
rjacobsen0 CreditAttribution: rjacobsen0 at ThinkShout for Mailchimp commentedIn order to test this I wrote a drush 9 command
drush bulk_create:mailchimp_audiences MyAudience 500
It outputs a cleanup command with all the ID's of the lists to be deleted when you're done testing. Here's the patch for it (attached). If you want to use it for testing, you have to put in your name and contact info (around lines 31 - 42 in file modules/contrib/mailchimp/src/Commands/bulkCreateMailchimpLists.php). It won't work without it.
This is not intended to be production code. Use for test only.
Comment #9
rjacobsen0 CreditAttribution: rjacobsen0 at ThinkShout for Mailchimp commentedTo reproduce this problem, if you don't have over 500 lists/audiences apply the patch drushBulkCreateCommand.patch and run 'drush bulk_create:mailchimp_audiences MyAudience 501' (You can change MyAudience to whatever name you want) and wait for lists to be created. Then go to your local site to admin/config/services/mailchimp/lists. You will wait a very long time (about 4 minutes)! But, it does eventually come back and show the list of Audiences including all 501 new audiences. It appears that this fix is no longer needed. Did I reproduce the problem correctly?
I continued testing by applying the patch 2870632-mailchimp-many-lists-6.patch (the latest solution) and clicked "Refresh Audiences from Mailchimp" button. It took about 4 minutes to complete. Then I wanted to try it again without the fix, so I rolled back the patches and re-applied the test patch (in other words the fix code was not present) and clicked "Refresh Audiences from Mailchimp" again. It again it came back after 4 minutes. It looks like the fix isn't needed any more, not even for speed.
Comment #10
rjacobsen0 CreditAttribution: rjacobsen0 at ThinkShout for Mailchimp commentedI did not commit this patch because it has already been fixed in some other way.