Performance problems with mailchimp_get_lists()

This is quite problematic. We have a domain access site with 31 domains, each with their own mailchimp list, but all on the same mailchimp account. There is a signup block in the footer of every page for each domain. Frequently when loading the block it will attempt to re-download all data about all lists including all interest groups and all mergevars, which can take around 5 minutes. This seems to be cleared out quite frequently. If multiple domains try at the same time to load the block we get the following error:

An error occurred requesting list information from MailChimp. "429: Too Many Requests - You have exceeded the limit of 10 simultaneous connections."

It seems it would be best to use a separate cache id per list, eg lists::bb49720d9f. This loads far quicker, but still very slow. It seems like it would be better to use something less volatile than cache tables. Any ideas?

My workaround in a custom module. Note that I am not using any of the sub-modules except for mailchimp_signup.

<?php
/**
 * Implements hook_module_implements_alter().
 */
function MYMODULE_module_implements_alter(&$implementations, $hook) {
  // Don't flush mailchimp cache!
  if ($hook == 'flush_caches') {
    unset($implementations['mailchimp']);
  }
}

/**
 * Implements hook_cron().
 */
function MYMODULE_cron() {
  // Refresh the mailchimp cache daily.
  if ($object = cache_get('lists', 'cache_mailchimp')) {
    if ($object->created < (REQUEST_TIME - 86400)) {
      mailchimp_get_lists(array(), TRUE);
    }
  }
}
?>

I applied the patches from #3 last week and it seems to have improved things on our staging site ... thanks.

Here's a patch that refactors mailchimp_get_list() so it no longer calls mailchimp_get_lists(). It didn't make sense to load all lists and interest groups then throw most of them away.

Loading interest groups is now optional and won't happen if a signup form isn't going to use them. Lists are also cached individually if loaded that way now.

Patches committed! Thank you, @torgosPizza. I'm seeing much better performance in local testing; hopefully this also looks good on your side.

Here is another patch which skips interest categories that are hidden in MailChimp.

This caused significant performance improvements for us, since we have a bunch of admin-only categories that we use for customer acquisition stuff, but which they'll never see, so it doesn't make sense to retrieve them with every mailchimp_get_lists() call.

Crap, this broke my custom module that called mailchimp_get_list() to retrieve all interest categories, including the hidden. See #1808704-30: Ability to put a Member into a MailChimp Group with a Rule Action.

@torgosPizza
I fixed the issue in my custom module now with making an extra API call (see 7a556dd and f126e67). The main reason why I used mailchimp_get_list() first, was because the data got cached. So when there is a taste in whether or not hidden groups should be loaded, you would need two cached versions.

@torgosPizza
I agree that I should make a patch. It would probably need an extra (optional) parameter to the mailchimp_get_list() function that is called $include_hidden_interests_groups.

Changing this back to needs work so the issue in #11 can be fixed.

This is happening on 8.x too!

I also saw #2870632: Large lists crash the cache rebuild & miss accounts with more than 500 lists which has interesting proposals. I think we should move all discussions to one issue and plan/decide on a fix and then work on implementing it. I think making some of what mailchimp_get_lists() does configurable could help. Not all sites need interest groups and not all sites need merge vars. That could help initially.

Then moving some/all of this logic to the cron should help. I think the other issue have good suggestion, so maybe close this one in favor of the other one?

I needed what was brought up in #15. I want to see the hidden interests.

Honestly not sure why we even hide them on this side of the fence as it seems like more of a front end issue then a backend one.

I also separated out the interest calls into there own functions and cache them. Not really what needed to happen but it seems cleaner it also gives better control to use the module calls to separate an ajax based form, calling them only when we need to. I think caching them will increase cache calls but in the end will likely improve performance. I use redis cache on most my installs so it may affect mysql differently.

Let me know.
-Jesse

After a day of debugging, I found the root cause of my marketing workflow to stop working for 9 months and it is the commits in this issue. You must provide a way to access the hidden interest groups. I use hidden interest groups as a way to trigger funnel marketing automation in Mailchimp. The groups are my funnels.

Updating the applicability of this.

I marked this issue as critical because the changes here were breaking changes.

I re-rolled the patch and made a PR on Github: https://github.com/thinkshout/mailchimp/pull/239

There is a bug in the patch of #18. I fixed it in my PR.

I have this patch running in my production site and I ran through a quick positive and negative test.

Rerolled the reroll from @dobe (comment #18). Changed the order of parameters so that the new parameter is the last one and it must have a default value (which it did). If these are not true then it is a breaking change.

Trying again. Patch file was empty.

A fix for the failing tests (works on my local).

Committing for the next release.