Mini Panels loading all displays on every page load

I was doing a performance audit on a website and noticed that panels_load_displays() was being called for a bunch of dids that were not relevant to the pages in question (see attached screenshot). As I traversed the code, I noticed that panels_mini_load_all() was getting loaded on every page load because it was called in panels_mini_block_list_alter().

This line inside panels_mini_load_all() is where every single did gets loaded: http://cgit.drupalcode.org/panels/tree/panels_mini/panels_mini.module#n347

$displays = panels_load_displays(array_keys($dids));

But I'm wondering if loading panels_mini_load_all() inside of panels_mini_block_list_alter() makes sense to begin with:

http://cgit.drupalcode.org/panels/tree/panels_mini/panels_mini.module#n152

function panels_mini_block_list_alter(&$blocks) {
  if (module_exists('page_manager')) {
    $current_page = page_manager_get_current_page();
  }

  // Load add at once to save time.
  panels_mini_load_all();

  foreach ($blocks as $key => $block) {
    if ($block->module != 'panels_mini') {
      // This block was added by a contrib module, leave it in the list.
      continue;
    }

    $panel_mini = panels_mini_load($block->delta);

I'd think that $panel_mini = panels_mini_load($block->delta); being loaded only when necessary, makes more sense from a performance perspective than preloading all mini panels on every page load, but seeing that this code was intentionally added to "save time", I figured I'd leave it up for discussion instead of just creating a patch for now.