Panelizer's permissions don't play well with IPE permissions

In the latest version of Panelizer, there's a mismatch in permissions between what Panelizer provides and what the IPE provides. Some access configurations can cause the IPE buttons to appear but not work (they generate the "You are not authorized to access this page" alert). I think this is because Panelizer's implementation of hook_panels_ipe_access is too broad. If a certain role has edit access to a content type, but the Panelizer permissions that are required to use the IPE on it are not granted, the buttons will appear when they shouldn't.

Versions in use:
CTools 7.x-1.11+0-dev
Panels 7.x-3.8+0-dev
Panelizer 7.x-3.4+15-dev

To reproduce:

Setup

1. Install latest Drupal core.
2. Download and install the latest dev versions of CTools, Panels, and Panelizer, including Page Manager and the Panels IPE.
3. Log in as an administrator.
4. Enable "node_view" page in Page Manager.
5. For both the Basic page and Article content types, enable Panelizer.
6. Check these boxes:
   • Panelize
   • Full page override
   • Provide an initial display named Default
7. In the Panelizer settings page for both Article and Basic page, set the full page override renderer to In-place-editor.
8. Create an Article node.
9. Create a Basic page node.

Roles and permissions

1. Grant all permissions to the administrator role
2. Create a role called "editor" and grant it some permissions:
   • Node: Grant all node creating, editing, and deleting permissions for Articles and Basic pages to the Editor role.
   • Panels: Grant Use the Panels IPE and Change layouts with the Panels IPE to the Editor role.
   • Panelizer: Grant the Administer Panelizer layout and content permissions for Article nodes to the Editor role. Do not grant panelizer permissions to Basic pages.
3. Create a user with the editor role.

Demonstrate the issue

1. Log in as the editor.
2. Visit the Article node.
3. Note the IPE buttons appear at the bottom of the screen.
4. Click the Customize this page button.
5. Note the IPE interface displays.
6. Click the Cancel button.
7. Visit the Basic page node.
8. Note the IPE buttons appear at the bottom of the screen. Expected behavior: Because editors are not granted the Panelizer Administer Content or Layout permissions, the buttons should not appear.
9. Click the Customize this page button.
10. Note you see the JS alert "You are not authorized to access this page."

The code in question is here:

/**
 * Implements hook_panels_ipe_access().
 */
function panelizer_panels_ipe_access($display) {
  // We only care about Panels displays from panelizer.
  if (isset($display->context['panelizer'])) {
    // The type array contains 3 elements, where the first is the full context
    // type (ie. 'entity:ENTITY_TYPE'), and the remaining two are the parts
    // seperated by ':', so 'entity' and the entity type name.
    $entity_type = $display->context['panelizer']->type[2];
    if ($handler = panelizer_entity_plugin_get_handler($entity_type)) {
      // Only allow access to use the IPE if the user has 'update' access to
      // the underlying entity.
      $entity = $display->context['panelizer']->data;
      return $handler->entity_access('update', $entity);
    }
  }
}

What this does is allow or deny access to the IPE strictly based on whether the current user has edit access to the node. If the current user has edit access but not the Administer Panelizer Content and/or Layout permissions, this hook causes the buttons to be displayed. But the Panelizer permissions kick in when you click the buttons, thus you get MENU_ACCESS_DENIED as a result of the page callback and you see the alert.

Instead, the hook should probably return FALSE if you don't have edit access, and NULL in all other cases.