"Select Allowed Layout" option in Panelizer is not respected in IPE. Page Manager defaults are used instead.

I did some more research here and can report the following:

Layouts for the IPE are set in ajax_change_layout() of plugins/display_renderers/panels_renderer_ipe.class.php in the Panels.module. The logic there goes like this:

if (isset($this->display->allowed_layouts)) {
  $layouts = $this->display->allowed_layouts;
}
else {
  $layouts = panels_common_get_allowed_layouts('panels_page');
 }

My guess the problem here is that the $this->display->allowed_layouts's isn't properly getting set in panelizer_panels_cache_get() since a quick look at that logic shows we have explicit handling of the allowed content types, but not the layout.

Here is a patch that likely does the trick. Uses the same pattern as the content types, but adds in the layouts too. It requires you clear your CTools cache to make it work, but I tested it on Panopoly and saw it setting correctly.