Config import does not work for custom view modes if bundles not selected

I've created three media entity bundles, and a custom view mode for media entities called "wysiwyg_embed." I've enabled and configured this view mode for all three bundles. I can create an embed button, and it works just fine.

When selecting bundles, the field description says: "If none are selected, all are allowed." And sure enough, this works as expected as well.

However, if I export the bundles, their fields, the view mode, and the embed button config files, and try to use them in a custom install profile, I get the following error:

Drupal\Component\Plugin\Exception\PluginNotFoundException: The "view_mode:media.wysiwyg_embed" plugin does not exist. in Drupal\Core\Plugin\DefaultPluginManager->doGetDefinition() (line 52 of core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php).

With some messing around, I discovered that the problem is solved if I select all three bundles. In other words, this export file fails:

uuid: 185f9187-e9ba-4d0e-9e68-3e06f561d7d4
langcode: en
status: true
dependencies:
  module:
    - entity_embed
    - media_entity
_core:
  default_config_hash: RyQcyZJRZRUMiVwHmk5MwiyiKPy5aOlq7yQ9hcMdoCY
label: Media
id: media
type_id: entity
type_settings:
  entity_type: media
  bundles: {  }
  display_plugins:
    - 'entity_reference:entity_reference_label'
    - 'entity_reference:media_thumbnail'
    - 'view_mode:media.wysiwyg_embed'
  entity_browser: default
  entity_browser_settings:
    display_review: false
icon_uuid: null

...while this export file succeeds:

uuid: 185f9187-e9ba-4d0e-9e68-3e06f561d7d4
langcode: en
status: true
dependencies:
  config:
    - media_entity.bundle.document
    - media_entity.bundle.image
    - media_entity.bundle.video
  module:
    - entity_embed
    - media_entity
_core:
  default_config_hash: RyQcyZJRZRUMiVwHmk5MwiyiKPy5aOlq7yQ9hcMdoCY
label: Media
id: media
type_id: entity
type_settings:
  entity_type: media
  bundles:
    - document
    - image
    - video
  display_plugins:
    - 'entity_reference:entity_reference_label'
    - 'entity_reference:media_thumbnail'
    - 'view_mode:media.wysiwyg_embed'
  entity_browser: default
  entity_browser_settings:
    display_review: false
icon_uuid: null

I'm not sure what the appropriate fix here is, but I know the field description kind of led me astray from a working config, as I expected that leaving all the bundles unchecked should produce the same result as having them all checked.