Extra info saved to fomatter settings

Problem/Motivation

When you use Dynamic Display formatter on view display with Layout Builder enabled and select Select view modes based on item delta, there are some extra info about buttons, saved into formatter settings:

             formatter:
                label: hidden
                type: dynamic_display
                settings:
                  view_mode: default
                  override: delta
                  bundle_modes:
                    image: default
                  delta_modes:
                    - small
                    - large
                  buttons:
                    add_more: 'Add delta'
                    remove: 'Remove last'
                third_party_settings: {  }

So it is failing validation with Config inspector with missing schema error.