Uninstalling module wipes out unrelated display configs.

Thank you for this module. It worked very well for me. I've written a patch to allow the option of "Render as SPAN with css background mask" which is super handy since you can then manipulate those via css easily. But upon uninstalling the module I found this hairy little issue. To reproduce:

1. Install the svg_image module
2. Navigate to the "manage display" UI of any content type with image fields.
3. Make and save several display changes like hide/show field labels and enable/disable entire fields.
4. Uninstall the svg_image module.
5. Navigate to the same "manage display" UI to review for lost display settings.

I believe this is has something to do with the module overriding core directly...making the uninstall routine become overzealous for some reason. I was able to get around losing work by doing the following juggling with my file based config backups. Prior to svg_config uninstall I...

1. Exported my configurations to file and make a copy of any config file named: core.entity_view_display.node.[BUNDLE].[DISPLAY-MODE].yml
2. From those copies, manually remove svg related config line from them including the - svg_image dependency under module: and each image field svg related configs like:
   

         svg_attributes:
           width: ''
           height: ''
         svg_render_as_image: true
   

3. With these modified config YML file backups off to the side, uninstall the SVG module.
4. Export what will now be the clobbered configs to file: drush cex
5. Now replace the clobbered configs with the manually edited copies that were placed off to the side.
6. Reimport the manually edited configs: drush cim