JavaScript should only be rendering on the page if the user has access to use panels_ipe

At the moment there are no less than 18 JS file includes from this module for anonymours users (which generally will never have the power to use panels_ipe). Drupal 8 made a lot of strides to remove JS from anonymours users, and it seems panels_ipe is adding it all back it.

<script src="/modules/panels/panels_ipe/js/panels_ipe.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/models/AppModel.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/models/BlockContentTypeModel.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/models/BlockModel.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/models/BlockPluginModel.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/models/RegionModel.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/models/TabModel.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/models/LayoutModel.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/views/AppView.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/views/BlockView.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/views/CategoryView.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/views/BlockPicker.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/views/BlockContentPicker.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/views/LayoutPicker.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/views/LayoutView.js?v=8.1.0"></script>
<script src="/modules/panels/panels_ipe/js/views/TabsView.js?v=8.1.0"></script>
<script src="/modules/panelizer/js/panels_ipe/panels_ipe.js?v=8.1.0"></script>
<script src="/modules/panelizer/js/panels_ipe/views/SaveTabView.js?v=8.1.0"></script>

Can we omit these files for users with no access?