Permission problem with Paragraphs

I use config pages with Paragraphs (and even Paragraphs inside Paragraphs).
I recently added restriction to roles to only be able to edit few config pages.
And unless giving them the 'edit config_pages entity' permission, they couldn't see the content of the Paragraphs.
I have few Patches with Paragraphs so it might be because of that, but it seems that Paragraph checks the Parent access which is a config_pages in this case and not a config_pages_type like in '/admin/structure/config_pages/types' for example, so the "'edit ' . $entity->id() . ' config page entity'" looks like 'edit 4 config page entity' in the end.