IPE Enabled Changes CSS Used?

Running into an issue when the IPE is enabled the CSS looks different from when the IPE is not enabled for non-logged in users. Looking through the issue tracker I did not find any previous solutions. Thought of hiding the IPE with CSS even though it is enabled but that would be a cheap fix. Anyone else run into this?