Hide exposed sort when view is empty

I have a view with a few exposed sorting criteria. Everything works ok for logged in users.

However, anonymous users do not have access to the fields in that view, so I just display a notice in the "empty text" informing they should be logged in to view the listings.

What is annoying is that sort dropdowns are also displayed on that page which kind of makes no sense :)

Is there a way to hide the sorts based on the fact that the view is empty?

Or, if there isn't at the moment, perhaps this can be made into a feature request?

Thank you!