Selective filter is restricted by contextual filter based on same field

Hi,
So here is my issue: my selective filter only suggests one value whereas there should be a few.
I made a few testing and I believe this is the due to the contextual filter I have, that is based on the same field.
The field is an unlimited entity reference field called "Managers", referencing Users. So each node of my view can have multiple managers and I would like the selective filter to suggest all the managers referenced by at least one of the node in the result list. This works if I don't have my contextual filter on. But the contextual filter allows me to list only the nodes for which the current user is a manager of. So the contextual filter is configured to provide a default value based on the current user...
As soon as I enable this contextual filter, only the current user is suggested in the selective filter :( So it looks like the contextual filter overlaps its filtering to my selective filter whereas it shouldn't. Because I want the current user to be able to see which nodes he/she is working on with some other Manager user.

I think I understand what the conceptual problem is. My contextual filter cannot be ignored. The nodes listed actually need to belong to the current user no matter what. So the selective filter should act like a AND filter. So the request is to list nodes for which Managers field contains Current user + selected User in selective exposed filter.

Maybe I need to change my view: remove the contextual filter and make it an exposed filter. I believe there is a trick to use the current user inside an exposed filter right?