Need filter: "While having role ?"

I have a

Flag: "browse anonymously" (user account checkbox)

Rule:
event: A user has been flagged, under "Invisible"
Action: Add "invisible" role and remove current/original role.

Views: Visitors
Filter: User: Roles (= original) only shows users with original role.

This all works to this point. Users with "invisible" role do not show up in the Visitors display.

Issue:
When users go back to their original role ("invisible" role removed - original added.) they then appear in the Visitors display of the users they visited.

Example:
UserA visits UserB as "Invisible". UserB does NOT see that UserA visited (this works). However, when UserA turns off "Browse anonymously" all users they visited as "invisible" instantly shows that UserA visited them.

Hoping someone can tell me what I'm doing wrong or what I should do.

Thank you

SL