Filter to only effect content types containing a certain field?

This has been driving me nuts:
All I want to do is have a fliter ONLY filter content with a specific field and not interfere with my other filters/fields.