Why does the field need to exist in the display to work?

Why are conditions solely evaluated when the fields are present in the display? It makes it confusing to setup - sometimes there are fields holding values that determine IF the fields show, but they aren't fields that should be in the display necessarily. This causes it all to fail until they are added, which requires work to go in and put in field permissions and perhaps CSS to hide it from view.

How come the conditions can't be evaluated at the entity level, where all the data is accessible upfront without display dependencies?