Removing an operation and its handler class puts a site in a totally broken situation, because clearing the cache which in theory should make the operations system forget about the operation instead causes a fatal because the handler class is not found.

I am guessing this is because of the order in which different caches are being cleared and rebuilt: something that looks at existing operation definitions is running before they are cleared.

One way -- I hope -- to deal with this would be for entity_operations_get_entity_operation_handler() to check whether the class exists, and if not, return a handler of class EntityOperationsBroken. This is the same way that Views and Flag work in this situation.