Buttons disappear after removing entity

In some circumstances, the form state contains empty entries for the list of referenced entities after removing a reference. Here is a way to reproduce it:

1. Add a single-valued entity reference field to a node type and set it to use Inline Entity Form (multiple widget).
2. Create a node and a new referenced entity.
3. Edit that node and remove the reference to the inline entity (it doesn't matter if the entity itself is deleted or not).
4. After confirming the deletion, the Inline Entity Form widget is empty, without the "Add new entity" and "Add existing entity" buttons.

There is already a check in place, but it only skips invalid entries:

if (!isset($value['entity'])) {
  continue;
}

However, the invalid entries are still in the $entities array and cause the cardinality comparison logic to fail and behave as though there is still a referenced entity.

The attached patch cleans up the $entities array after loading it from the form state.