No error thrown during validation if the referenced entity/entities don't exist

I tested both 7.x-1.2 and 7.x-1.x-dev, same behavior.
When we reference one or several entities, delete them in the meantime, and then save the form, entityreference ids are saved and the field validation doesn't throw any errors.

I'm using the following but it can be reproduced with wide range of settings:
- Create an entityreference field on article nodes (e.g.: target type: taxonomy term, widget: select list)
- Go to the form to create an article /node/add/article
- Fill in the required fields and reference an existing entity
- Open a new tab and delete the entity you just referenced
- Go back to the article form and save
- The article is saved and the referenced id as well. No errors thrown.

There is definitely some logic issue in entityreference_field_validate():
If we reference only unexisting entities, no error is thrown.
If we reference at least one existing and 1 or several unexisting entities, we get the "The referenced entity (@type: @id) is invalid." error as expected.

Looking at entityreference_field_validate() it seems removing the if (!empty($valid_ids)) could fix the issue.