Remove button broken when you have a field collection with required fields within a fieldable panel pane

Steps to reproduce:

1. Create a fieldable panel pane with a field collection. In that field collection have a required field.
2. Add a new pane of that type using panelizer.
3. Add all the required fields to the field collection and click Add another
4. Don't add all the required fields to the next set.
5. Click Save
6. Message should be shown saying fields are required.
7. Try to remove the row with empty required fields. It will fail here.