hook_inline_entity_form_entity_form_alter() not called on #ajax event?

I am using hook_inline_entity_form_entity_form_alter() and #ajax to load a set of values into a second select list based upon value from a first select list.

Let's say the first field is Team and the second field is Players to select from. The idea being that you can select a team, then select from it's roster of players. I followed the ajax dependent field example.

hook_inline_entity_form_entity_form_alter() works fine on the initial load, I can prepopulate the Team field with a list of values and add the #ajax item. But when the team is selected in the Team field and the #ajax callback is invoked, hook_inline_entity_form_entity_form_alter() doesn't get called again when the form-building function of the parent element is called.

Is this the way hook_inline_entity_form_entity_form_alter() and good ol' hook_form_alter() work or am I missing something? From what I understood, the modifications made to the form as a result of an ajax event are supposed to be made from within the form-building functions. Are the form altering hooks not called or is this an issue germane to IEF?

Structure would be:
Game
-> field_scoring (IEF)
Scoring
-> field_team
-> field_players

Thanks.