Fieldset displayed if there are empty sub-fieldsets

I have a horizontal tab that displays even when there are no fields to display. In my case, I have fields that aren't available to certain roles. However, those fields are organized into fieldsets. The horizontal tab displays for all users, and it looks like it is due to having these fields in a fieldset. If I remove the fieldset, the tab behaves correctly.

For example, I have a tab called "Contact Information". In that tab, all of the fields are contained within fieldsets. I have "address" and "phone" fieldsets. The fields in these fieldsets are only applicable to certain roles. For roles that do not need this information, they see the Contact Information tab, which is empty. If I remove the fieldset (but not the fields), the tab is not displayed to those users.

I don't have a patch yet. If I come up with one, I'll post it here. I think I may need to modify field_group_remove_empty_form_groups. If anyone has any ideas, please share. Thanks!