Glossary Pager does not reflect result-set generated by exposed filter.

The end goal here is to have 1 specific content type filterable on both a Taxonomy Term and the glossary style first letter.

I have the filtering working just fine and the the results when a user selects a term then selects a letter are correct.

However the query results that are passed to the pager show results where there should not be any.

For example
Title = OED, Taxonomy term = Encylopedia
Title = Boston Globe, Taxonomy Term = Newspaper

Filtering the view on Encylopedia gives
B(1) O(1)

where B(1) incorrectly is clickable and returns no results and O(1) correctly shows the OED entry.