Prioritize Facet Drill Down Order

I have a situation where a site has nodes containing two taxonomies: Brand and Category.

When I use a search to drill down by Category, the "Brand" facet is eliminated from the available facet blocks.

When I use a search by Brand, the "Category" facet is still available.

It looks like the Brand facet always gets interpreted with the highest priority. When the user selects a Category facet, the facet API assumes that the user did not want to search by Brand.

Is there any way to make it so that does not happen? Is there a way to control which facet is interpreted first, or to put two facets at the same priority level?

Is it possible to get a search by Category to show the Brands that might belong to nodes with that classification, and also get a searcy by Brand to show the Categories that might belong to nodes with that classification?