Create a list of taxonomy terms, but omit terms with no nodes or that have certain other terms?

I have two vocabularies, "contacts" and "category". I want to create a list of all "contacts" terms that are associated with a node (so "contacts" terms with no nodes are not included in the list). However, if a node associated with a "contacts" term (for example, "Jill") is also associated with a particular term from "category" (for example, "apple"), I want it to not be counted. Therefore, if all the nodes with the contacts term "Jill" also have the term "apple", then Jill would not show up on the list.

Is there any way to accomplish this?