Shipping Info and Quotes Show, even when there are no shippable products

We have one product type with an attribute where you can chose CD or Download as the delivery type. All of the File Download features are set to Shippable: NO.

Yet when we only have this product in the cart, with the Download attribute, it still shows the delivery pane, and still forces the user to select a shipping quote, and so all users are charged for shipping regardless.

("Hide shipping information when possible for carts with no shippable items" is checked in settings.)