Hidden filters with a fieldset still display the fieldset legend

Problem/Motivation

When adding a block with a hidden exposed filter, that has (for example) a range of values (From, To) the filter is hidden but the legend is still rendered

Steps to reproduce

1. Add a view with an exposed filter that has values "between", for example a date range
2. Add another exposed filter, simple value
3. Add an exposed filter block, hide the range filter
4. View it

Markup

<fieldset id="edit-date-wrapper--4" class="js-form-item form-item js-form-wrapper form-wrapper">
  <legend>
    <span class="fieldset-legend">Search Date Range</span>
  </legend>
  <div class="fieldset-wrapper">
    <div class="hidden-exposed-filter">