Core elements not removed when no metatag substitute provided

This is a follow-up to #1304038: Need to be able to disable meta tags provided by core (generator, shortcut icon, shortlink) which had only been partially resolved (for generator) and finally went in the wrong direction.

While you can override "core" metatags, you cannot actually disable them.
The current code in metatag_html_head_alter() only unsets elements added via drupal_add_html_head_link() if a metatag equivalent has been added earlier. Note the position of the continue statement:

/**
 * Implements hook_html_head_alter().
 */
function metatag_html_head_alter(&$elements) {
  // Remove duplicate link tags if found.
  $metatags = metatag_get_info('tags');
  foreach (array_keys($metatags) as $name) {
    if (!isset($elements['metatag_' . $name]) || $elements['metatag_' . $name]['#tag'] != 'link') {
      // Only check for link tags added by the metatags module.
      continue;
    }
    foreach (array_keys($elements) as $key) {
      if (strpos($key, 'drupal_add_html_head_link:' . $name . ':') === 0) {
        unset($elements[$key]);
      }
    }
  }

  // Remove the default generator meta tag.
  unset($elements['system_meta_generator']);
}

Since leaving a tag setting empty results in the tag not being added, the same metatag provided by another source won't get removed. It seems to me that any elements supported by metatag should be removed, regardless of wether they have been added earlier or not.
Filing as bug since the documentation claims this to be possible.