removeVariant() should remove variant from page and not just delete the variant.

\Drupal\page_manager\Entity\Page::removeVariant() currently only deletes the variant config entity, but should also remove that particular variant from the list of variants stored on the page.

eg:

<?php
  unset($this->variants[$variant_id]);
?>