Unexpected language prefixes on sitemap index

Note to self: the index generation seems to happen in src/Plugin/simple_sitemap/SitemapGenerator/SitemapGeneratorBase.php

  public function getIndexContent(): string {
    [...]
    // Add sitemap chunk locations to document.
    for ($delta = 1; $delta <= $this->sitemap->fromUnpublished()->getChunkCount(); $delta++) {
      $this->writer->startElement('sitemap');
      // THE URL TO THE CHUNK IS CREATED HERE
      $this->writer->writeElement('loc', $this->sitemap->toUrl('canonical', ['delta' => $delta])->toString());
      // @todo Should this be current time instead?
      $this->writer->writeElement('lastmod', date('c', $this->sitemap->fromUnpublished()->getCreated()));
      $this->writer->endElement();
    }
  }

Okay, so getIndexContent() from the snippet mentioned in my previous comment calls SimpleSitemap::toUrl(), which is this (in src/Entity/SimpleSitemap.php)

  public function toUrl($rel = 'canonical', array $options = []) {
    if ($rel !== 'canonical') {
      return parent::toUrl($rel, $options);
    }

    $parameters = isset($options['delta']) ? ['page' => $options['delta']] : [];
    unset($options['delta']);

    if (empty($options['base_url'])) {
      /** @var \Drupal\simple_sitemap\Settings $settings */
      $settings = \Drupal::service('simple_sitemap.settings');
      $options['base_url'] = $settings->get('base_url') ?: $GLOBALS['base_url'];
    }

    $options['language'] = $this->languageManager()->getLanguage(LanguageInterface::LANGCODE_NOT_APPLICABLE);

    return $this->isDefault()
      ? Url::fromRoute(
        'simple_sitemap.sitemap_default',
        $parameters,
        $options)
      : Url::fromRoute(
        'simple_sitemap.sitemap_variant',
        $parameters + ['variant' => $this->id()],
        $options);
  }

Looking at the routing file, the route normalizer is already disabled (suggested here: https://drupal.stackexchange.com/questions/246572/disabling-language-pre...)

simple_sitemap.sitemap_default:
  path: '/sitemap.xml'
  defaults:
    _controller: '\Drupal\simple_sitemap\Controller\SimpleSitemapController::getSitemap'
    _disable_route_normalizer: 'TRUE'
  requirements:
    # Sitemaps are accessible for everyone.
    _access: 'TRUE'

What would be the Right Way to ensure that the URL does not have a language prefix?

(For the time being I wrote a small patch for myself that is included in my composer.json which has a hard coded "remove /fi/ prefix" logic but that's obviously not the correct way to handle this...)

Cheers,
Markus