Sorting Taxonomy Term by Weight won't work

I'm seeing a problem in the code:

facets/src/Plugin/facets/processor/TermWeightWidgetOrderProcessor.php

/**
   * {@inheritdoc}
   */
  public function sortResults(Result $a, Result $b) {
    kint($a);
    $ids = [$a->getRawValue(), $b->getRawValue()];

    $entities = $this->entityTypeManager
      ->getStorage('taxonomy_term')
      ->loadMultiple($ids);

    // Bail out if we could not load the term (eg. if the result is not
    // referring to a term).
    if (empty($entities[$a->getRawValue()]) || empty($entities[$b->getRawValue()])) {
      return 0;
    }

    /** @var \Drupal\taxonomy\Entity\Term $term_a */
    $term_a = $entities[$a->getRawValue()];
    /** @var \Drupal\taxonomy\Entity\Term $term_b */
    $term_b = $entities[$b->getRawValue()];

    if ($term_a->getWeight() === $term_b->getWeight()) {
      return 0;
    }
    return ($term_a->getWeight() < $term_b->getWeight()) ? -1 : 1;
  }

The index field is based in taxonomy term name (field_category:entity:name), and the code above is assuming the field will be an id, which is not always the case, therefore, it can't find the right entities by name. I wish there could be a way to handle this better.