Entity::linkTemplates() has priority over uri_callback in Entity::urlInfo()

I wanted to use custom path for taxonomy terms that belong to specific vocabularies. Something like Forum module does.
In order to do this I followed the Forum module's example + the same approach was recommended to me on irc.
So I've implemented hook_entity_bundle_info_alter and added uri_callback to all bundles that I needed and of course created the callbacks themselves.

I have a View of entities that displays a link to taxonomy term that the entities belongs to. In my surprise the links were pointing to the same old url.

So I did a little bit of research and found out that it originated in Entity::linkTemplates(). This method first takes the links form the entity's annotation definition and if the link, which defaults to 'canonical' and which was used by the Taxonomy Views Field plugin, exists it returns it's route. If it does not exists it tries to use uri_callback for the bundle and if that does not exists it uses the uri_callback of the entity type.

Now the issue here is that 'canonical' is the default link name in any 'view' operation for any entity, basically.
This means that the uri_callback will never be taken into account until the link name is not present in entity's annotation.

So in order to achieve my goal I would have to somehow change this link's name in every place on the website that uses this method of retrieving entity url.

I think this totally kills the utilization of the hook_entity_bundle_info_alter for this purpose. Therefore I think that the order by which the route for the link is being determined should change from

1. links
2. bundle uri callback
3. entity uri callback

into

1. bundle uri callback
2. links
3. entity uri callback

When you think about it it kind of makes sens to do this change anyway because first we want to know it there is a specific configuration. Only after that we want to use the global configuration. Using links is like using 'cache' and using entity callback is like the last resort when everything else failed.