Comparator link alway show

It's not a bug since the module was designed to show remove link in the first row of the comparison table. If you want to customize the table, you can do it programmatically using hook_entity_comparison_rows_alter, see more in entity_comparison.api.php.

Fixed the issue.

So the issue seems to not be about the comparison table, but the view modes of the entity.

As can be seen in this guide as well https://www.webomelette.com/creating-pseudo-fields-drupal-8 when using the hook_entity_view() for extra fields the $display->getComponent has to be used to make sure that the extra field is only shown when the view mode is configured to show it (see patch in #3)

Without this patch the link to add (or remove) the entity from the comparison list is visible everywhere where the entity is rendered, no matter the view mode, but this couldnt have been intended.

Yupp, this is indeed a bug - patch works perfectly fine, thank you!