Entity reference of type "views" with huge amount of data results in Max Memory

Situation:
+- 9000 users

Entity type with entity reference field of entity type "user". The reference type used is "views"
Form display with Widget Select2: autocomplete on - "contains"

Problem:
When the select2 widget is initialised on the form, we get a "Fatal error: Allowed memory size exhausted in /var/www/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php on line 1276".
This is because the select2 widget will fetch all results from the view, without pagination and thus results in a load of +- 9000 users which is a bit much.

(possible) Solution:
The Select2 widget will look for all valid referenceable Entities.
This happens in the Element/Select2.php file on line 186. There we should find something to override the default behavior because now it acts more or less like a default select element where this issue would also be the case, hence the lazy load that is needed.