The whole theme/rendering of the token browser is problematic, it uses multiple theme functions that call each other, different ways to do the same (e.g. two different theme functions, but one can do the same as the other based on configuration).
It is also not possible to use the render cache with the current approach and the custom render cache that token implemented is possible anymore either, as the functions it used have been removed/moved into a service.
My suggestion would be to keep the browser modal link as the only theme function, convert the nested table theme functions to a service/function/#type that sets up the render cache and then uses #type/pre_render callbacks for the real work, allowing to rely on the render cache properly. Also, zero tests :)
I think that the custom overrides that token has for using render cache on POST would not be relevant if we default to the modal wherever it makes sense (= all forms).