Performance issues with large taxonomies

I've a taxonomy containing just over 1000 terms, at most 5 levels deep but mostly 2-3 levels. When I add a hierarchical taxonomy menu to the home page performance deteriorates significantly to the point the site becomes almost unusable. I've since moved the block onto a separate test page instead of the home page so I can test performance without breaking the site.

I've set the Cache config option to Permanent and I've added some calls to \Drupal::logger to time the build() function in the block. It takes 3-4 seconds which isn't fantastic but I had hoped the caching would take care of it. However after experimentation I've discovered that the build() function get's called again each time a new user hits the page, as if the logging is using the user ID as a context. The context is set to `url.path` in the code which means the hierarchy will rebuild on every page it appears on even though it's identical. So I'm not sure why `url.path` is set in the cache context for the block - is this necessary?

Having put the logging in, I've noticed something more concerning. Even when the build function is not called (i.e. the cache is hit), the page still takes ten or more seconds to load. Removing the block from the page does restore the performance. So, it seems that even when retrieving the cached copy of the block data, some code is still using lots of CPU cycles. I wonder if this is related to rendering the output in the Twig template, but if so I'm surprised this isn't also cached.

Any assistance would be much appreciated.