"Entity view" block showing first entity rendered on all pages

If you place the "Entity view" block using the core block system (ex. into the footer), and then go to a node view page, you'll correctly see the entity rendered again where you placed the block. However, when you go to any other page (including other node view pages) it'll continue to show the first node! If you clear the cache (ie. via drush cr) then reload the 2nd node view page, it'll switch to rendering that node on all pages.

FYI, if the first page you visit isn't a node view page, it'll cause a fatal error due to this core bug: #2620126: No way to handle 'optional' contexts without try/catch

I assume this is something with getting the render cache information wrong. This should be handled automatically by ContextAwarePluginBase but apparently isn't!