Problem/Motivation
Currently entity render caching sets #cache on the render array, but this only allows drupal_render() to cache the actual process of getting the HTML to a string.
We can also use #pre_render and defer the actual building of the bulk of the render array to cache misses as well.
The build and rendering process looks like this (in pseudo UML).

Note how the building of a renderable occurs before drupal_render() is invoked. This means that the cost of building an entity and its fields is incurred before the cache check occurs. Regardless of whether a cached entry is returned and rendering is avoided, the cost of building the entity and its fields is not avoided.
Proposed resolution
This issue proposes to move the costly building of an entity and its fields to after the cache check. To do this, we will undertake the building in a #pre_render callback on the entity. The flow looks a bit like this (again in pseudo UML).

API changes
Ideally, module developers will not notice this subtle shift in when the view and build hooks are invoked. To a developer, the build and render flow will appear to be the same.
This is not true for entities that override the EntityViewBuilder::view() and EntityViewBuilder::viewMultiple() methods. These methods, when the parent method is invoked, will return a minimal set of properties for the entity that include view mode, language and caching information. Entity classes should only override these methods in order to augment the information returned by EntityViewBuilder or to replace the pre render optimization strategy. Otherwise, the entity classes may provide an additional pre render function that will be run after the default EntityViewBuilder::buildEntity and/or EntityViewBuilder::buildEntityMultiple have run, providing the full entity (and its fields) for manipulation. Even better is to just implement the view and build hook and alter methods for specific changes in the build process.
Once the result of rendering is cached, the EntityViewBuilder::buildEntity and EntityViewBuilder::buildEntityMultiple code paths will not be run again until the cache for the entity is invalidated.
View and build hooks/alters
New hooks
hook_ENTITY TYPE_build_defaults_alterhook_entity_build_defaults_alter
Changed hooks
The signatures of the following hooks will change:
hook_ENTITY_TYPE_viewhook_entity_view
Before
function hook_entity_view(\Drupal\Core\Entity\Entity $entity, \Drupal\Core\Entity\Display\EntityViewDisplayInterface $display, $view_mode, $langcode) {}
After
function hook_entity_view(array &$build, \Drupal\Core\Entity\Entity $entity, \Drupal\Core\Entity\Display\EntityViewDisplayInterface $display, $view_mode, $langcode) {}
Entity view functions
The function entity_view() and entity_view_multiple() in entity.inc now only return basic caching information for the entity build. You cannot expect to have the full build information for the entity and its fields out of these functions. This change mostly affects tests. A new method on WebTestBase has been introduced to return the full renderable for an entity.
\Drupal\simpletest\WebTestBase::drupalBuildEntityView()
This returns what entity_view() used to return, namely the full renderable for an entity.