Changes to service definitions can cause fatal errors until services cache is cleared

It think what would help you is to leverage a deployment identifier, see settings.php. This allows you to directly invalidate the previous container without any additional step. The deployment identifier is designed in a way that it works across environments.
There are plans to expand that to twig templates as well, see #2752961: No reliable method exists for clearing the Twig cache

Given the nature of a service container, clearing the cache after changing a service definition as developer workflow seems entirely reasonable to me.

I had not known about the deployment identifier!. The example shows a dynamic identifier, is there a recommended way to set this up for custom projects that does not require editing the settings.php file each time? For discussion I've copied out the passage from the default.settings.php below:

/**
 * Deployment identifier.
 *
 * Drupal's dependency injection container will be automatically invalidated and
 * rebuilt when the Drupal core version changes. When updating contributed or
 * custom code that changes the container, changing this identifier will also
 * allow the container to be invalidated as soon as code is deployed.
 */
# $settings['deployment_identifier'] = \Drupal::VERSION;

@daniel.nitsche I like your idea.

Maybe during a composer update, we could write the composer.lock's content-hash to a file (ie composer.lock.hash) and then include it as the $settings['deployment_identifier']

So I was inspired to make my very first foray into GitHub and created Changes to service definitions can cause fatal errors until services cache is cleared #451 with two possible composer specific solutions.

Composer hash wouldn't work for changes to custom modules right?

@mstrelan
That's true. Changing a custom code won't affect composer.lock.

I've been thinking of using our OpenShift deployment-id env var as an identifier, but could it be bad performance to invalidate the container on each deploy rather than when needed?

So probably the best thing that we can do is to generate a hash from all services.yml -s' filemtime() (or something) in a codebase meanwhile a deployment is running - before updb/cim would run - and use that has as a deployment identifier. Or is there any other option that could solved this in Drupal and would not require additional steps from developers/devops?

I wasn’t aware of this problem, but when it happens, it sure is brutal. We just had this problem in Search API (#3126367: TypeError: Argument 2 passed to SearchApiConverter::__construct() must implement EntityRepositoryInterface) and I guess a lot of people got a small heart attack (or at least needed new pants).
Is there any way for contrib currently to address this, when making such a change? An update hook or something that would rebuild the service container cleanly?

If there is nothing, then I definitely agree that this is a major bug. And even if some workaround is available (for contrib developers, not the people deploying the changes), it would be great if this could somehow be fixed in Core so the workaround isn’t needed. I fear a lot of others are as clueless about this problem as I was until just now.

If this is technically not possible then the expected update/deploy steps for a site should be clearly documented, like

$ drush cr
$ drush updb -y 
$ drush cim -y

because currently, I am afraid that running an initial drush cr can break things because some services in Drupal are not actually stateless. If some cache entries related to a service get cleared it can behave differently, especially if there were a related change to the service since the last deploy - which is also the reason behind running drush cr in the beginning, because this is how you can convince the service container to rebuild service definitions. Another option is only clearing the cache entry of the service container with drush sqlq "TRUNCATE cache_container;".

Here is some reference that perfectly showcases that nobody has a bulletproof strategy for Drupal 8 deployments ;S
https://www.lullabot.com/articles/a-successful-drupal-8-deployment
https://medium.com/@ChandeepKhosa/drupal-8-development-deployment-a-step...
https://www.bounteous.com/insights/2020/03/11/automate-drupal-deployments/ (Run drush cim -y 3 times???)

I wholeheartedly agree with @mxr576, we really need clear documentation for the deployment steps.

because currently, I am afraid that running an initial drush cr can break things because some services in Drupal are not actually stateless. If some cache entries related to a service get cleared it can behave differently, especially if there were a related change to the service since the last deploy

Are there examples of this? I've been performing cache rebuilds before anything else after a deploy for years and haven't run into something like this. I don't doubt it, just curious.

It was a long time ago but I think an exampe could be when entity type definitions are changing and "drush cr" clears the "active entity type definition" from the (database) cache and the entity type manager reads the definition from the code, but an hook_update_N() or hook_post_update_N() within the same deploy would still need the cached definition to update the database.

@bkosborne here's a commit that introduced this problem for one of our projects: https://git.drupalcode.org/project/spamspan/-/commit/e0bc1308fad4fb7861c....

Drush command terminated abnormally due to an unrecoverable error.
Error: Uncaught Error: Class 'Drupal\spamspan\TwigExtension\SpamSpanExtension' not found in /var/www/code/drupal/web/core/lib/Drupal/Component/DependencyInjection/Container.php:259

Let me update my previous comment (#22) with additional input based a discussion with @alexpott on Slack.

Today another deployment failed that introduced a base field on a user entity with

Base table or view not found: 1146 Table 'user__ad_groups' doesn't exist: SELECT "t".*                     
  FROM                                                                         
  "user__ad_groups" "t"                                                        
  WHERE ("entity_id" IN (:db_condition_placeholder_0)) AND ("deleted" = :db_c  
  ondition_placeholder_1) AND ("langcode" IN (:db_condition_placeholder_2, :d  
  b_condition_placeho

when drush updb command just _started_ to run; because the baseFieldDefinitions() method was called on the entity _before_ an update hook could install the new field storage defintion. My assumption was that (as I said in #22) that happened because the deploy sequence we are still using is:

$ drush cr
$ drush updb -y 
$ drush cim -y

due to #19. BUT Alex said that

Interesting. I consider drush deploy to be the canonical sequence of commands. The first drush cr in the sequence above should not matter. The update process should not use anything from cache until the caches are cleared after hook_update_N.

and he also called my attention to that we are trying to add a base field to a user AND uid0 is also a user in Drupal which

FWIW user 0 is still loaded from the database so that’s probably why it is failing.

So there is definitely an issue with adding base fields to user entity that needs some proof as a test. It can be also verified why a full user load is happening when drush updb starts. This problem is probably not related to running an initial drush cr before updb.

And for the record, Alex also shared a possible workaround for this particular problem that I am leaving here as a record for the future:

A work around would be to store something in state. In your base field definitions check this… when false don’t add the definitions… and then true do… in your update function set it to TRUE and clear the static caches and sort out the definitions… you’d need to set the state to TRUE in the hook_install() as well.
And then in some future update remove the state thing and the code that checks in base field definitions.

Some more ideas on this here, #3364282: Allow modules to change the container cache key

A problem is that some low-level services are instantiated before any rebuild operation from e.g. drush cr would happen.
One such service is module_handler, but there are others.

Here's a snippet for modules to put in the .install file for an update hook:

/**
 * Rebuild the container cache.
 */
function MYMODULE_update_NNNNNNN() {
  \Drupal::service('kernel')->rebuildContainer();
}