InstallHelper::validateProfile() fails because extension list is cached

Clearing the cache won't help in all instances. See #3240928: Drupal\Core\Extension\ExtensionList::getPathnames() conflates cache and state values.

When using the install-profiles:switch drush command, it's possible to get the error below when a stagnant list of extensions is stored in state and the cache is empty:

$ ../vendor/drush/drush/drush install-profile:switch my-profile
 [warning] The following profile is missing from the file system: my-profile bootstrap.inc:195
 [warning] The following profile is missing from the file system: my-profile bootstrap.inc:195

In ExtensionList.php line 522:
                                       
  The profile my-profile does not exist.  
                                       

I believe resetting the extension list with the 'extension.list.profile' service is a viable workaround:

  public function validateProfile($profile) {
+   \Drupal::service('extension.list.profile')->reset();

    // Ensure the profile exists.
    if (!$this->profileList->exists($profile)) {
      throw new \InvalidArgumentException("The {$profile} profile does not exist.");
    }

We could look at core tests for ideas. I just opened a MR for the core issue, and tests are running right now.

Patch attached. I haven't figured out test coverage, however.

Steps to reproduce (with no patch initially):

1. Install Drupal 9 with Standard profile
2. Add Houston profile per Houston README
3. Enable Helper module
4. Run `drush install-profile:switch houston`
5. Observe the following error:
   

       In InstallProfile.php line 156:
                                          
         The houston profile does not exist.
       

6. Patch Helper with most recent patch
7. Run `drush install-profile:switch houston`
8. Observe the error no longer presents

Steps to reproduce (with patch initially):

1. Install Drupal 9 with Standard profile
2. Add Houston profile per Houston README
3. Enable patched Helper module
4. Run `drush install-profile:switch houston`
5. Observe the error no longer presents

Here's the call stack:

Drupal\helper\InstallProfile->validateProfile (/Users/my-user/Sites/helper-3201074/web/modules/contrib/helper/src/InstallProfile.php:156)
Drupal\helper\Commands\InstallProfileCommands->switch (/Users/my-user/Sites/helper-3201074/web/modules/contrib/helper/src/Commands/InstallProfileCommands.php:55)
call_user_func_array:{/var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php:257} (/Users/my-user/Sites/helper-3201074/vendor/consolidation/annotated-command/src/CommandProcessor.php:257)
Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback (/Users/my-user/Sites/helper-3201074/vendor/consolidation/annotated-command/src/CommandProcessor.php:257)
Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter (/Users/my-user/Sites/helper-3201074/vendor/consolidation/annotated-command/src/CommandProcessor.php:212)
Consolidation\AnnotatedCommand\CommandProcessor->process (/Users/my-user/Sites/helper-3201074/vendor/consolidation/annotated-command/src/CommandProcessor.php:176)
Consolidation\AnnotatedCommand\AnnotatedCommand->execute (/Users/my-user/Sites/helper-3201074/vendor/consolidation/annotated-command/src/AnnotatedCommand.php:313)
Consolidation\AnnotatedCommand\AnnotatedCommand->run (/Users/my-user/Sites/helper-3201074/vendor/symfony/console/Command/Command.php:255)
Drush\Application->doRunCommand (/Users/my-user/Sites/helper-3201074/vendor/symfony/console/Application.php:1027)
Drush\Application->doRun (/Users/my-user/Sites/helper-3201074/vendor/symfony/console/Application.php:273)
Drush\Application->run (/Users/my-user/Sites/helper-3201074/vendor/symfony/console/Application.php:149)
Drush\Runtime\Runtime->doRun (/Users/my-user/Sites/helper-3201074/vendor/drush/drush/src/Runtime/Runtime.php:118)
Drush\Runtime\Runtime->run (/Users/my-user/Sites/helper-3201074/vendor/drush/drush/src/Runtime/Runtime.php:48)
require (/Users/my-user/Sites/helper-3201074/vendor/drush/drush/drush.php:72)
{main} (/Users/my-user/Sites/helper-3201074/vendor/drush/drush/drush:4)

Here's where the exception is thrown in Helper by Drupal\helper\InstallProfile::validateProfile():

  public function validateProfile($profile) {
    // Ensure the profile exists.
    if (!$this->profileList->exists($profile)) { \\ <-- this is where it fails.
      throw new \InvalidArgumentException("The {$profile} profile does not exist.");
    }

$this->profileList is populated by the extension.list.profile service. Due to a bug (#3240928: Drupal\Core\Extension\ExtensionList::getPathnames() conflates cache and state values) in how Drupal\Core\Extension\ExtensionList::getPathnames() caches profile lists, the new profile, my_profile, is not registered by the extension.list.profile service. As a result, when Drupal\helper\InstallProfile::validateProfile() uses that out-of-date list, it can't find the new profile. Patch #6 provides a workaround by forcing the extension.list.profile service to rebuild the profile list before using it.

This discussion has been exclusive to the ::validateProfile() method; however, a workaround for the same core bug is already in place in the ::switch() method:

public function switch($profile, $schema_version = NULL) {
    $current_profile = \Drupal::installProfile();

    // Forces ExtensionDiscovery to rerun for profiles.
    $this->state->delete('system.profile.files'); // <-- WORKAROUND

    // Set the profile in configuration.

Do we need test coverage or is documenting the core issue sufficient?