Add a container_command and host_command Assessment plugin

In Phase II we may allow for modifications to the environment and restarting of the containers. E.G. apt-get install php7.1-gmp to add a particular extension and restart the containers. Or perhaps its a config change to mysql or php itself (disabling assertions etc).

Sweet. Bingo.

Plugins have underscores, config has hyphens.

I super like this idea, and think thats how it should be, but Im somewhat concerned that if we convert the codebase to have this standard, that we'll have a pile of older builds and older build.yml files that wouldnt be compatible with the new format. - but maybe we just add a thing where right after we try to discover if its a plugin and dont find one, we string replace _ for -'s for 180 days, and then after all the old builds have expired we remove it.

Also, found a potential use case for pre/post build.

Found this gitlab-ci.yml in a repo:
https://cgit.drupalcode.org/file_system_switch/tree/.gitlab-ci.yml?h=7.x...

So having the ability to make commits would be interesting. I doubt we can do that because we'd need maintainers ssh keys, but its something to think about.

Right, I was just saying that, thats a good idea for a standard, but we already have several config elements with underscores instead of hypens, and that if we adopt that standard, we should fix those config elements to be hyphens, but then existing drupalci.yml files wouldnt work because they'd all have underscores.

So I was just saying that at the point where we're recursing through the yaml, and matching keys to plugin names, anytime a key doesnt match a plugin name, we're assuming it means its a config element. So right there we could just convert whatever was submitted to be a hypen, so that it'll match the rest of the converted codebase.

#2953923: Convert configuration elements to be all hypens, ensure all plugin definitions use underscores Im def gonna do this.

Also, I've been testing this plugin out and running into a couple of rando issues, mostly with some oddball things we're doing elsewhere.

#2953897-7: drupalci.yml should not fail on container composer Is my latest, and its causing a bug with http://cgit.drupalcode.org/drupalci_testbot/tree/src/DrupalCI/Plugin/Bui... where pimple gets mad that we've set the db.results service and are trying to set it again. So Im working on a fix for that.

Also going to tweak some other config stuff, specifically '$is_extension_test' config should not be a config at all, and should instead come from codebase->getProjectType.

This one, keep working on.. I was getting confused between this and the other one

So when I said

Also, I've been testing this plugin out

I meant :
#2950939: Use the drupalci.yml to specify assessment if modified

Documenting the slack discussion:

Pre/post seem like they are the same thing, only separated by where they happen in the build.

Lets just make them command plugins, and remove the 'execution-environment: host' config and make them subclasses of each other. Seems nicer to say :

build:
  assessment:
    host_command.setup:
      die-on-nonzero: FALSE
      commands:
        - setup your/thing
        - setup other/thing
    container_command.critical:
      die-on-nonzero: TRUE
      commands:
        - showstopper -V

I'll gather this and start lookin.

Im starting to wonder if https://www.drupal.org/project/drupalci_testbot/issues/2847205 is something we should do first. execWithArtifact sure seems like its going down that path.

I saw these plugins trying to do a bunch of things they shouldnt need to be trying to do , and additional code for stuff that was already handled elsewhere, So, I went ahead and did some unifying of the underlying command infrastructure over in https://www.drupal.org/project/drupalci_testbot/issues/2847205

I even did the needful and updated CONTRIBUTING.md to reflect all these improvements:

https://cgit.drupalcode.org/drupalci_testbot/tree/CONTRIBUTING.md#n362

So, given that we've got this fancy api that does all the signal, artifact, and array handling for us, these plugins can just about be a passthrough mechanism.

*but wait, theres more*

I was thinking that in the vast majority of cases, the commands that are being executed need some variables from the underlying system, specifically, the source directory, and the project directory, in both the environment and on the host. -> perhaps we can either set some environment variables for whatever commands get executed, (proc_open?) or have some sort of rudimentary tokens that translate {SOURCE_DIR} into the sourceDir of the system under test. I dont think that changing directory to those locations is going to do what we want (especially with the docker container.. you cant just cd, because each command is like a new shell. you gotta cd somedir && do something else

Anyhow, did some other moving around, and now what remains is some command tokens, and making the tests test things.

Okay, so what we have now are two plugins under Utility: Command ('host_command'), and ContainerCommand ('container_command').

These rely on symfony process component to handle the execution of things.

Stuff I ran into:

Im setting some environment variables in each plugin such that consumers of these plugins can execute commands and have access to the following vars:

HOST_SOURCE_DIR
HOST_PROJECT_DIR
TEST_SOURCE_DIR
TEST_PROJECT_DIR

Which, now that I think about it shouldnt be obviously just SOURCE_DIR and PROJECT_DIR, and have different values depending on whether its host_command or container_command.

Anyhow, setting env vars on the symfony process must have done something wonky to composer such that the local composer commands were failing on a real testbot (but not on the vagrant box).

I got around that by not setting a blank array to the env, but that might mean that there are other scripts that fail when we pass in the envs in these plugins. Hard to tell. Maybe its just composer.

Okay, existing tests are passing, and, due to circumstances and not-quite thinking things through, this was merged into dev for testing, and then I had to get a separate concurrency bug fixed in d7, and so this ended up in production. So yeah, its there now.

Lets close this and carry forth in a followup: #2955089: Create tests that exercise host_command and container_command