Filter out optional config with missing dependencies

This has been asked for many times, and I have generally pushed back on it. @alexpott may also be able to provide clarification.

This is a tricky area because the way optional configuration works goes directly against the intrinsic goals and guardrails of a recipe.

A recipe needs to be, above all, predictable. Recipes can be applied to a site at any point in its lifecycle, and be composed into huge, complicated clusters and stacks. Both of these things mean that a recipe has to be predictable. You need to know exactly what it will do, just by reading it. Recipes must, in every respect, have a guaranteed outcome.

But, from a recipe's perspective, optional config is unpredictable. That's not to say optional config is random; it does operate by a known and clear set of rules. Nonetheless, exactly what optional config you will get really depends on the state of the system at any given point in time. What extensions -- including the install profile -- are installed? What config entities exist? Those can change over time, and whenever they do, it directly changes what optional config can be installed from that point on. Recipes have no way to know the state of the system, and alter themselves to fit it. If they did, that would violate one of the main principles of recipes, which is that they are dumb and cannot contain logic or make decisions.

In other words, if recipes installed optional config the "usual" way, applying a single recipe could produce different results depending on when you applied it. That harms, if not outright breaks, both predictability and composability. How can I build a recipe on top of another recipe which can produce different outcomes depending on factors which I, as the recipe author, have no way to know in advance?

If recipes' predictability and composability is harmed, a great deal of recipes' value is lost. That's why I am generally against the idea of having recipes be in the optional config business as we know it. And it's why I suspect the recipe system, as created by @alexpott, was designed this way.

Doing this would also open the door to an entirely new class of bugs which simply doesn't exist right now. And it would increase the complexity of the recipe system (which is currently fairly simple) by orders of magnitude, since now we have to deal with dynamic dependency resolution, which is problematic in the best of times.

Now, having said all that, I can see why the choice between either "import everything" (*) and "import some stuff" (an array of config names) is limiting. One thing I was considering was maybe introducing support for a new symbol (this is just a guess at syntax):

config:
  import:
    foo_module: '!'

This would mean "import all config from foo_module, except for its optional config".

To me, this:

• Improves DX, since you don't have to list individual entities if you just want the module's default config.
• Draws a clear distinction between optional and required config.
• Continues to give recipe authors the option to also import optional config (the behavior of * would be unchanged).
• Keeps recipes predictable, since you know what you're getting. Obviously you might get different config if foo_module changes the config it ships with, but that is a dependency versioning issue and does not make the recipe more sensitive to the state of the system.