Request for feedback: replace conditional split with partial split.

I think having partial splits would be an amazing power-up on config_split.
I could help new week on developing a poc, could you give me some guidance?

My first thoughts is that partial configs could completely replace conditional splits (aka graylist). My only concern is the upgrade path. Currently the existing sites could have a a file called node.type.recipe.yml under config/sync and another file called node.type.recipe.yml under the config/sites/my_site which it related to a config split.
According to your post, after the introduction of partial splits there would be a breaking change, so even if the 2.x release is a beta, there are + than 2900 sites using it, so IMO creating a 3.x branch could be justified and would let it be released with a more relaxed upgrade policy without breaking the existing sites.

@bircher thank you for the write up and the work that went into the poc. to me the changes for 2.x with partial splits read easier to comprehend and to wrap the head around for people, especially those getting into configuration splits (the the name partial split is also more approachable). and the quasi-patch form has a way better legibility and is way more explicit imho. the cognitive load is way less than in the current implementation. i like it a lot. especially that "added" and "remove" are named and distinguished and the changes are listed underneath each. i like that a lot! double thumbs up!

and one question in regards of partial splits: are they more robust, less error-prone and especially less maintainence effort for module maintainers compared to conditional splits especially if core api is used now?

Hello @bircher. Thanks for the updates. Then no concerns from my side regarding upgrade path.
I can't wait to see this feature in action. It's gonna be a game changer.

Really looking forward to seeing the proof of concept code! I'm also interested in helping.

Looks really interesting.

For my own understanding would the main benefit be that it's easier to interrogate a split manually to see what the difference is?

I see many use cases when having several splits for the same config entities. Here is an example:
You have 2 splits: A and B splitting 2 fields on the article content type.
Split A add the field A, and Split B adds the field B to the site. Now let's take a look on the minimum config entities involved:
- field.storage.node.field_a
- field.storage.node.field_b
- field.field.node.article.field_a
- field.field.node.article.field_b
- core.form_display.node.article
- core.view_display.node.article

for field.storage and field.field it is clear, there are independent, the full file need to be split in each config_split, but what happen with core.form_display core.view_display config entities? they have information needed for these fields to work. In the current implementation, using conditional split you will have 2 copies of the core.form_display.node.article config entity in both config_splits with the whole configuration, and you don't have control to merge them on config import (please correct me if I'm wrong). If you try to export configuration it will be even worst, the config entity will be placed in only one of both config_splits with the complete configuration.

I wonder partial splits will save the day for these use cases.

@bircher Thanks for you work on Config Split! This module has been great and super helpful in multiple instances.

I think the changes you are proposing make sense and the cleaner yml files will help people understand what's going on.

One question I did have - These changes seem mostly under the hood (code changes) will this impact UI/UX of using and configuring config split at all?

I have done a number of trainings on config management with config split and I'm happy to help with module and functionality testing if needed.

Thanks again & be well!

Fascinating!

I think there is a case for the partial config / quasi-patch to need to be able to not care what the values in the original sync directory version for a piece of config is. For example, taking the snippet from the nuvole article:

added: {  }
removed:
  dependencies:
    module:
      - menu_ui
  third_party_settings:
    menu_ui:
      available_menus: {  }
      parent: ''

What if the third_party_settings.menu_ui.parent had been changed to a different version in the original/upstream version of the config? Is the expected behaviour that the split/patch refuses to apply?

I have a use case where I use the same codebase for multiple brands' sites, but each brand has its own split, to override some things like some fields' default values. If the 'normal' value in the usual sync dir is changed, I'd still want the brands' specific value to be applied. This feels close to what you've hinted at, where there's a case for keeping config in a conditional split. But I think this case could still be fulfilled with the partial split / quasi-patch model, as long as the values in the config are ignored. For example:

added: {  }
removed:
  dependencies:
    module:
      - menu_ui
  third_party_settings:
    menu_ui:
      - available_menus # no longer mapped to a value
      - parent # no longer mapped to a value

I realise that's not simple either, as the menu_ui dependency there is a value to be removed, rather than being a key!

So although that's not a complete solution, hopefully that's a helpful use case to hear about?

Another use case: I've used the conditional split to attempt to keep reasoning about config simpler where possible. So, for example, changes to a webform might be imported via config sync, but as soon as changes are made manually on a production environment, I want to preserve those changes (without really needing to care as a dev). So having the webform in a conditional split, means that it's only once the production environment's version is different, that the webform would end up in the split, and therefore preserved. Until then, it is effectively 'controlled' by the version in the main sync directory, i.e. by developers using config management, rather than editors on production.

But I suspect this is a case where config_ignore could be used together with config_split. I suspect I'd look for workarounds, rather than being disappointed that config_split would no longer support that.

hi @bircher
Adding support for partial diff is an amazing enhancement to the config split module and whole config management flow in general.
For the backward compatibility point i would say as long as complete config can be replaced in partial diffs it should be fine.

I would also like to highlight https://www.drupal.org/project/config_update and https://www.drupal.org/project/update_helper module which uses a similar concept of config diffs to provide config updates, Going through the module might be helpful from implementation perspective.

At the end i would say a huge thank you for initiating this discussion.

I haven't taken the time yet to study the overall proposal so this comment is just about the config 'quasi-patch' portion.

Rather than comment extensively here, I've updated a relevant core issue, #2960941: Provide an API for persistent configuration alters, including for extension-provided config. tl;dr: Selective and persistent alters to existing config is a need that's occurred all over Drupal since the first 8.x release. The proposed format looks fine but it would be great if we didn't have to limit it to config_split or add yet another config diff format to the growing list of attempts in contrib. The ideal would be to fix this in core. Short of that, or as an initial step, to try to standardize in contrib.

@bircher - unfortunately the title of this issue was misleading as to the scope of the change that was just pushed in the last release. The newest release does not just replace conditional split with partial split (which I can understand the reasoning). It seems to also replace the complete split with the quasi-patch approach you outlined.

In comment 6 you said:

The only reason not to ditch the conditional split is if you need a copy of the config in the split folder for some config for some reason. I would like to understand that reason, because otherwise I don't have the motivation to maintain a feature that makes the UI more confusing and I don't think anyone would want.

And there may not be a reason to keep conditional split separate. But for complete splits, there are multiple use cases.

We've been using config split very successfully for feature bundling with multi site. In addition to splits by environment, we've created splits for specific packages of webforms per site. We have migration splits when we easily create a package of migration scripts. We use feature splits for new development. And the previous method where all the config files are still present and readable just as though they were in the main sync directory is crucial to all those use cases.

(And yes, I know that you can wrap migration scripts in a module instead, but config split worked well and was fast and convenient.)

We had a very unpleasant experience this week where we were running composer update locally and watched as all the custom migration scripts in the migration split folder were deleted. We always pre-backup and post diff the configuration directory for updates so there were no major consequences for us. But we all know there are site maintainers out there who apply minor module updates without those precautions. If any of them are using similar strategies they could have a very nasty surprise on their next set of contrib updates.

I've really loved the config split module and have found it versatile for a variety of use cases. But I do think that if you were going to make such a massive direction change and actually wanted feedback, the comment period should have been more than a few weeks and the title of this issue could have been significantly more clear that more than conditional splits would be affected. And then that massive change really should have coincided with a major version change to 3.x since folks tend to be more cautious when applying a major version change.

Edit: I do love the UI changes in the latest release.

I have actually gotten inconsistent results when upgrading. The first time I upgraded it removed my config directories. That upgrade was combined with the recent core security upgrade if that could be relevant.

The second time I upgraded only the config split module to more closely look at exactly what had changed and it did NOT remove my split directory configuration. Instead it simply added the "storage" key set to folder.

I'm going to continue doing more fine grained testing to see what the effect of this upgrade is on various use cases. My only caution to folks is this may be an instance where you want to isolate this module upgrade by itself.

@bircher thank you for providing more background on exactly what changed.

The rationale for the partial splits completely makes sense. The dependencies chaining is likely something that could have caught me, (e.g. split just the migration modules and assume all migrations would be split) and I'll make sure my splits list all split configuration explicitly.