Support for export filtering via Drush

There are already solutions for filtering on export.
Config Split, Drush CI Tools and drush itself support it.

Implementing that in here seems redundant and out of project scope to me.

Hi @iMiksu

You are right about the module name: it *really* sounds like something that should go both ways, export and import. And it was the intentions at the early state of the development.

The problem, up until now, has been that there was no way to hook into the process of exporting configuration. Not as far as I can see anyway. And that is the very same reason why the config_split have relied upon custom Drush commands and was only supported by using Drush, up until now.

Recent changes to Drush and D8 Core has now made this possible.

I've been in contact with the maintainer of the config_split module, and we have found away where each module can support each other, making config_split easier to use for smaller sites (hopefully), and give config_ignore the ability to interact with exported config.

You can read more about the config_split side of things here: http://nuvole.org/blog/2017/feb/27/introducing-config-filter.

I think the idea of filtering on export is important as:

• it can be confusing for a developer if they export from production for some reason and find new configuration that they have decided to ignore appear in their version control
• there is a benefit in terms of onboarding and training if the default drush config-export and default export in the UI are altered rather than having to train staff to use a separate command like drush cimy/cexy, etc

In drush, this is fairly straightforward. Here is a first patch to cover it for Drush.

For the UI, it needs a route alter as the export form is just a simple controller with a method that gets all config collections and writes them to a file. I propose to make that a separate issue as it needs a separate chunk of code and work to implement. Will create that in a moment.

Follow-up for ignoring via the UI here: #2857944: Support for export filtering via the UI

If we are going to filter also export, then we need to make this a flag for sure. Because I can understand that people might want to ignore importing deletions, but I believe in "all the config in git" and if I want to get the configuration from production I want all the config and merge it in the sites config that is also sent to staging.

Then a comment for the patch, I am working on getting the storage filters out of drush and into core. with the config_filter module being the middle station. So I created a 8.x-2.x branch here that depends on config_filter. This has then the added advantage to also work with the UI directly and with console eventually.

On 8.x-2.x you will want to implement also the write-related methods and do essentially what the graylist in config_split is doing.

but 8.x-2.x still needs the tests added again...

Just FYI for people looking for this feature:

Config Split allows the split config storage to be in the database. See #2865110: Allow split to live in the database.

So if you want to prevent drush to export some configuration to any files you can now do so.
This does not mean it is the same as Config Ignore though.

Oh, and drush has removed the storage filters, so the approach in the patch above won't work with future versions of drush. See [PR 2686](https://github.com/drush-ops/drush/pull/2686) So the recommended way is to use Config Filter now.

I have created a patch with my first attempt of filtering on export. Still needs a flag to tell it when the ignore on export should be done.

I applied #10 and exported through drush cex but the ignored configuration is still exported. Am I missing something? There is a mention of a flag but not sure what's the flag to use.

#12 worked for me, it filters the export and also the import.

Before, when I had an ignored config and removed the yml file from the file system, then it wanted to delete the config upon drush cim. Now it leaves the ignored config alone and doesn't remove it.

I did get notices on the configuration synchronization page. I've fixed these in the attached patch.

#12 works for me

#12 works great. Let's get this committed!

Thanks for the patch, the behaviour looks good.
However, we should make the export (write) filtering optional and configurable by adding a checkbox on the config ignore form.
Also the tests should be extended to cover this.

#12 worked for me too.

@Johnny Great, tested and patch #12 works without issues.

@bircher Thanks for considering this to merge into some future release :-)

Added new settings form variable to enable export filtering.

Tested #18 and it worked without any issues, the form field is nice to have :)

#18 works great for me with Drush 8.1.15.

Hi,
Thanks so much for continuing to work on this.
I have a few small comments and a big one at the end:

1. +++ b/config/install/config_ignore.settings.yml
   @@ -1 +1,2 @@
   +enable_export_filtering: ''
   

   The default should be false as the type should be boolean

2. +++ b/config/schema/config_ignore.schema.yml
   @@ -7,3 +7,6 @@ config_ignore.settings:
   +      type: string
   

   I think type: boolean would be better

3. +++ b/src/Plugin/ConfigFilter/IgnoreFilter.php
   @@ -186,6 +244,38 @@ class IgnoreFilter extends ConfigFilterBase implements ContainerFactoryPluginInt
   +    if (empty($prefix)) {
   

   We should probably return FALSE when export filtering is turned on independently of the prefix.

4. We need tests to cover the functionality to make sure that it behaves the way we expect and to make sure it will continue to do so in the future.

   Cases to cover:

   1. Config exists in sync folder and does not exist in the database.
   2. Config exists in sync folder and exists in the database (but different).
   3. Config does not exists in sync folder but exists in the database.
   4. Config does not exists in sync folder but exists in the database. But only part of the config is ignored.

   I think the patch already does it in the way we probably expect. But it would be really good to make sure of it with a test.

   The following could serve as an inspiration: https://cgit.drupalcode.org/config_split/tree/src/Tests/SplitFilterTest.php
   If nobody will add tests I will do that when I get to it. But I won't have a lot of time for it in the upcoming two month.

A use case I ran across is: having ignored configuration in a split.

Use case - Ignoring a settings config object for a module that is stored in a complete split. The settings object for this module is included in the ignore list.

Result - When the split is active, changes to the settings object are effectively ignored. However, when the split is deactivated and results are imported, the settings object should be deleted (because it is dependent on it's parent module), but it DOES NOT get deleted, but updated to a simple boolean value. This results in the import failing and a broken settings page for the module.

EDIT:

Workaround - Implement hook_config_ignore_settings_alter() and add logic for checkin whether or not the split is active. If it is inactive then remove all of the ignore settings for that settings object. The object will then be marked for deletion rather than being updated to the value of a boolean, resulting in a successful import.

Is there any use case for when a configuration file would normally be deleted be left not deleted? I think if a config object would normally deleted it should just pass through the ignore filter, yes?

Patch doesn't work if config to ignore belongs to a split.

Problems:
- 1) Splits filters run before config ignore filter, so database config is written to file in split directory even config is ignored.
- 2) IgnoreFilter try to read config from SourceFileStorage, but this storage is in "config sync" and split config isn't there, is in split directory.
- 3) If previous point (2) was resolved, there would have other problem. Current code get the config from source, and source has been written by split filter (point 1), so we have lost original data.

Reroll attached. Patch didn't apply for me with latest dev.

#24 doesn't apply for me, #18 still works against 2.1 though.

Just a note: export does not ignore language collection when the patch from #2973431: Ignore configuration storage collections is applied. Depending on which patch is committed first, the other one may need to be adjusted for the new functionality.

Having read this issue I was surprised to see nothing being exported when trying config_ignore without the patches here. Ignored config does not get exported for the following example settings lines:

webform.webform.*
captcha.captcha_point.*

So the wildcard seems to be the trigger here.

I'm assuming this is not by design. A separate bug report for this inconsistent behavior might be in order, but perhaps it affects this issue here as well.

Eric_A in my experience nothing exports until you change something else. Then the ignored config gets exported along with the other change.

@jansete I think in your case you could change the weight value of the Config Split plugins so that it was heavier than the Ignore plugin, so they would go last. You'd have to play around with it to see if it works for you.

#18 works fine for me as @benjy mentioned.

sourceWrite() must contain the same check as was added in https://cgit.drupalcode.org/config_ignore/commit/?id=0c74edb
Without this check, the following errors are possible:
TypeError: Argument 1 passed to Drupal\Component\Utility\NestedArray::getValue() must be of the type array, boolean given.

https://www.drupal.org/project/config_split_ignore can be used for ignoring configuration which is in splits.

@maximpodorov - Is config_split_ignore a module that extends config_split or is it a fork of config_split?

It depends on config_split and config_ignore by inheriting from config_ignore's plugin and by using config_split data.

Patch on #24 works great for me, do you have any plans to merge this eventually?

We're happily using patch #24. It makes sense to ship this feature in config_ignore: having one place to configure all config items to ignore on both import and export is great.

Yes this feature is going to be added sooner or later.
We need tests to cover at least the four cases described in #21.4.
Maybe someone who is using this patch knows of some other scenario that they wouldn't want to be broken and have tests for that too.
I can review patches but unfortunately I do not have the time right now to write the tests myself.

Hit an error trying to export ignore-filtered configuration when only a property of a config object instead of the whole config object (e.g. system.site:name vs system.site) is ignored and the sync file directory does not have the file. Here's an updated patch to address, as well as tweak the config schema for type. Still needs tests.

Just confirm #23
Those all patches are not compatible for me with config split.
Ignored config are still exported.
config_split_ignore module also din't resolved the issue.

The issue with config_split identified in #23 is due to the respective weights of the derivative config_split plugins (default of 0) and IgnoreFilter (set at 100) as suggested by #29.

Unfortunately, changing the plugin weights such that IgnoreFilter comes before the config_split derivative plugins would make the import process work incorrectly.

I think the way forward to resolve is to create a second filter plugin for config_ignore that handles the export. The weight of the new plugin should be less than 0.

Yes, it depends on how one expects this to work.
This complexity was the reason not to have filtering on export initially. And also why it has not been committed without tests that assert that it works as expected.
Config Ignore currently allows you to ignore for example webforms that were created on production and are not part of a config synchronisation. The ConfigImporter tells you if something is broken when some things are ignored so it is still somewhat safe.

Ignoring for exporting is another can of worms, it is totally possible to set up a "ignore split" and if you don't want the config files in your file system you can leave the folder empty and it will export it to the database (see #8). This is equivalent to the suggestion in #40 of having another IgnoreExportFilter plugin with a negative weight.

With Drupal 8.8 there is also a new API that allows us to simplify a lot of the code for that though we still need to agree on what should happen at least in the following cases from #21.4:

1. Config exists in sync folder and does not exist in the database.
2. Config exists in sync folder and exists in the database (but different).
3. Config does not exists in sync folder but exists in the database.
4. Config does not exists in sync folder but exists in the database. But only part of the config is ignored.

One other comment:

+++ b/src/Plugin/ConfigFilter/IgnoreFilter.php
@@ -192,6 +250,38 @@ class IgnoreFilter extends ConfigFilterBase implements ContainerFactoryPluginInt
+  public function filterDeleteAll($prefix, $delete) {
+    if (empty($prefix)) {
+      return FALSE;
+    }
+    return parent::filterDeleteAll($prefix, $delete);
+  }

This is probably not correct. filterDeleteAll allows a filter to trigger individual deletes for all the elements rather than the blanket deleteAll.
I think it should be:

if ($this->configuration['enable_export_filtering'] ) {
    return FALSE;
}
return parent::filterDeleteAll($prefix, $delete);

I will be available at DrupalCon Amsterdam for a sprint on this.

@bircher, not sure if this is the place for this, but I just want to thank you on all of the work you've done on Config Filter/Config Ignore, and CMI in general. There are so many projects that I have done which, without those two modules along with the paradigms and conventions that have been established pertaining to config management, would have been made significantly more difficult if not impossible to accomplish.

I, for one, am very excited about the changes coming to CMI in 8.8. Thank you!

While drush config:export works for me with the patch, I noticed drush config:pull doesn't - it still exports ignored config. I opened an issue with drush-ops about this:

https://github.com/drush-ops/drush/issues/4276

Re-roll.

I think the status of this needs updating?

Is this still relevant now that we have https://www.drupal.org/project/config_ignore/issues/3099674 or does #3099674 only apply to 3.x (and not 2.x)?

RE #46
The behaviour of 3.x is different from 2.x and #3099674: Create 3.x version using the new Drupal 8.8 API is a very different approach.
So this is still relevant for 2.x.

I think the question is.. Is 2.x still relevant now that we have 3.x?

2.x is still relevant as long as Drush 10 is a requirement. For certain environments like Acquia Cloud Site Factory, the drush version is locked at 9.x, so there's no way to update to 3.x of this module.

Config Ignore currently allows you to ignore for example webforms that were created on production and are not part of a config synchronisation. The ConfigImporter tells you if something is broken when some things are ignored so it is still somewhat safe.

I aslo wanted to filter on export, but I now so agree with this. If we want ignore configs so that changes on production don't get overwritten but still be able to export them, this is a valid use case and one should definitely account for.

Is 2.x still relevant now that we have 3.x?

As long as 2.2 is the suggested stable release, yes. I am working with clients who have security teams/restrictions that prevent us from using dev/beta/alpha etc versions of modules. Meaning I cannot just go use 3.x without a lot of red tape, so that leaves me with 2.x. Currently 2.x still needs ability to filter by drush exports and imports without needing patches... so.. yeah.

Like dorficus, I too am on ACSF and cannot use Drush 10 until they too support Drush 10 - and Acquia is promoting Config Ignore along with a secrets file on the server as a means to control site specific API keys or other sensitive configuration.

The patch in its current form may work for simple config management scenarios but it is not working when used with config split. Ignored config is still exported to the split directory, even while using config_split_ignore module. See #39 and #40 for more information. It also still needs tests.

In #40 @godotislate suggested to lower the weight below 0. As a test I lowered the weight to -1 with the latest patch but it is still not ignoring during drush config import run. I'm on Drupal 9.0.2 and Drush 10 with Config Split (It used to work prior to updating Drupal to 9.x).

I ran into an issue with #38 where ignored config would be exported with the option enabled when it did not yet exist. For example, when trying to prevent an API key from being exported in the config directory. Unsetting the element in `$data[]` when it doesn't exist in `$source[]` resolves the issue.

I was using the patch provided on #24 successfully with 2.2 version. I'm now on the path to upgrade my system to Drupal 9 and I've upgraded config ignore to 3.x-dev.

It seems that version 3.x doesn't require this patch, since it does not export by default the config files from the ignore list. Am I right?

Hello,

I am trying to write automated tests but I don't understand the error message and don't know if it will test that a file does not exist.

I attached a small diff.

1) Drupal\Tests\config_ignore\Kernel\IgnoreKernelTest::testExport with data set "test that an ignored config is not exported" (array(true), array('config_test.system'), array(array(array('Changed in the active config'))), array(array(false), array(false), array(false)), array(array(array('Changed in the active config'))))
config_test.system
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     'foo' => 'Changed in the active config'
-    0 => 'herp'
     '_core' => Array (...)
+    404 => 'herp'
 )

/project/app/modules/contrib/config_filter/tests/src/Kernel/ConfigStorageTestTrait.php:153
/project/app/modules/custom/config_ignore/tests/src/Kernel/IgnoreKernelTest.php:227

The patch provide in #54 would require a reroll as well.

Reroll the patch. Please review.

Remove the Needs reroll tag.

#61 applies for me and export filtering works after applying it. For testing this, I am using an existing codebase that was already using this feature via a previous version of the patch, which stopped applying after I updated to a more recent version of Config Ignore.

#61 working for me. Patched against the 2.3 release on core 8.8.8

Actually ignore #64. This isn't working for me after applying the patch.

I've added system.performance to my rules to ignore and checked the 'Enable export filtering' checkbox.

I can see that the above config is ready to be exported when I run 'drush config:export'

Drush version is: 8.4.0 if that makes any difference.

@blacklabel_tom,

It's possible that the drush version is making a difference. Drush 8 is supported, but not recommended for Drupal 8.4 or later.

I tried this patch on Drupal 9.1.3 and drush 10.3.6 and works fine for me.

https://www.drush.org/latest/install/#drupal-compatibility

On Drupal 9, the site frontpage is being set to /node despite me setting the value manually and having this as a config ignore value. Other items seem to be ignored (like blocks) but not anything in system.site:page:*

edit: Changing it to "system.site:page.*" fixes D9. Previous worked in D8.

Patch #61 is looking good to me, tested Drupes 8.9.13, Drush 9.7.2.

Thanks for the good work, this is a much needed patch IMO!

Patch from #61 worked great - thank you!

Patch from #61 worked for me too!

Tested, works as expected.

Yes to this. Works great.

The module description in the info file should really be updated to something like
"Ignore certain configuration during import and export."

Something is not quite right now. With the latest version of Drupal 8 and this patch, config-export and config-import keep trying to import values that have been ignored for over a year now.

edit:

Much like my previous comment about Drupal 9, I had to add these rules:

system.site:page.*
system.site:page.front
system.site:page.403
system.site:page.404

I replaced "system.site:page:*" which had worked for a very long time. Also, system.site:page.* was not working as a wildcard for those 3 values. What caused this change to happen?

Other settings I ignored with wildcard were also affected:

customtheme_1.settings:favicon:*
customtheme_2.settings:favicon:*
customtheme_3.settings:favicon:*
customtheme_1.settings:logo:*
customtheme_2.settings:logo:*
customtheme_3.settings:logo:*

edit: note that I had also just gone from 2.2 to 2.3.

Minimal change to #61.

RTBC +1

+1 for release.

Tested the patch in #75 on core 8.9.19 and drush 9.5.2

+1. Works on latest drupal 9

(deleted)

Maybe I'm misunderstanding what this is supposed to do, but drush cex is still exporting ignored configuration with this patch.

Patch does not apply to 3.0.0-beta1. Do we have to use 2.x then? Its confusing where this issue stands.

Using 2.x with the patch #75 under Drupal 9.3.3, I still see items marked 'updated' or 'created' in configuration export under Drush 11 even though they are marked as ignored. It looks like the code will include anything that existed previously in config (before ignoring it) - can that be suppressed or ignored?

edit: nm, was a bad paste into the config ignore UI.

Patch in #75 seems to work correctly.

One thing I found confusing was that existing config properties (e.g. system.site:mail) still kept its original value in the exported config file. Though an updated value (from a change in the UI) did not get exported, which is correct and what this patch tries to achieve, I somehow expected the property to be unset during the export. From what I understand that shouldn't make a difference as it would be ignored anyway during config import. It would though keep the exported config files cleaner. Maybe a candidate for an additional setting?

Apart from that, this patch looks good and solves a problem not otherwise fixable.

Patch on #75 works like a charm!

I was seeing the same issues as @KarlShea in #80 and @kevinquillen in #82. However, I realized I was running Config Filter 1.5, and after upgrading to D9 I bumped into https://www.drupal.org/project/config_filter/issues/3162628. Once I updated Config Filter things started working as expected.

Just in case anybody else runs into the same issue.

RTBC ++

#75 works great! Finally I have clean drush cex output.

RTBC +++ #75 work as expected!!

I have exactly the same issue mentioned in #80 and #82 with patched version 2.3 of config_ignore and version 2.2 of config_filter.

When I export, all the config items I have in ignore are exported, even with the tick checked to exclude from the export.

We are running version 9.2.13 (we have a vendor clash at the moment which prevents upgrading to Drupal 9.3.x).

Am I missing anything obvious here to make this work correctly and NOT export excluded config keys?

Am I missing anything obvious here to make this work correctly and NOT export excluded config keys?

#90 You need to enable the checkbox "Enable export filtering - By default, configuration values are ignored on import only. Enable to also ignore changes during configuration export operations." in the Config Ignore settings form.

Thanks @Baikho for replying. That box was checked already, but it still didn't work for me on export or import :-(

RTBC++

+1 for this. I was surprised to find this patch needed to be applied and assumed it was already going to work like that out of the box, so yay! :)

I assumed it was already going to work like that out of the box

I did too.

diff --git a/config/install/config_ignore.settings.yml b/config/install/config_ignore.settings.yml
...
+++ b/config/install/config_ignore.settings.yml
@@ -1 +1,2 @@
 ignored_config_entities: { }
+enable_export_filtering: false

I think we should change the config to enable this feature for new installs, instead of leaving it disabled.

Note that such a change would be Backward Compatible for existing installs because contents of config/install/ folder is only imported to active config when the module is initially installed. Additionally, if the settings page is accessed from existing installs, the value will remain unchecked until someone manually checks it (because the "true" value would not be present in active config).

I'm confused by the patch in #75.

In IgnoreFilter::sourceWrite it adds:

if (count($parts) == 1 && isset($source[$key])) {

}
else {
  $value = NestedArray::getValue($source, $parts, $key_exists);
  if ($key_exists) {
    // Enforce the value if it existed in the active config.
    NestedArray::setValue($data, $parts, $value, TRUE);
  }
  else {
    NestedArray::unsetValue($data, $parts);
  }
}

It is unclear to me why we copy the value into the export data if the key exists in the source. This prevents me from ignoring individual items in the export.

I think it should be simply the call to NestedArray::unsetValue.

See my added patch.

The situation confuses me more than I initially thought. Why are the proposed patches only excluding the keys from the export when they're found in the source?

I've refactored my patch so that it always removes the key from the export if it should be ignored.

The original patches confuse me even more. Why are they only excluding the value if it's already been exported before?

I've attached a patch that always unsets the value, and therefore always excludes it, if it should be ignored.

#75 works for me
Thank you

For the record, this was missing, interdiff between 75 and 99.

I am gonna remove the RTBC because my gut says we need to agree on whether 75 or 99 is the correct approach. (I really hope that we can finally close this issue soon...)

@mxr576: Thanks a lot for the interdiff!

I would like to invite everyone interested in this issue to test the 3.x branch and then get #3223748: Allow to configure different behaviors. moving to decide how strict we want to ignore things.

I tend to think that 75 is probably more correct.

-

Thank you for such a useful feature, it is very helpful and should be included in this module.
I have tested, for me it works perfectly

If anyone says "thank you, it works", please also mention what patch you used on which version. Thanks!

Patch does not apply to the 8.x-3.0 branch. The enable_export_filtering setting is now part of the module's schema, but it is set to false by default. I also can't find anywhere it is used and no way to change it on the settings form.

Any chance some of the maintainers could chime in? Maybe the setting was added in anticipation of a new release?

This issue is for 2.x in 3.x this has been implemented from the first tag on the branch. On the contrary one could not turn it off until 3.0

But now that 3.0 is released you should just update to it. If you want this patch then just updating will be enough, if you don't want it you switch to intermediate mode and clear out the export list.

@bircher My mistake and please accept my apologies if my question seemed like a complaint.

I searched for the this patch's config property and I thought I saw it in the 3.0 module's schema file, and then no references to the property in the code. However, I was incorrect and the enable_export_filtering property is not used or needed.

The 3.0 version of the module is working great, for both import and export, in simple mode. Thank you!!

I am marking this fixed in order to give everyone who contributed to this issue credits.
The 2.x branch for which this issue is will not get any further commits. And the 3.x branch now has this feature.

But a lot of patient people cared about this issue and I want to acknowledge their efforts. The only way to do this is according to the maintainer documentation is to mark the issue as fixed.

So thank you all for working on it and please upgrade to the latest 3.x release. YOur config is backwards compatible but please run the update hook as it will add a new setting where you can choose "intermediate" if you do not want this feature turned on by default (which is what the simple mode will do) (But do so via the UI the config schema is different for intermediate or advanced modes)