Export configuration YAML strings as multiline

Yes please, I had to deal with a git merge conflict on an exported webform which had several thousand characters in a single line, to say it was painful would be an understatement.

I'd absolutey love this too. We need to keep in mind though that changing this by default will cause a lot of config diffs!
I'm curious whether we could do the following things:

1. Keep the behaviour for existing sites the same by default.
2. Convert all the core configs over
3. Use the new behaviour by default

Does that sound like a solid plan?

+++ b/core/modules/user/config/install/user.mail.yml
@@ -1,28 +1,115 @@
+    [user:display-name],
+    ¶
+    A request to cancel your account has been made at [site:name].

The spaces on the blank line are also weird here. Looks tricky. Yeah folded style looks like what we want.

As I understood, Symfony doesn't support the folder style mentioned in the issue.

About spaces indentation. There is commit https://github.com/symfony/symfony/commit/81b59b9eca8af27403c1f24c28052a...
By default, indentation is 4 (https://github.com/symfony/symfony/blob/81b59b9eca8af27403c1f24c28052a96...)
This default indentation adds to the one we set on the Drupal side (https://github.com/symfony/symfony/blob/81b59b9eca8af27403c1f24c28052a96...)
Symfony adds this indentation even if an input is empty (https://github.com/symfony/symfony/blob/81b59b9eca8af27403c1f24c28052a96...)

Probably it can explain why spaces exist in black line.

Btw, there already is tool what can tidy up already exported yaml files (all, not only Webform's):
https://www.drupal.org/docs/8/modules/webform/webform-frequently-asked-q...

Setting to NW to change to folded style, or update the IS with remaining work. If symfony doesn't support folded style, is there anything to be done?

I decided to start using the Symfony YAML multiline formatter instead of my hack.

@see #2990866: Use native Symfony YAML formatter to tidy exported YAML

I am also seeing an extra return being appended to multiline values.

Instead of…

elements: |
  This is a 
  multiline value.
some_property: some_value

I am getting…

elements: |
  This is a 
  multiline value.
  
some_property: some_value

I am not seeing any major issues with the extra return but it is possible that it could cause problems for other applications.

Hi all, I've created a small module to enable this feature on configuration export: https://www.drupal.org/project/multiline_config

Thanks to all and specially to @andypost and @jrockowitz for the idea and code.

In this module I replace the encode method of the 'config.storage.staging' service to convert strings in multilines.

Also alter the 'config_single_export_form' output as @jrockowitz do in webform_devel module.

During the tests I've detected that if I manually edit the user.mail configuration (on mac), LF characters are replaced by CRLF and the conversion doesn't works.

Copy from \Symfony\Component\Yaml\Dumper::dump:

                if ($inline >= 1 && Yaml::DUMP_MULTI_LINE_LITERAL_BLOCK & $flags && \is_string($value) && false !== strpos($value, "\n") && false === strpos($value, "\r\n")) {
                    // If the first line starts with a space character, the spec requires a blockIndicationIndicator
                    // http://www.yaml.org/spec/1.2/spec.html#id2793979
                    $blockIndentationIndicator = (' ' === substr($value, 0, 1)) ? (string) $this->indentation : '';
                    $output .= sprintf("%s%s%s |%s\n", $prefix, $dumpAsMap ? Inline::dump($key, $flags).':' : '-', '', $blockIndentationIndicator);

                    foreach (preg_split('/\n|\r\n/', $value) as $row) {
                        $output .= sprintf("%s%s%s\n", $prefix, str_repeat(' ', $this->indentation), $row);
                    }

                    continue;
                }

Just to mention it, multiline_config is proving (so far) to be pretty fantastic, thanks.

We have 176 webforms, I will see what we can do with this.

As useful as this would be for us, the real solution would be patching Symfony to support the folded style and that is not something I will get involved with.

@andypost About the problem mentioned in #12: In my dup #2933070-5: Leverage symfony3 multiline literal blocks in Yaml dumps i upstreamed that to Symfony. Upstream fixed it meanwhile.
As we once forked the component, the bug may still persist, and we either must port the fix, or commit #3111008: Use native Symfony YamlFileLoader. It looks like this only is true for the loader.

Accordung to this PR this was fixed in Symfony 3.3.16/3.4.4/4.0.4.

Yay, green...

<3 great that this is fixed upstream, this is a nice improvement for Webform and similar large exports.

We still need a few things here - firstly we need to think about

• existing sites
• existing modules with config

At the very least we need a change record and a release note that after updating that people should expect changes in their config exports and that we recommend running a config export.

+++ b/core/modules/user/config/install/user.mail.yml
@@ -1,28 +1,115 @@
+    Thank you for registering at [site:name]. Your application for an account is currently pending approval. Once it has been approved, you will receive another email containing information about how to log in, set your password, and other details.
+    ¶
+    ¶
+    --  [site:name] team

This is a change. In the original configuration there is only one empty line but this patch is adding two.

Added draft change record: https://www.drupal.org/node/3114725

I don't think this affects existing sites or contrib modules, except that their config export will change when this has been released. Existing config that doesn't use the multiline syntax will still import correctly.

Removed the extra line noted in #33.

+++ b/core/profiles/demo_umami/config/optional/block.block.umami_banner_recipes.yml
@@ -28,6 +28,6 @@ settings:
-    pages: "/recipes\r\n"
+    pages: "/recipes"
     negate: false

Does this belong here?

ALl the rest looks straightforward code-wise. Changed test fixtures pass. So RTBC for me, minus the above.

Patch flying in without said change, hello bot.

Tested this manually and it's a holy bliss how many config item changes now get reviewable:

--- a/config-sync/language/de/webform.webform_options.days.yml
+++ b/config-sync/language/de/webform.webform_options.days.yml
@@ -1,3 +1,11 @@
 label: Tage
 category: 'Datum und Uhrzeit'
-options: "Sonntag: Sonntag\nMontag: Montag\nDienstag: Dienstag\nMittwoch: Mittwoch\nDonnerstag: Donnerstag\nFreitag:
+options: |
+  Sonntag: Sonntag
+  Montag: Montag
+  Dienstag: Dienstag
+  Mittwoch: Mittwoch
+  Donnerstag: Donnerstag
+  Freitag: Freitag
+  Samstag: Samstag

Note that strings containing \r\n will NOT get the bliss of this patch, only \n separated lines.

#37 works for me, thanks!

Actual code change is tiny and looks good. Using an upstream fix to provide this functionality. Yay.

The config changes look great. A preview of how awesome it's going to be when core is doing this for all multi-line config entries!

The only reason I hesitate to set this to RTBC is there are no tests or test changes in the patch. Granted, we don't necessarily need to be re-testing Symfony's own methods. But I'm slightly surprised we have 0 test coverage in all of core that cares about the format of the config export. Do we want tests for any of this?

Also, is user.mail.yml really the only config file in all of core that uses this? That's also slightly surprising. ;)

Not setting to NW nor tagging for 'Needs tests', but would love some feedback on these points before we RTBC.

Thanks!
-Derek

> Accordung to this PR this was fixed in Symfony 3.3.16/3.4.4/4.0.4.

8.9.x depends on symfony/yaml ^3.4.5
9.0.x depends on symfony/yaml ^4.4

Therefore even with minimum dependency installs this should work in all Drupal versions.

I also checked @dww's question in #41 of whether user.mail.yml is the only affected file:

$ find . -name *.yml | grep /config/ | xargs grep -l '\\n'
./core/profiles/demo_umami/config/optional/block.block.umami_banner_recipes.yml
./core/modules/node/config/schema/node.schema.yml
./core/modules/user/config/install/user.mail.yml

block.block.umami_banner_recipes.yml uses \r\n as seen in #36 - this can arguably be removed but is also out of scope.

node.schema.yml is a false positive as there is a comment referencing the "Drupal\node" namespace.

user.mail.yml is fixed in this patch.

Some observations with patch from #37 applied to 8.7/8.8 sites:

• After applying the patch a blank line is added at the end of a the multiline strings, but not to all files (though I can see absolutely no pattern)
• Shortly after I'm seeing some files being exported from active config where the last empty line is being removed (see list below) according to drush only, git does not show the change. Also many other files with that empty line remain in active config.
• On two different sites in active config dir in user.mail.yml I am seeing \r\n although it should be \n\n (at least I am pretty sure it wasn't changed on purpose). On both sites in the german translation I see \n\n which was correctly handled by the patch for one body field (cancel_confirm - the first one in the file and also alphabetically), but for the other body fields it is \r\n where nothing was done (as expected)

Also some contrib modules I came across use \r\n (markup, simple_block). Webform uses a simple \n

List of file where empty line was removed:

$ drush cex
 [notice] Differences of the active config to the export directory:
+-------------+------------------------------------------------------------------+-----------+
| Collection  | Config                                                           | Operation |
+-------------+------------------------------------------------------------------+-----------+
|             | config_snapshot.snapshot.config_sync.module.h4c_group            | Update    |
|             | config_snapshot.snapshot.config_sync.module.h4c_organisation     | Update    |
|             | config_snapshot.snapshot.config_sync.module.matomo               | Update    |
|             | config_snapshot.snapshot.config_sync.module.media_entity_browser | Update    |
|             | asset_injector.js.coo_map_dialog                                 | Update    |
|             | asset_injector.css.coo_site_overrides                            | Update    |
|             | asset_injector.css.coo_map_dialog                                | Update    |
|             | config_snapshot.snapshot.config_sync.module.user                 | Update    |
|             | views.view.media_entity_browser                                  | Update    |
|             | webform.webform.coo_survey                                       | Update    |
|             | webform.webform.organisation_contact                             | Update    |
| language.de | user.mail                                                        | Update    |
+-------------+------------------------------------------------------------------+-----------+

@anruether:
> After applying the patch a blank line is added at the end of a the multiline strings, but not to all files (though I can see absolutely no pattern)

This is the upstream bug i filed mentioned in #28/29. It will go away once we require newer symfony.

@geek-merlin worked on only D8. Now I have created another patch and its working on both D8 and 9.

Please ignore above file.

Patch is working on both D8 and 9 but test cases failed in D9.

working on it.

After applying patch #37 in drupal 9 it is working but there is blank line is added at the end of a the multiline strings, as you can see in patch-37.png.

And black line will be removed as mentioned in #45 ie . > This is the upstream bug i filed mentioned in #28/29. It will go away once we require newer symfony.

After applying patch #49 in drupal 9 working but there is no space between the multiline strings, as you can see in patch-49.png.

Attaching interdiff also.

It's hard to tell which elements are changed in a form, when you run a drush config:export --diff, since it is exported as one line, and thereby "everything" has changed. I wonder if this patch will allow Drush to do a better line-by-line comparison?

Is this functionality blocked until symfony 5?
Or could the patch be updated to fix the extra output, making this possible for symfony 4 compatibility?

Worked on this and found out i was wrong. The upstream fix only hid the problem.

Postponing on this new upstream issue.

Still needs a new stable release of Symfony, I guess this will be 4.4.19.

Also it looks like it still needs work.

The Symfony regression should have been fixed with this PR, which has been committed: https://github.com/symfony/symfony/pull/39683

@geek-merlin ¿Can we reopen this issue?

Symfony 4.4.19 is out so we can bump the minimum version of the YAML component to match.

Wow, so we are finally green? That'd be awesome.

I upgraded the CR with the '\r' limitation, to limit confusion.

As of upgrade path: This has a CR which describes the changes in the exported YAML. Nothing else changes.

As we do changes in the exported YAML quite often (mostly due to config changes, like int/bool changes, sort changes, ...), and we are all used to config export noise after updates, i do not see anything left to do.

No comment to #68, so RTBC for me.

This only helps for config without "\r", so gonna update some modules to filter that out.

To make that trivial, i opened #3202631: Add Textarea option to normalize newlines to \n. Let's try and land this together, review appreciated.

#64 is no longer applying to 9.2.x branch. Conflicts in composer.lock. Fun! ;)

Thanks/sorry,
-Derek

Working on the reroll.

Rerolled the patch for 9.2.x. Please review.

@ayushmishra206 you can update the lock file hash by doing composer update --lock

+++ b/composer.lock
@@ -570,7 +570,7 @@
-                "symfony/yaml": "^4.4",
+                "symfony/yaml": "^4.4.19",

this makes it look like we need to update the constraint in core/composer.json - which i think we do because we need bug fixes that release has.

Patch applies and is green.
The upstream library is required in the correct version (#75).
CR is prepared.
Back to RTBC.

Patch looks good but needs another re-roll.

And another..

I created a merge request to drupal project, please review it. https://git.drupalcode.org/project/drupal/-/merge_requests/499/diffs

I applied the patch and also I removed the composer changes because the latest composer version is "v4.4.21" on drupal project is higher than the current patch, see line composer.lock#L4292

            "name": "symfony/yaml",
            "version": "v4.4.21",
            "source": {

We should still raise the minimum version in composer.json even if we already ship a new enough version by default in the lock file. This ensures that a good enough version of the dependency will be installed if someone selects a different version (for example if we did min-max testing).

Right longwave, added constraint to core/composer.json and then update composer.lock

Thanks, the MR looks good to go.

Committed 677700a and pushed to 9.2.x. Thanks!

Tagging for release notes and adding a snippet.

Still seeing some new line characters in the commit in core/modules/user/config/install/user.mail.yml. Just checking whether these should be there? There are \n string sequences on lines 32, 37, 42 & 62.

Good point, and those aren't in #80, so something is wrong in the MR compared to the last patch. Reverted for now.

Fixed those \n sequences. Also cleaned up visibility.request_path.pages value in core/profiles/demo_umami/config/optional/block.block.umami_banner_recipes.yml which had trailing \r\n.

@catch

Good point, and those aren't in #80, so something is wrong in the MR compared to the last patch. Reverted for now.

For me, when I look in the Git repo, the last commit is still 677700aa263adba367edb6d38769a7a424885d0e, so the commit that committed this issue.

Are you sure you reverted it?

Indeed, no git revert made. That's why the MR is not mergeable.

Hope now all changes were made

The change to core/profiles/demo_umami/config/optional/block.block.umami_banner_recipes.yml is not necessary. If you install umami and then export the config with this change the /r/n is still there. So there's no need to change it.

The \r\n issue in config driven by textareas has a followup in #3202796: Default to normalize newlines in "text" config to \n, as per #96 we don't need to touch that here.

Reverted the \r\n change in core/profiles/demo_umami/config/optional/block.block.umami_banner_recipes.yml.

That looks better.

Committed/pushed to 9.2.x. Thanks!

This is a really useful improvement, for example when using drush config:export --diff to check if exported config looks correct. Example from webform.webform_options.titles.yml, you can now see the actual change much easier:

BEFORE

-options: "Miss: Miss\nMs: Ms\nMr: Mr\nMrs: Mrs\nDr: Dr\n"
+options: "Miss: Miss\nMs: Ms\nMr: Mr.\nMrs: Mrs\nDr: Dr"

NOW

 options: |-
   Miss: Miss
   Ms: Ms
-  Mr: Mr
+  Mr: Mr.
   Mrs: Mrs
   Dr: Dr

Big thanks to everyone who helped build this.

Should you want to use this in an earlier version of Drupal on a site managed with composer (made with drupal-composer/drupal-project or drupal/recommended-project), you could add "symfony/yaml": "v4.4.21 as 3.4.5" to the project root level composer.json and run composer update symfony/yaml. Then patch core/lib/Drupal/Component/Serialization/YamlSymfony.php and you are done. The API surface of this library is small and the consumers are few so I think going from v3 to v4 should be fine. 1c9842d189 introducing Symfony v4 didn't need any YAML related changes.

I'm seeing a small issue when exporting a yaml file that has multiple lines that is the end of a yaml config file. I ended up with:

data: |-
  label: Foo
  url: https://example.com
  body: baz

really should be:

data: |-
  label: Foo
  url: https://example.com
  body: baz
LF

Otherwise git complains and gives:
No newline at end of file

Re #105, here's a patch for drupal/core 8.9.14.

Also note that if the config contains return characters ("\r"), the exported config will not be in multi-line format. Sad face.

@DanChadwick For #109 see \Drupal\webform\Utility\WebformYaml::normalize

yes we probably need to normalize this, still an issue when \r is in there..

as a workaround I manually did a search replace for \r with nothing

had a curious issue where even with that yml file correctly formatted and re-exported, my custom module crashed on the same error having the culprit very large webform yml in config/install (import on install)

so I moved it to config/optional and used custom code to force load it (force import config yml) and curiously, it WORKED correctly without the sql error.

Of note PostgreSQL v12 has no issues either way in this use case, it was only MySQL v8.0.28 that I tested to have an issue (didn't try with others versions yet).

#3271926: mysql 1406 Data too long for column 'source' at row 1 insert into locales_source
(example contrib module)

Today a colleague reported a problem saving a webform translation and there's no drupal error message, no mysql error message, possibly some caught exception with no error. Not sure if their code base is tainted or not or badly patched hard to say.

I have noticed generally that for some reason webform module is still outputting \r\n even with latest webform module. This means very ugly git diffs (hard to read) and single line yml which is offensive to mysql when very long lines are trying to save to blob in the locales table.

Using recent core 9.3.16, core seems to allowing this, mysql doesn't like it on really big amounts of config, postgresql doesn't get phased as it allows larger blob or uses a different blob type.

https://www.drupal.org/project/drupal/issues/1885192#comment-14657855

Previously I had noticed an issue on import that I was able to manually resolve by replacing the \r\n with \n however colleague showed me a new issue where the save failes in the UI interface without error message in drupal, no mysql error message either.

I hope one day Microsoft will standardize on \n even though I'm not using Microsoft somehow their choices seem to linger on and annoy us regardlessly.

I'm tempted to suggest that Drupal core switch to a mediumblob , might really make the locales table a LOT bigger but at least MySQL wouldn't choke on >64k config.

or switch to Postgresql, which amazingly is actually seeming working mostly well with Drupal these days (some issues I've had with it as well but this blob issue on MySQL is a real annoyance).

these days I've been making some very large webforms that require translation.

@jrockowitz , @DanChadwick , I'm confused why the core fix is in yet the webforms module is still generating \r\n and why config yml webform translations are still extremely long single line processed strings when using the GUI interface.

a painful workaround for the webform module is to manually massage the .yml file so that it's multiline and then import that new webform translation with drush.

I've tried forcing cores filtering to do the replacement for webforms automatically but it seems that somewhere else the filtering is incorrect.

This is a bit of a deep dive issue, I'll keep digging but it's been a 5 year old issue ongoing.
 #2836206: [Translations] Translation of big webforms broken

Note: postgresql doesn't care about the length of the single line string , it's blob type handles rediculously long strings , this issue affects mysql

Note that this problem happens in core if you edit and save the User settings and check useruser.mail.yml

I think if there are any blank lines in your yml it will cause the whole file to use single-lines, and that's https://www.drupal.org/project/drupal/issues/3244757 - as best as i can see.

There is a SLIGHT chance you're not using the symfony yaml parser this could happen https://www.drupal.org/node/2769555#comment-11854900 <-- but that should be default.