Problem/Motivation

Diffs with different line endings lead to InvalidArgumentException : Invalid $mode 3 specified

This is because \Drupal\Component\Diff\DiffOpOutputBuilder::hunkOp does not handle \SebastianBergmann\Diff\Differ::DIFF_LINE_END_WARNING

Steps to reproduce

See unit test in MR.

Can be reproduced in diff module as per this test

Proposed resolution

Ignore line endings or handle them gracefully.

Remaining tasks

Check how line endings were handed in 10.0 and below.

User interface changes

API changes

Data model changes

Release notes snippet

Issue fork drupal-3389715

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mstrelan created an issue. See original summary.

mstrelan’s picture

Title: DiffOpOutputBuilder::hunkOp does not handle Differ::DIFF_LINE_END_WARNING » Diffs with different line endings leads to Invalid $mode 3 specified
Issue summary: View changes
Status: Active » Needs work
mstrelan’s picture

Status: Needs work » Needs review
mstrelan’s picture

acbramley’s picture

Status: Needs review » Reviewed & tested by the community

Simple fix, correctly fixes the failed test and fixes our issues on a client project.

larowlan’s picture

Status: Reviewed & tested by the community » Needs review

Left a question on the MR

mstrelan’s picture

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Restoring status.

mondrake’s picture

Status: Reviewed & tested by the community » Needs work

Thanks for working on this. It seems my initial implementation fell short on a number of things. Sorry for that.

The good thing is that we are adding test coverage for cases that were totally untested in the prior implementation - a more reliable pattern against possible future regressions.

In a couple of similar issues earlier, #3386482: DiffOpOutputBuilder does not correctly match the 10.0.x implementation and produces unexpected output and #3386707: DiffOpOutputBuilder does not correctly match the 10.0.x implementation and produces unexpected output (part 2), @alexpott also added the test case to the @legacy test for DiffEngine, so to show that the change is consistent with earlier behavior. I think it should be done here, too. NW for that.

mstrelan’s picture

Status: Needs work » Needs review

Added test coverage to DiffEngineTest

mondrake’s picture

Status: Needs review » Needs work

OK, so now we have the regression fixed.

Since this is making a 'change' out of two exactly same lines that only differ by the EOL (being UNIX-like or WINDOWS-like), I was wondering: this is technically correct, but for a human it would make no difference. So I checked usage of Diff, and I find one only in core, where the strings are certainly stripped of the '\n' (but I am unsure about the '\r'), BEFORE they are sent for diffing. So this one might be an edge case. I think that would deserve some comments somewhere re. how the array of strings passed to Diff should be preprocessed for EOL markings.

From ConfigManager:

  /**
   * {@inheritdoc}
   */
  public function diff(StorageInterface $source_storage, StorageInterface $target_storage, $source_name, $target_name = NULL, $collection = StorageInterface::DEFAULT_COLLECTION) {
    if ($collection != StorageInterface::DEFAULT_COLLECTION) {
      $source_storage = $source_storage->createCollection($collection);
      $target_storage = $target_storage->createCollection($collection);
    }
    if (!isset($target_name)) {
      $target_name = $source_name;
    }
    // The output should show configuration object differences formatted as YAML.
    // But the configuration is not necessarily stored in files. Therefore, they
    // need to be read and parsed, and lastly, dumped into YAML strings.
    $source_data = explode("\n", Yaml::encode($source_storage->read($source_name)));
    $target_data = explode("\n", Yaml::encode($target_storage->read($target_name)));

    // Check for new or removed files.
    if ($source_data === ['false']) {
      // Added file.
      // Cast the result of t() to a string, as the diff engine doesn't know
      // about objects.
      $source_data = [(string) $this->t('File added')];
    }
    if ($target_data === ['false']) {
      // Deleted file.
      // Cast the result of t() to a string, as the diff engine doesn't know
      // about objects.
      $target_data = [(string) $this->t('File removed')];
    }

    return new Diff($source_data, $target_data);
  }

Also, I think additional test cases should be provided for DiffFormatterTest, that is currently only testing array of strings free from any EOL markings.

eason xu’s picture

moshe weitzman’s picture

It seems like the additional tests are not coming quickly. Would maintainers consider fixing the bug without tests? Otherwise we are forcing everyone to live with a bug because we are worried that the bug might return one day.

micahw156’s picture

For what it's worth, we've been running the patch from #3 above in production for nine months with no ill effects.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

atropoides’s picture

StatusFileSize
new1.28 KB

Rerolled for D11.3

rclemings’s picture

The patch from MR4877 won't apply because /core/tests/Drupal/Tests/Component/Diff/Engine/DiffEngineTest.php doesn't exist in current Drupal 11.3.12. It also looks as if the patch for /core/tests/Drupal/Tests/Component/Diff/DiffOpOutputBuilderTest.php doesn't apply, although there are no errors (i.e. the code is unchanged after the patch with DiffEngineTest.php removed is applied).

mstrelan’s picture

Status: Needs work » Needs review

Agree with moshe in #14 and don't have much motivation to continue on this issue so I had claude assist to get it finished.

Rebased on main, the DiffEngineTest hunk is gone as that file was removed in #3424436: [11.x] Remove deprecated code from the Diff component. Updated the tests with more cases.

The IS is asking how this worked on 10.0 and below. The old DiffEngine had no line ending detection, mode 3 never occurred until #3337942: DiffEngine is deprecated and replaced with sebastianbergmann/diff.

Apparently ConfigManager cannot trigger this, as Yaml::encode escapes CRLF, so only direct callers can trigger it.

longwave’s picture

Status: Needs review » Reviewed & tested by the community

This looks correct to me, the new tests look more than enough, thanks!

amateescu’s picture

Version: main » 11.4.x-dev
Status: Reviewed & tested by the community » Fixed

Committed and pushed cc28583dbe1 to main and e5dd13300b2 to 11.x and aa675a94345 to 11.4.x. Thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

  • amateescu committed aa675a94 on 11.4.x
    fix: #3389715 Diffs with different line endings leads to Invalid $mode 3...

  • amateescu committed e5dd1330 on 11.x
    fix: #3389715 Diffs with different line endings leads to Invalid $mode 3...

  • amateescu committed cc28583d on main
    fix: #3389715 Diffs with different line endings leads to Invalid $mode 3...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.