Problem/Motivation

Compressed files, binaries, and databases are difficult to maintain and inspect.

You can see an example here:
See the change core/tests/fixtures/config_install/multilingual.tar.gz in this MR: https://git.drupalcode.org/project/drupal/-/merge_requests/5282/diffs#di...

The only changes were to update the compressed views config but there is no way for a reviewer to see that. The only way they could see what was changed would be to extract it before the commit, after the commit and do a manual diff.

Background information

This was first reported to the Drupal security team who determined this could be a public followup.

Steps to reproduce

Proposed resolution

Remaining tasks

Here are all of the zipped files found with
find . -iname *gz

#3455818: Decompress files for update_test_new_module
./core/modules/update/tests/update_test_new_module/8.x-1.0/update_test_new_module.tar.gz
./core/modules/update/tests/update_test_new_module/8.x-1.1/update_test_new_module.tar.gz
./core/modules/update/tests/update_test_new_module/8.x-1.1/update_test_new_module.zip
./core/modules/update/tests/update_test_new_module/8.x-1.0/update_test_new_module.zip

#3455819: Removed unused aaa_update_test.tar.gz
./core/modules/update/tests/aaa_update_test.tar.gz

#3403649: Rework database update tests so we don't have to ship database dumps in git
core/modules/system/tests/fixtures/update/drupal-11.3.0.bare.standard.php.gz
core/modules/system/tests/fixtures/update/drupal-11.3.0.filled.standard.php.gz

#3455820: Decompress files for config_install testing
./core/tests/fixtures/config_install/multilingual.tar.gz
./core/tests/fixtures/config_install/testing_config_install.tar.gz
./core/tests/fixtures/config_install/testing_config_install_no_config.tar.gz

User interface changes

None

API changes

May require test process updates since current tests grab config and or databases from the repository.

Data model changes

Unknown

Release notes snippet

TBD

CommentFileSizeAuthor
#10 3455714-nr-bot.txt90 bytesneeds-review-queue-bot

Issue fork drupal-3455714

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

nicxvan created an issue. See original summary.

nicxvan’s picture

nicxvan commented

I wasn't sure which component this should be for since most of it is related to testing.

nicxvan’s picture

nicxvan commented
Issue summary: View changes

nicxvan opened merge request !8457

nicxvan’s picture

nicxvan commented

I did a really rough POC for converting the multilingual config from a tar to just a directory that gets copied just to see if things work as expected.

This still needs cleanup, discussion and work.

I'm going to mark as needs review for the approach.

I also had to rebase, not sure how it was out of sync since I just created the branch, but that should be resolved now. I'm not super worried since this is meant more as a poc.

nicxvan’s picture

nicxvan commented

Also not sure why the title of the MR pulled from a cherry picked commit. I used the ui...

I'm asking in slack I assume something went awry, still, just a POC

nicxvan’s picture

nicxvan commented
Version: 11.0.x-dev » 11.x-dev

@fjgarlin helped, I created the issue with 11.0.x rather than 11.x-dev

nicxvan’s picture

nicxvan commented

Had to move the poc further along for tests to run.

nicxvan’s picture

nicxvan commented
Status: Active » Needs review

Tests are passing, I'm setting it to needs review for process.

I'm not sure if we need change records for this, I also can see each type of file we address being a separate issue.

needs-review-queue-bot’s picture

needs-review-queue-bot commented
Status: Needs review » Needs work
StatusFileSize
new 3455714-nr-bot.txt90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

nicxvan’s picture

nicxvan commented
Status: Needs work » Needs review
Issue tags: +no-needs-review-bot

aaa_update_test is going to be much more complex since it is explicitly checking the extraction process. We likely need to take a similar process to the db where there is a separate gitlab process to generate them periodically.

nicxvan’s picture

nicxvan commented
Title: Drupal should not commit compressed files, binaries, or databases » [META] Drupal should not commit compressed files, binaries, or databases
Issue summary: View changes

I'm going to create some child issues for each type of compressed files since each type requires different approaches,

nicxvan’s picture

nicxvan commented
Component: other » phpunit
Issue summary: View changes
nicxvan’s picture

nicxvan commented
Issue summary: View changes
nicxvan’s picture

nicxvan commented
Issue summary: View changes

nicxvan changed the visibility of the branch 3455714-drupal-should-not to hidden.

nicxvan’s picture

nicxvan commented
Status: Needs review » Needs work

I hid the branch here and moved it to the new child issue that I tidied up and is ready for review now.

nicxvan’s picture

nicxvan commented
Issue summary: View changes
nicxvan’s picture

nicxvan commented

I think this issue should have a test to confirm that there are no zips or tars in the repository to confirm closure of this issue.

nicxvan’s picture

nicxvan commented
Issue summary: View changes
nicxvan’s picture

nicxvan commented
Related issues: -#3403649: Rework database update tests so we don't have to ship database dumps in git
nicxvan’s picture

nicxvan commented
Issue summary: View changes

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

nicxvan closed merge request !8457

quietone’s picture

quietone commented
Title: [META] Drupal should not commit compressed files, binaries, or databases » [meta] Drupal should not commit compressed files, binaries, or databases
Issue summary: View changes
Status: Needs work » Active

There is also, core/modules/config/tests/fixtures/not_just_config.tar.gz

Changing title per Special titles.