Problem/Motivation

As we march towards better recipe authoring tools, the next logical step in having a good default content export system is the ability to export all content of a particular entity type and bundle.

Possible uses for this:

  • Let's say you make 10 sample blog posts in a site, and you want to export all of them into your blog recipe.
  • Or maybe you set up a megamenu with 30 links in it, and you want to export that entire menu in one fell swoop.
  • You want to export a collection of image media.

I'm sure we can think of others!

Proposed resolution

Core already has all of the necessary infrastructure for this. We just need an interface to do it.

So, let's alter the existing content:export command to make the entity ID optional. That opens use cases like:

  • To export all content: content:export node --dir=content
  • To export all content, and its dependencies: content:export node --with-dependencies --dir=content
  • ...of a particular bundle: content:export node --bundle=blog --dir=my-blog

For now, let's keep requiring the entity type ID as an argument -- only one entity type at a time can be exported. Maybe in a follow-up we can expand that to automatically export all content, indiscriminately. But that feels like a separate scope.

API changes

No API changes should be necessary. Combined with an entity query, core's default content exporter is already capable of this.

Release notes snippet

TBD

Issue fork drupal-3547119

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

phenaproxima created an issue. See original summary.

thejimbirch’s picture

thejimbirch commented
Component: recipe system » default content system

Moving the default content system.

With the default_content contrib module, we could export the following types.

node
taxonomy_term
file
media
menu_link_content
block_content
shortcut

I believe all except for file could have a bundle type and would be beneficial for recipe creators.

thejimbirch’s picture

thejimbirch commented
Related issues: +#2786479: Drush comand to export all items from entity with defined criteria

Adding an non-working issue in the contrib module for reference.

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Issue summary: View changes

svendecabooter made their first commit to this issue’s fork.

wotnak made their first commit to this issue’s fork.

wotnak opened merge request !13309

wotnak’s picture

wotnak
he/him
commented

Created MR !13309 with following changes:

  • added option to export all content entities of a given type
    • php core/scripts/drupal content:export entity_type_id --dir=...
    • --dir option is required
    • if --with-dependencies option is provided, then also exports dependencies
    • for example php core/scripts/drupal content:export node --dir=./content
  • added option to export all content entities of given type filtered by a bundle
    • php core/scripts/drupal content:export entity_type_id --bundle=entity_bundle_id --dir=...
    • entity_type_id argument is required
    • --dir option is required
    • if --with-dependencies option is provided, then also exports dependencies
    • for example php core/scripts/drupal content:export node --bundle=article --dir=./content
  • added option to export all content entities
    • php core/scripts/drupal content:export --dir=...
    • --dir option is required
    • --with-dependencies option is ignored
    • for example php core/scripts/drupal content:export --dir=./content
  • when exporting multiple entities, progress bars are shown, one per exported entity type
  • when verbose output option is used (--verbose or its shortcuts), instead of the progress bar, information about each exported entity is shown
  • added some additional test cases for new export options
wotnak’s picture

wotnak
he/him
commented
Status: Active » Needs review
phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Status: Needs review » Needs work

This seems like a great start -- and bless you for writing tests!!! -- but it's a little hard to read and understand, overall, because of the complexity. I think we might want to scale back a few things, at least for now:

  • The progress bar is a nice touch, but it adds weight to this MR and would be best done cleanly in a follow-up.
  • Let's not have the ability to just export all content, regardless of entity type. Such a feature has many implications and deserves careful consideration that we probably would want to think through in another issue. So, for now, let's require an entity type ID at the very least.
wotnak’s picture

wotnak
he/him
commented
Status: Needs work » Needs review

@phenaproxima Progress bar and option to export all content entities removed.

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Assigned: Unassigned » phenaproxima

Damn you are fast. Assigning to myself to review.

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Assigned: phenaproxima » Unassigned
Status: Needs review » Needs work

Reviewed, and looks better and better -- I think this will be even better if we use a generator function to yield the entities to export, rather than trying to load them all at once and do an array_chunk() on them. There is also more verbosity than is generally needed, and some changes which don't add anything to the patch and should be reverted. But it's shaping up!

wotnak’s picture

wotnak
he/him
commented
Status: Needs work » Needs review

Pushed changes including refactoring to a generator function and added replies to all comments.

ironnuts’s picture

ironnuts commented

Here is the output of the test-only test. Seems good:

PHPUnit 11.5.39 by Sebastian Bergmann and contributors.
Runtime:       PHP 8.4.12
Configuration: /builds/issue/drupal-3547119/core/phpunit.xml.dist
F.FF...FF                                                           9 / 9 (100%)
Time: 02:22.126, Memory: 8.00 MB
There were 5 failures:
1) Drupal\FunctionalTests\DefaultContent\ContentExportTest::testExportContent
Failed asserting that '\n
 [ERROR] taxonomy_term 42 does not exist.                                       \n
\n
' [ASCII](length: 83) contains ""taxonomy_term" entity with id "42" does not exist." [ASCII](length: 51).
/builds/issue/drupal-3547119/core/tests/Drupal/FunctionalTests/DefaultContent/ContentExportTest.php:97
2) Drupal\FunctionalTests\DefaultContent\ContentExportTest::testExportSingleEntityToDirectory
Failed asserting that '\n
 [OK] The file "druplicon_copy.png" was exported to                             \n
      /builds/issue/drupal-3547119/sites/simpletest/12527224/files/export-conten\n
      t.                                                                        \n
\n
' [ASCII](length: 245) contains "One entity was exported to" [ASCII](length: 26).
/builds/issue/drupal-3547119/core/tests/Drupal/FunctionalTests/DefaultContent/ContentExportTest.php:203
3) Drupal\FunctionalTests\DefaultContent\ContentExportTest::testExportWithDependencies
Failed asserting that '\n
                                                                  \n
  The --dir option is required when exporting with dependencies.  \n
                                                                  \n
\n
content:export [-W|--with-dependencies] [-d|--dir DIR] [--] <entity_type_id> <entity_id>\n
\n
' [ASCII](length: 293) contains "The --dir option is required when exporting multiple entities." [ASCII](length: 62).
/builds/issue/drupal-3547119/core/tests/Drupal/FunctionalTests/DefaultContent/ContentExportTest.php:236
4) Drupal\FunctionalTests\DefaultContent\ContentExportTest::testExportEntitiesFilteredByType
Failed asserting that '' [ASCII](length: 0) contains "The entity type "camels" does not exist." [ASCII](length: 40).
/builds/issue/drupal-3547119/core/tests/Drupal/FunctionalTests/DefaultContent/ContentExportTest.php:379
5) Drupal\FunctionalTests\DefaultContent\ContentExportTest::testExportEntitiesFilteredByBundle
Failed asserting that '\n
                                         \n
  The "--bundle" option does not exist.  \n
                                         \n
\n
content:export [-W|--with-dependencies] [-d|--dir DIR] [--] <entity_type_id> <entity_id>\n
\n
' [ASCII](length: 218) contains "The --dir option is required when exporting multiple entities." [ASCII](length: 62).
/builds/issue/drupal-3547119/core/tests/Drupal/FunctionalTests/DefaultContent/ContentExportTest.php:416
FAILURES!
Tests: 9, Assertions: 66, Failures: 5.
Exiting with EXIT_CODE=1
smustgrave’s picture

smustgrave commented
Issue tags: +Needs change record

So did some manual testing.

On a fresh 11.x install
Standard profile install
Created a few Articles and Basic page content
Ran php core/scripts/drupal content:export node --dir=my-content --bundle=page
Verified ONLY Basic pages were exported
Did the same for Articles

Appears to be working as expected!

Tagging for a CR though.

Leaving in review for others

smustgrave’s picture

smustgrave commented
Status: Needs review » Needs work

been almost a week to keep it moving moving to NW for the CR. I'll keep an eye out for it to come back around and ping in slack

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Issue tags: -Needs change record +Needs change record updates

Rather than a new CR, let's update the existing one: https://www.drupal.org/node/3533854

nicxvan’s picture

nicxvan commented

That might make sense, want to draft the change here so it's just a copy and paste when this gets in?

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Status: Needs work » Needs review

Simplified this a bit and added more test coverage. Incidentally, now that it's wrapping around a simple entity query, we get "export all entities of a type" for free. So I added a test of that. I also allowed multiple --bundle options to be passed.

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Issue tags: -Needs change record updates

Updated the change record with usage examples for this.

phenaproxima changed the visibility of the branch 3547119-add-a-command to hidden.

thejimbirch’s picture

thejimbirch commented
Status: Needs review » Reviewed & tested by the community

Everything is green, Change record updated, even better tests, and all of entity type!

Marking as RTBC.

Go team Default Content!

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs change record updates

Committed and pushed 3bfa87d35a5 to 11.x and 44d2c86e5cb to 11.3.x. Thanks!

Can we update the CR please. Once the CR update has been done this can be set to fixed.

  • alexpott committed 44d2c86e on 11.3.x 
    Issue #3547119 by phenaproxima, thejimbirch, wotnak, oily, smustgrave:...

  • alexpott committed 3bfa87d3 on 11.x 
    Issue #3547119 by phenaproxima, thejimbirch, wotnak, oily, smustgrave:...
alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
commented
Status: Needs work » Fixed
Issue tags: -Needs change record updates

Oh @phenaproxima had already made the changes even though this hadn't been merged... risky and interesting :) ... doesn't matter now!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented
Related issues: +#3554978: ContentExportCommand::loadEntities() should use loadByProperties()

Opened #3554978: ContentExportCommand::loadEntities() should use loadByProperties().

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented
Issue tags: +11.3.0 release highlights