Configuration management shouldn't export custom blocks as currently it will result in broken block.

A custom block is made of two entities, one for the placement and one for the actual content. Only the actual placement can be exported with cmi. The content can not.
Therefore this will result in "Block description Broken/Missing" error on site where the config is imported. And since there is no option to disable custom blocks from being exported through Configuration management this will brake the functionality.

Steps to reproduce

On Site A:

  1. Create custom block
  2. Assign this custom block to any region
  3. Export configuration of the site

On Site B:

  1. Import configuration from site A
  2. Go to Block layout and you will see custom block in correctly assigned region, however block won't work or actually show anything.
  3. Edit that block, you will see this messages:
    • Block description Broken/Missing
      This block is broken or missing. You may be missing content or you might need to enable the original module.
  4. Go under Custom block library and you won't see custom block.

Block layout got exported but block content didn't resulting in broken relationship.

Suggested solution

Don't export custom blocks through Configuration management.

Files: 
CommentFileSizeAuthor
#8 hello_world.tar_.gz570 bytesressa

Comments

info@websolutions.hr created an issue. See original summary.

cilefen’s picture

Title: Don't export Custom blocks » Custom blocks cannot be properly exported and imported
Component: block.module » configuration system
larowlan’s picture

Status: Active » Closed (works as designed)

This is how it was designed to work.

If you deploy your content with rest endpoint or deploy module, entity pilot module or use an update hook to create the block entity with the same uuid- it will repair itself.

I'm using this feature on several production sites and loving it.

tamerzg’s picture

Status: Closed (works as designed) » Needs work

Sorry but I don't agree this can be closed. Clearly its a usability issue since it results in Broken/Missing dependencies. I don't think solution is that user should know they need to use another contrib module (which seems to be in alpha!) in order to get this working.

morsok’s picture

Have to agree with @tamerzg here.

I think there would be two solution to this problem, either :
- Remove the export of the block placement (instance) from the config export
- Export the content of the block since it's so tightly coupled with the configuration

It's a real pain for developers new to Drupal 8 to figure out a way to deal with this situation when your first deployment miss the content and you get broken blocks all over the place. The documentation is clearly not cutting it and everyone seems to have a different way to deal with that. Maybe it's designed like that, but is it a good one ? There is room for reflexion here, I truly feel this is a bad DX. Maybe there is other solutions, maybe a clear documentation (cookbook style) would help ?

But in this state it's just plain frustrating and does not really feel 'finished'.

ressa’s picture

Thanks for weighing in @larowlan. I understand that content can't be part of the export > import workflow, but which of the solutions you outline would you recommend as the easiest and most foolproof? Do you have a simple example of an update hook which creates a block entity and adds some content to it?

ressa’s picture

To answer my own question, it's actually not too complicated to create a custom module which adds a block with some text. Just follow the instructions at Create a custom block.

ressa’s picture

FileSize
570 bytes

Here's a working "Hello Block" module example. NOTE: I am not sure what happened, but you need to rename the file to hello_world.tar.gz to be able to extract it.

  1. Enable the "Hello Block" module
  2. Assign the custom "Hello Block" block to a region
  3. Export configuration from site A
  4. Import on site B
  5. The block appears on site B, with the content from Site A's "Hello Block" module
larowlan’s picture

Perhaps you want https://www.drupal.org/project/simple_block then, which uses config to store the blocks instead of content.

jonathanshaw’s picture

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mlncn’s picture

Version: 8.2.x-dev » 8.1.x-dev

One other possible approach is to use the Default Content module to export the content. It's built for the content to be exported to an installation profile's 'content' folder, and then the module, if enabled, automatically brings the content in when the site is installed. Presuming that reinstalling the site would be a drastic step for getting one block's content (aside: we do find it useful to use installation profiles with default content for automated testing), it's also possible to import the content one item at a time, such as in an update hook, with the below code in your example.install or example.profile:


/**
 * Import a piece of content exported by default content module.
 */
function example_import_default_content($path_to_content_json) {
  list($entity_type_id, $filename) = explode('/', $path_to_content_json);
  $p = drupal_get_path('profile', 'guts');
  $encoded_content = file_get_contents($p . '/content/' . $path_to_content_json);
  $serializer = \Drupal::service('serializer');
  $content = $serializer->decode($encoded_content, 'hal_json');
  global $base_url;
  $url = $base_url . base_path();
  $content['_links']['type']['href'] = str_replace('http://drupal.org/', $url, $content['_links']['type']['href']);
  $contents = $serializer->encode($content, 'hal_json');
  $class = 'Drupal\\' . $entity_type_id . '\Entity\\' . str_replace(' ', '', ucwords(str_replace('_', ' ', $entity_type_id)));
  $entity = $serializer->deserialize($contents, $class, 'hal_json', array('request_method' => 'POST'));
  $entity->enforceIsNew(TRUE);
  $entity->save();
}

Export a custom block with an ID of 8:

drush dcer block_content 8

And used in your example.install file:

/**
 * Add the footer block content.
 *
 * Implements hook_update_N().
 */
function example_update_8001() {
  example_import_default_content('block_content/136efd63-021e-42ea-8202-8b97305cc07f.json');
}

See add content from the default content module in an update hook.

cilefen’s picture

Version: 8.1.x-dev » 8.2.x-dev

The 8.1.x branch is closed for bug fixes as described above.

ressa’s picture

To be able to place easily editable content via the block system within a dev > stg > prod workflow using drush config-export and drush config-import, you could create a "Text blocks" view with several blocks, and filter on "Title: starts with" and "Content type=Basic Page".

For example, to place a block on a Subject lists Views page, put some helper text in the blocks' "Empty view" field, like Create a page with Title "Subject list", and this in the Body field: "SOME CONTENT FOR THIS BODY FIELD". This requires of course that only a single Basic Page starts with the title "Subject list". Place the block in the region, and decide on which pages it should be shown.

Only show the Body field and an edit field, so that the editors have a link to edit the content of these texts.

This way a dev > stg > prod workflow is possible, keeping the content placed in blocks AND giving editors the option of updating the content, which will not be overwritten by a staging process like fx:

  # DEV
  drush @DEV-alias config-export
  git add -A :/
  git commit -m "added som stuff"
  git push

  # STG
  git pull
  drush @STG-alias config-import --preview=diff

I admit it is a bit clunky, but it seems to do the job. Also, this is really only an option if there are no more than a handful of pages.

icurk’s picture

One approach is to create update hook in which you create custom block programmatically. First, you must get uudi of your custom block and then on the creation of the custom block in the update hook you manually set value for uudi (you can also set values for other fields in your block). On site B custom block will be created with the same uuid as it is on the site A and you will get no error for "Block description Broken/Missing". Custom block on the site B will then be editable and nothing will be overridden in the next configuration import.

My only concern here is how safe it is to manually set uuid of the custom block and what are the chances that this will result in an error of duplicated uuid.

hoporr’s picture

I agree with tamerzg and morsok above: this issue needs to be fixed here in core.

I know how to programm custom blocks, but the point is that these blocks here are created using core's custom-block GUI.

So, like morsok said, either drop this placement-config from the sync altogether, or copy the content, or maybe create some block with a content-standin (which then could be changed manually on the target site) so that at least the block does exist -- but don't break the target site by placing a non-existing block!

I'll try one of the contrib modules for now, but I see these as workarounds to a fundamental flaw here.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

geerlingguy’s picture

Issue summary: View changes

This is probably the number one minor annoyance I've had with a pure CMI workflow in Drupal 8. It just feels wrong to have to either:

  1. Create a block on prod.
  2. Pull back the prod database.
  3. Place the block locally.
  4. Export local config.
  5. Push config to prod.
  6. Import config on prod so block is finally placed and won't get un-placed on future CMI full imports.

Or:

  1. Create a block locally.
  2. Place the block locally.
  3. Write update hook that creates the block again, but with a custom UUID (this just feels wrong).
  4. Export local config.
  5. Push config to prod.
  6. Run database updates on prod (this creates the block).
  7. Run config import on prod (this places the block).

I feel like we should either completely exclude custom blocks from CMI, or take the approach of the Simple Block module in core, for custom blocks...

kpv’s picture

An entity (block) could be created automatically when importing a config (which may be not a trivial task),
or we could add an option to set uuid while creating an entity (block) manually, so after importing a config the consistency could be restored.

hoporr’s picture

Should this issue not be elevated to 'critical' to get some more love from the powers to be?

By definition:
https://www.drupal.org/core/issue-priority
ciritical is when there is no workaround.

And in this case, the suggested "workaround" drags on major production environments, let alone it is not really feasible for extremely large DBs ( do you really want to install a 1GB+ DB to just fix a block? Try selling that to an enterprise customer, Drupal).

jonathanshaw’s picture

To quote more fully, to be critical an issue needs to "Render a site unusable and have no workaround".

This issue does not render sites unusable. Much as it's a big PITA for everyone trying to have good deployment workflows, it doesn't met critical criteria and escalating to that would just annoy people.

bdanin’s picture

Custom blocks conflict with config synchronization at this point, which creates a fundamental usability problem:
option 1: if using config-sync, then disable custom blocks from the site (or remove permissions so they cannot be created)
option 2: diligently manually config_ignore all newly created custom blocks (unsustainable on an active site with many users)

Perhaps a module could be written that would automatically add any custom block into config_ignore settings, which might make option 2 above more workable.

The problem: add and place a custom block, and it will be added to config on config-export. When importing into production, the new config will create a broken block in production, unless this block config is ignored through some manual workaround.

The only way to avoid this is to fully sync the database (eg: stage to production.) For most sites, this won't work.
Alternatively, we might back-port the site and database backward from production to stage or local, make config updates, sync back up to production. This still will not work because any new custom blocks created in this interim period on production will now be deleted on import.

To test a work-around, I tried to first create a custom block in local, immediately create the same custom block in stage (so they have the same machine name). However, when you run config-import, the config sync takes priority of the machine name. My newly created stage block is re-assigned a new machine name.
When I sync the config from local to stage I now have two configs for different blocks in stage, and the first one is a broken block placed into the site.

Here is a specific example:

Create and place a footercopyright custom block in both local and stage by copy-pasting the title and body.
Import the config into stage, and the footer block is replaced with a broken block.
Manually fix the block config by deleting the broken block created by config and re-place the originally created custom block.

Now, the config export from stage:
block.block.footercopyright_2 create
block.block.footercopyright delete

or re-import the original config:
block.block.footercopyright create
block.block.footercopyright_2 delete

Using another module, like simple_block, doesn't work for my current needs, and ideally core components such as these should not conflict.

In short, I think this should be upgraded to critical, because until it's fixed, it doesn't seem like we can use both config synchronization and custom blocks at the same time. If you have both of those things enabled on a site, then only a highly experienced Drupal developer should be implementing custom blocks, at which point it's probably best to write custom blocks in custom modules.

geerlingguy’s picture

@bdanin - While it is a major issue, and highly annoying, it doesn't meet the criteria for being 'Critical' (e.g. it's not bringing down people's production sites right now), especially since there are many ways to work around the problem for now (though none of them are very fun).

hoporr’s picture

Well, short of going to critical, there now is a wish list for Drupal 8.4:
https://www.drupal.org/node/2858592

If this matters to you, you may want to chime in there.

larowlan’s picture

If this matters to you, scream there. I did.

As I said in comment #3 this is 'works as designed'

When I was designing the block content to block module integration, my major bug bear was the way D7 tied it to the serial entity ID, e.g. the block equivalent of NID. I fought hard to retain IDs driven by UUID, knowing that you can deploy/create an entity with a UUID.

Deploying content between environments is not a core issue.

We got the 'broken' handler in so that your site doesn't blow up, there are several ways to deploy content in contrib and failing those, you can use an update hook like so (copied from an actual D8 site, works for *any* entity type:

/**
 * Creates the required block for the Footer contact details.
 */
function my_install_profile_update_8001() {
  $default_content = [
    // Keyed by entity type.
    'block_content' => [
        // Then by UUID.
        'b990e653-c29a-48d8-b90b-70e2676f6c6e' => [
          'info' => 'Contact us',
          'type' => 'contact_block',
          'field_something' => 'put your field values here',
        ],
      ],
    ];
  my_install_profile_create_content($default_content);
}

/**
 * Creates install content.
 *
 * @param array $content
 *   Content keyed by entity-type and UUID.
 */
function my_install_profile_create_content(array $content) {
  foreach ($content as $entity_type_id => $items) {
    $storage = \Drupal::entityTypeManager()->getStorage($entity_type_id);
    foreach ($items as $uuid => $item) {
      $entity = $storage->create($item + ['uuid' => $uuid]);
      $entity->save();
    }
  }
}

Finally

they are taking the wish list for Drupal 8.4 right now

- comments like this make me sad. There is no us vs them, there is no 'core devs' vs 'users', we are all drupal developers. By commenting here, you are taking part in the process and I resent the implication of there being a they, it implies there is a special class of users, there is not.

larowlan’s picture

I note there is also a ConfigImportEvents::MISSING_CONTENT event fired, which you could also react to instead of using an update hook, work is afoot in default_content module to support that.

But again, the presence of this event in the config API support my assertion that this is works as designed, and up to contrib to craft solutions for it.

gambry’s picture

I agree with @larowlan in #3 and #26.
Drupal core already gives all you/contribs need to deal with the problem, and ConfigImportEvents::MISSING_CONTENT is a perfect example (of many, i.e. update hooks).

Besides I don't understand the wonder. D7 (+features) works exactly in the same way and you need other contrib modules (i.e. FE Block, uuid) or update_hooks to create your missing blocks if placed, for instance, in panels or contexts.

It's clear there is a missing piece of the puzzle, but that shouldn't be found in core. And may not be code too.
In fact this thread has a lot of useful suggestions which require only a documentation page. :)

+1 for closing this issue.

hoporr’s picture

My issue is not that a custom block cannot be exported/imported, but that at present the resulting target system shows a broken block after import.

Now, if everybody can program hooks, and we want to write a new update_N hook for every custom block (do we?), we can fix that in code. However, as we are all aware, not all users can do that; for them the out-of-the-box solution impairs usability.

So if we define a custom block as being content (fine), as #3 and other argue, AND we want to keep custom content out of the configs (fine), then the question is: why is the placement info for that block exported at all? Why export ANY info about something that is deemed to be content?

If you really want to be consistent here as far as semantics, at least to me the logical conclusion would be to not write that info into the config at all. I'm happy with that.

But since core does include that config at present, one could argue as well for the other solution: you could say that the content is only what is really inside of the block; and the existence of the block is structural info that is part of the greater system. In that case, the solution would be, as presented above, to create a stub block.

Now, we can belabor this point forever, where do you draw the line between content and structure. I just want to be pragmatic, and get this going.

For that reason, I'd vote for just dropping the config -- it seems cleaner -- and push everything else dealing with content into the contrib space.

bdanin’s picture

I agree that new custom blocks should not be exported into config by default.

For now, I'm using a custom (in code) block similar to #8, along with config_pages, and then rendering fields with config_pages_config()
see specifics on rendering at stackexchange

When I export all my new config, fields in config_pages are created as empty, and my new block from the module code is enabled and placed in the right place. Then content authors can go into the config_pages UI and add content as expected.

Anyway, this method gives me exactly what I need, but it's not very straight forward nor easy for a novice. However, I'm not sure there is a simple core-based solution other than simply removing custom blocks from config-export, which supports the originally suggested solution. I now agree this is best:

Suggested solution

Don't export custom blocks through Configuration management

ravenstar’s picture

I'm grappling with this problem in my workflow as well. I'm very unsatisfied with the "works as designed" response as well as all the suggestions to write custom code to handle block creation or to omit blocks from the config export. Very simply, why can't empty block content be created when a new block's configuration is imported? I have not yet dug deep enough down into the import mechanism to know how to do this, but it seem obvious that this is far more desirable than either creating broken and unchangeable blocks or omitting blocks entirely and therefore omitting a very significant portion of a site's configuration. This isn't even any different from the behavior for existing blocks where content changes are not copied.

I'm sure the "works as designed" responses never intended to imply that creating broken blocks was an intentional design choice. I'll probably dig into this fix for my site, but if anyone wants to supply some advice to save me some time, I'd happily accept it and offer the patches back to community. Naturally I'm about to go on vacation so this will have to sit for a while, but perhaps we could mull over the implications of this change?

geerlingguy’s picture

Why can't empty block content be created when a new block's configuration is imported

This actually seems like a more reasonable solution to the problem, while keeping things the way they are.

The one difficulty with that approach is that there are myriad possibilities for fields, required fields, etc., and whatever system would create an empty block would need to account for that (akin to Devel Generate).

That would then create the problem of people not knowing why a block is empty on prod when they had content in it locally, though :/

kpv’s picture

Adding this just for reference https://www.drupal.org/project/recreate_block_content. Haven't tried it by myself yet.

jonathanshaw’s picture

Thanks @kpv, that's an interesting find.

It's a very simple module, it uses

          $block_content = BlockContent::create([
            'type' => $type,
            'info' => $block->id(),
            'uuid' => $uuid,
          ]);

so it doesn't attempt to account for field content in any ambitious way.

My dream for something more ambitious would be to serialize the content blocks on export, and then on import creating a new block from the serialized block, if a block with that uuid is not already present.

Even just exporting a label or other hint of what that block's content should be would be helpful.

simon.westyn’s picture

Thanks for sharing @kpv, works perfectly

Watergate’s picture

Thanks @kpv, module suits my use case.

mikeohara’s picture

Based on comments above, and my expectation, +1 to close

Joao Sausen’s picture

I agree that we should not export content at all, contrib modules (like recreate_block_content) should deal with it. +1 to close the issue.

jonathanshaw’s picture

Let's not close the issue but instead repurpose the issue into: document patterns for exporting custom blocks
To capture some of the strategies discussed in these comments.
It's clearly an important and difficult issue, even if the solution is not a core code patch.