layout_builder__layout_section column hitting database limit

Thanks for opening the issue! I'm not sure of the repercussions either, but seems like a reasonable change

Note that Layout Builder is entity type agnostic, so this will need to be generic and not tied to nodes

Hmmm, in doing some more research this might be way more tricky.
FieldType plugins can be used by much more than just entities. Others include Field Collection, Dynamic Entity Reference, Address, Views Reference, etc.

See #2339855: Handle changes to the schema provided by a field type and #937442: Field type modules cannot maintain their field schema (field type schema change C(R)UD is needed) for more.

I've asked @berdir for some guidance here.

Here's an example of this from contrib:
https://cgit.drupalcode.org/date_recur/commit/date_recur.install?id=8039...

Crediting @larowlan for the pointer

And @dpi wrote that code

This is terrifying stuff.

Still needs tests, but here's a new version based on the model that date_recur provides. (It's practically copy-and-paste.)

#23 is setting the new schema to use "normal" size, it should be "big"

Also let's not lose the change to core/modules/layout_builder/src/Plugin/Field/FieldType/LayoutSectionItem.php from the first patch

Right you both are. Corrected in this patch, plus another small mistake which I think broke the tests.

Hmm, do we really need longblob though?

A blob is 64kb = 0.064 mb
A mediumblob is 16mb.

That's already a 24 900 % increase.

@dpfitzsi Having trouble reproducing your bug.

Inline blocks are not serialized into that field, so I'm not sure why 10 blocks would overflow 64k
Can you share more information about this?

I can confirm this bug by having 12 custom blocks with 50 configured fields apiece in layout builder.

I know that sounds a bit over the top, but consider, many custom modules might ship blocks with that level of configuration that's dependent on context etc. In my situation, the blocks are being created by importing JSON schema, which may include many references to sub-schema. The block's interface makes it easy to add this amount of specific configuration to each instance.

You can work around it by doing the following:

• Writing custom code for your block that doesn't save defaults - that's an issue if you want the blocks to remain unchanging with their saved default settings when versions of the module / source change
• Ensuring there is no content in the config data you are saving - sometimes this can be impossible, depending on the use case
• Implementing a new system for block saving for your module

All of these are labor intensive for module developers vs:

1. Updating the field size to allow 16mb+ entries
2. Setting layout builder to not store block config in a single field and instead split it into regular block storage mechanisms

We're hitting this limit on a large university site.

The offending node had a revision whiched crashed the site when viewed. The current revision has 114 components, weighing in at 61 kB of data in the Layout builder field.

Providing some example data:

s:36:"74d1e195-5b37-46c0-9d81-0befc489e6fd";O:38:"Drupal\layout_builder\SectionComponent":5:{s:7:"*uuid";s:36:"74d1e195-5b37-46c0-9d81-0befc489e6fd";s:9:"*region";s:5:"first";s:16:"*configuration";a:8:{s:2:"id";s:19:"inline_block:editor";s:5:"label";s:36:"21f73fa9-aafb-4f14-a9d8-81869ea854ee";s:8:"provider";s:14:"layout_builder";s:13:"label_display";b:0;s:9:"view_mode";s:4:"full";s:17:"block_revision_id";s:5:"27055";s:16:"block_serialized";N;s:15:"context_mapping";a:0:{}}s:9:"*weight";i:0;s:13:"*additional";a:0:{}}

This is a single block placed. This example weighs 520 bytes. Multiplied by 114 equals 59280 bytes.

Setting status back to Needs work. If you think this extreme case is out of scope, feel free to change back.

The patch in #27 did not properly update the schema in my tests. I had to change to:

$schema_key = "$entity_type_id.field_schema_data.$field_name";

Added a related issue where large Charts Blocks are not able to be added through Layout Builder. In my case, 'size' => 'big', would definitely be my preference.

I applied the #38 patch in 9.1.10,but it was missing a reference.Now I uploaded a new one.

Patch from #42 fixed an issue for us.

Please ignore the patch I'm attaching here, it's for reference from composer. It contains just the changes needed at runtime, without the hook_update_N(). Could be useful for people who wants to avoid the nightmare with update numbers until the fix is committed to the upstream project. It assumes that you've copied the hook_update_N() from the #42 patch to a custom module.

Looks like no one addressed #28:

Hmm, do we really need longblob though?

A blob is 64kb = 0.064 mb
A mediumblob is 16mb.

That's already a 24 900 % increase.

A medium blob is 256 times larger than a blob. Not sure where the 24 900% number comes from. But in any case, I think you bring up a good point. Do we need to account for someone storing more than 256 times the amount of data than is currently allowed?. No, I don't think so. Going to a medium size seems totally fine.

However, Drupal does not currently support medium blob sizes (see #3118014: Additional blob field sizes in schema definitions), so we cannot go to medium.

I looked into the implications of using the different sizes. I don't know about other databases, but MySQL will use an extra 2 bytes of storage for each record when using a longblob vs blob. Presumably the same is true for MariaDB. Not sure on Postgres.

So, let's assume there are sites our there with some ridiculous number of entities that use layout builder. Let's say 10 million. That means increasing to a longblob from blob would add just about 10 MB storage requirements to the database. Not a big deal.

FYI: I also opened #3267444: To reduce database size, convert layout_builder__layout_section column to a hash of the content for discussion, but please don't let that derail this issue. Even if we end up wanting to do that issue, I don't think that should stop us from changing the column size in the meantime if it makes sense to do so.

I can confirm this bug by having 12 custom blocks with 50 configured fields apiece in layout builder.

I know that sounds a bit over the top, but consider, many custom modules might ship blocks with that level of configuration that's dependent on context etc.

Is it only the context mappings that are in the serialized section data, or is there other stuff that's stored in the section data for each block field as well? Per #29, the block fields themselves should not be getting serialized into there, other than perhaps the context mappings.

Patch #42 from @tammycao works as expected and planned to use in production

Still needs test coverage, also the hook_update_N() needs renumbering.

Updated hook_update_N(), still needs work for tests.

Hi all,

this is my first time writing an upgrade path test, I was following this doco,
https://www.drupal.org/docs/drupal-apis/update-api/writing-automated-upd...

need help to review the patch, I also added an interdiff file.

thanks.

this patch added a test group, ignore #52 above.

+  public function testUpdateHookN() {
+    $this->runUpdates();
+    $this->assertEqualLayoutSectionDataType('longblob');
+  }

Can we add an assertion before the updates run. To test for what it is now.

Addressed the Comment #55
Attached patch against Drupal 10.1.x

#56 isn't quite doing what #55 asked for, but it also shows that the test isn't testing what it thinks it is, since it'll pass even without the update running.

Consider looking at this related issue, which I think is a better idea.

Addressed #55 adding an assertion after to the check the expected value after test run.

Per #58 the tests should be looked at as they don't appear to be testing what's expected.

hii, The above patch failed because the branch has been updated with file layout-builder.php, so removing the file from the patch and adding assertion after to the check the expected value after test run, addressed #55.

Attaching interdiff, thanks.

Patch #42 works very well for us on D9.5.

We are using a heavily customized layout builder setup though.
It contains inline block_content for section configuration through https://www.drupal.org/project/layout_block
It also contains a few dozen block_content types for actual section content.
All of these blocks have 50+ fields on them. (some of them coming close to 100 fields)
We only had issues when using the section_library module. Related issue that reference me here: https://www.drupal.org/project/section_library/issues/3232071

Hi, could this patch considered as "safe" for production environment or is still under development? Thank you very much.

@FiNeX FWIW, we've been using this in production for a while now and I would consider it safe.

#3118014: Additional blob field sizes in schema definitions is now in RTBC. If it lands before this, I could easily see us switching from a blob:big to blob:medium.

This is ready for review again after a rebase + some changes. Tests pass green.

We should postpone this one then.

But pipeline is green so that's good and probably an easy update when the other lands.

I'd like to suggest reopening this issue since the previously linked issue is now needs work and labeled a feature request.

#3118014: Additional blob field sizes in schema definitions needs to land before we can work on this issue.