Killer feature: Fieldable Fields in core

Subscribe! :D

subscribing

KarenS and I discussed this possibilty at Drupalcon DC, first as a joke and then as a realization that it could be the way to implement "combo fields" like an address field compromised of sub-fields for street, city, state, and zip. Instead of trying to make all fields fieldable, I started just on a fieldable combofield type at Drupalcon Paris. I got the data storage part working, actually, so I can now make a combofield with any cardinality that itself has any number of other fields with any cardinality attached. I ran into a brick wall when it came to processing and rendering forms, and possibly on other issues, then got busy with other issues for code freeze and have not gotten back to it yet.

The module is combofield, http://drupal.org/project/combofield, available in CVS. The data storage code worked in Paris, and has tests, but has since become obsolete. The form processing code never worked. I'm happy to have some other people contribute, but I'll warn you that the code gets abstract and confusing pretty quickly. ;-)

If you want to do something specific like making one kind of field know how to have taxonomy fields attached to it, it will probably be easier than the generic combofield case, which itself is easier than the generic "make all fields fieldable" case.

In any event, any changes for this necessary for Drupal core are almost certainly too late for D7.

See, I knew Karen told me in Paris that someone was already experimenting with this idea...
Subscribing!

For the record, I'm not sure 'taxo term on field' is a good approach to profile categories. As Barry outlined, fieldable fields tend to get to your head pretty quick.
Different subtypes (bundles) of a 'profile' entity type look like a more straightforward approach for ^profile categories.

Field Collection is the latest in the contrib space. I not sure if this should/could happen for D8 core.

Reopen if anyone still thinks this is a good idea for core (which I doubt).

I still think it's a good idea, but it's late now

related issues:
#1792072: Merge field_collection module in Fieldable Fields killer feature in Drupal core

lpalgarvio: I agree that fieldable fields, in one form or another, can be great.

However, we can call it a killer feature, sing in harmony that this should be in core, and re-open a dozen of issues, but the module will still not write itself. Do you have any thoughts on what the best approach would be?

I don't think composed_field, double_field, multifield, serial or any other module that hacks the data to add fields inside fields is a proper solution, although it may bring performance benefits vs other solutions. I think the core devs will agree on this - i'ts not pretty nor clean.

Thats leaves us with a few solutions for data and/or visual grouping:

1) any entity + referenced node + fields
all kinds of support, but i'd say has the worst performance

2) any entity + referenced eck/custom entity + fields
quite some support, lighter than a full node

3) any entity + referenced field_collection + fields
quite some support too, but lacks some basic things like a title. has issues with lots of FCs and fields

4) any entity + field_collection + fields
same as previous solution, but we cannot reuse data

and then we have 5) Field Group, which achieves a different goal: visual grouping only

visual grouping only

Going with the most easiest method, that gives us results quicker, FIeld Group would be an awesome example of a module to merge into Fields in core. Combine that with the Elements (in core i think) and other code re-use and we got a clean solution that interfaces well in any entity and view mode, in Views, and also in Panels (with a sandbox module - gives us tabs in panels for example).

So that gives us a UI to group visually fields, in vertical tabs, horizontal tabs, multipages, fieldsets, and so on...

data grouping

For data grouping, we need to further investigate the other methods mentioned earlier. The contrib adaptation varies a lot, and so do the features.

But in the end, i think the best approach would be a full Entity type ("Multiple field") + bundle ("each multiple field configuration"), with support for most common things we find in nodes, products, field collections and so on:
- Title (and title module),
- entity translation,
- revisions,
- owner/author,
- date,
- view modes,
- any kind of field,
- properties (or whatever they call now),
- extra fields,
- etc...

But not paths, menus, and other aspects almost exclusive to nodes. They should also be light, but that may require patching in core (which we had recently, for example, #1040790: _field_info_collate_fields() memory usage).

In the end, i would say the ideal would be a mix of Field Collection with ECK/custom entity, but implemented with Entity Reference (solution 2 + 3) and not directly put on the host entity. That instantly gives us data re-usage too.

I also have the opinion that likewise we now have Entity Translation and Views in D9 core and the ability to create custom View modes, so we should also have the possibility of creating custom Entity types and Bundles, and export them as we wish via the CMI --> which ofc would give us an instant FC/ECK solution...

Btw, i'm frequently mixing Field Collections with Field Groups in complex setups.

I'm conducting a study for an internal project at my work, so i might provide some comparisons sometime in the future.

I think field collection would be the best candidate for a starting point out of the modules listed. It doesn't look like the others support all the same features such as truly heirarchical data with multple levels of multi-value fields.

There's already been some talk and work on porting it to Drupal 8. #1443960: Field Collection for Drupal 8

I tried looking at this a bit myself. At one point I had a working Drupal 8 installation... but when I tried that day the install process wouldn't complete. I saw on some of the Drupal 8 documentation page that module developers are recommended to wait more before attempting to port their modules over, so I got the impression that this work would be premature.

I think that wording is referring to the fact that if you start converting your modules now (esp. if they touch e.g. entity/field API, as this one does) it's definitely not going to be a "port it and forget it" kind of thing, as APIs are still in flux. However, I would say that most module developers are highly encouraged to start learning D8 now, and provide feedback on the APIs so we can fix them while we still can.

If you point me at that documentation I'm happy to help clarify it thus.

I see what you mean. The documentation I saw is "What about contributed modules?" on the sidebar at https://drupal.org/drupal-8.0 .

Might warrant looking if we should investigate if Multifield module is a good fit for D9 core.

It seems to me that field groups/bundles are the Drupal equivalent of Object Orientated Programming classes. Currently a node (or view) is such a group, we're just short of how to implement them better.

Perhaps Drupal 9 will be more consistent, so that we don't have "special" content type that define Comments, User and Taxonomy, but they are all just content types with different combinations of fields. After all, Comments are really only nodes with titles that are linked together, etc.

Each of these entity types are subclasses of fieldable entities. That's pretty consistent. Node's aren't really a generic entity type, they have data and operations that shouldn't be mixed with the other entity types. What would it mean to unpublish a user, for example?

Likewise, modules can make their own entity types that have custom functionality that nodes / users etc. don't need to support. Field collections are a good example of this as they double as both a custom entity and a field in their host. There's nothing else that functions like this and it wouldn't make sense for another entity type to do so, but there are advantages to having a special entity type that acts as data in a host node. For example:

• They can't be updated or deleted without the host knowing about it.
• Each revision of a field collection item corresponds to a specific revision in its host which allows nodes to be restored to the data that was in an older revision. This can't reliably be done with an entity reference field.
• You can clone a node with a field collection and the new node will have a new copy of the field collection item instead of a reference to the same one so they can be edited separately.

jmuzz wrote: "Likewise, modules can make their own entity types that have custom functionality"

I would separate the functionality from the entity types.

It bugs me that nodes all have "Promoted", "Sticky" and "Promoted" fields. Views doesn't need "Sticky" and "Promoted" to work, but adding such Boolean fields should be trivial, and supported by Views.

It bugs me that all nodes must have a title, when the NID is the unique primary field, and I'll add a title if I want one.

But it does all look a lot better in Drupal 8.

I can't think of a reason this would have to happen in D9; it seems like D8 provides all the plumbing needed. So for now moving back to 8.1.x until it's shown this can't be done in a non-BC-breaking way.

++ to a solution like Dave Reid's https://www.drupal.org/project/multifield

In case this issue is about entity to entity composite relationships...

The entity reference field in core points to a target entity without a revision id.
This functionality was dropped from core to reduce complexity in entity reference relationships as it also would confuse people with normal relationships.

Data grouping requires a composite relationship that is based on revisions. Thus reverts can work.

There is a contrib module created to make paragraphs module (composite entities with inline editing) work with this relationship:
https://www.drupal.org/project/entity_reference_revisions

This module would need to go into core or the entity reference module would need to be extended to support a composite relationship optionally.

There is a _LOT_ of work needed to make core properly work with revisions, offer a UI that works and properly deal with translations first. See the meta issue:
#2465901: [META] Make entity revision translation work
In case this is the approach to go, we should postpone this feature.

Core will need to define a new UI pattern to deal with inline editing / entity entanglement. It is not so easy to make users aware of what's going on.
Thoughts on a UI that supports a mixed setup where some referenced field items can be composite and others can be reused (same field instance) seems to be a bad idea in UX although it would be the most flexible approach. So if a reference is a composite, would be a semantic field storage decision that adds the target revision id. This can be an (unchangeable) field storage option.

Things for 8.1.x are generally postponed anyway, I think.

Btw. we used both field_collection module, which supports linking to revisions and entity reference with CPS and just storing an entity_id is enough if you track the revision somewhere else like in a changeset of changes together.

Overall we had less trouble with inline_entity_field PLUS entity_reference with CPS.

We are working hard to make things complete in the domain of paragraphs. Here is our experience with field_collection:
- It took more than 3.5 years and 300 replies to make field_collection partially multilingual. Still a followup is pending and there are many special cases that seem to not properly work yet. #1344672: Field Collection: Field translation (entity_translation) support.
- a revision tab for all entities #2452523: Offer a revisions tab for all entities with a revision handler #2462731: Check RevisionLogInterface for summary and provide
- diff integration for the revision tab to resolve composite relationships such as paragraphs (or field collections) #2433099: Provide a custom diff plugin
- a TMGMT integration on top of all these things #2462263: Support paragraphs translation.

We gave up to make this all work on 7.x with a field_collection approach. It's the first time that we see things really becoming ready and we will keep pushing until this is done in 8.x.

Would be nice if you could explain what you mean with "track the revision somewhere else".

#26: See http://drupal.org/project/cps for an idea how to track entities in multiple changesets / site versions with inheriting, language support, etc.

For multi-lingual we used CPS in combination with IEF (inline_entity_form) and custom 'content snippet' entities and yes language support in IEF is also buggy still.

The things we changed have been:

- Custom setting of the entity_translation form language on the loaded entities. (entity_translation sets the content language by replacing the entity form page callback, that is the MOST common bug I have seen for child-loaded entities).

- Pass the content language used on the parent by changing the ajax path to system/ajax/content-language/[langcode] and a custom language negotiator for CONTENT_LANGUAGE.

This was done in a field_widget alter hook IIRC.

And it works like a charm :)

I've been working on this for a work project.

It's not really like field_collection, in that it doesn't require a separate entity to attach fields to. It's similar to multifield in that way, I guess, but there's no intermediate step of going to some other admin UI to define your field. Actually, right now, it doesn't even have a UI at this point. You still have to define a field type plugin, but with this module, you extend ComplexFieldTypeBase and override ::getSubelements(), which returns an array describing your subfields. That method is used by ComplexFieldWidget and ComplexFieldFormatter to automatically generate the form and output for those subelements. Right now, the subelements can be anything in the FieldType plugin space (including those that have "no_ui = TRUE" in their annotation).

A UI would be relatively easy to build into the field storage configuration page, and might even end up simplifying the entire thing (because right now, there's a lot of hackery around setting up properties and schema and such - basically faking a good portion of the normal class hierarchy). My vision for the module is that I can - in the field UI - create a Complex Field, add and configure the storage settings for each of my sub elements, and click go, then go configure the Widgets and Formatters for the sub elements. Then when I view the node form that this Complex Field is attached to, it's essentially like seeing a bunch of fields in a fieldgroup. The module currently also supports fields with any cardinality, so you're not just limited to one complex field value for any given instance of the field.

Note that field storage/instances aren't created for the subfields - they're supposed to be included with the data of their parent field (i.e. not separate database tables - just new columns for the parent field).

As far as getting this into core, I'm not sure if it should be a separate module or just tossed into `core/lib/Drupal/Core/Field/Plugin/Field/{FieldFormatter,FieldType,FieldWidget}` directories. Either way would be fine, I suppose. I guess it's just a question of whether or not the field type should be possible to disable or not.

I know that there's at least one field in core that could benefit from this work, too: the Link field. That could be represented as a Complex Field with a StringItem and a UriItem attached. As noted in the IS, this could also serve as a good base for Address Field, and could completely replace Double Field. Then, with #1803064: Horizontal extensibility of Fields: introduce the concept of behavior plugins, it would be really easystraightforward to replicate something like Hierarchical Select. We have another module internally that's sort of like HS, and I'm hoping to rebuild it on top of Complex Field, so any work to make that happen may get some love here in the coming weeks.

Unfortunately, I can't release it just yet, as I'm waiting on my employer to approve the contribution, but I'm confident that it will be approved soon-ish, and I'll be able to post it as a new project node on Drupal.org. When that time comes, I'd really love help with making it usable and cleaning it up for consideration for Core. I'll update here when it's available.

^^^ sounds a w e s o m e ^^^

Having one standard way of doing this in core should help more contrib modules support this complex field, instead of trying to cater for field collection/multifield/etc. By the usage stat of these modules it's very clear there's a huge need for composite (complex) fields.

If it can be done without an additional entity, double kudos! even if there's some limitation we have to embrace, the performance gain/simplicity is worth it.

One very common use case is 'relationships with attributes', ex: an entity reference from a recipe node to an ingredient node where that reference holds the quantity and unit (2 cups of sugar). Right now you can use a custom field to represent that but you loose some contrib recognizing the 'reference' part of it.

I always thought nodes (entities) should be built, like you build PHP objects, and extendable in the same way. Methods become Rules, and so you can create any time of "record", and use rules to manage almost everything.

Then you wouldn't need "modules" to do stuff that can't be done in core, instead you'd have "entity/rule" profiles that do everything you want.

But I think I might be going off topic here.

when I view the node form that this Complex Field is attached to, it's essentially like seeing a bunch of fields in a fieldgroup.

(i.e. not separate database tables - just new columns for the parent field).

The module currently also supports fields with any cardinality, so you're not just limited to one complex field value for any given instance of the field.

brilliant! simply brilliant!

This would, in one fell swoop, completely supersede field_collection, multifield, & double_field and provide a single optimized method for handling this functionality whether in contrib or core. It would also address the potential performance issues with the field_collection entity based methodology as well as the loss of single table field storage that occurred when we went from CCK to field api.

Currently I usually code this into custom compound fields, which are easy enough to do, but it would be so much simpler, faster, and more maintainable with a plugin approach. Adding a UI for site builders is just the icing on the cake!

Glad you guys are excited about this. I'm so excited to share what I have so far. It's really really rough (and I can't under-emphasize that - what I have so far is a giant pile of hacks that could probably be better written by a drunken chimpanzee), but it's a starting point.

Still haven't heard whether or not it's okay to share from The Man yet, but I'm continuing to work on improving the module in my free time, so hopefully, it's in a slightly better state by the time I can share :)

My open source request has been approved, and the complex_field module is now available over at https://www.drupal.org/project/complex_field. PRs are welcome at https://github.com/cweagans/complex_field. Apologies for the admittedly rough code - it was written in a hurry with the goal of getting things working, rather than architectural correctness or maintainability.

There's enthousiasm and substanstial discussion in the comments but the issue summary tells me next to nothing :-)

Please update the issue summary with the basic idea/feature/pitch:
What is the problem, for whom is it a problem, why should we fix it, what's the proposed strategy to do it?

I can't speak for all of us, so I'm not updating the issue summary... but when I subscribed to this issue 7 years ago, I would have been happy with only half of what Paragraphs is today.

Paragraphs isn't just fieldable fields; it also allows nested fieldable fields, combining and re-ordering of different paragraph types, and it supports revisions and translations.

That's not to say Paragraphs should be in core; IMHO it is just fine living in contrib. Core provides the strong base of entities, fields, view modes and config management that is essential to build it.

I was working on a module that does something along these lines for D8 and someone pointed me to this issue. It's kind of like a cross between a custom custom Field type UI, and a Entity Fields UI Lite for the sub-fields... or something... Thought I'd post it here and figure out if I'm totally out to lunch!

https://www.drupal.org/sandbox/ceastwood/2901669

The idea is that the sub-fields are just database columns so they come with a different set of available widgets (instead of trying to support every field api field type). It's meant for simple, one-dimensional multi-fields. Anything more complicated I think it's better to use something like paragraphs anyway.

If https://www.drupal.org/project/paragraphs does fieldable fields, maybe the module paragraphs should be renamed as the title brings to my mind something related to writing.

I think there is a stark difference between fieldable fields and referenced entities (the way paragraphs does it). Both should be supported by core, but neither (in my opinion) needs to be in core.

As far as linked references go, paragraphs does a good job of this. The name might elicit thoughts of writing, but it is super flexible and I have used in in many occasions. This api has been built from core and works well as it is. Nothing further is required from core to support this.

The other other strategy is the true fieldable fields. Which the best candidate seems to be complex_fields and multifield. This is the approach that isn't fully supported with the current api and we should have an api module in core that allows for working on this type of approach. As it has already be stated, we already have complex fields with the link field. Date could also benefit from this.

The issue summary needs to be updated to better reflect this. I have made a simple update but it would be better if someone closer to this issue rewrote the IS.

In a recent meeting with the Office of the CTO at Acquia over content modeling in Drupal, @webchick indicated that she would like more data on the prevalence of the need for composite (or fieldable) fields, as currently provided by Paragraphs module (and previously by Field Collection), and she asked me to report back here. She posed the following questions: How many projects are doing fieldable fields for content modeling? What types of sites/projects, etc.? And why isn't Paragraphs a sufficient solution for it already, such that we should consider putting something in Core? I conducted a small scale, unofficial survey of Acquia Professional Services technical team members. Following is a summary of 14 responses:

How great is the need for fieldable fields?

• How often do your projects require a solution for fieldable fields for content modeling (not layout)?
  • 6 (42.9%) - Basically always
  • 6 (42.9%) - Most of the time
  • 1 (7.1%) - Some of the time
  • 1 (7.1%) - Seldom
  • 0 (0%) - Basically never
• Off the top of your head, what projects have you worked on that required fieldable fields? - 32 recent projects named specifically, including multiple platforms/distributions, and one response of "No really: all of them!".
• What types of clients/projects/use cases require fieldable fields? - Respondents indicated the need transcends verticals and size categories. They mentioned compound and semantic data elements, lists containing items with labels and supplemental information or options, products and e-commerce, named reference containers, and literal paragraphs of decomposed content.

Is the contrib solution sufficient?

• How well does Paragraphs module usually meet your needs?
  • 6 (42.9%) - OK—It has issues, but I can usually get around them with enough effort.
  • 4 (28.6%) - Well enough—It has issues, but they don't usually cause me trouble.
  • 2 (14.3%) - Very well—I usually use it without meaningful issues.
  • 2 (14.3%) - Poorly—It has issues, and its usually impossible or not cost effective to get around them.
  • 0 (0%) - It doesn't—I can't usually use it due to its issues.
  • 0 (0%) - n/a—I haven't needed it or have no opinion.
• Where does Paragraphs module cause you problems?
  • 8 (57.1%) - UX (User experience)
  • 4 (28.6%) - DX (Developer experience)
  • 6 (42.9%) - Workflow and revisioning
  • 5 (35.7%) - Multilingual
  • 6 (42.9%) - Performance and scalability
  • 4 (28.6%) - Migration
  • 3 (21.4%) - Module integrations (e.g., Views, REST, etc.)
  • 1 (7.1%) - Search
  • 1 (7.1%) - Overhead caused by number of entities created (Content Hub syndication, max_input_vars errors in extreme cases)

Any additional thoughts you'd like to share on the subject?

Respondents variously indicated the following: Fieldable fields adds a killer feature to Drupal's content modeling capabilities. It has been a requirement since the early days of Drupal 4.5. The biggest challenges related to the current Paragraphs implementation include the UX, integration with other critical systems, and the data storage model.

Summary and Recommendation

The sample size of my survey and the breadth of my personal experience are obviously insufficient to represent the total needs of the community. But they seem at least suggestive of some tentative conclusions:

• There is a significant need for the ability to model composite data in Drupal, more so as applications become larger and more complex.
• The Paragraphs module is a good option, but it has problems—at scale, integrating with Drupal Core and contrib modules, and its UX is poor. The nature and extent of the problems would probably require a significant refactoring to fix.
• Since Field Collection's project page states that it is likely to be deprecated in favor of Paragraphs, the major alternative for developers is currently a custom entity type, which is beyond the skill and budget of many project teams. Presumably a custom solution could be designed for better performance by using field properties instead of nesting for multiple values for known cardinalities, but it couldn't do that for unlimited value fields. And the probability of incompatibilities with other module APIs would be even greater.
• A solution in Core could presumably be made to interact better with Core systems and paradigms, but at the cost of complexity. Of course, it's just the absence of these particular complexities in Core that results in them being unconsidered and unaccounted for and thus unsupported when contrib tries to tackle them.

Conclusion: From a broad, community perspective, it seems like the best outcome would be either a partial rewrite of Paragraphs module that addresses the problems we've discovered with it or the addition of some comparable solution to Core.

@TravisCarden That is some good data. Even with a small sample size the distribution of the data gives good insights and I think you are right on in the assumptions taken from it.

Recently the solution I have found (in Drupal 8) has been a combination of Entity Construction Kit, Inline Entity Form, and Entity Browser. With that said, this gives me flexibility, but there are bugs absolutely everywhere and plenty of considerations that need to be made based on this solution. Luckily, none of the bugs are game-breaking and can be mitigated.

@TravisCarden: Thanks for the summary of where we are.

The key reason why Paragraphs is better than ECK + EntityReferences is the tight revision handling, that makes the whole thing worthwhile; #2859995: Add Entity Reference Revisions to Drupal CMS was opened to handle part of this equation.

Thank you @TravisCarden for this assessment and the interesting numbers. They will also help us (and we should do some more investigations) for setting our priorities right in Paragraphs maintenance.

There is nothing you can really rewrite in contrib to solve these problems. The Paragraphs (and Entity References Revisions) design tries to build on top of what Core provides and then workaround the limitations of core with minimum assumptions without breaking most expectations (by other Contrib, ...). Limitations of core are wide spread, just to name a few:

• The Field Widget design causes severe issues when nested (performance, complexity, delta handling).
• The multilingual situation is very complex. For instance it's hard to identify when a form is in translation or when source language. The Core multilingual form handling is too simplified.
• The Paragraphs UI introduces new patterns because Core doesn't provide UX patterns for composite handling.
• The performance is limited due to the AJAX (again, form builder) and validation and nesting
• Workflow and revisioning is our currently highest priority. Core added here new functionality without enough maturity, breaking many things at Paragraphs - expecially combined with multilngual.

We spent more than 18 man months of work in the problem domains mentioned above since we started the Drupal 8 port.

That said, Paragraphs does a great job with making things possible and removing a huge amount of complexity from site builders, developers, ... and DX wise small tasks are easy to resolve while deeper integrations require understanding of the core limitations and workarounds. We believe that most of the DX issues are simply in the nature of the additional complexity of a composite relationship and would likely exist in a different way when solving the use case without composites/Paragraphs.

All in all, i fully agree, core should be composite aware in storage and UI (form widget, UX) so contrib can build on top of this and risks are smaller core is breaking the ecosystem with future changes.

Thank you for the research and summary, Travis. My opinion more or less matches the answers given by the respondents.

I'm wondering what led you to the conclusion that "The nature and extent of the problems would probably require a rewrite to fix." The word "rewrite" suggests that the foundation of the module is not good enough. Maybe that is the case, but the answers do not substantiate it and it does not reflect my experience with Paragraphs. IMHO, saying it needs a rewrite without underpinning does not do justice to everything the maintainers have built so far.

Hello, @marcvangend. I meant no disparagement to either the Paragraphs module or its maintainers by suggesting a rewrite. (Indeed, I have used the module very happily, and I'm thankful to the contributers for providing it!) But I can see that my meaning could be made clearer, so I have updated my comment to use the language of a "significant refactoring" and a "partial rewrite". I will briefly explain the thinking behind the remarks: The UX challenge with Paragraphs is the most obvious to end users and so the most salient. It is also apparently intrinsic to the most obvious UI pattern, which means that a considerable reimagining of the interface may be needed. That would be a sizeable effort, no doubt, but what I have even more in mind when suggesting that considerable refactoring would be needed has to do with deep, invisible issues. For example, as I recall, Paragraphs keeps all of its entity data in a single database table, which is not a design optimized for scalability. It creates new entity revisions for every change to (or revision of) a parent entity and never deletes them, which exacerbates the "one table" design. (Infinite data growth potential with no control.) Others understand better than I the design implications for translations and workflow. In any case, these are some of the challenges that need to be solved, whether in Paragraphs or in a Core implementation, to provide a viable enterprise solution.

@TravisCarden: Paragraphs isn't the problem here, core doesn't provide a mechanism for automatically purging unwanted revisions and it always saves new records for every new revision, Paragraphs just compounds the problem because it's one of the only referenced entity system that properly handles revisions.

What would help is #2960887: Do not create field revisions when field data hasn't changed, and maybe a contrib module to automatically purge old revisions.

content entity builder + entity reference + Inline Entity Form is another choice.

Field collection, paragraphs, ECK, both allow you add field to entity. After a lot experience of this way.
I don't like the idea of one field one table. Actual one field 2 tables.
It is a big waste. waste a lot of storage space.
Performance is not good. suppose you have 100+ fields.
Updating is nightmare. Field collection is a example of update, it is not well supported in Drupal 8, update Drupal 7's field collection to Drupal 8 is a nightmare

I want to have full control with my data model: One entity one table. this is why I build content entity builder module.

Re #50
Interesting approach, thanks. Does your method support revisions and workflows?

I don't like the idea of one field one table. Actual one field 2 tables.
It is a big waste. waste a lot of storage space.
Performance is not good. suppose you have 100+ fields.

Did you test this in a side-by-side comparison of both methods? Can you share your measurements of performance and storage space?

I have real experience with 100+fields, it become very slow.
I have experience with storage 700K chess manual using node type in Drupal 8 with 40+fields,
After half a year I change it to a custom content type without bundle，no revision support.
Save me a lot of storage space.

Drupal 8 is not as scalable as Drupal 7. After the numer of chess manual reach 500K, the website become slow and slow. I have done some performance optimization for it, but the speed is still much slower than my expectation.

Paragraphs keeps all of its entity data in a single database table, which is not a design optimized for scalability. It creates new entity revisions for every change to (or revision of) a parent entity and never deletes them

In my opinion, the "never deletes them" part is the most critical problem with respect to including it in core. I don't think "never" is entirely true. It does delete them under some circumstances and #2771531: [META] Remove obsolete composite entities from existing storage (garbage collection) is working on improving that further. However, I opened #2997319: Delete records from paragraph field data tables that are orphaned from the host entity's default revision for one case that I believe to be a problem, since that leads to unbounded growth of paragraph data tables with core's default configuration (nodes saving a new published revision with each save).

I would suggest recognizing that there are two distinct issues:

• There are some circumstances where paragraph records may persist where they are no longer connected to a host entity or entity revision. These are bugs and should be handled as such.
• There are scenarios where people wish to purge old revisions of paragraph records. Drupal's core entity system does not have any system in place for automatically purging old revisions, so this would be a new feature and completely separate to whether or not Paragraphs-like functionality is added to core.

Lets not get these two confused.

Right. I think another way to look at #54 is:

• The paragraph "base_table", "data_table", and field data tables should not contain records for paragraph entities that are not referenced by the default revision of any host entity.
• The paragraph "revision_table", "revision_data_table", and field revision data tables should not contain records for paragraph entity revisions that are not referenced by any revision of any host entity.

To the extent that host entity revisions can't be purged, it of course makes sense that the paragraph data for those host entity revisions can't be deleted either.

Here's a quick update from Paragraphs and Core challenges:

We're at the edge of Core limitations to solve the remaining Paragraphs issues.
Even worse, Cores' adds new dimensions of complexity like Content Moderation that are effectively breaking contracts. At least breaking expectations that were developed from the plain Entity API and certain UI limitations in Core, allowing us certain workarounds in Paragraphs.

Suddenly, core states that Content Moderation is stable and Paragraphs could not technically declare incompatibility with it. People started to use it and their storage broke. This problem could not be resolved by Paragraphs on its own, instead, we needed about 6 Core patches to allow us to make Paragraphs compatible, problems that we discovered after 8.5 / Content Moderation was declared as stable.
After about 4 went in, we still need Core patches, see https://www.drupal.org/docs/8/modules/paragraphs/multilingual-and-conten...

This shows IMHO how important it would be that at least the ERR (storage) part of Paragraphs is a reference use case in Drupal Core. It's simply impossible to guarantee consistency for a composite storage model on top of an Entity system that doesn't. In fact it's also limitations about the whole Form builder + Form state and widget lifecycles work. I have the feeling that some problems can only be resolved nicely (without major performance limitations) if we break some existing contracts in Core.

Then looking at the numbers above pointing at unsatisfying UX, proofs that Core then also should declare UI patterns to solve the composite problems in the UI, be it Paragraphs or Inline Entity Form. Paragraphs is where the complexity of 10 dimensions comes all together where Core usually only combines a few of these in a specific context.

We start to feel the pain ourself as well with the growing garbage, but in fact none of our projects ever escalated this problem in a way that there were funds to solve it. And from the few contributors that worked on some related patches almost no one was willing to provide test coverage. As long as this situation doesn't change i tend to presume that the real need isn't as big as the wishes.

That said, i don't understand yet what the intention of some detail discussions here are. I outlined the identified Problem domains that Paragraphs need to deal with in comment #46 (maybe some are missing, like garbage collection) and the transition towards core would need to work through all of them. To start this journey we need full commitment from the Drupal project lead, many core subsystem maintainers, and a bunch of large scale projects (customers?) or other major beneficiaries (agencies?) that sponsor the involved maintainers. Without estimating in too much detail, but i would roughly guess that it is about 12 man months of work (for the storage), not days or few weeks. And much more if we would want to add the UI / widget as well. I'm all ears if larger orgs are willing to fund this work. Until now no one ever showed interest.

I'll respond more to #56, but for now, just want to mention that today another integration BC break got reported, this time for 8.6: #2997748: Views query alter for fields stored in dedicated tables is not working properly.

The objections of Miro Dietiker reminds me of the "Views in Core" discussion in advance of Drupal 7.

As we all now Views did not become a core module in Drupal 7 but in 8 due to its complexity and limited human power.

I totally agree that using different approaches together like multilanguage, content moderation, etc. has many unresolved problems.

Therefore this might rather happen in Drupal 10. But Paragraphs is really the future of Drupal.

The focus of this issues seems to have drifted wayyyy off course.

> One of the issues is dealing with profile module categories. IMO, those categories are best expressed as taxonomy fields. If fields could contain other fields, then profile categories would be simple to build with Taxonomy.

This is about adding fields to the *configuration*. So, adding a field to the user entity type, and then being able to set a tag on that field config.

The recent discussion is all about Paragraphs, which is compound field *values*.

#59 I'm not sure I get it. In D8 terms would we call this something like "allow config entities to have fields"? A field being a way of defining and interacting with a complex datatype; as config entities can have properties that have datatypes, why not allow them to use fields?

> In D8 terms would we call this something like "allow config entities to have fields"?

Yes, pretty much that.

> A field being a way of defining and interacting with a complex datatype; as config entities can have properties that have datatypes, why not allow them to use fields?

I'm not arguing for or against anything, I'm just saying that the focus of the issue has veered off-course. The issue summary and title is about "allow config entities to have fields", but recent discussion is all about Paragraphs, which is completely different.

> he issue summary and title is about "allow config entities to have fields",

I'm not sure sure that is really what this issue is about.

Having fieldable config entities would mean the actual field config entities themself would have fields. Which a) I don't really see how that could work and b) Doesn't seem that useful, honestly.

What fieldable fields really means IMHO is that the *field items* have fields, aka are complex structures. That's a completely different beast. And the current standard approach for that is having referenced entities with an inline widgets like paragraphs, field collection, ..., another approach in 7.x. was https://www.drupal.org/project/multifield but that had pretty severe limitations.

And core actually now has yet another approach for this, with non-reusable content blocks in layouts: #2957425: Allow the inline creation of non-reusable Custom Blocks in the layout builder.

> Having fieldable config entities would mean the actual field config entities themself would have fields. Which a) I don't really see how that could work and b) Doesn't seem that useful, honestly.

True -- the use case in the summary is for grouping fields on the user entity into categories.

That could be done instead with form display modes instead, just need a bit of extra code to turn form modes into tabs on the user entity.

Given that, should we close this as wontfix?

Just to be clear, most of the discussion has been about nested content field architectures, including the current contrib options of Field Collection, Multifield and Paragraphs, little consideration has been given to fields on configuration. Even the issue title is (paraphrase) "fields on fields".

@joachim, from comment #4 (9 years ago) this discussion has been about content structure and has nothing to do with config entities except the comment here and there that if fields have fields then config entities would also have fieldable fields. Maybe after 9 years, the IS needs an update, the title is still on point though.

Speaking to what is available on the market today, Paragraphs and its Paragraphs Library system definitely come the closest right now to achieving the balance of nested, flexible content structures and providing a way of making some of the content reusable, I highly recommend using them and then collaborating on improvements to the UX if needed.

@DamienMcKenna, I disagree. ECK + IEF + Entity Browser have in my opinion more flexibilty, depends on what you compare to each other.

The real dealbreaker concerns the revisions. ECK can't deal with that yet: #2788507: Add revision support

But as Miro mentioned in detail, Paragraphs also has difficulties with things like Content Moderation, etc.

I disagree with others and think it can only be solved meaningfully if a solution is fully integrated into the core, because it's simply such an important requirement and the dependencies to the core are very extensive.

To start this journey we need full commitment from the Drupal project lead

This is the key, no matter which way, whether Paragraphs, ECK or something new... The Drupal project lead must fully recognize the problem and be willing to provide a solution. After that you can attack technical details. Otherwise, we'll be going round in circles for a long time.

A large volume of Drupal sites require revision handling, which makes Entity Reference Revisions the only solution right now. Only Paragraphs and Bricks can work with Entity Reference Revisions. Entity Reference doesn't support revisions so any solution based on it is a non starter.

Correct, but in the end that doesn't help much for a clean stable solution if problems occur later as described by miro at #56.

All of the core bugs referenced by Miro in #56 have been committed. So yes, revisions matter and any nested-entities architecture which doesn't support references ala Entity Reference Revisions is creating a huge mess for clients to deal with.

Ok if that's defined so far, what are the next steps?

Multilingual Content Moderation now works in ERR and Paragraphs.

There are some limitations though, see:
#3004099: Allow to translate paragraphs from pending revisions

As far as i know IEF also supports ERR.

Also pinged @effulgentsia for the promised update in more depth.

For now we are pushing Paragraphs forward in small steps. We should maybe make it a much bigger initiative in all its dimensions. The problems and their severity (in core) seem not to be clear to most people until we have implemented solutions for them in Paragraphs.. Again the multilingual Content Moderation was much worse that expected and outlined in #56.

Anyone willing to dedicate a few weeks of work on solving the garbage collection problem in Paragraphs? Contact me!

Concerning using a custom entity, I think this is the best solution. What is really needed is a truly symmetrical fieldable n:m relation entity table, with widgets for both the n_target and m_target entities. I have done this kind of work before, for two projects, and even have a general module which does this. However it heavily relies on form hooks to produce the widgets. I have been thinking about redoing the whole thing now as follows:
1. A general translatable, bundleable, revisionable relation entity with slots n_target_id and m_target_id.
2. A selection field or widget which is automatically made available on the target entities. Probably two widgets are needed, a compact one, and one for long lists that works like views bulk operations.
3. Various views classes to handle the relations, etc.

This was fairly straight forward programming in N:M cases, however in a self-relation case, N:N, you end up with some more delicate issues in the widgets and views handlers.

You can do all of this with Paragraphs, but you only have a widget on one target entity then. and you end up with too many joins to really be efficient or transparent. I want to avoid endless joins, and just want three tables: The n_entity, the n:m entity, and the m_entity.

I am interested what others think about this. Does the need for this structure come up frequently enough to justify the effort?

@hawkeye.twolf, @ashrafabed and I met to discuss this further

@hawkeye.twolf has been experimenting on this in the rich_field project and I've been working on it in the Field union namespace. We've decided to join forces.

The plan from here is that I'm going to attempt to get my tests passing using his approach for instantiating sub-fields in the field-item constructor and try to remove the field proxy class I was using to get around computed property issues.

If that works, we can then combine the two approaches and will find a new namespace to work together in, possibly multifield.

Rewrote the issue summary to reflect the current thinking, approach and use-case

We created a slack channel at #multifield

For the record, its super easy to build compound fields with drush generate field. The wizard asks you to fill in how the number of sub-fields, and the particulars of each. See transcript below, for a compound field that nearly satisfies the use case in the OP. A web UI for this would be nice of course, but the framework is in good shape already IMO. I prefer this solution over paragraphs/field_collections for some use cases.

 $ drush generate field

 Welcome to field generator!
–––––––––––––––––––––––––––––

 Module machine name:
 ➤ recipe

 Field label [Example]:
 ➤ Ingredients

 Field ID [recipe_ingredients]:
 ➤

 How many sub-fields would you like to create? [3]:
 ➤ 3
–––––––––––––––––––––––––––––––––––––––––––––––––––

 Label for sub-field #1 [Value 1]:
 ➤ Quantity

 Machine name for sub-field #1 [quantity]:
 ➤

 Type of sub-field #1 [Text]:
  [1 ] Boolean
  [2 ] Text
  [3 ] Text (long)
  [4 ] Integer
  [5 ] Float
  [6 ] Numeric
  [7 ] Email
  [8 ] Telephone
  [9 ] Url
  [10] Date
  ➤➤➤ 6

 Limit allowed values for sub-field #1? [No]:
 ➤

 Make sub-field #1 required? [No]:
 ➤ Yes
–––––––––––––––––––––––––––––––––––––––––––––––––––

 Label for sub-field #2 [Value 2]:
 ➤ Unit

 Machine name for sub-field #2 [unit]:
 ➤

 Type of sub-field #2 [Text]:
  [1 ] Boolean
  [2 ] Text
  [3 ] Text (long)
  [4 ] Integer
  [5 ] Float
  [6 ] Numeric
  [7 ] Email
  [8 ] Telephone
  [9 ] Url
  [10] Date
  ➤➤➤ 2

 Limit allowed values for sub-field #2? [No]:
 ➤ No

 Make sub-field #2 required? [No]:
 ➤
–––––––––––––––––––––––––––––––––––––––––––––––––––

 Label for sub-field #3 [Value 3]:
 ➤ Name

 Machine name for sub-field #3 [name]:
 ➤

 Type of sub-field #3 [Text]:
  [1 ] Boolean
  [2 ] Text
  [3 ] Text (long)
  [4 ] Integer
  [5 ] Float
  [6 ] Numeric
  [7 ] Email
  [8 ] Telephone
  [9 ] Url
  [10] Date
  ➤➤➤ 2

 Limit allowed values for sub-field #3? [No]:
 ➤

 Make sub-field #3 required? [No]:
 ➤ Yes
–––––––––––––––––––––––––––––––––––––––––––––––––––

Would you like to create field storage settings form? [No]:
 ➤

 Would you like to create field instance settings form? [No]:
 ➤

 Would you like to create field widget settings form? [No]:
 ➤

 Would you like to create field formatter settings form? [No]:
 ➤

 Would you like to create table formatter? [No]:
 ➤


The following directories and files have been created or updated:
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
 • modules/recipe/config/schema/recipe.schema.yml
 • modules/recipe/css/recipe-ingredients-widget.css
 • modules/recipe/src/Plugin/Field/FieldFormatter/IngredientsDefaultFormatter.php
 • modules/recipe/src/Plugin/Field/FieldType/IngredientsItem.php
 • modules/recipe/src/Plugin/Field/FieldWidget/IngredientsWidget.php
 • modules/recipe/recipe.libraries.yml

I'd not noticed Drush could generate a composite field, thanks! I think the drawback of this for me might be inability to re-use sub-fields across multiple composite fields, which has considerable benefits when you want to manipulate all instances of a field in code via the field machine name. Does the framework support re-use?

Does the framework support re-use?

No it doesn't. It only supports a predefined list of "simple" field types.

 [1 ] Boolean
 [2 ] Text
 [3 ] Text (long)
 [4 ] Integer
 [5 ] Float
 [6 ] Numeric
 [7 ] Email
 [8 ] Telephone
 [9 ] Url
 [10] Date

For truly fieldable fields Paragraphs or entity reference + inline entity forms needs to be used.

You can't have it both ways, either you have an separate entity with the full flexibility that this offers or you have a combined field type with multiple properties.

Another important limitation of such an approach is that there is no (easy, official) way to extend such a field type, Drupal 8 does not have an API to change the schema/properties of a field type.

After using drush generate field and going through the steps, now do you use it? I get Unable to install modules recipe due to missing modules recipe.

Module enable should just work. Perhaps check to see if the generated files are where they were reported. If you have any more questions, please ask in the Drush issue tracker as we wont want to alert everyone here.

For the UI mentioned in comment #78, I think https://www.drupal.org/project/flexfield at least gets part of the way there for simple use cases where we don't need the full flexibility of a separate entity type...?

You can't have it both ways, either you have an separate entity with the full flexibility that this offers or you have a combined field type with multiple properties.

@Berdir, that is exactly what this issue is addressing. Giving us the best of both worlds, fieldible fields without the overhead/complexity of entities in core. Think Multifiled in core.

> Think Multifield in core.

That's exactly my point. Ever tried to add another column/field to a multifield? I had to do it once in 7.x for a custom project where I made it work with custom schema changes but that's so much harder to do in D8 core.

So again, either you have an entity type, with configurable fields and an API to add additional fields later on (and more challenging, change them), or you have a more efficient custom field type, but then that structure is pretty much set in stone.

There was an informal poll over Twitter about features people would like to see in Drupal 9.1 https://twitter.com/webchick/status/1235689652838776832 (there will be a formal poll not over (just) Twitter coming later this month), and this was one of the most highly requested features. @larowlan also expressed interest in this initiative. Just leaving a note. :)

Per #3109340: [policy] Decide whether to require json support for all database drivers for Drupal 10 we have an opportunity to use JSON features within all supported databases for Drupal 9.1. I have a (not yet proven) hunch that that might help resolve some of the performance/scalability concerns that have been raised in earlier comments and people's various contrib experience. I might be wrong on that, so a non-JSON storage solution is totally fine to pursue as well, so long as performance and scalability needs are addressed.

I would be open to an honest look at this. It will not be easy. There are some challenging issues to any form of paragraphs in core, but there are so many benefits that I think it's worth opening up again. 9.1 might be a bit aggressive, but if we can come up with a plan it may be possible in the D9 cycle over the next two years.

I think this could be implemented as an intermediary field type which "references" "delta" field values from other fields on the content type.

- All of the required fields are created independently on the content type as unlimited-value fields
- In the intermediary field type (maybe called "glue" or "web" multi-field), you choose which of the available fields on the content type should be "glue"d together (and how many field values per-field, per-group)
- The intermediary field suppresses the selected field types' regular widgets and re-renders them nested beneath a fieldset
- In the DB, the intermediary field's columns are, e.g.: intermediary_group_delta, field_name, field_delta
--- intermediary_group_delta = the current group of fields (e.g. group 1)
--- field_name = the machine name of one of the included ("referenced") fields on the content type
--- field_delta = the delta of the field's value which is part of this group

Pros:
- We are not creating a new entity type. This is instead a "glue" field which groups field values from various fields together
- If this field is deleted, all field values will still exist in their respective fields
- Fields can easily be added & removed

Cons/complexity:
- Core doesn't support rendering specific deltas of field values. The views module also faces this problem and has a (hacky?) workaround.

I've been thinking about that ^ solution for a while. Haven't had the time ($) to work on it, but I really don't see downsides to it. It's somewhat difficult to explain clearly without investing more time as well. Would love to work on it as a sponsored project if any orgs were open to that.

This conversation has diverged into discussing two distinct but related concepts.

Fieldable fields and compound fields.
Fieldable fields are a first class entity in themeselves with UI's for adding fields just like nodes. In the simplest of terms this is 'paragraphs in core'.

Compound fields are custom entities with prebaked fields.
Yes, if you have a dev available to write custom code, or a contrib module exists with one exactly how you want it, this solution works. But
This is not the issue this issue is about.

I'm updating the description to remove the words compound fields to reduce confusion between these two concepts.

The topic of compound fields is worthy of a documentation page if it doesn't have one already.

disambiguating field collections/fieldable fields from field groups.

I came across this presentation and slides, which covers this topic quite well.
Custom compound fields - DrupalCon Seattle 2019 - slides PDF

From reading through that the take is "compound fields" is unbrella term for field collections and custom field types that support multiple fields.

In case someone finds it reasonable that Field Bundle could be (or could NOT be) a candidate solution on entity-level (not field-level), please share your thoughts with us in #3240533: Consider shifting the work for the Field Bundle module towards creating fieldable fields in core.

@mxh It may be worth reaching out to @larowlan -- or visiting the #multifield channel in slack: https://drupal.slack.com/archives/CJHUL8VCN

To get a feel for the state of field_union and richfield modules. Given the field_union and field union and richfield modules are existing candidates, however appear to be stalled.

I don't have a reason off the top of my head for why field bundle would not be a viable candidate. In fact, there's no reason not to go
ahead and open a merge request and make a start.. There is absolutely no reasons to delay making a merge request even if it's barely started. There are good reasons to make the merge request early.

Will this be ready?

The Ideas project is being deprecated. This issue is moved to the Drupal project. Check that the selected component is correct. Also, add the relevant tags, especially any 'needs manager review' tags.

Changing to the standard issue template would also help other contributors.