The problem

We have a new configuration system, but we have not defined any standards in terms of how to implement it. I believe there are two main considerations that need to be addressed.

  • How much information should be stored in each file?
  • What is the naming convention for configuration files?

Proposed solution

This is just a stake in the ground to get discussion going.

  • Granularity
    • For configuration of 'things' (aka image styles, actions), we have one file per 'thing'. This has already been implemented in image styles and it seems to make a lot of sense.
    • For groups of settings (aka what is currently managed by system_settings_form()) I propose we keep one file per settings form per module implementing settings. I realize this is a somewhat arbitrary way to divide things up, but it mirrors the way the UI is organized which should improve findability and maintains a consistent organizational scheme. This is how the Performance settings were done in the initial patch. EXAMPLE: System module implements the settings form at admin/config/development/performance. This should be implemented as one XML file provided by the system module. Block module does a form_alter() to add the 'Cache blocks' setting to this form. That setting should be provided in a separate XML file provided by the block module. One file per form per module.
    • For settings NOT exposed in the UI, these should be grouped together in one more XML file, provided by the module responsible for those settings (in the case of core this will mostly just be system module.)
  • Naming
    • For 'things', files should be named <module>.<thing singular>.<machine_name>.xml. EXAMPLE: image.image_style.thubmnail.xml.
    • For 'settings', I don't have a good formal guideline, just that they should be named similarly enough to their accompanying form that it is recognizable. One thought would be to use form_id but that isn't always going to be ideal. Perhaps with a '.settings' suffix? EXAMPLE: system.performance.settings.xml, block.performance.settings.xml (note that right now we don't use the settings suffix).
    • I don't have a suggestion for the non-form things, other than it should be <module>.<something>.xml

Notes

  • Do we want to have a 'core' prefix for core-provided configuration? I think it is kind of pointless, but there might be some scenarios where it could come in handy.
  • I have heard some people suggesting that we should implement variables with one variable per file and I don't think this is workable. The number of files is already going to be enormous and I don't see a ton of gain towards adding thousands more to the mix.
  • If anyone brings up the possibility of changing file format in this thread, or challenges the reasoning behind XML being chosen, I will ask that they be banned. This is already going to be an enormous bikeshed, lets at least keep the bikeshedding on topic.
  • File names are our only querying mechanism at the moment, so naming them properly is very important. "give me the list of config files matching foo.bar.[^\.]*.xml" is all we have (for now) - additional filtering on specific properties inside the definition needs to happen in PHP, after unserializing the XML to PHP, and is therefore costly. This means that the file name for the configuration of a given "thing" must (of course) contain the thing's "primary key" (node type name, field name, view name...), *with a prefix* that unambiguously prevents clashes.
  • The automatic copying of module settings only has "by file" granularity. This means that the granularity of config files for a given subsystem is also determined by what we want modules to be able to specify as part of their initial config. This also has implications in terms of systems like Features, which may want to deploy only one setting in a file, rather than all of them.

Timebox

I'm going to let this issue run for two weeks, then take a decision based on feedback. This is not rocket science and we should be able to come to an agreement reasonably quickly. It is also really important to get a decision in place rather than spread the bikeshed across the 80 other issues where we are converting core subsystems to CMI.

Comments

heyrocker’s picture

Issue tags: +Configuration system

Tagging

heyrocker’s picture

Issue summary: View changes

Cleaning up html

webchick’s picture

"Do we want to have a 'core' prefix for core-provided configuration? I think it is kind of pointless, but there might be some scenarios where it could come in handy."

I don't think so. I think as much as possible, core should set the tone of what contrib should do, so just "module" is fine. Also makes it easier to move modules to contrib if we think that's a good idea.

"I have heard some people suggesting that we should implement variables with one variable per file and I don't think this is workable. The number of files is already going to be enormous and I don't see a ton of gain towards adding thousands more to the mix."

Ick. That sounds awful to me. From a DX perspective, I want to see related settings all together in the same file so I get the full context of what they do.

"If anyone brings up the possibility of changing file format in this thread, or challenges the reasoning behind XML being chosen, I will ask that they be banned. This is already going to be an enormous bikeshed, lets at least keep the bikeshedding on topic."

+1 :D Thanks, also, for time-boxing this discussion. I think that makes tons of sense. Whatever arguments are going to be made will have plenty of time to be made in that kind of timeframe.

On the specific proposal:

<module>.<thing singular>.<machine_name>.xml sounds good.

<module>.<something>.xml Do we need "something"? Why not just <module>.xml?

Crell’s picture

When we were working on Plugins back in Austin, we were looking at using

plugin.type.$scope.$type.xml (defines a type of plugin, eg "core.block" or "views.style".)
plugin.implementation.$scope.$type.$implementation.xml (defines an implementation, eg "core.block.whoisonline" or "views.style.slideshow".)
plugin.instance.$scope.$type.$machine_name.xml (defines an instance, eg "core.block.block_foo" or "views.style.some_id".

I still think that's a reasonably good model, as it is fairly self-documenting and makes discovery easier. I'd actually rather see most things end up in plugin-style implementation, and therefore plugin-centric config naming, rather than module-centric.

(For background on the above, see our writeup from November.)

heyrocker’s picture

@webchick I had thought that everything havings more than just the module name would make it a little easier to see what files are at a glance, but honestly I don't feel super strongly about it.

@crell I actually think the plugin model makes some sense, except for the fact that we don't currently have plugins and realistically won't for quite a while, and im unwilling to let this be a bottleneck for the CMI work. So I think we can keep that in our back pocket but for now we need to stay module-centric.

Crell’s picture

I wasn't suggesting we bottleneck on it, just providing a thought-through perspective and rough roadmap.

I think in general, config files should be named for the thingie that they are *relevant to*, not to *the thing that provided them*. So module-centric config files should be used only for configuration that is relevant exclusively in the context of that module. Things that piggy-back on something else should use that something else. Eg, node type information comes from a dozen places and modules, but should all be in one node type config object/file.

nicl’s picture

I don't understand fully how we are going to group config from the above (I'll need to read it more carefully when I have time). But does the system proposed prevent a kind of free-for-all like we have currently in the modules admin page - where module owners add so many random groupings that the groupings become useless?

JurriaanRoelofs’s picture

I propose we keep one file per settings form per module implementing settings.

There is also the theme settings form which might need some special attention. This is a single form that can be configured by themes, base themes, sub-themes, sub-sub-themes etc. and sub-themes inherit (default) configuration of base themes.

Also, some themes, like my responsive themes have several hundred settings, as opposed to older themes that just have several settings.

Just adding my 2 cents, which is probably an edge case.

yched’s picture

A couple technical constraints that affect granularity and naming - mostly pertain to 'things' files rather than 'settings' files.
Sorry, a bit long, but I don't think we have those written down anywhere right now, and they're an important part of the heuristics around 'config files layout'.

  1. File names are now our only querying mechanism.
    E.g : give me the list of all "things" (e.g node types, or image styles, or views, or field instances on a given bundle...)
    In D7 : "SELECT * FROM {node_types}" - possibly with restricting with a couple WHEREs if needed
    In D8/CMI : "give me the list of config files matching foo.bar.[^\.]*.xml" is all we have - additional filtering on specific properties inside the definition needs to happen in PHP, after unserializing the XML to PHP, and is therefore costly.
  2. This means that the file name for the configuration of a given "thing" must (of course) contain the thing's "primary key" (node type name, field name, view name...), *with a prefix* that unambiguously prevents clashes :
    e.g node.bundles.[type_name].xml rather than node.[type_name].xml, so that an hypothetical node.some_other_stuff.xml stays out of our "node types list".
  3. We should IMO favor grouping of related stuff under a given prefix :
    node.bundles.page.fields.body.xml : the definition of the 'body' field instance on the 'page' node type.
    node.bundles.page.contrib_foo : The settings on page node handled by contrib_foo module.
    So that deploying the 'page' node type means pushing node.bundles.page.*
  4. The automatic copying of module settings only has "by file" granularity :
    xml files provided in sites/[site]/[module]/config/*.xml are copied as is over to the main file store on module install. Adding a config snippet *within* an existing file is not supported.

    This means that the granularity of config files for a given subsystem is also determined by what we want modules to be able to specify as part of their initial config.

    For example, within Field API, I'd love to move "display settings for the various view modes" out of the $instance definition, and into separate node.bundles.page.displays.[some_view_mode].xml files, listing all the display settings for all fields and properties of a page node in a given view mode. Less memory bloat.

    But this means that a module that wants to add a new field instance to 'page' nodes would need to:
    - ship the instance definition in a config/node.bundles.page.fields.my_field.xml file - no problem
    - *insert* a snippet for the new field within the existing node.bundles.page.displays.full.xml - not doable currently, would have to be done "old-style", by api calls in hook_install().

EclipseGc’s picture

Yeah, so a comment on yched's point. What he's suggesting IS doable but has to be done in hook_install/uninstall. This isn't necessarily a bad thing, but whatever.

yched hits on this some, but containing information in a single config file per "thing" actually feels nicer since we have a single canonical place to find out anything about that "thing" from a configuration stand point. Doing a lot of this with blocks right now, I would be aghast if some module came along and started saving config data for my block (via it's form alterations) in a separate CMI file. And this dovetails nicely to injecting configuration where we can. I don't know if it's reasonable to discuss that at this point, but injecting a single CMI object and having that be the canonical source (as opposed to letting the form name the CMI object it wants) feels a lot cleaner to me. Again that may not be practical, and actually falls into the blocks initiative to some extent (though to WHAT extent is still up for discussion), but I just wanted to mention that form elements asking for (for example) $form_state['config']->get('my_value'); is WAY more palatable to me than having a $config = config('my.config.file'); call at the top of the form. I won't be a pain on this point no matter what, but if it's practical, now is probably the time to discuss it.

From a naming convention perspective, I think heyrocker and yched pretty much beat that horse to death successfully, and I have nothing to add.

Eclipse

yched’s picture

I would be aghast if some module came along and started saving config data for my block (via it's form alterations) in a separate CMI file

That's a real question IMO. Do we want to standardize on xml files for "things" as "free for all dumping grounds" for 3rd party extensions of those "things".

That is, should image.module take care of saving and loading image.style.[style_name].xml files containing altered-in 3rd party properties that are not part of the official structure for an image style ?

{user}.data, anyone :-) ?

(also, i think it was stated in #1270608: File-based configuration API that drupal_alter() on configuration was nonsensical)

[edit: Field CRUD D7 kind of does that : {field_config}.data holds "the rest" - everything in the $field array that doesn't have it's own column in the table. That's probably one of the worst design decision we made, and I have currently ripped that off in the $field->save() OO methods in my "Field API as plugins" sandbox]

sun’s picture

re #10: The answer might be "yes".

E.g., merely consider field (instance) configuration. Forgetting about technical details for a moment, it's a giant hook_form_alter(), since no field is actually owned by Field API. Thus, storing configuration for a field means to store configuration for another field type module, field widget module, and field formatter module.

That said, Field API's architecture would allow it to load the individual configurations for each module separately during regular runtime. That's however not really or not reliably the case for other (non-field) configuration that is enhanced by other modules.

Splitting configuration means that we'll lose the envisioned namespacing hierarchy. Configuration for a field field is not found in entity.node.bundle.article.field.file.instance.attachment but instead in file.field.file.instance.attachment, whereas the latter needs to store pointers to the entity types and bundles. That said, the latter might even be more correct, as I'm not sure how we'd store configuration for field instances that are attached to multiple bundles with the former.

Lastly, note that the previously discussed drupal_alter() was with regard to loading configuration; i.e., invoking drupal_alter() after a configuration object has been loaded. This question here is on the opposite end, saving configuration.

sun’s picture

Issue summary: View changes

Added timebox

cosmicdreams’s picture

for a naming convention, I would prefer <module>.<thing singular>.<unique id>.xml which is nearly exactly what the OP suggests. Instead the unique id can be a UUID when one is available.

yched’s picture

@sun

consider field (instance) configuration. it's (... kind of ...) a giant hook_form_alter(), since no field type is actually owned by
Field API. Thus, storing configuration for a field means to store configuration for another field type module, field widget module, and field formatter module.

Well, precisely not :-) The structure of a $field or $instance, even in D7 big-array-style is very predictable, and precisely isolates the spots that are 'field type'-dependent, or 'widget type'-dependent :
$field['settings']
$instance['settings']
$instance['widget']['settings']
$instance['display'][$view_mode]['settings']
And you know precisely what to expect in there : an array whose keys are the defined in hook_[field|field_widget`field_formatter]_info() (or associated _info_alter() hooks)

With $field and $instance as classed objects (most probably w/ protected properties & getters / setters), I'm not even sure how to deal with "stash your own 3rd party properties somewhere, I'll save and load (and not mess with) them for you" - other than reserving a well-defined $field->external property / <external> entry in the XML to serve as dumping ground, which then furiously resembles the $user->data / {users}.data stuff we all love.

Configuration for a field field is not found in ...[snip]

FWIW, and for the sake of discussion, here's the config layout I had in mind and discussed with @heyrocker back in BADcamp :
- field.[field_name].xml
$field definition - does not refer to any instance
- [entity_type].bundles.[bundle_name].instances.[field_name].xml
$instance of a field on a given bundle - refers to a field name (id, actually)
Possibly, not sure :
- [entity_type].bundles.[bundle_name].displays.[view_mode].xml
display settings and weights for all the 'components' of an entity in a view mode, including non-fields (which are currently stored in {variables}, will need a new home)
- [entity_type].bundles.[bundle_name].form.xml
same for entity create/edit form

--> Deploying the 'page' node type means shipping node.bundles.page.*, plus, indeed, packing along the corresponding field.[field_name].xml files. Fields are cross-bundles by design, and we don't want to duplicate the definitions within each bundle "namespace".

This, of course, assumes that contribfoo.module, that has stuff to say about page nodes, is able to secure a non-clashing space at node.bundles.page.contribfoo.[...].xml.
OR, as EclipseGc mentions, that node.bundles.page.xml can somehow carry the extra stuff from contribfoo and friends.

the previously discussed drupal_alter() was with regard to loading configuration; i.e., invoking drupal_alter() after a configuration object has been loaded. This question here is on the opposite end, saving configuration

I don't think I see the difference, nor why the latter would be more doable that the former. My point in #1270608-110: File-based configuration API was that the only stuff we currently know how to alter() is either :
1) _info() hooks or similar (hook_menu) - lives in code, altered in code: simple.
2) runtime stuff that doesn't get saved back (a form before it's displayed, a query before it's run, the display params for a field before its formatted...)

Crell’s picture

Another thing to consider that I discussed with Eclipse earlier today is that there are many good use cases for "nesting" configuration. Eg, a text format should consist of 3-4 filters, each with a configuration. There's no good reason to give each of those configurations its own existence, but the text format itself is the amagamation of those. So we'll want to store that as 3 config objects (one for each filter) aggregated into a single text format (with machine name), with an order.

Views and blocks/layouts are other examples of that pattern.

Not sure how that impacts this discussion, but "config objects that do not exist except as a subset of another config object" is a not uncommon use case that should be considered.

yched’s picture

One input format in one single file : fully agreed, just like image styles are currently handled.

[edit: and I'd not consider that "the configuration of a filter within a given input format", "... of an image effect within a given image style", "... of a display within a given view", are config objects to begin with. The atomic config "thing" is the input filter, the image style, the view]

Shameless plug to #1470824: XML encoder can only handle a small subset of PHP arrays, so switch to YAML, though - storing an arbitrary list of elements (filters in an input formt, image effects in an image style) in CMI currently requires painful legwork from the client subsystem.

heyrocker’s picture

I can see how we have a lot of considerations to go through for 'thing' configurations, especially for more complex use cases. I'd like to find a time in Denver to talk this out, as it seems like having a white board in front of everyone would be very useful. We can even Skype in yched if possible, I'd really like him to be a part of the discussion given his experience with the most complex use case we face. Yched if there's anyone you can recommend who will be in Denver, then that would be great as well.

There is one comment I'd like to respond to from Crell that I think bears some further discussion

I think in general, config files should be named for the thingie that they are *relevant to*, not to *the thing that provided them*. So module-centric config files should be used only for configuration that is relevant exclusively in the context of that module. Things that piggy-back on something else should use that something else. Eg, node type information comes from a dozen places and modules, but should all be in one node type config object/file.

One big issue with this is that it makes uninstalling modules and rolling back changes far more difficult if everything is merged into one file. Right now, if you uninstall a module, all you have to do to cleanup is delete its config files. If you want to roll back changes, you can just grab the default config the module provided and copy it back over the original one. If we are merging all these changes into a single file, our lives are going to become immensely more complicated. This does mean that users will have to push more than one file to push a change to a node type or view or whatever, but honestly I don't think that is a horribly big deal. You check what has changed into git, and pull on the other side. If you don't know you or aren't using version control on a simple site, just push the whole directory. I'm happy to take this tradeoff to keep our architecture a lot simpler.

heyrocker’s picture

So to this point most of the discussion has been around the 'thing' config. What about settings? Does anyone have any issues with what I proposed above? Getting some agreement on this would allow me to start making a huge pile of 'Convert to the config system' issues, and the great thing about these is that they would be really easy for anyone with basic knowledge to do. While I realize that organizing settings around the forms on which they are implemented and the modules that implement them is not a particularly meaningful way to arrange things, but it is easy to explain and follows what we currently have. If we want to reorganize later thats fine, but any objections to going forward with this for now to keep things moving?

Crell’s picture

So as a guide, there's several use cases to consider.

1) Installing/uninstalling a module. Greg is correct that this is totally easy when config files are all namespaced by module, and a PITA when namespaced by "thing affected by it".

2) Loading, editing, saving configuration. This is much much easier when configuration is clustered by the thing being loaded, edited, and saved. It's a huge hassle to try and get all of the information about a node type right now (in D7), because it's scattered across about 4 different variables with different internal structures, some namespaced by node type and some in one big array, plus some other stuff that's off in SQL tables... and that's just core. Once you add contribs, it gets even worse. Rather, creating and deploying (and "featurizing", even if manually) a node type should all be happening in a single logical object, even if that object may end up as multiple files on disk. (Although IIRC we do have a 1:1 mapping from config object to config file, which makes total sense.)

So we're going to get screwed either way in some use cases. Can we do a little of each, depending on which is a better trade-off in each case? Vis, there's a ton of variable settings right now that are clearly module-dependent (mostly things handled by system_settings_form()), for which clustering by module is fine. However, for Views, Node Types, Plugins, and other stuff like that, it's much more convenient to cluster by thingie being configured rather than by providing module. (I'm thinking default-publish state in core, auto_nodetitle in contrib, comment settings, etc.)

Greg, didn't we have support for both approaches back at DrupalCamp Denver, when we were still thinking in terms of hierarchy?

heyrocker’s picture

Honestly I don't remember what all we talked about back then and I don't think we ever got much into these kinds of discussions. I'm still thinking this through, not sure what kind of ideas I have here.

beejeebus’s picture

i think whatever we do, we've got to support modules being able to add 'snippets' to other config files without having to run code during hook_install(). otherwise, we end up with 'here's the defaults from the module. except for the other bits.', which is kind of worse than useless IMO.

catch’s picture

Another thing here, this might have already been covered.

Using the node types example, it's mostly variables + hook_node_type_$op(). It's very easy to do dynamic defaults with the variables system since it defaults to whatever you passed in, which just happens to be a very annoying pattern that people have tried to get rid of for years as well.

If we move to providing those extended defaults via snippets in config files, then they can't be in the format [entity_type].bundles.[bundle_name] - since the module can't possible know the value of [bundle_name] (or [entity_type] if a site is using ECK).

That leaves acting on hooks more or less as happens now - makes it easy to add things dynamically, but harder for export/import (and sometimes harder to grep for where things came from too - if you load a node type there's no predictability about where the data came from at all).

Either that or supporting some kind of wildcard when loading in defaults - so we'd fall back to wildcard file if a more specific file doesn't exist.

For the one-off settings form, something along the lines of $module.settings.$group.xml (with $group mostly mapping to form ID) seems reasonable? I think it'd be great if we could try to come up with something usable for that so follow-up issues can be opened, then keeping working on the 'things' which is a lot harder to get right. Modules may wan to form_alter settings forms but that's a much more limited use case and it'll likely end up a sub-set of what we do for things.

heyrocker’s picture

This weekend I starting opening conversion issues for various settings forms in order to take advantage of a dozen or so new contributors who showed up at my code sprint. In general they have been naming these files $module.$group.xml, however I agree that $module.settings.$group makes sense. Does anyone else have a problem with this? If not I'll go through and reroll those patches and post a followup issue to change the ones already committed.

Also thanks everyone for keeping this discussion focused and on topic, I think its going really well. I appreciate it.

Crell’s picture

I think that's reasonable for a first pass. Once we get to the more complicated things like plugins, node types, etc. we can try to figure out how to make object-centric files work, since I do think there's a good use case for both of them.

effulgentsia’s picture

$module.settings.$group sounds good to me. In general, I prefer sticking to singular when the contents of a file is about a single thing, or in other words, change #13 to bundle, instance, and display (as per #11) rather than the plural forms. However, in the case of settings, a file can contain more than one setting, so here the plural is justified.

For example, within Field API, I'd love to move "display settings for the various view modes" out of the $instance definition, and into separate node.bundles.page.displays.[some_view_mode].xml files

In http://groups.drupal.org/node/160144#comment-715359, EclipseGc indicates that this might actually become the province of Block/Layout configuration, and not Field API. If this needs more discussion, let's open a side issue for it, but just mentioning it here so that we don't dwell on this use-case if it's about to go away.

Another thing to consider that I discussed with Eclipse earlier today is that there are many good use cases for "nesting" configuration... There's no good reason to give each of those configurations its own existence... Views and blocks/layouts are other examples of that pattern.

Above is from #14, and maybe my understanding of this is different than what Crell intended, but I would think in the case of Views, that we'd want each View Display as its own "thing" rather than an entire View as a single "thing".

For #18.2, would it make sense to allow modules to provide *files* that are grouped by module, but for the *active store* to aggregate and group by object? Or is breaking the 1:1 mapping between files and active store a big no no?

EclipseGc’s picture

@effulgentsia

I think views will actually be a single configuration object. The idea is that multiple plugins might support that, but a single config will be broken into multiple arrays as necessary and passed into the plugin system when needed. So no, I'm pretty sure the idea is that you want the whole view as a single CMI object. The same is true of layouts.

Eclipse

yched’s picture

@effulgentsia :

[field display settings for the various view modes] might actually become the province of Block/Layout configuration, and not Field API

Yes, I saw that thread, and even though I still have to assemble some articulate feedback, I very much like the idea.

I don't think this changes the nature of my point 4) in comment #8, though. Even if field display settings are held in Layout land, this is still config in a CMI file somewhere. A module that creates a field instance to page nodes will need to specify its display settings, be it within node.bundle.page.display.[view_mode].xml or in some.layout.config.file.xml.

What I mean is that the ability or not to provide *snippets* of a config file in the /config folder at module install has an impact on the possible layouts of the config files for a subsystem (field display settings or layouts, in this case).
Hence, that's part of the heuristics for "leveraging CMI for 'things' config"; there might be other similar cases in other subsystems, my "field display settings" example is just an example.

effulgentsia’s picture

Yep. Also, I see a node type as similar to a View. So, if we think a View should be a single big config object (as per #25), then I would also expect us to think that a node type (with all its .instance.*, .form.*, and display/layout info) is a single big config object (.field.* is separate though, since that has identity independent of any particular node type). OTOH, if we partition a node type into separate mini objects (instances, form, displays/layouts, etc.), then I would expect us to also partition a View into mini objects.

Node types and Views are great use-cases to consider in terms of being "large" objects composed of many parts that have no meaningful identity on their own, but that can be provided by and extended by many modules.

andypost’s picture

Having big config objects could bring a task of sync/locking to prevent concurrency|race issues

heyrocker’s picture

Note that there is some discussion happening at #1599554: Tutorial/guidelines for how to convert variables into configuration which overlaps this issue a bit.

sun’s picture

Priority: Major » Normal

This is sorta sorted out for now, at least for the current settings form conversion issues. Since we're still discussing the overall architecture for dynamic configuration in #1668820: Concept, base class, and interface for Configurables (e.g., node types, image styles, vocabularies, views, etc), let's downgrade this issue's priority, so as to not block other core patches that are ready in the queue right now.

heyrocker’s picture

Status: Active » Closed (fixed)

I think we've pretty well got this sorted at this point given we've now converted almost all of core.

Gábor Hojtsy’s picture

A doc like this would still be very useful for contrib, no?

Gábor Hojtsy’s picture

Issue summary: View changes

Rolled in some comments from yched