Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Problem/Motivation
- Leverage symfony's event model to make #1515312: Add snapshots of last loaded config for tracking whether config has changed, #1842956: [Meta] Implement event listeners to validate imports and #1808248: Add a separate module install/uninstall step to the config import process easier
- Reduce the amount of procedural code in the Configuration managemet sub-system to make it more consistent
- config.inc contains a mish-mash of config_sync_*() and config_import_*0 functions
- Reduce amount of unnecessary config import / sync code loaded during normal runtime
Proposed resolution
- refactor code to create ConfigSync class to manage configuration import and installs
Remaining tasks
- needs validation of approach
- needs tests for Symfony events
Comment | File | Size | Author |
---|---|---|---|
#80 | 75-80-interdiff.txt | 17.49 KB | alexpott |
#80 | 1890784.80.drupal8.import-refactor.patch | 70.95 KB | alexpott |
#75 | 73-75-interdiff.txt | 1.88 KB | alexpott |
#75 | 1890784-75-import-refactor.patch | 68.45 KB | alexpott |
#73 | 70-73-interdiff.txt | 6.4 KB | alexpott |
Comments
Comment #1
alexpottHere's an initial patch the converts config_import, config_install_default_config, and ViewTestData::importTestViews to use a new ConfigSync class to manage and perform the synchronisation tasks.
It also fixes a bug in ViewTestData::importTestViews - the current version only works for a single module in the modules array - fortunately we only ever pass it a single module.
Comment #4
alexpottThis patch contains the fix from #1889752: Remove unnecessary manifest creation in config_install_default_config() and so was broken in the same way as explained in http://drupal.org/node/1889752#comment-6949574
Additionally the processed list was not correctly maintained so config entities were imported twice resulting in views with no uuid.
Comment #6
alexpottDoing a retest as install works locally for me.
Comment #7
alexpott#4: 1890784.4.drupal8.config-sync.patch queued for re-testing.
Comment #8
moshe weitzman CreditAttribution: moshe weitzman commented#4: 1890784.4.drupal8.config-sync.patch queued for re-testing.
Comment #10
gddRerolled
Comment #11
gddComment #13
gddSo four hours ago I was getting on a plane and the patch in #4 failed to apply. So I rerolled it, incorrectly, without the new files. So now I just started over with a fresh checkout and a fresh branch ... and the patch in #4 now magically applies. So I'm throwing it back to the bot.
Comment #14
gdd#4: 1890784.4.drupal8.config-sync.patch queued for re-testing.
Comment #15
tim.plunkettIt conflicted with #1831264: Use the Entity manager to create new controller instances, rerolled.
Comment #16
gddI'm not sure I see where this patch is a net win, but then again I don't see moving procedural code to objects as being a net win in and of itself in general. Is there a specific reason why this is better encapsulated into objects than just leaving it in the include file?
Some general comments:
- Calling a function called setChanges() with no parameters strikes me as odd. I'd actually expect the change list to be generated automatically when the ConfigSync object is constructed since in the majority of cases you'll just want to construct and sync. If you want to set a different change set, use setChanges() then.
- I think doSync() would make more sense just being named sync().
- I do like the switch to using events, as it makes our notification system more consistent with what we're doing for overrides.
Comment #17
Anonymous (not verified) CreditAttribution: Anonymous commentedthis isn't a switch to events. it's an addition of an event - nothing equivalent existed before this patch.
i agree with #16 - i don't see the benefit here with OO. i'd rather we dropped this patch entirely and changed it to just the change to add the sync event.
if we are going to OOify this, then i think we need to register this with the container, and inject the event dispatcher? and invoke Crell on it?
Comment #18
alexpottI started this patch after spending some time thinking about the config validation patch. I realised that in order to mirror the rest of the Config sub system we'd need to use Symfony events to be more consistent. Once I was going down this route I wanted to pass the event something to act on and that could provide functions to get information about and manage the synchronisation - hence the conversion to OO.
I've been working on the addition of a validate event and moving the config name validation to an event subscriber to try to show the value of this approach. In doing this I've created a config_sync service on the container. This has turned out to have a significant benefit. At the moment when we press submit on the ConfigUI we calculate the changelist to build the form and then we rebuild it completely during config_import. Registering the config_sync service allows us to do this only once.
I think we might discover further advantages to an OO approach - for example - creating an event to be able to defer changes or reorder the changelist depending on dependencies.
Comment #19
Anonymous (not verified) CreditAttribution: Anonymous commentedi'm mostly meh about the OO conversion, so i don't mind if it goes in. wouldn't be my first choice, but meh ;-)
if you want an event fired after import, lets discuss that.
if you want to implement deferring, reordering based on dependencies etc during import, lets discuss that.
but - can we please not push functional changes as side effects of implementation details?
Comment #20
alexpottThis patch:
All I'm trying to do is to refactor config import to point where we can solve some of the issues we have in the follow up patches. I was not saying that this patch should implement deferring or re-ordering. What I was trying to say is that we should try to make these easy to implement. In my opinion if we keep the current procedural approach this will lead to extra complexity.
The interdiff is bigger that the patch because some of the new classes have been renamed.
Comment #21
sunI absolutely and totally agree that config.import/config.sync should be a service. @alexpott and I discussed this at DrupalCon Munich already, with the important conclusion that turning this into a classed service will be the only way for contrib/custom sites to invent a better configuration import/sync/staging implementation in D8. Procedural functions can't be swapped out.
And as things currently stand, contrib will have to re-invent in D8, in order to fix and close the gaps that we failed to address in core...
(I, for one, will probably work on an implementation that replaces all machine names with UUIDs in config names across the board, in order to truly guarantee uniqueness.)
So, thanks for kick-starting this, Alex. Will try to review this patch some more in-depth as soon as possible. Any chance you could push your code into the cmi sandbox? :)
Comment #22
alexpottBranch pushed to CMI sandbox... http://drupalcode.org/sandbox/heyrocker/1145636.git/shortlog/refs/heads/...
Comment #23
Crell CreditAttribution: Crell commentedsun is correct. A straight conversion of procedural functions to injected DIC services automatically gets you:
1) Lazy-loading code.
2) Lazy-initializing services (when you need it it's initialized, but you don't spend time initializing until you need it).
3) Contrib can rip it out and replace it if it wants, as long as it follows the interface.
That's a strong case for service-ifying most systems all on its own, IMO.
I don't fully understand the problem space, but since I was invoked, some additional cleanliness/architectural comments below. From my incomplete read through, this does seem like a reasonably good use of events.
Don't bother with the hard coded drupal_container() call. If the event dispatcher is required, then it's required. Passing it in is what the DIC is for.
Shouldn't this be config.importer? It's an object that does things, so it has an "er" suffix for thing it does.
This should be done in the container configuration. You can call:
And you're good.
If this is something that gets set per-use case, then it should either be multiple DIC registrations or a factory in the DIC, I think. (Although I don't know the specific use case in detail.)
Ibid.
Or, really, this is in a test so you shouldn't be calling the container in the first place. Especially inside DUTB. Just make objects yourself and use 'em. That's how you keep unit tests unit-y.
Comment #24
alexpottThanks for the great review @crell!
Patch attached tries to implement all your suggestions:
Unfortunately the interdiff can't be constructed since this is a reroll.
Pushed rerolled branch to heyrocker's sandbox.
Comment #26
alexpottThis failed on
Drupal\text\Tests\TextSummaryTest
The test did not complete due to a fatal error. Completion check TextSummaryTest.php 169 Drupal\text\Tests\TextSummaryTest->testOnlyTextSummary()
Does not occur locally - sending for a retest.
Comment #27
alexpott#24: 1890784.24.drupal8.config-sync.patch queued for re-testing.
Comment #28
alexpottRerolled for #1515312: Add snapshots of last loaded config for tracking whether config has changed and #1889620: config_install_default_config() overwrites existing configuration.
Config snapshot creation is now an event subscriber to the import event.
Comment #29
alexpottForgot to attach diff of 24 to 28 - it's not a true interdiff as this was a reroll...
Comment #30
alexpottThe
removeChangelist
sucks...Patch attached fixes this and removes the config.installer service from the container as this is only used in one place and does not need to injected.
Comment #31
chx CreditAttribution: chx commentedInvoking https://twitter.com/chx/status/242326777538695168/photo/1 , trying to imagine what stof would do with this API.
What about
Comment #33
alexpottThanks for the review @chx
Comment #34
chx CreditAttribution: chx commentedI think this is RTBC but as sun written this , he should review it.
Comment #35
Crell CreditAttribution: Crell commented#33: 1890784.33.drupal8.config-import.patch queued for re-testing.
Comment #37
alexpottWorking on a reroll... Changes due to the config context patch :)
Comment #38
alexpottRefactored the patch again due to the impact of context and other changes in core since this #33. This really requires a re-review... and interdiff is not possible because of the re-roll.
Summary of changes to #33:
Comment #40
alexpottOkay some views test's have an unmet dependency on the theme system being configured...
And config factory reset needs to clear the static cache not just reinitialise the config. I think this is the correct thing to do because currently we are reinitialising all the config stored in the static cache on a reset() but we have no idea if we're actually going to use it again.
Comment #41
alexpottComment #42
moshe weitzman CreditAttribution: moshe weitzman commentedThis one has gone a bit quiet. Do we still want it?
Comment #43
moshe weitzman CreditAttribution: moshe weitzman commentedI nIRC, @alexpott confirms that we still want this and he offerred to reroll tomorrow.
Comment #44
alexpottRerolled...
Comment #45
alexpottAfter chatting to @timplunkett in IRC decided to remove the config_import function so everything is contained with the ConfigImporter class.
Comment #47
alexpottRerolled...
Comment #48
tim.plunkettThis is beautiful. Once this is in and I reroll #1945398: Convert config_admin_import_form to a new FormInterface implementation and routing definition., it will be clear how nice this is. It really makes it clear how you can interact with the config import process, and will be immensely useful to contrib enhancements of this workflow.
I'll do a thorough review tonight.
Comment #50
alexpottMessed up the reroll of core/lib/Drupal/Core/CoreBundle.php
Some more DrupalUnitTestBase tests have undeclared dependencies on the theme system...
Patched attached fixes both these issues.
Comment #51
tim.plunkettThis makes me wonder if there is another way to pass the string to the service, so the constructor could be the same and you wouldn't need to always pass 5 of the same services
These could be
$this->container->get('config.importer');
, right?$this->container
This is a bit weird to change here, but I like the new name better anyway.
Comment #52
damiankloip CreditAttribution: damiankloip commentedI have probably missed something but if it's a reset, can it not just set $this->cache = array()?
Docblock
huh? :)
Adds
Isn't it the same to += the two arrays together, as then only keys that aren't already set will be added?
This would be changed to
strpos($name, 'manifest.') === 0
. It's all preference though I guess.Seems strange doing count there? Would we usually do !empty()? Either way, this will work just fine.
This snippet gets used a few times, not sure if it's worth trying to abstract this out at all? Probably not actually.
Maybe we should add some class constants for this event string?
What happens if an exception is not thrown? The tests will still pass. What we have done in tests before is set a variable and then evaluate after the try/catch.
Comment #53
alexpottThanks for the reviews!
In reply to @timplunkett #51
1. @timplunkett "This makes me wonder if there is another way to pass the string to the service" ... suggestions welcome :)
2. Fixed - did it this way due to @crell's last comment in #23 ... but that was before so many dependencies
3. Fixed
4. I had change
importChange
toimportUpdate
to match the $op's being stored in the changelists... create, delete and change just didn't seem right... have a look inConfigImporter::importInvokeOwner()
In reply to @damienkloip #52
1. Yep you missed that here we're clearing the static cache of a specific config object not all... see
ConfigFactory::reset()
2. Fixed
3. Fixed
4. Fixed
5. Nope we have numeric keys so this doesn't have the desired effect of unique array values.
6. The strpos version is not quite equivalent... the current version is also only testing the first 9 characters of the config name
7. You can't use
empty()
to test function returns - see http://php.net/manual/en/function.empty.php8. I tried but I agree probably not :)
9. The point here is that the events are based on the ConfigImporter instance's servicename... I don't think these should be class constatns
10. Added a $this->fail() if the exception is not thrown.
Whilst adding the docblock for ConfigImporter (@damienkloip's point 2) I realised that the comparison of configuration stored in different config storage controllers is a different concern than actually doing the import. The attached patch implements a StorageComparerInterface and 2 classes that implement it ConfigComparer and CongirComparerManifest - the difference being that one uses manifests to decide what to do for config entities. An additional advantage of this approach can be seen in the ConfigSnapshotTest where comparing the snapshot storage to active and staging is very simple (contrast with the patch in #50).
Comment #54
alexpottAnd one more thing...
After chatting with @beejeebus in IRC he pointed out that injecting a config context here made no sense as it only makes sense to run the import with a FreeConfigContext.
Comment #55
tim.plunkettI think entityManager is a better name here, and I think its wrong to typehint with an interface that doesn't have the method we want on it. Either we need an interface for EntityManager::getStorageController(), or we should just typehint with the class.
This is way out of scope for this issue, but the coupling of CMI to Config Entities here always bothered me greatly. Is it possible that the ConfigEntity system could subscribe to the event instead (in a follow-up).
It would break the $handled_by_module part, but I'm still unsure if we need that.
Needs typehinting, and we should move it out of the interface (we don't usually put __construct in interfaces)
public?
Comment #56
alexpott@timplunkett / #55
1. Agreed and fixed
2. Yep we could add an event which always modules to do handle an import instead of the general process - definitely out of scope here.
3. Fixed
4.
static function getSubscribedEvents() {
is how all the other core events are...Comment #57
tim.plunkettThis looks awesome. Thanks so much @alexpott!
Comment #58
YesCT CreditAttribution: YesCT commentedalexpott asked me to give this a standards look.
usually @see's are together at the end of the doc block, but iirc, they can also be placed inline if that helps communicate better.
small change in formatting for lists in comments.
http://drupal.org/node/1354#lists
changed to be consistant.
. Used ...
->
used
taking out the $lock
http://drupal.org/node/1354#var
one missing type in @param.
also, one has change instead of update.
Making them match (and also match the order in the hasChanges definition:
hasChanges($changelists = array('delete', 'create', 'update'))
=======
posting what I have so far. I can get back to this in a couple hours.
Comment #59
YesCT CreditAttribution: YesCT commentedThere is the config importer, comparer, and interface.
I updated to be more consistant, I wonder if it can use a {@inheritdoc}.
updating target comment to be "to write".
fixed copy and paste error. "to read" -> "to write".
needed to delete the @param for target that was removed.
this is also missing the @return.
Maybe this is an unrelated change?
I think this can be an {@inheritdoc} but maybe this was left here to be part of cleaning up this later. So I'll leave it.
I think these are usually alphabetical.
The Allowed... sentence seems about the function in general so moved it up to the function doc block.
added a new line at the end of the class because, mostly we do, and also, other classes in this patch do.
Comment #60
shnark CreditAttribution: shnark commentedI made the small change about the doc block and changed it to an {@inheritdoc} like YesCT sugjested in comment #59.
Comment #61
YesCT CreditAttribution: YesCT commentedTalking to alexpott and EllaTheHarpy in irc, we agree the hasChanges in ConfigImporter is different than the hasChanges in ConfigComparer, so renaming it with a better name: hasUnprocessedChanges.
I dont see a hasChanges in any of the use classes. Why does this work?
Is it because it's on an object of type ConfigImporter?
Then it might should be the new name hasUnprocessedChanges()
Comment #63
tim.plunkettI think these also needed switching?
Comment #64
YesCT CreditAttribution: YesCT commentedyeah, I convinced myself, and the testbot confirms. I'll do it. (since I broke it *grin*)
Comment #65
YesCT CreditAttribution: YesCT commentedHere it is.
Comment #66
alexpottReviewing the improvements made by YesCT and EllaTheHarpy I discovered a mistake I made in the documentation... well at one point it was true but now it a LIE :)
Comment #67
tim.plunkettGreat docs and naming improvements!
And the code still looks great.
Comment #68
YesCT CreditAttribution: YesCT commentedlooks good to me too.
Comment #69
aspilicious CreditAttribution: aspilicious commentedVery minor
param names don't match :(
Creates?
Comment #70
Anonymous (not verified) CreditAttribution: Anonymous commentedi think is ready to go.
rerolled for the doc only changes in #69.
Comment #71
marcingy CreditAttribution: marcingy commentedThis is likely to have a conflict with #1987660: Convert config_sync() to a new style controller. This should be committed first as the other issue is just moving code hunks it isn't rtbc but just in case.
Comment #72
YesCT CreditAttribution: YesCT commentedComment #73
alexpottTalking with @catch about this... he didn't like passing the service name in... and I think @timplunkett wasn't sure about it either.So I've implemented this using class constants.
Comment #74
tim.plunkettShouldn't this be
static::SERVICE_NAME
? That's what we use elsewhere.Comment #75
alexpottYep... doh!
Comment #76
tim.plunkettRTBC if green! This feels much better.
Comment #77
Anonymous (not verified) CreditAttribution: Anonymous commentedit hurts my head to ask this question, but:
does this mean that validation and import run against different config contexts? it hurts my head because i don't really know what that means, but my gut feeling is that is probably not a good idea.
Comment #78
alexpott@beejeebus I did consider this but the reason I did not do this is that I think it should the event listener's job to worry about context if it needs to... If it does it is probably incorrect :)... Because validators should be using the storagecontrollers attached to the storagecomparer to read config in. After all most validators are probably not going to look at what's there but what's coming and therefore will have to read from staging. Additionally what they are most likely to want to compare to active is stuff like machine names and uuid's and context messes will these... well... all bets are off.
Comment #79
catchLooks much better with the constants, a couple of things still confuse me bit:
Installing default config is sufficiently rare that I completely agree it doesn't deserve it's own service, however it's weird that there's a StorageComparer service and the class is also instantiated directly - does it need to be a service at all then, why not instantiate directly everywhere?
Class constant rather than a property on the instance.
This defines a service name, but it's not registered as a service. I don't think these are service names at all, but rather operation names or similar.
Comment #80
alexpottRemoved the services added by the patch... Catch is right these will not be used often enough to justify their existence.
Renamed SERVICE_NAME to ID as that's what it is an identifier.
Comment #81
tim.plunkettHaving them as a service allows the importing to be swapped out. I'm not certain enough of our choices to say that contrib won't have some better ideas.
However, I leave this to catch. I'm fine with either, just worried a bit.
Comment #82
catchI think if we allow the importing to be swapped out, we'd also want to allow the defaults installation to be swapped out as well. It's the mismatch rather than them being services that bothers me (although I can't see another reason than swapping the implementation for them to be such).
This feels like something we could change at any time but leaving this open for a bit longer in case people want to discuss.
Comment #83
catchWent ahead and committed/pushed this - if we want to make these services that'll be a simple follow-up - they're only called a couple of places.
Comment #84
effulgentsia CreditAttribution: effulgentsia commentedCommitted, so no conflicts to avoid anymore.
Comment #85.0
(not verified) CreditAttribution: commentedAdd issue that is postponed on this.