Problem/Motivation
ContextHandler::getMatchingContexts() is used to find which context objects match the needs of a given plugin.
Part of this filtering process includes checking the constraints of the available contexts, and comparing them to the any constraints on the definitions specified by the plugin.
For example, a Block plugins can specify that it requires an Article node.
If one of the available context objects provides an Article node, then it will be included in the result.
If another available context object provides a Basic Page node, it will be excluded.
Scenario:
A Block plugin specifies that it will accept a either a Book node or an Article node.
An available context object provides an Article node.
ContextHandler::getMatchingContexts() is called
Expectation:
The result will include the available context object, as it satisfies the requirements of the plugin
Result:
The result is empty, because the available context object was not both a Book and an Article
This is due to the following line:
if ($context_definition->getConstraint($constraint_name) != $constraint) {
return FALSE;
}
An additional problem:
Constraints expect to evaluate against a corresponding data object. In the case of bundles (and a few other constraint validators) that would be an entity. The context system can be functional before we have a given entity (say during administration), so constraints need to be validated against contextual metadata. If a context is available that documents it is a member of a particular bundle, our validators should be able to use that information.
Proposed resolution
Fix the ContextHandler to delegate Constraint validation to the validation system and allow constraints to be directly validated against a real or generated data object.
Remaining tasks
TBD
User interface changes
None
API changes
API addition to ContextDefinitionInterface, but with a 1:1 addition to ContextDefinition, so no BC issue
Data model changes
None
| Comment | File | Size | Author |
|---|---|---|---|
| #90 | 2671964-context-90.patch | 28.68 KB | tim.plunkett |
| #90 | 2671964-context-90-interdiff.txt | 1.14 KB | tim.plunkett |
| #82 | 2671964-context-82.patch | 28.56 KB | tim.plunkett |
| #82 | 2671964-context-82-interdiff.txt | 1.22 KB | tim.plunkett |
| #79 | ContextDefinitionInterface.png | 35.66 KB | larowlan |
Comments
Comment #2
eclipsegc commentedSooo I investigated being able to validate directly against other definitions, unfortunately the RecursiveContextualValidator::validateNode() makes a direct call to TypedDataInterface::getCanonicalRepresentation() which it then passes the result of as what we're validating from that point forward. This means that I'm not going to be able to reliable use the validator without inventing the data type generator stuff I suggested as the other potential possibility... YUCK.
Eclipse
Comment #3
eclipsegc commentedOk, so after a bit more digging, TypedDataManager::getCanonicalRepresentation only attempts to get the value if the data definition specifies that things be unwrapped. All the data types I looked at don't specify this, which means they unwrap by default, thus value, so I've introduced a new data type that does not unwrap for canonical representation, and that means our data type itself is passed to the constraint validators. This is great because it means that we can type check the passed object and validate for both the object or a representation of the object. I'm not sure where to put the data type right now and I have overridden the Bundle, EntityChanged and EntityType validators. I'll provide all that code shortly, but I wanted to put the ContextHandler changes up first just for sanity.
Eclipse
Comment #4
imalabyaComment #6
eclipsegc commentedThis patch introduces a new data type plugin which will not unwrap for validation. Updated some of the entity constraint validators to compensate properly for this. Others may need updating. Let's see what testbot says.
Eclipse
Comment #8
tim.plunkettThis really needs tests to prove the bug.
Comment #9
eclipsegc commentedYeah, I'm working on tests now. Unfortunately the existing test make up non-existent constraints, so it's not immediately clear how things break unless you're familiar with how constraints are intended to be used. We weren't when we were working on the ContextHandler. :-(
I'm working on tests now.
Eclipse
Comment #10
dsnopekEclipseGC explained this to me on IRC, and I sort of get it. But the issue summary here is really difficult for me to parse, and I think should be updated to describe the problem more clearly.
Comment #11
eclipsegc commentedOk, let's see how testbot likes this!
Eclipse
Comment #12
eclipsegc commentedComment #13
eclipsegc commentedComment #14
tim.plunkettIf this were a new test, I would say "this should use a @dataProvider". But you're removing one instead! Why?
Same here!
Comment #15
eclipsegc commentedIt was easier for me to read, I figured it would be for others too. I'm happy to go back to one, but a big array of arrays is kinda hard to read. That's all.
Eclipse
Comment #16
eclipsegc commentedSwitched it back.
Eclipse
Comment #17
eclipsegc commentedOk, added docs to all the things, tried to not rewrite all the methods and parameters in the test methods. Should be a little cleaner.
Eclipse
Comment #18
eclipsegc commentedComment #19
tim.plunkettSure, we all love the new array syntax. But now I can't tell what you're changing here. Or in this entire test.
All of this can't be 100% wrong...
Comment #20
eclipsegc commentedSure, it wasn't wrong for the old ContextHandler, but the new class is actually going to delegate validation of constraints to the actual system that does that job, so made up constraints won't work any longer, and since the specific problem I wanted coverage for happens within entity bundles, I had to convert to real data types as well, so "wrong" is a strong word, but it's not going to work in the new version of the class appropriately.
I've included a FAIL version of the test against the old ContextHandler just to illustrate the sort of thing that SHOULD work but currently doesn't.
Eclipse
Comment #22
eclipsegc commentedComment #23
fagoYeah, this needs to do proper matching. We are working on having a type-inheritance systems for that in Rules.
While the premise is correct, I don't think the conclusion is. Constraints are for validating data, yes. When you need to check whether some context definition matches with some data definition that's something different - I don't think it makes sense to use the constraints which already have the logic for checking the data, for checking the metadata also.
I think the case of bundles can be covered perfectly fine with a data type based matching system, e.g. entity:node:page matches entity:node. Regarding, the more complex case you suggested, having a context plugin that requires a node with a certain field X: I'm not convinced we need to have support for that, e.g. Rules never was able to express that, still the matching was powerful enough in D7.
However, one thing that we had in d7 which we are not use it's required to bring back in D8 is support for having a contextual plugin with a context that must be of a list of allowed types, e.g. entity:node:page OR entity:node:article. Instead of that, constraints might be a good way to express that as you can specify that you require a node with the constraints of having either an article or page bundle. However, the specified constraints are metadata, so I could see us making use of them for matching metadata in combination with metadata assertions. Metadata assertions, are what Rules makes for pre-computing the available context: e.g. when you have a node type condition checking for a page, it can provide the assertion that after its execution the passed context is not only of type entity:node but also has the constraint "Bundle is page" fulfilled. Given we have that in place (we are working on assertions still) we could leverage that for context-fullfillment-checking by matching the asserted constraints with the needed constraints.
Thus, we'd not execute the constraints at configuration time, we'd only match the arrays of known, asserted constraints with the array of required constraints. Would that meet your use-case also?
@core-context-handler:
We actually do not use that, it has not much logic we could re-use, so it was simpler to have our own code for mapping context for now. FYI: It's no service, but a (re-usable) trait: https://github.com/fago/rules/blob/8.x-3.x/src/Context/ContextHandlerTra... Would be good if we could merge those approaches in the long run though.
Comment #24
fagoBesides, I've troubles finding issues related to the Context System in core. Should we start using some tag at least to make those issue findable? Having a component would probably be better, but that doesn't match the code-structure. (Would have been nice there as well.)
Comment #25
eclipsegc commentedNo, CTools could not use that. The above code actually fulfills the exact situation you laid out as "missing" and will allow a context of type entity:node of bundle page or article to match on a given plugin. CTools does do the exact field example I gave, we in fact already have code for it right now and just didn't do the constraints properly because of this bug, which could lead to some runtime nastiness, which core only saves us from because it's block placement is so naive. A PageManager or Panelizer based block placement system needs to be a lot less naive, and we have the code to support that, if only the context handler is imbued with this logic.
I don't think updating a handful of constraint validators to respond appropriately to metadata is onerous. The diffs in the patch show that this is pretty straight forward and it wouldn't actually affect the runtime evaluations against real objects since we're doing an instanceof check against the value to evaluate. I'm sorry to hear that Rules doesn't make use of this class at all, but CTools (and core) make use of it pretty regularly, and it really needs to do proper constraint checking for that. I really think this is a big step in the right direction, and apparently won't adversely affect Rules. I'd love to get this class to a point that you all feel you can depend on it. Perhaps we could have a dedicated separate conversation about that? I'd like that very much.
Eclipse
Comment #26
eclipsegc commentedOh, also, on the topic of the entity:node:page level validation. I don't really see how this is possible. I can't attach multiple requirements to a plugin. The plugin requires 'entity:node' of bundles page or article, not entity:node:page || entity:node:article with some special handling. And with the approach in this patch, it's future proof against contrib constraints, instead of special casing the constraint we use most often with custom logic.
Eclipse
Comment #27
eclipsegc commentedOk, made up a new tag. :-)
Eclipse
Comment #28
dawehnerAt that point there hasn't been too much discussion, basically fago made a comment and disagreed with the idea.
In general it seems to be that we are extending the behaviour of the context handler, so I'm wondering whether a class ContextHandlerWithValidationConstraints would be a usable alternative.
This comment can be dropped now.
This doesn't make sense for me.
\Drupal\Core\TypedData\DataDefinitionInterface::getConstraintdocuments to return a array (the constraint configuration) and $constraint->type is a string:\Drupal\Core\Entity\Plugin\Validation\Constraint\EntityTypeConstraint::$typepoints to ctools still
+1 to not mock in a provider
I agree with tim, let's not convert everything. Why? It moves focus of reviews from fundamental reviews to codestyle only.
Comment #29
lokapujyavalidated=>validate (be validated)
Comment #30
eclipsegc commented28.1.) OK
28.2.) Yes the return value says that, but the EntityType constraint returns a string. I've stared at it in a debugger a lot, and just to confirm that I wasn't crazy went ahead and did so again just now. So either EntityType constraint plugin is wrong, or the return docs should say mixed.
28.3.) fixed
28.4.) Still learning these provider things
28.5.) Well I converted it because I rewrote it all. With the last two sets of changes to undo that though, sure. That being said, I completely rewrote providerTestFilterPluginDefinitionsByContexts() to be a more realistic test with a full list of plugins and a more nuanced set of contexts that will exercise exactly the problem that this issue is trying to address, so I did not return it to any former state since there is nothing the same about it.
29.1.) fixed
Eclipse
Comment #31
eclipsegc commentedFixed another unnecessary change.
Eclipse
Comment #32
fagoRight, still we should try to do not fork the way context is handled needlessly and use the same way of matching context as far as possible. Such that the context system works the same way for users/devs in both cases + we have the chance to unify code in future.
Right. That's what I'm saying that we do not support right now and I'm fine with, in particular if we can get to "the plugin requires 'entity:node' of bundles page or article".
If the approach taken does not work with a general symfony validation constraint, then it looks like mis-use of the validation API to me. It does not make much sense to use symfony validation constraints for contexts IF you have to modify them in some special way to make it work imo.
Afaics, my suggestions would achieve the same without the need for extending symfony validation constraints. Let's discuss the idea to see whether it can meet your requirements also.
Seeing that in the validator is quite a no-go imo, as we did quite some work to make coding validators so easy that you know that there is NO wrapped object going to be passed to you. We should not go back to a situation where sometimes wrapped objects are passed and sometimes not.
Comment #33
eclipsegc commentedCore's Context class actually does the typed data work for us when there's a value, so I'm handing off to that functionality instead. It seems worth looking at that class further to see if the data_type_wrapper instantiation could be added there instead. I'll look at this as I continue working to make this simpler.
Eclipse
Comment #34
eclipsegc commentedOops, unrelated crap in the diff.
Eclipse
Comment #39
eclipsegc commentedComment #40
jhedstromPatch no longer applies.
Comment #41
jofitzRe-rolled.
Comment #43
jofitzAddressed test failures.
Corrected coding standards errors.
Comment #45
jofitzUnable to replicate the test failures on my local build.
Comment #46
eclipsegc commentedOk, so this has been a long time coming on my part, but after a lot of soul searching and even getting another patch into core, I think I have a solution that should satisfy all involved.
The entity system recently got the ability to generate sample entities for given bundles. This new patch undoes all the changes to all constraint validators and removes the new Data Type object I previously added. Instead we generate sample entity or an EntityAdapter in the degenerate case of not knowing or needing a bundle. This way we can pass an object that will properly validate with the constraint validators. These are essentially run time entities which are never saved or used except to determine if the provided context satisfies available plugins. In addition to this, the ContextHandler identifies and generated TypedData objects as necessary for non-entity contexts as well, so things like strings or other data type primitives should all work as expected.
Eclipse
Comment #47
phenaproximaI ain't qualified to RTBC this, but honestly, it makes a lot of sense to me. I don't really see anything wrong this patch.
Micro-optimization: rather than call array_keys() here, we could do:
!array_key_exists('EntityType, $definition->getConstraints())Another micro-optimization: can we call $definition->getConstraints() once and reuse the return value?
Once again, let's call $context->getContextDefinition() once and use that value.
Comment #48
jofitzApplied @phenaproxima's micro-optimisations.
Comment #49
tim.plunkettPoints 1-3 all need test coverage, we had 100% before and I think its worth the extra effort here.
This
The `else` case of this, aka the final return statement
The `else` case of this
Nit: empty line between these, please
This could all be done for you if the class extended \Drupal\KernelTests\Core\Entity\EntityKernelTestBase
I think you'd just need to keep `node` in the $modules line
Nit: assertFalse
Okay not gonna highlight all these
Wait why can't this go back to being a dataProvider? Those work for KTB too
The @covers need to be updated to include the new protected methods
I notice we call getConstraintsFromContextDefinition() twice in the same code path. Why not pass the result of the first call into getTypedDataFromContext() to avoid the need for the second one?
The entity-specfic-ness of this code worries me.
Why is ContextHandler left holding the bag on so much of this?
Why are getConstraintsFromContextDefinition() and getTypedDataFromContext() part of this service?
Is there a need for other handling similar to this, for non-entity stuff?
Comment #50
tim.plunkettTried to rewrite the IS
Comment #51
tim.plunkettI think ideally, all of this would be a new service, and the changes to ContextHandler would consist entirely of this:
Comment #52
eclipsegc commentedTim and I talked a bunch and I extracted the entity specific bits into another service for the time being. I've not yet added test coverage for that service. What do we think of the basic structure of this now?
Patch got a bit simpler since I'm not rewriting the tests any more, so the patch is actually smaller than the interdiff would be.
Eclipse
Comment #54
tim.plunkettI really like moving this responsibility out of ContextHandler.
I've gone a step further, and moved it to ContextDefinition itself.
Opened #2671964: ContextHandler cannot validate constraints for moving the entity logic out of this class.
Restored a lot of the original tests, but added a new one for the new parts.
Once again, the interdiff is bigger than the patch, so it is not included.
Comment #55
tim.plunkettDiscussed the failing test with @EclipseGc and came up with this solution.
Comment #58
eclipsegc commentedI was thinking about this bit and... we could yield here so that we don't have to generate all bundles we just have to generate until we find a bundle that works. Thoughts?
Eclipse
Comment #59
tim.plunkettHaven't figured out the block fail yet, but here's the yield.
Comment #61
eclipsegc commentedOk, so we need to encapsulate the yields so that we don't invoke every member of the array return needlessly. I wrapped it in an anonymous function in order to achieve this but given some of the black magic happening here I wanted to see if this reduces us back to the 5 failures of earlier before working further.
Eclipse
Comment #63
eclipsegc commentedI've not looked at all the failures, but the BlockTest failure is related to the UserRole condition, which given the code we're touching here likely means it's context related. I'll see what I can do to investigate further.
Eclipse
Comment #64
eclipsegc commentedOooook... So I think this is a user password validation related error. So I cheated, and just told the "satisfied by" code to return true for all user requirements vs user contexts. If this passes all tests, then I'll look at the password validation process on user entities and see if there's something really wrong there.
Eclipse
Comment #65
eclipsegc commentedOk, looking at the password validation stuff, I see that there's a switch for just not validating password. I've added that to the current user context provider. Let's see if this has a negative or positive effect on tests.
Eclipse
Comment #66
tim.plunkettSomewhere I missed adding the test coverage to prove that the
yieldis actually a benefit.While doing that I was able to confirm that the change in #61 is not needed. Undoing that.
Additionally, I found the root issue causing the password fails.
\Drupal\Core\TypedData\Plugin\DataType\Map::getValue()loops over each property, but only currently works for those properties that happened to be instantiated already.Switching to iterate over the full list of properties causes the comparison in
\Drupal\user\Plugin\Validation\Constraint\ProtectedUserFieldConstraintValidator::validate()to correctly pass.Comment #68
tim.plunkettThat exposed a lot of other bugs, so opened #2934192: FieldItemBase::getValue() returns partial or full values depending on state for that. Restoring the workaround with an @todo pointing there.
Comment #69
eclipsegc commentedI think we need an rtbc at this point.
Eclipse
Comment #70
samuel.mortensonLooks good to me 👍
Comment #71
larowlanThis is a BC break.
I discussed this with @catch and we were of the opinion that in order to maintain BC we might need to add a new interface for this method, and have ContextDefinition implement it.
What would the implications of that be?
What are the implications on contrib if we don't?
We have the 'contributed blocker' tag but there isn't any detail on what it blocks.
Thanks
Comment #72
tim.plunkettThere is a 1:1 relationship here.
Comment #73
tim.plunkettI spent a good amount of time on the IS, it explains why this issue is essential.
Without this fix, context definitions are pretty useless. Which is why no one has implemented the interface besides this single class.
It blocks any contrib module from being able to use the entire subsystem.
Comment #74
larowlanwhat if there is more than one value?
if the first one is valid, we return true without checking the later ones.
that is this issue?
this feels back to front.
we're embedding specific knowledge of an 'EntityType' constraint in ContextDefinition.
Feels like there is a missing concept between the constraint and the definition.
Shouldn't the context definition constructor add this for us, then we don't need to embed this logic
Or does this indicate there is a missing EntityContextDefinition that sub-classes ContextDefinition and auto-adds this constraint definition.
I note that
\Drupal\layout_builder\Plugin\Derivative\FieldBlockDeriver::getDerivativeDefinitionsadds the Bundle constraint - but not the EntityType? Should it?If someone already added an alternate EntityType constraint - would this clobber it?
tl;dr if we expect a context with naming
entity:{type}will have anEntityTypeconstraint, shouldn't it happen before here?Comment #75
larowlanThere was no discussion here of why #51 was abandoned
Comment #76
tim.plunkett#74
1) One is enough
2) That should have linked to #2932462: Add EntityContextDefinition for the 80% use case
3) That's what that ^ was about
4) Also covered by the above.
#75
If you'll note, the diff of ContextHandler is almost identical to #51. But instead of delegating to a service, it's delegating directly to the ContextDefinition object in question.
Not playing status wars anymore.
Comment #77
larowlanRight, but keeping it somewhere else outside the context definition would remove the API change and that whole debate becomes moot.
However, in light of the right @todo link, the current approach here (the API addition) gives a clear path forward for #2932462: Add EntityContextDefinition for the 80% use case with a clean API design so that's reason not to make it a separate service.
I think that as per #72, there is a 1:1 relationship.
Given the full scope of the interface:

I just don't see anyone implementing it without sub-classing the ContextDefinition class.
Comment #78
larowlanPutting back to RTBC to get opinions from other committers.
We still need the @todo links fixed, but that can remain RTBC
Comment #79
larowlanComment #80
larowlanComment #81
larowlanComment #82
tim.plunkettFixing NID
Comment #83
catchI'd forgotten about the explicit 1-1 rule when discussing this last night. Given that, I think this should be OK to go into 8.5.x too.
Comment #84
tim.plunkettReflected that in the IS
Comment #85
larowlanI know this is an underused API, but can we get a change record for the new functionality, as its a vast improvement over the current logic.
Thanks
Comment #86
eclipsegc commentedAdded a change notice here: https://www.drupal.org/node/2934902
Eclipse
Comment #87
berdirRelated question, I was wondering about hierarchy in regards to context data types before.. lets say I have something requires a type "entity" and I have a node, aka "entity:node". Clearly that's not supported yet, is there any issue/discussion on supporting that?
This isn't making it any worse than it is now, so obviously not a blocker.
similar here, if we look explicitly for entity:, what if I have a context that accepts any entity...
not sure if/how this is relevant for the test, but it's worth pointing out that getBundleInfo() always returns a bundle, if there is none, then there is a default/fallback bundle where bundle === entity_type_id, see \Drupal\Core\Entity\EntityTypeBundleInfo::getAllBundleInfo(). Might result in a more realistic test to simulate that behavior?
Comment #88
tim.plunkettComment #89
larowlanFor #87.3
Let's fix that
Comment #90
tim.plunkettDoesn't affect the test, but might as well do it.
Comment #91
eclipsegc commentedNo behavioral change, and the test looks more complete. Re-RTBCing.
Eclipse
Comment #92
larowlanCrediting @fago, @dawehner, @phenaproxima and @Berdir as their reviews shaped the patch
Comment #94
larowlanDiscussed the earlier concerns of @fago and @dawehner with @EclipseGc.
For posterity, the patch changed tack in #46 to address those concerns and the final solution doesn't special case and validation constraints or validators.
Instead, it relies on the ::generateSampleValues functionality since added to core, so that validators don't have to do anything different.
This is a better solution as we don't embed any knowledge of the context api in the validation api.
Committed as 0c1de8e and pushed to 8.5.x.
Published change record, unpostponed issues blocked on this.
Comment #95
dsnopekWow! This has huge implications (the good kind :-)) for Panelizer in Drupal 8 :-)
Comment #96
jonathan1055 commentedI think the commit in #93 above has caused new test failuers in Rules #2936553: Rules kernel tests fail with "Creating default object from empty value", judging by the timing of the commit and the backtrace. Not saying that the fault does not originate within Rules code, but this core change definitely is involved in it.
Comment #97
eclipsegc commentedThe fault is almost certainly with Rules. This system was not capable of filtering plugins by context appropriate until now. I'm sure Rules was attempting to compensating for it.
Comment #98
tim.plunkettIn fact, from the description in the issue it doesn't even sound like a bug in Rules, but a failed assumption in the Rules test setup
Comment #99
jonathan1055 commentedThanks EclipseGc and tim.plunkett.
I was asking on this issue because the new line
$current_user->_skipProtectedUserFieldConstraint = TRUE;definitely causes the "Creating default object from empty value" error. Is there any way that:should ever validly return NULL i.e. not a user object? Adding the temporary line
fixes the problem and the Rules tests run OK. But obviouly that's not the solution. I don't know enough about the contexts to work out where the real source of the problem is, so just thought I would ask here, as this is where that new line came from.
Jonathan
Comment #100
tim.plunkettI guess this could be broken for anonymous users.
Comment #101
larowlanCan we get a major follow up filed to add add the diff at #100?
Comment #102
tim.plunkettDone #2936642: Getting runtime contexts will generate an E_WARNING for anonymous users
Comment #103
larowlanthanks
Comment #105
agentrickardApparently, this change breaks certain uses of COnfig Entities, since they do not require the getIterator() method.
See:
#2941319: Error: Call to undefined method Drupal\domain\Entity\Domain::getIterator()
#2759267: Undefined method YamlForm::getIterator()
I have a temporary fix here: #2941457: EntityAdapter::getIterator requires undocumented methods
Without the fix, Config Entities will throw fatal errors in unpredictable areas, such as Context negotiation. The only other fix is to break BC and force Config Entities to add the method.
Comment #106
tim.plunkettThe bug in EntityAdapter was introduced in 2014 via #2002138: Use an adapter for supporting typed data on ContentEntities, but only exposed to wider runtime code by this issue.
I think that the "temporary fix" issue is actually a long-term fix, even after #1818574: Support config entities in typed data EntityAdapter happens.
It is now RTBC.
Comment #107
cilefen commentedDid this issue cause the linked regression?
Comment #108
abrammAnother regression: #2958785: ContextDefinition isSatisfiedBy() check fails for context using inherited class.
Comment #109
timodwhit commentedThere was a major performance regression with this change that is visible in layout builder. Issue created:#2981889: Performance Degradation in Layout Builder and other places likely