Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Problem/Motivation
I opened #3134245: Run drush imports as user=1 to solve part of the problem. But that solution is very much a sledgehammer solution when perhaps a more valid solution is to just switch the user if entity validation is enabled. This does the latter.
Proposed resolution
Switch to the entity owner if it implements EntityOwnerInterface
Test the latest patch by enabling validation on the destination entity i.e. validate: true
.
source:
plugin: d7_node
node_type: page
process:
title: title
type: type
body: body
destination:
plugin: 'entity:node'
validate: true
Remaining tasks
User interface changes
API changes
Data model changes
Release notes snippet
Comment | File | Size | Author |
---|---|---|---|
#85 | diff-78-85.txt | 1.68 KB | mikelutz |
#85 | 3134470-85.drupal.patch | 23.63 KB | mikelutz |
Comments
Comment #2
heddnComment #3
heddnComment #4
Kristen PolAssigning to myself for review.
Comment #5
Kristen PolFrom May 7th:
Thanks for the patch.
1) Reviewed the code and it looks straight forward. I compared it to the
AccountSwitcherInterface
usage incore/lib/Drupal/Core/Cron.php
and see they are doing dependency injection in the constructor but this is adding it via create. I'd need to look at more examples which I can't do at the moment (getting pulled away). I imagine the code is perfectly fine though. :)2) Patch applies cleanly:
3) What is the best way to manually test this?
==
UPDATED (May 8th): I searched the codebase for something like:
and didn't see anything. I like to review existing code that's similar when reviewing to help me validate the approach.
Comment #6
Kristen PolComment #7
Kristen PolOk, I was being too hasty yesterday and now see that this code is using some of the same code from the referenced migrate_tools module patch in #3134470: Switch to entity owner in EntityContentBase during validation which was based on the default_content module.
Looking at this code again and at other migrate code, unfortunately, it's not clear to me why
accountSwitcher
isn't part of the constructor and thegetAccountSwitcher
function is used.Comment #8
heddnSo, the reason I didn't do constructor injection is that while constructors are not technically part of BC, there are several cases where contrib and custom implementations extend the constructor and add their own additional arguments. If we did that here, we'd break lots of people. The solution I used still uses injection, but is not going to break many thousand of sites.
Comment #9
Kristen PolThanks for the explanation. Now I have dumb question...
Can't it not be added to the parameters in the constructor but still initialized in the constructor (presumably if people are extending the class they are calling the parent as this is a best practice, no?) and then the function wouldn't be required as it would always be set? Or we can't assume they would call the parent constructor?
Comment #10
heddnManually test... build out a source that sets the format on a text field. Make sure the format is one that requires elevated access. Additionally, make sure
validation: true
on the destination. Then run the migration via drush. Drush always runs as anonymous. Without the patch, validation will fail. With the patch, validation will succeed if the user associated with the entity has access to add the format.Comment #11
heddnHere we add some tests. Interdiff is the test only patch.
Comment #12
heddnSelf review, to pick up on next re-roll:
Comment #13
heddnre #9: can we add the injection in constructor (not create) function?
We could move it over, but I prefer not to call
\Drupal::service()
unless I need to. The constructor doesn't have access to the container. Whilecreate
does have access. And adding the getter lets me easily stub out the service in tests if I ever need to do that.Comment #14
Kristen PolThanks for the test.
1) Patch applied cleanly:
2) Reviewed the code and could follow most of it though it would be nice to have some comments :) though there aren't many in the other tests.
Unclear why this was deleted though I assume it wasn't needed.
Nitpick: extra new line.
IMO this comment isn't clear. Not sure what's better but maybe something along the lines of the following, unless I'm not understanding the test:
Tests validation where user without filter format permission saves content.
3) Still have question from #9.
4) Moving back to "Needs work" in case any of 2) should be addressed.
Comment #15
Kristen PolWe crossposted so #14.3 was answered in #13. Thanks!
Comment #16
heddnAdds comments to the test to respond to most of the feedback.
#9: see #13
#14.1: this is not removed but became
$this->installConfig(['field', 'filter_test', 'system', 'user']);
#14.2: fixed
#14.3: fixed
Comment #17
Kristen PolThanks for the updates.
1) Reviewed the interdiff and the changes cover the items noted in #14.
The comments help a lot! :)
Ah, see that
installConfig
was moved afterinstallSchema
. Thanks for pointing that out.2) Patch applies to 8.9 (with offsets), 9.0, and 9.1.
3) Reviewed the patch again. Only a couple nitpicks I missed before caught my eye. Not sure they need to be changed but I'll move back to needs work in case they should be.
Sorry, should have caught this before.
Nitpick: over 80 characters.
Same as above.
4) Not sure this will need manual testing before this can considered for RTBC.
Comment #18
heddn80 char limit is for comments, not code. And it would be caught by PHPCS if it were a concern for the testbot. Switching back to NR for any other feedback.
Comment #19
Kristen PolSorry, lack of sleep is becoming apparent here! Ignore me ;)
@heddn In your opinion, does this need manual testing prior to RTBC? Thanks.
Comment #20
heddnre manual testing: I don't think so.
Comment #21
Kristen PolConsidering all the above, marking RTBC. Thanks for your patience!
Comment #22
quietone CreditAttribution: quietone as a volunteer commentedI'd really like to see a failing patch and I decided to make it myself. I started from the patch in #16 and made a failing patch and re-uploaded 16.
And then I found I have some questions.
This test is done twice in the same method. Can't we do it just once? Should we check if the entity owner is the same as the current user before switching and switching back?
Should be 'Gets'.
Does this mean user and user roles?
Missing trailing ','
I must be reading this wrong. The user is set to $normal_user but the content type that is owned by $normal user fails to migrate?
Comment #24
Kristen PolNice! Glad this is getting more eyes on it :)
Comment #25
quietone CreditAttribution: quietone as a volunteer commentedSetting to NW for #22
Comment #26
heddnre #22.5:
The fail test will fail on entity 1, because it isn't switched to user 1 (admin user) and it attempts to run validation as an anonymous user. This user doesn't have the needed access.
But then the test goes further to assert that entity 2 will fail, because a non privileged user (user 2) doesn't have access to use "filter_test" filter. Both test cases are needed.
Fixes other feedback from #22, points 1-4. I'm not sure I like the combined check any more then what was there before, but it does only execute things once.
$switch_user = (($entity instanceof EntityOwnerInterface) && $user = $entity->getOwner());
Comment #27
Kristen PolThanks for the update.
1) Reviewing the interdiff in #26 and the items in #22, it appears that they are all addressed. I noticed one missing Oxford comma but I saw this so I don't want to hold this up further:
https://www.drupal.org/drupalorg/style-guide/words-phrases
If this patch ends up being updated again, add Oxford comma after "roles".
Reading again, although I understand what "non-permissioned" means here, it turns out it's not commonly used. Only 4,800 results in Google for that. It's not used in core anywhere. Probably want to reword that. Perhaps just remove it since IMO the expanded text covers it.
The second user import should fail validation because they do not have access to use "filter_test" filter.
2) Patch applies cleanly to 8.9, 9.0, and 9.1.
Comment #28
Kristen PolMoving to "Needs work" #27 as I think the rewording in 27.1.2 would make it easier to read.
Comment #29
longwaveInstead of having two variables here, $switch_user as a boolean flag and then $user as the object, why not just use $user as both the flag and the object?
Also I think we tend to use $account for generic account objects and $user as the actual "current user" though not sure this is set in stone, not sure if that would make it clearer here or not.
Comment #30
ridhimaabrol24 CreditAttribution: ridhimaabrol24 at Srijan | A Material+ Company for Drupal India Association commentedComment #31
ridhimaabrol24 CreditAttribution: ridhimaabrol24 at Srijan | A Material+ Company for Drupal India Association commentedComment #32
heddnNit: this is now 81 chars. How about (53 chars): "Filter format access is impacted by user permissions."
Comment #33
ridhimaabrol24 CreditAttribution: ridhimaabrol24 at Srijan | A Material+ Company for Drupal India Association commentedAltered the comment as suggested in #32.
Comment #34
Kristen PolThanks for the update.
1) Reviewed the interdiff in #33 and see it addresses item from #32.
2) Patch applies cleanly to 9.1.
3) Tests pass.
4) Marking RTBC based on this review and the comments above.
Comment #35
alexpott@heddn if they override the constructor then they'll be calling parent::__construct() - if we add AccountSwitcher to our __construct as $account_swticher = NULL and then trigger a deprecation when called without it we're fine and we'll break no one.
This pattern of doing pseudo setter inject in ::create() methods is okay for custom or contrib where you want to insulate yourself from potentially breaking upstream changes but I don't think we should use it all the time in core.
Comment #36
heddnI'm not sure if we should re-order the arguments on EntityComment, EntityRevision and EntityUser. But this picks up the rest of the requested changes.
Comment #37
Kristen Pol@heddn There's only an interdiff. Did you want to post the patch too or are you looking for feedback on the changes first?
Comment #38
heddnOops.
Comment #40
heddnComment #41
alexpottYeah we shouldn't re-order the arguments. In an ideal world the comment and user classes would be final and their constructor marked as @internal so we can change the order. But that's not the world we're in.
This is no longer needed.
Do we have to be concerned about migration dependencies. Like is the user guaranteed to exist at this point?
I was pondering about this from a migrate dependencies perspective and then realised that actually injecting this service is pointless because the user entity is not a EntityOwnerInterface entity.
Kinda amusing to have to make this change for something that for this object will never be used.
Comment #42
heddnRe #41.2: yes we do have to concern ourselves with this. Ergo the check for account in the next line.
Comment #43
alexpott@heddn re #41.2 probably worth adding a comment in the code about that then.
Comment #44
heddnComment #46
heddnComment #47
quietone CreditAttribution: quietone as a volunteer commentedA brief look at the recent changes, since the RTBC.
If I am reading the policy correctly this needs to have when it will be removed and a CR.
%thing% is deprecated in %deprecation-version% (free text describing what will happen) %removal-version%. %extra-info%(optional). See %cr-link%
Doesn't this need a trigger_error as well?
Comment #48
heddnCR added and linked and deprecation test added.
#47.2: I don't think it is needed. In fact, I removed it from a few other places just now. The parent class will throw the deprecation.
Comment #49
alexpottI'm wondering if this behaviour should be optional. For example, imagine permissions are being changed as part of the migration. Also maybe for performance reasons or other reasons someone if running the migration as user 1.
Comment #50
heddnThey can always not activate validation. But if you want entity validation, then you need to use the right user ID.
Comment #51
heddnBTW, the only way to run (via drush) as user 1 is via the account switcher. For a very long time, these migrations have been run as anonymous, not even the admin user. This would be different via migrate_drupal_ui, but then you really aren't so worried about performance or anything at that point, because entity validation (currently) isn't even enabled for that UI.
Comment #52
quietone CreditAttribution: quietone as a volunteer commentedI read the CR and it looks fine to me and does the patch. I don't see anything still to do.
However, I have one question. I applied that patch and ran phpcs on migrate/destination/EntityContentBase.php and get the following error and there are not coding standard errors reported by drupalci. Does the message need to change?
Comment #53
alexpottIs this the case? We don't switch before saving a node that was created by someone else.
Comment #54
heddnI'm not exactly sure what to suggest then. In the case where someone is saving content that is owned by another user, we do run validation. And if the user doesn't have access to full html in the body field, then the save will fail. We could always run it as root, but that seems like cheating and it would hide valid validation errors.
Comment #55
alexpott@heddn yeah I think this is a tricky thing. It's going to depend so much on how your new site is configured as to what is the correct thing to do. I'm not sure it is possible for there to be a right way.
Comment #56
heddnre #53: usually user migrations are supposed to happen before all other migrations. For just that reason. If someone has entity validation configured for a migration, then they want to validate the entity in the most valid way. Which seems to me as the entity owner.
From #3134245-7: Run drush imports as user=1, some other use cases that might help guide us:
The later issue w/ path alias makes me wonder if using user = 1 is the better solution. Instead of the entity owner.
Comment #57
heddnComment #58
Wim LeersCan we point to an example validation constraint that requires this? (using
@see
)🤔 This implies that every migration using a
EntityContentBase
subclass as a destination for which the entity type implementsEntityOwnerInterface
MUST have a required migration dependency on thed7_user
migration.Is it at all possible to add an
assert()
that adds the logic that checks that?Preferably in
\Drupal\migrate\Plugin\migrate\destination\EntityContentBase::__construct()
, where aMigrationInterface
object is readily available, and also to avoid us asserting that for every entity we validate, which is pointless.🤓 s/filter format/text format/
🤓 The second sentence here doesn't really make sense.
👎 Not specifying a failure message here (that second parameter to
assertSame()
) yields far more helpful failing test output.Could you please remove that? 🙏😊
Comment #59
Kristen PolBack to needs work per #58.
Comment #61
heddnAll feedback from #58 addressed.
For #58.2: I don't think we can make this assumption. We cannot assume all migrations are for drupal upgrades. Folks can also migrate from any source they want for any motive they want.
Comment #63
heddnComment #64
heddnComment #65
Wim Leers#61: RE #58.2: Right, that makes sense! 👍
My key remaining concern is then that we ensure that the test coverage actually tests what we need it to test. @quietone already asked this question and proved it in #22 👍
I would RTBC this … but unfortunately at least the first of the following remarks must be addressed before this can get committed. The remainder are nitpicks that I am fine with not being addressed, since they're only in tests.
This now needs to be updated to
9.2.0
.Missed this spot in #58.
Related nit: s/who has filter permission/who has permission to use this text format/
Comment #66
quietone CreditAttribution: quietone as a volunteer commentedWhat about the points raised in #56 about files and path aliases? In that comment heddn suggests that path alias should be owned by uid 1. Will enabling validation on path alias migration break things for people?
I re-read the issue and reviewed the patch, finding a few things, showing that I have been dabbling in coding standards issues. The comments in test, MigrateEntityContentValidationTest::testEntityOwnerValidation have improved since my last look at this and it is much easier to understand. And along with the suggestion by Wim Leers it will be even better.
Null is allowed here. So methinks this should be
@param \Drupal\Core\Session\AccountSwitcherInterface|null $account_switcher
Same here
Next I read the CR. I do like that it is concise but in this case perhaps too concise. I think that the phrase , "When validation occurs during a migration" needs explanation. I suggesting adding an example, as in the IS here that shows how one enables validation. And even a link to the CR that added validation would be nice, https://www.drupal.org/node/3073707.
Comment #67
heddnMostly updates to comments that I'd consider novice issues.
Comment #68
quietone CreditAttribution: quietone as a volunteer commentedI remembered there was a tag for updating the change record. See #66
Comment #70
sudiptadas19 CreditAttribution: sudiptadas19 at QED42 for Drupal India Association commentedAll the feedback from #65 and #66 addressed.
Comment #71
sudiptadas19 CreditAttribution: sudiptadas19 at QED42 for Drupal India Association commentedFixed PHPCBF issue.
Comment #72
sudiptadas19 CreditAttribution: sudiptadas19 at QED42 for Drupal India Association commentedFixed PHPCBF whitespace issues.
Comment #73
gabesulliceNice! This test case demonstrates exactly the same issue that I encountered myself and this is exactly the solution I was going to propose before I found this issue :D
Nit: it won't be required before Drupal 10. The phrasing below matches other, similar deprecation examples.
The overwhelming majority of them do not have a period after the
so I removed that dot as well.IMHO, this can be RTBC'd because everything else looks good here. I made the change above myself so that this doesn't die by a death of a 1000 nits.
Comment #74
gabesulliceI've updated the CR
Comment #75
gabesulliceIMO, this is a bug since it is almost impossible to validate imported entities which have a text format that the anonymous user isn't allowed to use (which is often the case)
Comment #77
larowlanOne question and a couple of nits
should we wrap this in a try catch and then do the switch back in a finally?
just in case something in validation throws an exception?
nit; should we use
assert($filter_test_format instanceof FilterFormatInterface)
instead?nit: similarly here?
Comment #78
gabesulliceThanks @larowlan!
This is wise. I added a
finally
block but left off thecatch
block for the reasons in the patch comment. This means that the account will be switched back whether there is an exception or not and the exception will bubble up just as it does today.Every row import() is wrapped in a try/catch in
MigrateExecutable
, which captures the exception and stores it in the migration messages table. If I caught it, I would have had to either silence the exception or duplicate that logic.Comment #79
heddnAll feedback addressed. Back to RTBC.
Comment #80
catchThis looks like probably the only fix available within migrate, but feels like it's an issue with filter format validation in general.
FilterFormat::access() accepts an $account parameter, however we can't pass that to ::validate().
This wouldn't have come up when we were doing validation in form API, it's because we're also trying to handle REST requests, but migrate is neither and feels like this kind of use case has been missed.
Comment #81
catchThere's another problem here as well. There's no guarantee that the entity owner has access to the text format at the point the content is migrated, for various reasons:
1. The target site has different permissions for the text format.
2. The user is no longer in the role that has permissions for the text format.
3. The content was edited by an admin user, and the text format was changed to one that the entity owner doesn't have access too.
... and etc.
So I think this means we'd need a way for validation to bypass the access check in text format validation altogether. Moving back to needs review for more discussion.
Comment #82
gabesulliceMigrate feels like an exceptional case. Validating entities instead of specific forms means Drupal can safely handle updates by users via HTTP requests (which includes form POSTs), that seems like right use case to optimize for. Everything else I can think of falls into developer-only scenarios. E.g. drush hooks, cron jobs, migrations, etc.
This is a best-effort resolution which makes a reasonable assumption IMO. Entity validation during migrations is an opt-in feature too, which makes it even more acceptable. Just disable it if its an enormous problem.
1 & 2: most Drupal-to-drupal migrate stuff in core assumes you're upgrading from D7 to D8+ without making any changes so this is not out of the ordinary. Whether that was a good idea is a much, much bigger fish to fry.
3: This is likely a one-off situation. The DX will be:
Comment #83
heddnThe validation errors surfacing from 82.1-3 are exactly the types of errors I would want to surface. It gives me the chance to fix the data in D6/D7 before I migrate. Or fix the destination system if their permission model is incorrectly setup. Without the validation, the data is imported silently and it isn't until a few days/weeks later that an editor goes into an old page and finds they can't edit the data.
Also, while typically the issue is for filter format, it could also surface w/ field permissions as well. What if there is a field that only editable by the node owner? But not by other users?
And yes, turning on entity validation is optional. And turned off by default. Because of just these types of reasons we've been discussing.
Comment #84
m.stentaSetting this back to RTBC. I believe @catch's concerns were addressed by @gabesullice's and @heddn's comments.
Comment #85
mikelutzReroll
Comment #86
marvil07 CreditAttribution: marvil07 as a volunteer and at Adapt commentedSmall +1 on the re-roll, I arrived to exactly the same patch, and it looks good, even if I have not examined it in details, so it testbot confirms it is likely ready.
Comment #88
catch#82 and #83 makes sense - either don't validate, or validate with the correct user.
Committed 8065ee3 and pushed to 9.3.x. Thanks!
Comment #90
quietone CreditAttribution: quietone at PreviousNext commentedPublish CR