The XML::fetchNextRow() method currently returns a SimpleXMLElement when the current selector doesn't render to a string and has children. This seems to be in conflict with core's MigrateExecutable::import() which starts by calling $source->rewind(). This will result in an exception being thrown ("Migration failed with source plugin exception: Serialization of SimpleXMLElement is not allowed"), and the migration is interrupted.
Quickly googling for this problem indicates that SimpleXMLElement can't really be serialized, so returning that as an object is simply not possible. I'm proposing that instead of returning the object, the code should be altered to return the element's XML, which can easily be converted back to the object with simplexml_load_string().
Since might be relevant, in this migration I've got a set of XML files in an import directory (crawled by the hook_install hack), and one of my processors needs to use the XML element, as the specific child being imported depends on a Drupal variable (the XML element contains different translations of a string, and which one gets imported depends on the current site's language).
| Comment | File | Size | Author |
|---|---|---|---|
| #14 | 2837684-14-migrate-plus-xml-return-as-xml.patch | 1.5 KB | michelle |
| #7 | interdiff-2837684-0-7.txt | 2.08 KB | holist |
| #7 | 2837684-8-migrate-plus-xml-return-as-xml.patch | 1.5 KB | holist |
| migrate_plus_xml_return_asxml.patch | 1.06 KB | jcnventura |
Issue fork migrate_plus-2837684
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
jcnventuraComment #3
itsekhmistro commentedReviewed. The exception is thrown by Row->rehash() method.
So migrations are affected only if the param `track_changes` is enabled in migration configs.
The patch works for me.
Note, In result of this patch Process plugin might retrieve string value instead of SimpleXMLElement object previously.
The $value can be easily converted back to SimpleXMLElement if needed.
Comment #4
mikeryanWe are explicitly allowing the elements through per #2792765: Allow XML elements to pass through as field values to process plugins - this change would break BC (it would break anyone currently using that functionality).
Is this only a problem (as itsekhmistro suggests) when rehashing? If so, is there a way for us to do the XMLization only in that case? Perhaps with by overriding the Row used in our source plugin with an extension that overrides rehash() to unwind any SimpleXMLElements within?
Comment #5
jcnventuraIt probably is related to track_changes, as the migrations where the exception is thrown also have this flag enabled. If I have time, I'll try to re-implement it taking into account your suggestions.
Comment #6
agileadamI had to use this patch to get track_changes to work (worked great!) and did experience an issue with some custom code. I have updated my code to work nicely with or without the patch, but it would be nice to work out a way to do this without breaking BC. I tried to find some indication that a (re)hashing is taking place but didn't see anything at first glance.
Comment #7
holist commentedI added the same fix to
SimpleXml::fetchNewRow()as well.I agree that the proper solution would be to just provide something serializable for
$row->rehash(). I was already thinking of how overridingSourcePluginBase::prepareRow()could work to this end, but it was a bit too complicated for my time frame here to make up a solution becauseRow::rehash()manipulates the source data directly.Comment #8
tuwebo commentedHello,
Patch in #2837684-7: Migration failed with source plugin exception: Serialization of SimpleXMLElement is not allowed is working fine for me, thanks.
Any updates here?
Comment #9
heddnGiven the concern and complications here, should we add a test?
Comment #10
pookmish commented#7 worked for me as well. I agree a test would be ideal
Comment #11
pookmish commentedTip: I found if you are planning to do a `sub_process` process, before the process configuration add a callback plugin as such:
Comment #12
duaelfrGood catch here!
For those who have issues with
track_changes, have a look at #3050924: Migration: Serialization of 'SimpleXMLElement' is not allowedI don't think #7 is a good approach as it forces workarounds like #11.
Let say I have a source like this:
I my migration I do the following:
I expect it to work but it does not and I don't expect to have to use a callback to reload something that has been serialized somehow.
Comment #13
das-peter commentedI've just posted a patch in #3076534-8: Xml parser doesn't create array of values if value doesn't render to string that casts complex values into multi-level associative arrays. As far as I understand this is what other plugins expect in "multivalue" cases.
Comment #14
michelleI did a re-roll from #7. This just makes that patch apply on 6.0.x and doesn't do anything about the following discussion.
Comment #15
jumpsuitgreen commented@Michelle Thanks for posting. Your patch (#14) resolved our issue.
We have a D10 site with migrations using the URL source plugin to fetch XML file elements. Empty elements would trigger this error:
Migration failed with source plugin exception: Serialization of 'SimpleXMLElement' is not allowed in /var/www/web/core/modules/migrate/src/Row.php line 391After applying patch #14, the migration runs smooth.Comment #16
joachim commented@das-peter did you mean the patch from #3076534: Xml parser doesn't create array of values if value doesn't render to string should fix the issue here?
I tried the patch from #3076534: Xml parser doesn't create array of values if value doesn't render to string for my issue and it didn't fix it.
Comment #17
joachim commentedI also tried the patch from #14, and the values I get in my source row don't look right any more:
Comment #18
joachim commentedI'm trying to understand the exact circumstances that produce this error, as some of my XML migrations are working fine.
Is it caused by the value-holding XML item having an attribute, for instance?
e.g.:
This works ok, because the xpath selector is 'property[@field="field-alpha"]' and the resulting item has no attributes, and so renders to a string:
But this doens't work:
Comment #19
joachim commentedComment #20
joachim commentedI can see a partial fix: replace:
with:
MR coming.
It's a fix in that the use case I gave in #18 now works.
But it's not really a complete fix because this arm of the condition will still produce something that will crash Migrate:
This code was added in #2860785: Allow XML elements to pass through as field values to process plugins (for SimpleXML processor), but it's not clear how back when it was added it wasn't causing this crash!
Comment #21
joachim commentedI'm not sure about where to put a test, as the tests in tests/src/Kernel/Plugin/migrate_plus/data_parser are unit-ish tests -- they don't test the data parser plugins in the context of a migration. Where should tests that cover integration with the migration system go?
Comment #23
heddnIs there any real world testing for MR 109? What's in there looks reasonable.
Comment #24
heddnThe problem seems to also exist in src/Plugin/migrate_plus/data_parser/Xml.php. Can we add test coverage for this and post fixes for both places?
Example xml fixture to use:
Comment #26
jerech commentedI've updated the MR. Instead of changing children() to hasChildren() (which broke BC for elements with attributes), I implemented a SerializableSimpleXml wrapper. This correctly fixes the Row->rehash() serialization exception while fully preserving BC for downstream plugins. I also added the test coverage with the XML fixture you provided. Ready for review!
Comment #27
heddnComment #28
jerech commentedComment #29
heddnThere are a lot of stray commits/changes included here. Can we rebase things?
- Modeler API commit (1f22c59)
- efc64aa "Fix Snippet process plugin throwing warning on invalid YAML instead of exception"
Comment #30
jerech commentedComment #31
heddnI've spent some time on this because it keeps stalling, and I think I know why. There are three separate bugs tangled in this issue, and every patch so far fixes one and regresses another.
A. Serialization. Row::rehash() cannot serialize a SimpleXMLElement. Only bites with
track_changes: true. This is the issue title.
B. Attributes. children() is truthy even when it holds zero children, so
is treated as a complex element. joachim's #18 and #20.
C. Shape. sub_process needs arrays; neither objects nor strings satisfy it. das-peter's
#12, and really #3076534.
I'm going to post some tests. Then I suggest we might deprecate these troublesome plugins and add a replacement.
Comment #33
heddnI've pushed tests. What do we think about deprecating and replacing? What would that look like?
Comment #34
jerech commentedI completely agree with your analysis in #31. Trying to fix the existing xml and simple_xml data parsers in place has proven to be a game of whack-a-mole with BC breaks. Deprecating the current plugins and introducing a new, robust replacement is definitely the smartest path forward.
Regarding what the replacement would look like, I propose we create a new data parser plugin (e.g., xml_array or xml_parsed). Unlike the current parsers that yield SimpleXMLElement objects, this new parser should inherently cast the parsed XML elements into standard PHP arrays (for complex elements/children) and scalar strings (for values/attributes) before yielding the row.
This approach solves all three issues at once:
Serialization (A): Arrays and strings are natively serializable, so Row::rehash() with track_changes: true will work flawlessly.
Attributes (B): We can build the new parser's logic to explicitly separate attributes from child elements into an intuitive array structure (e.g., pushing attributes into an @attributes key, similar to how JSON conversion handles it).
Shape (C): By yielding standard associative arrays, it will play perfectly with the sub_process plugin right out of the box without requiring custom callbacks.
If you are on board with this concept, I can start drafting this new data parser plugin against your new MR (!162) so we can get those failing tests to pass using the new architecture. Let me know your thoughts!
Comment #35
heddnThe concept I agree with. Naming is hard. I'm not sure I like the suggested names, but I also didn't come up with any better ones either. Go forward with a new plugin and we can debate naming later.
Comment #36
heddnAs a first step, post the expected array output structure. Also, decide if you'll use XMLReader vs simple_xml().
Comment #37
jerech commentedHere is the proposed approach for the first step.
1. XMLReader vs SimpleXML:
I propose a hybrid approach. We should use
XMLReaderfor the primary traversal of the document to prevent memory limit exhaustion on massive XML migrations. Once it iterates to the targetitem_selector, we pass that specific node tosimplexml_load_string()to cast it into our array structure. This gives us streaming memory efficiency + easy row parsing.2. Expected Array Structure:
To solve the attribute handling (Issue B) and shape requirements for
sub_process(Issue C), we need a predictable format.Given this XML fixture:
The new parser would yield this array structure:
This isolates attributes under
@attributes, maps mixed nodes to@value, and natively supportssub_processiteration without needing custom callbacks.Let me know if this looks solid, and I can start drafting the new plugin against MR !162!
Comment #38
heddnI reworked the tests in MR !162 so they don't assume a solution. They run a real migration with
track_changes: trueon a fixed feed and assert on the destination rows. Each candidate configuration is one data-provider row, so the candidates are compared on the same outcomes.I checked each candidate with a throwaway implementation. And propose sub-fields as a possible solution.
Proposal: sub-fields
Let a field declare what to read from each element it matches:
A field with sub-fields always yields a list of arrays of strings, even for a single match. That fixes A and C. Attributes and namespaced children are ordinary selectors (
@target,payload/dc:title), and those already work today.Backwards compatibility
fieldschange, and existing migrations behave exactly as before. The module's existing parser and source tests give identical results with the prototype applied.children()check stays as it is (B). Changing it changes pass-through output (#2837684-25: Migration failed with source plugin exception: Serialization of SimpleXMLElement is not allowed). Attribute selectors and sub-fields avoid it.Edge cases
The rule: if the right value is clear, it arrives. If it isn't, the migration fails with a message naming the culprit. Losing data silently is never acceptable. The edge cases are covered in
XmlComplexElementEdgeCaseTest.