The XML::fetchNextRow() method currently returns a SimpleXMLElement when the current selector doesn't render to a string and has children. This seems to be in conflict with core's MigrateExecutable::import() which starts by calling $source->rewind(). This will result in an exception being thrown ("Migration failed with source plugin exception: Serialization of SimpleXMLElement is not allowed"), and the migration is interrupted.

Quickly googling for this problem indicates that SimpleXMLElement can't really be serialized, so returning that as an object is simply not possible. I'm proposing that instead of returning the object, the code should be altered to return the element's XML, which can easily be converted back to the object with simplexml_load_string().

Since might be relevant, in this migration I've got a set of XML files in an import directory (crawled by the hook_install hack), and one of my processors needs to use the XML element, as the specific child being imported depends on a Drupal variable (the XML element contains different translations of a string, and which one gets imported depends on the current site's language).

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

jcnventura created an issue. See original summary.

jcnventura’s picture

itsekhmistro’s picture

Status: Needs review » Reviewed & tested by the community

Reviewed. The exception is thrown by Row->rehash() method.
So migrations are affected only if the param `track_changes` is enabled in migration configs.

  ...
  source:
    track_changes: true
...

The patch works for me.

Note, In result of this patch Process plugin might retrieve string value instead of SimpleXMLElement object previously.
The $value can be easily converted back to SimpleXMLElement if needed.

mikeryan’s picture

Priority: Major » Normal
Status: Reviewed & tested by the community » Needs work

We are explicitly allowing the elements through per #2792765: Allow XML elements to pass through as field values to process plugins - this change would break BC (it would break anyone currently using that functionality).

Is this only a problem (as itsekhmistro suggests) when rehashing? If so, is there a way for us to do the XMLization only in that case? Perhaps with by overriding the Row used in our source plugin with an extension that overrides rehash() to unwind any SimpleXMLElements within?

jcnventura’s picture

It probably is related to track_changes, as the migrations where the exception is thrown also have this flag enabled. If I have time, I'll try to re-implement it taking into account your suggestions.

agileadam’s picture

I had to use this patch to get track_changes to work (worked great!) and did experience an issue with some custom code. I have updated my code to work nicely with or without the patch, but it would be nice to work out a way to do this without breaking BC. I tried to find some indication that a (re)hashing is taking place but didn't see anything at first glance.

holist’s picture

Status: Needs work » Needs review
StatusFileSize
new1.5 KB
new2.08 KB

I added the same fix to SimpleXml::fetchNewRow() as well.

I agree that the proper solution would be to just provide something serializable for $row->rehash(). I was already thinking of how overriding SourcePluginBase::prepareRow() could work to this end, but it was a bit too complicated for my time frame here to make up a solution because Row::rehash() manipulates the source data directly.

tuwebo’s picture

heddn’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Given the concern and complications here, should we add a test?

pookmish’s picture

#7 worked for me as well. I agree a test would be ideal

pookmish’s picture

Tip: I found if you are planning to do a `sub_process` process, before the process configuration add a callback plugin as such:

field_foo_bar:
  -
    plugin: callback
    callable: simplexml_load_string
    source: foo
  -
    plugin: sub_process
    key: bar
    process: ...
duaelfr’s picture

Good catch here!
For those who have issues with track_changes, have a look at #3050924: Migration: Serialization of 'SimpleXMLElement' is not allowed

I don't think #7 is a good approach as it forces workarounds like #11.

Let say I have a source like this:

<Item>
  <Links>
    <Link>
      <Url>http://youtube.com/foo</Url>
      <Title>Foo video</Title>
    </Link>
    <Link>
      <Url>http://youtube.com/bar</Url>
      <Title>Bar video</Title>
    </Link>
  </Links>
</Item>

I my migration I do the following:

source:
  item_selector: /Items/Item
  fields:
    -
      name: links
      selector: Links/Link
process:
  field_link:
    plugin: sub_process
    source: links
    process:
      uri: Url
      title: Title

I expect it to work but it does not and I don't expect to have to use a callback to reload something that has been serialized somehow.

das-peter’s picture

I've just posted a patch in #3076534-8: Xml parser doesn't create array of values if value doesn't render to string that casts complex values into multi-level associative arrays. As far as I understand this is what other plugins expect in "multivalue" cases.

michelle’s picture

I did a re-roll from #7. This just makes that patch apply on 6.0.x and doesn't do anything about the following discussion.

jumpsuitgreen’s picture

@Michelle Thanks for posting. Your patch (#14) resolved our issue.
We have a D10 site with migrations using the URL source plugin to fetch XML file elements. Empty elements would trigger this error:
Migration failed with source plugin exception: Serialization of 'SimpleXMLElement' is not allowed in /var/www/web/core/modules/migrate/src/Row.php line 391 After applying patch #14, the migration runs smooth.

joachim’s picture

@das-peter did you mean the patch from #3076534: Xml parser doesn't create array of values if value doesn't render to string should fix the issue here?

I tried the patch from #3076534: Xml parser doesn't create array of values if value doesn't render to string for my issue and it didn't fix it.

joachim’s picture

I also tried the patch from #14, and the values I get in my source row don't look right any more:

  "mail" => "<property name="Email"/>"
joachim’s picture

I'm trying to understand the exact circumstances that produce this error, as some of my XML migrations are working fine.

Is it caused by the value-holding XML item having an attribute, for instance?

e.g.:

This works ok, because the xpath selector is 'property[@field="field-alpha"]' and the resulting item has no attributes, and so renders to a string:

		<property name="field-alpha">
			<value>
                            value-to-be-migrated
			</value>
		</property>

But this doens't work:

		<property name="field-alpha">
			value-to-be-migrated
		</property>
joachim’s picture

Version: 8.x-3.x-dev » 6.0.x-dev
joachim’s picture

I can see a partial fix: replace:

          if ($value->children() && !trim((string) $value)) {

with:

          if ($value->hasChildren() && !trim((string) $value)) {

MR coming.

It's a fix in that the use case I gave in #18 now works.

But it's not really a complete fix because this arm of the condition will still produce something that will crash Migrate:

          if ($value->hasChildren() && !trim((string) $value)) {
            $this->currentItem[$field_name][] = $value;
          }

This code was added in #2860785: Allow XML elements to pass through as field values to process plugins (for SimpleXML processor), but it's not clear how back when it was added it wasn't causing this crash!

joachim’s picture

Status: Needs work » Needs review

I'm not sure about where to put a test, as the tests in tests/src/Kernel/Plugin/migrate_plus/data_parser are unit-ish tests -- they don't test the data parser plugins in the context of a migration. Where should tests that cover integration with the migration system go?

heddn’s picture

Is there any real world testing for MR 109? What's in there looks reasonable.

heddn’s picture

Status: Needs review » Needs work
Issue tags: -Needs issue summary update

The problem seems to also exist in src/Plugin/migrate_plus/data_parser/Xml.php. Can we add test coverage for this and post fixes for both places?

Example xml fixture to use:

  <?xml version="1.0" encoding="utf-8"?>
  <items>
    <item id="1">
      <values>
        <value name="alpha"/>
      </values>
    </item>
  </items>

jerech made their first commit to this issue’s fork.

jerech’s picture

Status: Needs work » Needs review

I've updated the MR. Instead of changing children() to hasChildren() (which broke BC for elements with attributes), I implemented a SerializableSimpleXml wrapper. This correctly fixes the Row->rehash() serialization exception while fully preserving BC for downstream plugins. I also added the test coverage with the XML fixture you provided. Ready for review!

heddn’s picture

Status: Needs review » Needs work
Issue tags: -Needs tests
jerech’s picture

Status: Needs work » Needs review
heddn’s picture

Status: Needs review » Needs work

There are a lot of stray commits/changes included here. Can we rebase things?

- Modeler API commit (1f22c59)
- efc64aa "Fix Snippet process plugin throwing warning on invalid YAML instead of exception"

jerech’s picture

Status: Needs work » Needs review
heddn’s picture

I've spent some time on this because it keeps stalling, and I think I know why. There are three separate bugs tangled in this issue, and every patch so far fixes one and regresses another.

A. Serialization. Row::rehash() cannot serialize a SimpleXMLElement. Only bites with
track_changes: true. This is the issue title.
B. Attributes. children() is truthy even when it holds zero children, so
is treated as a complex element. joachim's #18 and #20.
C. Shape. sub_process needs arrays; neither objects nor strings satisfy it. das-peter's
#12, and really #3076534.

I'm going to post some tests. Then I suggest we might deprecate these troublesome plugins and add a replacement.

heddn’s picture

I've pushed tests. What do we think about deprecating and replacing? What would that look like?

jerech’s picture

I completely agree with your analysis in #31. Trying to fix the existing xml and simple_xml data parsers in place has proven to be a game of whack-a-mole with BC breaks. Deprecating the current plugins and introducing a new, robust replacement is definitely the smartest path forward.

Regarding what the replacement would look like, I propose we create a new data parser plugin (e.g., xml_array or xml_parsed). Unlike the current parsers that yield SimpleXMLElement objects, this new parser should inherently cast the parsed XML elements into standard PHP arrays (for complex elements/children) and scalar strings (for values/attributes) before yielding the row.

This approach solves all three issues at once:

Serialization (A): Arrays and strings are natively serializable, so Row::rehash() with track_changes: true will work flawlessly.

Attributes (B): We can build the new parser's logic to explicitly separate attributes from child elements into an intuitive array structure (e.g., pushing attributes into an @attributes key, similar to how JSON conversion handles it).

Shape (C): By yielding standard associative arrays, it will play perfectly with the sub_process plugin right out of the box without requiring custom callbacks.

If you are on board with this concept, I can start drafting this new data parser plugin against your new MR (!162) so we can get those failing tests to pass using the new architecture. Let me know your thoughts!

heddn’s picture

Status: Needs review » Needs work

The concept I agree with. Naming is hard. I'm not sure I like the suggested names, but I also didn't come up with any better ones either. Go forward with a new plugin and we can debate naming later.

heddn’s picture

As a first step, post the expected array output structure. Also, decide if you'll use XMLReader vs simple_xml().

jerech’s picture

Here is the proposed approach for the first step.

1. XMLReader vs SimpleXML:
I propose a hybrid approach. We should use XMLReader for the primary traversal of the document to prevent memory limit exhaustion on massive XML migrations. Once it iterates to the target item_selector, we pass that specific node to simplexml_load_string() to cast it into our array structure. This gives us streaming memory efficiency + easy row parsing.

2. Expected Array Structure:
To solve the attribute handling (Issue B) and shape requirements for sub_process (Issue C), we need a predictable format.

Given this XML fixture:

<item id="1" type="article">
  <title>Migrate Plus</title>
  <links>
    <link target="_blank">http://example.com</link>
    <link>http://drupal.org</link>
  </links>
</item>

The new parser would yield this array structure:

[
  '@attributes' => [
    'id' => '1',
    'type' => 'article',
  ],
  'title' => 'Migrate Plus',
  'links' => [
    'link' => [
      0 => [
        '@attributes' => ['target' => '_blank'],
        '@value' => 'http://example.com',
      ],
      1 => 'http://drupal.org',
    ],
  ],
]

This isolates attributes under @attributes, maps mixed nodes to @value, and natively supports sub_process iteration without needing custom callbacks.

Let me know if this looks solid, and I can start drafting the new plugin against MR !162!

heddn’s picture

I reworked the tests in MR !162 so they don't assume a solution. They run a real migration with track_changes: true on a fixed feed and assert on the destination rows. Each candidate configuration is one data-provider row, so the candidates are compared on the same outcomes.

I checked each candidate with a throwaway implementation. And propose sub-fields as a possible solution.

Proposal: sub-fields

Let a field declare what to read from each element it matches:

fields:
  -
    name: links
    selector: links/link
    fields:
      - { name: url, selector: url }
      - { name: title, selector: title }
      - { name: target, selector: '@target' }

A field with sub-fields always yields a list of arrays of strings, even for a single match. That fixes A and C. Attributes and namespaced children are ordinary selectors (@target, payload/dc:title), and those already work today.

Backwards compatibility

Edge cases

The rule: if the right value is clear, it arrives. If it isn't, the migration fails with a message naming the culprit. Losing data silently is never acceptable. The edge cases are covered in XmlComplexElementEdgeCaseTest.