Problem/Motivation

One common case when manipulating a DOMDocument starting from an html field, e.g. node body, during a migration is to replace one specific bit inside it.

Proposed resolution

A new migrate process plugin that can manipulare a DOMDocument and then return the object for more processing or exporting into a html string.
This is naturally dependent on #2958281: Allow manipulating html from process plugins.

Completed tasks

  • Add tests.

Remaining tasks

User interface changes

N.A.

API changes

Not really an API change, but process plugins seem to assume that strings are passed along.
This new process plugin instead produces a DOMDocument on import, but a normal string on export.

Data model changes

N.A.

Comments

marvil07 created an issue. See original summary.

marvil07’s picture

Assigned: marvil07 » Unassigned
Status: Active » Needs review

Here a dom_str_replace migrate process plugin.

Also, thanks to benjifisher for some suggestions about this.

marvil07’s picture

marvil07’s picture

Adding a parent ticket.

benjifisher’s picture

Issue summary: View changes
Issue tags: +Needs tests
benjifisher’s picture

Issue tags: -Needs tests
StatusFileSize
new7.63 KB
new10.7 KB

The attached patch makes several changes compared to the one in #3:

  1. Add test coverage for the new DomStrReplace class (dom_str_replace plugin).
  2. Add a base class that can be used by DomStrReplace and other classes that work with DOMDocument objects.
  3. Add a new property $xpath to the base class.
  4. Replace the method getNodeList() with $this->xpath->query(...)
  5. Coding standards: replace null with NULL.
  6. Add an init() method to the base class to set the two properties.
  7. Instead of an explicit test $value instanceof \DOMDocument in the validate() method, rely on the type hint of the init() method.
  8. Minor change to the logic when selecting str_replace(), str_ireplace(), or preg_replace().
benjifisher’s picture

StatusFileSize
new7.63 KB

Oops, I used .patch instead of .txt on the interdiff.

benjifisher’s picture

The attached patch fixes coding standards found by the testbot.

I am attaching interdiffs comparing it to the the two previous patches (from #3 and #6).

Status: Needs review » Needs work

The last submitted patch, 8: 2958285-dom-string-replace-8.patch, failed testing. View results

benjifisher’s picture

Status: Needs work » Needs review
StatusFileSize
new1.6 KB
new9.27 KB
new10.81 KB

I think this should fix the failing tests.

Again, I am attaching an interdiff comparing to the patch in #3 as well as one for the previous patch.

Status: Needs review » Needs work

The last submitted patch, 10: 2958285-dom-string-replace-10.patch, failed testing. View results

benjifisher’s picture

Status: Needs work » Needs review
StatusFileSize
new465 bytes
new9.25 KB
new10.8 KB

Try again. Once again, I am attaching an interdiff for the patch in #3 as well as one for my previous attempt.

Status: Needs review » Needs work

The last submitted patch, 12: 2958285-dom-string-replace-12.patch, failed testing. View results

benjifisher’s picture

Status: Needs work » Needs review
StatusFileSize
new1.23 KB
new9.49 KB
new11.03 KB

You cannot catch a TypeError, so let's go back to the $value instanceof \DOMDocument test.

benjifisher’s picture

Issue tags: +midcamp2019
heddn’s picture

Status: Needs review » Needs work
  1. +++ b/src/Plugin/migrate/process/DomProcessBase.php
    @@ -0,0 +1,47 @@
    +      throw new \InvalidArgumentException('Dom plugins only work with \DOMDocument objects.');
    
    +++ b/src/Plugin/migrate/process/DomStrReplace.php
    @@ -0,0 +1,201 @@
    +      // @todo Move out once another mode is supported.
    

    Add an issue for the follow-up.

  2. +++ b/src/Plugin/migrate/process/DomProcessBase.php
    @@ -0,0 +1,47 @@
    +  protected function init($value) {
    +    if (!($value instanceof \DOMDocument)) {
    +      throw new \InvalidArgumentException('Dom plugins only work with \DOMDocument objects.');
    
    +++ b/src/Plugin/migrate/process/DomStrReplace.php
    @@ -0,0 +1,201 @@
    +    try {
    +      $this->init($value);
    +    }
    +    catch (\InvalidArgumentException $e) {
    

    Pass in as much context for the exception as possible. So we can print the migration and destination field, etc.

  3. +++ b/src/Plugin/migrate/process/DomStrReplace.php
    @@ -0,0 +1,201 @@
    +      // @todo Move out once another mode is supported.
    

    Add an issue for the follow-up.

  4. +++ b/src/Plugin/migrate/process/DomStrReplace.php
    @@ -0,0 +1,201 @@
    +        throw new MigrateException("Configuration option '$option_name' is required.");
    

    InvalidPluginDefinitionException?

  5. +++ b/src/Plugin/migrate/process/DomStrReplace.php
    @@ -0,0 +1,201 @@
    +    foreach ($this->xpath->query($this->configuration['expression']) as $html_node) {
    

    Would it make sense to use array_map and iterator_to_array?

benjifisher’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new8.28 KB
new12.29 KB

@heddn:

Thanks for the review, and for discussing this issue in person at MidCamp. I have implemented most of your recommendations.

  1. I added #3042833: Process non-attribute strings in the dom_str_replace process plugin, and I added an @see comment after the @todo comment.
  2. I did not see any way to get the current migration. Maybe using getPluginDefinition()? But maybe that is not really needed, since the Migrate system (or the drush commands from the Migrate Tools module) will make it clear which migration is active. I did pass $destination_property to the init() method, and simply used $this->getPluginId() in the base class: once the object is instantiated, it "knows" the data of the derived class.
  3. I think this is a repeat of (1).
  4. Done. I had to look at the constructor for the InvalidArgumentException class to find that it requires two arguments.
  5. I do not think that array_map() would be useful here. I think of that as a way tp generate a new array (with the same keys as the input) by transforming the values. Here, for each $html_node, we modify $this->document, the DOMDocument object. Also, I think there could be performance problems if we used iterator_to_array(). We do not need to collect all the $html_node elements in a single array.

For the record: as we discussed in person, the big change in error handling is that we do not catch the exceptions in (1) and (4). The reason is that these exceptions will be thrown when the migration that uses this plugin is mis-configured. In this case, it is better to stop the execution than to continue throwing errors for each row.

I also took the opportunity to expand the test coverage. We now test all of the validation checks in the constructor. I did this by removing one test method and adding a data provider, so we get more tests with only a few net lines added and less duplicated code.

Since the base class is not a plugin, I guess it should be moved to src/ from src/Plugin/migrate/process/. If I do that, then would you also like me to rename the class? Perhaps DomProcessPluginBase instead of DomProcessBase?

benjifisher’s picture

StatusFileSize
new889 bytes
new12.33 KB

This should fix the coding standards problems.

marvil07’s picture

@benjifisher, thanks for improving the code here, and special kudos for the added tests.
@heddn, I'm glad you could review the changes.

I have been looking at the changes, and they look great.
I have only added a minor change to make the introduced parent base class abstract, which is not really needed since it does not declare a plugin, but it will leave it clear that it is meant to be extended, which also reinforces the name of the class, and follows the pattern in core for the base process plugin class.

heddn’s picture

This looks good. Thanks for all your contributions to migrate!

On commit I will switched from throwing an invalid argument exception to throwing a skip row exception in init(). After thinking about this, we don't want to kill the entire migration if a single row or two isn't DOM parsable.

-      throw new \InvalidArgumentException($message);
+      throw new MigrateSkipRowException($message);

  • heddn committed 9af1f59 on 8.x-4.x authored by marvil07
    Issue #2958285 by benjifisher, marvil07, heddn: Allow replacing based on...
heddn’s picture

Status: Needs review » Reviewed & tested by the community
heddn’s picture

Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.