Problem/Motivation

Currently the way we handle exceptions on configuration errors is inconsistent in our process plugins. Sometimes an exception is thrown in the constructor, sometimes we throw exceptions for invalid configurations in the constructor, sometimes we don't do it until the transform method is called. There is some inconsistency in which exceptions get thrown as well.

All plugins should do the following:
If the plugin configuration is invalid, and this can be determined from the configuration alone with no value to transform, the plugins should throw a consistent exception in the constructor, specific exception to be determined.

If the configuration is valid, but certain inputs are not, then the plugin should determine this in the transform method and throw a consistant exception there, exception tbd

In most cases, moving or changing the exception will be a breaking change, so we will likely have to add @trigger_errors in all the places we want to put new exceptions warning people that we will be adding the new exceptions in the next major version, Drupal 10 currently if we can get this into Drupal 9.4. We should have a version of the patch ready for D10 as well that actually adds the excpetions.

Steps to reproduce

Proposed resolution

Modify these process plugins to use InvalidArgumentException.
ProcessField
Explode
FormatDate
MachineName
MakeUniqueBase
StaticMap - Breaking change, add @trigger_error in the constructor if bypass and default value are set.

Remaining tasks

Review
Commit
Update Change record

User interface changes

API changes

Data model changes

Release notes snippet

Issue fork drupal-3247950

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mikelutz created an issue. See original summary.

mikelutz’s picture

Status: Active » Needs review
quietone’s picture

Issue tags: +Bug Smash Initiative

Nice cleanup! and as a bonus this simplifies testCallbackExceptions

I guess this needs a CR so people know to catch a different exception?

mikelutz’s picture

I've added a change record, I've also moved the exception from an overridden constructor into the transform method. While all of this, (throwing a different exception and throwing the exception in a different place is technically a behavior change, it brings this plugin in line with the other migration plugins. During any normal migration workflow, whether through the core ui or drush or with the common contrib migrate modules, this change should be completely non-disruptive. I can't think of any practical way that this change could cause a problem. You would have to be running a custom executable, specifically handling this plugin separately for some reason which I can't fathom. Even so, the new exception would still be caught and handled correctly along with the other plugins now, and I consider the fact that it sent out the wrong exception in the wrong place to be a bug that this is fixing.

quietone’s picture

Status: Needs review » Reviewed & tested by the community

@mikelutz, thanks!

I applied the MR locally, read the code and ran the tests. It all looks good to me.

Then I read the CR and thought it could use a little improvement in the way it explains what Callback does that is different than other process plugins.

All together I think this is good to go.

benjifisher’s picture

We have discussed these exceptions before. The way I remember it, @heddn convinced me that we should throw different exceptions in different cases.

If you can tell from the configuration alone that there is a problem, then throw an \InvalidArgumentException from the constructor. This has to be fixed in the code. It will fail for every row, and there is no point in throwing an exception (and logging a message) for every row.

If it depends on the value passed to the transform() method, then throw a MigrateException there. Some rows may fail, others may not.

Found it: #2882276-71: Extended callback process plugin to call functions with multiple parameters is pretty close to what I just said. Search that issue for "InvalidArg" and you will see it in a few other comments, including #69, which refers to #2936463-27: Throw exception when "callable" plugin is called with undefined on invalid callback where @heddn asked for InvalidArgumentException in the Callback constructor.

On #2882276 we also discussed whether to use something other than InvalidArgumentException. @heddn suggested "invalid plugin exception", but the closest I could find was InvalidPluginDefinitionException. No one made a follow-up issue.

If you agree with this approach, then instead of using MigrateException in Callback, we should move the first test in Explode to the constructor and change the exception.

Maybe also FormatDate. I do not know why the transform() method returns an empty string in some cases before validating the configuration.

MachineName throws MigrateException from the constructor.

MakeUniqueBase is the first one where I have noticed an @throw comment in the doc block. I guess we should clean that up. The validation could be moved to the constructor, and we could use whatever exception we agree on for constructors. Substr is similar.

MigrationLookup has a couple of @throw comments after {@inheritdoc}. We are not supposed to do that, right?

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

StatusFileSize
new6.9 KB

@benjifisher, thanks for doing the research on this. On reflection, what heddn says seems right to me.

With that in mind I started working on a patch, updating the process plugins mentioned in #7. I've used InvalidArgumentExcpetion because I haven't put any thought into if it should be that or InvalidPluginDefinitionException. I also searched for all process plugins in core that have an throws statement. There were 12 to check and of those only ProcessField is a candidate for a change. The patch does not do that one or MakeUniqueBase.

If anyone has a custom process plugin that creates one of these process plugins and is catching the MigrateException, this change would cause it to break.

Regarding the @throws after @inheritdoc, the coding standards for inheritdoc do not explicitly say this should not be done.

danflanagan8’s picture

Status: Needs review » Needs work

@quietone, that patch looks really good.

I wonder if static_map should be updated too. It throws an exception if default_value and by_pass are both set...but only if the value does not get mapped. I would argue that the exception should get thrown in the constructor since it would make it easier to nip this problem in the bud.

Other than that it seems like we still have to move exceptions to the constructor in:

substr
MakeUniqueBase

I'll set to NW.

Also tagging for IS update, since the direction of this issue has changed dramatically.

danflanagan8’s picture

mikelutz’s picture

Title: Callback process plugin should throw a MigrateException on invalid input » Throw consistent exceptions on invalid process plugin configurations.
Category: Bug report » Task
Issue tags: +Needs change record

I think to move some of these we are going to have to ease into it. static_map, in particular. we need to @trigger_error in the constructor if bypass and default value are set, warning that having both is unsupported and will throw whatever exception in Drupal 10, and then add the exception in D10. Because danflangan8 is right, we should throw an exception earlier, but if all your values map, you can currently get away with it, so adding the exception is a breaking change.

Will need Change Record(s) too.

mikelutz’s picture

Issue summary: View changes
quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new4.4 KB
new11.62 KB

Add in core/modules/field/src/Plugin/migrate/process/ProcessField.php and the related test.

I think that leaves static_map and MakeUniqueBase to resolve.

Status: Needs review » Needs work

The last submitted patch, 14: 3247950-14.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review

Retesting because the test failure was due to #3255836: Test fails due to Composer 2.2 which has been fixed.

danflanagan8’s picture

Status: Needs review » Needs work

The changes for core/modules/field/src/Plugin/migrate/process/ProcessField.php look great. Thanks, @quietone!

I think that leaves static_map and MakeUniqueBase to resolve.

Setting back to NW to address these.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new7.4 KB
new19.02 KB

Here is a first attempt at static_map and make_unique_entity_field.

danflanagan8’s picture

Status: Needs review » Needs work

Getting close!

In addition to the "custom command failure", there are a couple things that caught my eye.

+++ b/core/modules/migrate/src/Plugin/migrate/process/StaticMap.php
@@ -163,9 +177,6 @@ public function transform($value, MigrateExecutableInterface $migrate_executable
-        if (!empty($this->configuration['bypass'])) {
-          throw new MigrateException('Setting both default_value and bypass is invalid.');
-        }

I don't think we want to remove this since the current behavior is to throw an exception. It would make more sense to change MigrateException to InvalidArgumentException now and then remove this in 10.0.0. when the constructor can start throwing the exception.

+++ b/core/modules/migrate/src/Plugin/migrate/process/MakeUniqueEntityField.php
@@ -101,6 +102,12 @@ class MakeUniqueEntityField extends MakeUniqueBase implements ContainerFactoryPl
+    if (isset($configuration['start']) && !is_int($configuration['start'])) {
+      throw new \InvalidArgumentException('The start position configuration key should be an integer. Omit this key to capture from the beginning of the string.');
+    }
+    if (isset($configuration['length']) && !is_int($configuration['length'])) {
+      throw new \InvalidArgumentException('The character length configuration key should be an integer. Omit this key to capture the entire string.');
+    }

Should this go in the constructor of MakeUniqueBase? There might be tricky BC implications with this one since it's a base class.

murilohp’s picture

StatusFileSize
new18.75 KB
new597 bytes

Here's a new patch fixing the coding standards from #18, I'll leave this issue in NW since we have to fix #19.

Thanks!

murilohp’s picture

Status: Needs work » Needs review
StatusFileSize
new18.31 KB
new3.62 KB

Addressing the changes from #19 on this new patch, and moving back to NR thanks!

danflanagan8’s picture

The static_map stuff in #21 looks good to me.

As far as the MakeUniqueBase stuff, in #19 I was hoping to get a little discussion going as opposed to asserting that the change should be made. From #19:

Should this [the exception throws] go in the constructor of MakeUniqueBase? There might be tricky BC implications with this one since it's a base class.

Currently, the InvalidArgumentExceptions get thrown by the transform method of MakeUniqueBase. The intention certainly appears to be that extensions of MakeUniqueBase declare the exists() method and let the base class handle the transform method. But transform is not final, so there's nothing stopping anyone from overriding transform too. Even if that doesn't make much sense.

I *think* from a BC standpoint, I'd like to see deprecation notices in the MakeUniqueBase::__construct() and leave MakeUniqueBase::transform untouched until 10.0.0. Maybe we could even make transform final in 10.0.0. I have no idea how that would work though.

Also, let it be know that I'm happy to be wrong about any of this! Leaving as NR to get additional input.

quietone’s picture

StatusFileSize
new4.87 KB
new18.67 KB

@danflanagan8, I think your right.

#22. This makes the suggested changes. Tests pass locally but I am not running the tests because the deprecation notice for MakeUniqueBase needs work. I've had a long day and just not up to thinking up some text.

quietone’s picture

Title: Throw consistent exceptions on invalid process plugin configurations. » Throw consistent exceptions on invalid process plugin configurations
danflanagan8’s picture

Status: Needs review » Needs work

Setting back to NW since @quietone indicated a desire to update the exception message. Currently it is:

Having a non integer length position is deprecated in drupal:9.4.0 and will throw an InvalidArgumentException in drupal:10.0.0.

I guess I would propose (with little conviction mind you!):

Using a non-integer value for length is deprecated in drupal:9.4.0 and will throw an InvalidArgumentException in drupal:10.0.0. See https://www.drupal.org/node/3248473

It looks like the deprecation notice in the static_map constructor needs the "See" added too.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new7.69 KB
new19.52 KB

@danflanagan8, thanks! I forgot to come back to this.

This patch uses the text from #25 and add the link to the change record to the deprecation message in StaticMap.

murilohp’s picture

StatusFileSize
new19.39 KB
new618 bytes

Hey, just fixing the code standards here.

danflanagan8’s picture

Status: Needs review » Needs work

Very close to the finish line here! I'm having second thoughts about the following change to static_map though:

+++ b/core/modules/migrate/src/Plugin/migrate/process/StaticMap.php
@@ -164,7 +178,7 @@ public function transform($value, MigrateExecutableInterface $migrate_executable
       if (array_key_exists('default_value', $this->configuration)) {
         if (!empty($this->configuration['bypass'])) {
-          throw new MigrateException('Setting both default_value and bypass is invalid.');
+          throw new \InvalidArgumentException('Setting both default_value and bypass is invalid.');
         }
         return $this->configuration['default_value'];

I'm the one that proposed this back in #19, but now I'm pretty sure this line should not be changed prior to 10.0.0. MigrateExceptions are handled gracefully, while InvalidArgumentExceptions cause the whole migration to quit immediately. We are changing MigrateException to InvalidArgumentException in a number of other places where we know that every single row will fail. So at that point there's arguably no behavior change.

In this static_map case though, this exception will not (likely) happen on every single row. That's because the configuration currently only gets validated if the source value is empty, not on every single row. Therefore there is a significant behavior change possible if we change from MigrateException to InvalidArgumentException before 10.0.0. We should continue throwing a MigrateException until 10.0.0.

So sorry about my bad advice in #19. But I think this is going to be the last change required.

murilohp’s picture

Status: Needs work » Needs review
StatusFileSize
new18.71 KB

No worries @danflanagan8! Here's a new patch.

murilohp’s picture

StatusFileSize
new1.58 KB

Ooops forgot the interdiff

quietone’s picture

Issue summary: View changes
StatusFileSize
new1.61 KB
new17.42 KB

I was reviewing this and saw that I left the changed exceptions in MakeUniqueBase::transform. See #22. The exceptions will be changed in D10.

I did a little work on the IS as well.

quietone’s picture

Issue tags: -Needs change record

I've updated the change record, it could use reviews.

Status: Needs review » Needs work

The last submitted patch, 31: 3247950-31.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new1.89 KB
new17.21 KB

Arg, I got called away and forgot I hadn't adjusted the test.

danflanagan8’s picture

I think the patch is ready.

I checked out the change record and made a couple three little edits. But I have a question about this statement from the change record:

Previously, both exceptions were caught and logged in similar ways in MigrateExecutable, so when executing migrations in normal migrate workflows, no action should be required.

Is that true? I thought that an InvalidArgumentException brought everything to a grinding halt. I see this block of code in MigrateExcecutable::import():

        try {
          foreach ($pipeline as $destination_property_name => $plugins) {
            $this->processPipeline($row, $destination_property_name, $plugins, NULL);
          }
          $save = TRUE;
        }
        catch (MigrateException $e) {
          $this->getIdMap()->saveIdMapping($row, [], $e->getStatus());
          $msg = sprintf("%s:%s: %s", $this->migration->getPluginId(), $destination_property_name, $e->getMessage());
          $this->saveMessage($msg, $e->getLevel());
          $save = FALSE;
        }
        catch (MigrateSkipRowException $e) {
          if ($e->getSaveToMap()) {
            $id_map->saveIdMapping($row, [], MigrateIdMapInterface::STATUS_IGNORED);
          }
          if ($message = trim($e->getMessage())) {
            $msg = sprintf("%s:%s: %s", $this->migration->getPluginId(), $destination_property_name, $message);
            $this->saveMessage($msg, MigrationInterface::MESSAGE_INFORMATIONAL);
          }
          $save = FALSE;
        }

It looks to me like InvalidArgumentException is not caught when thrown by MigrateExecutable::processPipeline. I don't have any "real" migrations at the ready though.

joachim’s picture

I'm not really following why we're throwing InvalidArgumentException instead of InvalidPluginDefinitionException.

Is it because InvalidPluginDefinitionException is for cases where the plugin definition in the annotation is incorrect, as opposed to the configuration that's passed to a plugin instance?

quietone’s picture

@joachim, I think the answer is in comment #7 of this issue, the links in the issue point to original point being made by heddn.

joachim’s picture

I read that, but what it says about the exception class is this:

> On #2882276 we also discussed whether to use something other than InvalidArgumentException. @heddn suggested "invalid plugin exception", but the closest I could find was InvalidPluginDefinitionException. No one made a follow-up issue.

and I don't really understand. If InvalidPluginDefinitionException isn't correct because this is about plugin *configuration* rather than *definition*, it seems to me we need an InvalidPluginConfigurationException in the core plugin component. What does the comment about a follow-up mean?

joachim’s picture

Just spotted that Libraries module has an exception class exactly for this purpose: https://api.druphelp.com/api/libraries/src%21Plugin%21MissingPluginConfi...

mikelutz’s picture

Status: Needs review » Needs work
Issue tags: +Portland2022, +Needs change record updates

Is that true? I thought that an InvalidArgumentException brought everything to a grinding halt. I see this block of code in MigrateExcecutable::import():

I swear I didn't make this up, but looking at it, I have no idea where I got this from..

If we are going to throw invalidArgumentExceptions in the constructor, we should probably catch that when we create the plugins, and gracefully reset the migration status, and bow out nicely...

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.