there's a need to occasionally limit machine names to a select number of characters. see #2244805: Fix d6_contact_category exceeding bundle's max length for one such example, where the bundle machine name is required to be 32 characters or less.

additional options should be added to the DedupeBase process plugin to substr the destination machine name value.

CommentFileSizeAuthor
#23 interdiff-2247903-21-20.txt4.09 KBbdone
#23 2247903-dedupe-substr-21.patch14.68 KBbdone
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 72,036 pass(es). View
#20 interdiff-2247903-20-17.txt1.41 KBbdone
#20 2247903-dedupe-substr-20.patch15.32 KBbdone
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,672 pass(es), 4 fail(s), and 0 exception(s). View
#18 interdiff-2247903-17-10.txt962 bytesbdone
#18 2247903-dedupe-substr-17.patch15.97 KBbdone
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Failed to run tests: failed during invocation of run-tests.sh. View
#16 interdiff-2247903-10-8.txt6.46 KBbdone
#16 2247903-dedupe-substr-10.patch15.79 KBbdone
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Failed to run tests: failed during invocation of run-tests.sh. View
#10 interdiff-2247903-10-8.txt6.46 KBbdone
#10 2247903-dedupe-substr-10.patch15.79 KBbdone
None View
#8 interdiff-2247903-8-5.txt3.68 KBbdone
#8 2247903-dedupe-substr-8.patch15.4 KBbdone
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 70,987 pass(es), 0 fail(s), and 1 exception(s). View
#5 2247903-dedupe-substr-5.patch15.03 KBbdone
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 70,954 pass(es), 0 fail(s), and 1 exception(s). View
#2 imp-2247903-machine-name-substr.patch6.52 KBbdone
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 69,589 pass(es). View
Members fund testing for the Drupal project. Drupal Association Learn more

Comments

bdone’s picture

working on a patch for this...

bdone’s picture

Status: Active » Needs review
FileSize
6.52 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 69,589 pass(es). View

here's a first pass at this one. a sample usage, from #2244805: Fix d6_contact_category exceeding bundle's max length, could look like below where we are adding length...

process:
  ...
  id:
    -
      plugin: machine_name
      length: 32
      source: category
benjy’s picture

+++ b/core/modules/migrate/lib/Drupal/migrate/Plugin/migrate/process/MachineName.php
@@ -37,7 +38,20 @@ public function transform($value, MigrateExecutable $migrate_executable, Row $ro
+      $new_value = substr($new_value, $start, $length);

Lets use Unicode::substr()

We also need to discuss how the truncating will work with dedupe. This process plugin is nearly always used with dedupe and that currently has no context of the maximum length of the machine name.

benjy’s picture

Status: Needs review » Needs work

We discussed this on IRC and decided that the best approach would be to:

  1. Add a default length of 32 to the machine name plugin
  2. Add a length configuration option to DedupeBase
  3. Update any migrations using the machine name and dedupe plugins to specify a length of 32 to dedupe.
  4. Update DedupeBase so that it can handle creating a non duplicate value at the correct length.
bdone’s picture

FileSize
15.03 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 70,954 pass(es), 0 fail(s), and 1 exception(s). View

worked on comments in #4 a little bit. this does not yet address bullets #1 and #4, but i'm posting what i have so far.

bdone’s picture

Title: Add a substr options to MachineName process plugin » Add a substr options to DedupeBase process plugin
Issue summary: View changes

updated title, based on #4.

benjy’s picture

Thanks for working on this.

  1. +++ b/core/modules/migrate/lib/Drupal/migrate/Plugin/migrate/process/DedupeBase.php
    @@ -31,6 +33,16 @@ public function transform($value, MigrateExecutable $migrate_executable, Row $ro
    +      throw new MigrateException('Input character length should be an integer.');
    

    Should we expand on this to say that "null" or no value will capture the entire string?

  2. +++ b/core/modules/migrate/tests/Drupal/migrate/Tests/process/DedupeEntityTest.php
    @@ -82,10 +83,44 @@ public function testDedupe($count, $postfix = '') {
    +    if (isset($start)) {
    +      $configuration['start'] = $start;
    +    }
    +    if (isset($length)) {
    +      $configuration['length'] = $length;
    +    }
    

    I can't see where we use configuration where we couldn't simply use $start and $length?

Other than above, It's looking good so far.

bdone’s picture

FileSize
15.4 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 70,987 pass(es), 0 fail(s), and 1 exception(s). View
3.68 KB

1. Should we expand on this to say that "null" or no value will capture the entire string?

does this patch read any better?

2. I can't see where we use configuration where we couldn't simply use $start and $length?

not quite sure what you mean here @benjy. add the start config key alongside length, for each migration using dedupe_entity, or something else?

benjy’s picture

@bdone, ignore my last comment I missed this line in dreditor as it was greyed out.

plugin = new DedupeEntity($configuration, 'dedupe_entity', array(), $this->getMigration(), $this->entityQueryFactory);

The error messages ready better, thanks.

bdone’s picture

thanks @benjy.

Update DedupeBase so that it can handle creating a non duplicate value at the correct length.

here's a go at addressing that one.

bdone’s picture

i think this patch now has everything from the list in #4, except #1...

Add a default length of 32 to the machine name plugin

i do not know how to address this one. any suggestions here would be appreciated.

bdone’s picture

Status: Needs work » Needs review

The last submitted patch, 5: 2247903-dedupe-substr-5.patch, failed testing.

The last submitted patch, 8: 2247903-dedupe-substr-8.patch, failed testing.

bdone’s picture

bdone’s picture

FileSize
15.79 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Failed to run tests: failed during invocation of run-tests.sh. View
6.46 KB

same patches as #10. trying another test.

Status: Needs review » Needs work

The last submitted patch, 16: 2247903-dedupe-substr-10.patch, failed testing.

bdone’s picture

Status: Needs work » Needs review
FileSize
15.97 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Failed to run tests: failed during invocation of run-tests.sh. View
962 bytes

i managed to track down why MigrateDrupal6Test was failing. trying once more...

Status: Needs review » Needs work

The last submitted patch, 18: 2247903-dedupe-substr-17.patch, failed testing.

bdone’s picture

Status: Needs work » Needs review
FileSize
15.32 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 71,672 pass(es), 4 fail(s), and 0 exception(s). View
1.41 KB

re-rolled patch for psr-4 and fixed failing test issue in while loop.

bdone’s picture

Status: Needs review » Needs work

The last submitted patch, 20: 2247903-dedupe-substr-20.patch, failed testing.

bdone’s picture

Status: Needs work » Needs review
FileSize
14.68 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 72,036 pass(es). View
4.09 KB

one more try with unit test passing and an earlier substr of value...

mikeryan’s picture

Status: Needs review » Reviewed & tested by the community

Looks good to me - the one open issue here is the idea of defaulting machine names to 32 characters, but I also don't see how that would work without introducing complection between the machine_name and dedupe_entity plugins.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed df301e7 and pushed to 8.x. Thanks!

  • Commit df301e7 on 8.x by alexpott:
    Issue #2247903 by bdone: Fixed Add a substr options to DedupeBase...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.