At https://www.drupal.org/node/2560435#comment-10282161, webchick expressed concern about the fact that the per-migration message tables do not use the PSR-3 compatible logging interface the rest of core does. Let's look at how they were used in D7 migrate. When messages were logged for specific items being migrated, they were stored in the migrate_message table corresponding to that migration, keyed by the source ID for each item (note that each migration has its own table, because the source key schemas vary for each migration). From the migrate dashboard, you could click the # messages column to arrive at a view of the messages:

We want to be able to display the messages for a given migration, and we want to be able to sort/query on the specific source IDs. Also, outside of the migration tools themselves, it was very common to query the message table directly for QA purposes - e.g., "SELECT CONCAT('http://example.com/node/', sourceid1),level,message FROM migrate_message_blog WHERE message LIKE '%missing related article ID%'" to generate a spreadsheet for manual review linking straight to source pages.

Another aspect of migration message handling that doesn't map well to simply using dblog is that we don't want to accumulate messages. That is, if we run the user migration, get a notice, fix the problem and rerun it, we should now see no messages. So, as the migration process fetches each source item to process, it deletes existing messages for that item.

That all being said, could we reimplement the message table handling as an implementation of LoggerInterface? Looking at the dblog implementation as an example:

  • We can pass the migration and source ID to log() in $context.
  • From the passed migration, we can derive the migration-specific table to use (because one migration might have an integer source ID, another might have a string, still another might have a multi-column key).
  • Add a delete() method to remove the messages for a given source ID.
  • Add a retrieval API (which we just did for the existing implementation: #2560435: Need an API for retrieving migration messages.

So, at this point I'm not seeing a big upside to this approach, but opening the issue for discussion...

CommentFileSizeAuthor
migrate_messages_example.jpg31.16 KBmikeryan
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mikeryan created an issue. See original summary.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan at Virtuoso Performance commented
Version: 8.1.x-dev » 8.2.x-dev
Issue tags: +neworleans2016, +Migrate BC break
mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan at Virtuoso Performance commented
Version: 8.2.x-dev » 9.x-dev
Status: Active » Postponed

It doesn't seem like there's a lot of interest in this - we're trying to get the migration APIs locked down for 8.2.x, and it's not looking like this one is going to happen. Postponing for 9.x.

catch’s picture

catch
he/him
Primary language English
CreditAttribution: catch at Third and Grove commented
Version: 9.x-dev » 8.3.x-dev

This could be done in a minor release, it doesn't have to be done of course, but I don't see anything that couldn't be done with a bc layer.

heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC commented
Issue tags: -Migrate BC break

Per #5, removing BC breaking tag.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan as a volunteer and at Virtuoso Performance commented
Status: Postponed » Active

This could be pursued if it comes with a BC layer.

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
CreditAttribution: claudiu.cristea as a volunteer and at Webikon for European Commission and European Union Institutions, Agencies and Bodies commented

My 2 cents from recent projects. The content managers/admins are not tech guys. Sometimes, it's even hard for them to understand how we split the whole migration in small migrations. When it comes to main objects like node articles or events it's clear for everybody, but when discussing about migrating text formats or other more technical migrations (field instances?) they are lost. What they desperately need from us is a good list of data inconsistencies. Our migration log should be able to produce such a list on a per-migration basis but the migration should be just a specific filter. More important is to be able to output and export the entire unified log, regardless of migration. We should think on how to store all the entries in a single storage, so we can query in multiple ways. I think a content entity is needed and let Views to the rest.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC commented
Status: Active » Closed (won't fix)

This won't be happening any time soon. Ideally we'll incubate this in contrib first as well. So going to close this issue down. If this does get fixed, it will be brought in via contrib and not this issue.

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
CreditAttribution: claudiu.cristea as a volunteer and at Webikon for European Commission and European Union Institutions, Agencies and Bodies commented
Status: Closed (won't fix) » Active

This won't be happening any time soon.

I don’t think this is a good reason to close this issue. I had several migrations with similar requests. Let’s keep it open, at least, to track and evaluate possible candidates from contrib.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Wim Leers’s picture

Wim Leers
Location Ghent 🇧🇪🇪🇺
CreditAttribution: Wim Leers at Acquia commented
Related issues: +#3095456: [Plan] Discuss strategy for long-term use of entity validation in the migration system, +#3063856: Add ability to view migrate_message table data

What they desperately need from us is a good list of data inconsistencies.

Related: #3095456: [Plan] Discuss strategy for long-term use of entity validation in the migration system.

Our migration log should be able to produce such a list on a per-migration basis but the migration should be just a specific filter. More important is to be able to output and export the entire unified log, regardless of migration. We should think on how to store all the entries in a single storage, so we can query in multiple ways. I think a content entity is needed and let Views to the rest.

Related: #3063856: Add ability to view migrate_message table data.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.