Core migration issues

Next video meeting 2025-04-24

Hello all, it’s time for the biweekly migration subsystem meeting. The meeting will take place in slack in various threads. This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every second Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. #3456078: [meeting] Migrate Meeting 2024-07-18 2100Z)See the parent issue for an idea of the typical agenda.
Transcript will be exported and posted to the agenda issue. For anonymous comments, start with a 👤 emoji. To take a comment or thread off the record, start with a 🚫 emoji.

0️⃣ Who is here today?

benjifisher Hi! Sorry, I am running a little late. :alarm_clock:
mikelutz (he/him) o/
dinarcon :wave:
Dan Davis Greetings

1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.

benjifisher Convert MigrateSource plugin discovery to attributes
benjifisher SQL source plugins: allow defining conditions and join in migration yml
benjifisher Documentation: using hook_query_tag_alter()
dinarcon Moving \Drupal\migrate_drupal\Plugin\migrate\source\d8\Config source plugin to migrate module. Similar to[#3498915]

2️⃣ Action items. To be added later.

benjifisher https://www.drupal.org/project/drupal/issues/3421014: update the MR after latest review. @godotislate (probably next Monday, or anyone else who can get to it sooner)
benjifisher https://www.drupal.org/project/drupal/issues/3069776: Get the issue ready for review now that the blocker has been fixed. Volunteer needed.
benjifisher Somewhere in the guide https://www.drupal.org/docs/8/api/migrate-api/migrate-source-plugins, add an example of using hook_query_tag_alter() using the query tags added by SqlBase::prepareQuery(). For bonus points, give examples using a traditional hook implementation and an OOP hook. @benjifisher
benjifisher Open an issue to move the d8_config source plugin from the migrate_drupal module to the migrate module, similar to[#3498915]. @benjifisher

3️⃣ Statistics

Migrate Initiative Meeting TBA
Migrate Initiative Meeting Google sheet for recording stats: https://docs.google.com/spreadsheets/d/1o0Rjlc1vnnLP5bM5P-SMMyGzqn7258hi...
benjifisher All open: 356
benjifisher Fixed since last week's meeting: 4 (not counting the issue for the meeting).
benjifisher RTBC: 4, 1 of which is Major, and has not been updated in 3 months.
benjifisher NR: 5, 2 of which are Major.

4️⃣ Comment in this thread if you are looking for ways to contribute. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...

5️⃣ Previous minutes.

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/3497391 (edited)
benjifisher It still needs a transcript.
Dan Davis I am adding this transcript now
Dan Davis Updated.

6️⃣ Announcements

7️⃣ Convert MigrateSource plugin discovery to attributes

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/3421014 (edited)
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1739455578919459?thread_ts=... (edited)
benjifisher I reviewed and tested over the weekend. I think it is close, but I asked for some minor changes.
benjifisher @quietone made the most important change, but there are a few still pending. @godotislate, can you take a look?
benjifisher Feel free to discuss "perhaps" and "probably" in my comment.
benjifisher This issue was unblocked when[#3258581] was fixed.
godotislate @godotislate, can you take a look?maybe on the holiday Monday (edited)

8️⃣ SQL source plugins: allow defining conditions and join in migration yml

Migrate Initiative Meeting https://www.drupal.org/project/drupal/issues/3069776 (edited)
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1739456239488759?thread_ts=... (edited)
benjifisher This issue is unblocked because[#2833060] was recently Fixed.
benjifisher I have not looked at it recently. How close is it to being ready?
benjifisher We agreed to work on[#3500698] as a follow-up issue. That decision made it possible to finish #2833060.
benjifisher @heddn, do you know how close this issue is to being ready?
heddn @benjifisher which issue? 3069776 is pretty close
benjifisher Yes, that is the one I meant.

9️⃣ Documentation: using hook_query_tag_alter()

Migrate Initiative Meeting Do we have any examples of this in our documentation?
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1739456449824939?thread_ts=... (edited)
benjifisher I asked about this in the previous meeting, but I did not get any answers. @mikelutz (he/him), @dinarcon?
dinarcon Link to thread in previous meeting for reference https://drupal.slack.com/archives/C226VLXBP/p1738273717740989
dinarcon I use this heavily when debugging migrations... It lets me see the source query.
use Drupal\Core\Database\Query\AlterableInterface;

/**
 * Implements hook_query_TAG_alter().
 */
function udrupalcom_query_migrate_alter(AlterableInterface $query) {
  /** @var \Drupal\migrate\Plugin\MigrationInterface $migration */
  $migration = $query->getMetaData('migration');

  if (in_array('Drupal 7', $migration->getMigrationTags())) {
    var_dump((string) $query);
  }
  
}

(edited)

dinarcon Reading the original thread, the request is for an example of altering the query...
dinarcon I sometimes do the following to only import a subset of records... That way I have some sample data for my migrations without having to wait for all records to be imported.
use Drupal\Core\Database\Query\AlterableInterface;

/**
 * Implements hook_query_TAG_alter().
 */
function udrupalcom_query_migrate_alter(AlterableInterface $query) {
  /** @var \Drupal\migrate\Plugin\MigrationInterface $migration */
  $migration = $query->getMetaData('migration');

  if (array_key_exists('high_water_property', $migration->getSourceConfiguration())) {
    $query->range(0, 100);
  }
  
}

(edited)

benjifisher This is related to the issues in 8️⃣. The SqlBase base class adds tags to the query, and we want to clean up that class. But in the mean time, we should make sure that we have examples of what those tags are good for. (edited)
benjifisher @dinarcon, do you mention that debugging code in any of your articles?
dinarcon I updated my code snippets to note that, in fact, I am using the migrate tag added in \Drupal\migrate\Plugin\migrate\source\SqlBase::prepareQuery  My examples are mostly for debugging purposes. I also use the migration object added as metadata. (edited)
benjifisher I was beginning to question my short-term memory ...
mikelutz (he/him) I mean, the documentation for using the hook is here: https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Database%... adds the tag “migrate” and “migrate_” by default to queries that extend it and don’t override it in prepareQuery().  I don’t know that we document that we add that query tag.  It’s not in the SqlBase file documentation.
mikelutz (he/him) I don’t think it’s necessary to document how to use the hook, that is documented elsewhere and not really a migrate thing. What should be documented is simply the fact that SqlBase adds those tags by default to migration queries.
benjifisher I think it would be helpful to add under https://www.drupal.org/docs/8/api/migrate-api/migrate-source-plugins, not as API documentation.
dinarcon I have not included those snippets in articles written so far, but I plan to include debugging tips in the last article of the current D7 to D10 migration series.
benjifisher Maybe add an @see comment to the doc block for prepareQuery().
dinarcon The one to see the query as a string might be useful in general, but the one to limit the records to process in the presence of high_water_property might not be good advise. I mostly use it to get sample content, but it does not take into account relationships among entities so things could break or not function as expected. (edited)
benjifisher New documentation page: https://www.drupal.org/docs/8/api/migrate-api/migrate-source-plugins/mod..., corrections, and other comments are welcome!
benjifisher I tested using an OOP hook. As I hoped, it lets you target a tag like migrate_d7_node:page directly. You cannot do that with hook_query_TAG_alter() since : is not allowed as part of a function name.
benjifisher I also took the opportunity to mention the batch_size option. I have long thought that this option needs to be more widely known.
benjifisher @nicxvan: ^^
benjifisher @mikelutz (he/him): What is the best practice for the parameter type declaration? If you want to use hasTag(), then you should specify AlterableInterface (as in database.api.php). But if you want to use condition(), then it should be SelectInterface.
benjifisher I guess AlterableInterface&SelectInterface. I have never used an intersection type before (and I had to look up the syntax). I guess there is a first time for everything.

🔟 Moving \Drupal\migrate_drupal\Plugin\migrate\source\d8\Config source plugin to migrate module.

Migrate Initiative Meeting We already moved the content_entity source plugin.
Migrate Initiative Meeting (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1739458332693849?thread_ts=... (edited)
benjifisher I agree. Let's create an issue for it.
dinarcon FWIW, it was mentioned at[#3315257]#comment-15126074
benjifisher I would like to finish converting source plugins to attributes before working on this. We almost ended up reverting content_entity to source_module:migrate_drupal in that issue.
benjifisher Other than that conflict, moving content_entity was pretty easy. One thing we decided was to keep the version in migrate_drupal intact (but deprecate it) instead of making it a wrapper for the new version in the migrate module. That way, we could add parameter and return type declarations to the new version without breaking BC.
benjifisher I added[#3506605] and already added a couple of comments. We can continue the discussion there.

1️⃣1️⃣ Wrap-Up

Migrate Initiative Meeting Thanks for coming all! See you in 2 weeks
benjifisher I will add some action items under 2️⃣ after my day job.

Participants:

Comments

benjifisher created an issue. See original summary.

benjifisher credited heddn.

benjifisher’s picture

Issue summary: View changes
Status: Active » Needs review
smustgrave’s picture

Status: Needs review » Fixed

Seems all threads were captured

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.