Problem/Motivation

Migrations (YML) files can be stored either under config/install or under migrations folder.
The first will import migration config into Drupal upon enabling the custom migration module.
The later will make migrations available in drush ms and changes will be understood with a simple drush cr instead of uninstalling/re-installing the migration custom module. I believe this makes it by far more convenient than adding migrations under config/install.

$ tree
.
├── faq_csv_migrate.info.yml
├── migrations
│   └── users.yml
└── README.md

On the second occasion migrations added under the migrations folder are never listed in the UI, although they are executed with the patch in #2924296: Batch (VBO) functionality for Web UI executions.

Proposed resolution / User interface changes

Also display migrations found under the migrations folder in the UI.

Additional Information

YML under config/install folder:
migrate tools ui

YML under migrations folder:
migrate tools ui

Drush (on both cases):

$ drush ms
 ------------------- ------------------ -------- ------- ---------- ------------- --------------------- 
  Group               Migration ID       Status   Total   Imported   Unprocessed   Last Imported        
 ------------------- ------------------ -------- ------- ---------- ------------- --------------------- 
  Users (users)       users              Idle     20      20         0             2019-11-01 23:36:35  

 ------------------- ------------------ -------- ------- ---------- ------------- ---------------------
CommentFileSizeAuthor
migrate_tools_ui.png13.3 KBbserem
users_under_config_install.png14.17 KBbserem
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

bserem created an issue. See original summary.

bserem’s picture

bserem CreditAttribution: bserem as a volunteer commented
Issue summary: View changes
heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC commented

This is somewhat intentional. I'm not sure we want to web UI littered with hundreds of drupal provided migrations.

bserem’s picture

bserem CreditAttribution: bserem as a volunteer commented
Status: Active » Closed (works as designed)

Oh I see! That actually makes sense.

Thanks

stefan.korn’s picture

stefan.korn
Primary language German
Location Jossgrund
CreditAttribution: stefan.korn at Stefan Korn - Webentwicklung commented
Category: Bug report » Feature request
Status: Closed (works as designed) » Active

Sorry on excavating this. But I am currently also thinking about this and it seems to me it would be a improvement if the Web UI shows the same as drush migrate-status. Though one could probably not get with the current approach of the Web UI and using EntityListBuilder for displaying the list.

How would the web UI get cluttered by this? I think drush migrate-status is not cluttered by hundred of drupal provided migrations?

And as per this change record https://www.drupal.org/node/2668742 it seems that the migrations are now plugins and not config entities anymore. But it seems migrate_plus does introduce migration config entity type (again)? I am a bit confused about this.

Maybe one should at least try to list the migrations from the migrations folder below the migration config entities from migrate_plus.

jigarius’s picture

jigarius
Location Montréal
CreditAttribution: jigarius as a volunteer commented

I'm facing a similar problem as well. My migrations are all defined in a module like "foo/migrations", however, I want the site's administrators to be able to see and run them in the UI. Thinking aloud, maybe this could be a setting? Show non-config migrations in the UI?

bserem’s picture

bserem CreditAttribution: bserem at Annertech commented

@jigarius when you are done with developing your migration, assuming it will not change, you can move it to `config/install` and enabled the module that provides it. This way it will be visible in the UI.

It will become configuration, so any updates on the file itself will not be reflected, but you can re-install the module or write an update hook to force it (I think that's possible).

natts’s picture

natts
He/they
Primary language English
Location London
CreditAttribution: natts at natts.com commented

I also want my migrations, located in the current correct place (which is 'modulefolder/migrations'), to be listed in the UI.

Moving them to config/install is not best practice any longer. Re-installing modules or writing update hooks is messy, laborious and unnecessary. I don't have to do this for Twig templates, for instance, or new/updated classes in src, so why for migrations?

If there really is some kind of issue about about having too many migrations shown, then just have tabs or another UI mechanism to filter them.

JonMcL’s picture

JonMcL CreditAttribution: JonMcL commented

Adding my use case for this ..

We are in active development of a new Drupal application. Migration source is an Oracle database (non Drupal). We do not have any migrations created by Drupal itself. All of our migrations are being created like foo_migrate/migrations/*.yml. These migration configs are very rapidly changing and will go through many rounds of refinement before they can be considered ready for config/install or config/sync directories.

Our issue is that all our testing and staging environments are in Kubernetes clusters and ssh access is not readily available. So no easy drush execution.

It would be great to have a UI that can run theses migrations. I fully understand it is not a simple problem to solve. The current lists are based on displaying migrate_plus.migration configs while migrations that exist inside of a migrations/ folder are actually plugins. So maybe a separate UI that only lists plugins might be needed.

jigarius’s picture

jigarius
Location Montréal
CreditAttribution: jigarius as a volunteer commented

Indeed, bserem. That's how I had them, but then they need to change so config/install is not possible in my case. I have considered some other solutions which I'll share here.

After every time I do a drush config:import I can do a drush config:import --partial --source=modules/custom/foo_migrate/config/install to update my migration configuration.

I have also considered setting up symlinks to all my migration YML files inside Drupal's config sync directory. At the point of writing this comment, I think the config_ignore module can only ignore imports and not exports. Thus, a config:export would overwrite my symlinks. However, if it were to support "ignore during export only" some day then my symlinks approach might work.

IMHO, all migrations are migrations irrespective of how they're being created. So it'd be great to have a similar UI to manage them all. But maybe some people might want to hide certain "internal" migrations from the UI? In which case, a parameter like "hidden: true" in the migration YML might help.

That said, for now, I'm just going to live without these migrations in the UI. Sad, but true.

bserem’s picture

bserem CreditAttribution: bserem at Annertech commented

You can use hook_install, like this for example:

/**
 * Update config from config/install.
 */
function hook_update_8201() {
  $message = NULL;

  if (\Drupal::moduleHandler()->moduleExists('views')) {
    $config_path = \Drupal::service('extension.list.module')->getPath('EXAMPLE') . '/config/install/views.view.EXAMPLE.yml';
    $data = Yaml::parseFile($config_path);
    \Drupal::configFactory()->getEditable('views.view.EXAMPLE')->setData($data)->save(TRUE);
    $message = 'The EXAMPLE views have been updated.';
  }
  else {
    $message = 'Oh no! Something went wrong! Is views installed?';
  }
  return $message;
}

This is from a module of mine (admin_feedback) and worked fine. You'll have to update the paths to reflect migrations instead of views.