Implement basics of panopoly_migrate for migrating from Panopoly 1.x to 2.x

Some notes about what I've gone through so far:

• In panopoly_demo, we declared migration plugins in YAML files in the migrations/ directory - I started down that road for panopoly_migrate, but I think in the end it'd be better to provide config entities (like those from Migrate Plus), because then users can modify them with Migrate UI (eventually, when that actually works ;-)). However, everything in migrate is flux (see #2677198: Rewrite to use Migrations as Plugins and #2677196: Re-introduce migration config entity talking about putting the config entity in Migrate API, rather than Plus) so it might be "safer" to stick with the core supported YAML-based discovery of migration plugins until that stuff settles down?
• We might need to have separate modules for each panopoly_* module, ie. panopoly_migrate_pages, panopoly_migrate_core, etc, which is what I did in my WIP, but it'd be nice to avoid that if there's a way
• Because Migrate Tools' drush commands depend on config entities from Migrate Plus, and I'm using the YAML-based discovery, I had to create some simple drush commands in panopoly_migrate to actually run the migrations. :-/ This is pointing me more to using the config entities from Migrate Plus

An interesting complication regardless of how we're storing the migrations: the user will need to configure the source_base_path property in order for the file migration to work. If we provide config entities, well, we'll need to tell the user to change it. If straight migration plugins, then we'd need a alter hook to change the plugin definition, which would have to pull the value from configuration somewhere.

We really want the migration to not work until the user has configured this path, or somehow disable the files migration until they configure it?

Punting on the "how structure it question" and just trying to get the migration running. It's still not finishing, this patch includes mostly changes to help debugging.

There is another general problem I've been trying to come up with a solution for: how to have specific mappings/migrations for things in Panopoly, while generally importing the things we don't have new Panopoly equivalents for?

For example: content types.

Panopoly defines equivalent content types in Panopoly 2.x, like: Content Page and Landing Page. But the site builder might have added their own other content types. So, we'd want to have a generic content type migration to move over all content types, but to skip Content Page and Landing page, actually, even better would be to record the id mapping in the migration, but not actually overwrite anything on the destination. This way, anything that needs a content type can just using the 'migration' process plugin and it'd know how to convert the old type to the new type, without having to always specify an additional 'static_map' process plugin to map the known types.

This gets more complicated if the site builder has some other content types that they want to migrate specifically, but still may want to take advantage of the generic migration for the others.

With destination plugins for content entities, we can specify a configuration key called 'overwrite_properties' and if we give an empty array, I think it won't update anything on existing entities, and just add the mapping. However, there's no equivalent for config entities - those always overwrite everything.

There's an event, MigrateEvents::PRE_SAVE_ROW, that's called just before import where we could do some magic to unset all the destination properties if we see a certain configuration and a certain destination id?

This problem gets more complicated for other examples, like roles.

We will have equivalent roles in Panopoly 2.x with new default permissions set (once there's a solution to #2729261: Get permissions included in Drupal 8 features). But we've always let users change the default permissions, and so we'd want to migrate the permissions, and then somehow merge them into the new set of defaults. And that'll be tricky because it'll be hard to know that a user disabled a default permission on the Panopoly 1.x site, and so we shouldn't re-enable it on Panopoly 2.x, versus a permission that didn't used to be set by default, that we now want to set by default. :-/

This gets even more complicated with child distributions, where they may have their own set of defaults, that could be different than either Panopoly 1.x's and Panopoly 2.x's...

There's really two use-cases we want to cover:

1. Providing a set of default migrations that will "just work" for the average site built on the Panopoly 1.x profile (ie. not a child distribution)
2. Providing some reusable defaults and tools to make it possible to build the same kind of thing for a child distribution

And in both cases, we want to allow the site builder to customize the migration to some of the unique specifics on their site.

I think the solution for now, might be to just NOT provide general migrations for content types and roles (even if that means these migrations won't actually work in practice for a real site that has added new content types or roles) just so that we can move on and prove out the migrations for all the Panels-y stuff like FPP, Panelizer, etc.

Digging into this more, I think the solution to my customization problem (and maybe even the "what form should our included migrations take" problem) could be figuring out how to integrate with the Migrate Upgrade and core "Migrate Drupal UI" modules so that the configuration is dynamically generated. I'm not yet sure exactly how those modules generate their configuration, though, and what hooks or events we'll have to be able to modify it.

I've started down the road of trying to get Migrate Upgrade to generate migrations (in Migrate Plus configuration entities) that we can then run with the Drush commands from Migrate Tools.

This is a great article that walks through how this works:

https://drupalize.me/blog/201605/custom-drupal-drupal-migrations-migrate...

Not uploading a new patch because my new code is only sort of half working, and I want to be able to remove some of the old code before uploading again so it's not a totally mess. :-)

Ok, I've got a patch that implements the new approach of working with migrate_upgrade. It won't actually complete due to some errors, but it has much less code and we're moving in the right direction. I also realized that enabling the demo content is problematic when doing an upgrade, so we really need to implement #2755621: Optionally enable/disable demo content during install too

Small changes to get the code in the last patch actually working!

This will actually migrate the custom Panelizer display for an individual Content Page or Landing Page, although, loads of blocks will be omitted because we aren't mapping the old ids to the news ids, so, really only 'entity_field' blocks work. Next up would be migrating FPPs to custom blocks and then mapping those ids, so FPPs will come over!

About to start digging into this again and couldn't remember how any of it worked, so I added a README.txt for myself and others :-)

@DamienMcKenna: I'm going to work on some Panopoly-specific migrations here (as I figure out how to do this) but then we can abstract it into something generic. I think it should be in D8 branch of FPP, because that's what provided the entity type - and then after migrating you could disable it. Putting it in CTools would bring that baggage to all sites using CTools, even if they aren't migrations.

Here's a new patch which will import the Text widget from panopoly_widgets, that is, an FPP! We'll do all the panopoly_widgets types like this, but we could put code in FPP that would try to automatically migrate an FPPs, and we'd use an alter hook to remove the automatically generated ones for any widget from Panopoly (so we can control how the fields are mapped) just like we do for the node types from panopoly_pages.

However, when working on this, I can see that the custom_block entity is super limited... We might need another module to extend it and bring it closer to feature parity with FPPs so we have a place to put all the data from the FPP entities!

Anyway, that's something for the future - here I'm just going to try and make something that works for all of Panopoly's types.

We need UUID on custom block revisions to support the way we use vuuid:xxx to refer to blocks in FPP on D7 - here's the core issue #1812202: Add UUID support for entity revisions

Gah! I attempt to upload this patch earlier, but apparently it errored out or something. It adds improved metadata migration and all the FPP types from panopoly_widgets

And here's a patch that adds an event that can be used to extend how panes are mapped to blocks in Panels displays! Next is to have the FPP integration actually use it keep the FPPs on the Panels displays they are placed on.

I'm having some problem with my local test site that's causing the d7_user migration to always fail, and unfortunately, this causes the Panels display migration to not even run, so I'm having trouble getting started on the FPP conversion on the Panels display :-( I'm not really doing anything special, but I can't seem to Google up anyone else having this problem with d7_user