File migration with MountManager

Problem/Motivation

Create a submodule that facilitates file migration from one defined Flysystem scheme to another using Flysystem ’s Mount Manager.

Mount Manager is a League\Flysystem wrapper class that makes it easy to work with multiple defined Flysystem file system instances from a single object. The League\Flysystem\MountManager is an easy to use container allowing you to simplify more complex cross file system interactions.

Drupal\Flysystem has a submodule that can be enabled that provides a user interface for selecting files contained in one defined Flysystem scheme, and migrated them to another defined Flysystem scheme. The selected files will be added to a queue, and will get processed by Drupal cron in batches that can be defined through the user interface.

The speed of the file transfer will depend on a number of items:

• Speed of internet connection for remote file mount schemes
• Size of files
• Quantity of files
• Responsiveness of the remote file system object API

For large numbers of files or large sized files, you may find that manually migrating them outside of Drupal to your remote object store, and then utilizing Flysystem’s flysystem:reconcile Drush command is faster, as the file transfer doesn’t have to go through Flysystem itself. See the README for more details on how this works.

Proposed resolution

The module would work by creating a file management interface in the Admin UI that:

• Provides a selector for the source schema (dropdown)
• Provides a selector for the target schema (dropdown)
• Provides a button that loads a selectable list of files from the source schema that can be selected to migrate to the target schema.

You then:

• Select the files from the listing that you wish to migrate to the target schema.
• Select the batch size you wish to use for the migration queue using the Batch size file.
• Save the populated form.

The migration batch will be added to a queue, and will get processed during Drupal cron runs, and the migration batch will display on the “Mount Manager Migrations” listing page. You can monitor the progress of the migration by visiting this page where you will see:

• The source scheme
• The target scheme
• The batch size configured
• The total number of files batched for migration
• The number of files migrated
• The number of files left to migrate.
• The status of the migration (Queued, In Progress, Complete).

Once any queued migrations have reached completion, the module may be uninstalled until the next time it is needed.

The migration process will take care of the following:

• Moving the selected files from the source scheme to the target scheme.
• Updating the scheme for the file in the file managed table.

Note that files will be deleted from the source scheme once the file has been migrated to the target scheme.

Remaining tasks

1. Create the submodule.
2. Write Tests.
3. Write Documentation.

User interface changes

New administrative interface for creating and monitoring schema to schema migrations via Mount Manager.

API changes

Mount Manager entities that can “technically” be leveraged by Drupal’s Migrate API to build larger scale migrations that are managed with the Migrate module processes.