Migrate Source Directory

Use a directory of files as the source for a migration.

Common use cases include:

• Import a set of files from a directory as Drupal file entities
• Find all Markdown files in a directory and create a content entity for each one

Source plugin configuration options:

source:
  plugin: directory
  track_changes: true
  # One or more directories to scan for files. Will recurse into subdirectories.
  directory:
    - /path/to/files/for/import
  # (optional) A regex mask to apply to file paths, relative to the directory(s)
  # above. Only files that match this pattern will be processed. Use to do
  # things like only process files with a given extension.
  file_mask: '/(.*\.(mp3|m4a|wav)$)/i'
  # (optional) How deep to recurse into subdirectories. -1 (default) for no
  # limit.
  recurse_level: -1
  # (optional) prefix to add to the sourceID when processing files. The default
  # for sourceID is the path to the file relative to the directory set above.
  id_prefix: 'optional_sourceid_prefix'
  # (optional) List of one or more strings that a file must contain in order
  # to be considered for processing. This probably only really works with text
  # like files since it uses strpos().
  file_must_contain_string:
    - '<div class="chapter">'

This plugin provides the following source fields:

• sourceID: Unique ID of the file (filepath)
• source_file_basename: Base name of the file
• source_file_extension: File extension
• source_file_filename: Filename and extension
• source_file_mtime: Time the file was last updated
• source_file_path: Path without filename
• source_file_pathname: Path to the file
• source_file_realpath: Absolute path to the file
• source_file_size: File size
• source_file_type: File type

Create Drupal File entities from a directory of files

Use the plugin like so to import files in to Drupal:

id: directory_mp3
label: 'Import audio files'
source:
  plugin: directory
  constants:
    uri_file: 'public://'
  track_changes: true
  directory:
    - /path/to/files/for/import
    - /path/to/additional_files
  # (optional) A regex mask to apply to file paths, relative to the directory(s)
  # above. Only files that match this pattern will be processed. Use to do
  # things like only process files with a given extension.
  file_mask: '/(.*\.(mp3|m4a|wav)$)/i'
  # (optional) How deep to recurse into subdirectories. -1 (default) for no
  # limit.
  recurse_level: -1

process:
  source_full_path: source_file_path
  uri_file:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/uri_file
        - source_file_filename
    -
      plugin: urlencode
  filename: source_file_filename
  uri:
    plugin: file_copy
    source:
      - '@source_full_path'
      - '@uri_file'

destination:
  plugin: 'entity:file'

migration_dependencies:
  required: {  }
  optional: {  }

To reference these files in other migrations, use the source property sourceID. This is the path to the file relative to source.directory.

Create Drupal content entities from a directory of files

You can also use this to iterate over a set of files, and import the content of those files as Drupal content entities. For example; a directory of Markdown files could be imported as nodes with a migration like this:

id: markdown_import
id: markdown_import
label: 'Import markdown files'

source:
  plugin: directory
  track_changes: true
  directory:
    - /path/to/files/for/import
  # All files with a .md extension, excluding README.md files.
  file_mask: '/^(.*\.md)(?<!README\.md)$/i'
  recurse_level: -1

process:
  type:
    plugin: default_value
    default_value: "basic_page"
  title: title
  body/value: body
  body/format:
    plugin: default_value
    default_value: "markdown"
  changed: last_updated

destination:
  plugin: 'entity:node'

migration_dependencies:
  required: {  }
  optional: {  }

And either a corresponding implementation of hook_migrate_prepare_row(), or a custom source plugin class that extends \Drupal\migrate_source_directory\Plugin\migrate\source\Directory and implements the required logic in its own prepareRow() method.

Example:

function hook_migrate_prepare_row(Row $row, MigrateSourceInterface $source, MigrationInterface $migration) {
  if ($migration->id() == 'markdown_import') {
    $content = file_get_contents($row->getSourceProperty('source_file_realpath'));

    // The first line of the file, if it starts with `#` is the title.
    $title = strtok($content, "\n");
    if (substr($title, 0, 1) == '#') {
      $row->setSourceProperty('title', trim(substr($title, 1)));

      // Remove the first line (we're done with it), and any empty newlines
      // above/below the resulting content.
      $content = trim(substr($content, strpos($content, "\n") + 1));
    }

    // And the rest is the body.
    $row->setSourceProperty('body', $content);
  }
}