Problem/Motivation

File migration configuration settings are confusing and need better documentation.

source

destination

  • source_base_path - This is the absolute path to the site for the content to be migrated. Depending on the setting of the "source_path_property" (see next item) this can be either a local filepath (/var/www/mysite.dev) or a uri (http://mysite.dev) - no trailing slash in either case.
  • source_path_property - can be set to "filepath" or "uri".
  • destination_path_property - can be set to "filepath or "uri", defaults to "uri" (under what conditions would this need to be overridden?)
  • move - set whether source files are moved or copied from the source - defaults to "FALSE" (copied).
  • urlencode - set whether source file URL needs to be urlencoded prior to copy/move - defaults to FALSE.
  • rename - set whether a file is renamed or replaced copied from source to destination if the file already exists - defaults to FALSE (replaced).

Proposed resolution

Figure out what all the parameters mean, and document them here: https://www.drupal.org/node/2257723

Remaining tasks

Post list of all configuration settings then have big brains fill in any blanks.

User interface changes

None.

API changes

None.

Comments

ultimike’s picture

ultimike
Primary language English
Location Florida, USA
CreditAttribution: ultimike commented
Issue summary: View changes

A couple of notes:

  • In Drupal\migrate\Plugin\migrate\destination\EntityFile, there is no default value for "rename" in the constructor. Any objections if I create a patch to add it (and default it to FALSE, which is it's implicit value anyway)?
  • The file migration configuration parameters are confusing the way they are structured because different parts of the source path a split between source and destination configuration values. For example, consider a file that lives at an absolute source path of http://mysite.com/sites/mysite.com/files/images/cheese.jpg. To configure a file migration, I believe it would look like this:
    source: 
  conf_path: sites/mysite.com
destination:
  source_base_path: http://mysite.com

    I think it is a little counterintuitive that different parts of the source URL to a file would be split between the "source" and "destination" configuration values. Not sure if there is anything we can do about it, but I figured I'd mention it.

ultimike’s picture

ultimike
Primary language English
Location Florida, USA
CreditAttribution: ultimike commented
Title: Making sense of the File migration configuration settings » Making sense of (and documenting) the File migration configuration settings
benjy’s picture

benjy CreditAttribution: benjy at CodeDrop commented

I believe some of these settings came across from D7 so mikeryan might be the best person to comment on some of them. I'm guessing destination_path_property is configurable to be flexible between different media storage such as core files, file entity etc.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan at Acquia commented

I haven't looked closely at file migration - urlencode is the only one I see that comes directly from D7. First thing, we need to distinguish between configuration that's part of the migrate module's file destination plugin, from stuff that migrate_drupal adds for Drupal 6/7 file sources. Question: is https://www.drupal.org/node/2257723 supposed to be for migrations to Drupal 8 in general, or just for D6/D7 upgrades? The title suggests the former but the content seems to focus on the latter...

Anyway, I don't know the D8 file plugin well enough (yet) to definitively explain how it constructs the paths, but I suggest looking at https://www.drupal.org/node/1540106 for a model - this gives examples of how source_dir, destination_dir, and destination_file apply in D7.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan at Virtuoso Performance commented
Status: Active » Postponed
Related issues: +#2695297: Refactor EntityFile and use process plugins instead

This is changing shortly - once #2695297: Refactor EntityFile and use process plugins instead is committed we should update the documentation accordingly.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous at Netuxo Ltd (RIP) commented

Are you sure there shouldn't be a trailing slash after source_base_path? When I did not add a trailing slash, migration of files did not work, and I could see in the migrate_message_upgrade_d6_file table messages that file /path/to/files/without/slashfiles/ does not exist. So it did just add "files/", and did not put a slash between the source_base_path and "files". Adding the trailing slash to source_base_path made it work.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan at Virtuoso Performance commented
Status: Postponed » Active

#2695297: Refactor EntityFile and use process plugins instead is in, unblocking. It ended up going into 8.2.x and later, so do we need to document both the old and new ways?

@Andreas Speck - It actually shouldn't matter if you provide a trailing slash, it will be added in if necessary. Definitely in the 8.2.x version, at least, but it certainly looked like 8.1.x was ensuring a trailing slash (adding it if necessary).

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

xenophyle’s picture

xenophyle CreditAttribution: xenophyle commented

Based on the issue mentioned in #8, this documentation is now completely wrong. I would like to fix it when I have the chance, but if someone else does it first, I won't be mad.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan as a volunteer and at Virtuoso Performance commented

Ideally that page would be deleted entirely, since there is no longer any configuration specific to entity:file, but in the meantime I've edited it to remove the obsolete configuration.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

quietone’s picture

quietone CreditAttribution: quietone as a volunteer commented
Issue tags: +drupal.org documentation

This is d.o documentation (tagged) and should be in the api docs as well.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

masipila’s picture

masipila CreditAttribution: masipila as a volunteer commented
Status: Active » Closed (outdated)

I'll close this issue as outdated (as there are no longer any configuration options).

I'll open a new issue for writing a handbook example on how to migrate files from non-Drupal source systems.

Markus

masipila’s picture

masipila CreditAttribution: masipila as a volunteer commented

I opened a separate issue for documenting an example on how to migrate files from non-Drupal source, see #2953166: Migrate handbook page for migrating files from non-Drupal source