Find a better way to preserve files

I'd like a little more information - whatever migration initially migrated the file, if it had preserve_files set, should have added the file_usage row so MigrateFileFid wouldn't... Ahhh, I see it, because we set the count to 1 the fid rollback decrements and deletes it. OK, definitely something to look into.

Well, now I'm having a similar problem in the FILE_EXISTS_REUSE case. I've never been entirely comfortable with using file_usage to accomplish this - for one thing, when can Migrate release its usage so people can manually delete files after migration? For another, it's tough to get the count just right.

When migrating files in their own migration (MigrateDestinationFile), ugly as it is we probably should replace the file_delete() call with our own version of file_delete() hacked to respect the preserve_files setting. Dealing with migrating files directly into a file/image field will be tougher - since we're not the ones calling file_delete() when the referencing entity is rolled back, we'll probably have to maintain the file_usage stuff as best we can for that case. This is another reason to consider migrating files in their own migration, rather than directly into the field, as a best practice, so some documentation/education is an element in minimizing the issues here.

OK, I've committed the following changes:

1. For MigrateDestinationFile migrations, if preserve_files is set we use a custom fileDelete() method that skips deleting the actual file, it only cleans up the database.
2. For file/image field migrations, the code to set file_usage is now also called in the MigrateFileFid and FILE_EXISTS_REUSE cases (it was previously skipped).

Leaving open as a documentation issue - I want to highlight the behavior, and also emphasize using MigrateDestinationFile as a best practice.

How exactly is this supposed to work? I'm using migrate_d2d to create a DrupalFile6Migration (to migrate my files):

$file_arguments = $common_arguments + array(
  'machine_name' => 'File',
  'description' => t('Migration of Files from Drupal 6'),
  'user_migration' => 'User',
  'default_uid' => 1,
  'source_dir' => 'sites/default/files/rs6',
  'destination_dir' => 'public://legacy',
);
Migration::registerMigration('DrupalFile6Migration', $file_arguments['machine_name'], $file_arguments);

Then, in my DrupalNode6Migration override class, in the __construct method, I simply map the field with it's source migration and file_class of "MigrateFileFid":

// Logo field.
$this->addFieldMapping('field_logo', 'field_logo')
  ->sourceMigration('File');
$this->addFieldMapping('field_logo:file_class')
  ->defaultValue('MigrateFileFid');

I import all my files, which works great. Then I import my nodes, which also works. However, when I rollback my nodes, the corresponding logo files get deleted.

Where do I indicate that the files should be preserved on rollback?

You might try:

$this->addFieldMapping('field_logo:preserve_files')
     ->defaultValue(TRUE);

Thanks, Mike. However, the files were still deleted. Here's the code I used:

$this->addFieldMapping('field_logo', 'field_logo')
  ->sourceMigration('File');
$this->addFieldMapping('field_logo:file_class')
  ->defaultValue('MigrateFileFid');
$this->addFieldMapping('field_logo:preserve_files')
     ->defaultValue(TRUE);

For what it's worth, looking at my Migrate UI field mapping page, it shows the following alert (with the code above):

"field_logo:preserve_files" was used as destination field in "" mapping but is not in list of destination fields

Since I'm using the MigrateFileFid file_class, won't this disable any customized field mapping for the file?

I see it, although MigrateFileFid is calling markForPreservation, the preserveFiles flag is actually implemented in MigrateFile, that stuff needs to move down to MigrateFileBase...

Thanks.

This works for me.

migrate/plugins/destinations/file.inc
641c641
< $file->preserve_files = FALSE;
---
> //$file->preserve_files = FALSE;

This line prevents the markForPreservation to work in import

53 protected function markForPreservation($fid) {
54 if (!empty($this->preserveFiles)) {

I also got the same problem when i rollback article images.

we are at a point where we are having to write a module to store the file references and put them back in when doing an import, this is really wrong and I can't figure out why migrate is doing this.

I'm having the same problem, and in a big migration (about 100,000 nodes and 300,000 media files) is quite an issue when I need to rollback the nodes. I haven't found a solution yet, if I do I post it around here.

I am bumping this to major (its a critical bug in my opinion though) since this bug is actually quite destructive - it deletes the files from the file system not just the references to them in the database.

This should address the issue with MigrateFileFid respecting preserve_files.

For documentation purposes, you'll want to add the parent class fields() in MigrateFile::fields():

return parent::fields() + array(
  'destination_dir' => t('Subfield: <a href="@doc">Path within Drupal files directory to store file</a>',
    array('@doc' => 'http://drupal.org/node/1540106#destination_dir')),
  'destination_file' => t('Subfield: <a href="@doc">Path within destination_dir to store the file.</a>',
    array('@doc' => 'http://drupal.org/node/1540106#destination_file')),
  'file_replace' => t('Option: <a href="@doc">Value of $replace in that file function. Does not apply to file_fast(). Defaults to FILE_EXISTS_RENAME.</a>',
    array('@doc' => 'http://drupal.org/node/1540106#file_replace')),
);

Otherwise, your patch didn't work for me. The files are preserved upon rollback of my File migration, but not upon rollback of any migrations dependent upon the File migration (where the File migration is a sourceMigration() for one of the dependency migrations).

If I comment out the line as in comment #10 above, the files are preserved and dependent migrations won't delete the files.

Here's an updated patch which incorporates these two additions.

Nevermind, I take that back. I forgot to add the "preserve_file" in my dependent class. Yep, your patch works great, except you need to add the parent fields. Thanks!

Committed, with the parent fields added.