I am running a D6 to D8 migration and I believe I have everything migrating correctly except files. The fields do get created but no files get migrated. I also verified that in files_managed it does not have any of my files.

Any idea what it can be or any help on how to debug this?

Here are some steps I did to migrate:
- installed migrate modules
- ran drush migrate-upgrade --configure-only --legacy-db-url=mysql://mydb:mydb@localhost/mydb --legacy-root=/var/www/files/
- drush config-export --destination=/tmp/migrate
- copied migrate files to custom module
- created custom group
- ran drush migrate-import --group=d6soinc

The two fields that have the files issue:

  • field_image
  • field_upload_files

migrate_plus.migration.update_d6_file.yml

uuid: 84172432-fdf3-41b7-95d2-97118d01a1cd
langcode: en
status: true
dependencies: {  }
id: upgrade_d6_file
migration_tags:
  - 'Drupal 6'
migration_group: mygroup
label: Files
source:
  plugin: d6_file
process:
  fid: fid
  filename: filename
  uri:
    plugin: file_uri
    source:
      - filepath
      - file_directory_path
      - temp_directory_path
      - is_public
  filemime: filemime
  filesize: filesize
  status: status
  changed: timestamp
  uid: uid
destination:
  plugin: 'entity:file'
  urlencode: true
  source_base_path: /var/www/files/
migration_dependencies:
  required: {  }
  optional: {  }

migrate_plus.migration.update_d6_node_page.yml

uuid: 1239f8c0-ba73-4587-9d4c-a5ff713d683a
langcode: en
status: true
dependencies: {  }
id: upgrade_d6_node_page
migration_tags:
  - 'Drupal 6'
migration_group: mygroup
label: 'Nodes (page)'
source:
  plugin: d6_node
  node_type: page
process:
  type:
    plugin: default_value
    value: page
  nid: tnid
  vid: vid
  langcode:
    plugin: default_value
    source: language
    default_value: und
  title: title
  uid: node_uid
  status: status
  created: created
  changed: timestamp
  promote: promote
  sticky: sticky
  body/format:
    plugin: migration
    migration: upgrade_d6_filter_format
    source: format
  body/value: body
  body/summary: teaser
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
  field_image:
    plugin: d6_cck_file
    source: field_image
  field_upload_files:
    plugin: d6_cck_file
    source: field_upload_files
destination:
  plugin: 'entity:node'
  default_bundle: page
migration_dependencies:
  required:
    - upgrade_d6_user
    - upgrade_d6_node_type
    - upgrade_d6_node_settings
    - upgrade_d6_filter_format
  optional:
    - upgrade_d6_field_instance_widget_settings
    - upgrade_d6_field_formatter_settings
    - upgrade_d6_upload_field_instance

Comments

albertski created an issue. See original summary.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Active » Postponed (maintainer needs more info)
Related issues: +#2640842: Make related file migration ID configurable in d6_cck_file

In terms of the files themselves not being migrated, I'm guessing that the source_base_path of /var/www/files/ is incorrect. The Drupal 6 file URIs are of the form /sites/default/files/..., and the source_base_path is prepended to that - if you're going to migrate the files from a local directory, they need to be below sites/default, and you need to provide a source_base_path of the parent directory of sites.

Note that once you fix that, the file fields won't be populated - see #2640842: Make related file migration ID configurable in d6_cck_file.

albertski’s picture

albertski commented

Thanks Mike for the input.

I updated migrate_plus.migration.update_d6_file.yml from:

source_base_path: /var/www/files/

to:

source_base_path: https://www.mysite.com

I also applied patch from #2640842: Make related file migration ID configurable in d6_cck_file but I still don't get any files.

This is my response from the upgrade_d6_file during migration:

 [debug] Calling Drupal\migrate_tools\MigrateExecutable::import() [11.46 sec, 45.03 MB]
 [status] Processed 2391 items (0 created, 0 updated, 2391 failed, 0 ignored) - done with 'upgrade_d6_file' [42.4 sec, 46.8 MB]
mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

When providing more information to an issue marked "Postponed (maintainer needs more info)", be sure to reset the status to Active.

When there are errors during import, they are saved in the migrate_message table for the migration and can be viewed with the drush migrate-messages command:

drush mmsg upgrade_d6_file

What do the messages look like? Please don't paste all of them;)...

Note that https might not work out-of-the-box - see http://stackoverflow.com/questions/12612484/copy-file-from-https-url.

albertski’s picture

albertski commented

This is some of the output from drush mmsg upgrade_d6_file:

    2d3ec2b0c547e819346e6ae03f881fd9f5c978ff3cbe29dfb807d40735e53703  1    File https://www.mysite.com/default/files/image.jpeg could not be copied to public://image.jpeg. 
  e2ae6a3570a53932d73194d032328e43e6da28aa4d2eb460a02995cf295f72a8  1    Could not create directory 'public://layout'    
  54b5253e8750d02ef1116a242cc374a0f067161446cc3c439d34e1e0cce58bc7  1       Could not create directory 'public://uploaded_files'

I do see that the folders: layout and uploaded_files did get created and they seem to have valid write permissions.

albertski’s picture

albertski commented
Status: Postponed (maintainer needs more info) » Active
mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

Can you verify https://www.mysite.com/default/files/image.jpeg is readable (go to the URL as it is in the message table with a browser)?

As for the "Could not create directory" - I can't think of a reason for that if the directories are actually there with the proper permissions.

albertski’s picture

albertski commented

Actually the errors I posted were wrong, they were like this:
File https://www.mysite.comdefault/files/image.jpeg could not be copied to public://image.jpeg.

So the problem was source_base_path was missing the trailing slash.

Updated migrate_plus.migration.update_d6_file.yml from:
source_base_path: https://www.mysite.com
To:
source_base_path: https://www.mysite.com/

The files did get migrated and can see them when going to /admin/content/file but they aren't there when you edit the node. I did apply patch 30 from #2640842: Make related file migration ID configurable in d6_cck_file.

Not sure if this an issue but during migration now I get an error that Mysql Server has gone away during the user import :

[debug] Calling Drupal\migrate_tools\MigrateExecutable::import() [1117.1 sec, 50.38 MB]
 [error]  PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /var/www/site/web/core/lib/Drupal/Core/Database/Statement.php:59
Stack trace:

I just rerun the migration and it continues. I think it may be because I am running this migration via Vagrant and it is running out of memory.

albertski’s picture

albertski commented

I read through the comments in #2640842: Make related file migration ID configurable in d6_cck_file and I was able to get the files to show up in my nodes.

I updated migrate_plus.migration.update_d6_file.yml from:

id: upgrade_d6_file

To:

id: d6_file

And the files showed up correctly in my nodes.

Thanks for your help Mike.

albertski’s picture

albertski commented
Status: Active » Closed (fixed)