Our Drupal 6 website has a huge nodes(about one million nodes), we are preparing to migrate to Drupal 8. When I run "drush migrate-upgrade", I can see "... Killed", meanwhile, the script will be interrupted. By debuging, I find the drush script crashes at: return new \IteratorIterator($this->query->execute()); in the class of SqlBase when migrating nodes to Drupal 8. Could you improve the node migrate plugin to process the huge node migration.

Thanks for your hard work.

Comments

merryfreespace created an issue. See original summary.

cilefen’s picture

cilefen CreditAttribution: cilefen commented
Version: 8.0.0 » 8.0.x-dev
Category: Feature request » Bug report

Can you post a stack trace and whatever error message resulted?

merryfreespace’s picture

merryfreespace CreditAttribution: merryfreespace commented

The below is error messages:

$ drush migrate-upgrade --legacy-db-url=mysqli://root:merry@192.168.1.2/drupal6 --legacy-root=http://drupal6.dev
Upgrading d6_date_formats
Upgrading d6_dblog_settings
Upgrading d6_file_settings
Upgrading d6_search_settings
Upgrading d6_system_cron
Upgrading d6_system_date
Upgrading d6_system_file
Upgrading d6_system_image
Upgrading d6_system_image_gd
Upgrading d6_system_logging
Upgrading d6_system_maintenance
Upgrading d6_system_performance
Upgrading d6_system_rss
Upgrading d6_system_site
Upgrading d6_upload_field
Upgrading d6_url_alias
Upgrading d6_user_mail
Upgrading d6_user_settings
Upgrading menu
Upgrading menu_settings
Upgrading search_page
Upgrading taxonomy_settings
Upgrading text_settings
Upgrading update_settings
Upgrading d6_filter_format
Missing filter plugin: filter_null. [error]
Upgrading block_content_type
Upgrading block_content_body_field
Upgrading d6_custom_block
Upgrading d6_user_role
Upgrading d6_block
Upgrading d6_file
Upgrading d6_user_picture_file
Upgrading user_picture_field
Upgrading user_picture_field_instance
Upgrading user_picture_entity_display
Upgrading user_picture_entity_form_display
Upgrading d6_user
Upgrading d6_node_type
Upgrading d6_node_settings
Upgrading d6_field
Upgrading d6_field_instance
Upgrading d6_field_instance_widget_settings
Upgrading d6_view_modes
Upgrading d6_field_formatter_settings
Upgrading d6_node__page
Upgrading d6_node__poll
Upgrading d6_node__story
Killed

catch’s picture

catch
he/him
Primary language English
CreditAttribution: catch commented
Title: Upgrading D6 to D8 will crash with huge nodes » Upgrading D6 to D8 will crash with lots of nodes

Looks like issue is the number of nodes, not that individual nodes are huge.

merryfreespace’s picture

merryfreespace CreditAttribution: merryfreespace commented

The node story has the majority of our content nodes. There are 800,000 nodes which belong to it.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan at Acquia commented

There's not much to go on here... @merryfreespace, can you try it with the drush -d (debug) flag and post the output from around "Upgrading d6_node__page" forward? That might give some more hints.

One guess for a cause would be #2309695: Add query batching to SqlBase - since the database query fetches all results into memory on execute, a query of many nodes (@merryfreespace, how many nodes are there in the failing migration?) could exhaust memory.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan at Acquia commented
Status: Active » Closed (duplicate)

Actually, looking back at the issue summary and seeing that it was traced to \IteratorIterator($this->query->execute()), I'm confident that it is memory exhaustion on the query - closing in favor of the existing issue.