Migrating data from Amazontools (Drupal 5) to Amazon module (Drupal 6)

The Amazontools module (AAT) enables users to create certain nodes with attached product information from Amazon; it also allows to add "Related links from Amazon" to any other content type. Additionally there are several 3rd party modules like the ISBN/ASIN field that use the AAT storage backend and add further functionality. For example, the ISBN/ASIN offers a CCK field that accepts - you might have already guessed - an Amazon ASIN. Sadly AAT will not be upgraded past Drupal 5, so all created content will be lost when upgrading to Drupal 6 or beyond. Luckily there is the Amazon module which offers a migration path. This handbook page documents a well-tested migration procedure which allows D5 users to finally upgrade their sites to Drupal 6 without data loss.

What is being migrated from D5's 'Amazontools' (AAT) and 'ISBN/ASIN' field modules:

'amazon' and 'amazon_node' nodes from AAT are being migrated to D6;
"Related Amazon links" attached to any other content type are converted into D6 ASIN fields;
'ISBN/ASIN' CCK fields, attached to any other node, are also converted into D6 ASIN fields.

What isn't migrated when upgrading 'Amazontools' and 'ISBN/ASIN' field modules from D5 to D6 (this list might not be complete):

Ratings from AAT;
Blocks from AAT (e.g. "Amazon: Random review", "Amazon: We recommend", "Amazon: Random book")

Your ISBN/ASIN fields from D5 will be directly converted into D6 'Amazon' ASIN fields; data from 'amazon' and 'amazon_node' nodes and from the "Related links from Amazon" will be moved into a newly created CCK 'Amazon' ASIN field called field_legacy_asin; this field's name can not be changed.

Step by step instructions

Prerequisites:

As of this writing, this procedure requires amazon 6.x-1.x-dev
(optional) If you have a old site, e.g. where files are located at system/files, fix these old file storage paths in D5; you'll need access to your database and a tool like Search & Replace Scanner (to correct references to the old storage path(s) in your nodes); the recommended path for files in D6 is ./sites/default/files;
Make a full site backup, including database, configuration files, and "files" folder; verify that it can be restored; you can do this manually, or use a handy shell script like the Site backup script;
(highly recommended) Clone the D5 database: If your D5 site uses database_drupal, set up a second database like database_d6_drupal; you'll want to have the D5 and D6 site running in parallel to compare how everything works and looks in D6; for database setup instructions consult the file INSTALL.TXT in your Drupal site root;
(optional) Put your site in maintenance mode at ./admin/settings/site-maintenance, if you don't do a parallel setup;
(optional, but highly recommended) Switch to a core theme; 'Garland' is recommended;
(optional, but highly recommended) Disable all contributed (non-core) modules on the D5 site; don not uninstall any module you want to have on D6!

Upgrade Drupal core and a bare minimum of contributed modules:

Unpack a full D6 archive in your site's root folder;
Recursively copy the "files" folder from the D5 site to sites/default on the new site;
Run update.php to update Drupal core; this might take about an hour on bigger sites;
(optional) Update essential modules, e.g. image.module or 3rd party input formats;
Install CCK module into sites/all/modules (e.g. with Drush: drush dl cck);
Enable content.module;
Run update.php;
(optional) Enable other core modules from CCK (no other CCK modues!);
Run update.php;

Migrate the Amazontools data to Amazon module:

Install Amazon module into sites/all/modules.
Create the amazon and amazon_node content types (exports and additional instructions: see comment #62);
Enable Amazon API from amazon.module (NOT the ASIN field yet);
Run update.php; ignore errors and/or warnings during database update;
(optional) Enable the 'Amazon' sub modules 'Amazon Filter', 'Amazon media', 'Amazon search';
Run update.php;
Enable 'Amazon field' (= "ASIN field") from 'Amazon' module;
Run update.php. This may take a while as it processes nodes with attached ISBN/ASIN field (from D5); it's displaying a progress bar and the node it is currently processing; ignore warnings like user warning: Unknown column 'field_asin_asin' in 'field list' query: INSERT INTO content_field_asin (vid, nid, field_asin_asin) SELECT vid, nid, field_asin_asin FROM content_type_buch in /var/www/drupal6/sites/all/modules/cck/includes/content.admin.inc on line 1553. or user warning: Can't DROP 'field_asin_asin'; check that column/key exists query: ALTER TABLE content_type_buch DROP field_asin_asin in /var/www/drupal6/includes/database.mysql-common.inc on line 322.
Enable the "Amazon Legacy Import" module from 'Amazon' module;
Visit Admin->Settings->Amazon Legacy Importer (located at ./admin/settings/aat_legacy) to convert the old amazonnodes and amazon related links data from AAT; it'll display a screen "Migrating amazonnode field values into field_legacy_asin CCK fields" with progress bar and the node it's currently processing; this might take a long time (depending on the amount of 'Amazontools' nodes and you processor's speed, this might take even several hours);
Run update.php;
Visit your Amazon settings page at admin/settings/amazon. You'll have to change the locale, if you're not afilliated with Amazon.com ('US')
You can safely disable the "Amazon Legacy Importer" which is not needed anymore

Upgrade remaining contributed modules:

Enable other CCK modules (like date);
Run update.php;
Move all other modules back into sites/all/modules and enable them as needed; it's recommended to enable one contibuted module after the other, each followed by running update.php;
Run update.php;

Following these steps might take half a day or longer, so better do this offline. Also, this might cause heavy cpu and db load; if you're monitoring MySQL, you might encounter 1500-2000 qps or more.

Verifying the migration success

After the import has finished, you might want to verify the success of the migration. Keep in mind, that all migrated nodes are "normal" CCK-based content types with no proprietary properties (as AAT nodes) anymore. You can do anything with them what you can do with any other CCK node type:

Check you watchdog log for errors and warnings at ./admin/reports/dblog
Compare the number of 'amazon' and 'amazon_nodes' in D5 and D6; that kind of information you can get directly from querying your database (with SQL) or with the contributed >Systeminfo module; in my site, I get an exact match (e.g. 'amazon' -> 1626 nodes, 'aazon_node' -> 13782 nodes both on D5 and D6);
Filter out 'amazon' nodes at ./admin/content/node; review some nodes; check if the ASIN data has been pulled in correctly; by default, you can expect a small cover image and a linkt to Amazon's product page; for affiliates of Amazon.de (Germany) usually no pricing information is displayed, even for newly published books; also note, that the cover images by default are not hyperlinked to Amazon's product page as they were in AAT; you might want to change this in your theme's template file (would this the way to go for this?);
Filter out 'amazon_node' at ./admin/content/node; review some nodes;
(optional) Use Views Bulk Operations (VBO) to filter down your content more selectively;
Identify where CCK fields of the type 'Amazon item' are used at ./admin/content/types/fields. They're labeled either something like 'field_asin' (the field's name you provided for 'ISBN/ASIN' fields in D5), or 'field_legacy_asin' ("Related Amazon links" from AAT); the latter will include the content types 'amazon' and 'amazon_node'. Review some nodes of these content types and compare them to their D5 aequivalent.
Check the content types with attached ASIN fields at ./admin/content/node-type/{your-content-type}/fields, look out for errors and warnings; also, check the display settings at ./admin/content/node-type/{your-content-type}/display;

Polishing up

The AAT blocks can be manually rebuilt with views:

"Amazon: Random review" selects one random 'amazon' node;
"Amazon: Random book" selects one random 'amazon_node' node (not necessarily limited to books!)
"Amazon: We recommend" selects one 'amazon' node with a freely selectable minimum rating; this requires 'Amazon' module and something like 'Voting API' plus 'Fivestar' in D6

Other cleanup work after a D5->D6 migration:

Views: All views based on data from AAT will break, they're not converted/migrated automatically. Also note that D5 had Views 1, D6 has Views 2, so you'll have to rework your views anyway.
CCK: Many CCK fields behave differently on D6; you'll want to visit every content type and check the field's configuration as well as the display settings. E.g., on all my migrated sites fieldsets were excluded from display after the upgrade, so neither the fieldsets itself nor the fields contained in it showed up. Also check the weight of the newly created and imported ASIN fields, they might have a value of "10" (on the very bottom).
Theming: ASIN fields from 'Amazon' module might be formatted strangely; especially 'amazon' nodes do look very differently after the migration; fix this in your theme's CSS, if you liked the AAT's formatting better (I'll post suggestions later). Since 'field_legacy_asin' is now a multivalue field, it might be difficult to mimic AAT's (single value) appearance
Taxonomy: If your 'Amazon' and 'Amazon_node' nodes had tags, remember to visit ./admin/content/taxonomy and enable the proper categories for those content types (the old settings are not migrated automatically)
i10n/i18n: If you are using a non-English (= localized) version of Drupal, the contributed module's translations might disappear after the Upgrade. Re-importing the translation packages can be done automatically with the Localization client.

Enhance your site's structure

CCK allows you to create customized semantic content types like 'DVD', 'Movie', or 'Hardware' - e.g. for producr reviews; AAT, on the other hand, is an "old world" module that accumulated anything in one node type 'amazon'. The migration procedure documented above does not enhance your site's structure semantically, meaning: it only migrates the nodes as they were in D5 to D6.

If you want to fully enter the "new world" of CCK and fields in core (D7 and beyond), you might want to work on restrcuturing your site's content. The Node convert module allows to convert the node from it's current node type (e.g. 'amazon'), to another node type (e.g. 'movie_review'). Also it gives the ability of transfering all the custom CCK field values of the current node type to the destination CCK fields. Other tools like Nodetype (which also allows bulk operations; patch that adds support for Actions/Triggers) do not necessarily convert any CCK fields.

However, to sort a bunch of 'amazon' style nodes into distinct sets of semantic nodetypes, you'll might need something to allow you to filter out DVDs, Movies, or Hardware. A (not yet existing) action for VBO might do the trick.