Weblinks contenttype link-URL not possible to Migrate

migrate_d2d is not going to build in support for every contrib module: http://www.commitstrip.com/en/2015/04/07/just-an-exception-they-said/

It looks like the weblinks module stores its data in a custom table, rather than making use of the field API. Thus, there needs to be a field handler to expose the available columns from the table, and to write them when mapped. This should really be part of the weblinks module, but you could implement it in your own custom module (ideally, if the weblinks maintainer is not going to do it, would be if you submitted a patch to weblinks with the field handler).

Then, on the destination side, you would extend the node class to expose the source fields and add them to the node query (or, if you may have more than one weblink per row, add them to $row in prepareRow()).

It looks like the weblinks module stores its data in a custom table

Yes, that's true.

In the meantime I got a first draft of the Web Links migration code working.

But I still wonder about the following mapping warning messages:

"click_count" was used as source field in the "click_count" mapping but is not in list of source fields
"click_count" was used as destination field in "click_count" mapping but is not in list of destination fields
"last_checked" was used as source field in the "last_checked" mapping but is not in list of source fields
"last_checked" was used as destination field in "last_checked" mapping but is not in list of destination fields
"last_checked_info" was used as source field in the "last_checked_info" mapping but is not in list of source fields
"last_checked_info" was used as destination field in "last_checked_info" mapping but is not in list of destination fields
"last_click" was used as source field in the "last_click" mapping but is not in list of source fields
"last_click" was used as destination field in "last_click" mapping but is not in list of destination fields
"last_status" was used as source field in the "last_status" mapping but is not in list of source fields
"last_status" was used as destination field in "last_status" mapping but is not in list of destination fields
"metatag_abstract" was used as destination field in "" mapping but is not in list of destination fields
"metatag_author" was used as destination field in "" mapping but is not in list of destination fields
"metatag_canonical" was used as destination field in "" mapping but is not in list of destination fields
"metatag_copyright" was used as destination field in "" mapping but is not in list of destination fields
"metatag_description" was used as destination field in "" mapping but is not in list of destination fields
"metatag_generator" was used as destination field in "" mapping but is not in list of destination fields
"metatag_image_src" was used as destination field in "" mapping but is not in list of destination fields
"metatag_keywords" was used as destination field in "" mapping but is not in list of destination fields
"metatag_news_keywords" was used as destination field in "" mapping but is not in list of destination fields
"metatag_original-source" was used as destination field in "" mapping but is not in list of destination fields
"metatag_publisher" was used as destination field in "" mapping but is not in list of destination fields
"metatag_revisit-after" was used as destination field in "" mapping but is not in list of destination fields
"metatag_robots" was used as destination field in "" mapping but is not in list of destination fields
"metatag_shortlink" was used as destination field in "" mapping but is not in list of destination fields
"metatag_standout" was used as destination field in "" mapping but is not in list of destination fields
"metatag_title" was used as destination field in "" mapping but is not in list of destination fields
"reciprocal" was used as source field in the "reciprocal" mapping but is not in list of source fields
"reciprocal" was used as destination field in "reciprocal" mapping but is not in list of destination fields
"taxonomy_weblinks:additional_field" was used as destination field in "" mapping but is not in list of destination fields
"taxonomy_weblinks:weight" was used as destination field in "" mapping but is not in list of destination fields
"url" was used as source field in the "url" mapping but is not in list of source fields
"url" was used as destination field in "url" mapping but is not in list of destination fields
"urlhash" was used as source field in the "urlhash" mapping but is not in list of source fields
"urlhash" was used as destination field in "urlhash" mapping but is not in list of destination fields

Any ideas why these messages will be issued?

Bad coding practice. I didn't pass the $row by reference outside of prepareRow()

What is effectively meant by this?

Then, on the destination side, you would extend the node class to expose the source fields and add them to the node query

By extending the source query the source fields become exposed and can be mapped in the extended Node class. But what else needs to be done to make these extra fields appear in the Migrate UI Dashboard Edit Fields Mapping list?

Do you know of any examples of MigrateFieldHandler extensions supporting multiple extra fields?

On the source side, in addition to adding fields to the query, to expose them to the UI you need to add them to $this->sourceFields in your migration. E.g.

class MyCustomNodeMigration extends DrupalNode6Migration {
  public function __construct(array $arguments) {
    parent::__construct($arguments);
    $this->sourceFields += array(
      'field1' => t('My first custom field'),
      'field2' => t('My second custom field'),
    );
    ...
  }

Now, in terms of the query, are weblinks rows one per node, or can you have multiple rows for a node? If no more than one per node, you can add the join in query(). However, if there can be multiple rows, that won't work right because then the query will end up return multiple complete rows per node to migrate - instead, you'll need to add the weblinks data to $row in prepareRow().

On the destination side, I may have mislead you - I think a destination handler may be more appropriate here. See migrate's sources/destinations/path.inc for a good simple example.

On the source side, in addition to adding fields to the query

I got that part already working.

Now, in terms of the query, are weblinks rows one per node,

Web Links rows are only one per node. So far I got that working as well.

I think a destination handler may be more appropriate here.

Thanks. I will look at your proposed example.

When I understand you right a destination handler exposes extra fields from a custom table to Migrate? And to make a destination handler available for migration purposes I have to create a weblinks.migrate.inc file including hook_migrate_api and the destination handler code itself. Correct?

Correct - the destination handler exposes any custom Drupal-side fields to the UI, and needs to be registered in hook_migrate_api in the migrate.inc file. The handler code itself doesn't need to be in the migrate.inc, but it does make it simpler to have all the migration code together in one file if there isn't too much to it (the handler class shouldn't be very large).

Thanks for your feedback.

In the meanwhile I got the destination handler working and the migration of Web Links works so far.

But there is still another issue: associated taxonomy vocabularies with several terms per node (free tagging): for these vocabularies only one term gets migrated. All other terms will be ignored. I thought Migrate and/or migrate_d2d takes of this case as well.

What do you propose? Do I have to implement a hook_migrate_prepare_node function to catch these vocabularies and migrate all the used terms there?

Martijn, thanks for keeping us posted. But no worries, I haven't spent any time for investigations yet.

Are you all set with the weblinks migration? Can we close this?

Thanks.