Help Migrating an Associated Image Set into ImageField/FileFields

I have to do this in the next couple days too. I can give you my thinking, but would also like to hear from mike as to best practices.

I've found that there are two distinct ways you can import complex content:

1) All at once: Use the 'prepare' hooks to load additional complex content that can't be represented directly in the view (even with relationships). Examples would be user roles (more than 1 role per user id) and files (more than one fid per node), so things that have multiple values per sourceid. Since much of this complex or multiple content can be easily stored through drupal content "save" functions (node_save), this can means all you have to do is get the additional data and store it in the object correctly, a dpm() of an existing object will help you know what format the extra filed needs to be in.

Benefits:
* The full content object is saved all at once.

Drawbacks:
* no mapping data is generated for the extra complex content, making it harder to track what files made it and what didn't, adn for future reference, etc.
* you have to handcode your queries (not using views)
* it's hard to un-save/re-import, say just the files if you realize something screwed up; you'd have to fix it and then reload all the nodes.

2) Each complex content as it's own base table and view: This is my new preferred method now that mike is making the map tables very accessible, and after a whole lot of experimenting. I'll use the drupal database model as the example, but other cms probably store information about multiple fields similarly)

a) Load tables: Load your multiple field's base table into tw, here we will use the "files" table as the example. Analyze it and add nid and uid as FK (fid is already the primary key). In drupal, extra data (vid (node vid), description, and list (show it listed in node) is stored in the file_revisions table, so let's add that too, setting vid to a FK. Now, assuming you already setup a content set for nodes, you will now have a nodes_node_map table already loaded and analyzed.

b) Set relationships: Set the following relationships (left => right)
node.nid => node_node_map.nid //get the new node id's for content that has been imported.
node.vid => file_revisions.vid //get all the files for the current revision of a node
file_revisions.fid => files.fid // get the actual file data.

c) Create the files view. .. [saving for now while i test this out, more later today]

[update 4-15-09]

My attempts to create a view of files (from an old drupal database ) failed.

1) using d47_files as the base table:
I could get a d47_files.nid =>d47_node.nid relationship, but my d47_files.fid => d47_file_revisions.fid didn't work when i got to the view, probably because the d47_file_revisions.fid isn't unique.

2) So, I tried to use d47_file_revisions.fid as the base table, but not relationships to d47_files.fid or d47_node.vid would show up. here, I'm not sure why that would be, unless that since there is no unique fields in the base table, no relationships can be formed? Mike?

Well, in the two big projects we've done, two approaches:

In one project, the old CMS has tables tracking files associated with stories, much like Drupal's {files} and {upload} tables. For that one, in the prepare_node hook we grab the data from the old CMS and massage it into {files} and {upload} rows.

In another, there's no helpful metadata, just <img> and <a href> tags. I extended auditfiles to scan bodies for references, scan the files directory for the actual files (we copied all the referenced images etc. in there first), and make everything consistent ({files}, {upload}, and the references in the node bodies). I've been reluctant to commit my enhancements without the maintainer's pre-approval (both because I basically tripled the size of the module, and because it needs some work, particularly in terms of scaling), but I'll try him again...

updated #1 - mike any help on creating a view from the files base tables? I'd prefer this method, but can do the prepare_hook route I guess.

I'm not entirely clear on what you're trying to do - I take it file_revisions was an equivalent to node_revisions under D4.7? So long ago...

If the idea is to add a destination type of files, going 1-to-1 from a D4.7 files row to a D6 files row, and info for file_revisions need to be in the view, you need to reduce the view so you only get one file_revisions row for each file. In the node->node_revisions case, joining on vid accomplishes this - is there a version id field in the D4.7 files and file_revisions tables? If not, how did Drupal know what the current revision was?

FYI, Mike's enhancements to auditfiles are committed in its dev snapshot.

We don't actually give any help here for imagefield/filefield. Reopening. I will be looking into this soon. Note that imagefield has a devel_generate include which does programmatic saving of images. Lets borrow that code.

Subscribing. I will be looking into this soon, maybe I'll be able to help here too.

Note: I think i ran into some issue with the filefield api in that I couldn't set (override) all the values I wanted to...
Here is the related issue: #439952: Global user used to set the $file->uid, Looks like getting the right user was fixed, but being able to set the other properties of $file (like i think date?? (... been a while) isn't possible through the api. You'd have to override though sql if it's really important ( wasn't critical for us).

Patch attached.

It provides a new field for content sets called 'FileField: @label source file path'. This should be mapped to a column containing the path to import, for example "/tmp/uploads/photo.jpeg".

In theory, it should support default values, and multiple filefields per content type (but not multi-valued fields). I've only tested it with a single filefield per node, so reviews are appreciated.

Fixed a small bug when the sourcefile was an empty string:

this looks great. fyi, the 'destination' is no longer part of the hook name. see #552928: Strip "destination" from hook and function names

Committed, thanks!

Something really minor I found while reading the code for this patch

good catch, thanks.

Committed, thanks!

moving this to migrate_extras since that is where the filefield code is now.