How to determine reason for unprocessed files [#2476077]

Finally ran my first file migration from Drupal 6 to Drupal 7 using UI.

3100 Items
3036 Processed
62 Unprocessed
2498 messages

I am simply trying to figure out what those 62 files are, and why they did not process.

I looked at the "messages" link for the file migration task. They all say something like:

2701 Informational Deleted documents from index with query id:"pyli4t/file/3782" OR sm_parent_document_id:"pyli4t/file/3782" Don't know what to do with this, or how to query for unprocessed files.

I looked at the migrate_map_43b7eadb1file table for the migration task, but don't see anything that indicates what files were not processed.

I looked at the migrate_message_43b7eadb1file table and was able to locate the two files that failed using this SQL statement: SELECT * FROM `migrate_message_43b7eadb1file` WHERE `level` != 4 But, nothing about unprocessed files.

That's all I can think to do. Any suggestions?

Comments

Comment #1

mikeryan

he/him

English

Murphysboro, IL, USA

CreditAttribution: mikeryan at Acquia commented 24 April 2015 at 16:54

Status:

Active

» Postponed (maintainer needs more info)

Side issue first - that informational message is coming from some contrib module (something search-related, perhaps) and not related to migration per-se. Grep your codebase for "Deleted documents from index" to see what it might be.

So, the unprocessed count is precisely the count of source records, minus the number of records in the map table (so, you will never find an "unprocessed" record in the map table. All processed records are recorded in the map table with an appropriate status.

The first question is, are there really unprocessed records, or is the source count off? The most common cause of "unprocessed" records being reported is a malformed source query that has more than one row with the same primary key (fid in this case) - this can happen when joining to another table that has multiple rows per "base" row. Elsewhere you've been speaking about extending migrate_d2d classes and modifying their queries - could you have done that for the file query, and joined to another table?

The next step is to try to identify specific unprocessed files. If your D6 and D7 databases are on the same MySQL server and accessible with the same MySQL user, you can do something like:

select d6.fid 
from d6.files f
left join d7.migrate_map_myfilemigration map on f.fid=map.sourceid1
where map.sourceid1 is null

If the dbs are not on the same server, you could try dumping the D6 files table (or just the fids) and loading it into a temp table on the D7 server.

Comment #2

SomebodySysop CreditAttribution: SomebodySysop commented 25 April 2015 at 00:42

Yes, I have extended the node table query, but not the file migration query. The total number of files in the database is 3100, so the number represented in the migrate d2d UI is correct.

I checked the files table and none of the file ids are null, nor are any of the filenames null. I only found one record that had an empty filepath.

The tables are on two separate databases, but you've given me an idea. I'm going to copy the entire D6 database to the same server where the migration is running. I noticed that when migrating files (one at a time), occasionally a node would fail with this error:

Could not save to map table due to NULL value for key field nid

I could resolve the problem by re-saving the source node on the source site -- it would then migrate over the next time migrate d2d UI was executed.

This could be that's what's happening with the files: Some hiccup resulting from the fact that the source database is connected on a remote server over the Internet.

Going to make these changes, rollback and try again to see if this makes a difference. Will report back results.

Thanks!

Comment #3

SomebodySysop CreditAttribution: SomebodySysop commented 25 April 2015 at 20:33

Status:

Postponed (maintainer needs more info)

» Fixed

Thank you for the query. Using it, I was able to find the files that did not process. They were physically copied over, but not inserted into the database.

These are mostly images created using the old D6 Image module. I won't bore you with the details, but apparently there was some module mis-configuration that caused these particular images to have incorrect filenames in the D6 database, hence they could not be migrated into D7 database. Push come to shove, I can manually re-enter.

That solves this mystery.

Thank you for all your assistance.