Just playing with these hooks for the first time in Drupal 6 and just found an interesting limitation, you can not extend the existing values. I'm returning some values to append to the data array, and by doing so, I nuked the existing values loaded by FileField.
The quick workaround was to change my implementation of the hook_file_load() slightly, then modify the other code to move this into the FileField data array to integrate with FileField UI Extras, [I'm getting too many additional fields per image but the workflow is so easy to use!]
$file->data['filefield_terms'] = filefield_terms_get_terms($file);
$file->filefield_terms = filefield_terms_get_terms($file);
Would it be worth doing an array_merge_recursive() instead of just array_merge() for all merges when loading this data? If so, I guess this should be lodged against Drupal 7 core, but I haven't looked closely at that code yet.