Using Feeds to import files with a specified path and name.

SITUATION

Spring 2025 I wanted to use Drupal 11 FEEDS to import public files from a decommissioned website to a specific public path and specific filename on a new site similar to the old site. I have a public files site folder from a no longer hosted Drupal 7 website (yes this should have been done last before Drupal 7 end of life.) I hope I have found a way that is not as elegant as I would have liked but may work for you. Yes the migration tools may be better, but we wanted to do FEEDS and have the CVS files for the content anyways. It is probable that there was a much better approach, if so please document and post.

CHALLENGES

·         I could not use feeds “FILE” processor to import files to same specified paths in the public file system but instead had to have the destination in an entity field and extend Drupal with File (Field) Paths to allow it to modify the file field path with tokens using field data from the feed.

·         My experience with Drupal FEEDS is that it works adequately importing files from a website URL and most googled articles I read have done it that way. Please document and post how if you can assure a more elegant way.

·         Some of the bodies of nodes and other places have URL links to the files hard coded, and the public file location was not sites/default/files.

APPROACH

·         Setup a separate source website to offer the files.

·         Create a CSV file with values for source url, destination path and name to be used during import with Drupal Feeds.

·         Extend the new Drupal on the destination website with the module "File (Field) Paths"  https://www.drupal.org/project/filefield_paths

·         Setup a web server as a source to host the files so that the CSV value for source URL is correct.

·         For the destination Drupal website, optionally you may want to change value of $settings['file_public_path'] to be like the old website if new site refers to old paths in links. This approach may resolve old links but create new headaches.

·         We will be using Feeds to import files into an entity so that we can declare field path. Setup a custom Media Type as the entity to be fed.

·         Setup the Structure for Feed type.

·         Add Feed for Content

·         Backup everything

·         Run Feed

·         Verify

WEBSERVER FOR SOURCE

I used the old Drupal folder [drupalroot]/sites/example.com/files and set up the drupal7 document root so that on the webserver to serve files like sites/example.com/files/testfolder/test1.png with url like http://old.example.com/sites/example.com/files/testfolder/test1.png 

Note for this tempoary website hosted with Apache software I restricted access by IP and set AllowOverride NONE with OPTIONS +Indexes -FollowSymlinks -Includes -ExecCGI

WEBSERVER FOR DESTINATION

·         The new site may want the public folders in a different location than default, sites/default/files and if so you can declare the public folder in settings.php

·         If your site has links hard coded in html you could change/move the public folder to be similar to previous website;  find . sites/default/files | cpio -pvmud sites/example.com/

·         Then change appropriate settings.php for $settings['file_public_path'] = 'sites/example.com/files';

CSV FILE FOR FEEDS

Before turning off the old site I had setup Views Data Export and made a CSV file with information about the files, so I have the path and name of the files that I can modify a copy to have information for a FEEDS csv file with the website addresses of the files, the file name and path.

If you can still access the old site with Views Data #xport you may get what you need without tampering. You may can edit the CSV with other tools for find-replace-concatenate needed values, or you can try Feeds Tamper with what you got to get to what you need for FEEDS to accomplish the tasks.

Here is an example of the CSV file to use for feeding (note description can be empty. We just need something to feed the description field for the media even if it is null.

“uuid”, “dest-name”, “dest-path”, “source-url”, “description”

"453dc7ca-8a2b-43ff-acf3-1a4352dc5407", "test1.png", "/testfolder1",”http://old.example.com/sites/example.com/files/testfolder1/test1.png”, “”

"fbbb4ff0-8f19-4b0d-a142-7d5d31503a4a" “test2.png”, “/testfolder1/testfolder2”, http://old.example.com/sites/example.com/files/testfolder1/testfolder2/test2.png”, “Test image 2”

Note I had a UUID field in my CVS that was unique, you may not have it if so maybe you can use file id, FID.  It is a GOOD thing to have a unique field and map it that way in Feeds. I really do not want duplicates of files if we import a file withs same path/name more than once, example; image.jpg image_0.jpg image_1.jpg. 

EXTEND DRUPAL

The Drupal module “File (Field) Paths” allows you to automatically sort and rename your uploaded files using token based replacement patterns. Extend Drupal with the module "File (Field) Paths"  https://www.drupal.org/project/filefield_paths

Batch restructuring of the file field path is possible by selecting additional options on the file field we will setup in the MEDIA custom type file field, but not recommended for production site. Note by default this module will not rename nor move the file path unless we enable development settings ( I did not need to do this )… The secret sauce was the File (Field) Paths module and having path prefix and file name as fields in my CSV file... and importing to something like Media that uses that module.  Just importing into files did not allow me to take advantage of that modules features.

DESTINATION ENTITY (media type)

We will use a MEDIA type to be the entity to have a file field to receive the file field from FEEDS. It would have been nice to just process to FILE type instead of MEDIA type, but to specify file name and path per item I found the need to have it be an entity. An example of setting up a custom MEDIA type:

Structure -> Media types -> Add media type

Name: FeedFiles
Description: A media type to receive files from FEEDS with added fields for destination path and name.
Media source: File
Media Source configuration, Field with source information:  - Create -
Field mapping (name/mime/size):  - Skip field -
Publishing options: Published

->Create a new field -> Plain Text
Text (plain), Label: dest-name
-> Continue
Field Storage: 255, Limited: 1
-> Save Settings

->Create a new field -> Plain Text
Text (plain), Label: dest-path
-> Continue
Field Storage: 255, Limited: 1
-> Save Settings

File (field_media_file)  -> Edit

Label: File

Upload destination: Pubic files

Allowed number of values: Limited, 1

Required field: checked

Allowed file extensions:

---

There are a lot of used extensions, https://en.wikipedia.org/wiki/List_of_filename_extensions

It is important to have all the file extensions that might be in your CVS value for file name or the item will fail to import. Results of a Google search for allow all says “You can also leave the field blank to allow all extensions, but this is generally not recommended for security reasons.” BUT I was unable to leave it blank nor use a wildcard? In this situation I think we are limited to about 255 characters for this field. Here is a list of some file extensions that covers most documents, images and videos and is less than 255 characters:

aac, ai, avi, cdr, cpt, css, csv, doc, docx, eps, gif, indd, iso, jpg, jpeg, mov, mpeg, mp3, mp4, m4a, m4v, odf, odg, odp, ods, odt, pdf, pps, ppsx, ppt, ps, png, psd, rar, raw, rtf, tar, tgz, tif, tiff, txt, wav, webp, webm, wmv, xls, xlsx, zip, 7z

“check” Enable File (Field) Paths>, expand collapsed input and change File (Field) Path settings to use tokens from the other media fields.
(Browse Available Tokens (expand))

File path: [media:field_dest_path]
   File path options (leave unchecked for our approach)

File name:  [media:field_dest_name]
   File name options (leave unchecked for our approach)

Enable Description field
( I had this unchecked but Feeds will still want to map this I think)

Click “Save Settings”

STRUCTURE FOR FEED TYPE

Note some of these settings, like language and existing may be different for you. For an example I am picking values that worked for me.

Structure -> Feed types -> Add feed type   (click)

Name: FeedFiles
Description: Import Public Files
Explanation or submission guidelines: Bulk feed import public files with destination path and name for each.

Fetcher: Upload file

Parser: CSV

Processor: Media

Media type:  FeedFiles

 ->Settings -> Import Period: Off

-> Fetcher setttings -> Allowed file extensions: txt csv
-> Fetcher setttings -> Upload directory: private://feeds

-> Parser Settings -> Default delimiter: ,

-> Processor Settings -> Language: Not applicable

-> Processor Settings -> Insert new media items: “check” Insert new media items

-> Processor Settings -> Update existing media items: “check” Update existing media items

-> Processor Settings -> Previously imported items: Keep

-> Processor Settings -> Expire media items: Never

-> Processor Settings -> “check” Owner: Feed author

-> Processor Settings -> Advanced Settings:  I left “Authorize” checked, the others unchecked.

-> Log Settings:  I did not change any settings, left it with everything checked.

“Save and add mappings” click and you will go to the next screen to add mappings.
Now we are on the “Mappings FeedFiles” page, /admin/structure/feeds/manage/feedfiles/mapping

We need to map three values in our CSV file. Note File will want two sources (id and description), so four sources will be mapped:

“-Select a target –“ and choose “File (field_media_file)”

  “-Select a source” for File Id: New CSV Column source, source-url

  “-Select a source” for File Id: New CSV Column source, description

“-Select a target –“ and choose “dest-name (field_dest_name)”

  “-Select a source” for File Id: New CSV Column source, dest-name

“-Select a target –“ and choose “dest-path (field_dest_path)”

  “-Select a source” for File Id: New CSV Column source, dest-path

You should set one of the fields as unique so that if we import two or more times we can try to avoid multiples of the file. If you don’t have a UUID but have a File ID maybe you can substitute that, fortunately I have a UUID for my files.

(OPTIONAL, map UUID field for unique)

“-Select a target –“ and choose “UUID”

  “-Select a source” for File Id: New CSV Column source, uuid

In the mapping configuration for uuid, ensure the "Unique" checkbox is selected. This tells Feeds to use this field for uniqueness checks.

Click “Save”

If your CSV file is adequate, maybe will not have to make and settings for “Tamper” ... Note if you have any fields that are not limited to one, you will probably be adding Tamper Explode plugin.

I did not need to make any changes to the other tabs; Custom sources, Manage fields, Manage form display, Manage display.

Reminder, to define a unique field in Drupal Feeds, you need to specify which field in your source data (e.g., CSV, RSS) should be used to identify and update existing entities during import. This unique field will prevent duplicates and allow Feeds to efficiently update existing records. In my case UUID, if not that maybe the file or node or entity id?

ADD FEED CONTENT

Content -> Feeds -> Add feed  (click)

Click on “FeedFiles”

Title: FeedPublicFiles

File:  Upload your CSV File  (recommend a test file with two rows.)

Import Options -> UNCHECK “Active” to disable periodic import for this feed.

Use down arrow and change “Save and import” to “Save” unless you are feeling froggy and have a good backup. Click “Save”

BACKUP EVERTHING

Sometimes mor often than not when bulk feeding it can go badly wrong. Recommend backing up the entire site, including files and database. Even if the issues can be corrected manually it can be a lot less effort to just delete, restore backup and try again with feeds.

RUN FEED

Content -> Feeds -> FileFeeds

Cross fingers and click “Import”

“Are you sure you want to import the feed FeedPublicFiles?”
Click Import

VERIFY

Hopefully you get no error messages or a few that are correctable without having to erase and replace the Drupal files and database from a backup.  Read the logs, look at the results, verify the files have the name and path they should and see if you need to try again or worse case delete site and restore backup.

Note you can look Content-> Media and review the content for the new media type.  Be sure to check the URL of the image.

OTHER THOUGHTS

Note for configuration of Media Type file fields you can change setting to rename file. Should we? I think not. I will leave that setting alone for now.

We were fortunate that each field in our CSV only has one value. Feeds will support uploading multiple images for a CSV file using the Tamper and “Explode.”  I had bad luck with using a comma for Explode because some of my fields that had muliple values had commas in content so I used ALT-31 instead, but that is another discussion.

It can be useful if your previous website was extended with UUID fields, include that in the CSV and map that in Feeds. I have tried that and it worked well enough for me.

REFERENCES and EXTRA

Having issues with some fields with multiple values having a comma and Feeds Tamper Explode plugin gets confused. Instead of ,%s I am using alt-31 as the delimiter for my Tamper Explode.  Unlikely/hopefully none of the fields that I am applying Tamper Explode have that character!

Note for Feeds Parser setting you can’t put a custom value in, only have five default delimiter choices. Gues I will stay with the comma. But I can enter a custom value for Feeds Tamper Explode plugin and I used alt-31 as the delimiter inside fields with multiple values.  I had to adjust my Views DataExport and my Feeds Processor and Tamper Explode accordingly.

https://stackoverflow.com/a/41555511

Least used delimiter character in normal text < ASCII 128

….I would choose "Unit Separator" ASCII code "US": ASCII 31 (0x1F)

In the old, old days, most things were done serially, without random access. This meant that a few control codes were embedded into ASCII.

ASCII 28 (0x1C) File Separator - Used to indicate separation between files on a data input stream.

ASCII 29 (0x1D) Group Separator - Used to indicate separation between tables on a data input stream (called groups back then).

ASCII 30 (0x1E) Record Separator - Used to indicate separation between records within a table (within a group). These roughly map to a tuple in modern nomenclature.

ASCII 31 (0x1F) Unit Separator - Used to indicate separation between units within a record. The roughly map to fields in modern nomenclature.