File target: add support for local file path as source

Yes, you can just specify a file URL in your source and map that to "File ID". By default the target is configured to reference by filename.
In the beginning I found it confusing too that you need to map an url to a file ID. I had considered to change that until I noticed that file ID made more sense as that what gets actually saved. See #2867838: Change label of file target's property 'target_id' to 'File URL' as the input is expected to be an URL.

Example CSV for importing nodes (or other entities):

Just to add an information here, I add the exact same error than saranya.
My images were in /sites/default/files and the import failed (with a mapping between the image URL and the File ID)
As soon as I moved them to the root folder, in a custom folder called "import", it worked (with absolute URL http://mysite.com/import/image.jpg, not with relative URL like /import/image.jpg)
I di not check but maybe the "files" folder has specific access and Feeds can not upload from it?
Just food for thoughts ;)

The D8 version of Feeds seems to ony support download from urls, not local paths. It tries to get a file with the method \GuzzleHttp\ClientInterface::request():

protected function getContent($url) {
  $response = $this->client->request('GET', $url);

  if ($response->getStatusCode() >= 400) {
    $args = [
      '%url' => $url,
      '@code' => $response->getStatusCode(),
    ];
    throw new TargetValidationException($this->t('Download of %url failed with code @code.', $args));
  }

  return (string) $response->getBody();
}

The D7 version however did also support local paths. See FeedsEnclosure:

// Copy or save file depending on whether it is remote or local.
if (drupal_realpath($this->getSanitizedUri())) {

So let's turn this into a feature request to add support for local file paths.

I have an urgent need to be able to import from local files. I was doing this on D7 wherein I used a Feed to import into a File field from the private:// scheme. What I would do is place the file son the server and have Feeds read a Feed file and translate those into Nodes with a File Attachment with the source being a local file in the private::// scheme. I really need to be able to do this in D8.

Edit: there are 1790 files in this feed. I just wanted to note that because downloading each from a URL is not workable.

I have the need to import local files too (around 50000 files, public:// scheme). Any update on this feature request?

I highly doubt that this patch is a proper solution, but it works for me and I can import my local files (public:// scheme) with this patch. Use with caution and at your own risk.

@gkaas
Thanks for your contribution! So you basically want to preserve the file path provided by the source (when it starts with 'public://' or 'private://') and ignore the field's settings of where to save the file?
Else I think that the method getContent() should return the file contents. But preserving the file path would indeed be a lot faster as in that case the file does not need to be loaded into memory. Hm...

I see some other interesting things in the patch, like mime_content_type(). Did not know that PHP function existed. Could possibly fix issues like #2611014: Feeds doesn't import remote images which have no proper extension (such as ASP) or #2977565: Unable to download image from RSS feed when it contains a query string.

Thank you @MegaChriz,
Yes, I indeed want to preserve the file path provided by the source. I'm migrating ~50000 images from D6 to D8 and I have a seperate script that reads the image data from the D6 database, saves some metadata to the images, copies the images to the new D8 public:// folder (and keep the original directory structure) and it generates the CSV file I use as import file. Because I want to preserve the original directory structure of our image archive I want to ignore the field's setting of where the file should be saved (the images are already where they should be). Not sure if this is a common use case though.

I didn't look into the other issues you mention yet, but hopefully mime_content_type() helps fixing these issues.

There are more file target related issues. Tagging this with “Feeds file target”.

from #2 above, @MegaChriz says:

you can just specify a file URL in your source and map that to "File ID".

is this still available.

If i read this right, when i set Target to a File field, i have an option under Configure for File ID, but for this (and any other option) i get the same Target fields:
File (base64) (field_file): Base 64 string
File (base64) (field_file): Filename
File (base64) (field_file): Description

perhaps just confusing UI and i can set my path to file as any of the 3?

i found the solution somewhere; which was to remove the 2 file plugins from the feedsdev module.

I can now import files via absolute path to that file being used on another site. Guessing i am just lucky that those files were (incorrectly) stored as Public or Feeds likely wouldn't have access to them. But that's an issue for another day.

@liquidcms
Yes, in Feeds DEV I added indeed alternative plugins for file fields. And discovered you could only have one plugin per field type. There's a patch included with Feeds DEV to allow multiple Target plugins per field type (I noted this on its project page). You would need to apply it if you want to use both the default file target plugin and the alternative one. But you can indeed also remove the alternative file target plugins from Feeds DEV. :)
The reason the feature is not part of Feeds yet is because I'm not happy yet with the current implementation.

i tried patch, didn't work (i think patch failed).. so tried removing the 2 plugin files.. which did work.

@MegaChriz @liquidcms - Thanks for this discussion. I have some questions, hoping someone can help me out as I'm confused.

I have a large number of files (thousands) I'm going to be importing via Feeds and having this create new nodes and attach said files to the node via the File field. This works as expected in current D8 Feeds with this dev version.

My problem is that it copies the files from my directories and puts them all in the basic default/files folder instead of just using the existing files in their (complicated) directory structure. Is there some way to do this with a patch or code hack? I did try using the "File ID" Method mentioned above but it still just copied my files to the default/files folder. It seems Liquidcms got it to work by removing something in the module? I'm not sure where/what.

Would appreciate any tips or tricks, many thanks!

My comment was regarding how to get the Feeds import to accept a full path to a file on a different site (the D7 site i am migrating from).

In my case i am creating a different structure for where these files belong. In D7 site they were attached to a node bundle inside a Field Collection and stored in Public file space (improper design). In the D8 site we have this type of doc files attached to a different bundle and we are not planning to try to map the old D7 files to this bundle. Instead i have created a Paragraph to hold the file with some other metadata. That file field uses Filefield Paths module to locate the file in Private in a folder based on meta data imported from the D7 along with the file (basically based on the user's id number).

The Feeds import creates the "file" Paragraph.

I then wrote a small Action plugin to run as a VBO action on the new Paragraphs to attach them to the new User field.

Vrajak, #18

please have a look at file (field) paths module and see if that solves your issue?
https://www.drupal.org/project/filefield_paths

I have some questions about importing images using feeds.

My situation:
I have a ton of images that sit in a structure, such as /galleries/UNIQUEID/**filename**.JPG
They have no metadata and are attached to "parent content" identified by UNIQUEID.
I will have separately imported the "parent content", in which the UNIQUEID is available as feeds item GUID.

Pictures are currently on another server.

How should I proceed to import them with feeds?

Should I make a CSV file like this:

"UNIQUEID";"Filenames"
UNIQUEID;PATH/UNIQUEID/file-1.jpg,PATH/UNIQUEID/file-2.jpg,PATH/UNIQUEID/file-3.png,PATH/UNIQUEID/file-4.jpg

Will something like this work? (please note the use of semicolon as a separator and comma as a separator for filenames)

PATH:
are only URLS accepted?

then the processing:
will each file wander through the same filters and processes as uploaded files? i.e. being resized, getting default alt and title values ?

storage: will the import use the settings specified for the image field?
(I will use this module to use a similar file structure as before: https://www.drupal.org/project/filefield_paths)

do you have any other advice?

@manarak
Importing files in Feeds could use improvement. Right now source files need to come from an url and only certain protocols work. See #2969401-5: cURL error 1 on CSV import with image field mapped from field with url.

Filefield Paths

I believe there were some issues as well when using file import in combination with the Filefield paths module, though it's possible these issues only exist in certain cases in the D7 version of Feeds.

File target issue serries

For all file target related issues in Feeds, see https://www.drupal.org/project/issues/search?issue_tags=Feeds%20file%20t...
I hope I can address these series of issues sometime in the future. It's not that high on my priority list at the moment, sorry.

#6 @Toki - was struggling with this until I read your suggestion. Worked perfectly. Thank you.

Hi all!
#6 confirmed working - ty Toki , nattyweb !

#20 - I have used that module to great effect on my system. I am current SFTPing the files to my public scheme and using feeds to grab them and put them in a directory structure in my private scheme. But what I really want to do is read the files during feeds import from the local file system, rather than from a URL.

I am going to look through this thread carefully, but at first glance, I don't think anyone has solved this?

@mitchems
I see that there is a patch available in #10. I haven't tried that one, but it perhaps it would work in your situation?
Setting the issue status to "Needs review". Perhaps the status should be "Needs work", as we would also need to have an automated test for this. I think a Kernel test would be good.

The patch in #10 apparently no longer applies, so it at least need to be rerolled. Since I'm focussing on other Feeds issues, I haven't checked the original patch closely yet, so not sure if the implementation should also be different.

#10, which I re-rolled in merge request !28, worked for me. I needed to use the full URL as the filename (I was pointing to files stored in AWS S3FS).

Cool, @andileco
I guess the next step would be to have tests for this feature :).

Reroll for the current version of feeds.

One issue with the current patch (#32) is that local files are deleted upon import, which may not be what the user wants. I'm trying to figure it out why the files are getting deleted but I haven't identified the cause yet.

Steps to reproduce:

1. Create a CSV feed type and use an image file ID.
2. In the CSV file, use the private:// path.
3. Import the feed. The files on the local file system will be deleted.

Another issue with the current approach: if you set "Update existing media items" to "Do not update existing media items", it works fine. However if you select "Update existing media items", you will get a warning: "cURL error 1: Protocol "private" not supported or disabled in libcurl (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)"

Hi,

I have used the patch for private files folder but on save getting permission issues for image upload. By adding below code with the #32 patch it worked for me.

By applying #32 patch and my changes in single patch.

Rebased again to bring up upstream changes, so the patch can apply.

The patch is available at https://git.drupalcode.org/project/feeds/-/merge_requests/28.patch, but I'm saving a static copy here as well.

Still seems to be an issue with Drupal 10. How do I use the feeds module to import feeds with images from a ../private/ directory? I have tried private://image.jpg and I get:

cURL error 1: Protocol "private" not supported or disabled in libcurl (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for private://image.jpg

Still an issue with D10 and commerce imports
I'm working locally and views data export only gives local file paths, so I downloaded all the images and placed them in the folders that matched the csv structure. The import fails with cURL error 3 but succeeds if image field is not required and I leave it out of the import.

I guess I'll be editing my csv files or is there another way to add the local file path?

fixing some codeing standard

Thank you for a great module, which has been my go-to for years now. I am currently converting a site from D7 to D10 and really need to be able to import local files using the public:// path. Neither the patch in #45 nor the merge request will apply in Drupal 10.3.1. Any chance of an update?

@tsurvuli
Hm, perhaps if you need this quickly, could you try the patch/code applied on Feeds 8.x-3.0-beta4 instead? Biggest things that you'll miss is the possibility to skip entity validation and importing users with roles (a change in Drupal 10.2 broke role import for Feeds 8.x-3.0-beta4).

All my focus is on getting Feeds (and some modules that extend Feeds) ready for Drupal 11.

@tsurvuli please try the patch in #41. It does apply to Drupal 10.3.1

I have no idea why the patch in #45 was placed in this issue.
@rohit.rawat619 could you open up another issue and place your patch there. It is not germain to this issue.

@SocialNicheGuru
Looking at the patch in #45 now, I see it is indeed only coding standard fixes. There's no need to open a new issue for that. Except a few that are explicitly ignored, all coding standard issues have been fixed.

@SocialNicheGuru, for what it's worth I was able to successfully apply the patch in #41, but only after I removed the patch from issue 2928904, which I had already added. For some reason, if either patch is present in the composer file, the other won't apply. Right now I have 2928904's patch in my composer file. (And I have the beta4 version of Feeds.)

Since I am trying to create both a series of media importers that will import files as media, and also a node importer that will then import those local media files into node content, it would be extremely helpful if both patches were available to use at the same time. Thanks for the quick responses from you and @MegaCriz.

that's what happens when waiting for 7 years before merging patches

The status is incorrect; this is not ready for review. It still needs tests (#31) and there are the issues I pointed out in #33 and #34.

Setting the uid to 1, because I had issues with private files.

Conflicts with #2928904: Add a mapping target to media field

I'm working on moving file resolving logic out of the File target in #3565186: Add service for resolving files. This is because the Media target also needs file resolving features. (The media target is handled in #2928904: Add a mapping target to media field).

In my enthousiasm I already experimented with allowing local file paths as source. I've been testing that feature the last couple of days and I think it works quite well. Though it may not cover some edge cases reported here:

• #12 Ignore directory setting and use file at originally location
  If the file to import is for example "public://foo/text.txt", but the configured destination directory on the file target is "public://bar", FileResolver will copy the file to "public://bar/text.txt". The file on "public://foo/text.txt" is not removed. This perhaps prevents the issue reported in #33? #33 reports that files get deleted on import.
• #34 cURL error 1: Protocol "private" not supported or disabled in libcurl
  Not sure when that happens.
• #54 Issues with private files
  With testing I did notice some validation issues on private files. The workaround seems to be to give the feed owner permission to access the private file. Because during my testing so far, if the file was new, the file owner was set equal to the feed owner.
• The implementation does not use mime_content_type() and does not set "filemime". It looks like that the File entity sets this by themselves in \Drupal\file\Entity\File::preCreate().

I feel like these edge cases would be better handled in follow-ups. Let's first get absolute paths and public uri's supported! 😃

In 2968671-tests I've added test coverage for this feature, but the implementation of the feature itself is currently in #3565186: Add service for resolving files.

#3565186: Add service for resolving files has been merged, this issue stays only open for adding additional test coverage and perhaps a small fix should the tests reveal a bug.

The tests + a small addition to the file resolver service have been scheduled for merging!

Thanks to all people who contributed to a fix in the past. The actual implementation was added to the File Resolver service in #3565186: Add service for resolving files.