Provide an option to delete the uploaded file after import is done

Here is a patch (for the latest 7.x-2.x-dev, as well as for 7.x-2.0-alpha5) which provides an option to delete the file immediately.

One issue with the patch, though, is that we can't really guarantee the deletion will happen. For example, if the batch import gets interrupted midway, the code for deleting the file will never run, so the file will remain in place until another import is run. This could be an issue if you're relying on this to avoid having sensitive/private data (which may be present in the import file) publicly accessible.

Semi-related issue: #1147734: Import files in public://feeds/ vs. private://feeds/

A caveat about this patch is that it will not work well (will not actually delete the file) if you are using something like http://drupal.org/project/file_lock on your site. You'd need to figure out how to configure that module to specifically exclude the uploaded feeds import files.

Personally, I'm moving on to #1147734: Import files in public://feeds/ vs. private://feeds/ for my use case instead... but I think the patch here could still be useful for certain situations.

See #955236: Handle fatal errors for entity create with better error handler function.

The option to delete import files after update would be essential in order to avoid processing existing files (e.g. CSVs) in a directory all over again.

I gave it a try and Feeds will try to re-import the file(s) again. If the Importer is configured to update existing entities and the entity has been manually updated in the mean time (hash has changed) then the Importer will override these changes again.

Files should be marked as processed or be deleted in order to avoid this.

Any plans to enable the "delete after import" option in the file fetcher?

Does this also handle multiple files, e.g. if a directory was read?

I'm attaching a rerolled patch, so it should apply cleanly against the latest 7.x-2.x. I haven't yet made any changes or updates, beyond just rerolling it.

This works OK for me if the file is uploaded via a form.

If the FileFetcher gets the files from a given path though, the files are not deleted ($source_config['fid'] == 0).

I added a check for when multiple files are imported (e.g. they are uploaded via FTP), however my solution is not very robust.

Imho this fix should also be able to remember a list of csvs that are being imported and delete them when finished (not only the single file case).

#10 Worked for me nicely when using the stand alone form, thanks!

This is very important because sometimes you have sensitive data to import and you don't want the files hanging around on the server.

I'm using the latest dev, and experienced the following:

I uploaded a CSV (using standalone form) and it imported fine. I realised there was a mistake with some of the data, so I used the 'Delete items', which worked fine.

The CSV file was still listed on the import page. I clicked browse, chose the same (now changed file), but when I clicked Import I got told that it couldn't be uploaded and it needed to be a .csv extension.

There is no way to delete that file. I renamed my file and tried again and then it worked, and I noticed that it also deleted the old file.

So basically, this is a bug. You cannot upload a file with the same name, and there is no option to delete that file. The only thing you can do is to rename the file every time you import.

This patch applies cleanly to the latest dev release as of January 28, 2016

Patch in #16 works wonderfully.

I noticed an issue when the importer is configured to be ran in the background and when periodic import is enabled. When a second import is ran using cron the following PHP notice was logged:

Notice: Undefined index: source in FeedsFileFetcher->fetch() (regel 68 van /Websites/drupal/drupalsites/drupal7/sites/all/modules/wip/feeds7/feeds/plugins/FeedsFileFetcher.inc).

I've also worked on tests for this issue. The tests currently fail on the error noted above. The tests cover the following cases:

• Import a file using the UI and import in one run.
• Import a file in the background and needing two cron runs to complete.

Setting status to "Needs review" so the testbot runs the tests.

I fixed the reported PHP notice by setting the source configuration to the defaults instead of an empty array in FeedsFileFetcher::afterImport().

As marktheshark reports in #11, the setting has no effect when used in combination with the option "Supply path to file or directory directly". Therefore, I propose to disable the "delete_uploaded_file" option when that other option is checked. See attached patch.

I also noticed an issue when using the Feeds Preview module: the file did not get deleted when first previewing it. But that's a Feeds Preview issue, so no blocker for this issue.

Opinions?

I'd agree that Feeds Preview should take care of that edge case.

Committed #21. Thanks all!

Note: I will create the Feeds Preview module later.

For the Feeds Import Preview problem, I have opened a new issue with a patch: #2770771: Previewing uploaded file causes it to be saved as temporary when using the file fetcher

I seem to have a problem with the committed version (b35a9fe).

When I do a

$importer->fetcher->setConfig($feeds_importer->config['fetcher']['config']);

with

'config' => array(
    'allowed_extensions' => 'txt csv tsv xml opml',
    'delete_uploaded_file' => TRUE,
    'direct' => FALSE,
    'directory' => 'public://feeds',
    'allowed_schemes' => array(
        'public' => 'public',
    ),
),

delete_uploaded_file never gets set in the FeedsFileFetcher Object. I'm not quite sure why.

I set the default to TRUE for now.