I have been successful in using feeds for importing multiple images into an image field.

If anyone coming from Google is reading this, the key is to prepare your CSV like this (note the commas and semicolons):

"uid";"image_url";"title";"alt"
12345;url1,url2,url3,url4;title,title,title,title;alt,alt,alt,alt
and use feeds tamper to apply "explode" on image_url, title and alt

but this is not what I want to ask here

I am now faced with the task of importing thousands of images which are currently stored in a file structure similar to this:
/gallery/UNIQUE_ID/*filename*

and I need them like this:

UNIQUEID1;*filename*,*filename*,*filename*,*filename*,*filename*,*filename**filename*,*filename*
UNIQUEID2;*filename*,*filename*,*filename*,*filename*,*filename*
...

I can easily get the list of files with their full path using for example dir /b /s > file.txt in CMD (windows) or similar on unix, but there is lots of cleanup to do. like... a bit too much

so does anyone know a way to do it quicker?

if that's impossible, I would like to know if it's possible to *append* images to existing content in an imagefield ?

using excel, I can easily produce something like the following, but so far my tests to append new images to an image field have been unsuccessful. Is there a way to do this?

UNIQUEID1;*filename*
UNIQUEID1;*filename*
UNIQUEID1;*filename*
UNIQUEID1;*filename*
UNIQUEID1;*filename*
UNIQUEID1;*filename*
UNIQUEID2;*filename*
UNIQUEID2;*filename*
UNIQUEID2;*filename*
UNIQUEID2;*filename*
UNIQUEID2;*filename*

Many thanks for any help or ideas

Comments

manarak created an issue.

megachriz’s picture

megachriz
Primary language Dutch
commented

The 'append value' feature has been requested a lot in the past, but it is something Feeds cannot do at the moment without custom code (or hacking the code).

What Feeds does for each import row, is first emptying each field being mapped to and then set the values for them. This prevents having duplicate values and allows users to import empty values.

There are a few challenges to overcome in order to support this, all related to updating existing entities.

Challenges

  1. On subsequent imports you would get duplicates
    Say you're importing fruits on a multivalue field and on import 1 these are apple, banana and orange. In the append values workflow you would get three fruits imported in one field.
    You change the source: you remove the orange and replace it with a lemon. Now on import 2 you'll get: apple, banana, orange, apple, banana, lemon. That's because apple and banana are still in the source file and get appended again.
  2. Emptying fields
    In the append values workflow this would be harder to do. As you saw in the previous example, the orange remained on the field, despite that it no longer appeared in the source file.
  3. Some fields cannot hold more than one value
    If you would have the append values workflow for every field, then on subsequent imports you'll get errors for fields that have a limit on how many values are allowed. For example: the node title field can only hold one value.

I can imagine there might be more things to overcome, but these above where the ones I could think about right now.

I've came up with some ideas to overcome these challenges but these may introduce other challenges:

  1. Clean up duplicates
    Before saving, Feeds could make sure that on multivalue fields every value may only appear once. But this would limit use cases where you actually want to have the same value appear more than once on a list. So I think this is a bad idea.
  2. Empty field only once for each entity during an import
    This idea might come a lot closer to a solution. When during the same import, Feeds comes across an entity a second time, it doesn't empty the fields again.
    Challenge in these are the following:
    • Feeds would need to keep track of entities it has seen.
    • You can only append values if they come all in one import. If you have multiple sources and need to perform multiple imports to import these, then appending values would still not work.
  3. For each field being mapped, add a config option to say if you want it to append values or not
    This could fix the issue for fields that can only contain 1 value. Maybe autohide the option for single value fields.

For D7, a module called "Feeds empty" was created to start exploring the append values workflow. I'm not sure how well it worked for D7, but it has no D8 port that I know of.

Possible workaround

In EntityProcessorBase::map(), the method clearTarget() gets called:

if ($mapping['target'] == 'feeds_item') {
  // Skip feeds item as this field gets default values before mapping.
  continue;
}

// Clear the target.
$this->clearTarget($entity, $this->feedType->getTargetPlugin($delta), $mapping['target']);

As you see in this code, clearing targets is already skipped for the feeds_item field target. So a quick fix for you for now could be to add a second condition in which clearing values is skipped. For example, if your field is called 'field_image':

if ($mapping['target'] == 'field_image') {
  // Don't empty the field 'field_image'.
  continue;
}

I haven't tested if this would work. And it can happen you run in one of the issues mentioned above (like getting duplicates for example).

I hope this helps you further.

drupaldope’s picture

drupaldope commented

Thank you for your reply MegaChriz

I feel starting to tinker with feeds behaviors opens the door to too many unpredictible results.

In cases of site migrations, I guess most people are in the same situation as me, i.e. all pictures will have t be imported only once, so finding an elegant way to generate the CSV file might be what will help most users.

If anyone has an idea or code snippet in VBS or PHP to generate the CSV file from a flat text file containing picture URLs that include the GUID, please post here!
Thank you

I will try to write something myself, but don't hold your breath ;-)

drupaldope’s picture

drupaldope commented

Nailed it!

here is a piece of PHP code that will produce the desired CSV list, adapt for your needs

$rii = new RecursiveIteratorIterator(new RecursiveDirectoryIterator('galleries'));

$files = array();
$i = 0;
$uid = 0;
$newfilename = '';

foreach ($rii as $file) {
    if ($file->isDir()){
        continue;
    }
    $filename = $file->getPathname();

//skip unwanted files and directories
if(preg_match('(Thumbs|backup)', $filename) != 1) {
    $firstslash = strpos($filename,"\\")+1;
    $secondslash = strpos($filename,"\\",$firstslash);
    $length = $secondslash-$firstslash;
    $newuid = substr($filename,$firstslash,$length);

    if ($newuid != $uid){
    	if($i>0){echo $newfilename."<br>";}
    	$newfilename = $newuid.";".$filename;
    }
    else {
    	$newfilename .= ",".$filename;
    }
	$i++;
	$uid = $newuid;
}
}
Qadees’s picture

Qadees commented

@Manarak,

Sorry to ask, how do I apply this PHP code?

drupaldope’s picture

drupaldope commented

@Qadees

I run wampserver on my Windows machine, so I have local PHP.

I put all the directories and files into a local wampserver directory, and the PHP script at the directory root, and then navigate the browser to it.

feel free to ask for more details if I was unclear

wampserver_website_webroot / galleries / unique_ID / files
wampserver_website_webroot / PHPscript

Qadees’s picture

Qadees commented

Thank fo reply Manarak,

I put all the directories and files into a local wampserver directory, and the PHP script at the directory root, and then navigate the browser to it.

Does that mean you did not use feed module to achieved the append function?

If you use feed module, do you hack the code that handles the Import?

drupaldope’s picture

drupaldope commented

@Qadees

yes, I use the feeds module, and no, I don't hack the code.

the code I provided here just helps me putting the filenames into the correct format, so that I can make the CSV file for feeds to import multiple images into one image field.

igorgoncalves’s picture

igorgoncalves commented

Just a update about this issue, that @drupaldope comment helps me solve on my case.

I have a csv like:

code_id,member_id
123,0001
124,0001
123,0002
223,0002

where code_id its a field of my taxonomy and member_id also a custom id field from our user.

so, i had to handle the csv as it come, do all the referals from code and member ids to their respective entities and save all terms to the user.

The steps was to:

  1. Create a custom module with one ExampleUserSyncSubscriber.php
  2. add my main function on FeedsEvents::INIT_IMPORT
  3. do all the entityqueries to get tids and uids without duplications
  4. group into final array all uids and theirs respective tids
  5. group all tids like "tid|tid|tid|tid"
  6. after all we ended up with a clean array with one uid by row containing all tids and the member_id value
  7. rewrite the csv file from the $feed->getSource() to add a uid column and rewrite the code_id column as now we are grouping the values with |
  8. create new Feed entity and relate uid = uid, code_id = tid, and member_id = user_member_id
  9. with tamper module explode the code_id value by "|"