By building upon the entity API we could add a generic entity processor. In particular that would be useful for entity types being created via the entity API itself, e.g. profiles, messages or field-collections. However, it could also serve as an alternative implementation for the core-entities.

Related to #1005128: Rules integration & enable modules to customize imports, one could use Rules for creating the entities instead of directly the entity API. So we could benefit from the Rules form elements it implements for the entity property info data types, thus we get proper form elements for inputting dates, formatted text, entity references, .. In addition we would could benefit from Rules' input evaluation system and get stuff like token replacements for free. For that we could just re-use the 'entity_create' action + the form Rules provides for it.

Anyway, here is a first patch attached which just uses the entity API and implements simple very simple form elements. It should basically work, I've not really tested it though.

This is related but not identical to #1257314: Ability to attach a feeds source to any entity, not only nodes.

Files: 
CommentFileSizeAuthor
#244 feeds-entity_processor_fix_entity_property_name_error-1033202-235.patch16.53 KBjamesdixon
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity_processor_fix_entity_property_name_error-1033202-235.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#242 feeds-entity_processor_fix_required_mapping_fields-1033202-234.patch16.45 KBjamesdixon
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es).
[ View ]
#233 feeds_entity_processor-1033202-229.patch8.33 KBjamesdixon
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es).
[ View ]
#228 feeds_entity_processor-1033202-93.patch8.11 KBjamesdixon
PASSED: [[SimpleTest]]: [MySQL] 4,505 pass(es).
[ View ]
#217 feeds_entity_processor-1033202-217.patch16.03 KBkreynen
PASSED: [[SimpleTest]]: [MySQL] 4,507 pass(es).
[ View ]
#208 Phone Number Importer.png108.12 KBthat0n3guy
#207 feeds_test.xml_.zip933 byteskyleoliveira
#203 Screen Shot 2013-07-24 at 6.42.06 AM.png217.71 KBkreynen
#203 feeds_entity_processor-1033202-203.patch15.9 KBkreynen
PASSED: [[SimpleTest]]: [MySQL] 4,507 pass(es).
[ View ]
#200 feeds_entity_processor-1033202-196-200.diff.patch1.52 KBj0rd
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_entity_processor-1033202-196-200.diff.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#200 feeds_entity_processor-1033202-200.patch15.13 KBj0rd
PASSED: [[SimpleTest]]: [MySQL] 4,505 pass(es).
[ View ]
#196 feeds_entity_processor-1033202-196.patch15.27 KBguillaumev
PASSED: [[SimpleTest]]: [MySQL] 4,505 pass(es).
[ View ]
#194 feeds-missing-arg-1033202-193.patch2.95 KBjames.williams
PASSED: [[SimpleTest]]: [MySQL] 4,353 pass(es).
[ View ]
#192 feeds-existingid-1033202-192.patch1.71 KBjames.williams
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-existingid-1033202-192.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#182 feeds-1033202-124-fix.patch1.95 KBSteven Jones
PASSED: [[SimpleTest]]: [MySQL] 3,919 pass(es).
[ View ]
#164 feeds-entity-warning-1033202-164.patch546 bytesvordude
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-warning-1033202-164.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#163 feeds_ui-genericentity-configfix-1033202-163.patch1011 bytesj0rd
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_ui-genericentity-configfix-1033202-163.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#148 Screen Shot 2013-01-23 at 5.24.26 PM.png62.24 KBcoreycondardo
#146 feeds-entity-processor-1033202-146.patch15.62 KBmErilainen
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-146.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#141 error.txt35.79 KBgmario
#141 Schermata del 2012-12-17 18:46:59.png181.69 KBgmario
#139 feeds-entity-processor-1033202-139.patch4.02 KBgmario
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-139.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#129 feeds-entity-processor-1033202-129_0.patch11.39 KBrickmanelius
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-129_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#124 feeds-entity-processor-1033202-124.patch11.31 KBimclean
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-124_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#118 feeds-entity-processor-1033202-118.patch7.72 KBspotzero
PASSED: [[SimpleTest]]: [MySQL] 3,847 pass(es).
[ View ]
#116 feeds-entity-processor-1033202-116.patch15.63 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-116.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#110 feeds-entity-processor-1033202-111.patch15.69 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-111.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#109 feeds-entity-processor-1033202-110.patch9.11 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-110.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#107 1033202-107-feeds-generic_entity_processor.patch18.56 KBjlyon
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-107-feeds-generic_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#106 feeds.jpg52.15 KBbpmbriguy
#102 FeedsGenericEntityProcessor.png28.09 KBndf
#101 FeedsGenericEntityProcessor.png28.09 KBndf
#98 1033202-98-feeds-generic_entity_processor.patch9.04 KBgilgabar
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-98-feeds-generic_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#69 1033202-69-feeds-generic_entity_processor.patch7.96 KBelliotttf
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-69-feeds-generic_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#61 feeds-ui-admin-allow-single-class-supporting-multiple-entities.patch1.18 KBmukesh.agarwal17
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-ui-admin-allow-single-class-supporting-multiple-entities.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#60 1033202-48-generic-entity-feeds_drush-make-alpha4.patch8.35 KBpatcon
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-48-generic-entity-feeds_drush-make-alpha4.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#48 1033202-generic-entity-feeds.patch8.38 KBwesnick
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-generic-entity-feeds.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#41 1033202-entity_processor.patch8.36 KBBevan
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#22 feeds.1033202-22.patch18.33 KBjamsilver
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-22.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#22 feeds.1033202-22-no-prefix.patch18.29 KBjamsilver
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-22-no-prefix.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#20 feeds.1033202-20.patch8.8 KBjamsilver
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-20.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#20 feeds.1033202-20-no-prefix.patch8.77 KBjamsilver
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-20-no-prefix.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#19 feeds.1033202-19.patch8.78 KBjamsilver
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-19.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#19 feeds.1033202-19-no-prefix.patch8.75 KBjamsilver
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-19-no-prefix.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#18 feeds_1033202-18_add_generic_entity_processor.patch7.53 KBjamsilver
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_1033202-18_add_generic_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#5 feeds.1033202-5.patch7.94 KBGrayside
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-5.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#3 feeds_entity_processor_1.patch7.27 KBdasjo
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_entity_processor_1.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#1 feeds_entity_processor.patch6.49 KBfago
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_entity_processor_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
feeds_entity_processor.patch6.47 KBfago
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Comments

fago’s picture

StatusFileSize
new6.49 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_entity_processor_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

ops, used the wrong creation function.

Grayside’s picture

Status:Needs review» Needs work

I tried configuring a Message importer, but when I attempt to save the processor to use the Message Processor, I get a couple errors.

Missing Feeds plugin FeedsEntityProcessorMessage. See message. Check whether all required libraries and modules are installed properly.

Notice: Undefined index: FeedsMissingPlugin_feeds_form in drupal_retrieve_form() (line 736 of /home/abr/workspace/d7/includes/form.inc).
Warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'FeedsMissingPlugin_feeds_form' was given in drupal_retrieve_form() (line 771 of /home/grayside/workspace/d7/includes/form.inc).

Three of the first, followed by confirmation that I changed the processor plugin. Then the actual error. On page refresh, no processor is actually selected.

dasjo’s picture

Status:Needs work» Needs review
StatusFileSize
new7.27 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_entity_processor_1.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

grayside, i ran into your problem once as well but using current versions it works.
also there was a problem with the feeds_ui distinguishing plugins by class, which is fixed in the attached patch

Steven Jones’s picture

Subscribe

Grayside’s picture

Status:Needs review» Needs work
StatusFileSize
new7.94 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-5.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Okay, tweaked a comment and made entity input format configuration reference the current entity, instead of sticking with the default "nodes" from FeedsProcessor.inc.

Continuing to test with the Message module, when I reached the mapping phase, I discovered that this integration does not identify the difference between entity properties and entity property bundles (not up on the correct terminology yet, sorry!) That's to say, Message has an "Arguments" property which can be a fairly arbitrary set of values, but the Mapper had no insight into this. In Message's case, this might require some smarter mappers, I explain it here for completeness' sake.

I tried jury-rigging a mapping anyway, and attempted to import a feed. The error that happened shortly into the batch process was massive, including gigantic entity definition arrays of doom--the piece actually explaining what went wrong here:

Fatal error: Nesting level too deep - recursive dependency? in /home/abr/workspace/d7/sites/message.d7/modules/feeds/plugins/FeedsProcessor.inc on line 148 Call Stack #TimeMemoryFunctionLocation 10.000157076{main}( )../index.php:0 20.04052213208menu_execute_active_handler( )../index.php:22 30.04102259820call_user_func_array ( )../menu.inc:501 40.04102260144system_batch_page( )../menu.inc:0 50.04102260144_batch_page( )../system.admin.inc:2282 60.04112261124_batch_do( )../batch.inc:81 70.04112261124_batch_process( )../batch.inc:162 80.04222286964call_user_func_array ( )../batch.inc:285 90.04222287160feeds_batch( )../batch.inc:0 100.05333270400FeedsSource->import( )../feeds.module:150 111.18873642488FeedsProcessor->process( )../FeedsSource.inc:350 121.29584036432var_export ( )../FeedsProcessor.inc:148

Deciding to switch the experiment to a more stable use case, I tested my system with a standard Node Processor, then the Entity Node Processor. Standard Node Processor worked fine, the Entity Node Processor ran into some problems.

Indicated as successes:

Feed Node Github has been created.
Failed importing 31 node

Indicated as warnings:

Warning message
Unknown data property feeds_item.

<repeated what I assume is 31 times>

Indicated as errors:

SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1

@dasjo: What type of entity did you use?

dasjo’s picture

grayside, i will see if i can test into this upcoming weeks but can't promise.

generally, feeds seems to get a push again due to the OA/MN migration which sounds great!

jucallme’s picture

Keeping an eye on this

e2thex’s picture

Assigned:Unassigned» e2thex
febbraro’s picture

sub

e2thex’s picture

First of all I really like this path for the processors, and if we can get this working I think the current processors might go away (are atleast go away on the back end).

I have jumped in testing with a basic cvs file that I want to move into nodes

I am getting the same error as in #5
Warning message
Unknown data property feeds_item.
SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1

And I think the issue is that in FeedsProcessor newItemInfo there is the line

<?php
$entity
->feeds_item = new stdClass();
?>

but entity is a EntityDrupalWrapper instance, and when calling a property, it is using its __get method, which is trying to build a new EntityDrupalWrapper. (__get is part of the parent EntityStructureWrapper)

Looks like there might be something in EntityStructureWrapper::__get that lets you by pass this for dpm to work, I wonder if that could be more general (maybe a way to create a property besides just declaring it?)

fago’s picture

>First of all I really like this path for the processors, and if we can get this working I think the current processors might go away (are atleast go away on the back end).

Great, so let's make it work again. :)

>but entity is a EntityDrupalWrapper instance, and when calling a property, it is using its __get method, which is trying to build a new EntityDrupalWrapper. (__get is part of the parent EntityStructureWrapper)

hm, $entity shouldn't be a wrapper but a usual entity object. If it is, this certainly causes troubles. The wrapper allows you only to set properties which are declared accordingly in the entity's property info.

e2thex’s picture

Here is where we build the new entity (from the patch)

<?php
 
/**
   * Creates a new entity in memory and returns it.
   */
 
protected function newEntity(FeedsSource $source) {
    return
entity_property_values_create_entity($this->entityType(), $this->config['values']);
  }
?>

looks like entity_property_values_create_entity returns a EntityDrupalWrapper (from entity.property.inc)

<?php
/*
 * ....
 * @return EntityDrupalWrapper
 *   An EntityDrupalWrapper wrapping the newly created entity or FALSE, if
 *   there were no information how to create the entity.
 */
function entity_property_values_create_entity($entity_type, $values = array()) {
?>
fago’s picture

oh, good catch. We should use

entity_property_values_create_entity($this->entityType(), $this->config['values'])->value()

then.

kapayne’s picture

sub

Shadlington’s picture

Subbing

7wonders’s picture

sub

Peter Törnstrand’s picture

Subscribing.

jamsilver’s picture

StatusFileSize
new7.53 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_1033202-18_add_generic_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

OK - new patch incorporating the suggestion in #13 and adding two new lines

+++ feeds.plugins.inc 20 Feb 2011 07:26:30 -0000
@@ -130,6 +130,29 @@ function _feeds_feeds_plugins() {
+          'description' => 'Create and update ' . $entity_info['label'] . 's.',
+          'help' => 'Create and update ' . $entity_info['label'] . 's from parsed content.',
+          'plugin_key' => 'FeedsEntityProcessor',
+          'handler' => array(

Adding 'plugin_key' entry - needed to make ctools plugin work where key is different to handler file.

+++ plugins/FeedsEntityProcessor.inc 20 Feb 2011 07:26:31 -0000
@@ -0,0 +1,157 @@
+  protected function newEntity(FeedsSource $source) {
+    $entity = entity_property_values_create_entity($this->entityType(), $this->config['values'])->value();
+    $entity->language = LANGUAGE_NONE;
+    return $entity;
+  }

There is an issue with finding a general way of filling all default properties of an entity when creating it. Not sure where the good solution is for this problem, but adding this language line was needed in my use case, and is probably a common one.

Powered by Dreditor.

jamsilver’s picture

StatusFileSize
new8.75 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-19-no-prefix.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
new8.78 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-19.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

More work on it. Added hook_entity_insert/update/delete so feeds keeps track of the entities its imported.

jamsilver’s picture

StatusFileSize
new8.77 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-20-no-prefix.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
new8.8 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-20.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Again more work: Fixing bug with call to entity_delete_multiple.

Grayside’s picture

Generally a good idea to double check with the assigned developer before doing work on the issue. It's probably safe to assign it to yourself if you want to keep pushing a little more, since @e2thex seems to be radio silent here.

jamsilver’s picture

StatusFileSize
new18.29 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-22-no-prefix.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
new18.33 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.1033202-22.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Bit more work - getting updates to work on existing entities, adding GUID field.

I'm just doing this because I really need to get it working for my own needs asap, and I thought that others may find it useful to find a working patch in this queue. In the long run e2thex is welcome to use as much or as little as my work as he wants.

podarok’s picture

subscribe

muschpusch’s picture

Tried the patch from #22 and tried to import profile fields (profile2) from CSV:

PHP Fatal error: Call to undefined method FeedsMissingPlugin::getMappings() in [..]modules/feeds/feeds_ui/feeds_ui.admin.inc on line 488

I disabled profile2 and switched to profile and chose "Mapping for Entity processor User" but still no new fields in the mapping section :(

wolan’s picture

Also tried the patch #22 but am getting an error when importing from csv to profile2 entity:

Invalid data value given. Be sure it matches the required data type and format.
Status message Failed importing 2 profile
Error message SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 1

Anyone else experiencing this behavior?

/wolan

inversed’s picture

subscribe

muschpusch’s picture

It actually works with d7 Core user account fields.

scotthooker’s picture

I am getting the same error "PHP Fatal error: Call to undefined method FeedsMissingPlugin::getMappings() in [..]modules/feeds/feeds_ui/feeds_ui.admin.inc on line 488" when attempting to make a feed with an entity processor for node.

Niklas Fiekas’s picture

Intresting. Subscribe.

Dave Reid’s picture

Some of this has already been committed (the hook_entity_insert/update/delete implementations were committed in another issue).

Niklas Fiekas’s picture

Niklas Fiekas’s picture

This could also solve #1041804: Feeds and Media Module: A Drupal 7 Media Utopia?, couldn't it? (Media now only requires files to be imported, no specific Media stuff.)

webankit’s picture

+1

elly’s picture

Subscribey!

BarisW’s picture

Subscribing.

Niklas Fiekas’s picture

How is it going? (Is there a patch of the current state against Feed's HEAD?) How can I help?

hellomobe’s picture

Applied patch #22 - get the following error
EntityMetadataWrapperException: There is no property for this entity. in EntityStructureWrapper->get() (line 381 of ...sites\all\modules\entity\includes\entity.wrapper.inc).

To recreate: I went to feed-importers ->add new -> processor -> selected entity field-collection

torgosPizza’s picture

Sub.

thepak’s picture

sub

Bevan’s picture

The patch needs re-rolling. The patch from #22 does not apply to 7.x-2.0-alpha4 (which is same as 7.x-2.x-dev at the time of writing):

patching file feeds.1033202-20-no-prefix.patch
patching file feeds.module
Hunk #1 FAILED at 572.
1 out of 1 hunk FAILED -- saving rejects to file feeds.module.rej
patching file feeds.plugins.inc
Hunk #1 succeeded at 129 (offset -1 lines).
patching file feeds_ui/feeds_ui.admin.inc
Hunk #1 succeeded at 420 (offset 4 lines).
Hunk #2 succeeded at 429 with fuzz 2 (offset 4 lines).
patching file plugins/FeedsEntityProcessor.inc

Removing assignee (e2thex) since (s)he only touched this issue twice and that was four months ago, on March 15 and 17. And I may re-roll the patch in the coming days or weeks (but do not count on it).

Bevan’s picture

Status:Needs work» Needs review
StatusFileSize
new8.36 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

#1066806: Use hook_entity_insert/update/delete rather than type-specific hooks added hook_entity_insert/update/delete since comment #22. (The patch-conflict was something else in the same file, feeds.module.) #1066806 (da8d974) excluded this from each implementation:

<?php
if (!in_array ($type, array ('node', 'user', 'taxonomy_term'))) {
  ...
}
?>

The attached patch is like the patch from comment #22 but with the following changes;

  • hook_entity_insert/update/delete are not added to feeds.module, since they are already there
  • // $Id$ is removed from plugins/FeedsEntityProcessor.inc, since CVS tags are deprecated
  • feeds.1033202-20-no-prefix.patch is excluded from the patch file, since it is not required and patch files in patch files is very confusing! :)

It is a git patch and thus applies with patch -p1.

I do not know how to test this, since my interest is in #1063434: Add Feeds integration to FieldCollection. And that appears to require further work beyond the scope of this issue and patch. (If someone could confirm, that would be great.)

philipz’s picture

I'm trying patch from #41 on Profile2 profile entities and after some trials of mapping / settings configuration it finally starts importing profile's fields.
The problem is Entity processor settings form where required fields of Type and User show up.
In order to import any fields to a profile I had to set up the User field to uid of one of existing user accounts.
This of course doesn't make sense because ideally I would set that uid binding in Mapping form taking the uid's from my csv file.
Using uids here also isn't perfect solution as I would prefer to use email or username field to bind imported profiles to user accounts...

I'm going to try and use this general entity processor to write my own only for Profile2 profiles entities. Then hopefully I will understand it better and maybe figure out how this could be done with general approach (or that it's not possible).

Is someone trying simillar thing ?

Grayside’s picture

Status:Needs review» Needs work

Tried to apply the patch, needs a reroll.

jyee’s picture

FWIW, the patch in #41 was generated from the drupal root (named "app") and not the feeds module root. This means that if you run patch -p1 from the feed module root (as is the Drupal standard), it will install a plugin in your feeds directory at feeds/app/sites/all/modules/contrib/feeds/plugins. You will need to move the FeedsEntityProcessor.inc file from that directory back to your feeds/plugins directory, otherwise the include will not be found and you will receive errors such as: "Missing Feeds plugin FeedsEntityProcessorProfile2. See profile_importer. Check whether all required libraries and modules are installed properly."

tim.plunkett’s picture

Bevan’s picture

@jyee: Ooops! My fault. patch -p7 < 1033202-entity_processor.patch from the feeds module directory should workaround that issue until someone rerolls the patch.

phquelen’s picture

sub

wesnick’s picture

StatusFileSize
new8.38 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-generic-entity-feeds.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

I have rerolled this patch in the standard fashion, I have been using it and it seems to work fairly well. It is worth noting that you must define your entities' metadata properties and their setters to have any success with this.

dasjo’s picture

thanks wesnick

wesnick’s picture

Status:Needs work» Needs review

I have been using this patch for a few weeks and it works well. Marking "needs review"

webadpro’s picture

wesnick, Do all I need is your patch or is there another patch I must also apply to get this working, because after applying your patch I get all kinds of errors.

eg.

Missing Feeds plugin FeedsEntityProcessorUser.

Thanks

Mohammed J. Razem’s picture

subscribing

wusel’s picture

I have patched Feeds from #48.

Then I created a new Feeds-importer with 'Entity processor Profil' to import the Profile2-fields ("complete_name","birthday","tel_1") from the CSV-file:

"member_nr","email","username","password","complete_name","birthday","tel_1"
"1001","new.tester@example.com","new tester","test","New Tester","1999-07-13","00000 12345-213"
"1002","old.tester@example.com","old tester","test","Old Tester","1945-07-12","00000/54321-0"
"1003","another.tester@example.com","another tester","test","Another Tester","1985-07-11","00000 678901"

What shall I put in the field 'User' on the page 'Settings for Entity processor Profil' to connect the new Profille-fields to the existing users?
I don't know, what "The owner of the profile. Expected data type: user. Just enter the identifier of the entity, i.e. User ID" means in this case. I assume, that the CSV-file contains no field "UID"! The field "member_nr" is a historical member-ID and not the UID.

Please help me.
Thanks in advance

wesnick’s picture

@webadpro: Make sure to clear your caches; since the classes are generated dynamically, you need to make sure that entity info is recached and all class definitions are added to the registry. Sometimes it requires two cache clears to fully refresh cached data correctly.

@wusel: The settings page is for you to add default values for required entity properties. It doesn't matter what you put here, just as long as it is the correct datatype. If your CSV does not have the values that correspond to these required fields, then the created/updated entity will get these default values. Best case, just ensure that your CSV has the appropriate values. If you want to match up existing users with values on a CSV you need to have a uid field that maps to the User's uid, and be sure to add it to the mappers section. If they are new users, then you do not need this. Once feed creates entities from a feed, this particular entity will exist in the feeds_item table so any future imports/updates will update the previously created entity with the fields from the CSV.

TimelessDomain’s picture

The generic entity processor sure is pulling in a lot more targets. Very nice. I haven't been able to test all of them, and am unsure of the format for some targets. I will post back my results. We need to compile a list of what generic entities this opens up for feeds, so that we can keep on top of each one. I started a list of them below, but did not add the targets within the entities yet. We need a documentation page for this.

Would it be possible to have a generic sub-entity processor (creating new entities from a field that is embedded within another entity)? At the beginning we could just do 1 level (which is what media feeds is doing http://drupal.org/project/media_feeds ) then eventually we could go deeper.
I don't think that Media Feeds allows the ability to fill in additional fields of the sub/child-entity, but this function would be ideal.
This is definitely where feed processing is going, so why not leave the design open to be adapted to this or get it done now to give our community a headstart on the rest. thanks

Entity processor Node
Create and update Nodes from parsed content.

Entity processor Order                                          (Ubercart)
Create and update Orders from parsed content.

Entity processor Profile
Create and update Profiles from parsed content.

Entity processor Profile type
Create and update Profile types from parsed content.

Entity processor Rules configuration
Create and update Rules configurations from parsed content.

Entity processor Taxonomy term
Create and update Taxonomy terms from parsed content.

Entity processor Taxonomy vocabulary
Create and update Taxonomy vocabularys from parsed content.

Entity processor User
Create and update Users from parsed content.

Entity processor Wysiwyg profile
Create and update Wysiwyg profiles from parsed content.

Grayside’s picture

Hi Timeless,

I don't think Feeds should keep track of individual entity types, except perhaps ones in Core. Feeds should concentrate on providing generic support for Entities, and other modules should do any further finessing of the integration.

Recursing entity processing seems reasonable, but I think it should be addressed in a followon issue dependent on this one.

andreij’s picture

Same error of #37 EntityMetadataWrapperException: There is no property for this entity. in EntityStructureWrapper->get() (line 408 di /modules/entity/includes/entity.wrapper.inc)

Debugging the code the get function in entity.wrapper.inc is called with a null parameter. Where should I look to try correct this bug?

andreij’s picture

The problem arises from this line:

$id_info = $wrapper->$name->get($info['entity keys']['id'])->info();

the get function throws:

EntityMetadataWrapperException: There is no property for this entity. in EntityStructureWrapper->get() (line 402 of /opt/local/apache2/htdocs/soraimar/sites/all/modules/entity/includes/entity.wrapper.inc).

$info['entity keys']['id'] is in fact null, what should it be?

patcon’s picture

Was looking into what had happened to the FeedsDataProcessor in plugin in D7, and stumbled across this train of threads -- REALLY exciting stuff guys :)

patcon’s picture

StatusFileSize
new8.35 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-48-generic-entity-feeds_drush-make-alpha4.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

In case anyone finds it helpful, here's th patch from 48 rerolled for alpha4 which will work with drush_make

mukesh.agarwal17’s picture

StatusFileSize
new1.18 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-ui-admin-allow-single-class-supporting-multiple-entities.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Great work already. I went through the patches and I note a common thing, i.e. we are trying to use a single processor class for all our entities. The only issue with that is the feeds_ui form for processors. Using any of the above patches, create your custom import and go to the list of processors, select your "Entity processor MY_ENTITY" and save, it will not show the same as default value after save. The reason is the following line (in function feeds_ui_plugin_form in feeds_ui_admin.inc):

<?php
 
foreach ($plugins as $key => $plugin) {
   
$form['plugin_key'][$key] = array(
     
'#type' => 'radio',
     
'#parents' => array('plugin_key'),
     
'#title' => check_plain($plugin['name']),
     
'#description' => filter_xss(isset($plugin['help']) ? $plugin['help'] : $plugin['description']),
     
'#return_value' => $key,
     
'#default_value' => ($plugin['handler']['class'] == get_class($importer->$type)) ? $key : '',
    );
  }
?>

The default value sets the value from handler's class which is the same FeedsEntityProcessor for all the entities. Attached is the patch which works for me.

Also, I think entity import should be a separate module altogether. It is not a generic requirement and should stay out of feeds.

mukesh.agarwal17’s picture

Also, not every entity needs properties and fields, they could exist as data (which might get used in some computations not necessarily related to display). All of the above patches assume that only the properties need to be imported. And therefore the line:

<?php
    $wrapper
= entity_metadata_wrapper($this->entityType(), NULL, $info);
    foreach (
$wrapper->getPropertyInfo() as $name => $property) {
      if (!empty(
$property['setter callback'])) {
       
$targets[$name] = array(
         
'name' => $property['label'],
         
'description' => isset($property['description']) ? $property['description'] : NULL,
        );
      }
    }
?>

in getMappingTargets function. I'm currently working on making this more generic to allow all types of data with the entities importable (including fields). If anyone has already worked on it, please share the details.

mrfelton’s picture

Status:Needs review» Needs work

After applying the path from #60, and setting my processor to the Entity Field Collection processor, and then going to the settings page for the processor (/admin/structure/feeds/tc_feeds_rankings/settings/FeedsEntityProcessorField_collection_item) I get the following warning repeated 3 times:

Missing Feeds plugin FeedsEntityProcessorField_collection_item. See tc_feeds_rankings. Check whether all required libraries and modules are installed properly.

As well as the following errors:

Notice: Undefined index: FeedsMissingPlugin_feeds_form in drupal_retrieve_form() (line 750 of includes/form.inc).
Warning: call_user_func_array() expects parameter 1 to be a valid callback, function 'FeedsMissingPlugin_feeds_form' not found or invalid function name in drupal_retrieve_form() (line 785 of includes/form.inc).
Notice: Undefined index: mappings in FeedsQueryPathParser->sourceForm() (line 124 of sites/all/modules/contrib/feeds_querypath_parser/FeedsQueryPathParser.inc).
Warning: Invalid argument supplied for foreach() in FeedsQueryPathParser->sourceForm() (line 127 of sites/all/modules/contrib/feeds_querypath_parser/FeedsQueryPathParser.inc).
Notice: Undefined index: FeedsMissingPlugin_feeds_form in drupal_retrieve_form() (line 750 of includes/form.inc).
Warning: call_user_func_array() expects parameter 1 to be a valid callback, function 'FeedsMissingPlugin_feeds_form' not found or invalid function name in drupal_retrieve_form() (line 785 of includes/form.inc).

Note that the patch in #60 did not apply cleanly. It wanted to make some changes to feeds.info, but they were completely out of sync with the .info file in the latest -dev code. But, I don't think that is related to the problem here.

afestein’s picture

I can confirm the exact same issue as #63. I tried applying it to 7.x-2.0-alpha4. From what I can gather it's supposed to create a file "plugins/FeedsEntityProcessor.inc" but that's not happening.

I'm patching via OSX Terminal - what could we be doing wrong here?

colan’s picture

Title:add a generic entity processor» Add a generic entity processor
Issue tags:+D7 stable release blocker

Adding tag.

patcon’s picture

Title:Add a generic entity processor» add a generic entity processor
Issue tags:-D7 stable release blocker

@mrfelton sorry, #60 is against alpha4, not dev. it was just a reroll of a previous, so that I could put it in a makefile
@afestein #60 was rolled for drushmake, which might be applying the patch differently than you are manually. look into the patch command's "p" flag

"patch -p0 < test.patch" should work from the module's root dir

bibo’s picture

Status:Needs work» Reviewed & tested by the community

Patch from #60 + additional comments from #66 works for me.

cthiebault’s picture

Patch #60 on alpha4 with field_collection dev generates the same error as #37:

EntityMetadataWrapperException: There is no property for this entity. in EntityStructureWrapper->get() (line 402 of /sites/all/modules/entity/includes/entity.wrapper.inc).
elliotttf’s picture

Status:Reviewed & tested by the community» Needs review
StatusFileSize
new7.96 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-69-feeds-generic_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Re-rolled the patch against 7.x-2.x.

Code functionality was not changed; however, the .info file no longer includes plugins in its files array.

adam_b’s picture

I've been following this thread with interest, because I have a lot of data which I need to import into Field-Collection entities. The structure isn't complex -- each row has one date (year-only) field and three text fields -- but obviously none of them are presently visible in the field-mapping interface.

I'm a product manager rather than a developer, so I haven't tried any of the patches yet. Would anyone be able to give me an idea of how close we are to being able to do what I need? Should I just roll a patch and try it? Would a financial contribution speed up the process?

Many thanks.

bibo’s picture

Should I just roll a patch and try it? Would a financial contribution speed up the process?

@adam_b: If you can apply the latest patch in #69, go for it. elliotttf might also like a contribution?

Our contribution will be testing/reviewing this patch as soon as we can in the next few days. Entity support is really crucial for feeds. Hoping to get this RTBC asap.

bibo’s picture

Assigned:e2thex» Unassigned

Unassigning e2thex , who apparently has not been around for 10 months. This was already suggested.

There have been a bunch of patches by different people, is there someone who this assign should belong to?

cthiebault’s picture

Patch #69 still does not work with field collection... but that's normal as code functionality was not changed.

EntityMetadataWrapperException: There is no property for this entity. in EntityStructureWrapper->get() (line 402 of /sites/all/modules/entity/includes/entity.wrapper.inc).
staniel25’s picture

I almost have it. Added patch #69. Now, the Entity tables now show up and are selectable.

However, the table fields are not showing up in the 'Target' selection list, as I would expect.

Any suggestions?

Tx Nube

killtheliterate’s picture

@jdschwam is that list being built with a different query?

staniel25’s picture

No special queries from my side. I think the query for Target values comes from the #69 patch.

I am wondering if my db table has been set up correctly an an Entity. Anybody have a suggestion on how to verify this?

Thanks in advance for any help.

turoibz’s picture

Category:task» support

I was following the great article by Yuriy Gerasimov about Entities on Trellon's website (here is the link http://www.trellon.com/content/blog/creating-own-entities-entity-api ). It gives you a great explanation about how to create your own entities.

I have applied the #69 and now I can see my entities on the processor selector. But none of my fields are available on the Mapping section. Note that the fields on my entity are declared in code so far.

I wonder why is this issue. I hope someone can explain.

Thanks in advance.

kreynen’s picture

I had to move the FeedsEntityProcessor.inc into the plugins dir after applying the patch, but that may have just been the way I applied the patch. I also extended FeedsEntityProcessor with class FeedsEntityProcessor[ENTITY] in the module I defined the entity. After that, I could map any fields added to the entity, but not fields defined by the entity. The only field showing up in the mapper was the [ENTITY]_id, but mapping that returns...

Entity property [ENTITY]_id doesn't support writing.

It also returns that error when trying to map an id to any of the entities already defined (comments, og, etc). Worth noting that Media does not show up in the list a targetable entities, but anything defined by extending the EntityAPI does. Not sure what would be required to add to Media to make it targetable, but that would helpful to at least define what's required to target the entity with the FeedsEntityProcessor.

echataig’s picture

Same error as #73 here when I select Entity Field Collection Processor after applying patch #69.
However, I see that #74 partially succeded.

Is there a chance that versions of modules I use could lead to this error ?

I'm using Feeds 7.x-2.0-alpha4+40-dev.

bradjones1’s picture

Category:support» feature

Reclassifying as this isn't a support thread; it has a pending patch needing review.

likewhoa’s picture

Status:Needs review» Needs work

This is missing two plugins in order to work with profile2, they are:

  1. FeedsEntityProcessorProfile2
  2. FeedsEntityProcessorProfile2_type

Also getting Fatal error: Call to undefined method FeedsMissingPlugin::getMappings() in sites/all/modules/feeds/feeds_ui/feeds_ui.admin.inc on line 502
when clicking on mappings. This is with latest -dev and patch from #69 applied.

axel.rutz’s picture

crosslinking:

This is related but not identical to #1257314: Ability to attach a feeds source to any entity, not only nodes.

johnv’s picture

@likewhoa,

Feeds mappers are better off in their dedicated module, so each module maintains its own Feeds support.
Therefor, shouldn't the Profile-Feeds support be in the Profile2-module? And then the status back to Needs Review?
(BTW, I see support-requests for Profile2 popping up in other modules, too, so perhaps Profile2 doesn't set some variables.. )

franz’s picture

@johnv, I think this is not the case. This patch generates those classes for all entity types, so that's why it tries to find those classes. Maybe this is a case of clearing all caches? Either way, the fatal error is just a derivative of the first, since the proper class is not found, FeedsMissingPlugin is used instead.

likewhoa’s picture

@johnv yes i agree each contrib module should support their own feeds mapper because feeds module itself shouldn't have to cater to every module which wants a mapper but just provide the necessary api for them to use and implement.

@franz I can rule out cache which doesn't remedy the problem.

js’s picture

Thank you for this work.

I have installed patch in #69 and creating a feed into a "data table". The table shows up nicely in "select a processor", but I don't have the option to select certain fields as a "unique target".

I have used the data patch
http://drupal.org/node/1551430 #4.

I am also getting the following error, and don't yet know if it is related:

Notice: Undefined property: stdClass::$headers in http_request_get() (line 98 of ../sites/all/modules/feeds/libraries/http_request.inc).
Warning: array_change_key_case() expects parameter 1 to be array, null given in http_request_get() (line 98 of /var/www/nolet/html/sites/all/modules/feeds/libraries/http_request.inc).
Download of failed with code -1002.

js’s picture

I haven't sorted out the logic yet, but the problem is that I am not setting up fields correctly when I create the data table structure.

js’s picture

Does someone have feeds working with /project/data?

I have used the patches for /project/data from here:
http://drupal.org/node/1551430

What seems to be the problem is that guid and url are not set for the feeds_item object.

I am very confused by where this should be mapped. The FeedsEntityProcessor.inc appears to me to work differently than the FeedsNodeProcessor.

I would appreciate help.

js’s picture

I appreciate all the work and help with feeds and data. I have it working and for me for scale, this is substantially better then creating nodes.

Should the this be camelcase like the parent?

+ public function entitysave($entity) {

I think the parent is: entitySave

lalweil’s picture

The entity processor seems to be working fine for me, but Data Tables aren't showing up as options. I have tried several different patches listed on http://drupal.org/node/1551430 without any success. I have created a couple of different data tables, and I still don't see anything.

Am I missing something?

Thanks for all the work!

emilyf’s picture

i patched with #69 against feeds dev using -p1 flag and it works correctly. then extended FeedsEntityProcessor with FeedsEntityProcessor[ENTITY].inc in my module but the only field that is mappable is the id.

i then integrated this patch: http://drupal.org/node/1063434#comment-5217400
as the other fields in my module are field collection fields.

now my field collections are also mappable, and data is coming in.

liquidcms’s picture

just weighing in with my findings so far:

node (question)
- questions (f collection) : 3 allowed

collection (questions)
- question (text) : 1
- tip (text) : 1
- options (f collection) : unlimited

collection (options)
- option (text)
- value (int)

i added the patch for field collections linked in #91

- this now lets me see the fields within my first collection: question and tip
- but i don't see how i would select delta to map to which specific on of these i am writing to (??)
- i also don't see the fields within the collection field (so i guess no support for recursively descending through the entity?) - granted this might be hard to map out from the other end (CSV file) anyway

other:
- on manage fields pages for collections i get numerous "Missing Feeds plugin FeedsEntityProcessor." errors
- i have not tried to actually import anything yet (no delta control so wouldn't make sense to import)

i added patch from #69 above and adds nothing to UI; but possibly would make things work (??)

i don't understand emily's comment above "then extended FeedsEntityProcessor with FeedsEntityProcessor[ENTITY].inc in my module" - perhaps this helps with something (??)

i could also throw some funding $$ at this issue if it would help (or coding effort)

wiherek’s picture

I applied patch from #69 to the latest dev version. I am getting two errors now, one of which is critical.

The minor:
drupal (and drush upon enabling the module) displays this warning:
Missing Feeds plugin FeedsEntityProcessor. Please contact your site administrator.

The critical error:
After importing some rows from a csv file, I get:
EntityMalformedException: Missing bundle property on entity of type node

I checked in the database and both 'bundle' and 'type' cells have false as value.

wiherek’s picture

Status:Needs work» Active

Ok, the problem was related to custom entities created by modifying the Model Entities module. We tried using Feeds with entities created with Entity Creation Kit module, and that works without errors.

greggles’s picture

Status:Active» Needs review

After reading the comments in ~81-92 I'm not sure if this is "needs review" or "needs work" but it's definitely not just active :)

scottrigby’s picture

@Greggles Agreed!

I should mention I've been using #69 with success (along with #661606: Support unique targets in mappers).

Though I continue to get this warning about a dozen times on every cache clear:

Missing Feeds plugin FeedsEntityProcessor. Please contact your site administrator.

If someone else can confirm this, we should probably change to 'needs work'. Otherwise I may be missing something minor in my configs or setup (because it does seem to work perfectly otherwise).

imclean’s picture

Status:Needs review» Needs work

I'm also seeing the error "Missing feeds plugin FeedsEntityProcessor" when clearing the cache, as is at least one other.

"Needs work" seems to make sense.

gilgabar’s picture

StatusFileSize
new9.04 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-98-feeds-generic_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

The 'Missing feeds plugin FeedsEntityProcessor" errors look like they are a result of a bug in feeds_entity_property_info_alter() introduced in #1424992: Add implementation of hook_entity_info_alter() so that $node->feed_nid is available to Rules. The call to feeds_plugin() is passing in the class name as the plugin name. This works in most cases because the plugin name and class name are usually the same. It breaks in this case because the class 'FeedsEntityProcessor' is used for all of the feeds entity plugins, none of which use that exact name.

Changing that to use the plugin name fixes the missing plugin errors, but results in another set of errors. "Notice: Undefined index: type in FeedsEntityProcessor->entityType() (line 18 of /sites/all/modules/contrib/feeds/plugins/FeedsEntityProcessor.inc)." One of those for each entity plugin. This also appears to be due to the hook_entity_property_info_alter() implementation. It creates an instance of a processor with a fake importer for each plugin, so that it can check for the existence of the 'entityType' method and then add a feed_nid property to each of the entities supported by a plugin. The problem with this is that the fake importer appears to think it is configured to use a node processor, so when it gets to calling the entity class, which expects the plugin to have a type property, it doesn't find a type because the node plugin does not have one.

I would suggest removing the checking of processor plugins in feeds_entity_property_info_alter(). The addition of support for all entities with this patch should make that redundant.

I haven't tested extensively, but a revised patch is attached.

gilgabar’s picture

Status:Needs work» Needs review

And of course I forgot to set it to 'needs review'.

axel.rutz’s picture

@gilgabar: i thank you wholeheartedly for that good explanaition.

I would suggest removing the checking of processor plugins in feeds_entity_property_info_alter(). The addition of support for all entities with this patch should make that redundant.

I support this and it would be in line with #1739842: Deprecate some processors in favor of generic entity processor.

So we still need this reviewed and tested!

ndf’s picture

StatusFileSize
new28.09 KB

Followed #77 Turoibz instructions and patched against Feeds 7.x-2.0-alpha5.

Tests:
- Well Known Entities 'Node', 'User' and 'Comment'.
- Entity from Code.
- Entity from Code with Fields attached via UI.
- ECK (Entity Construction Kit) Entity with Fields.

My findings:
- For well known Entities 'user', 'comment', 'node' all properies and fields are working.
- For Entities build with ECK properies and fields are working.
- For coded Entities attached fields are not working / recognized automaticly. Properties do work.

NB:
- For ECK Entities getting recognized you need to flush cache.

Conclusion for sitebuilder and coders: use ECK if you want an extensible and 'feedable' site.

I am very interested in #62 Mukesh.agarwal17 not only have a generic entity processor, but also a generic attached field processor. My expectation (or wish) is that when we create a new entity with fields it is 'magicly' processable by Feeds.

ndf’s picture

StatusFileSize
new28.09 KB
axel.rutz’s picture

thank you niels for your tests. this really brings us much closer to takeofff!

For coded Entities attached fields are not working / recognized automaticly. Properties do work.

any ideas from folks here why this is the case and how difficult this ist to implement?

gilgabar’s picture

@nielsdefeyter could you clarify at what point the fields are breaking for the case of coded entities?

Re: #62, fields are added as targets via hook_feeds_processor_targets_alter().

I haven't had a chance to test a full import yet, but the field targets are showing up for my custom coded entities, as well as a contrib coded entity (profile2), so I'm curious if you are seeing the fields listed as available targets at all, or if the import is failing at some point after the feed is configured, or something else is going on.

miiimooo’s picture

I've just tested this as well thanks to tests from #101

Using ECK, git 7.x-2.x version of feeds and feeds_sql the initial import works a charm for text and integer properties and fields.

I'm a bit confused about the update feature. Does this require the GUID or the Id to be set as unique target? Using the Id I get "SQL violation for the primary key" warnings. It looks like it's trying to create new items instead of updating the existing ones.

bpmbriguy’s picture

StatusFileSize
new52.15 KB

I was able to successfully use an ECK entity and the feeds entity processor. I am importing a CSV file with a primary key, text, and integer fields. In this case my primary key is "ProviderID"

While using the update feature, I mapped ProviderID to GUID and set it as unique. Then I also mapped ProviderID to my entity ID.

jlyon’s picture

StatusFileSize
new18.56 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 1033202-107-feeds-generic_entity_processor.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Here is an addition to the patch in #98 to always include a select box to choose the bundle on the config form. This was all I needed to be able to map to the attached fields for an entity in code that was generated from the Model module, as well as for the redhen_contact entity that is a part of the Redhen module.

scottrigby’s picture

@jlyon that should probably be a separate issue. Cross-referencing #661606: Support unique targets in mappers & #1711648: Abstract bundle handling.

btw, interdiffs are really helpful on large patches like this, so people can see at a glance what you're proposing ;)

twistor’s picture

Status:Needs review» Needs work
StatusFileSize
new9.11 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-110.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Ok, so I finally got around to re-reviewing this after initially reviewing it some time ago. I am sold, however, there's no way that this is going to get into the 2.x branch. I poked around the patch to get it to work on current dev, and for multi-columned fields like the Link field. It's awesome, but I think it's time for 3.x.

Actually, there is a way to get this into 2.x. The plugin would have to only declare itself for entities that are supported already, so it would have to be last and do some checks. The field handling that it does itself would have to be removed because it is very broken and doesn't allow for a bunch of fun, new things that are in Feeds. To do that we would need to extend Feeds api to be able to determine the plugin type and key externally to a plugin. Also, it would have to handle inheritance correctly, which, currently it basically ignores.

Attached is my *very* quick hacking to prove to myself that this is the way forward.

twistor’s picture

StatusFileSize
new15.69 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-111.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Errr, this one.

pcambra’s picture

Looks great, but there are some extra spaces and a dpm left behind in getMappingTargets.

This processor will kill all custom processors (such as commerce_feeds ones) so it'd be great to have it in feeds itself.

Cebra’s picture

I have applied the patch on the current dev-version. Do I have to do anything else to use feeds with a custom entity? Because I don't find a custom entity processor in the importer settings of feeds. Thanks

patcon’s picture

@cebra, if it's like the old one, you might need to create a bundle, if i remember correctly. Might be wrong, but just skim the comments above and they should recap the expectation :)

Cebra’s picture

Thx. I created a bundles but I don't see any custom entity processor in the feeds importer settings.

I used the patch111 on the dev-version from september, 8th. But I received the messages:

git apply -v ../feeds-entity-processor-1033202-111.patch
feeds/../feeds-entity-processor-1033202-111.patch:18: trailing whitespace.
feeds/../feeds-entity-processor-1033202-111.patch:254: trailing whitespace.
feeds/../feeds-entity-processor-1033202-111.patch:298: trailing whitespace.
feeds/../feeds-entity-processor-1033202-111.patch:302: trailing whitespace.
feeds/../feeds-entity-processor-1033202-111.patch:311: trailing whitespace.

warning: squelched 1 whitespace error
warning: 6 lines add whitespace errors.

Could it be that the patch hasn't been applied?

Cebra’s picture

Ok.... I've found my mistake. ;-) Now the patch works fine.

But when I select an entity processor and click on the mapping-link afterwards there is a failure, so that my screen keeps blank. I thought it is an url problem, but I'm not sure.

twistor’s picture

StatusFileSize
new15.63 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-116.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

This should get rid of the failures, but it's still rough.

Cebra’s picture

Works fine! ;-) Thank you!

spotzero’s picture

StatusFileSize
new7.72 KB
PASSED: [[SimpleTest]]: [MySQL] 3,847 pass(es).
[ View ]

Attached is a patch, based heavily on patch 116, that provides a working generic entity processor (at least everything I've tested so far is working), without requiring any other changes than adding the FeedsEntityProcessor.inc file (other than having to mention it in "feeds.plugins.inc" of course).

Also, this FeedsEntityProcessor has the entity type selection as a settings option, so it doesn't clutter the "processors" tab with a entity processor for each entity type.

I have to admit though, that I'm a little confused at what a lot of the changes to feeds in patch 116 do, since, as demonstrated by this patch, they don't appear to be necessary for a generic entity processor.

I haven't been following this issue for very long, so please let me know if I'm way off in my understanding of how things should work.

yang_yi_cn’s picture

Hi spotzero,

Do you know why that in the file FeedsEntityProcessor.inc, near the end, this line was commented out?

<?php
//drupal_alter('feeds_processor_targets', $targets, $type, $info['bundle']);
?>

I'm using the patch and it works fine with my ECK created entities until I try to map fields from the source to some location fields, which location_feeds module supports by calling the feeds_processor_targets hook. I copied the module for my own customization and changed the supported type from "node" and "user" to my own ECK types. Originally it didn't work until I find out that the patch commented out the hook calling code.

Just FYI, after I uncomment the code, location mapping is working for me. (location CCK field, attached to an ECK created entity)

yang_yi_cn’s picture

also a minor error, I think might because the Feeds API changed? Getting a lot of warning using the patch 118:

Warning: Missing argument 5 for FeedsEntityProcessor::setTargetElement(), called in trunk/htdocs/sites/all/modules/contrib/feeds/plugins/FeedsProcessor.inc on line 410 and defined in FeedsEntityProcessor->setTargetElement() (line 137 of trunk/htdocs/sites/all/modules/contrib/feeds/plugins/FeedsEntityProcessor.inc).

The original interface in FeedsProcessor.inc is like this:

<?php
 
/**
   * Set a concrete target element. Invoked from FeedsProcessor::map().
   *
   * @ingroup mappingapi
   */
 
public function setTargetElement(FeedsSource $source, $target_item, $target_element, $value) {
    switch (
$target_element) {
      case
'url':
      case
'guid':
       
$target_item->feeds_item->$target_element = $value;
        break;
      default:
       
$target_item->$target_element = $value;
        break;
    }
  }
?>

but in the FeedsEntityProcessor.inc provided by the patch, it has an extra "mapping" argument which is generating a lot of errors.

<?php
 
public function setTargetElement(FeedsSource $source, $target_item, $target_element, $value, $mapping) {
?>

The "$mapping" is not used in the function anyway, so it should be removed.

Cebra’s picture

I received an error when importing an XML file where one node of the mapping is missing. It don't know if this problems occurs due to problem in the XPath Parser, the feeds module or this patch here.

Could anybody help me?

EDIT: Seems to be an entity problem. Replacing the entity with a node (custom content type) everythings works fine. But I need entities.

brayo4’s picture

i get this error with patch from #118:

Parse error: syntax error, unexpected T_STRING in \sites\all\modules\feeds\plugins\FeedsEntityProcessor.inc on line 69

yang_yi_cn’s picture

BTW all my test are only with CSV imports, and they are going well so far.

imclean’s picture

StatusFileSize
new11.31 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-124_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

A combination of #116 and #118 as the entity type selection settings option is quite handy.

I'm trying to get it to work with the Data module for scheduled and manual imports. See #1551430: implement hook_entity_property_info for Feeds integration

imclean’s picture

I'm back to #116 again for testing but the question in #119 is a good one: why is drupal_alter() commented out? This changes the structure of the entity.

imclean’s picture

The drupal_alter() line is required to enable other modules to modify entity target mappings.

g089h515r806’s picture

I have test with the entity processer patch, it does not works for field collection.
So I create a module http://drupal.org/project/field_collection_feeds

rickmanelius’s picture

FYI #124 no longer applied cleanly for 7.x-2.0-alpha7. Working on a re-roll...

rickmanelius’s picture

Status:Needs work» Needs review
StatusFileSize
new11.39 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-129_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

There were just some minor cleanup issues to get #124 to apply cleanly to the latest... setting to needs review (particularly with respect #116, which appears to be working for at least one other person).

mzwyssig’s picture

I have this error when I apply the patch:

$ git apply feeds-entity-processor-1033202-129_0.patch
docroot/sites/all/modules/feeds/feeds-entity-processor-1033202-129_0.patch:82: trailing whitespace.
 
docroot/sites/all/modules/feeds/feeds-entity-processor-1033202-129_0.patch:201: trailing whitespace.
   
docroot/sites/all/modules/feeds/feeds-entity-processor-1033202-129_0.patch:203: trailing whitespace.
   
warning: 3 lines add whitespace errors.

I applied it on the latest dev version.

kreynen’s picture

I was able to apply the patch cleanly to both 7.x-2.0-alpha7 and 7.x-2.x-dev using...

patch < feeds-entity-processor-1033202-129_0.patch

The only issue I had after the patch with the base functionality was the same issue I pointed out in #78 where FeedsEntityProcessor.inc is created in the root module dir instead of in the plugins dir with the other processors.

Once installed, I can map to all the fields defined using the Field API. I do NOT see fields I've defined within the entity like title and description. With an entity like file, I can only map to File type and fid. With Comment, I get a much more complete list to fields to map to.

What controls which fields are mappable by default.

Also, when selecting an Entity type in Entity processor Settings, the fields for that entity's form are loaded after the select list option is selected. Values are required for any form fields that are required which seems odd. What's even more confusing is that fields with formatters like date or term reference show up as text fields.

imclean’s picture

Also, when selecting an Entity type in Entity processor Settings, the fields for that entity's form are loaded after the select list option is selected.

This one of the reasons I went back to the patch in #116. Perhaps give that one a go, using git apply to apply the patch.

kreynen’s picture

#1431952: Date field will only set in Rules if it already has a value

I think this may be an issues for mapping dates on entities in Feeds as well. Text fields work fine. Adding a date to the mapping returns...

Unable to set the data property value as the parent data structure is not set.

twistor’s picture

Title:add a generic entity processor» [Meta] Generic entity processor
Status:Needs review» Needs work

Alright, I have started a branch, 1033202-entity-processor. That way we don't have to keep re-rolling this.

The current branch is where I think this should be going.

The dropdown for selecting entity types is clever, but is absolutely not going to happen. In fact, I am going to add a feature at some point that doesn't let you change the processor at all if it has imported entities. There's a lot of work around this that needs to be done, that's not directly related to this patch. Please create new issues based on this branch, and we'll reference them here.

I consider this to be the only blocker to a Beta, since we will end up deprecating all other processors, and hopefully removing them before launch.

Repeated: Please create new issues based on this branch, and we'll reference them here.
#1739842: Deprecate some processors in favor of generic entity processor

spotzero’s picture

Twistor,

Can you clairify:

The dropdown for selecting entity types is clever, but is absolutely not going to happen. In fact, I am going to add a feature at some point that doesn't let you change the processor at all if it has imported entities.

Does it mean:

The dropdown for selecting entity types will never be included in a release, for reasons I didn't mention. Also before this patch is included in a release, there needs to be a feature that doesn't let you change the processor at all if it has imported entities.

Or:

The dropdown for selecting entity types is clever, but can't be included until a feature that prevents you from changing the processor at all if it has imported entities is implemented.

imclean’s picture

"Absolutely not going to happen" indicates to me it won't be included. There is a UI issue with it, as mentioned previously. It also makes no sense if other processors are going to be deprecated in favour of the entity processor. In that case you'd have only one processor in the initial list and a drop down consisting of everything else.

spotzero’s picture

Fair enough, I agree that there really isn't a reason to include it if all processor are replaced with this one.

gmario’s picture

hello everybody,

it is from some days that I'm trying to make to work this patch, but apart from of an initial success there is not way :)

I describe what i do:

1) install drupal 7.17

2) downloaded following modules:
job_scheduler (7.x-2.0-alpha3)
schema (7.x-1.0-beta4)
ctools (7.x-1.2)
entity (7.x-1.0-rc3)
views views_ui (7.x-3.5)
data data_ui (7.x-1.0-alpha4)
feeds feeds_ui (7.x-2.0-alpha7)

3) apply patch #129

4) enabled modules in this order:
job_scheduler
schema
ctools
entity
views views_u
data data_ui
feeds feeds_ui

5) from data section create a table (with primary key)
6) from feeds section created a feeds importer with "import from file", "cvs parser" and "Entity processor" but in "entity processor" settings there is not track of table created in 5)

I have done the same try for patch #116

Am I making some mistake? Should I to apply some other patch?

Thanks in advance for any help.

gmario’s picture

StatusFileSize
new4.02 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-139.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

an update related to the previous post that may be useful to someone else.

I've taken entity, feeds, data end views modules from their respective repository ( maybe an operation unneeded for all modules) then I've applied to feeds patch in #129. that has solved the problem.
That patch, anyway, doesn't can be applied as is to feeds but it needs to be correct. Here the version that worked with me: patch

imclean’s picture

gmario, your patch doesn't include the .inc file. Also, please see comments from #134 and later. Based on this info, the most recent valid patch is #116. Please ignore mine in #124 and later ones which aren't based on #116.

gmario’s picture

StatusFileSize
new181.69 KB
new35.79 KB

Hi imclean,

it is more than a week that i'm trying to make to work data and feeds modules together. I'm read this post, this and this too.

There is a bit of confusion in me now :)

Here what i have done:

installed a fresh drupal 7.17 site
installed ctools, schema, job_scheduler, entity modules in a standard way by using drush, while data and feeds modules from git.

Then accordingly to this comment (posted by you :) ) I've applied patch #116 to feeds and patch #48 (following what that has been said in posts) to data in data I have also applied patch for labels.

enabled ctools, schema, job_scheduler, entity, views, views_ui
enabled data, data_ui, data_entity and feeds, feeds_ui

Now I've created a table from data_ui (doing so an error is appeared in reports - file attached)

Then I have created a feeds importer in which there is not listed "data entity processor". (image attached)

What I doing wrong? :)

thanks a lot for your help.

a si biri
gmario

js’s picture

I had this working months ago, but because the patch didn't make it into an update, my feeds broke.

For me, the work around was to use
feeds_processor_targets_alter
to add the data fields back into $entity and they are currently saving into the Data table fine.

imclean’s picture

Hi gmario,

- Use the Feeds patch from #116 above.
- For the Data module, the patch in #1551430: implement hook_entity_property_info for Feeds integration has already been committed and nothing else required for basic feeds integration. Use the latest dev to be sure.
- Create your Data table and specify a Primary Key within the settings. The PK field must be an integer (or Serial)
- Clear your cache

After the above steps, your data table should appear as a processor ("Entity TableName Processor").

Check Data's issue queue, there are some other useful patches but nothing specifically required for a straight feeds import.

gmario’s picture

Thanks imclean, now it works.

your help it was really appreciated.

a presto
gmario

mErilainen’s picture

The patch in 116 worked for me also, but I had to do the same and enable the drupal_alter() in FeedsEntityProcessor.inc to make it work with another patch for file_entity module (See issue #1430190: Add feeds mappers for files for more information).

EDIT: I think there is some issue when trying to update the imported files. I get a database error for each existing image, so the GUID mapping is not working properly. It should update the existing file, not to add a new one.
Error message:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'public://path_to_file/12867588.jpg' for key 'uri'

mErilainen’s picture

StatusFileSize
new15.62 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-processor-1033202-146.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Here is the same patch from #116, but with the drupal_alter() included, because several comments mentioned that it should be included and it works in my case also. Just something that can be referenced.

alibama’s picture

just to be clear - this patch http://drupal.org/node/1616680 got me through the organic groups referencing when patching my entityreference module (drushed it in today + latest feeds) whereas this patch did not work - just an fyi for anyone else who may follow in the near future

coreycondardo’s picture

StatusFileSize
new62.24 KB

Trying to use this patch to import multiple profile entities to attach to multiple users.

Putting a specific UID into the User field of Entity Parser profile settings (see screen shot) gets me one user.

Is there any way that I can put a variable into that field to allow the import of multiple profiles and tie them to the associated users?

The use case being that we have thousands of profiles to import and tie to thousands of users.

Please and thank you.

alibama’s picture

yes - it works fine, however that's a feeds tamper question - use the explode tamper and you should be fine

oh yeah - btw I've never gotten this patch to do exactly what i wanted, so refer to the link three up... that worked like a charm (please excuse me - if someone else has this all dialed in then that's awesome, just haven't gotten lucky myself at least with the organic groups stuff)

imclean’s picture

No need to keep rerolling the patch, see #134.

Check out the branch 1033202-entity-processor:

http://drupal.org/project/feeds/git-instructions

imclean’s picture

I've had some problems with that branch so am still using #116.

So much has changed with Feeds since that patch was created. I've tried updating it to the current 7.x-2x version on git but am getting memory errors relating to _ctools_export_unpack_object().

axel.rutz’s picture

Todd Young’s picture

I have applied the patch in #116 against latest 2.x and it seems to work, but I'm ignoring a fail while patching the .info file.

Is this going to make it into a dev version, or a 3.x, etc?

lolcode’s picture

Subscribing. Edited: I had posted a comment about an issue with the entity processor branch but after further checking my issue belonged to location feeds.

j0rd’s picture

Same problem as #120

Warning: Missing argument 5 for FeedsEntityProcessor::setTargetElement(), called in trunk/htdocs/sites/all/modules/contrib/feeds/plugins/FeedsProcessor.inc on line 410 and defined in FeedsEntityProcessor->setTargetElement() (line 137 of trunk/htdocs/sites/all/modules/contrib/feeds/plugins/FeedsEntityProcessor.inc).
webadpro’s picture

Which patch should we be using on which branch ?

I've tried patch #116 on 2.3 and 1033202-entity-processor but its not applying correctly.

j0rd’s picture

@webadpro, I'm currently using the branch from GIT specific for this patch.

I also tried to merge it with HEAD, but that caused me some problems, so currently I'm just using the branch.

I did a large import (~50,000 or so nodes) and it worked as expected.

webadpro’s picture

J0rd, Thats exactly what I have done also. I've used the branch in place.

Thanks for the confirmation also. :)

vinoth.3v’s picture

Applied the patch #129 against feeds_entity_processor branch.

Fatal error: Call to undefined method EntityValueWrapper::getPropertyInfo() in feeds/plugins/FeedsEntityProcessor.inc on line 152 (If I select EntityProcessor)

with another entity, I was unable to select any fields to match.

kreynen’s picture

I've been using the git branch without any patches with both http://drupal.org/project/cm_airing and http://drupal.org/project/media_event. It would be great to have a 3.x tag to file specific issues against, but since that doesn't exist I'm adding them here.

- Fields defined in the entity aren't mappable. The only solutions I could find were to write a custom processor or remove the fields from the entity's schema and redefine them using field API.
- Taxonomy's assigned to entity's aren't filterable. This has nothing to do with Feeds, but is something some people working with entities will run into. #1826068: Create an entity generic version of views_handler_field_term_node_tid solved this issue for me. It would be great to see more testing on that so the patch gets into Views.

Other than that, everything is working well with both new entity feeds as well as existing node feeds.

agn507’s picture

Noticed a small error on the entity branch. An error in FeedsEntityProcessor.inc on line 76

if ($this->config['authorize'] && !empty($entity->uid)) {

about authorize not being defined. Making sure that authorize was set in the defaults fixed the problem.

  public function configDefaults() {
    return array(
      'values' => array(),
      'authorize' => TRUE,  //was missing
    ) + parent::configDefaults();
  }
jshimota01’s picture

subscribing d7 +1

I'm interested in testing this as it moves further along. I have a devel machine setup for it now using php5.29, sql and apache (wamp 2h)

j0rd’s picture

StatusFileSize
new1011 bytes
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_ui-genericentity-configfix-1033202-163.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

I think the branch has a bug where you're the appropriate defaulted "Entity Processor" is never set via the feeds_ui.

I've created a patch which resolves the issue for me.

I think it may be the same problem as experienced by comment #3

There appears to be another problem where you can not map fields on the entity.

This patch is created against the entity-generic branch.

PS. I'm using messages as my entities.

vordude’s picture

StatusFileSize
new546 bytes
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity-warning-1033202-164.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

On the Access check before entity save (the method entitySaveAccess()) if $this->config['authorize'] isn't set, then you get a warning.

Here's a patch to make the warning go away.

I think this doesn't create an issue by bypassing a check if it does, slap my hand and I'll fix it.

franz’s picture

Status:Needs work» Needs review

Status:Needs review» Needs work

The last submitted patch, feeds-entity-warning-1033202-164.patch, failed testing.

MegaChriz’s picture

Status:Needs work» Needs review

I think the patches are made against the 1033202-entity-processor branch, therefore they fail to apply on the 7.x-2.x branch. Setting issue status back to "needs review".

j0rd’s picture

My patches are against the 1033202-entity-processor branch, which is the one I think most people in here are using.

I personally can't really patch against -dev....as it's moved on since this generic entity branch was created and work when merged with -dev.

franz’s picture

Well, can't you try to merge with -dev, fix it and post a patch? We'll have to do it sometime anyways. Or you can rebase from -dev.

MegaChriz’s picture

In #167 I didn't meant the patches should be made against 7.x-2.x-dev (at least, not right now), I meant that the tests were failing because the testbot was testing the patch against the wrong branch, resulting in the (in this case) wrong issue status "needs work", so I had set the issue status back to "needs review".

I agree with franz that at some point the patch should be made against the 7.x-2.x-dev, but I think it should be done when the feature is considered complete. I have not read the whole issue and therefore I'm not sure in what state the feature currently is. Hm, sounds like this issue could use an issue summary. I add the tag "needs issue summary" to this issue.

imclean’s picture

Reading the comments by twistor in #134:

Alright, I have started a branch, 1033202-entity-processor. That way we don't have to keep re-rolling this."

Please create new issues based on this branch, and we'll reference them here.

Repeated: Please create new issues based on this branch, and we'll reference them here.

He even has an example. The only problem is the new branch can't yet be selected in the version dropdown for a new issue.

MegaChriz’s picture

@imclean
Alright, that means that this issue should be used as a sort of "index" issue which links to several subissues. This is something that should be noted in the issue summary (I can not create one right now, cause I have not read the whole issue yet). A workaround for the problem that the branch can't be selected is to create a link to this issue in each subissue (on the top of their issue summary), for example:

This is a subissue of #1033202: [Meta] Generic entity processor

Problem/Motivation

(issue details here)

franz’s picture

Component:Code» Generic entity processor

I added a component, hope this helps. Then new issues can go in there.

franz’s picture

Just to clarify, the "Version" drop-down can only select branches/tags that follow the naming conventions for releases.

imclean’s picture

I have to admit, I still don't understand the future of the new branch. Feeds has changed so much since it was created.

How does or will the entity branch relate to the main Feeds branch?

j0rd’s picture

@imclean, correct me if I'm wrong, but I still don't believe you can import generic entities with the main branch?

I could be wrong, but I tried a massive import a couple weeks ago where I needed it, and the main branch didn't have this feature.

Maybe I didn't know where to look, but that seemed to be the case....and I went back to using this branch.

imclean’s picture

You're right, it doesn't support generic entities yet. But I suspect any code developed in the new branch would need a bit of a rewrite before it could be included in the main branch. That was my main concern, I'm not really sure how these things work.

rooby’s picture

I am using the special branch and the latest version of the data module and I get the option to use my data entity however not all the fields are available as targets.

I have a data table with 2 columns. An int field (PK) and a varchar field.
Only the int field is available as a feeds target.

Can anyone shed some light on why?

Is it safe to assume that this new branch will eventually be rolled into the main one?

rooby’s picture

Seems the reason is that in FeedsEntityProcessor::getMappingTargets() it does:

<?php
   
foreach ($wrapper->getPropertyInfo() as $name => $property) {
      if (!empty(
$property['setter callback']) && empty($property['field'])) {
       
$targets[$name] = array(
         
'name' => $property['label'],
         
'description' => isset($property['description']) ? $property['description'] : NULL,
        );
      }
    }
?>

It never passes empty($property['field']) because $property['field'] is always set to TRUE for my data table columns.

Surely other people are seeing this, or is it just me?

rooby’s picture

Related - I have made a patch to address the issue in #179 for the data module #1957022: Entity property info declares the table columns as fields - This feeds entity support doesn't work properly for me with the data module without it.

juampy’s picture

We had to restore Feeds plugin info files as per #1201638: Plugins should be listed in info file, but the branch works well so far.

Are we far from merging this branch on top of 7.x-2.x? People using it (including me) will miss fixes/new features from the stable branch and the more time we spend the harder it will be to merge.

Steven Jones’s picture

StatusFileSize
new1.95 KB
PASSED: [[SimpleTest]]: [MySQL] 3,919 pass(es).
[ View ]

Sorry to add to the noise, but here's a patch that fixes the strict warnings about the method declarations being inconsistent.

xjm’s picture

Tag renaming. :)

xjm’s picture

kyleoliveira’s picture

This looks like it would be super handy, but I'm kind of lost in all the comments. How close is this to being put into the main branch?

Also: Has anyone figured out how to map multiple subentities in the same import? (e.g. a single importer that would import Users and their associated Profile2 profile(s)). Yes, I've seen the Migrate-based solution, but I'm trying to do this all with Feeds.

j0rd’s picture

@kyleoliveria If someone was merging this into mainline, we'd be seeing patches.

So far it looks like most people in this issue are simply happy to continue to use the branch.

Eventually I assume someone will step up and merge it (like i've tried), but so far this hasn't happened.

I personally need some fixes on this branch, so that I can get my RSS feeds importing properly into message entities (used to work, but broke)...so I'm waiting on a fix or a merge. Eventually I'll get around to doing it myself if no one else steps up, but that won't happen until I finish a bunch of other work.

stopshinal’s picture

I'm struggling to use this branch successfully. What is the structure of the data Feeds expects for an entity reference?

I have an entity reference field that links to a taxonomy. I've tried providing feeds with the vocabulary name as well as the vocabulary ID for this field. It does not seem to import.

Is there an entity ID I should be using instead?

Ideally I would like to have the feed creates new taxonomy if it is not already present. I'm not sure how I am going to get over the hurdle of having all of my data (taxonomy vocabulary) in plain text - and not in ID's.

rerooting’s picture

Quick thought - have you looked at http://drupal.org/project/feeds_entityreference by chance?

rerooting’s picture

And http://drupal.org/project/feeds_tamper_string2id per some discussion in the issues.

@Stopshinal I think your question pertains more to mapping to entityreference field types, whereas this issue thread is a discussion and development of a generic entity processor for feeds, which is a somewhat related but fundamentally different topic. I hope these links are helpful for you (I'm about to tackle the same issue today myself!)

stopshinal’s picture

@rerooting - thanks so very much.

I suppose I'm curious what this new branch solves for me (in the context of my destined goal) Otherwise I should probably just move to the regular dev branch, correct?

I'm excited by these two other thinks you provided. I was aware of the feeds_entityreference, but though this branch solved those issues. The feeds_tamper_string2id itringues me!

Which combination will you be using? (Which feeds branch, etc)

stopshinal’s picture

Using the entity branch of feeds, as well as the string2id has worked PERFECTLY for me. Not sure about any other branches.

james.williams’s picture

StatusFileSize
new1.71 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-existingid-1033202-192.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Importing feeds using this entity processor fails when there are already existing entities that are not in the feeds_item table (e.g. when they existed before feeds was being used to import that type of entity).
This is because the FeedsEntityProcessor does not implement the existingEntityId() method. Note that the FeedsNodeProcessor does do this, and checks the node table for any matching node IDs or titles.
I've produced a rough solution, based on the FeedsNodeProcessor that checks for the ID, but given that every type of entity has potential to have quite different data structures, it won't always work.

Status:Needs review» Needs work

The last submitted patch, feeds-existingid-1033202-192.patch, failed testing.

james.williams’s picture

StatusFileSize
new2.95 KB
PASSED: [[SimpleTest]]: [MySQL] 4,353 pass(es).
[ View ]

In comment #182, there's a patch to fix the problem raised in comments #120 & #155 about missing arguments. That one missed a couple of calls to setTargetElement() that also missed the new argument. Here's a patch that includes those calls.

j0rd’s picture

Status:Needs work» Needs review

for testbot.

guillaumev’s picture

StatusFileSize
new15.27 KB
PASSED: [[SimpleTest]]: [MySQL] 4,505 pass(es).
[ View ]

Here is an attempt to merge the entity_processor branch back into the 7.x-2.x branch.

j0rd’s picture

Much appreciated @guilaumev. I'll try this out next week and let you know how it goes. I recommend others do the same.

guillaumev’s picture

Just pushing this up as it would be great if this patch could be committed before it can not be applied anymore because of other changes committed...

j0rd’s picture

@guillaumev could you post the commit in 2.x-dev you patched against for others future reference. I don't believe testbot has this information.

j0rd’s picture

StatusFileSize
new15.13 KB
PASSED: [[SimpleTest]]: [MySQL] 4,505 pass(es).
[ View ]
new1.52 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_entity_processor-1033202-196-200.diff.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Alright, I've tested @guillaumev's patch for the generic entity processing. While it applies cleanly, it has a minor issue with the bundle handling. I've resolved that. I'm attaching two files to this post.

feeds_entity_processor-1033202-196-200.diff.patch
This is the differences I've added to guillaumev's patch.

feeds_entity_processor-1033202-200.patch
All the changes from 196 with my minor fixes.

With my patch, I can now import RSS feeds into "Message" entities provided by the message module (aka a generic entity).

So for me, this works. Can we get others to test. There's many other people in here which need this patch. We need to get this in soon before this patch and -dev diverge again.

stewart.adam’s picture

Status:Needs review» Needs work

Looks pretty good, I had a small issue with it though: when selecting the 'Entity processor User', I was prompted to select a bundle when editing the mapping. When clicking the link to bring me to the processor's setting page, there is no option to select a bundle (probably because only one exists). When selecting 'Entity processor node', I do see a bundle selector with each of my content types listed.

aaronpinero’s picture

I am working in Drupal 7.22. I have created user profiles using the core Profile interface. I would like to be able to create and update users by importing a CSV file using Feeds that will populate the user profile fields. I applied the patch in #200 to the feeds module but I am still not seeing data from an imported CSV file applied to the correct profile fields. There are no errors reported by Drupal (in fact, the Feeds importer tells me that my users were successfully updated) but a review of the individual user accounts shows that this is incorrect.

kreynen’s picture

StatusFileSize
new15.9 KB
PASSED: [[SimpleTest]]: [MySQL] 4,507 pass(es).
[ View ]
new217.71 KB

Overall, the patched version of the Feeds 2.x branch works well for importing file entities directly using MediaRSS. I made one minor change to expose the Feeds views handlers to File views so that we could filter files by the source feed. The only other thing I noticed was in the UI. When using the Entity File Parser, the Bundle option is displayed twice.

This was added to the patch...

diff --git a/views/feeds.views.inc b/views/feeds.views.inc
index 01ba7d4..0836f0e 100644
--- a/views/feeds.views.inc
+++ b/views/feeds.views.inc
@@ -166,7 +166,7 @@ function feeds_views_data() {

   // Add a relationship for each entity type relating the entity's base table
   // to the feeds_item table whre feeds_item.entity_type = 'entity_type'.
-  foreach (array('node', 'taxonomy_term', 'user') as $entity_type) {
+  foreach (array('node', 'taxonomy_term', 'user', 'file') as $entity_type) {
     $info = entity_get_info($entity_type);
     $data['feeds_item']['table']['join'][$info['base table']] = array(
       'left_field' => $info['entity keys']['id'],
@@ -319,6 +319,7 @@ function feeds_views_data() {
       'type' => 'LEFT',
     ),
   );
-


   return $data;
}

j0rd’s picture

Status:Needs work» Needs review

getting testbot in here.

Need more people to install these patches and test. Let's get this in quick before branches diverge again.

Most importantly, since this is adding new functionality, is making sure it doesn't regress previous behaviour. Does everything else not using this new processor continue to work for you guys?

kyleoliveira’s picture

I'm getting "Invalid data value given. Be sure it matches the required data type and format." errors when I try to import Users with the Entity processor User processor and the last patch. It looks like it doesn't like email addresses despite the fact that I'm using normal email addresses and I've set the mappings (seemingly) correctly. I'm using the Feeds XPath module to read in an XML file, but it seems to be working correctly as well. Has anyone else successfully imported users' email addresses with the last patch? I'm doing this on simplytest.me, so I may just have set something up incorrectly.

kreynen’s picture

@kyleoliveira

Does importing users work with the XML and the "old" User import? If you export your Feed Importer and attached that along with a sample of the XML with just demo data, I'd be happy to test.

kyleoliveira’s picture

StatusFileSize
new933 bytes

It does work with the "old" User import. Also of note, the Entity-based importer does not allow you to set the Email or Username as Unique whereas the old (classic?) version does and the Entity-based importer lets you map a target for uid (which I haven't tested yet).

I've attached an example xml file. The importer is as follows:

$feeds_importer = new stdClass();
$feeds_importer->disabled = FALSE; /* Edit this to true to make a default feeds_importer disabled initially */
$feeds_importer->api_version = 1;
$feeds_importer->id = 'user_xml_importer';
$feeds_importer->config = array(
  'name' => 'User XML Importer',
  'description' => '',
  'fetcher' => array(
    'plugin_key' => 'FeedsFileFetcher',
    'config' => array(
      'allowed_extensions' => 'xml html htm',
      'direct' => 0,
      'directory' => 'public://feeds',
      'allowed_schemes' => array(
        'public' => 'public',
      ),
    ),
  ),
  'parser' => array(
    'plugin_key' => 'FeedsXPathParserXML',
    'config' => array(
      'sources' => array(
        'xpathparser:0' => '@active',
        'xpathparser:1' => 'userName',
        'xpathparser:2' => 'email',
        'xpathparser:3' => '@id',
      ),
      'rawXML' => array(
        'xpathparser:0' => 0,
        'xpathparser:1' => 0,
        'xpathparser:2' => 0,
        'xpathparser:3' => 0,
      ),
      'context' => '/users/user',
      'exp' => array(
        'errors' => 0,
        'debug' => array(
          'xpathparser:0' => 'xpathparser:0',
          'xpathparser:1' => 'xpathparser:1',
          'xpathparser:2' => 'xpathparser:2',
          'xpathparser:3' => 'xpathparser:3',
          'context' => 0,
        ),
      ),
      'allow_override' => 1,
    ),
  ),
  'processor' => array(
    'plugin_key' => 'FeedsEntityProcessorUser',
    'config' => array(
      'mappings' => array(
        0 => array(
          'source' => 'xpathparser:0',
          'target' => 'status',
          'unique' => FALSE,
        ),
        1 => array(
          'source' => 'xpathparser:1',
          'target' => 'name',
          'unique' => FALSE,
        ),
        2 => array(
          'source' => 'xpathparser:2',
          'target' => 'mail',
          'unique' => FALSE,
        ),
        3 => array(
          'source' => 'xpathparser:3',
          'target' => 'guid',
          'unique' => FALSE,
        ),
      ),
      'update_existing' => '2',
      'input_format' => 'plain_text',
      'skip_hash_check' => 0,
      'bundle' => 'user',
      'values' => array(
        'mail' => '',
        'roles' => array(
          3 => 0,
        ),
        'status' => '0',
        'theme' => '',
      ),
    ),
  ),
  'content_type' => '',
  'update' => 0,
  'import_period' => '-1',
  'expire_period' => 3600,
  'import_on_create' => 1,
  'process_in_background' => 0,
);
that0n3guy’s picture

StatusFileSize
new108.12 KB

#203 is forcing me to specify an entity ID (which is bad). See the attached screenshot.

The entity is created via ECK.

kreynen’s picture

Obviously there are additional issues that need to be resolved to get the entity processor merged with the 2.x branch, but trying to track progress on resolving these as comments in a thread with 200+ comments in an issue started 2+ years ago is going to be difficult. So far we've identified 3 "merge blocking" issues...

  • Bundles must be selected twice in File Entities
  • Email and username issues with User Entities
  • Default ID required for ECK Entities

None of these seem like they are going to be very difficult to resolve, but I'm guessing new issues will also be found as more testing is done. The 2.x branch of Feeds is still in alpha, but stable. An unpublishable branch for the entity work already exists, but new issues can't use that as their version nor can non-git savvy users help test that branch. While the generic entity processor is available as a Component for issues, breaking these issues up and working on them independently based on a patch this large is going to create a lot of work merging changes back into a single patch... and the 7.x-2.x branch can continue changing.

Adding entity support seems like an improvement worthy of a dev-only, 3.x branch. Once the entity work is completed, those changes could be back ported to the 2.x release.

While we've had 5 or 6 developers contributing to this patch in the last 6 months, without version control, packaging, and issue tagging it seems unlikely that we'll be able to get the momentum necessary to finish this. I created #2054709: Create 3.x branch for Entity Processor requesting a 3.x branch be created and someone w/ maintainer access commit to reviewing patches related to the entity processor.

MegaChriz’s picture

@kreynen
Another possibility is to move the work over to a sandbox project, so a 3.x branch can be reserved for, well a complete new version.

kreynen’s picture

Sandboxes don't have packaged downloads or version stamped releases which limits testing to git-savvy users. If we can't manage this within the Feeds project, GitHub is a better alternative to Drupal.org sandboxes.

guillaumev’s picture

The issue I see with creating a 3.x branch of Feeds to add the Entity processor is that, given the number of comments in this issue, it seems like many people need this sooner rather than later, and I doubt this 3.x branch will be released soon.

I might be missing something here, but what about doing the following:

  1. Committing the patch in #203 to the 2.x-dev branch
  2. Fixing the small bugs one by one, creating one issue for each one of them

My point here is that:

  • This patch does not break anything in the old feeds importers
  • The small bugs identified do not seem critical at all to me and can probably be fixed quickly once the patch is in place in the 2.x branch

We could eventually add an "(EXPERIMENTAL)" in the labels of the entity feeds importers, so that we make sure users know the risks, but I think creating a 3.x branch for this is being a bit too careful maybe.

j0rd’s picture

I'm with Guillaumev on this. Providing this new processor does not break any of the old ones, it should be merged in to 2.x . Then additional bugs could be found and fixed as needed.

I'm personally running the patch on production right now, and it's working fine for my needs.

guillaumev’s picture

Also running the patch in production, and working fine...

kyleoliveira’s picture

Sounds good to me (that's what dev is for anyhow, right?). Who needs to do what to make this happen?

dsnopek’s picture

Status:Needs review» Reviewed & tested by the community

Well, we can mark this patch as RTBC. :-) Then we need to get the maintainer to take a look at it and hopefully commit it.

kreynen’s picture

StatusFileSize
new16.03 KB
PASSED: [[SimpleTest]]: [MySQL] 4,507 pass(es).
[ View ]
new318.5 KB

God I love community driven decision making :)

I wasn't suggesting pushing the 3.x branch to a stable release, but using the branch as... oh, I don't know... a branch to work on this feature and then merge the 3.x work with any changes made to the 2.x branch while the remaining bugs are resolved. Drupal's move to git was a big improvement over CVS, but the link between packaging dev snapshots, issue tagging, and git tags is still really limiting. I'm suggesting options that work with Drupal.org's limitations rather than push changes with known bugs into the current, relatively stable branch.

I think everyone following this issue wants to see entity support in Feeds, but I know I have no idea what issues are preventing a beta release of the 2.x branch. I didn't want to make that work more difficult and don't have time to contribute to it so I was suggesting an option that got me what I wanted without causing more problems for someone else. I don't care if this gets committed to 2.x or 3.x, but that it gets committed or a list of what needs to be corrected before it will be committed is created.

We've been using the entity patch, then branch, and now patch again with media driven sites in production environments since March. The entity processor has issues, but it's better than alternatives for directly importing media/file entities...

https://vimeo.com/67261530

We just launched http://okv.se/play using File Entity processor to directly import 1,500+ YouTube videos to file/media entities.

I think the suggestion to label the entity processors as EXPERIMENTAL is great. I've updated the patch to include that in the title and reiterate it in the processor's description.

If the maintainers won't commit this to either the 2.x or a new 3.x branch, can we at least get a list of what would need to be resolved before this can be committed so we can work through that list?

inventlogic’s picture

Just checking regarding patching of the git version of feeds 1033202-entity-processor that includes this patch.

Is that a feature frozen version that will receive no further updates for future bug fixes or is it an active branch that is patched?

kreynen’s picture

@inventlogic,

The entity processor branch hasn't been updated in 8 months... http://drupalcode.org/project/feeds.git/log/refs/heads/1033202-entity-pr...

So it's missing everything that's been committed to the 7.x-2.x branch in the last 8 months http://drupalcode.org/project/feeds.git/log/refs/heads/7.x-2.x

AND the improvements that have been added to this patch like being able to related entities to their parent Feed the way you can w/ nodes.

I think most developers who have contributed to the entity processor are now using the current 7.x-2.x brach w/ this patch.

inventlogic’s picture

@kreynen

Thanks for the info.

Reading through this issue I figured the patch at #116 was still the valid one to use against 7.x-2.0-alpha8.

However it failed on several chunks.

Which patch should I be using?

kreynen’s picture

@inventlogic, I'm patching against the 7.x-2.x branch from git, but that is currently the same as the dev snapshot since nothing has been committed since July 6. The patch won't apply to 2.0-alpha8 because of the changes made between April and July. If you really want to help test this patch, I would encourage you to clone 7.x-2.x from git https://drupal.org/project/feeds/git-instructions.

kreynen’s picture

Media 2.x went alpha1 yesterday!

Would be really great to have a non-patched solution to direct file_entity field mapping and imports. @twistor has been committing regularly to the 8.x branch, but nothing's been done with the 7.x branches in 30+ days. 1139 open issues in the queue is also a little overwhelming. Anything people interested in seeing this move forward can do to reduce that number makes it easier for maintainers to focus on issues that are RTBC.

petria’s picture

Can anyone help me to get this patch working with external images?
(https://drupal.org/node/2066741)

gmasky’s picture

I am trying to import data into an entity form.

I applied the patch at #217

At Processor-change-Entity processor Entityform - EXPERIMENTAL I get the following message

Missing Feeds plugin FeedsEntityProcessorEntityform. See entity_importer. Check whether all required libraries and modules are installed properly.

and

Notice: Undefined index: FeedsMissingPlugin_feeds_form in drupal_retrieve_form() (line 764 of C:\wamp\www\aloysius7\includes\form.inc).
Warning: call_user_func_array() expects parameter 1 to be a valid callback, function 'FeedsMissingPlugin_feeds_form' not found or invalid function name in drupal_retrieve_form() (line 799 of C:\wamp\www\aloysius7\includes\form.inc).

The Settings for Entity processor Entityform is blank and

Mapping gives me the error Fatal error: Call to undefined method FeedsMissingPlugin::getMappings() in C:\wamp\www\aloysius7\sites\all\modules\feeds\feeds_ui\feeds_ui.admin.inc on line 491

Any help will be highly appreciated

guillaumev’s picture

@gmasky Did you try clearing the caches ?

gmasky’s picture

Yes I have cleared the cache. I get the message

Missing Feeds plugin FeedsEntityProcessorEntityform. See entity_importer. Check whether all required libraries and modules are installed properly.

on the front page after I clear the cache

jrao’s picture

This patch doesn't seem to work with entity properties of the type entity, the default values form validation can't handle it, and even if we bypass that, the mapping doesn't seem to work either.

jamesdixon’s picture

StatusFileSize
new8.11 KB
PASSED: [[SimpleTest]]: [MySQL] 4,505 pass(es).
[ View ]

I was able to get this working with 7.x-2.x for my entity. I used kreynen's patch in #217 and added some code to fix:

EntityMalformedException: Missing bundle property on entity of type partprofile. in entity_extract_ids() (line 7693 of ...

Comment #93 was having this issue as well. The problem was the bundle was getting missed when the entity info hook has $info['entity keys']['bundle'] set.

tannerjfco’s picture

Status:Reviewed & tested by the community» Needs work

Tested patch in #228 against latest 2.x-dev, I'm also getting the same error messages as mentioned in #224, along with the following warning:

Missing Feeds plugin FeedsEntityProcessorNode. See exhibitor_list. Check whether all required libraries and modules are installed properly.

kreynen’s picture

@gmasky and @Tannerjf, can you provide the exports from your exhibitor_list and entity_importer Feed Importers or the steps someone can use to reproduce this error? Did these importers exist before the patch? Were they created with a previous patch applied?

apmsooner’s picture

I applied the patch from #228 to most recent dev version of feeds. Patch applied fine, I see the experimental processors, etc... however I get the same message stated previously about missing plugin. This is what it says in my log entries:
Plugin Entity processor File - EXPERIMENTAL of plugin type feeds:plugins points to nonexistent file sites/all/modules/feeds/plugins/FeedsEntityProcessor.inc for class handler handler.

I didn't have feeds previously installed on this test site nor was there any previous patches applied. Only the one from #228. Hopefully that helps....

Also, these errors appear on the feeds admin ui:

    Notice: Undefined index: FeedsMissingPlugin_feeds_form in drupal_retrieve_form() (line 763 of /Users/apmsooner/Sites/project/includes/form.inc).
    Warning: call_user_func_array() expects parameter 1 to be a valid callback, function 'FeedsMissingPlugin_feeds_form' not found or invalid function name in drupal_retrieve_form() (line 798 of /Users/apmsooner/Sites/project/includes/form.inc).
apmsooner’s picture

Here is my export although there really isn't much to it as I can't modify settings and mapping fields leads to WSOD.

$feeds_importer = new stdClass();
$feeds_importer->disabled = FALSE; /* Edit this to true to make a default feeds_importer disabled initially */
$feeds_importer->api_version = 1;
$feeds_importer->id = 'files';
$feeds_importer->config = array(
  'name' => 'Files',
  'description' => '',
  'fetcher' => array(
    'plugin_key' => 'FeedsHTTPFetcher',
    'config' => array(
      'auto_detect_feeds' => FALSE,
      'use_pubsubhubbub' => FALSE,
      'designated_hub' => '',
      'request_timeout' => NULL,
    ),
  ),
  'parser' => array(
    'plugin_key' => 'FeedsSyndicationParser',
    'config' => array(),
  ),
  'processor' => array(
    'plugin_key' => 'FeedsEntityProcessorFile',
    'config' => array(),
  ),
  'content_type' => '',
  'update' => 0,
  'import_period' => '-1',
  'expire_period' => 3600,
  'import_on_create' => 1,
  'process_in_background' => 0,
);
jamesdixon’s picture

StatusFileSize
new8.33 KB
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es).
[ View ]

I apologize, I'm pretty new to writing patches. I forgot to include --staged in git diff so it would include the new file. Please try this one.

apmsooner’s picture

Thanks @jamesdixon, that patch took care of the issue. Not sure how to create a file entity processor though, it's asking for required fields for file id, file size, file owner and feed nid. What are you doing for that?

apmsooner’s picture

I can put bogus values in there just to get the Entity Processor File settings form to save but trying to import gives an error of: Entity property name doesn't support writing.

If it can't write the fileid and it's required in the settings form, not sure how I could use the fileentity processor. Anyone else tried this and have tips?

jamesdixon’s picture

@apmsooner: The required settings form may need work. I think what's happening is the EntityProcessor is allowing you to specify default values for each field if your Feeds Source doesn't have a value for that field.

It must be pulling these form items from the entity form config, and if the entity form config has a required value, then it will require a default value. We should probably make all those default value settings optional.

I have not tried using fileentity with this feed processor. What is the exact error message you are receiving, is it "Entity property name doesn't support writing."

apmsooner’s picture

Yup, Entity property name doesn't support writing.

This error occurs even if I don't map anything but the guid. The default required values is the root of the problem from the start. Not sure what will happen after the required restriction is removed but happy to test out if we can get to that stage.

markusa’s picture

Great feature for the feeds module.

Already used the branch discussed in the issue to successfully import a list of taxonomy vocabularies.

To be able to import as any entity is awesome!!

Thanks for you your work

jshimota01’s picture

What chance this patch (or the 116 patch) get put into the current dev for availability to non-programmers? I could really use this patch without having to learn how to patch!

gmasky’s picture

@jshimota01 patching is not that difficult. This is helpful if you use Windows https://drupal.org/node/620014

scottalan’s picture

I have been trying to get the patch in #116 to apply but have been unsuccessful thus far. Has anyone else had an issue getting the patch to apply?

Tried using drush:

projects[feeds][type] = "module"
projects[feeds][subdir] = "contrib"
projects[feeds][download][type] = "git"
projects[feeds][download][url] = "http://git.drupal.org/project/feeds.git"
projects[feeds][download][branch] = "7.x-2.x"
projects[feeds][download][revision] ="13837134da3fb6fb572384c9c1864da92c14bd1e"
; https://drupal.org/node/1033202#comment-6510204
projects[feeds][patch][] = "https://drupal.org/files/feeds-entity-processor-1033202-116.patch"

and

curl http://drupal.org/files/feeds-entity-processor-1033202-116.patch | git apply -

Errors:

error: patch failed: feeds_ui/feeds_ui.admin.inc:430
error: feeds_ui/feeds_ui.admin.inc: patch does not apply
error: patch failed: includes/FeedsConfigurable.inc:277
error: includes/FeedsConfigurable.inc: patch does not apply
error: patch failed: plugins/FeedsFetcher.inc:112
error: plugins/FeedsFetcher.inc: patch does not apply
error: patch failed: plugins/FeedsParser.inc:52
error: plugins/FeedsParser.inc: patch does not apply
scottalan’s picture

Issue summary:View changes

crosslinked

jamesdixon’s picture

Issue summary:View changes
StatusFileSize
new16.45 KB
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es).
[ View ]

@apmsooner - I believe the required fields issue existed earlier, and by opening up the allowed fields for mapping my patch made it worse. I've added some code to make those default mapping fields optional.

Also my two previous patches needed running together to work properly, I combined them into a single patch attached with the code to remove the required default mapping fields.

jamesdixon’s picture

Status:Needs work» Needs review
jamesdixon’s picture

StatusFileSize
new16.53 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-entity_processor_fix_entity_property_name_error-1033202-235.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

This last patch should fix any "Entity property [propertyname] doesn't support writing" people were having. Like #235

dobe’s picture

Patch #244 fixed my "Entity property [propertyname] doesn't support writing" as well as cleaned up the default settings.

sisko’s picture

Apologies for this ultra newbie question but to which file(s) do I apply the patch?

sisko’s picture

Apologies for this ultra newbie question but to which file(s) do I apply the patch?

jamesdixon’s picture

@sisko you download the 7.x-2.x-dev branch and apply the patch to it. Here is a link to download http://ftp.drupal.org/files/projects/feeds-7.x-2.x-dev.tar.gz.

sisko’s picture

@JamesDixon: Thanks for clarifying.

Just confirming the patch is executed against the entire Feeds folder I:E
patch feeds.patch feeds/* ???

jamesdixon’s picture

@sisko apply the patch inside the feeds folder. I am a noob when it comes to patching so I used git instructions to apply patches here https://drupal.org/project/feeds/git-instructions

sisko’s picture

@JamesDixon: Thank you for the assistance. It's very appreciated. I shall give your directions a try asap.

jamatulli’s picture

I've been able to get the Entity Processor to show up for a custom table created with the Data module. Unfortunately the mapping can not be completed because it only shows the first field in the table.

jamatulli’s picture

Correction: It is getting the first field marked Primary Key.

jamesdixon’s picture

@sisko Great let us know if it works for you.

@jamatulli I had the same issue with earlier patches. Please try version 7.x-2.x-dev with patch 244 and let us know if it works.

imclean’s picture

@jamatulli, there are a number uncommitted patches for the Data module which may help. Please try the label patch and the primary key patch from the Data module's issue queue.

#1538588: Add Entity Label field (Was: allow entity ref fields to point to data items)
#1671380: Data Entity allows entities to be created with entity ids that are not integers

jamatulli’s picture

Currently running 7.x-2.0-alpha8+8-dev which is the current dev and this was patch #244. I will take a look at those Data Entity issues and patches. It will most likely take me a couple days.

dobe’s picture

The patch worked good for using feeds to import some entities. I was unable to get it to work with commerce orders though.

agoradesign’s picture

The patch worked also good for us, importing some ECK-made entities

jlyon’s picture

@imclean: Thanks! feeds-7.x-2.x-dev with the patch from #244 and your two patches from #255 seem to be working pretty well for me (I'm importing from and xml file into a data table). I had to manually apply the patch from #1671380: Data Entity allows entities to be created with entity ids that are not integers because it conflicted with the first one.

I'm still having getting a PRIMARY KEY error when I try to update the enties by running a second import without first deleting the existing data, though:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '69365' for key 'PRIMARY'

This is likely a data module issue, so I'll follow up in one of those queues once I've had some time to track down the cause.

imclean’s picture

@jlyon, good to hear. Could you please provide feedback in both those issues?

In regards to the integrity constraint violation, check the actual schema of your feeds_item table:

https://drupal.org/comment/6757420#comment-6757420

The feeds errors ... were caused by the entity_id field in feeds_item not being set as a PRIMARY key. It is in the schema for feeds but wasn't in my database.

To fix the errors, make sure your database table feeds_item matches the schema in feeds.install. This may have been changed between feeds versions.

jamatulli’s picture

Finally got around to applying the two patches for Data and I had no luck. I also had to apply the second one line by line and assumed they were additive.

I imagine what became data_entity.admin.inc:line 81 should be what is prescribed in the "label callback" patch (i.e. $header = array(t('Name'), t('Locked'), t('Required'), t('Label'));) as opposed to being additive (i.e. $header = array(t('Name'), t('Locked'), t('Required'), t('Label'),t('Entity ID'));). This seems rather inconsequential to the issue I am having though and predictably neither version had an impact.

I may try to restart from the beginning and see if I can get it work.

imclean’s picture

Because neither of the Data patches I liked to have been committed each one needs to be applied to a clean version of Data from git.

They're not large so the second one you do could be done by hand.

jamatulli’s picture

Success. At least with the listing of particular fields from a Data table. I used #1538588: Add Entity Label field (Was: allow entity ref fields to point to data items) but not the other patch. It was probably something in my hand application that wasn't working.

lquessenberry’s picture

Are there pushes to implement this into Feeds permanently? I sure would love to be able to safely import CSV files into my own ECK Entities without patching.

dobe’s picture

As far as I know. But it needs testing from the community. Please patch and let us know if you find any issues with ECK.

lquessenberry’s picture

Thanks! I am testing it now and so far it's doing really well.

bobojo’s picture

I have been working on this all day, and I just can't figure out what's wrong. I'm trying to create some forums using the Entity processor for taxonomy terms. It says that it has created the terms, but when I looked in MySQL, only the taxonomy_heirarchy table is being changed. I'm using Feeds SQL with some very simple mapping to set the tid, name, description, and parent. Any ideas why this wouldn't be working properly?

Edit: I just got it to work by removing the mapper to manually assign a tid. I need this mapper to work, so I'm going to keep testing it, but I thought it would be good to mention that in case it helps find the problem.

axel.rutz’s picture

Profile2 mapper: Did not succeed updating profil2 entitite; mapped user id did not overwrite the predefined one.

EDIT: it looks like the problem is that i can't set UID as unique target.

axel.rutz’s picture

Status:Needs review» Reviewed & tested by the community

Nevertheless i'm all for the approach outlined in #212:
* commit this as experimental
* handle the finetuning in dedicated issues
So setting RTBC to attract maintainers.

kreynen’s picture

In 10 days this issue will be 3 years old. Despite contributions from dozens of developers, this doesn't seem to be any closer to being committed. It has been moved to RBTC several times. The first was almost 2 years ago https://drupal.org/comment/5524356#comment-5524356

@twistor is the only active maintainer and hasn't responded to any of the patches that would add entity support in over a year (https://drupal.org/comment/6798294#comment-6798294).

The last push to get something committed was 5 months ago and despite reaching out to @twistor directly to review the patch, there was still no response https://drupal.org/comment/7710367#comment-7710367

Judging by @twistor's commits, he seems focussed on a D8 release. That is great, but doesn't solve the problem of entity support for D7.

If none of the current maintainers are interested in supporting this feature in this project, I think it's time to add additional maintainers or fork the project to entity_feeds or something like that. Trying to troubleshoot issues with specific entities based on patches really isn't an option.

We've been using a patched version of Feeds to import Media entities directly (vs. using something like https://drupal.org/project/media_feeds to add media to a field using a node) for almost 2 years. I can't speak for all possible use cases, but entity support is very reliable for Media.

twistor’s picture

Assigned:Unassigned» twistor

I shall take a gander soon.

alibama’s picture

seriously - chuck this dude a few bucks. https://www.gittip.com/twistor/ just sayin'

twistor’s picture

Alright, I made a few changes and went ahead and committed this with an experimental tag on the processors.

Let's start with the followups.

twistor’s picture

Assigned:twistor» Unassigned
Status:Reviewed & tested by the community» Fixed
Issue tags:-Needs issue summary update

Big thanks to everyone who helped. I know there was a lot of people.

http://drupalcode.org/project/feeds.git/commit/f6aa61d

guillaumev’s picture

Awesome, thanks a lot !

fago’s picture

heureka!

Patrick Danielsson’s picture

Thanks, this is amazing!!

But I can't find the fields for Profile 2 when I want to map the fields, what is wrong? I can only choose the "standard fields", like ID, username etc.

I have chosen the Profile processor, and the Main profile.

Thanks!

dasjo’s picture

yay, glad to see this in now :)

after 278 comments, i'd suggest opening one or more follow up issues because this one's already too big

axel.rutz’s picture

YAY!!!!!

likewhoa’s picture

Mission Accomplished!

alibama’s picture

well - entityforms don't actually work yet, but it's at least a start...

kreynen’s picture

@alibama Please open an issue specifically for any entity that can't be imported in the Feeds queue or that module that controls that entity.

To use the File Entity processor w/ Media requires an additional module like https://drupal.org/project/feeds_media_internet_files to handle mapping the Filename. Without that, Filename shows up the mapping UI, but is empty in the resulting file entity.

It's possible a helper module like that will be required for entityform, but I haven't look at that entity specifically.

ndf’s picture

Great!

scottrigby’s picture

Excellent!
twistor++

Patrick Danielsson’s picture

Maybe i'm the one who does something wrong, but I can't add any Data Processors, i've done everything correct if I read the thread, and I have made a new Data table in the Data module, but still nothing is showing up in Feeds.

Leo Pitt’s picture

@Patrick Danielsson - are you using the 7.x-2.0-dev branch?

Patrick Danielsson’s picture

I'm using this version of Feeds:
http://drupalcode.org/project/feeds.git/commit/f6aa61d

And i'm using Data 7.x-1.x-dev, is there another version I should use.

kreynen’s picture

PLEASE open new issues for problems w/ Data (or really problems w/ anything). This specific issue has been considered fixed since Entity support was added to the 7.x-2.x branch. 7.x-2.0-dev isn't a branch. It's a snapshot release of the 7.x-2.x branch. You should be using a dev snapshot for 2014-Jan-16 and use 7.x-2.x-dev when tagging the issue.

Status:Fixed» Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

travist’s picture

This commit causes some major problems when installing the Taxonomy module after the feeds module is enabled.

Please see #2223853: Installing taxonomy module after this module causes a fatal.

Should I reopen this issue, or just move forward with the issue mentioned above?

rooby’s picture

Stick with the other issue.

robertwb’s picture

PEBKAC error RESOLVED - sorry for the spam.

I am unable to see any parsers for entities such as Profile2 in my Feeds Importer config screens. Structure -> Feeds Importers - [My Feed Importer] -> Processor .

I have applied the most recent dev module and verified that the code from this patch seems to be in there (found "node_form_feedsentityprocessor_feeds_form_alter" anyhow).

Is there something really simple that I am missing?

crystaldawn’s picture

What needs to be done to get Profile2 entities to show up in the list of processors. I can see several but none list Profile2. These are a few I can see:

Content entity processor - EXPERIMENTAL
Node processor

The above 2 processors to me would indicate that this patch is functioning (I am using the dev branch) since I can see 2 processors that are essentially the same thing. Node processor and Content entity processor are pretty much identical. I would expect to see a Profile2 EXPERIMENTAL processor as well, but it's not in the list. Why is it not listed?

Is this left up to the profile2 project (which will likely never happen), or is it suppose to show all entity api defined entities (which is what I would expect, and thus this patch should display profile2 defined entities).

Status:Closed (fixed)» Needs review

Status:Needs review» Needs work
dsnopek’s picture

Status:Needs work» Closed (fixed)

New issues should be new issues! Marking as closed again.

twistor’s picture

For those following along at home, this is being reverted. Weigh in here #2469219: Remove the Generic Entity Processor.