A recurring request for Feeds is that actual feed data should be analyzed and then the results from it should be offered as a mapping source.

This makes every sense particularly for CSV format data where all fields are known per definition (ok, not if we don't include a header row - but that's currently not supported by Feeds at all).

This could be supported by optionally adding the mapping stage as a second step form to the import form or to a node form.

Tasks:

- move mapping form into FeedsProcessor to have it available on admin UI *and* on standalone forms and node forms.
- Offer checkbox on admin mapping form "Define mapping on import"
- If this option is selected, send user to mapping form after submission of the import or node form.

Files: 
CommentFileSizeAuthor
#90 feeds.651478.90.patch34.38 KBroderik
FAILED: [[SimpleTest]]: [MySQL] 2,536 pass(es), 1 fail(s), and 0 exception(s).
[ View ]
#82 feeds.651478.82.patch34.45 KBroderik
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.82.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#73 feeds.651478.72.patch35.96 KBroderik
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.72.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#67 feeds.651478.67.patch35.51 KBroderik
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.67.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#69 feeds.651478.69.proposal.patch35.98 KBroderik
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.69.proposal.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#68 feeds.651478.68.patch35.48 KBroderik
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.68.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#66 feeds.651478.66.patch35.4 KBroderik
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.66.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#65 651478-60_mapping_on_import.patch38.13 KBksalbrecht
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-60_mapping_on_import.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#63 651478-59_mapping_on_import.patch57.92 KBrbayliss
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-59_mapping_on_import.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#58 651478-58_mapping_on_import.patch42.72 KBalex_b
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-58_mapping_on_import.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#48 feeds-mapping-on-import-651478-48.patch43.35 KBmorningtime
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds-mapping-on-import-651478-48.patch.
[ View ]
#47 feeds.651478.47.patch44.17 KBadityakg
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.47.patch.
[ View ]
#39 feeds.651478.39.patch43.84 KBadityakg
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.39.patch.
[ View ]
#35 651478-35_mapping_on_import.patch33.05 KBalex_b
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-35_mapping_on_import.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#34 651478-34_mapping_on_import.patch32.01 KBalex_b
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-34_mapping_on_import_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#33 feeds.651478.33.patch33.58 KBadityakg
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.33.patch.
[ View ]
#33 feeds-import-form.png23.08 KBadityakg
#33 feeds-node-form.png22.79 KBadityakg
#31 feeds.651478.31b.patch33.59 KBadityakg
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.31b.patch.
[ View ]
#29 feeds.651478.29.patch18.96 KBadityakg
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.29.patch.
[ View ]
#28 feeds.651478.28.patch15.59 KBadityakg
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.28.patch.
[ View ]
#5 feed_example.patch3.28 KBfunkmasterjones
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feed_example.patch.
[ View ]
#5 feed_example.png42.56 KBfunkmasterjones

Comments

netentropy’s picture

is this currently a feature available in dev version or in the works?

B-Prod’s picture

Subscribing

Why the mapping form should appear after the import form? It could be a part of the feeds fieldset or a new tab for feed nodes. This last solution seems better for me.

shrimphead’s picture

Subscribing

TimG1’s picture

Subscribing.

funkmasterjones’s picture

StatusFileSize
new42.56 KB
new3.28 KB
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feed_example.patch.
[ View ]

Heres a little util to help with mapping

In the admin ui mapping area, a textfield appears, enter the url/path and a parsed example will be given, you can then use this info to do your mapping.

It works with any parser and FeedsHTTPFetcher(tested) and FeedsFileFetcher(untested). It is independent of the processors/mappers

This is a very simple hack, it doesn't provide new sources so its best used when there are no sources (text entry instead of select box) like with exhaustive xml parsers

This is not a final solution, but rather a little pill to help relieve some headaches (at least mine anyways)

Let me know what you think...

Cheers

NOTE:
the patch was made in windows and on top of the dev version of feeds (not alpha1), not sure if this will cause any problems but the fix is simple enough it could be done by hand

whatdoesitwant’s picture

subscribing, will check patch. +1 for original approach though

geerlingguy’s picture

alex_b’s picture

#7: This feature is essentially a requirement for #631104: Extensible XML parser (mapping more sources).

alex_b’s picture

#5: I'm open to an example feed on the mapping page. However, in order to implement it, we should move the mapping forms into processors (just like the config forms are in processors). Thus sth like an extensible XML parser or an exhaustive parser could be realized as an add-on module.

As to requiring an example feed for CSV parser, I'm not convinced of the advantages of the current implementation in the patch in #5. Having to add an example feed first does seem slightly worse than being able to freely enter the titles of headers (even if it's slightly better it's much more code to maintain for a not 100 % solution of the problem).

A 2 step form could be a better solution here as well. First enter example feed, second pick mappings. The same 2 step form could be used on the end user facing UI if desired (just as I described in my initial description of the issue).

netentropy’s picture

Since I am experimenting with both, you might look at the node import module, it has a page where you first can load (in this case the csv file) and then it pulls the fields on for mapping before import

alex_b’s picture

#10 - I'm familiar with node import, also user import. And yes: the 2 step import process (1: upload&parse, 2: map&import) is exactly what I find very useful for some use cases. It's exactly what we should be shooting for.

This 2 step import process should be useable on both - the frontend and the backend. So the frontend (one off import forms, feed nodes) should support an "add feed, submit, map, submit" workflow while the backend should support a very similar workflow aiding creating mappings for unpredictable sources (similar to what's suggested in #5).

1)

The first step is going to be to move mapping form handling from feeds_ui into the plugins (most likely FeedsProcessor.inc). Once it's there, mapping can be used for both, frontend and backend tasks and moreover it can be modified better by extending classes.

2)

Introduce a two step form handling on the mapping form that first imports and parses a given source and then uses the results of the parsing stage to present a mapping form (with dynamically generated source options). At this stage a user or a site builder populates the mapping form. If this form is being used on an importer form (admin/build/feeds/...) the mapping is merely saved (as it is right now). If this form is being used in an import, the next step would be to use this mapping for the import.

3)

Once mapping forms are handled by FeedsProcessors and a two step form handling is implemented, we can give the site builder a new option on the importer's settings form:

[x] Mapping on import
If checked, offers a mapping form between submitting a source and import.

4)

Parsers like e. g. CSV parser can start using mapping on import on importer configuration forms. I'm leaning towards not making this an optional step for site builders but a required one. So *if* a specific parser needs an example feed to offer mapping sources, it should *require* it. This is to avoid alternative usage patterns that are difficult to maintain.

drewish’s picture

subscribing

derekwebb1’s picture

Category:feature» bug

Yeah, current mapping is unusable. Scanning the source would be a perfect solution.

Right now, it maps "Description" to <content: encoded> which is full of additional garbage that I really don't want in teasers or even in the node proper.

This is a RSS 2 feed. I really just want the Description to be the actual <description>, not the <content: encoded>. I may try out this patch at http://drupal.org/node/651478#comment-2466100 as that sounds pretty promising.

Thanks for your efforts.

Cheers, Derek

PS: setting this as a bug since it is not a feature request but rather a request for a feature to be fixed and made functional.

An example of the RSS feed:

<item>
<title>The title proper</title>
<link>Some title link</link>
<guid isPermaLink="true">http://example.com.com/somepage.html</guid>

<description>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas est ligula, iaculis sed volutpat eu, rhoncus ac lectus. Etiam sed lacus et turpis elementum fringilla id a mauris. Donec sed nisl justo. Proin vestibulum sagittis sapien et accumsan. Aliquam non elementum est. Donec fermentum varius euismod. Etiam pharetra mattis ultrices.</description>

<content:encoded>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas est ligula, iaculis sed volutpat eu, rhoncus ac lectus. Etiam sed lacus et turpis elementum fringilla id a mauris. Donec sed nisl justo. Proin vestibulum sagittis sapien et accumsan. Aliquam non elementum est. Donec fermentum varius euismod. Etiam pharetra mattis ultrices. Vestibulum rutrum volutpat nisl, id ultricies eros aliquam sed. Proin interdum, elit eget malesuada rutrum, quam mi ullamcorper quam, at accumsan mauris mauris ut tellus. Donec a felis in neque fringilla aliquet ut vel risus. Morbi ultrices ipsum quis elit ullamcorper volutpat. Curabitur id tellus non mauris varius semper nec non turpis. Integer ac mi orci. Sed sem purus, vulputate ac iaculis id, cursus molestie justo. Pellentesque at sodales leo. Etiam purus sapien, dapibus non ornare non, eleifend adipiscing nibh. Donec malesuada hendrerit felis, ut porta augue tempor eleifend. Integer pellentesque lectus id est dictum dapibus. Nullam et dolor et sem rhoncus congue quis vitae magna. Duis egestas libero at elit rutrum a tempus enim scelerisque. Maecenas mauris neque, gravida id ornare nec, tempor in lectus. Ut at quam mauris, eu hendrerit tellus. Morbi eu mi et lorem dapibus sollicitudin vitae vel felis. Suspendisse id ligula magna. Aliquam porttitor aliquam aliquet. Integer quis dolor nec odio accumsan convallis. Aenean eros augue, pharetra quis pulvinar dapibus, sagittis sed ipsum. Ut id sapien massa. Maecenas vel libero faucibus sem varius pharetra a eu nisl. Fusce nec eros diam, nec ullamcorper dolor. Suspendisse bibendum, quam tristique elementum lobortis, enim odio adipiscing est, sit amet cursus erat nibh et quam. Vivamus lobortis gravida lobortis. Proin sed bibendum justo. Donec consectetur mauris at risus molestie quis adipiscing mi rhoncus. Curabitur pellentesque congue elit suscipit facilisis. Maecenas vel erat nec enim gravida tempor sit amet eu felis. Duis non commodo risus. Nullam aliquam, velit sit amet dignissim rutrum, eros mauris imperdiet felis, a tincidunt eros erat quis neque. Mauris lacus orci, varius non condimentum tristique, varius ut magna. Vivamus blandit volutpat rhoncus. Praesent vitae quam leo, a tempus augue. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Nunc adipiscing ante nec est adipiscing dignissim. etc. etc. etc. etc. etc. etc. ad nauseum....

dvbii’s picture

Version:6.x-1.x-dev» 6.x-1.0-alpha11
Category:bug» feature

suscribing!

alex_b’s picture

Version:6.x-1.0-alpha11» 6.x-1.x-dev

This remains a feature request for -dev.

funkmasterjones’s picture

I made a generic xml parser with an updated mapping example patch found here:http://drupal.org/node/631104#comment-2600030. (screenshots are there too)

The xml parser should work on any feed and it will create all mapping sources for every element. I think this closely resembles FeedAPIMapper mapping on import functionality

I can't say its perfect because I made it only for my purposes but it seems to be quite flexible, so feel free to use it and improve upon it.

alexb, I hope that this code will be useful in creating a true 2 step mapping on import process as it provides example preview and generates dynamic source targets

infojunkie’s picture

subscribe

webdeli’s picture

subscribe

AntiNSA’s picture

subscribing

srobert72’s picture

Subscribing

morningtime’s picture

+1

mirzu’s picture

subscribe

adityakg’s picture

subscribing.

morningtime’s picture

Your XML maps fine, but has little to do with mapping on import, which takes place on the node form itself. Now you still have to create a custom feeds importer for each XML file. If this mapping could be done on the Feed Nodes, it would save an unnecessary extra step.

FeedAPI could do this: Feed Node with remote XML > Custom Mapping after detecting XML > Import data as Nodes

Feeds module requires an extra step: Feed Importer for each custom mapping(!) > Custom XML Mapping > Create Feed Node > Import data as Nodes

FreddieK’s picture

Subscribing

andrewlevine’s picture

subscribe

twistor’s picture

subscribe

adityakg’s picture

Assigned:Unassigned» adityakg
StatusFileSize
new15.59 KB
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.28.patch.
[ View ]

I am working on this. I followed the steps specified by #11, but instead I put the mapping form handling in FeedsConfigurable.inc and introduce new method FeedsConfigurable::renderMappingForm() which can be used by FeedsSource or FeedsProcessor.

Progress:
Moving Mapping Form handling out of feeds_ui. (DONE)
Introduce two steps mapping form when importing, both importing by standalone form or by using node. (IN PROGRESS)

adityakg’s picture

StatusFileSize
new18.96 KB
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.29.patch.
[ View ]

Sorry, please ignore the patch above. It contains some bugs that I didn't realize beforehand.

alex_b’s picture

Status:Active» Needs work

Great to see this moving :-)

- I get a 'Call to undefined function _feeds_processor_format_options()' when editing the default configuration 'Feeds'.
- The mapping form is specific to FeedsProcessors, let's keep it there. FeedsConfigurable is what its name says: the base class for configurable entities.
- Let's not put the mapping form into the config form, that's going to be too crowded in processors like FeedsNode processor.
- Please watch out for whitespace, there shouldn't be any trailing spaces in your patch, there should always be one empty line at the end of a file.

adityakg’s picture

Status:Needs work» Needs review
StatusFileSize
new33.59 KB
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.31b.patch.
[ View ]

Finished the issue's specification.
1. Now 2-step form will be encountered by user if 'Define mapping on import' option in mapping is checked.
2. mappingForm() is moved to FeedsProcessor detaching all dependencies with feeds_ui module EXCEPT 2 files: feeds_ui.js and feeds_ui.css

There are a lot of subscribers here, please review it :)

dawehner’s picture

+++ feeds.module 6 Jul 2010 09:33:11 -0000
@@ -125,7 +125,25 @@
+        'page arguments' => array('feeds_import_mapping_form', $importer->id, 1),

Just general notice, if page arguments get integers, they treat them as dynamic argument placeholders.

+++ feeds_ui/feeds_ui.admin.inc 6 Jul 2010 05:37:04 -0000
@@ -382,8 +373,8 @@
+  }  ¶

sry, tabs again.

+++ includes/FeedsSource.inc 6 Jul 2010 06:39:56 -0000
@@ -287,5 +287,5 @@
-  }
+  }  ¶

here is a tab.

+++ plugins/FeedsProcessor.inc 6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@

+  // Get the FeedsImporter object
+  $importer = feeds_importer($this->id);

Use spaces instead of tabs here.

+++ plugins/FeedsProcessor.inc 6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@

+  // Get the FeedsImporter object
+  $importer = feeds_importer($this->id);

...
+function feeds_mapping_form_help() {
+  return t('
+  <p>
+  Define which elements of a single item of a feed (= <em>Sources</em>) map to which content pieces in Drupal (= <em>Targets</em>). Make sure that at least one definition has a <em>Unique target</em>. A unique target means that a value for a target can only occur once. E. g. only one item with the URL <em>http://example.com/story/1</em> can exist.
+  </p>
+  ');

i would move

out of the t() function. see user_help as an example.

+++ plugins/FeedsProcessor.inc 6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@
+        '#title' => 'Define mapping on import',
+        '#description' => 'If this is checked, the mapping form here will be disabled and instead user will be asked for the mapping configuration after the first import.',

This texts should be translatable.

Powered by Dreditor.

+++ feeds.module 6 Jul 2010 09:33:11 -0000
@@ -125,7 +125,25 @@
+        'page arguments' => array('feeds_import_mapping_form', $importer->id, 1),

Just general notice, if page arguments get integers, they treat them as dynamic argument placeholders.

+++ feeds_ui/feeds_ui.admin.inc 6 Jul 2010 05:37:04 -0000
@@ -382,8 +373,8 @@
+  }  ¶

sry, tabs again.

+++ includes/FeedsSource.inc 6 Jul 2010 06:39:56 -0000
@@ -287,5 +287,5 @@
-  }
+  }  ¶

here is a tab.

+++ plugins/FeedsProcessor.inc 6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@

+  // Get the FeedsImporter object
+  $importer = feeds_importer($this->id);

...
+function feeds_mapping_form_help() {
+  return t('
+  <p>
+  Define which elements of a single item of a feed (= <em>Sources</em>) map to which content pieces in Drupal (= <em>Targets</em>). Make sure that at least one definition has a <em>Unique target</em>. A unique target means that a value for a target can only occur once. E. g. only one item with the URL <em>http://example.com/story/1</em> can exist.
+  </p>
+  ');

i would move

out of the t() function. see user_help as an example.

+++ plugins/FeedsProcessor.inc 6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@

+  // Get the FeedsImporter object
+  $importer = feeds_importer($this->id);

Use spaces instead of tabs here.

+++ plugins/FeedsProcessor.inc 6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@
+        '#title' => 'Define mapping on import',
+        '#description' => 'If this is checked, the mapping form here will be disabled and instead user will be asked for the mapping configuration after the first import.',

This texts should be translatable.

Powered by Dreditor.

adityakg’s picture

StatusFileSize
new22.79 KB
new23.08 KB
new33.58 KB
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.33.patch.
[ View ]

Thanks dereine for the review. #32 issues have been fixed. Did minor tweaks and fixing coding conventions/tabs/etc. Now the 'mapping on import' form will be available as a tab in the node-edit page and import form.

Please see the attached image to be more clear.

alex_b’s picture

StatusFileSize
new32.01 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-34_mapping_on_import_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

So, I started to review then ran out of time. All I got to do is consolidate the feeds_get_form() wrappers. In #33 configFormValidate() and configFormSubmit() was called on the mappingForm().

aditya_kristanto: please review. These changes may open up some adjustment work in the rest of the patch for you.

alex_b’s picture

Status:Needs review» Needs work
StatusFileSize
new33.05 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-35_mapping_on_import.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

This is going to be larger refactoring work.

When editing a mappin gon import, the patch currently stores mapping for the importer (= all sources), where it should store it only for the source at hand.

This will mean that a processor should have source config 'mappings' that is populated when mapping on import and used over the processor's general config if available.

This will mean that we'll have to pass a FeedsSource object into map() to be able to check whether there is a $source->config['mappings'] and use it over $this->config['mappings'] if present.

This will mean that we have to rethink how a mapping is populated on form submission. Currently addMapping(), addUnique() etc. are processor-level methods. Trying to retool them to store source-level information gets ugly very quick - we should instead move to an entirely different paradigm where we save the entire mapping configuration with each form submission. Thus we can populate with $object->addConfig(array('mappings' => $mappings)); no matter whether $object is FeedsSource or a FeedsProcessor and thus elegantly copy the entire configuration from FeedsProcessor to FeedsSource with the first override.

Before we tackle Mapping on import, we should address these issues:

#849834: Generalize feeds_config_form() to feeds_form()
#849840: Mapping form: submit full mapping on every submission

Further issues I found with the patch in #33:

- mapping form on admin/build/feeds does not work as expected
- feeds_mapping_form() needs to accept $form_state, not $form, function signature can be simplified.
- When mapping on import is selected, we should not hide the mapping form on a processor, but allow the user to define a default.

FWIW, I attach here the patch showing how far I got with refactoring before I gave up for the above listed reasons.

adityakg’s picture

@alex_b: sorry i was away yesterday.

Thanks for the review. I completely agree with #849840. Removing addMapping() etc will certainly reduce the complexity of the mapping form, will do it in a day or two.

I completely missed the fact that it is tied to importer instead of source! I will look through it after #849840.

alex_b’s picture

#850638: Introduce FeedsSource::preview() will make this patch a smidge easier.

adityakg’s picture

Another patch. Needs review. Now it stores the mapping configuration in FeedsSource instead of FeedsImporter.

#849840: Mapping form: submit full mapping on every submission is also fixed. FeedsSource and FeedsProcessor is modifying their mapping configurations without the help of addMapping, setUnique and removeMapping.

@todo: Move FeedsDataProcessor::addMapping() to FeedsDataProcessor::mappingFormSubmit()

adityakg’s picture

StatusFileSize
new43.84 KB
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.39.patch.
[ View ]

Sorry, forgot to attach the patch.

andrewlevine’s picture

@alex_b

Just a general question related to this patch: I'm not sure what the rationale is for having settings in the source vs. the importer. Although it seems like some settings are better suited to be in the source, as far as I can tell, all the settings in the importers could just as easily be in the source as well. The reason I bring this up is that I have several sources which utilize the same mappings, so obviously for my case it is better to have the mappings in one place (the importer), saved as an exportable. Have any comment?

I think mimicking the views model of Displays might make more sense: There would be a default importer (what is now the importer), and as many other importers as needed of different types (what are now sources). The different types could require settings that the default didn't (eg a feed URL/file location). Besides these settings that are required to be different than the default, the importers would inherit the default settings (think menu settings in a Views page display).

alex_b’s picture

andrewlevine:

I guess that much is clear: Importer level settings are the settings that you'd want to be in place for all imports. Source level settings are the settings that you are going to decide on on a per import basis.

This distinction is sometimes a fine line and often you could argue that you'd like to be able to change just about anything of an importer level setting on a per import (per source) basis. In fact, FeedAPI supported that. Out of the box, FeedAPI supported overrides of importer settings on a per import basis but it did not support well different UIs for those different usage contexts of the same settings form. the result was messy long forms on feed nodes.

Feeds now distinguishes between importer level settings (configForm()) and import level settings (sourceForm()). This approach recognizes the fact that configuring an importer (a *site builder* task) and importing content (a *site manager* task) are two very different user stories.

The reason why making the mapper form now usable on a source level is hard is because the handling of the mapper form never assumed to be used on a source level. Simpler forms, like for instance the delimiter selection on the CSV parser where very easy to expose on a per source level.

In regards to a Views UI like behavior with default importers and overrides: I don't see that happening. Importer configurations re fairly simple, I think that being able to copy them is already helping a lot in cases where similar importers need to be configured.

andrewlevine’s picture

@alex_b:

I certainly do not want to bring up features that have already been implemented and cut or that would complicate the UI for site managers. I just want to mention that since here we have to implement a system in which a source configuration option overrides an importer configuration option, it might be a great time to at least allow that system to be exposed for other options. Then in the future we could possibly be able to expose other options to both the sourceForm and configForm in the future.

An idea: when you call getConfig() on a source, instead of just getting the source config it gets the values from the importer config and merges over the values from the source config. configForm (renamed to importerForm?) and sourceForm would still be independent, but the APIs that deal with the data (getConfig/addConfig/setConfig) would be joined. FeedsSource::save() would only save the keys whose values were in sourceDefaults.
This would have some nice side-effects:
- Mapping in FeedsSource happens the exact same way as mapping in FeedsImporter
- Custom sourceForm configuration defaults can be saved in an importer exportable
- Any importer configuration option can be overridden by a FeedsSource through the normal sourceForm

Something to think about...I don't think it would be too much more work than doing the same thing but just for mappings.

Summit’s picture

Subscribing, greetings, Martijn

meatbag’s picture

Subscribing

loliearn’s picture

Subscribing

aaron_michels’s picture

subscribing

adityakg’s picture

Status:Needs work» Needs review
StatusFileSize
new44.17 KB
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds.651478.47.patch.
[ View ]

Hi all,

Re-rolled the patch to CVS HEAD and did minor modifications. It should now have working mapping on import on all processors included with Feeds. Setting to needs review.

morningtime’s picture

StatusFileSize
new43.35 KB
FAILED: [[SimpleTest]]: [MySQL] Invalid patch format in feeds-mapping-on-import-651478-48.patch.
[ View ]

@adityakg, I tested your patch. At first I couldn't apply it, so I made the changes manually.

But after doing so, it works perfect! I even use it in combination with a custom XML xpath parser (available in another feeds issue). I had to slightly change that xmlparser to use the custom mapping (and add the custom xpath config setting). But it was easy to do. I have tested this patch now on multiple feeds with 100+ items per feed. Custom mapping is now working perfectly on my system. Drupal 6.17, using feeds_imagegrabber and feeds_xmlparser add-on modules.

From my opinion: go ahead, put this in feeds.

(Attached is the same patch from #47, only re-created for my personal purpose. No changes.)

morningtime’s picture

One more thing, a feature request:

I use the Node Clone module, http://drupal.org/project/clone, and when I clone a feed node with custom mapping, that mapping is reset to the default.

It's not a bug, but would save time to copy the existing custom mapping to the cloned node.

adityakg’s picture

@morningtime:

Thanks for the review. The patch should be able to cleanly patched on the current CVS HEAD of feeds, but maybe not for the older version of feeds. We will need more review (especially on the code and the design aside from the functionality that you have tested).

Unfortunately the feature request might not be done atm, at least not before this is incorporated into Feeds. But afterwards you can always open up a new feature request issue :)

morningtime’s picture

@adityakg Great, I'm trying to build a "copy mapping from other node" autocomplete type field. That would simplify things. I'll post it soon as I get around to it.

morningtime’s picture

One thing, a little "bug", I noticed is: each time you access the node's Mapping tab, the FeedsHTTPFetcher is called and re-downloads the file. This should not be necessary, it could re-use the already existing file. Since the feed is already fetched on node creation, you can avoid re-fetching on the Mapping tab.

elliotttf’s picture

Status:Needs review» Reviewed & tested by the community

I can also confirm the patch from #47 works.

As for #52, this might actually be desirable behavior. If your original feed file changed for some reason (uploaded to the server by a script or whatever) you would want to verify against the latest file, not necessarily one that already exists. If it's not a huge waste of resources it's probably not a major issue.

morningtime’s picture

I have found 2 tiny bugs (though they don't influence operation):

1. In combination with the domain access module, I can only set mappings on import from the main domain (id=0). On subdomains, I cannot map on import (I try to add a new source->target mapping, click save, page reloads without my new mapping - no error, no message). I have to open the node on the main domain to make it work. There are no watchdog errors, no drupal message.

2. Mapping on import tries to preview the mapping sources, which works great for example with the FeedsCSVParser. You get a nice select input dropdown. However, when I use a custom FeedsXMLParser, I get the following error in watchdog:

Invalid argument supplied for foreach() in /var/www/vhosts/123.com/httpdocs/sites/all/modules/patched/feeds/includes/FeedsBatch.inc on line 372.

That line is:
foreach ($this->items[0] as $k => $v) {

The watchdog bug could be prevented by testing for array before the foreach: "if (is_array($this->items[0])) { ... }"

I think the FeedsXMLParser is too custom to work with this getMappingSources(). So we need a check to prevent a PHP error.

--------

Everything works fine, but there are these little quirks.

morningtime’s picture

@#53: depends on how big your feeds are. I implemented a custom logic in FeedBatch.inc, to only re-download once every expiry period. Otherwise I'd download 500MB feeds on every feeds cron run. In my extreme case, that would be every minute.

rbayliss’s picture

Thanks so much for this patch in #47. It seems to work perfectly. One issue: I accidentally put in a URL of feed://domainhere/feedhere (rather than http://domainhere/feedhere). When I went to mapping, I got an immediate WSOD with no watchdog messages as soon as it tried to check the format of the feed. Obviously this is a case of user error, but it would be awesome if it failed a little more gracefully (the standard import gives a drupal message of error -1003).

PHP log errors:
[23-Aug-2010 14:15:26] PHP Fatal error: Call to undefined function drupal_get_bootstrap_phase() in /Sites/Multisite/includes/common.inc on line 3874

imclean’s picture

Trying to apply patch #47 on CVS but getting 2 failures: FeedsBatch.inc @ 4, FeedsTermProcessor.inc @ 6.

I'm also having trouble working out where the changes go. To make it easier to patch by hand, please use -p with the diff command.

alex_b’s picture

Status:Reviewed & tested by the community» Needs work
StatusFileSize
new42.72 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-58_mapping_on_import.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Rerolled after recent commits caused a conflict (#57).

Running tests with the patch applied results in 2363 passes, 534 fails, and 19 exceptions.

I'd love to get first the prerequisites for this patch committed:

#849834: Generalize feeds_config_form() to feeds_form()
#849840: Mapping form: submit full mapping on every submission

(both quite straightforward) and then roll this patch. I personally don't plan to work on this in the next weeks.

AntiNSA’s picture

I really hope thiscan be made part of feeds. two weeks is so long. so are you saying this patch works now?

AntiNSA’s picture

warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'FeedsNodeProcessor_feeds_config_form' was given in /home/cyberfan/htdocs/includes/form.inc on line 376.
warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'FeedsHTTPFetcher_feeds_config_form' was given in /home/cyberfan/htdocs/includes/form.inc on line 376.
warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'FeedsNodeProcessor_feeds_config_form' was given in /home/cyberfan/htdocs/includes/form.inc on line 376.

rbayliss’s picture

Hey Alex,
Thanks for re-rolling this. If I understand correctly, these two prerequisites are dealt with in this patch. You're asking for them to be separated out and moved into their respective issues?

alex_b’s picture

#61: would make working with this patch much easier, yes.

rbayliss’s picture

StatusFileSize
new57.92 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-59_mapping_on_import.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Ok, I think I satisfied the prerequisites on

#849834: Generalize feeds_config_form() to feeds_form()
#849840: Mapping form: submit full mapping on every submission

Attached is a patch rolled against HEAD + these two patches. Ideally this patch would be rolled directly against HEAD, but I don't have time to make that modification at the moment. I'm getting a lot of failed tests, but most of them are coming from moving the mapping/config forms to the plugin specific URL's.

alex_b’s picture

Assigned:adityakg» Unassigned

#849834 and #849840 are committed now, #849840 with some modifications, so #63 does not apply anymore. Don't have time to work on this patch ATM.

Setting to unassigned as adityakg hasn't been around for a while.

ksalbrecht’s picture

StatusFileSize
new38.13 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 651478-60_mapping_on_import.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

-Rolled against head and modified addMapping function in tests.

Testing comes up with 16 failures, all from sitemap parser.

roderik’s picture

StatusFileSize
new35.4 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.66.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

alex_b is churning out code like there's no tomorrow. Here's another one, rerolled against the latest HEAD which has #908964: Break out job scheduler applied.

FWIW, I reviewed the patch (and learned a lot) at the end of august, after reading the comment in #50. (Only after the review, I saw that #849840 was still open and needed work before this :) -- and got distracted before I could do proper work on it.) Just reviewed it more quickly again.

The code looks good to me but I didn't know any Feeds internals beforehand -- and I haven't had much exposure to OO programming.

The only question that I had, as a result of looking at FeedsProcessor::getMappings() and FeedsProcessor::mappingFormSubmit(), was:
"don't we want to define 'source'? Like a 'protected $source' in the definition of class FeedsProcessor? Because it seems to be the only variable that is set without being defined in a (parent) class."
But I'm not even 100% sure if that makes sense.
So I won't change the status.

roderik’s picture

Status:Needs review» Needs work
StatusFileSize
new35.51 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.67.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

...and between the time I checked out HEAD and uploaded this patch here, alex_b had done a zillion CVS commits :)

re-rerolled.

note: patches in #67 - #69 apply cleanly to 1.0-beta6 & 1.0-beta7. I didn't spot these releases until later :)

roderik’s picture

Status:Needs work» Active
StatusFileSize
new35.48 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.68.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

...and after that porting, I actually tested stuff :)

Changed one "$values['op'] == 'Add'" to $values['op'] == t('Add')", which makes the mapping form work in non-English environments too.
(The rest of the patch is the same, except you may need a '-p1' option to apply this file. alex_b's latest changes to HEAD didn't introduce differences here.)

I know I'm creating 'noise' here, but wanted to upload a patch this little fix separately... before recoding something which I want to ask your opinion about, in a little while.

roderik’s picture

Status:Active» Needs review
StatusFileSize
new35.98 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.69.proposal.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

So here's a proposal / question if I am approaching things the right way... (I haven't had that much exposure to feeds)

I'm fiddling with a parser (working version of Feeds XPath Parser) which preprocesses input for all mapped fields, and discards the rest of the input - because it doesn't want to bother processing stuff it doesn't need.
This is a valid approach, right?

This presents an issue in the 'mapping on import' stage. It discards (does not return) all source fields it finds in the XML, which aren't mapped already... which means you can't add any mappings for those fields :)

So,
- in the call from feeds_import_mapping_form to ...parser->parse(), should we signify to the that we're in the pre-mapping stage (meaning "we're really only interested in calling $batch->getMappingSources() later, so make sure you populate $batch with all the source fieldnames and don't bother parsing any info you don't need) ?
- should we do that using some extra property in FeedsImportBatch? (I guess from a purely conceptual standpoint, it's not a 'Batch' property. But otherwise we'd need a third argument to all FeedsParser::parse() functions, which is more intrusive for existing code.)
- if we do that... then $loc_id is not needed anymore as the 2nd argument to FeedsProcessor::mappingForm, right?

You can see what I did, if you compare the patch in #68 against this one. Few changes.

(Detail: the behaviour in FeedsProcessor::mappingForm may have changed slightly for so-far-unused edge cases. That's because the concept of arguments changed slightly:
- $loc_id, which I deleted, was 'the location from where the mappingForm is called' (default 'config' -- set to 'import' for mapping-on-import)
- $batch->context, which I introduced, is 'the context in which a batch is run' (default 'import', or actually unset: when called from the config screen $batch is not set at all -- set to 'mapping' for mapping-on-import)
I don't think it makes a practical difference.)

Summit’s picture

Status:Needs work» Needs review

Subscribing, greetings Martijn

tema’s picture

Subscribing

jibberish’s picture

sunbscribing

roderik’s picture

StatusFileSize
new35.96 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.72.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Re-rolled against beta10.

(Nothing changed except the context lines around some changes.
State is still 'same as when adityakg last worked on it, except the change in comment #69')

JoshuaBud’s picture

Being able to map after selecting the feed based on available feed fields is going to ROCK! Subscribing! I tried to manually apply the patch above and got an error after getting it mostly working.

Everything was fine but when I click the Mapping tab under the Feed content node that I added and RSS feed too I get the following error.

Fatal error: Call to undefined method FeedsNodeProcessor::mappingForm() in /home/appvent/public_html/go-protest/sites/all/modules/feeds/feeds.pages.inc on line 216

Maybe I made an error somewhere else but I took my time applying it to the latest beta version of Feeds module.

;$Id: feeds.info,v 1.5 2010/09/15 19:27:42 alexb Exp $
name = Feeds
description = Aggregates RSS/Atom/RDF feeds, imports CSV files and more.
package = Feeds
dependencies[] = ctools
dependencies[] = job_scheduler
core = 6.x
php = 5.2

; Information added by drupal.org packaging script on 2010-10-29
version = "6.x-1.0-beta10"
core = "6.x"
project = "feeds"
datestamp = "1288369547"

atolborg’s picture

Is this feature obtainable now?

I am pulling in events from Facebook using the parser_ical. Its working fine, but I would like to map more fields - e.g. the event organizer.

Since I am not very experienced, I would like to know if its 'safe' to apply the path and if this will give me access to map the entire ical feed?

morningtime’s picture

Has anyone attempted a patch for D7 mapping on import?

claar’s picture

Exactly what I need. Subscribing.

Taxoman’s picture

Subscribing

cmcintosh’s picture

I am doing work over at the old feedapi_mapper module to migrate it into the new Feeds API. I will have support for mapping to various types of fields along with an API to extend it for other types of custom fields.

cmcintosh’s picture

looks like reading through it would be better to work and commit against this issue. I would also like to download files via the mapper. It could be handy for things like images that sometimes goes away after a time.

uxjam’s picture

Also wondering if this is getting into a D7 version of Feeds - would be great for working with non-standard RSS feed fields.

roderik’s picture

StatusFileSize
new34.45 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds.651478.82.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

re-roll for beta11.
Still: see #69 for outstanding comments.

pyrello’s picture

Subscribing... we are trying to move from node import to feeds and this would be crucial to us being able to do that.

danielb’s picture

Subscribing. I recently discovered this module, and I'm writing a feeds importer for node export. Not comfortable with the competence of users to be able to map this stuff ahead of time. Also am familiar with the approach 'node import' used.

kirkcaraway’s picture

Does anyone know if this patch is going to be incorporated and released? Thanks

morningtime’s picture

Version:6.x-1.x-dev» 7.x-2.x-dev

+1 for D7 version of this

roderik’s picture

Version:7.x-2.x-dev» 6.x-1.x-dev

Of course. And I surely would if I had time.

It's just that 'version tags' in an issue follow the work being done. Not people's wishes. Anything else confuses the work process.

Summit’s picture

Yes please would love to see this on D7, especially Xpath parser doesn't give any log if it is working ok!
Thanks a lot in advance for building this for D7.
greetings, Martijn

kvyn’s picture

D7 +1

roderik’s picture

StatusFileSize
new34.38 KB
FAILED: [[SimpleTest]]: [MySQL] 2,536 pass(es), 1 fail(s), and 0 exception(s).
[ View ]

re-roll against 6.x-1.0-beta13; still see #69 for outstanding comments.

jshimota01’s picture

subscribing D7 +1

Status:Needs review» Needs work

The last submitted patch, feeds.651478.90.patch, failed testing.