Mapping on import

is this currently a feature available in dev version or in the works?

Subscribing

Why the mapping form should appear after the import form? It could be a part of the feeds fieldset or a new tab for feed nodes. This last solution seems better for me.

Subscribing

Subscribing.

Heres a little util to help with mapping

In the admin ui mapping area, a textfield appears, enter the url/path and a parsed example will be given, you can then use this info to do your mapping.

It works with any parser and FeedsHTTPFetcher(tested) and FeedsFileFetcher(untested). It is independent of the processors/mappers

This is a very simple hack, it doesn't provide new sources so its best used when there are no sources (text entry instead of select box) like with exhaustive xml parsers

This is not a final solution, but rather a little pill to help relieve some headaches (at least mine anyways)

Let me know what you think...

Cheers

NOTE:
the patch was made in windows and on top of the dev version of feeds (not alpha1), not sure if this will cause any problems but the fix is simple enough it could be done by hand

subscribing, will check patch. +1 for original approach though

Possible duplicate of #631104: Extensible XML parser (mapping more sources)?

Since I am experimenting with both, you might look at the node import module, it has a page where you first can load (in this case the csv file) and then it pulls the fields on for mapping before import

subscribing

Yeah, current mapping is unusable. Scanning the source would be a perfect solution.

Right now, it maps "Description" to <content: encoded> which is full of additional garbage that I really don't want in teasers or even in the node proper.

This is a RSS 2 feed. I really just want the Description to be the actual <description>, not the <content: encoded>. I may try out this patch at http://drupal.org/node/651478#comment-2466100 as that sounds pretty promising.

Thanks for your efforts.

Cheers, Derek

PS: setting this as a bug since it is not a feature request but rather a request for a feature to be fixed and made functional.

An example of the RSS feed:

<item>
<title>The title proper</title>
<link>Some title link</link>
<guid isPermaLink="true">http://example.com.com/somepage.html</guid>

<description>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas est ligula, iaculis sed volutpat eu, rhoncus ac lectus. Etiam sed lacus et turpis elementum fringilla id a mauris. Donec sed nisl justo. Proin vestibulum sagittis sapien et accumsan. Aliquam non elementum est. Donec fermentum varius euismod. Etiam pharetra mattis ultrices.</description>

<content:encoded>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas est ligula, iaculis sed volutpat eu, rhoncus ac lectus. Etiam sed lacus et turpis elementum fringilla id a mauris. Donec sed nisl justo. Proin vestibulum sagittis sapien et accumsan. Aliquam non elementum est. Donec fermentum varius euismod. Etiam pharetra mattis ultrices. Vestibulum rutrum volutpat nisl, id ultricies eros aliquam sed. Proin interdum, elit eget malesuada rutrum, quam mi ullamcorper quam, at accumsan mauris mauris ut tellus. Donec a felis in neque fringilla aliquet ut vel risus. Morbi ultrices ipsum quis elit ullamcorper volutpat. Curabitur id tellus non mauris varius semper nec non turpis. Integer ac mi orci. Sed sem purus, vulputate ac iaculis id, cursus molestie justo. Pellentesque at sodales leo. Etiam purus sapien, dapibus non ornare non, eleifend adipiscing nibh. Donec malesuada hendrerit felis, ut porta augue tempor eleifend. Integer pellentesque lectus id est dictum dapibus. Nullam et dolor et sem rhoncus congue quis vitae magna. Duis egestas libero at elit rutrum a tempus enim scelerisque. Maecenas mauris neque, gravida id ornare nec, tempor in lectus. Ut at quam mauris, eu hendrerit tellus. Morbi eu mi et lorem dapibus sollicitudin vitae vel felis. Suspendisse id ligula magna. Aliquam porttitor aliquam aliquet. Integer quis dolor nec odio accumsan convallis. Aenean eros augue, pharetra quis pulvinar dapibus, sagittis sed ipsum. Ut id sapien massa. Maecenas vel libero faucibus sem varius pharetra a eu nisl. Fusce nec eros diam, nec ullamcorper dolor. Suspendisse bibendum, quam tristique elementum lobortis, enim odio adipiscing est, sit amet cursus erat nibh et quam. Vivamus lobortis gravida lobortis. Proin sed bibendum justo. Donec consectetur mauris at risus molestie quis adipiscing mi rhoncus. Curabitur pellentesque congue elit suscipit facilisis. Maecenas vel erat nec enim gravida tempor sit amet eu felis. Duis non commodo risus. Nullam aliquam, velit sit amet dignissim rutrum, eros mauris imperdiet felis, a tincidunt eros erat quis neque. Mauris lacus orci, varius non condimentum tristique, varius ut magna. Vivamus blandit volutpat rhoncus. Praesent vitae quam leo, a tempus augue. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Nunc adipiscing ante nec est adipiscing dignissim. etc. etc. etc. etc. etc. etc. ad nauseum....

suscribing!

I made a generic xml parser with an updated mapping example patch found here:http://drupal.org/node/631104#comment-2600030. (screenshots are there too)

The xml parser should work on any feed and it will create all mapping sources for every element. I think this closely resembles FeedAPIMapper mapping on import functionality

I can't say its perfect because I made it only for my purposes but it seems to be quite flexible, so feel free to use it and improve upon it.

alexb, I hope that this code will be useful in creating a true 2 step mapping on import process as it provides example preview and generates dynamic source targets

subscribe

subscribe

subscribing

Subscribing

subscribe

subscribing.

Subscribing

subscribe

subscribe

I am working on this. I followed the steps specified by #11, but instead I put the mapping form handling in FeedsConfigurable.inc and introduce new method FeedsConfigurable::renderMappingForm() which can be used by FeedsSource or FeedsProcessor.

Progress:
Moving Mapping Form handling out of feeds_ui. (DONE)
Introduce two steps mapping form when importing, both importing by standalone form or by using node. (IN PROGRESS)

Sorry, please ignore the patch above. It contains some bugs that I didn't realize beforehand.

Finished the issue's specification.
1. Now 2-step form will be encountered by user if 'Define mapping on import' option in mapping is checked.
2. mappingForm() is moved to FeedsProcessor detaching all dependencies with feeds_ui module EXCEPT 2 files: feeds_ui.js and feeds_ui.css

There are a lot of subscribers here, please review it :)

+++ feeds.module	6 Jul 2010 09:33:11 -0000
@@ -125,7 +125,25 @@
+        'page arguments' => array('feeds_import_mapping_form', $importer->id, 1),

Just general notice, if page arguments get integers, they treat them as dynamic argument placeholders.

+++ feeds_ui/feeds_ui.admin.inc	6 Jul 2010 05:37:04 -0000
@@ -382,8 +373,8 @@
+  }  ¶

sry, tabs again.

+++ includes/FeedsSource.inc	6 Jul 2010 06:39:56 -0000
@@ -287,5 +287,5 @@
-  }
+  }  ¶

here is a tab.

+++ plugins/FeedsProcessor.inc	6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@
+  	
+  	// Get the FeedsImporter object
+  	$importer = feeds_importer($this->id);
+  	

Use spaces instead of tabs here.

+++ plugins/FeedsProcessor.inc	6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@
+  	
+  	// Get the FeedsImporter object
+  	$importer = feeds_importer($this->id);
+  	
...
+function feeds_mapping_form_help() {
+  return t('
+  <p>
+  Define which elements of a single item of a feed (= <em>Sources</em>) map to which content pieces in Drupal (= <em>Targets</em>). Make sure that at least one definition has a <em>Unique target</em>. A unique target means that a value for a target can only occur once. E. g. only one item with the URL <em>http://example.com/story/1</em> can exist.
+  </p>
+  ');

i would move

out of the t() function. see user_help as an example.

+++ plugins/FeedsProcessor.inc	6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@
+        '#title' => 'Define mapping on import',
+        '#description' => 'If this is checked, the mapping form here will be disabled and instead user will be asked for the mapping configuration after the first import.',

This texts should be translatable.

Powered by Dreditor.

+++ feeds.module	6 Jul 2010 09:33:11 -0000
@@ -125,7 +125,25 @@
+        'page arguments' => array('feeds_import_mapping_form', $importer->id, 1),

Just general notice, if page arguments get integers, they treat them as dynamic argument placeholders.

+++ feeds_ui/feeds_ui.admin.inc	6 Jul 2010 05:37:04 -0000
@@ -382,8 +373,8 @@
+  }  ¶

sry, tabs again.

+++ includes/FeedsSource.inc	6 Jul 2010 06:39:56 -0000
@@ -287,5 +287,5 @@
-  }
+  }  ¶

here is a tab.

+++ plugins/FeedsProcessor.inc	6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@
+  	
+  	// Get the FeedsImporter object
+  	$importer = feeds_importer($this->id);
+  	
...
+function feeds_mapping_form_help() {
+  return t('
+  <p>
+  Define which elements of a single item of a feed (= <em>Sources</em>) map to which content pieces in Drupal (= <em>Targets</em>). Make sure that at least one definition has a <em>Unique target</em>. A unique target means that a value for a target can only occur once. E. g. only one item with the URL <em>http://example.com/story/1</em> can exist.
+  </p>
+  ');

i would move

out of the t() function. see user_help as an example.

+++ plugins/FeedsProcessor.inc	6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@
+  	
+  	// Get the FeedsImporter object
+  	$importer = feeds_importer($this->id);
+  	

Use spaces instead of tabs here.

+++ plugins/FeedsProcessor.inc	6 Jul 2010 09:35:00 -0000
@@ -273,4 +276,215 @@
+        '#title' => 'Define mapping on import',
+        '#description' => 'If this is checked, the mapping form here will be disabled and instead user will be asked for the mapping configuration after the first import.',

This texts should be translatable.

Powered by Dreditor.

Thanks dereine for the review. #32 issues have been fixed. Did minor tweaks and fixing coding conventions/tabs/etc. Now the 'mapping on import' form will be available as a tab in the node-edit page and import form.

Please see the attached image to be more clear.

@alex_b: sorry i was away yesterday.

Thanks for the review. I completely agree with #849840. Removing addMapping() etc will certainly reduce the complexity of the mapping form, will do it in a day or two.

I completely missed the fact that it is tied to importer instead of source! I will look through it after #849840.

Another patch. Needs review. Now it stores the mapping configuration in FeedsSource instead of FeedsImporter.

#849840: Mapping form: submit full mapping on every submission is also fixed. FeedsSource and FeedsProcessor is modifying their mapping configurations without the help of addMapping, setUnique and removeMapping.

@todo: Move FeedsDataProcessor::addMapping() to FeedsDataProcessor::mappingFormSubmit()

Sorry, forgot to attach the patch.

@alex_b

Just a general question related to this patch: I'm not sure what the rationale is for having settings in the source vs. the importer. Although it seems like some settings are better suited to be in the source, as far as I can tell, all the settings in the importers could just as easily be in the source as well. The reason I bring this up is that I have several sources which utilize the same mappings, so obviously for my case it is better to have the mappings in one place (the importer), saved as an exportable. Have any comment?

I think mimicking the views model of Displays might make more sense: There would be a default importer (what is now the importer), and as many other importers as needed of different types (what are now sources). The different types could require settings that the default didn't (eg a feed URL/file location). Besides these settings that are required to be different than the default, the importers would inherit the default settings (think menu settings in a Views page display).

@alex_b:

I certainly do not want to bring up features that have already been implemented and cut or that would complicate the UI for site managers. I just want to mention that since here we have to implement a system in which a source configuration option overrides an importer configuration option, it might be a great time to at least allow that system to be exposed for other options. Then in the future we could possibly be able to expose other options to both the sourceForm and configForm in the future.

An idea: when you call getConfig() on a source, instead of just getting the source config it gets the values from the importer config and merges over the values from the source config. configForm (renamed to importerForm?) and sourceForm would still be independent, but the APIs that deal with the data (getConfig/addConfig/setConfig) would be joined. FeedsSource::save() would only save the keys whose values were in sourceDefaults.
This would have some nice side-effects:
- Mapping in FeedsSource happens the exact same way as mapping in FeedsImporter
- Custom sourceForm configuration defaults can be saved in an importer exportable
- Any importer configuration option can be overridden by a FeedsSource through the normal sourceForm

Something to think about...I don't think it would be too much more work than doing the same thing but just for mappings.

Subscribing, greetings, Martijn

Subscribing

Subscribing

subscribing

Hi all,

Re-rolled the patch to CVS HEAD and did minor modifications. It should now have working mapping on import on all processors included with Feeds. Setting to needs review.

@morningtime:

Thanks for the review. The patch should be able to cleanly patched on the current CVS HEAD of feeds, but maybe not for the older version of feeds. We will need more review (especially on the code and the design aside from the functionality that you have tested).

Unfortunately the feature request might not be done atm, at least not before this is incorporated into Feeds. But afterwards you can always open up a new feature request issue :)

I can also confirm the patch from #47 works.

As for #52, this might actually be desirable behavior. If your original feed file changed for some reason (uploaded to the server by a script or whatever) you would want to verify against the latest file, not necessarily one that already exists. If it's not a huge waste of resources it's probably not a major issue.

Thanks so much for this patch in #47. It seems to work perfectly. One issue: I accidentally put in a URL of feed://domainhere/feedhere (rather than http://domainhere/feedhere). When I went to mapping, I got an immediate WSOD with no watchdog messages as soon as it tried to check the format of the feed. Obviously this is a case of user error, but it would be awesome if it failed a little more gracefully (the standard import gives a drupal message of error -1003).

PHP log errors:
[23-Aug-2010 14:15:26] PHP Fatal error: Call to undefined function drupal_get_bootstrap_phase() in /Sites/Multisite/includes/common.inc on line 3874

Trying to apply patch #47 on CVS but getting 2 failures: FeedsBatch.inc @ 4, FeedsTermProcessor.inc @ 6.

I'm also having trouble working out where the changes go. To make it easier to patch by hand, please use -p with the diff command.

I really hope thiscan be made part of feeds. two weeks is so long. so are you saying this patch works now?

warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'FeedsNodeProcessor_feeds_config_form' was given in /home/cyberfan/htdocs/includes/form.inc on line 376.
warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'FeedsHTTPFetcher_feeds_config_form' was given in /home/cyberfan/htdocs/includes/form.inc on line 376.
warning: call_user_func_array() [function.call-user-func-array]: First argument is expected to be a valid callback, 'FeedsNodeProcessor_feeds_config_form' was given in /home/cyberfan/htdocs/includes/form.inc on line 376.

Hey Alex,
Thanks for re-rolling this. If I understand correctly, these two prerequisites are dealt with in this patch. You're asking for them to be separated out and moved into their respective issues?

Ok, I think I satisfied the prerequisites on

#849834: Generalize feeds_config_form() to feeds_form()
#849840: Mapping form: submit full mapping on every submission

Attached is a patch rolled against HEAD + these two patches. Ideally this patch would be rolled directly against HEAD, but I don't have time to make that modification at the moment. I'm getting a lot of failed tests, but most of them are coming from moving the mapping/config forms to the plugin specific URL's.

-Rolled against head and modified addMapping function in tests.

Testing comes up with 16 failures, all from sitemap parser.

alex_b is churning out code like there's no tomorrow. Here's another one, rerolled against the latest HEAD which has #908964: Break out job scheduler applied.

FWIW, I reviewed the patch (and learned a lot) at the end of august, after reading the comment in #50. (Only after the review, I saw that #849840 was still open and needed work before this :) -- and got distracted before I could do proper work on it.) Just reviewed it more quickly again.

The code looks good to me but I didn't know any Feeds internals beforehand -- and I haven't had much exposure to OO programming.

The only question that I had, as a result of looking at FeedsProcessor::getMappings() and FeedsProcessor::mappingFormSubmit(), was:
"don't we want to define 'source'? Like a 'protected $source' in the definition of class FeedsProcessor? Because it seems to be the only variable that is set without being defined in a (parent) class."
But I'm not even 100% sure if that makes sense.
So I won't change the status.

...and between the time I checked out HEAD and uploaded this patch here, alex_b had done a zillion CVS commits :)

re-rerolled.

note: patches in #67 - #69 apply cleanly to 1.0-beta6 & 1.0-beta7. I didn't spot these releases until later :)

...and after that porting, I actually tested stuff :)

Changed one "$values['op'] == 'Add'" to $values['op'] == t('Add')", which makes the mapping form work in non-English environments too.
(The rest of the patch is the same, except you may need a '-p1' option to apply this file. alex_b's latest changes to HEAD didn't introduce differences here.)

I know I'm creating 'noise' here, but wanted to upload a patch this little fix separately... before recoding something which I want to ask your opinion about, in a little while.

So here's a proposal / question if I am approaching things the right way... (I haven't had that much exposure to feeds)

I'm fiddling with a parser (working version of Feeds XPath Parser) which preprocesses input for all mapped fields, and discards the rest of the input - because it doesn't want to bother processing stuff it doesn't need.
This is a valid approach, right?

This presents an issue in the 'mapping on import' stage. It discards (does not return) all source fields it finds in the XML, which aren't mapped already... which means you can't add any mappings for those fields :)

So,
- in the call from feeds_import_mapping_form to ...parser->parse(), should we signify to the that we're in the pre-mapping stage (meaning "we're really only interested in calling $batch->getMappingSources() later, so make sure you populate $batch with all the source fieldnames and don't bother parsing any info you don't need) ?
- should we do that using some extra property in FeedsImportBatch? (I guess from a purely conceptual standpoint, it's not a 'Batch' property. But otherwise we'd need a third argument to all FeedsParser::parse() functions, which is more intrusive for existing code.)
- if we do that... then $loc_id is not needed anymore as the 2nd argument to FeedsProcessor::mappingForm, right?

You can see what I did, if you compare the patch in #68 against this one. Few changes.

(Detail: the behaviour in FeedsProcessor::mappingForm may have changed slightly for so-far-unused edge cases. That's because the concept of arguments changed slightly:
- $loc_id, which I deleted, was 'the location from where the mappingForm is called' (default 'config' -- set to 'import' for mapping-on-import)
- $batch->context, which I introduced, is 'the context in which a batch is run' (default 'import', or actually unset: when called from the config screen $batch is not set at all -- set to 'mapping' for mapping-on-import)
I don't think it makes a practical difference.)

Subscribing, greetings Martijn

Subscribing

sunbscribing

Re-rolled against beta10.

(Nothing changed except the context lines around some changes.
State is still 'same as when adityakg last worked on it, except the change in comment #69')

Being able to map after selecting the feed based on available feed fields is going to ROCK! Subscribing! I tried to manually apply the patch above and got an error after getting it mostly working.

Everything was fine but when I click the Mapping tab under the Feed content node that I added and RSS feed too I get the following error.

Fatal error: Call to undefined method FeedsNodeProcessor::mappingForm() in /home/appvent/public_html/go-protest/sites/all/modules/feeds/feeds.pages.inc on line 216

Maybe I made an error somewhere else but I took my time applying it to the latest beta version of Feeds module.

;$Id: feeds.info,v 1.5 2010/09/15 19:27:42 alexb Exp $
name = Feeds
description = Aggregates RSS/Atom/RDF feeds, imports CSV files and more.
package = Feeds
dependencies[] = ctools
dependencies[] = job_scheduler
core = 6.x
php = 5.2

; Information added by drupal.org packaging script on 2010-10-29
version = "6.x-1.0-beta10"
core = "6.x"
project = "feeds"
datestamp = "1288369547"

Is this feature obtainable now?

I am pulling in events from Facebook using the parser_ical. Its working fine, but I would like to map more fields - e.g. the event organizer.

Since I am not very experienced, I would like to know if its 'safe' to apply the path and if this will give me access to map the entire ical feed?

Exactly what I need. Subscribing.

Subscribing

I am doing work over at the old feedapi_mapper module to migrate it into the new Feeds API. I will have support for mapping to various types of fields along with an API to extend it for other types of custom fields.

looks like reading through it would be better to work and commit against this issue. I would also like to download files via the mapper. It could be handy for things like images that sometimes goes away after a time.

Also wondering if this is getting into a D7 version of Feeds - would be great for working with non-standard RSS feed fields.

re-roll for beta11.
Still: see #69 for outstanding comments.

Subscribing... we are trying to move from node import to feeds and this would be crucial to us being able to do that.

Subscribing. I recently discovered this module, and I'm writing a feeds importer for node export. Not comfortable with the competence of users to be able to map this stuff ahead of time. Also am familiar with the approach 'node import' used.

Does anyone know if this patch is going to be incorporated and released? Thanks

Of course. And I surely would if I had time.

It's just that 'version tags' in an issue follow the work being done. Not people's wishes. Anything else confuses the work process.

Yes please would love to see this on D7, especially Xpath parser doesn't give any log if it is working ok!
Thanks a lot in advance for building this for D7.
greetings, Martijn

D7 +1

re-roll against 6.x-1.0-beta13; still see #69 for outstanding comments.

subscribing D7 +1

I needed this for a project I'm working on, so here's a D7 version.

I've 2 patches attached, one against the latest 7.x-2.0-alpha9 (tested) and one against 7.x-2.x-dev (untested, but probably works).

One thing I have noticed is that when a user defines mappings at time of import, those mappings are saved to the master parser mappings, and are loaded when the user (or a different user) is uploading a new file which may be in a different format. Changing this back to needs work so mappings always have to be redefined afresh.

subscribing D7 +1

People interested in this issue may also be interested in my proposal for a new UI for the D8 version of Feeds, where the source can be supplied at the beginning of the process (after choosing the fetcher): #2443471-9: Implement configurable parsers.. The proposal has some rough edges: I'm not yet sure how to define the UI for two workflows in Feeds: providing the source at the beginning of the process (and let Feeds autodetect the available mapping sources) vs defining an import "template" (in other words: planning an import) and provide the file (or a serie of files) afterwards. Feeds currently is focussed on the second workflow, while the first one would be a lot user friendly for most use cases. Any help there or discussion is appreciated.

Rerolling without knowing what this patch actually does.