Port FeedAPI Mappers: date

Alex,

I just want to check this with you before I start any work on it, am I correct in saying that if I port "feedapi_mapper_date.inc" (i.e. from the FeedAPI module) to Feeds, then along with the "iCal parser" module I should be able to import iCal info from another site and map the date information on one of my event nodes (CCK)?

I just want to double check I understand this correctly and I have not missed any other file or module required before I start working on it.

Nick

The ical parser (in HEAD) is roughly integrated with feeds and dumping everything out ready for mapping. So yes in theory once the mapper bit is migrated it's ready to roll.

It may want some tweaking the way the information is presented from the parser for mapping, but that should be easy. But at the moment, I've certainly not had time, to get it from the parsed feed to the field (the mapping bit :) for the date section of the feed items. So if you have that would be brilliant. Any changes to the parser let me know :-)

Hi ekes,

Thanks for that. Same problem here, lack of time. I figured if I used FeedsAPI, it would be quicker, but eventually I would be switching to Feeds so therefore might as well put something back in to the community whilst helping to develop my site. Will see if I can find time this weekend to have a crack at it.

Nick

Has anyone started to work on this ?

@mongolito404 I was planning on starting last week, just have not had the time.

Hi,

This may be me missing something, again! I have started to work on this, porting the date mapper. My first hurle, and I ahve abrely started, is that I cannot get a date to show in the source. I have created an Event type, it has a date field, I create a new feed type to import nodes to, but I do not see the date field listed in the source.

Can anyone tell me what I am doing wrong?

Please note, I have not started the coding yet as this is the first thing I need to see before I can start work on the mapping.

Regards,
Nick

You have to implement hook_feeds_node_processor_targets_alter($targets, $content_type) to see your date field as a target. Sources are item fields provided by the parser. The parser don't know about content types, it would not provides any sources for their fields.

Hi,

Just got it. I was not expecting to have to do that. Think that is something that needs to be documented.

Thanks,
Nick

@mongolito404, I have just re-read what you said, and must have misread it. I do not ahve a problem with the target, that appears to be working okay. I understand that the source doesn't know the content type, hence the reason we use mappers.

The problem I have is that the source I am expecting to see is not that and I am not sure if it is me missing something or a bug.

I am using the iCal Parser (http://drupal.org/project/parser_ical), which I have read should be compatable with Feeds. I have created a news feed, associations and event type with rel. fields. I see the field/entry to map to with the date mapper in the target. I do not see anything representing a date in the source though which is what I would expect from the iCal Parser.

So the question is, is it the iCal parser that should be putting an entry in the source drop down?

1. I am assuming YES. If so, can someone else please confirm that this is a bug before I chase it up within that module.
2. If NO, can someone point out what idiotic thing I am doing wrong?

Cheers,
Nick

Hi,

Okay, think I have figure it out. I did not realise the iCal Parser adds the dropdown box (see attached images), thought this was Feeds. So with the iCal parser enabled, it tells the mapper what the sources are. There is no way to enter one manually. Am I right in assuming I should be able to enter in to the source box, the name of the field on the feed I want to map? i.e. VEVENT?

Nick

Hi All,

Okay the code I have so far is below, not a lot I know, but still learning the Feeds API system so bear with me. I am not able to do any more work until Sat. so if anyone wants to pick it up, code for date.inc is:

<?php
// $Id$

/**
 * @file
 * On behalf implementation of Feeds mapping API for ?????????????
 */
 
 /**
 * Implementation of hook_feeds_node_processor_targets_alter().
 *
 * @see FeedsNodeProcessor::getMappingTargets().
 */
function date_feeds_node_processor_targets_alter(&$targets, $content_type) {
  $info = content_types($content_type);
  $fields = array();
  
  // The $info['fields'] holds CCK fields such as date and links.  
  if (isset($info['fields']) && count($info['fields'])) {
    foreach ($info['fields'] as $field_name => $field) {

      // Check to see if the field type is a date (CCK).
      if ($field['type'] == 'date') {
      
        // Set the name for the source label to be the label of the field or the field name if no label exists
        $name = isset($field['widget']['label']) ? $field['widget']['label'] : $field_name;
	
	//Sets the targets to map
	$targets[$field_name.'#date'] = array(
          'name' => $name . ' (' . t('date').')',
          'callback' => 'date_feeds_set_target',
          'description' => t('The Date for the CCK !name field of the node.', array('!name' => $name)),
        );

      }
    }
  }
}

/**
 * Callback for mapping. Here is where the actual mapping happens.
 *
 * When the callback is invoked, $target contains the name of the field the
 * user has decided to map to and $value contains the value of the feed item
 * element the user has picked as a source.
 *
  * In plain English, maps the source elemnts to the target.
 */
function date_feeds_set_target(&$node, $target, $values) {
}

Nick

Hi,

So I mnaged to sneak in a little more work, only a few lines plus a patch file. It does work, but is very limited and basic. Needs a lot of refinement and work, but it is a start. I am using the output of the iCal Parser to feed in to this mapper. Do look at it and add to it.

Nick

There is 3 type of date fields: date, datetime and datestamp (deprecated but used to support legacy data).

If some parsers output dates as start date and end date, a single target is needed. But parsers could also output two separated sources (CSV for instance). So two targets are needed. Worse, parses could also output the date and time as separated sources. So we the mapper would need separated targets for these. So I think we need the following targets
- Dates (start date, end date) as a single fields => $field[0]['value'] and $field[0]['value2']
- Start date => $field[$i]['value']
- End date => $field[$i]['value2']
- Start time => $field[$i]['value']
- End time => $field[$i]['value2']

Using Drupal and PHP date functions, it should be possible to parse a time source and to its overwrite only the time parts of the full datetime already sets in the field to merge a two separated sources in a single field 'value'.

@mongolito404, I now see that in the old FeedAPI mapper. Am disecting the old mapper and slowly porting it over. I ahve attached an updated copy of the code. Still not a lot, but only doing short periods at it when I get the chance. Anyway, with any luck that should now be on the right lines. Do let me know if I have made any more idiotic mistakes, wouldn't be so bad if I could spend more time at it i one go...

Nick

subscribing
Good work, from a newbie who is really looking forward to the Feeds iCal parser integration!

Hi all,

Just a quick update, I won't be doing anymore work on this for a couple of days.

Nick

If so, I will work on it.

@mongolito404, if you could that would be great. I will ahve some time at the weekend, but nothing before then.

Nick

Here is the current state of my work.

I simplified it. It only supports full date time (no date only or time only). It should also supports iCal but it has not been tested.

The provided test case tests for many different date formats, many of them fail because of missing code.

The current todos with this code are:
- In date_feeds_node_processor_targets_alter: Only provides "end date" target if field allow it.
- In date_feeds_set_target :
- Add support for the field format when parsing values
- Add support for common formats when parsing values on PHP < 5.3 (see _create_date_from_known_formats)
- Write _create_date_from_known_formats for PHP < 5.3
- In FeedsMapperDateTestCase
- Test for iCal
- Test end date

I'm not sure if support for date formats must be included in the mapper. I think parsers must handle the date formats required by the file formats they support. The common syndication and SimplePie parses do it and provide their dates as UNIX timestamp. Generic parses like CSV should provide configurable data conversion. For date by probably for other types as well (probably using a hook or plugins).

@mongolito404, wow, that is impressive. Will test it out late, obviously I need to read up on the Feeds API!

Hi mongolito404,

That is some nice work, did not relise the testing was there as well. Anyway, will ahve a play over the weekend and see if I can add some more of the missing code.

Nick

I think the timezone conversion is wrong.

From what I'm reading, I *think* you should be using

  $field_timezone = timezone_open($field['timezone_db']);

instead of

  $field_timezone = timezone_open(date_get_timezone($field['tz_handling'], date_default_timezone_name()));

as the data is going into a node_save() style save rather than a drupal_execute() style save.

I am testing it with imported date fields that look like this:

12/08/2009

All of the imported fields look like that. They all import properly, so at least in that case it looks good to me.

Another minor nit that is rather annoying is the use of a variable that doesn't seem to be set anywhere:

          'name' => $name . ' (' . $sub_field .')',

-- which is making it hard to see which is which. Suggest just using (begin) and (end) respectively...

Hey all! I'm looking into this too. @mongolito404, could you upload the date.csv simpletest file you're using?

Thanks,
David

Hey! Here is a direct port of the old feedapi date mapper to feeds--it keeps the identical functionality of date parts and that date creation class and so on. I have lightly tested it so far, including with the new ical parser, and it gets stuff through on so on. I haven't tested timezone stuff thoroughly yet.

Does anyone know if there were any issues with the old feedapi date mapper? The code possibly could be cleaner, but I guess there were use cases that had to be supported for only mapping various parts at a time, like the time from one field and the date from another? I am going to work on this a bit more, including testing and possibly contemplating just scrapping this old method and rearchitecting, perhaps along the lines of mongolito's slimmed down version. Any input on important use cases, past precedent for this mapper, or other suggestions would be very helpful.

David

For some context, the reason the date mapper ballooned into its current state is encapsulated in this lengthy issue: #215979: Create a CCK date field mapper. Basically, the use cases for mapping certain parts and similar stem from certain csv type date exports. I'm going to test the mapper, and if it works I'd be in favor of just leaving it basically as is, at least for now.

Slightly tweaked patch, includes raw field for the RSS parser published date so that the date mapper can use it correctly if there are timezone conversions or similar.

Subscribing....

Hey, currently working on a complete refactor that will be very pretty and OO, although at this point it will probably require PHP 5, perhaps (but hopefully this can be avoided) 5.2.

David

I have been using the patch from #20 but #29 seems to be newer and quite different.

Which one are we all thinking will be the winner here? I'm re-rolling my Feeds and want to test whichever one is more likely to be the final accepted solution.

#20 has been working well for me but my date needs are very basic.

Hey, preliminary working 3rd version of patch. This one is kinda completely different from the other two...again :-( It is now object oriented and does cool stuff like merging date parts intelligently. I will do some more testing and add comments + my simpletests tomorrow. Also, iCal parsing is now contained in its own class that extends these, and it should live in the parser_ical module. I'll submit that patch there too soon, and I'll post the link.

David

Significantly improved code factorization and brevity; clarity fixes as in Alex's patch; added date mapper test that should pass. See relevant ical parser patch ticket #614120: 2.x feeds module version. I will add a test for that one soon.

David

All done but failing the tests on not-my-setup :-)

Fixed fatal error.

I installed the module (6.x-1.0-alpha9) and cant find my cck date fields in the mapping settings.
Did I miss to download a mapper.inc somewhere?

... UPDATE: sorry, just saw that this is part of 6.x-1.0-alpha10
Is it stable enough to work with?