[meta] Feeds extensible parsers 8.x roadmap [#2465535]

Plan overview

The next step for working towards a stable D8 release is fixing the UI for the XML parser (see #2917924: Implement the XML Xpath parser UI). This task depends on #2669974: Implement the CSV parser UI.

After that, the user interfaces of the other parsers need to be updated as well. Lastly, some fixes that were added to the 7.x-1.x version need to be ported over. And then it is matter of keeping Feeds extensible parsers up to date with changes that happen in Feeds.

See also the roadmap of Feeds: #1960800: [meta] Feeds 8.x roadmap

Feeds extensible parsers 8.x M1 - The XML Xpath parser UI

Goal: have an usable XML Xpath parser, preferably with functional tests

See #2917929: Plan for 8.x-1.0-alpha1 release

Remaining tasks

#2917924: Implement the XML Xpath parser UI

Feeds extensible parsers 8.x M2 - Implement the UI for other parsers

Goal: make all parsers usable
#2997320: [meta] Add UI tests for parsers

Feeds extensible parsers 8.x M3 - Port remaining fixes

Goal: port the remaining fixes

A few fixes have been added to the 7.x-1.x version since the 8.x port started.

Issues/commits:
#2494185: JMESPath parser requirement for "Context" makes value inaccessible?
#2512428: JMESpath exceptions should be exposed to the user
http://cgit.drupalcode.org/feeds_ex/commit/?id=f36c61f7223d316f3059251ec...
#2314157: Add expression validation for JSONPath. using Flow\JSONPath\JSONPath.
#1517642: Strip ASCII characters below 32
#1998398: Add option to decode HTML entities in XML.

Feeds extensible parsers 8.x M4 and beyond

Goal: stable release

Keep up to date with changes that happen in Feeds.

Original report by MegaChriz

I made a rough start with porting this module to Drupal 8:
https://www.drupal.org/sandbox/megachriz/2465523

It isn't functional at all at this point. What I've done so far:

Added namespaces to files.
Removed "FeedsEx" from class names.
Moved parser classes to \Drupal\feeds_ex\Feeds\Parser.
Added annotation to parser classes.

I haven't touched the implementation yet, so that is still based on Feeds D7.

Volunteers are welcome to help with the port.

Comment	File	Size	Author
#8	feeds.feed_type.boom_level_plist.yml	2.26 KB	MegaChriz

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

twistor CreditAttribution: twistor commented 9 April 2015 at 18:44

@MegaChriz, I've made you a maintainer of feeds_ex. You can merge in your branch here whenever you feel like it. I'm going to be working on regular Feeds for a bit, but will circle around back to this port at some point.

Thanks!

Comment #2

MegaChriz CreditAttribution: MegaChriz commented 9 April 2015 at 18:52

Great! It will probably take a while before I get feeds_ex to a point where it is usable. I barely know its code at this point. I just started the port as I like to use it in a hobby project. I think working on feeds_ex will also help me to get familiar with the D8 version of Feeds.

Comment #3

twistor CreditAttribution: twistor commented 9 April 2015 at 19:09

Yes, there's no rush at this point. Feeds isn't quite stable yet so things can change.

Also, some things from feeds_ex will get added to Feeds proper once I figure them out. Mostly around how the parsers are configured. I want to change the CSV parser in Feeds to use a similar thing so that the CSV fields are not configured in the mapping. That way, the CSV fields can have machine names, and will make Feeds Tamper's life a lot easier. It will also allow people to map from fields on the Feed entity, as well as CSV fields. That's currently a bit wonky in Feeds 7.

Some of the XML helpers are also already in Feeds 8.

Lot's of work to do.

Comment #4

MegaChriz CreditAttribution: MegaChriz commented 6 May 2015 at 19:41

I'm well on it's way in porting Feeds Extensible Parsers to Drupal 8! The units for the XML parser and the HTML parser are ported to PHPUnit and most test methods are passing when the lines about fetcher config in Drupal\feeds_ex\Feeds\Parser\ParserBase is commented out. I haven't paid attention to the UI yet. I'd like to get the unit tests done first and make them run without PHP fatal errors.

I have a few questions about the code:

In Drupal\feeds_ex\Feeds\Parser\ParserBase::parse() there is code trying to get fetcher config to "set a link":
```
$result = new ParserResult();
// Set link.
$fetcher_config = $feed->getConfigurationFor($feed->importer->fetcher);
$result->link = is_string($fetcher_config['source']) ? $fetcher_config['source'] : '';
```
But the link property from Drupal\feeds\Result\ParserResult was removed in commit da3b00d0. It seems to me that this code is no longer relevant because of this removal, though the xml parser test explictly tests this piece in testLinkIsSet(), so it has at least some level of importance in the 7.x-1.x version. I tried to figure out for what purpose this link property was ever set, I went all the way back to this commit in Feeds XPath Parser. My first thought was that maybe is used as a mapping source, but that appears not to be the case.

What should I do with this piece of code?
I can't get the tests with the Japanese encoded file passing locally yet. The fetched result gets empty after Drupal\feeds_ex\Utility\XmlUtility::removeDefaultNamespaces() is called. Has this to do with my local setup? (I've seen that the XmlUtility class can be replaced with the XmlParserTrait, but I keep that for later.)

Having the CSV parser work in similar way as the extensible parsers looks like a smart thing to do, I'll come back to that later.

Lot's of work to do indeed. Up to the JSONPath parser!

Comment #5

giuliovale CreditAttribution: giuliovale commented 3 February 2016 at 12:00

Any news about D8 porting?

Comment #6

webteqit CreditAttribution: webteqit commented 4 June 2016 at 08:38

Morning All,

I have just made a foray into Drupal for a project I am working on and must say I love it and all the great work you guys are doing on the Modules.

Just wondering if there is any update on the port for the Feeds extensible parsers to Drupal 8?

I don't think I would be able to contribute much from a coding perspective, but am more than happy to test the current release, if it's at all possible for me to get a copy?

Thanks in advance

Comment #7

joelhsmith CreditAttribution: joelhsmith commented 12 October 2016 at 20:24

I love this module. I would love to see in Drupal 8, How is the port going?

Comment #8

MegaChriz CreditAttribution: MegaChriz commented 27 October 2016 at 12:32

File	Size
feeds.feed_type.boom_level_plist.yml	2.26 KB

The port is currently postponed on #2443471: Implement configurable parsers.. And that issue is postponed on taking a design decision and also on finding volunteers who want to help with the development of Feeds for D8. I've paused my work on the D8 version of Feeds at the moment (with the exception of trivial patches), mainly because I'm a bit overwhelmed with the amount of work there is to get it done and I haven't got enough courage and energy yet to start on it on my own. For now I have chosen the easier path; and that is working on the D7 version of Feeds (to hopefully fix all the major bugs in it).

The state of the port itself: it is currently based on an old version of D8 (I think it is 8.0.0-beta9) and an old version of Feeds (a dev version from about the same time as the beta9 release of D8). Only the logics of the module have been ported: there is no UI yet. With the exception of the points noted in #4, I had the module in a "working" state at some point. This means having the configuration entered manually in a YAML file, followed by importing it using Drupal's config management UI. See attachment for a sample configuration file.

Comment #9

MegaChriz CreditAttribution: MegaChriz commented 6 November 2017 at 09:03

Title:	Port Feeds extensible parsers to Drupal 8	» [meta] Feeds extensible parsers 8.x roadmap
Category:	Task	» Plan
Issue summary:	View changes

I've updated the roadmap to get an overview of the long term plan for Feeds extensible parsers. First, some changes are needed in Feeds as explained in #2917928: Plan for 8.x-3.0-alpha1 release. Then we can work on the plan for the first alpha release of Feeds extensible parsers: #2917929: Plan for 8.x-1.0-alpha1 release.