Hi,
Drupal 8 is a great chance to rework architecture of Feeds module/ecosystem, so let's discuss about it.

I suggest to use Rules as base for importers. Rules has tools for near everything we need except batch support and actions plugin to supply Feeds fetchers and parsers. This idea is described here: https://www.drupal.org/sandbox/bobik/1960618

Comments

twistor’s picture

I actually played around with this a long time ago. It's an interesting idea to use Rules, but it doesn't pan out in the long run. The problems:

  • Performance: Rules is a bit slow. For its intended use it's fine, but it won't scale to a site with thousands of feeds.
  • Configuration: Configuring Rules is painful. It wold make the learning curve too large for people trying to use Feeds.

I'll keep this open for other architectural suggestions/ideas.

andypost’s picture

@twistor what do you think about to reuse some parts of core's aggregator and migrate module?

It looks that feeds should extend aggregator for 8.0 but also leverage migrate process plugins

twistor’s picture

I did a lot of work on aggregator with Paris in the hopes of being able to do just that. Sadly, it wasn't enough. The core aggregator module still has hardcoded mappings and is designed with RSS in mind only.

Migrate has been something I've been thinking about for a while. I last looked into it in D7, and it wasn't flexible enough yet. I haven't looked at the changes in D8 lately to see what's what. What I remember from my previous investigation was that Migrate sources combined the transport and payload in a way that's not flexible enough for Feeds.

adub’s picture

Seems that D8 has feed reading in core (including pubsubhubbub) via Zend? e.g. https://api.drupal.org/api/drupal/core%21vendor%21zendframework%21zend-f...

twistor’s picture

We're using Zend's feed parser already.

The pubsubhubbub from Zend isn't usable for a number of reasons. It requires interfaces that Drupal doesn't include. It doesn't support signing. It doesn't support newer versions of pubsubhubbub, specifically v0.4. Mostly though, it's tied to the Zend framework for handling post-backs which is 90% of the work.

gagarine’s picture

EDIT. Question was already answered, should read it more carefully. Sorry.

Honza Pobořil’s picture

What do you (Feeds maintainers) think about replacing specific features by more generic features? For example:

  • Node processor, Term processor, xxx processor → Entity processor
  • GUID and URL mapping target → any field can be unique
  • Term, User, File, Reference mapper → generic reference mapper (with search by any field on referenced entity)

Drupal 8 core has done many similar steps by itself, so Feeds can do it also. This could decrease complexity of Feeds, increase code reuse, so there will be smaller codebase to maintain and less bugs happens. And as a bonus, porting to D8 will be easier.

MegaChriz’s picture

Interesting thoughts, Bobík.

Node processor, Term processor, xxx processor → Entity processor

In D7 we faced some issues with defining a generic entity processor: some entity types required extra values on entity creation. For example, a Field collection item entity requires that the property "Host entity" is set. If this is still the case in D8, it may be a better idea to keep the separate processors. Something worthwile to investigate.

GUID and URL mapping target → any field can be unique

This looks like a great idea for field targets. Note that there can also exist custom targets. For example a target that does some extra processing before setting the field. Or a target that sets multiple fields. I think these custom targets should not be setable as unique by default. Since all properties of an entity are fields now in D8 (base field or configurable field), it's possible that these custom targets would only live in contrib space.

Term, User, File, Reference mapper → generic reference mapper (with search by any field on referenced entity)

There is already one single target for the Entity reference field. Adding a search by any field on the referenced entity sounds great.

An idea I also had was to investigate if it is possible to have a generic field target. Via the REST API it is possible to set fields. Does that happen in a generic way? Also in D7, it was possible in Rules to set any entity property.

MegaChriz’s picture

Category: Task » Plan
andypost’s picture

For mappings there's already typed data plugins in core.
Processors looks like fits into serializer module area

MegaChriz’s picture

@andypost
It would be nice if Feeds could reuse more components that already available in Drupal core, but it might make things also more complex, I think. In the code base of the D8 version of Feeds there is already architecture available for mapping and processors, so I feel it makes more sense to focus on other parts of Feeds first, like the UI for example. Maybe you (or someone else) wants to write a prototype of a mapping solution or processor solution using components from Drupal core? This way it could become a more realistic goal.