[Policy, No patch] Consider adopting parts of migrate system in the underlying parts of feeds

The Feeds 8.x-3.x version will remain an independent solution.

A few months ago, I've spoken with the maintainer of the Migrate module, and there is a plan to make a serious attempt to replace parts of Feeds with Migrate components. The Migrate module maintainer is open for changes to the Migrate module to support Feeds. The plan is that the Feeds version that will use Migrate components will be labeled as 8.x-4.x. The plan is also to first reach a "working" version of Feeds 8.x-3.x, which means a working CSV parser, a working XML parser and a port of Feeds Tamper. Nice to have features like unpublishing/deleting nodes that no longer appear in the feed or language support as in the 7.x-2.x version may not make it in the 8.x-3.x version (or maybe in a very basic form, if possible).

Unfortunately, due to other obligations (I'm in the process of moving to an other house) I've zero time to work on Feeds at the moment. Feeds is for about 90% a hobby project for me and in the past few years I've spent near the maximum of my free time to work on it (in my most active period about 7 hours a week). It's unlikely I will we able to spend as much time a week as I used to in the nearby future. In other words: as long as I'm the only one to take the lead on the development of Feeds, it is unlikely that a Feeds version based on Migrate will come anytime soon (if at all).

I'm happy that you would like to help with the development.

MegaChriz - How would you feel if we worked with you and the migrate maintainers to start the 8.x-4.x a little before your 3.x branch is prime time?

@joshmiller
That would be great, thank you. Keep in mind though that at least until November I likely won't have the time to coordinate the project (I'm not sure how much work moving to an other house will take until it is done). I still know very little about the D8 version of Migrate too. The Migrate maintainer's suggestion to get up to speed with Migrate was to join the Migrate online meetings. I signed up for these, but I haven't found the time yet to actually join them.

This isn't a bad idea. I spent a good deal of time writing a massive migration from a non-drupal to D8 system. The one thing the migrate module was really missing a UI that allowed for periodic imports from different files. The file name was typically hard coded into whatever migrate module you wrote.

I don't mind not having a UI to configure fields. And you can 'tamper' everything in prepare_row. But just having a few things from the feeds module and you're there.

@joshmiller,
we had code sprint for feeds8 at BADCamp last week, and we are actively working on the module. Next big step for us is to add parsers, and it would be awesome if we can reuse what is in core for migrations. It would be great to work on this together.

@irinaz

Good news !!!!!!
I am impatient to test Feeds8 with new parsers (csv, xml)

We are also making some strides in migrate_tools to add some more UI tools. In November of 2017, we add a migration runner. And there are several follow-up issues to continue beefing up its UI features. Then there is https://www.drupal.org/project/commerce_export, which is working some UI enhancements to provide an easy way to import data into Commerce. There's somewhat of a roadmap for the UI aspects of migrate in this blog post: https://www.mtech-llc.com/blog/edys-meza/running-custom-migrations-web-ui

@heddn, welcome to this discussion!! We have now weekly meetups at https://drupal.slack.com/messages/C34CECZAL channel every Thursday between 19:00 and 20:00 UTC, maybe you can join us sometime so we can discuss direction and architecture. I hope to be testing new https://www.mtech-llc.com/blog/lucas-hedding/two-down-one-go this week to see how much of Migrate UI can be reused in some way in Feeds.
Thanks!

#DrupalCon2018 - Lots of conversations about how to approach integration of feeds UI and migrate in core resulted in a placeholder for new module to start building Feeds-type UI to Migrate module https://www.drupal.org/project/feeds_migrate.

It looks like this module does not belong fully neither to Feeds Module nor to Migrate or Migrate Tools.

I'm sorry I wasn't at DC Nashville to influence more collaboration. I really hate to see "yet another" module in the place. Any chance to reconsider doing this in migrate_tools instead? Does it really need to be a new module that does the same thing as migrate_tools?

If that will make it more efficient to add it to Migrate tools I am happy to go this path.

@heddn
As far as I know Migrate, I see two possible approaches to this issue:

1. Rebuild the Feeds UI in Migrate Tools

   Cons:
   Specific Feeds features that are not matured in Migrate yet will be lost and need to be rewritten. Think of periodic import, unpublish/delete nodes not in feed, target configuration, pubsubhubbub, expire, update items that are already on the target site. I cannot oversee how hard it will be to port these features over to Migrate.

2. Incorporate Migrate features into Feeds
   Feeds would keep doing its own thing but it would use Migrate's process features to deliver the data to fields. This could be with Migrate process plugin 'Get' and with a Migrate destination (I haven't figured this out yet).
   At a later stage we can build adapters for other Migrate concepts, like MigrateSource.

   Pro:
   Feeds is more free to follow its own direction.

   Cons:
   There is a risk that Feeds would not use Migrate to its full potential: it is possible that only some Migrate features would be compatible with Feeds and others won’t. Think of "rollback" and "chained migrations".

However, it seems to be targeting site builders more than developers. Having both Feed and Migrate in the name of the module make big difference for end users.

• Periodic import - API supported, just need a cron system to execute
• Unpublish/delete nodes not in feed - supported w/ patch #2809433: Migrate support for deleting items no longer in the incoming data
• Target configuration - What is this?
• PubSubHubbub - What is this?
• Expire - What is this?
• Update items that are already on the target site - Supported even in D7 migrate

I think target configuration is possible. I need more description though before I can confirm. Can I get a better description of Expire and PubSubHubbub? I cannot determine what features are needed for those to work.

I cannot comment about option #2.

@heddn,
@MegaChriz,
@Irinaz

On drupal 7 i always used feeds module to import datas from differents sources: file xml, file csv...
But on drupal 8, after having tested the 2 solutions, feeds and migrate, the migrate module seems to be more advanced. Migrate is stable and several related modules/possibilities with migrate, are useful: migrate commerce, migrate medias...

But, we have not an efficient UI with migrate, it's the bad point. Maybe, an interesting thing could be to use the good ideas of the feeds UI. A good UI for migrate module could help users to use migrate (less user friendly).

A common work on migrate will be an interesting point, and working on an unique robust solution is important for Drupal.

@kumkum29 - let's do that - here is placeholder - https://www.drupal.org/project/feeds_migrate
@heddn - I talked to Mike Ryan and about Tools being module for developers and Feeds_migrate is more for site builders.

Next Thursday, July the 19th, a Migrate workshop is planned at 17:00 UTC. Attending this is useful for people who want to learn more about Migrate and for people who want contribute to bringing Feeds and Migrate closer to eachother.

Contact Lucas Hedding (via twitter or drupal) to join the workshop.

You could also spread the word by retweeting the following tweet:
https://twitter.com/irinazaks/status/1017481569743732736

Just to mention it, when the first alpha was released I suddenly wondered if it was time well spent, if instead effort should have been put into Migrate to make it cover Feeds' use cases, so I'm super happy to see some collaboration starting - kudos!

@DamienMcKenna,
Feeds was designed for sitebuilders and UI is one of it's core components. Feeds is essential for adoption of Drupal 8 for this group. Migrate is very powerful developers' tool, but does not cover some of the features that are needed when building import via UI. We started new module that should be integration of Feeds UI with migrate back end

https://www.drupal.org/project/feeds_migrate

@joshmiller, here is proposed approach for building 8.4 version https://www.drupal.org/project/feeds_migrate/issues/2986949