Move Feeds into core
#1827164: Integrate Feeds into core
I'm looking for some insight into feeds 7.x-2.0 release plans. I've used it quite heavily over the past couple of weeks and applied quite a few patches to get the functionality I needed. I added some notes at http://drupal.org/node/1301604. Not sure who are the current maintainers and their thought on 2.0 release are, but below are mine. If this seems like a worthwhile discussion, we can move it to the g.d.o. wiki. Maybe we need a 7.3 placeholder for pushing issues forward.
Keep core feeds project as simple as possible. Its heavily used and easy to build on top of. Move as much extra functionality as possible to feeds_tamper, other field modules (references, etc.), and special use case modules.
Definitely in
- Fix simpletests for D7. Many are broken: #1652764: Fix code-style and #1653018: Fix failing tests.
- Error Catching: #1110762: Feeds does not catch file exceptions properly in file mapper and FeedsParser.inc getFile() function , #955236: Handle fatal errors for entity create with better error handler function
- Important features: #1127696: Attach multiple importers to one content type (in D7)
Hopefully postponable
When we have #1033202: [Meta] Generic entity processor we should do #1739842: Deprecate some processors in favor of generic entity processor and mark these as not release critical:
- Language Issues #1183440: Multilingual Feeds - Make field import language-aware , #1247536: Allow users to choose node import language / #840142: How can the created nodes be assigned their correct language? (and other "core" fields...), #1228570: Add user language to FeedsUserProcessor
- User Imports: #1300764: Feeds User Processor User Update Feature Needs clarity in user interface
Maybe In, maybe separate module
- Support for hierarchical taxonomy imports. #1152940: Feeds term import with hierarchy and weight
Surely important, but can be done after release
- #1362378: Provide entity post-save hook
- #1369874: Don't roll own CSV parser; use PHP's native one
- One to many relationships. #759966: CSV: Support one to many relationships, #661606: Support unique targets in mappers
- Multivalue fields. #1039134: Multi Tag import when importing a node, #1165506: Add pipe delimiter to csv parser, [#] Feeds tamper can resolve some of these.
- Edge Case? #1190470: URL prefix and suffix for HTTPFetcher
Comments
Comment #1
Dave ReidWould be great to get some kind of confirmation or affirmation from phase 2 on this direction. Anyone?...
Comment #2
Anonymous (not verified) CreditAttribution: Anonymous commentedWe're holding another module maintenance jam at Phase 2 during lunch on Wednesday. I'll make sure we talk about this, so we can get you some feedback on our thoughts and what direction we'd like to go in.
Comment #3
febbraro CreditAttribution: febbraro commentedI dont have any comprehensive plan for Feeds 7.x-2.x just yet, there are too many outstanding issues/patches to focus forward just yet. I have mobilized a few guys from my team to help tackle the long list of outstanding patches and once we make some good headway there we can start to formulate more of a plan for moving forward. All that to say we will dig into outstanding patches including the "Definitely In" section above.
As of this morning I have all unit tests running cleanly (please verify)
I also have a group of us spending our lunch hour or so right now reviewing patches.
Comment #3.0
febbraro CreditAttribution: febbraro commentedadded simpletests
Comment #4
odavy CreditAttribution: odavy commentedWill the generic entity processor ever be rolled in to this module? The patch seems to work well...
Comment #5
colanWe should stick these in there as well:
...as other issues either could or do depend on them. We should add any others that are framework-ish.
I think the biggest job that has to happen (ideally before we get this thing out), is cleaning out the issue queue. I'm finding tonnes of duplicates, and those are metric tonnes! We could probably have a summer student work on this exclusively. :)
Comment #6
Anonymous (not verified) CreditAttribution: Anonymous commented@colan, We've been working on testing patches in the past week or so.
If you see any issues that are duplicates, please feel free to close that issue and in the comments, refer to the issue that asks the same question. We'll be doing this as we go through the issue queue as well, but any help you can give us would be greatly appreciated!
Comment #7
colanWay ahead of you - been doing that for the past week at least. ;) Thanks for testing patches - wasn't sure if anyone was still maintaining this!
Comment #8
Anonymous (not verified) CreditAttribution: Anonymous commented@colan, Thanks! I wasn't sure by your comment above. We really appreciate your help!
Comment #8.0
Anonymous (not verified) CreditAttribution: Anonymous commentedfixed an issue link
Comment #9
colanMarking #1400042: Create stable release of Feeds for 7.x as a duplicate of this issue.
Comment #10
Xen CreditAttribution: Xen commentedMay I suggest that someone who has the appropriate authority (febbraro?) tags the important issues with 7.x-2.0 blocker?
Gives quick entry for anyone wanting to give a hand.
Comment #11
colanFrom greggles' duplicate:
Xen: I think anybody can add tags; extra permissions not required.
Comment #12
colanMaking sure tag can be added.
EDIT: I was able to add the tag to some issues (including this one), and I don't have any extra permissions. So other folks can help with the ones I didn't get to.
Comment #13
Xen CreditAttribution: Xen commented@colan
Wasn't so much a question of permissions, but more to get the stamp of approval from the maintainers on the selected issues. If everybody just adds their favorite issue, we'll never get anywhere.
Comment #14
twistor CreditAttribution: twistor commentedMight I add:
#996808: Update existing doesn't reset targets that have real_target set.
#712304: Batch import does not continue where it left off, instead starts from the beginning
I actually think there are multiple issues here, but I've been getting a lot of complaints from this one:
#625196: Fatal errors (Unsupported operand types) and warnings (Argument is not an array) in FeedsConfigurable.
Comment #15
Xen CreditAttribution: Xen commented@twistor
That's why we need a maintainer to draw out a roadmap.
Comment #16
pcambraI think drush integration is a must, and it is quite there #608408: Drush integration for Feeds
Comment #17
colanXen: That's a good point, but is might save the maintainer(s) work if we tag what we think is appropriate, and then he/she/they edit it.
Comment #18
Xen CreditAttribution: Xen commented@colan:
Well, if someone would gather together all the contenders and make a list for the maintainers to give the nod, it might.
Would be nice if the maintainers would show up and give their input.
Comment #19
febbraro CreditAttribution: febbraro commentedGiven the fact that the issue queue is so huge, I don't think any one person will have time to evaluate all of the things that should go into a stable release. We would certainly appreciate any support in identifying all of the critical issues for a 2.0 stable release with the 'D7 stable release blocker' tag. These need to be real critical issues though, not just that something specific (or obscure) does not work for a particular use case.
In any case, tagging the issues will help us focus our efforts and allow us to get there faster. Thanks.
Comment #20
Xen CreditAttribution: Xen commented@febbraro
Having everybody and their dog tag their favorite issues with the 'D7 stable release blocker' is just going to make another big list of issues. Somebody has to play hardball and decide which of those issues is *really* needed for a stable release, and what'll have to wait.
There's been issues mentioned here that's not really needed to get a stable D7 release out the door, and the more that's added to the list, the longer it'll take.
Having a clear "these are the issues that the maintainer says needs to be fixed" list rather than a big "these are the issues that some random thought should be fixed" list is a bigger motivator for anyone that wants to dig in.
Comment #21
twistor CreditAttribution: twistor commented@Xen, Issues can be un-tagged just as easily as they can be tagged. A maintainer just asked for assistance finding the critical issues as this queue is huge.
Might I suggest a few guidelines:
Comment #22
Xen CreditAttribution: Xen commented@twistor
I'm just asking for someone to take a bit of charge here. It's been six months since the last alpha, there's still major functionally issues, and everybody agrees that we need to get things moving and prioritize.
So why is #1369874: Don't roll own CSV parser; use PHP's native one considered a blocker? I'm all for using the standard PHP parser, but considering the problem of too many issues, is it really needed for the first D7 release? If the old code works, let it be and refactor it in a later release.
It's fine if people can help out pointing out the issues, but someone has to decide whats considered a blocker, because we can't all do it.
Comment #23
colan@Xen: Then this one is something that can be dropped from the draft list once its complete. But let's add a refactor tag to this type of issue, so that these don't get lost.
@febbraro: Do you want more co-maintainers? I (and probably others) would be happy to help, as it seems like you're overwhelmed.
Comment #24
zilla CreditAttribution: zilla commentedthat would be awesome (a stable release) - though 'job scheduler' is still in alpha. could this be configured to work with elysia cron instead? http://drupal.org/project/elysia_cron (seems that it would present potential for more granular control in large import scenarios or predictable intervals at bigger scale)
Comment #25
emackn CreditAttribution: emackn commentedHi guys. I'm the lead for Feeds module here at Phase2, so with guidance from febbraro, and your input, I'm trying to get this stuff moving. I've been committing smaller changes over the last few months, while trying to get more familiar with the code base. But anyway, just wanted to chime in here so you know where to direct your questions and concerns.
Comment #26
zilla CreditAttribution: zilla commentedthanks for the intro - it sounds like the floating question is where things stand for a stable 7x release, whether it's soon or later this year, any insights would be greatly appreciated!
Comment #27
johnbarclay CreditAttribution: johnbarclay commentedHope things are going well. Another approach to getting a beta out and perhaps make the core feeds module maintainable would be to split out the feeds modules as such:
feeds
feeds_ such as:
feeds_user
feeds_node
feeds_taxonomy
feeds_webservices
In the long run, this might make the module releases more viable because "feeds" could be released as stable without all these other dependencies. In this model, the modules like feeds_user would also be architected as contrib feeds modules are expected to be and would server as better templates for contrib modules. It might also help to get core feeds in core drupal.
I have some funding to do some work on feeds if there can be some hint of an exit strategy for 7.2, otherwise I just need to work around feeds altogether. I'm just not in a position to build on top of a module that has path out of dev.
Comment #28
franzemackn, I'm also in for co-maintainer role, if it's needed.
I don't think #1033202: [Meta] Generic entity processor, #1362378: Provide entity post-save hook and #1369874: Don't roll own CSV parser; use PHP's native one should be listed as blockers at all. New features need to be pushed into 3.x, and we should work hard on fixing the bugs at this time, as we all need a stable realease of 2.x. I'm tired of using -dev release or patching already committed fixes in the old release, so maybe we should setup a sprint or something to get this done.
Comment #29
twistor CreditAttribution: twistor commented@franz, I agree that those issues shouldn't be blockers. Although, they don't need to be pushed back to 3.x either. New features, as long as they are backwards compatible, can be considered.
Count me in for a sprint.
Comment #30
franzI think we should add a new alpha release at this time. Already a handful of bugfixes in.
Comment #31
gregglesIt's 11 months since the last alpha - makes a ton of sense to me.
Comment #32
franzGot a new release rolled, I hope beta doesn't take another year.
Comment #33
franzWe could definitely use a sprint here, how can we schedule it? Please ping me on irc #drupal-contribute if you're interested.
Comment #33.0
franzAdded items from my comment & fixed typos, formatting.
Comment #34
Ivan Simonov CreditAttribution: Ivan Simonov commentedI am with you guys.
Comment #35
muka CreditAttribution: muka commentedme too!
Comment #35.0
franzAdding issues
Comment #36
geek-merlinSPRINT@MUNICH?
hey, it's late but not too late: i can imagine getting quite some stuff done at an in-place-sprint around drupalcon munich, alongside the multilingual folks. (Drupal 8 Multilingual Initiative sprint before/after DrupalCon Munich 2012, needs sponsors | Drupal Groups)
it's about this weekend 18./19. and maybe the next one 24./25./26.8.
anyone out there who likes that idea too?
Comment #36.0
geek-merlinAdded one more ticket
Comment #36.1
geek-merlindeleted fixed issues
Comment #36.2
geek-merlinreordered issues
Comment #36.3
geek-merlinremoved fixed issue
Comment #36.4
geek-merlinadded issue
Comment #37
geek-merlinOK, i'm at the sprints now and donating spare cycles to clean up this.
As i'm not maintainer i opened up a sandbox.
This is alsy my application for maintainer status.
EDIT:
* cleaned up description, removed already fixed issues.
* propose that we remove more issues from "blocking" as soon as the needs-review-patch in #1033202: [Meta] Generic entity processor is committed. so folks go test that thingee!
* now looking after release blockers
* i'm sticking to this issue's description and reordering it so it is clearer why something is in and out.
* we need manual test of #955236: Handle fatal errors for entity create with better error handler function to fly it in.
* added #1739842: Deprecate some processors in favor of generic entity processor to decide direction
And here's the list of patches i consider ready from minisprint branch:
* #1364116: Option to skip hash check on re-import
* #1698076: Force update for TermProcessor should default to FALSE.
* #1739704: Node lookup by title in Nodeprocessor does not respect nodetype
* #1110762: Feeds does not catch file exceptions properly in file mapper and FeedsParser.inc getFile() function
These need more thorough review:
* (i needed this myself and use it in prod, but not all mappings, so tests needed:) #1689374: Map feed Title and feed URL
* this is a monster patch so i don't trust it until bit more testing: #1127696: Attach multiple importers to one content type (in D7)
Comment #38
podarok#1827164: Integrate Feeds into core
feature request
Comment #38.0
podarokfixed html
Comment #39
rcross CreditAttribution: rcross commentedthe request to integrate feeds into core has been pushed to D9.
Can we get an update on either a plan of action here or if this is just generally considered unmaintained at this point? We really need a stable release so that the security team will address any security issues with this module.
Comment #39.0
rcross CreditAttribution: rcross commentedUpdated issue summary.
Comment #40
tvl CreditAttribution: tvl commentedI don’t have right to complain about releases and bug fixes, as I didn’t commit anything, but it has been a while since the last alpha8 release (13 months).
May I ask if there is a plan for a new alpha or, even better, a beta release (for D7)?
Comment #41
Xen CreditAttribution: Xen at Reload commentedJust release a 2.0 version.
There haven't been a commit for 3 months, it's not like there's a flurry of development at the moment..
This is a four year old module that haven't had a *single* stable release yet. People are using the alpha in production and have done for years.
Yes, there's known issues and things that could be improved, but it's not like the 2.0 version is supposed to be the final one.
Else change the maintenance and development status to reflect reality.
Comment #42
MegaChriz CreditAttribution: MegaChriz commentedPlans for a new alpha release: #2290465: [Meta] Create beta 1 release
Comment #43
gregglesIt's really time to just cut a release. All software has bugs. All software is missing some features. Even if there are things that would be nice to fix prior to a stable release they can wait for the 7.x-2.1 release.
There are about 90,000 sites using feeds. They should not be exposed to having security issues reported in the public queue like #2495145: Possible XSS in PuSHSubscriber.inc.
Comment #44
twistor CreditAttribution: twistor commented@greggles, I agree, I plan on getting it done this weekend.
Comment #45
MegaChriz CreditAttribution: MegaChriz commentedAlthough I agree Feeds is in high need of a stable 7.x-2.0 release, I personally would prefer another alpha release first because I'm a bit concerned about how secure Feeds currently is. For example, an user with the permission to administer Feeds but without the permission to administer users, has still the ability to import users (and he/she can even control the roles these users get).
An other concern is that there are still some issues open that are marked as stable release blockers. At least one of them, #1183440: Multilingual Feeds - Make field import language-aware, requires an API change: #2333029: Extend mapping API to allow for defaults and multiple callbacks. (I see twistor is currently actively working on that, that's great :)). I think we should investigate first if these issues are still valid blockers and if they are not, remove the tag.
@greggles, twistor
What do you think about my concerns? Would fixing these be worth enough to postpone a stable 7.x-2.0 release?
Comment #46
sonicthoughts CreditAttribution: sonicthoughts as a volunteer commentedPlease understand how much dependency there is on patches. Date ical requires alpha8 + patch to import dates! : #2237177: Apply this patch to Feeds 2.0-alpha8 to fix broken date import.
I'd suggest a beta version - why not - feeds is already in so many production sites. I realize there are guidelines but consider the poor mortal system builders too :).
Comment #47
twistor CreditAttribution: twistor commented@MegaChriz, yes that's closer to what I had in mind. I want to get rc1 out the door, and try to shoot for stable in a month after that.
We shouldn't be too scared of a stable, we keep backwards compatibility anyway.
Comment #48
klausiI just filed another security issue: #2502419: Log messages XSS attack vector
Please release Feeds 2.0 so that we can finally have security team support and fix stuff like this in private.
Comment #49
MegaChriz CreditAttribution: MegaChriz commented@klausi
I agree that Feeds needs a 7.x-2.0 release and we are doing our best to get there. I feel that we are very near to get one of the last stable release blockers in: #1183440: Multilingual Feeds - Make field import language-aware, though it is a tough one to test. I hope I can find enough time to get through that one quickly.
Comment #50
klausiThat issue looks like a complex change/addition, so I would NOT release that with Feeds 2.0. Security support is more important right now and we should not throw even more changes at users with the 2.0 release (which will be a security release everyone has to apply).
Comment #51
sonicthoughts CreditAttribution: sonicthoughts as a volunteer commented+1 for @klausi - how do you determine that this is a requirement for a stable 2.0?
Comment #52
twistor CreditAttribution: twistor commented@klausi, the security release isn't going to be 2.0. It's going to be alpha-9. There are already too many changes in dev to make a security release, so we're going to have to cut a separate branch just for it.
Comment #53
klausiOK, so I'm now in the uncomfortable situation that I cannot downgrade to the alpha9 security release because it does not contain all fixes of the dev version.
If I install the most recent dev version I still get an email from sites that I maintain that security releases are available. Can you do a fake commit to the 7.x-2.x dev branch to make the dev release newer than alpha9? Or can you release 2.0 now?
Comment #54
twistor CreditAttribution: twistor commentedThe dev release should have been more recent, I'll find something.
Comment #55
Liam Morland@MegaChriz wrote in #2786487: Create stable release:
Comment #56
kborowycz CreditAttribution: kborowycz commentedAll the conversations I'm seeing for a stable release are well over 6 months old. Where is this at for a stable release?
Comment #57
MegaChriz CreditAttribution: MegaChriz commentedSee #2826403: Plan for Feeds 7.x-2.0-beta4 release for plans for the next beta release. After that I'll have to investigate what still needs to be done. #1995728: [META] Cron import not working on 7.x-2.0-alpha8 and later is marked as critical.