Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
For work on implementing the UI of the CSV parser, see:
#2669974: Implement the CSV parser UI
For all UI designs and discussion about them see:
#2917405: UI designs
For the plan of the first next steps to take, see:
#2917928: Plan for 8.x-3.0-alpha1 release
For the long term plan of porting Feeds to D8, see:
#1960800: [meta] Feeds 8.x roadmap
Comment | File | Size | Author |
---|---|---|---|
#29 | Screenshot from 2017-04-27 14-42-14.png | 149.04 KB | bucefal91 |
#29 | Screenshot from 2017-04-27 14-23-28.png | 173.91 KB | bucefal91 |
#29 | Screenshot from 2017-04-27 14-18-02.png | 175.78 KB | bucefal91 |
#28 | EditTabNewFeed.png | 122.39 KB | irinaz |
#28 | EditTabUpdate.png | 123.69 KB | irinaz |
Comments
Comment #1
marc.groth CreditAttribution: marc.groth commentedI'd love to help fix this up but I have no idea how it's anticipated to work i.e how will you know what options to give without having a file to map against?
If we could get some indication of how this should work perhaps we can help get it working. With D7 you enter a column name manually whereas here it looks like it should be smarter by offering a select list (which is currently blank as per the original issue).
Any guidance is much appreciated.
Cheers,
Marc
Comment #2
queenvictoria CreditAttribution: queenvictoria commentedAs we can't know (in all cases) the contents of the CSV (as it mayn't have been uploaded yet) it seems to me that it *has* to be a text field with a column name (or number as indicated by the help text in the "No headers" field under Parser settings).
This patch jumps through some loops to detect if the Mapping sources are empty. Strangely, even those CSV sets this to `FALSE`, the `feed_type->getMappingSources()` call returns an array, length 1, with an empty value.
Comment #3
queenvictoria CreditAttribution: queenvictoria commentedI've deleted this comment because it wasn't correct and it should probably have been in a different issue.
Comment #4
queenvictoria CreditAttribution: queenvictoria commentedI've updated this patch to reduce warnings.
Comment #5
MegaChriz CreditAttribution: MegaChriz commentedIn #2465535: [meta] Feeds extensible parsers 8.x roadmap twistor says that configuring the CSV parser will get similar to how you configure the Xpath parser from Feeds Extensible Parsers:
Comment #6
twistor CreditAttribution: twistor commentedIt's time to get this done.
Comment #7
twistor CreditAttribution: twistor commentedHere's a quick stab at this. I'm not happy with it yet, and open to ideas.
Specifically, the naming needs work. Dynamic mapper doesn't really make sense since it's just the source. Configurable is already overloaded, but could work. UserConfigurableSourceMapper? Not sure.
Also, the form structure is tied too closely to the FeedTypeForm implementation.
Comment #8
twistor CreditAttribution: twistor commentedAlso, this doesn't actually implement the parsing for the CSV parser, just the configuration. Once the configuration is sorted, the actual parsing will be easy.
Comment #9
MegaChriz CreditAttribution: MegaChriz commentedI have been thinking about how to best go forward with configurable parser sources. This comment may get a bit out of focus for this issue, though.
Two workflows
I figured that there are actually two types of workflows for which Feeds can be used:
Example use cases:
Example use cases:
Extract column names from source instead of typing it manually?
For the first workflow, it makes much more sense to provide a file at the beginning of the process. In this case the system (Feeds) could extract the parser sources from the provided file, so the user doesn't have to type over the column names from the source file (in case of a CSV file). This would be a great UX improvement, in my opinion.
Or keep only the method to type in sources manually?
For the second workflow, the user wants to setup a sort of "import template" and may not even have the file to import yet. For these cases, it may be undesired to have the user to provide a file immediately.
Feeds (and the 8.x-3.x version in particular) currently focusses on the second workflow. This makes use cases that more belong to the first workflow a longer process.
Mocks
I've tried to create some mocks in which the two workflows could go hand in hand. I'm not happy with them yet, but maybe this helps to come up with new ideas. I would like an interface like Panels has.
The mocks are inspired by the WP All Import plugin for Wordpress.
Step 1: Fetcher: File upload
A file can be optionally uploaded, so the sources can be extracted from it by the parser.
Step 2: Parser: CSV
When selecting a parser, the parser immediately will try to parse the provided file in step 1 (if it was provided). In case of the CSV parser, this means it will prefill the sources config table with the found column names. Below the config table the complete parser result is shown, so you can see what's in the file.
Step 2: Parser: XML
When selecting the XML parser (which will be provided by Feeds Extensible parsers), the complete contents of the file is shown in a box. When typing in a different "Context" the box is updated with only the XML relevant within that context. Below the box, a list of xpaths can be typed in. Any change will directly update the parser result so you have instant feedback on what the set of records to import look like.
The box with the XML contents and the parser result will not be shown if no file was provided during the fetcher step.
Step 3: Processor
The processor step wouldn't change much. Note that I removed the "Replace" option so that "Update existing" becomes a single checkbox. I personally still do not know the advantages of the "Replace" option. I also added the option "Import new items" (this idea comes from #1286298: Don't create new items, only update existing).
Step 4 would be the mapping configuration. I have no mock created for that step. Step 5 would be a screen where the final import can be done (following workflow 1) in case a file was provided in step 1. The final import can be skipped (for users that want to import series of files, following workflow 2). In this case or if no file was provided, the user will be redirected to the page where it can add a new feed (admin/content/feed).
Comment #10
twistor CreditAttribution: twistor commentedThat's is nifty! How do you envision these steps would be outlined? How can the user get back and forth between steps? I guess, what does the outside of the UI look like? Tabs? I'd like to try and stick with Drupal's default elements as much as possible, but am open to other things.
I think we can get part-way there with an "Upload sample file" type form. There's an old issue somewhere that has been trying to do that for ages.
For mapping, I've been thinking about implementing an auto-mapping feature. Actually, I did, but then ripped it out. The basic idea is that parsers would provide suggestions on what type of fields their sources should go, and something would look at the entity fields and match them up as best as possible. This would only work for non-configurable parsers, but would still be helpful.
Comment #11
kclarkson CreditAttribution: kclarkson commentedJust noticed this as I am preparing to try a simple feeds import into D8.
@MegaChriz I think the mock ups look great!! I like the simplicity.
As far as the navigation between these pages, I agree with @twistor. I would like to see the continuation of the left hand side Tabs for the Navigation between these pages though.
Comment #12
Honza Pobořil CreditAttribution: Honza Pobořil as a volunteer commentedIf there is discussion about improving UI, how about to make it more easy to chain multiple importers? Internally it is just needed to integrate with Rules - to trigger importer by Rules action (#2446307: Trigger Feed Import Using Rules) and to have Rules as a processor (module there).
It will be useul for use cases where we need to load additional data for every item by additional request, or recursive importing. Example: Import FB events from group by /{group-id}/events and then for every event load attendees by /{event-id}/attending.
Currently we have go throught multiple UIs to setup Rules actions adn Feeds importers. It will be much more easier if Rule configuration screen will be part of the Feeds processor configuration.
Comment #13
MegaChriz CreditAttribution: MegaChriz commented@Bobík
Chained importers and involving Rules in the process sounds interesting (but also complicated). I've a hard time though to visualize how that would work. It sounds also like it should be an add-on instead of a Feeds core feature (because of the complicateness). Maybe we can discuss this better in an other issue? It seems like there is also discussion about this subject in #2300927: New architecture for new Drupal or is that a different issue?
Comment #14
Honza Pobořil CreditAttribution: Honza Pobořil as a volunteer commented@MegaChriz: Chainging could be done, if there will be proper plugins implementing Rules. Rules are flexible enough to do any usecase, just Feeds are hardcoded to only few usecases.
It will be nice to rewrite Feeds architecture from 4 steps workflow (fetch, parse, process and map) to something flexible like Rules (any number of steps, branching and still support batching). But I am skeptical, because I saw many "fuck off"s in this issue queue to feature requests requiring architecture improves.
If Feeds core developers are happy do make big changes, I will support it, but for now Rules integrations seems more realistic.
Comment #15
eugene.ilyin CreditAttribution: eugene.ilyin as a volunteer and at DrupalJedi commentedPatch from #4 helped me to configure mapping.
It looks like this
It works, is it possible to commit it?
Comment #16
MegaChriz CreditAttribution: MegaChriz commentedI have temporary renamed the CSV parser to "CSV (not working yet, do not use)" since a lot of people report that it is not working.
@eugene.ilyin
The patch in #4 works indeed for now, but I and twistor think that this solution is inflexible in the long run. The D7 version, where sources were also entered manually for the CSV parser on the mapping form, showed us some problems. For example, additional mapping sources added by other modules couldn't be used. Feeds Tamper had to add special case solutions for the CSV parser. We should try to avoid these problems in the D8 version. The sources should be configured as part of the parser settings, similar to how the D7 version of the Feeds extensible parsers does. Or better: read them directly from a provided file as proposed in #9.
I'm looking for suggestions to improve the UI that is used in #9.
Comment #17
josmera01 CreditAttribution: josmera01 at Globant commentedHello MegaChriz
I'd like contrib and add the option for the csv type to the feeds module. Are you working on this option? or I could begin working on that.
Comment #18
ftyre76 CreditAttribution: ftyre76 as a volunteer commented1) All the patches I tried to use on this page failed. I tried to manually implement the changes, but there were considerable changes to interfaces used in twistor's patch (it has been several months since the patch was added to the discussion so no big surprise there).
2) It seems to me that the building / configuring sources should be on its own tab.
3) I think we need to build the manual way of configuring a feed type and then work on automating that process (to hopefully get unstuck).
4) I don't understand why the form isn't using the content type we are importing into to determine how to build the sources configuration settings.
Comment #19
g089h515r806 CreditAttribution: g089h515r806 commented#4 works for csv parser.
Comment #20
veronicaSeveryn CreditAttribution: veronicaSeveryn commentedAgreed. Have been using Feeds with Patch from #4 and it works well for me too.
Tested on DEV branch from 2016-07-12 and from 2016-10-04
Comment #21
MrPeanut CreditAttribution: MrPeanut commentedExcuse my ignorance, but I've applied the patch from #4. I can type in the source field now, but my node fields (other than body, title, and some other default options) don't appear as targets. Am I missing something or do additional fields not appear at this time?
Comment #22
MegaChriz CreditAttribution: MegaChriz commented@MrPeanut
Additional fields should be available once you selected a content type. It could be though that not all field types are supported yet as target. I think currently none from contrib.
Comment #23
MrPeanut CreditAttribution: MrPeanut commentedThanks, MegaChriz. You were right — date range is not yet supported.
Comment #24
kshm CreditAttribution: kshm commented@MegaChriz I know there's a possibility this is getting worked on this weekend so I wanted to give you what I had done from some work over the last couple days re: the new workflow. This is by no means complete, but might be a good jumping off point for getting the multi-step forms ironed out.
So far I've gotten the fetcher, parser and processor forms separated and tabbed (per twistor's recommendation). It's set up, currently, to redirect from the fetcher > parser > processor forms on submission with the option of using the tabs to reach each form individually once a feed type is created.
I didn't have time to integrate any additional fetcher configuration so that will need to be added next before moving onto the configurable parsers aspect.
I hope this is useful!
Comment #25
thomasmurphy CreditAttribution: thomasmurphy commentedHi, what resources are currently needed to resolve this ticket? This page
https://www.drupal.org/node/2604358
mentions the need for a UX developer, does that mean a module developer to build the UI?
Comment #26
MegaChriz CreditAttribution: MegaChriz commented@thomasmurphy
For Feeds in D8 I had in mind that the process for the site builder should become easier. Feeds should help the site builder with importing its content. For example: when someone has a CSV file to import, Feeds could extract the columns from the file, make a proposal of how things should be mapped based on the column names and show a preview of the import. Problem is that Feeds should also target users that want to build the mapping first so they know what the source file should look like.
I'm looking for an UI designer who could design an UI that would target both type of users. I'm currently in contact with someone who is working on a proposal, but more input is always welcome!
@kshm
Sorry, I haven't found the time yet to look at your work. I hope to make time for that next week, though there are also some other issues in Feeds that I would like to give attention.
Comment #27
thomasmurphy CreditAttribution: thomasmurphy commented@MegaChriz we can probably donate some time from one of our UI designers, I'll look into it. In terms of what Feeds could potentially be for Drupal 8, what I've generally heard from the Drupal community is that there is an urgent need for a bare-bones version of feeds 8.x that does essential things for skilled users, before too much energy goes into creating a better experience for less advanced users, but I suppose you'll get a better picture from the survey. I'll join the conversation on slack soon to see if I can help with planning :)
Comment #28
irinaz CreditAttribution: irinaz as a volunteer commentedHere is another version of mockups for CSV parser
When new feed is being set up a user is directed to mapping tab in a way similar to “Save and Add Fields” for content type
Comment #29
bucefal91 CreditAttribution: bucefal91 at Websolutions Agency commentedI know this issue is long already so I'll try to be short. I am right now right at the stage where I must decide whether to do my CSV import via Migrate API or Feeds. I would classify my usecase as #2, i.e. I have a template and every now and then other people will upload distinct CSV files with the same column names.
If I were to do it via Feeds, I would like to see the following UI.
Step 1. Set up the fetcher and processor
Step 2. Mapping
Step 3: Importing a file
You create a feed importer as normal, but now you have an option to upload a sample file. This way implicitly we can quickly determine whether we are facing use case #1 or use case #2 (if the guy uploads a sample file this means he's case #1 and vice-versa).
If the guy didn't upload a sample file there's just not much to alter/improve in the ongoing screens because we have no additional knowledge on which to build our better UX
However, if the guy did upload a sample file, then:
I've tried to combine #9 and #28. I also think these mockups while do embrace both use cases they do not put any hard boundaries between the 2 use cases and do all its UX implicitly (the opposite would be asking the user as the 1st step: are you use case #1 or use case #2?).. and not the least, we do not go too far away from what already exists in D8 feeds thus keeping the amount of work reasonable.
Comment #30
MegaChriz CreditAttribution: MegaChriz commentedAwesome, bucefal91!
Note that the dropdown should also be available when a parser is chosen that provides predefined sources, like the parser for RSS does.
Comment #31
kumkum29 CreditAttribution: kumkum29 commentedHello,
Is there a possibility of importing the data via a csv file? Do you think create a next version of "feeds" with this very useful option (and xml) ? I see some posts and patchs with this problem. Apparently there is a problem with the UI.
Thanks for your replies on this important subject !!!
Comment #32
MegaChriz CreditAttribution: MegaChriz commented@kumkum29
I'm planning to pickup the development of Feeds again soon. First I will release Feeds 7.x-2.0-beta4 (planned for this Sunday). After that I will reorganize the issues a bit and I'll try to gather all the information required for the next steps in one place. I hope that once the plan with substeps are written, other developers will chime in to help get CSV parser, XML parser and Feeds Tamper done.
Summarized plan of the next steps:
https://www.drupal.org/node/2604358#comment-12258151
Comment #33
kumkum29 CreditAttribution: kumkum29 commented@MegaChriz
Thanks for this information. I will follow this post and test the next feeds versions / patchs.
(Good news for Feeds Tamper and the XML parser, useful tools for feeds)
Comment #34
thomasmurphy CreditAttribution: thomasmurphy commented@MegaChriz Do you want us to put some effort into this issue at Drupal South code sprint? If so could I get an overview of what progress you have made so far, eg is the functionality from
https://www.drupal.org/sandbox/megachriz/2465523
now incorporated into feeds 8, or are additions required?
Also, what's the level of coordination, if any, with the d8 tamper API?
Comment #35
irinaz CreditAttribution: irinaz as a volunteer commented@thomasmurphy we are also working on this right now, can we get in touch with you directly? We are in BADCamp, and plan to have another code sprint tomorrow or Saturday.
here are most recent issue links
UI - https://www.drupal.org/node/2917405
csv import - https://www.drupal.org/node/2669974
email me irinaz@iringweb.com
Comment #36
MegaChriz CreditAttribution: MegaChriz commentedThis issue has gone into multiple directions. It tried to be these three things:
Now being worked on in #2669974: Implement the CSV parser UI
This can now be discussed in #2917405: UI designs
An overview of the D8 porting status: #1960800: [meta] Feeds 8.x roadmap
Plan for the first alpha release: #2917928: Plan for 8.x-3.0-alpha1 release
Therefore, I'm closing this as a duplicate now.