Support from Acquia helps fund testing for Drupal Acquia logo

Comments

marc.groth’s picture

I'd love to help fix this up but I have no idea how it's anticipated to work i.e how will you know what options to give without having a file to map against?

If we could get some indication of how this should work perhaps we can help get it working. With D7 you enter a column name manually whereas here it looks like it should be smarter by offering a select list (which is currently blank as per the original issue).

Any guidance is much appreciated.

Cheers,

Marc

queenvictoria’s picture

Status: Active » Needs work
FileSize
1.69 KB

As we can't know (in all cases) the contents of the CSV (as it mayn't have been uploaded yet) it seems to me that it *has* to be a text field with a column name (or number as indicated by the help text in the "No headers" field under Parser settings).

This patch jumps through some loops to detect if the Mapping sources are empty. Strangely, even those CSV sets this to `FALSE`, the `feed_type->getMappingSources()` call returns an array, length 1, with an empty value.

queenvictoria’s picture

I've deleted this comment because it wasn't correct and it should probably have been in a different issue.

queenvictoria’s picture

I've updated this patch to reduce warnings.

MegaChriz’s picture

In #2465535: [meta] Feeds extensible parsers 8.x roadmap twistor says that configuring the CSV parser will get similar to how you configure the Xpath parser from Feeds Extensible Parsers:

Some things from feeds_ex will get added to Feeds proper once I figure them out. Mostly around how the parsers are configured. I want to change the CSV parser in Feeds to use a similar thing so that the CSV fields are not configured in the mapping. That way, the CSV fields can have machine names, and will make Feeds Tamper's life a lot easier. It will also allow people to map from fields on the Feed entity, as well as CSV fields. That's currently a bit wonky in Feeds 7.

twistor’s picture

Title: CSV parser mapping not working » Implement configurable parsers.
Assigned: Unassigned » twistor
Category: Bug report » Task
Priority: Normal » Major

It's time to get this done.

twistor’s picture

Status: Needs work » Needs review
FileSize
11.16 KB

Here's a quick stab at this. I'm not happy with it yet, and open to ideas.

Specifically, the naming needs work. Dynamic mapper doesn't really make sense since it's just the source. Configurable is already overloaded, but could work. UserConfigurableSourceMapper? Not sure.

Also, the form structure is tied too closely to the FeedTypeForm implementation.

twistor’s picture

Also, this doesn't actually implement the parsing for the CSV parser, just the configuration. Once the configuration is sorted, the actual parsing will be easy.

MegaChriz’s picture

I have been thinking about how to best go forward with configurable parser sources. This comment may get a bit out of focus for this issue, though.

Two workflows

I figured that there are actually two types of workflows for which Feeds can be used:

  • Import a single source.
    Example use cases:
    • User has a single CSV-file with data that he/she wants to import.
  • Import multiple sources.
    Example use cases:
    • User wants to import multiple RSS feeds.
    • User wants their users to import data for a specific content type.

Extract column names from source instead of typing it manually?

For the first workflow, it makes much more sense to provide a file at the beginning of the process. In this case the system (Feeds) could extract the parser sources from the provided file, so the user doesn't have to type over the column names from the source file (in case of a CSV file). This would be a great UX improvement, in my opinion.

Or keep only the method to type in sources manually?

For the second workflow, the user wants to setup a sort of "import template" and may not even have the file to import yet. For these cases, it may be undesired to have the user to provide a file immediately.

Feeds (and the 8.x-3.x version in particular) currently focusses on the second workflow. This makes use cases that more belong to the first workflow a longer process.

Mocks

I've tried to create some mocks in which the two workflows could go hand in hand. I'm not happy with them yet, but maybe this helps to come up with new ideas. I would like an interface like Panels has.
The mocks are inspired by the WP All Import plugin for Wordpress.

Step 1: Fetcher: File upload

A file can be optionally uploaded, so the sources can be extracted from it by the parser.
fetcher

Step 2: Parser: CSV

When selecting a parser, the parser immediately will try to parse the provided file in step 1 (if it was provided). In case of the CSV parser, this means it will prefill the sources config table with the found column names. Below the config table the complete parser result is shown, so you can see what's in the file.
csv parser

Step 2: Parser: XML

When selecting the XML parser (which will be provided by Feeds Extensible parsers), the complete contents of the file is shown in a box. When typing in a different "Context" the box is updated with only the XML relevant within that context. Below the box, a list of xpaths can be typed in. Any change will directly update the parser result so you have instant feedback on what the set of records to import look like.
The box with the XML contents and the parser result will not be shown if no file was provided during the fetcher step.
xml parser

Step 3: Processor

The processor step wouldn't change much. Note that I removed the "Replace" option so that "Update existing" becomes a single checkbox. I personally still do not know the advantages of the "Replace" option. I also added the option "Import new items" (this idea comes from #1286298: Don't create new items, only update existing).
processor

Step 4 would be the mapping configuration. I have no mock created for that step. Step 5 would be a screen where the final import can be done (following workflow 1) in case a file was provided in step 1. The final import can be skipped (for users that want to import series of files, following workflow 2). In this case or if no file was provided, the user will be redirected to the page where it can add a new feed (admin/content/feed).

twistor’s picture

That's is nifty! How do you envision these steps would be outlined? How can the user get back and forth between steps? I guess, what does the outside of the UI look like? Tabs? I'd like to try and stick with Drupal's default elements as much as possible, but am open to other things.

I think we can get part-way there with an "Upload sample file" type form. There's an old issue somewhere that has been trying to do that for ages.

For mapping, I've been thinking about implementing an auto-mapping feature. Actually, I did, but then ripped it out. The basic idea is that parsers would provide suggestions on what type of fields their sources should go, and something would look at the entity fields and match them up as best as possible. This would only work for non-configurable parsers, but would still be helpful.

kclarkson’s picture

Just noticed this as I am preparing to try a simple feeds import into D8.

@MegaChriz I think the mock ups look great!! I like the simplicity.

As far as the navigation between these pages, I agree with @twistor. I would like to see the continuation of the left hand side Tabs for the Navigation between these pages though.

Honza Pobořil’s picture

FileSize
35.19 KB

If there is discussion about improving UI, how about to make it more easy to chain multiple importers? Internally it is just needed to integrate with Rules - to trigger importer by Rules action (#2446307: Trigger Feed Import Using Rules) and to have Rules as a processor (module there).

It will be useul for use cases where we need to load additional data for every item by additional request, or recursive importing. Example: Import FB events from group by /{group-id}/events and then for every event load attendees by /{event-id}/attending.

Currently we have go throught multiple UIs to setup Rules actions adn Feeds importers. It will be much more easier if Rule configuration screen will be part of the Feeds processor configuration.

Mockup of Rules processor

MegaChriz’s picture

@Bobík
Chained importers and involving Rules in the process sounds interesting (but also complicated). I've a hard time though to visualize how that would work. It sounds also like it should be an add-on instead of a Feeds core feature (because of the complicateness). Maybe we can discuss this better in an other issue? It seems like there is also discussion about this subject in #2300927: New architecture for new Drupal or is that a different issue?

Honza Pobořil’s picture

@MegaChriz: Chainging could be done, if there will be proper plugins implementing Rules. Rules are flexible enough to do any usecase, just Feeds are hardcoded to only few usecases.

It will be nice to rewrite Feeds architecture from 4 steps workflow (fetch, parse, process and map) to something flexible like Rules (any number of steps, branching and still support batching). But I am skeptical, because I saw many "fuck off"s in this issue queue to feature requests requiring architecture improves.

If Feeds core developers are happy do make big changes, I will support it, but for now Rules integrations seems more realistic.

eugene.ilyin’s picture

FileSize
17.14 KB

Patch from #4 helped me to configure mapping.
It looks like this

mapping

It works, is it possible to commit it?

MegaChriz’s picture

I have temporary renamed the CSV parser to "CSV (not working yet, do not use)" since a lot of people report that it is not working.

@eugene.ilyin
The patch in #4 works indeed for now, but I and twistor think that this solution is inflexible in the long run. The D7 version, where sources were also entered manually for the CSV parser on the mapping form, showed us some problems. For example, additional mapping sources added by other modules couldn't be used. Feeds Tamper had to add special case solutions for the CSV parser. We should try to avoid these problems in the D8 version. The sources should be configured as part of the parser settings, similar to how the D7 version of the Feeds extensible parsers does. Or better: read them directly from a provided file as proposed in #9.

I'm looking for suggestions to improve the UI that is used in #9.

josmera01’s picture

Hello MegaChriz
I'd like contrib and add the option for the csv type to the feeds module. Are you working on this option? or I could begin working on that.

ftyre76’s picture

Status: Needs review » Needs work

1) All the patches I tried to use on this page failed. I tried to manually implement the changes, but there were considerable changes to interfaces used in twistor's patch (it has been several months since the patch was added to the discussion so no big surprise there).
2) It seems to me that the building / configuring sources should be on its own tab.
3) I think we need to build the manual way of configuring a feed type and then work on automating that process (to hopefully get unstuck).
4) I don't understand why the form isn't using the content type we are importing into to determine how to build the sources configuration settings.

g089h515r806’s picture

#4 works for csv parser.

veronicaSeveryn’s picture

Agreed. Have been using Feeds with Patch from #4 and it works well for me too.
Tested on DEV branch from 2016-07-12 and from 2016-10-04

MrPeanut’s picture

Excuse my ignorance, but I've applied the patch from #4. I can type in the source field now, but my node fields (other than body, title, and some other default options) don't appear as targets. Am I missing something or do additional fields not appear at this time?

MegaChriz’s picture

@MrPeanut
Additional fields should be available once you selected a content type. It could be though that not all field types are supported yet as target. I think currently none from contrib.

MrPeanut’s picture

Thanks, MegaChriz. You were right — date range is not yet supported.

kshm’s picture

@MegaChriz I know there's a possibility this is getting worked on this weekend so I wanted to give you what I had done from some work over the last couple days re: the new workflow. This is by no means complete, but might be a good jumping off point for getting the multi-step forms ironed out.

So far I've gotten the fetcher, parser and processor forms separated and tabbed (per twistor's recommendation). It's set up, currently, to redirect from the fetcher > parser > processor forms on submission with the option of using the tabs to reach each form individually once a feed type is created.

I didn't have time to integrate any additional fetcher configuration so that will need to be added next before moving onto the configurable parsers aspect.

I hope this is useful!

thomasmurphy’s picture

Hi, what resources are currently needed to resolve this ticket? This page
https://www.drupal.org/node/2604358

mentions the need for a UX developer, does that mean a module developer to build the UI?

MegaChriz’s picture

@thomasmurphy
For Feeds in D8 I had in mind that the process for the site builder should become easier. Feeds should help the site builder with importing its content. For example: when someone has a CSV file to import, Feeds could extract the columns from the file, make a proposal of how things should be mapped based on the column names and show a preview of the import. Problem is that Feeds should also target users that want to build the mapping first so they know what the source file should look like.
I'm looking for an UI designer who could design an UI that would target both type of users. I'm currently in contact with someone who is working on a proposal, but more input is always welcome!

@kshm
Sorry, I haven't found the time yet to look at your work. I hope to make time for that next week, though there are also some other issues in Feeds that I would like to give attention.

thomasmurphy’s picture

@MegaChriz we can probably donate some time from one of our UI designers, I'll look into it. In terms of what Feeds could potentially be for Drupal 8, what I've generally heard from the Drupal community is that there is an urgent need for a bare-bones version of feeds 8.x that does essential things for skilled users, before too much energy goes into creating a better experience for less advanced users, but I suppose you'll get a better picture from the survey. I'll join the conversation on slack soon to see if I can help with planning :)

irinaz’s picture

Issue summary: View changes
FileSize
167.48 KB
123.69 KB
122.39 KB

Here is another version of mockups for CSV parser

When new feed is being set up a user is directed to mapping tab in a way similar to “Save and Add Fields” for content type
edit tab for new feed
mapping csv tab for new feed
update tab for   feed

bucefal91’s picture

I know this issue is long already so I'll try to be short. I am right now right at the stage where I must decide whether to do my CSV import via Migrate API or Feeds. I would classify my usecase as #2, i.e. I have a template and every now and then other people will upload distinct CSV files with the same column names.

If I were to do it via Feeds, I would like to see the following UI.

Step 1. Set up the fetcher and processor
 processor

Step 2. Mapping
 Mapping

Step 3: Importing a file
 Importing a file

You create a feed importer as normal, but now you have an option to upload a sample file. This way implicitly we can quickly determine whether we are facing use case #1 or use case #2 (if the guy uploads a sample file this means he's case #1 and vice-versa).
If the guy didn't upload a sample file there's just not much to alter/improve in the ongoing screens because we have no additional knowledge on which to build our better UX
However, if the guy did upload a sample file, then:

  • We can automatically build a "smart" mapping so he does not have to start from an empty screen but with what we think is a reasonable default. This part can actually be put off for later and currently we can just put some kind of method stub in an interface or so.
  • We can also offer a dropdown list of source fields based on the headers of the sample file. I would still leave the option to type in a column name just in case our user is an advanced one and he just knows what he's doing.
  • Lastly, when importing a feed, we should give the user an option:
    • to run the importer on the sample file
    • to review the sample file so he can quickly remind himself what are the expected column names

I've tried to combine #9 and #28. I also think these mockups while do embrace both use cases they do not put any hard boundaries between the 2 use cases and do all its UX implicitly (the opposite would be asking the user as the 1st step: are you use case #1 or use case #2?).. and not the least, we do not go too far away from what already exists in D8 feeds thus keeping the amount of work reasonable.

MegaChriz’s picture

Awesome, bucefal91!

Note that the dropdown should also be available when a parser is chosen that provides predefined sources, like the parser for RSS does.

kumkum29’s picture

Hello,

Is there a possibility of importing the data via a csv file? Do you think create a next version of "feeds" with this very useful option (and xml) ? I see some posts and patchs with this problem. Apparently there is a problem with the UI.

Thanks for your replies on this important subject !!!

MegaChriz’s picture

@kumkum29
I'm planning to pickup the development of Feeds again soon. First I will release Feeds 7.x-2.0-beta4 (planned for this Sunday). After that I will reorganize the issues a bit and I'll try to gather all the information required for the next steps in one place. I hope that once the plan with substeps are written, other developers will chime in to help get CSV parser, XML parser and Feeds Tamper done.

Summarized plan of the next steps:
https://www.drupal.org/node/2604358#comment-12258151

kumkum29’s picture

@MegaChriz

Thanks for this information. I will follow this post and test the next feeds versions / patchs.
(Good news for Feeds Tamper and the XML parser, useful tools for feeds)

thomasmurphy’s picture

@MegaChriz Do you want us to put some effort into this issue at Drupal South code sprint? If so could I get an overview of what progress you have made so far, eg is the functionality from
https://www.drupal.org/sandbox/megachriz/2465523
now incorporated into feeds 8, or are additions required?
Also, what's the level of coordination, if any, with the d8 tamper API?

irinaz’s picture

@thomasmurphy we are also working on this right now, can we get in touch with you directly? We are in BADCamp, and plan to have another code sprint tomorrow or Saturday.
here are most recent issue links
UI - https://www.drupal.org/node/2917405
csv import - https://www.drupal.org/node/2669974
email me irinaz@iringweb.com

MegaChriz’s picture

Issue summary: View changes
Status: Needs work » Closed (duplicate)

This issue has gone into multiple directions. It tried to be these three things:

  1. Implement an UI for the CSV parser
    Now being worked on in #2669974: Implement the CSV parser UI
  2. General UI discussion
    This can now be discussed in #2917405: UI designs
  3. Planning for the next step
    An overview of the D8 porting status: #1960800: [meta] Feeds 8.x roadmap
    Plan for the first alpha release: #2917928: Plan for 8.x-3.0-alpha1 release

Therefore, I'm closing this as a duplicate now.