Implement the CSV parser UI

Some initial comments:

1. +++ b/src/Entity/FeedType.php
   @@ -141,6 +141,13 @@ class FeedType extends ConfigEntityBundleBase implements FeedTypeInterface, Enti
   +  ¶
   

   Trailing spaces.

2. +++ b/src/FeedTypeInterface.php
   @@ -123,5 +123,13 @@ interface FeedTypeInterface extends ConfigEntityInterface {
   +  ¶
   

   Trailing spaces.

3. +++ b/src/Feeds/Parser/CsvParser.php
   @@ -7,30 +7,31 @@
   +class CsvParser extends PluginBase implements ParserInterface, FeedPluginFormInterface {
   

   I think if you don't extend ConfigurablePluginBase you wont be able to export this settings.

4. +++ b/src/Feeds/Parser/CsvParser.php
   @@ -130,31 +128,8 @@ class CsvParser extends ConfigurablePluginBase implements FeedPluginFormInterfac
   -  public function buildConfigurationForm(array $form, FormStateInterface $form_state) {
   

   Why are you removing this code?

5. +++ b/src/Plugin/Type/PluginBase.php
   @@ -218,5 +218,8 @@ abstract class PluginBase extends DrupalPluginBase implements FeedsPluginInterfa
   +  ¶
   

   Trailing spaces

I'm sorry to say, but the CSV sources should not be defined on the mapping form anymore. I know this is how it works in D7, but that caused inflexibilities which I pointed out in other issues. First #2656496: Parser not getting updated should be fixed. The CSV sources need to be defined on the parser settings screen.

The CSV sources should be defined in the parser settings section on the feed type form. This cannot be done until #2656496: Parser not getting updated is fixed, because the parser config form is not loaded when the CSV parser is selected.

See image:

This is a sketch of how the parser sources could be defined in CSV, but I'm not comfortable with the UI design yet:

This issue is actually a duplicate of #2443471: Implement configurable parsers., but since that issue is going much further we can keep this one open for now.

Hi, just stopping by to see what the progress is on the issue. Can you give an estimate when csv will be ready to use?

What is the current state of this? Are there still plans regarding CSV support for the future of this module?

@Daniel Schaefer
Yes, there are still plans for CSV support in the future. I cannot give any estimates however, as I'm currently not actively working on the D8 version of Feeds.

Are you aware of any possible alternatives? We need to import Drupal 7 nodes, however we did not want to migrate the entire site for the sake of a clean and fresh install.

@Daniel Schaefer
See my answer in #2696247: Importing nodes alternatives to Feeds?.

Just FYI I would I will be reading the threads for csv support. I have a number of instances which are coming up where I need to load bulk content in csv or xml format. I am examining migrate module, but customarily used feeds in 7 for these bulk imports. Use me for testing.

I'm with RKopacz. I have a dozen sites that depend on Feeds pulling data from CSV files many times a day. I'm not a coder but anything I can do to help move the CSV function along including testing etc., I'm in. I know you're all busy, but I'm a bit surprised that this features implementation has languished for so long. Would have thought there was more demand for it.

@tedwyer I just succeeded in importing 138 records to a D8 site from a CSV using the migrate module in core, with the help of the migrate source csv file. I'd be happy to share it in the right place, once I got the YML file sorted it was pretty straightforward. I was even able to import a field with multiple items, but it took me a while to figure it out.

Thanks RKopack. I'd love to see what you're doing. Not sure what you mean by right place but I'm certainly interested.

RKopacz, thanks again. Already had had a look at that CSV module but had another look and I think I can figure it out. Your direction to the yml example will help a lot.

TD

@RKopacz indeed, I would love to see how you did it, very interested!

I managed to use the latest dev to import a few thousand nodes of varying content types. Some of which have up to 25 fields. I used the patch from #4 here Implement configurable parsers. It was applied manually as things have changed since then.
Not sure if its meant to work right now, but for me it did the job nicely.

For those interested, @Jeff Burnz and @tedwyer the migration module works pretty well with CSV files. This particular example module can be modified to meet whatever needs you have.

https://github.com/heddn/d8_custom_migrate

I had a special need to migrate data to a multiple value field (ie a field which can accommodate unlimited number of items), and after a lot of scratching around, found the solution. I am pasting the Process part of the YAML file which goes in the config/install directory of the custom module, which was adapted from the above code.

process:
  # The content (node) type we are creating is 'founder'.
  type:
    plugin: default_value
    default_value: founder
  # Most fields can be mapped directly - we just specify the destination (D8)
  # field and the corresponding field name from above, and the values will be
  # copied in.
  title: name
  field_founder_first_name: first_name
  field_founder_last_name: last_name
  field_founder_birth_death: birth_death
  body: biography
  field_founder_references:
    plugin: explode
    limit: 100
    source: references
    delimiter: '|'

Note the field called field_founder_references. Using the explode plugin which is part of the Migrate Source CSV module. One caveat is that that particular module requires the delimiter to be offset in single quotes. I've seen this configuration also without the single quotes, but with Migrate Source CSV you need the single quotes.

All the other fields are straight plain text fields, and the text to the right of the colon for each item are the column labels in the CSV.

I'm going to be attempting more complex fields, such as date fields, image fields, and pulling in taxonomy fields (there is a nice, well documented set of example modules in the Migrate Plus module which gets into it deeper).

I loved using feeds in D7 for CSVs, but it looks like Migrate is the future, and once it gets a UI so that content managers can start doing regular uploads, it should be a great solution.

#2656496: Parser not getting updated was done, so this issue can be reopened.

Wouldn't the patch from #2443471-4: Implement configurable parsers. solve this issue temporarily? There's already a big "DONT USE THIS BITES" label when you choose CSV, I'm guessing there shouldn't be too much trouble with nuking people's importers whenever a definitive solution rolls out.

Just my two cents (since I can't actually help out, probably worth even less), but this still seems like one of the bigger issues with D8 to me right now.

Hi,

Any one has successfully used this module to import from CSV file to create nodes under Drupal 8?

Eager to do CSV import in my project, but the module setting page says do not use it for CSV.
Thanks.

@huida #21, I am successfully using the patch from #4 on https://www.drupal.org/node/2443471 to import nodes from a D6 site without issue on the latest dev. Images are created, taxonomy terms, etc. There will of course still be some bugs and unfinished work on this module while work is going on. Highly recommended you make a backup before starting.

Hello,

I want to import nodes from a csv file, but I don't see how to configure the importer. In the mapping settings, the source list is always empty... How to call the datas of my csv file in this list? It's a little different in comparison to D7 interface...
Can you help me to test this module?

thanks.

Just a side note for those needing a D6->D8 CSV solution today: Over on the D8 Migrate front, the Migrate Source CSV plugin module (https://www.drupal.org/project/migrate_source_csv) is working very well and has a stable release. The YAML configuration is trickier than feeds, but once you get over the learning mountain it's very robust.

I'm still looking forward to a feeds release though.

I've set a goal to fix this issue during the Global Drupal sprint weekend. I'll be in Amsterdam on Saturday. Who wants to help getting this issue done, join the #feeds channel at https://drupal.slack.com or send me a mail via my contact form.

@TrevorBradley

If we want import contents in different languages, the solution with migrate module (& migrate source csv) seems to be not very functionnal... At this moment, there is no valid solution (feeds/migrate) to import multilingual contents...

Hi @MegaChriz,

Di you manage to work on this during the Global Drupal sprint weekend? Is there a solution now for CSV import? I hope so. Please keep us updated.

Thanks!

@memoday
No, during the global sprint weekend in January we did not manage to get this job done. Instead we took a step back to create a survey from which the results can be seen in #2847682: Drupal 8 survey Feeds/Migrate usage and functionality. A few have been working on new UI designs for the CSV parser in #2443471: Implement configurable parsers.. I hope to gather enough energy and time to implement them. Goal for having these finished is the end of June 2017. I hope I can make that, as I'll also be on holiday for two weeks in June (and I'm also trying to finish some stuff for the D7 version of Feeds, which also slows this down). If I can find help from other coders, it should be possible.

Made a start with a possible implementation. This adds an option "New source..." to the mapping sources select list. It is currently non-functional.

@MegaChriz Thanks for the update. I really hope I can help out with the code, but I am not familiar with D8 yet. I hope other coders will jump in and give you a helping hand. Thanks again for your efforts and the great module.

Not finished yet, but a step closer. This should save the custom entered sources. I haven't tested yet if this makes the CSV parser work with these custom sources. Also some methods need to be added FeedTypeInterface to handle these custom sources.

@MegaChriz Thank you so much for the update. Should I install the dev version and test it? Or should I just apply the patch in "interdiff-2669974-29-31.txt"?

Thanks again!

@memoday
Great that you want to give it try! When testing, it would be best to focus on the UX first in this case, though it would also be helpful to know if custom sources do work with the CSV parser.
Best would be to get the latest dev first and then apply the patch. Interdiffs are meant to see the changes between two patches, see also https://www.drupal.org/documentation/git/interdiff.

Hi @MegaChriz,

I have just found out that for some reason, we cannot uninstall 'installed' modules :-(

I deleted the old dev version, uploaded the new dev version and applied the patch, but whenever I go to the "Mapping" tab, I get this error message:

"The website encountered an unexpected error. Please try again later."

All other tabs are working fine, except for the mapping tab. Any idea?

@memoday
Clear all caches, if not done before. You can do this at admin/config/development/performance.
If there is still an error on the mapping page after clearing caches, turn on error reporting at admin/config/development/logging to see which error occurs on the mapping page. You can also check the "Recent log messages" (admin/reports/dblog) to see which error occurred.

Uninstalling Feeds is indeed a bit hard to do in D8. You first have to manually delete all feeds at admin/content/feed and delete all feed_item fields. You can see on /admin/reports/fields where the field is used. Hopefully there is an API in Drupal core that provides an easier way of removing these automatically upon uninstall.

I had to delete all legacy Feeds code in the modules folder, then remove it from the database using 'drush pmu feeds', then composer require the latest dev, followed with applying the patch in #31, which... works!

Is there documentation hiding somewhere for adding Feeds 8.x support to a new target field type? I'd like to get something set up for Address module.

@caldenjabobs
No, there isn't any official documentation yet for Feeds D8. It would be a good idea to start with that. I have made a list of conceptual differences between the D7 and the D8 version. Maybe I should start with creating a guide for Feeds 8.x-3.x that only contains that (as a start).
If you want to write your own FeedsTarget plugin, you could use some of the existing FeedsTarget plugins in feeds/src/Feeds/Target as example. "link" and "image" would probably be good examples. These FeedsTarget plugins have support for multiple "subfields". For example, for the image field mapping to "ID", "alt" and "title" is all made possible within one FeedsTarget plugin.

@megachriz
That would be really helpful.

Working on the plugin now :)

Hey is that Feeds Slack channel still active?

@caldenjacobs
I've added a list of conceptual differences between the D7 and the D8 version of Feeds to the documentation:
https://www.drupal.org/docs/8/modules/feeds/conceptual-differences-with-...

The Feeds slack channel is still there, but so far only used occasionally.

@MegaChriz,

You are a saint sir!

We also heavily relied on feeds in D7 for importing over 100 events per year as well as all of our content.

I have applied the patch and was able to run a CSV import for users so it appears to be working!!!! I repeat we have lift-off and you can import using a CSV file!!!!!

Here are a couple of issues I am seeing.

1. The user processor doesn't recognize the role as a target. You have the status available but no roles.

2. When adding a source during mapping, every time I add a new source I receive the following error:

Notice: Undefined index: exists in Drupal\Core\Render\Element\MachineName::validateMachineName() (line 251 of core/lib/Drupal/Core/Render/Element/MachineName.php).
Drupal\Core\Render\Element\MachineName::validateMachineName(Array, Object, Array)
call_user_func_array(Array, Array) (Line: 277)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object, 'feeds_mapping_form') (Line: 119)
Drupal\Core\Form\FormValidator->validateForm('feeds_mapping_form', Array, Object) (Line: 571)
Drupal\Core\Form\FormBuilder->processForm('feeds_mapping_form', Array, Object) (Line: 314)
Drupal\Core\Form\FormBuilder->buildForm(Object, Object) (Line: 74)
Drupal\Core\Controller\FormController->getContentResult(Object, Object)
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 574)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
call_user_func_array(Object, Array) (Line: 144)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 64)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 99)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 78)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 50)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 656)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)
Warning: call_user_func() expects parameter 1 to be a valid callback, no array or string given in Drupal\Core\Render\Element\MachineName::validateMachineName() (line 252 of core/lib/Drupal/Core/Render/Element/MachineName.php).
Drupal\Core\Render\Element\MachineName::validateMachineName(Array, Object, Array)
call_user_func_array(Array, Array) (Line: 277)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object) (Line: 235)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object, 'feeds_mapping_form') (Line: 119)
Drupal\Core\Form\FormValidator->validateForm('feeds_mapping_form', Array, Object) (Line: 571)
Drupal\Core\Form\FormBuilder->processForm('feeds_mapping_form', Array, Object) (Line: 314)
Drupal\Core\Form\FormBuilder->buildForm(Object, Object) (Line: 74)
Drupal\Core\Controller\FormController->getContentResult(Object, Object)
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 574)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
call_user_func_array(Object, Array) (Line: 144)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 64)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 99)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 78)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 50)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 656)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

3. When trying to run the import again to update previous users:

The user staff failed to validate with the following errors:
feeds_item.0.url: This value should be of the correct primitive type.
Please check your mappings.

4. After selecting deleting items from the feed it appears to delete all users including, the Admin user 1, even if the csv file does not have the Admin user in it.

@kclarkson
Thanks for testing!

1. I think the role target hasn't been implemented yet.
2. Thanks for reporting. I'll look into it the next time I continue with this issue (this issue still feels like a mountain to climb for me).

Hi MegaChriz,

Sorry for my belated follow-up on this. It was very busy at my end.

I managed to map sources, but for some reason I cannot seem to locate the import page itself. Where is it located? In D7, it's located at www.sitename.com/import/

Where can I find it in D8?

Thanks!

After creating a feed type at /admin/structure/feeds, you need to create and run the actual feeds at /admin/content/feed.

Thanks a million for your support @jwineichen! I managed to import a small tab-delimited file just fine.
This is great progress.

Thanks @MegaChriz for your work!

Thanks @jwineichen! and MegaChriz. CSV importer works! We import the results of Bing News Search API's for many SoCal news sites many times a day and the importer handled an average .csv with 20 varied fields without issue. Just begun testing and will let the list know of any issues. Again, thanks.

T Dwyer - Indian Hill Media

Are we supposed to use this in conjunction with #2443471-4: Implement configurable parsers or is it a separate thing?

@jwineichen
That's a separate thing.

Gotcha. Thanks, @MegaChriz.

I was able to get everything set up and import content with the #31 patch. I'm getting similar "undefined index" errors @kclarkson got in #41. Everything still works though and I can then use the newly added source for a different mapping without getting the error. I was surprised to find that we can map to paragraphs fields (at least text fields) without any additional work. That's really exciting.

Running patch #31.

Using on a content-type instead of users. Not getting the "undefined index" as others have mentioned. I get the following instead:

An AJAX HTTP error occurred.
HTTP Result Code: 200
Debugging information follows.
Path: /batch?id=43&op=do_nojs&op=do
StatusText: OK
ResponseText:
( ! ) Fatal error: Method Drupal\Component\Render\FormattableMarkup::__toString() must not throw an exception in /home/ubuntu/workspace/web/modules/contrib/feeds/src/Feeds/Processor/EntityProcessorBase.php on line 386
Call Stack
#TimeMemoryFunctionLocation
10.0000242800{main}( ).../index.php:0
20.0047728512Drupal\Core\DrupalKernel->handle( ).../index.php:19
30.01111896696Stack\StackedHttpKernel->handle( ).../DrupalKernel.php:656
40.01111896808Drupal\Core\StackMiddleware\NegotiationMiddleware->handle( ).../StackedHttpKernel.php:23
50.01111897304Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle( ).../NegotiationMiddleware.php:50
60.01121897736Drupal\page_cache\StackMiddleware\PageCache->handle( ).../ReverseProxyMiddleware.php:47
70.01121898624Drupal\page_cache\StackMiddleware\PageCache->pass( ).../PageCache.php:78
80.01121898704Drupal\Core\StackMiddleware\KernelPreHandle->handle( ).../PageCache.php:99
90.01512210648Drupal\Core\StackMiddleware\Session->handle( ).../KernelPreHandle.php:47
100.01732380136Symfony\Component\HttpKernel\HttpKernel->handle( ).../Session.php:57
110.01732381088Symfony\Component\HttpKernel\HttpKernel->handleRaw( ).../HttpKernel.php:64
120.04954352672call_user_func_array:{/home/ubuntu/workspace/vendor/symfony/http-kernel/HttpKernel.php:144}
( ).../HttpKernel.php:144
130.04954352872Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}( ).../HttpKernel.php:144
140.04954353128Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext( ).../EarlyRenderingControllerWrapperSubscriber.php:97
150.04954365088Drupal\Core\Render\Renderer->executeInRenderContext( ).../EarlyRenderingControllerWrapperSubscriber.php:124
160.04964365584Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}( ).../Renderer.php:574
170.04964365632call_user_func_array:{/home/ubuntu/workspace/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php:123}
( ).../EarlyRenderingControllerWrapperSubscriber.php:123
180.04964366040Drupal\system\Controller\BatchController->batchPage( ).../EarlyRenderingControllerWrapperSubscriber.php:123
190.05024463056_batch_page( ).../BatchController.php:55
200.06427884280_batch_do( ).../batch.inc:77
210.06427884592_batch_process( ).../batch.inc:95
220.865011407648call_user_func_array:{/home/ubuntu/workspace/web/core/includes/batch.inc:252}
( ).../batch.inc:252
230.865011408368Drupal\feeds\FeedImportHandler->batchProcess( ).../batch.inc:252
240.865011408464Drupal\feeds\FeedImportHandler->doProcess( ).../FeedImportHandler.php:157
250.865711418576Drupal\feeds\FeedHandlerBase->dispatchEvent( ).../FeedImportHandler.php:266
260.865711418576Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch( ).../EventDispatcherTrait.php:37
270.865711418712Drupal\feeds\EventSubscriber\LazySubscriber->Drupal\feeds\EventSubscriber\{closure}( ).../ContainerAwareEventDispatcher.php:111
280.871311902112Drupal\feeds\Feeds\Processor\EntityProcessorBase->process( ).../LazySubscriber.php:92
291.181816087216Drupal\feeds\Feeds\Processor\EntityProcessorBase->entityValidate( ).../EntityProcessorBase.php:135

@tedwyer @Topplestack Try this module Content Import to import nodes from CSV file to any content type

@dhayanandan_k I saw that The biggest downside of that is having to have the first row have field names, which isn't very user friendly for those that will be importing periodic content. It's also a bit of a pain for the systems exporting the csv files. It will require a user to go in and run a find and replace before performing the import. I think I might be able to use it as a stop gap, but I'd still like to see either feeds continue to be worked on or migrate adopt some of the feeds functionality.

@dhayanandan_k Thanks for the tip but don't think this works for us. I need to do this programmatically from scripts sourced on the server (not local computer). Need mapping and feeds drush commands is a big bonus.

Updated the issue summary: defined the goal and requirements for this issue and the steps to be taken to fix this issue.

We are currently working on this issue, feel free to contact me.

Hi all,
I have a couple of issues by Selecting the source in Mapping(None to select the source), Is need to add the source in manually?? kindly refer the image I attached below for more information. . And another issue is by CSV I could upload the file and the error is "CSV (not working yet, do not use)".

Reverted issue title.

I am waiting for this module. If I can help by testing please let me know.

Trialling patch #31 today - it introduces an error in testing:

Fatal error: Call to undefined method Mock_FeedTypeInterface_bc694287::getMappingSources() in .../modules/contrib/feeds/src/Feeds/Parser/CsvParser.php on line 39

Investigating to see if this is genuine regression or just the test needing an update now.. I suspect the latter

I have the same test fail with patch#31, but successfully imported a CSV to nodes without errors. The UI suggests that multiple tag creation can happen on import, but this does not occur, only one tag of the form "tag1,tag2,tag3" is created. Should I create a separate sub issue for that..? @dman suggests using whatever tamper approach we choose overall for this specific purpose.

patch #31 working ok, successfully importing csv and populating fields as mapped.
Update of existing nodes, took a little tweaking, requiring Feed item / GUID target to be set as unique to work as expected.
Could create a separate issue for that perhaps?
Looking good otherwise

If I can help with testing, please let me know.

@dman, @thomasmurphy, @hughmccracken
Thanks for testing the patch in #31 and confirming that the sample source file can be imported without errors. However, the patch as is cannot be committed yet because it breaks tests, it contains undocumented API additions and the machine name callback is not implemented yet.

So the next tasks for fixing this issue are:

1. Investigate and fix the test failure.
2. Implement the machine name callback.
3. Clean up/document the API additions.

Reflected this in the issue summary as well.

@dman
Do you want to fix the test failure?

@hughmccracken

...requiring Feed item / GUID target to be set as unique to work as expected.

It sounds like you are running into the following bug: #2857692: Update existing nodes fails when not mapping to "Feeds Item: Item URL": This value should be of the correct primitive type.
If you want to help with fixing that bug, that would be great!

@Lea Sky
Thanks for your offer to help on this issue! You could help with trying to reproduce the errors reported in #41 and #51 and then provide the steps here of how they can be reproduced.

Tried the patch in #31 with content import, everything was imported as expected and generally works. I haven't encountered any of the issues listed above.
I just had a slight hiccup with the importer telling me my entity reference title was empty when it was not but the nodes where created and referenced correctly anyway. EDIT: My mistake, nodes where not imported because of a whitespace at the end of the referenced entity title, so everything works as expected!

Hello,

the patch in #31 seems to enable/activate the CSV parser.
Dou tou think create a next version with this useful parser ?

Thanks.

I have updated the patch from #31 (https://www.drupal.org/project/feeds/issues/2669974#comment-12107320).
Looks like tests failed because of incorrect line endings in the files:

tests/resources/csvs/newlines.csv
tests/resources/csvs/quotes_and_newlines.csv

I tried to convert the line endings in my editor, but I had no luck with that since the line endings are already LF with me.

As already reported in #62, I see that the testbot reports the following error:

PHP Fatal error: Call to undefined method Mock_FeedTypeInterface_544eeffa::getMappingSources() in /var/www/html/modules/contrib/feeds/src/Feeds/Parser/CsvParser.php on line 39

https://dispatcher.drupalci.org/job/drupal8_contrib_patches/19410/console

You can come there in the following way:

1. On the issue, underneath the patch file, click on one of the (failed) test results. For example, you'll come here: https://www.drupal.org/pift-ci-job/851600
2. Click on 'View results on dispatcher', which will lead for example to https://dispatcher.drupalci.org/job/drupal8_contrib_patches/19410
3. In the menu on the left, click on 'Console Output' for details about the test run. This will be for example https://dispatcher.drupalci.org/job/drupal8_contrib_patches/19410/console

Thank you for your help, MegaChriz!

Looks like \Drupal\feeds\FeedTypeInterface missing getMappingSources() method.

@max-kuzomko
Great! All test failures have been fixed. If we can tackle the remaining tasks, the patch can be committed!

Great news! Does that mean that patch will be pushed to this version https://www.drupal.org/project/feeds/releases/8.x-3.x-dev? Many thanks!

@irinaz
Ideally, the last three remaining tasks (machine name callback, code cleanup/documentation, automated tests) also get done before commit, but I understand the importance of having an UI for the CSV parser. I'll give it two days to see if these tasks get tackled. Else the patch will be committed as is.

I'll work on implementing the machine name callback now.

This should do the trick for the machine name callback.

This adds methods for managing custom sources. To be sure I implemented these methods correctly I added unit tests for these methods. And order to create these unit tests I had to move calls to the global Drupal class (like \Drupal::service() and \Drupal::moduleHandler) to separate methods. Hopefully this didn't break other things.

A random test failure occurred. Back to "needs review". I will commit it tomorrow if I (or someone else) don't discover any disruptive things.

I checked all the code changes. About the changes in CsvParser.php I'm not 100% certain yet, these could be verified if we had UI tests. I postpone that to a follow up. I did make two small changes in the code, so that needs one last patch to verify nothing gets broken. One change is the title of the CSV parser (I removed "do not use" text from it ;)) and the other is a small cleanup in the FeedType unit test.

Alright, committed #84!

I'm going to create the follow-ups now.

The followups have been created:

#2938505: Add UI for editing and removing custom (CSV) sources
#2938500: Add UI tests for the CSV parser

@MegaChriz

thanks for your work on this module.
I will test shortly the new version.