How to import/create 500.000 nodes in the best way?

By Transmitter on 15 Apr 2016 at 14:35 UTC

Hello,

I've got an csv and json file with 500.000 data sets which I'd like to import into D8.
Now the CSV via Feeds doesn't work yet - so I transformed them into RSS - but that takes only roughly 20.000 items at a time (and that takes roughly 1 hour to import).
So I thought I could create them programmatically from within Drupal or via REST.
Assuming via Drupal is fast, I wanted to try this.
Easy enough:
http://stackoverflow.com/questions/24172791/how-to-programmatically-crea...
But what I couldn't find out:
Do I need to create a module and put the code into a module or can I somehow just copy index.php to createnode.php and get the code there working?
It's a local test and playing environment - so I don't worry about beauty/security. ;)

Thanks,
Transmitter

Comments

Why don't you try Migration

joshi.rohit100

he/him

Hindi

Delhi, India

commented 15 April 2016 at 18:39

Why don't you try Migration way ?

I agree, the Migration module

Jaypan commented 15 April 2016 at 23:09

I agree, the Migration module is a good way to go.

I don't know .. migrate seems

Transmitter commented 16 April 2016 at 02:29

I don't know .. migrate seems to be very mighty, little documented and quite cluttered.

So far I installed: Migrate Plus, Migrate Tools, Migrate Source CSV and tried my luck with this one:
https://github.com/heddn/d8_custom_migrate

On migrate_tools it's written:

Note that the UI is not currently functional with 8.1.x.

So down to drush trial and error:

$ drush migrate-status
$ drush migrate-messages
Missing argument 1 for drush_migrate_tools_migrate_messages() migrate_tools.drush.inc:357                                                                                                                               [warning]
Migration  does not exist

Not sure where to go from here - is there any working example/tutorial for 8.1? I know .. quite early but hope dies last ;)

Well when i put this in

joshi.rohit100

he/him

Hindi

Delhi, India

commented 18 April 2016 at 11:11

Well when i put this in google serach "drupal 8 migrate csv", I got lot of results and out of them, two I am pasting -

https://www.drupal.org/node/2574707 and https://ohthehugemanatee.org/blog/2015/05/02/how-to-build-a-new-source-f...

Yes, I find a lot of results

Transmitter commented 22 April 2016 at 06:28

Yes, I find a lot of results as well in Google. Just the quality is really, really low.
There isn't any working example.

Here:
https://www.drupal.org/node/2574707
JonasMuc commented short keypoints on the 8.1 version - none really helpful to get a global picture, but a few details.
His link (yielding a 404 but correcting it to https://www.drupal.org/node/2625696 ) contains mostly dev info - interesting but not helpful for site builders.

https://ohthehugemanatee.org/blog/2015/05/02/how-to-build-a-new-source-f...
Is almost a year old. 8.1 has been released a few days ago - so I'm wondering how useful this info still is.
Yet, I went through it with 0 result. Ok, I can copy methods into a class - a complete file would have been nice.
Nothing is being said about folder structure, where to place a yml file, how to name it.
That article is geared towards module developers I guess - not towards someone who wants to build a site and import data.
At the very end, I read that this stuff is now a module - yay! So In installed this as well - but then again .. drush ms shows nothing. Ok .. maybe that stuff isn't a module after all and it's more like a template and I still have to build my own submodule? Who knows?

At last I find https://github.com/wunderkraut/migrate_source_example the first more or less useful thing with actual instructions since this one: https://github.com/heddn/d8_custom_migrate comes completely without any info on how to use it, where to install it, how to customise it. Again - geared towards modules developers I'd say - completely wrong audience. Anyway, installed it, enabled it, drush ms shows nothing again. (yes, I created the people content type, yes I moved people.csv to sites/default/files/csv - not sure whether that is correct since nothing is mentioned anywhere but I assume that would be fine?)
Back again to wunderkraut: So installed it, looks fine in Drupal so far. Sticking to the documentation on github I tried drush ms - again, nothing.
Maybe drush ms isn't correct and I'm supposed now to use sth. else? Who knows - nobody would write anything.

And so it goes with many other pages that I went through with this topic.
Of course I understand the Drupal developers are busy coding and getting everything right asap.
But that is just one part of a software product - if nobody can use it, the whole code (or the product) isn't really useful. I like Drupal a lot and maybe I just need to wait another couple of weeks or months until this area gets useful for users/sitebuilders as well. But certainly, if sth. is 'ready' (at least for testing) it would be nice to know how to use it.