Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
D7 had drush support for exports, are there plans to have the same for D8? Especially for batched exports.
Issue fork views_data_export-2887450
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
mscalone CreditAttribution: mscalone as a volunteer commented+1. great feature, extremely useful
Comment #3
alexdoma CreditAttribution: alexdoma at Drupal Teams commentedHi, you can use VDE drush add-on for drush support
Comment #4
mscalone CreditAttribution: mscalone as a volunteer commentedgreat, thank you for the contribution!
The problem is that I need to use batch exporting because the dataset is so big that I have memory problems in standar mode.
Do you plan to add support to batch processing ?
Comment #5
alexdoma CreditAttribution: alexdoma at Drupal Teams commented@mscalone Yes, we planned to add batch support soon.
Comment #6
a.ilyin CreditAttribution: a.ilyin as a volunteer commented@mscalone Hi!, we've updated our module VDE drush add-on. In current version it supports batched export for views data export. Please read README file or find information on module page on how to use it. Also feel free to create issues and bug reports for it.
Hope this module will help you.
Comment #7
mscalone CreditAttribution: mscalone as a volunteer commentedthanks, @a.ilyin I tested last version with batch support but I'm still experimenting memory problems. I read the code and realise its like an emulation of batch processing, and I thinks that's not enough in this case. I think a version that uses queue/batch api its needed in this case.
thank you very much.
Comment #8
a.ilyin CreditAttribution: a.ilyin as a volunteer commentedThanks for your response @mscalone.
Could you provide more information, please?
How many items do you have for export, how many fields and the number of rows set for Batch size.
Also can you create an issue for our module?
Thanks!
Comment #9
leymannxIt seems a little bit inefficient to create a new module for Drush 9 support. Instead you should provide a patch and help to maintain this module. What if tomorrow Views data export incorporates their own Drush 9 commands? VDE drush add-on would be useless then at least, or conflicting at worst.
Please provide a patch in favor of an already existing well-known and well-maintained module.
Comment #10
leymannxComment #11
leymannxComment #12
a.ilyin CreditAttribution: a.ilyin as a volunteer commentedHi @mscalone!
We've added Queue API support to our module. Please try to use it
Comment #13
a.ilyin CreditAttribution: a.ilyin as a volunteer commentedWe've also created patch for views data export module to use our module as submodule of views data export.
Thanks to @leymannx for suggesting to do so here
Comment #14
a.ilyin CreditAttribution: a.ilyin as a volunteer commentedComment #15
mscalone CreditAttribution: mscalone as a volunteer commentedThank you @a.ilyin
I'm having the following error with a view that exports a field that is a relationship to a taxonomy term:
[error] The "field_xxxxxxx" entity type does not exist. [0.66 sec, 36.14 MB]
If I remove the taxonomy field and the relationship the patch works correctly:
You can test it adding a simple relationship to a taxonomy and then adding some field of the taxonomy (ex. language) and do the export.
I din't test another kind of relationship but may be have the same issue.
thanks again!
Comment #16
frantisekivanko CreditAttribution: frantisekivanko commentedThank you @a.ilyin for the patch.
I'm having the following error with a view batch export method.
Problem is in function entityCacheDisable
The standard export method is working correctly.
I do have a relationship to author in data export view.
Can you pls tell me if it is possible that batch export will work also with relationships?
Thanks !
Comment #17
frantisekivanko CreditAttribution: frantisekivanko commentedFixed entityCacheDisable function.
Added try / catch to handel PluginNotFoundException.
Comment #18
p4trizio CreditAttribution: p4trizio at Elicos commentedHello, thank you for this submodule. In my case, I have almost 20k product to export in batch.
I applied patch in #2789531 to get batch feature and it works great if I use it in frontend.
Unfortunately, when I use drush, rows are repeated for every chunk as if it starts again and again with an offset 0 and a limit 100 (that's my view batch size).
Anybody has the same issue? thanks
Comment #19
p4trizio CreditAttribution: p4trizio at Elicos commentedHi, small fix to address problem described in #18
Comment #20
leisurman CreditAttribution: leisurman commentedViews data export says to use
drush views-data-export [view-name] [display-id] [output-file]
. When I do I get the errorCommand "views-data-export" is not defined.
. I am using drush version 9. This post is about a new module to provide drush 9 support. This post was closed. Can the Views data export module provide drush 9 support within its module?Comment #21
p4trizio CreditAttribution: p4trizio at Elicos commented@leisurman the command is vde_drush:views-data-export
drush vde_drush:views-data-export [view-name] [display-id] [output-file]
or use the alias
drush vde [view-name] [display-id] [output-file]
Comment #22
leisurman CreditAttribution: leisurman commented@p4trizio. Thank you for your help. My View name is atest1 and my display is rest display with the machine name 'rest_export_1'
And I tried.
drush vde_drush:views-data-export atest1 rest-export-1 myfile.json
and
drush vde atest1 rest-export-1 myfile.json
I get error: The view atest1 does not have the rest-export-1 display.
I also tried using undercore in my display name. and then ran.
drush vde_drush:views-data-export atest1 rest_export_1 myfile.json
and
drush vde atest1 rest_export_1 myfile.json
I get error: Incorrect display_id provided, expected a views data export display, found rest_export instead.
My Views format is set to data export, api jsn, and its using fields. I am using Drupal v8.5.8 and Lightning 8.x-3.104.
My task was to create a table list that displays over 3000 nodes and loads all at once. The problem was the view query was slow at 50 seconds. I then used a View to output the 3000 nodes in a static json file. Now the page loads this static json file in 4 seconds. We want to automate and run the view to output the json file every hour. Then the page will load this static json file.
Comment #23
leisurman CreditAttribution: leisurman commented@p4trizio, My fault. I was using a rest display. I created a data export display and then it worked using this
drush vde atest1 data_export_1 myfile.json
. Thank you for your help!Comment #24
B-Prod CreditAttribution: B-Prod commentedHi have the following error when exporting a CSV from a view:
[error] Error: Call to a member function __toString() on string in Drupal\vde_drush\Commands\VdeDrushCommands->viewsDataExport() (line 201 of /app/web/modules/contrib/views_data_export/modules/vde_drush/src/Commands/VdeDrushCommands.php)>/code>
This may occur when there is no result.
Comment #25
longwave#19 works well for me in conjunction with the batch operations patch, but not even sure this needs to be a separate submodule, this could just be merged into Views Data Export itself? It will then automatically work if Drush 9 is installed and do nothing if Drush 9 is not installed.
Comment #26
B-Prod CreditAttribution: B-Prod commentedHere is a patch that fixes:
Comment #27
Manuel Garcia CreditAttribution: Manuel Garcia as a volunteer and at Appnovation for Pfizer, Inc. commentedPatch on #26 included the
.DS_Store
binary file by mistake, which is why it fails to apply.I manually removed this hunk from the patch file and it applied fine.
So here is the regenerated patch. I thought it would be useful to have the interdiff between it and the previous patch #19 so I generated it as well.
Comment #28
Manuel Garcia CreditAttribution: Manuel Garcia as a volunteer and at Appnovation for Pfizer, Inc. commentedre #25 I totally agree, there is no need for a separate module imho.
Comment #29
Manuel Garcia CreditAttribution: Manuel Garcia as a volunteer and at Appnovation for Pfizer, Inc. commentedThought I'd cook up the patch making the command be part of views_data_export itself and not an extra module, so here it is.
A brief summary of what I did:
views-data-export
Comment #30
Manuel Garcia CreditAttribution: Manuel Garcia as a volunteer and at Appnovation for Pfizer, Inc. commentedMissed one spot :)
Comment #31
mellowtothemax CreditAttribution: mellowtothemax commentedThis does not run with Drush 10. Any suggestions?
Regards,
Comment #32
steinmb CreditAttribution: steinmb as a volunteer commentedSince Drupal 9 req. Drush 10 it think we should change the issue topic and make sure it works with Drush 10.
Comment #33
josephdpurcell CreditAttribution: josephdpurcell at Bounteous commentedThe documentation states that this command is available. So, I ran it and discovered the drush command is not available. This is a great feature! If I get a chance, I'll try to report back if I was able to use the patch. Thanks for working on this!
Comment #34
patrickfwestonHi all,
I applied the patch in #30, but I was getting errors due to a missing
drush.services.yml
file and the dependency injection it provides:ArgumentCountError: Too few arguments to function Drupal\views_data_export\Commands\ViewsDataExportCommands::__construct(), 0 passed in /app/docroot/core/lib/Drupal/Component/DependencyInjection/Container.php on line 265 and exactly 4 expected in Drupal\views_data_export\Commands\ViewsDataExportCommands->__construct() (line 68 of /app/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php)
I've added to the patch in #30 to include this file. I was able to get this running on a Drupal 8 site using Drush 9 as a result.
To use the command provided by the patches in this queue, ensure you've disabled and removed the
vde_drush
contrib module and have cleared cache. Then, run a Drush command like so:drush views-data-export [view_id] [display_id] [path to export file]
Comment #35
lee.cocklin CreditAttribution: lee.cocklin commentedApplied patch from #34 to a Drupal 8.8.10 site with Drush 10 and it runs just fine. No issues to report.
Comment #36
mellowtothemax CreditAttribution: mellowtothemax commentedI also tried #34 and it works with Drush 10.
However the output is slightly different to the output of the view preview and the manual download. Not really sure why but for some reason when exporting to xml as in my case, the closing
</response>
tag is absent and the link to product field (in this case for commerce products) returnshttp://default/
instead of the site url.Comment #37
longwaveRe #36 you have to pass the
-l
or--uri
option to drush to tell it the base URL, it has no way of knowing otherwise.Comment #38
mellowtothemax CreditAttribution: mellowtothemax commentedThanks longwave that that fixed the url issue.
Comment #39
chiefme CreditAttribution: chiefme commentedDrupal: 8.9.6
PHP 7.3.19
Tried both: 8.x-1.0 and 8.x-1.x-dev
Could not apply patch! Skipping. The error was: Cannot apply patch... (#34)
Updated:
Everything is ok in local machine and other places where I tested, can't find out why this only occurs on client's server.
Could this be related with PHP?
Solved:
Stupid problem.. problem was because client VPS wasn't installed or disabled "patch"
Comment #40
pivica CreditAttribution: pivica commentedGot the same error as in 36:
> Not really sure why but for some reason when exporting to xml as in my case, the closing tag is absent
Here is a patch that should fix this and also fix while loop for a queue - that if check was wrong and was missing the last queue item. Tested this against custom XML format implementation and seems it is working great now.
Comment #42
error84 CreditAttribution: error84 commentedJust to confirm patch #34 works for:
Drupal: 9.1.4
Drush: 10.4.0
PHP: 7.4.12
Tested with batched export to CSV.
Had to do 'drush cr' after patch was applied via composer.
Comment #43
PhilippVerpoortJust to say that I can also confirm that patch #34 works for me with:
Drupal: 8.9.11
Drush: 10.4.3
PHP: 7.2.24
Also had to clear caches but worked like a charm after that. Only used CSV exports though.
Hoping that this will get finalised and committed soon! :)
Comment #44
steinmb CreditAttribution: steinmb as a volunteer commented#40 raises a concern but the patch fails. I would speed things up if this could be rerolled and tested by end users.
Comment #45
Megha_kundar CreditAttribution: Megha_kundar commentedComment #47
tim-dielsPatch #40 not working as some files are not inside the patch
Comment #48
steinmb CreditAttribution: steinmb as a volunteer commented@tim-diels Try the reroll in #45
Comment #49
tim-dielsSorry I mean that patch #45 is not working. Going to hide this as the patch is completely wrong. Using patch #34 at this moment and works.
Comment #50
BS Pavan CreditAttribution: BS Pavan at Srijan | A Material+ Company commentedHi @tim-diels
I have rerolled the patch as per #40 and tried to fix the issue. I think the issue is because of count which is mentioned in
https://www.drupal.org/project/views_data_export/issues/3172799#comment-...
Please check once.
Comment #52
BS Pavan CreditAttribution: BS Pavan at Srijan | A Material+ Company commentedComment #53
simgui8 CreditAttribution: simgui8 as a volunteer and commented#52 applies correctly on 8.x-1.0 and fixes "views-data-export" is not defined.
Thanks
Comment #54
ivnish CreditAttribution: ivnish commented#52 works
Comment #55
doidd CreditAttribution: doidd commentedI add one row to file src/Plugin/QueueWorker/FileWriter.php to support cron queue.
Comment #57
hfernandes CreditAttribution: hfernandes at ImageX commentedHello,
This patch was not working in my case - I have a views configured to export in batches, but I'm still facing PHP memory limit issues.
I decided to come up with a new proposal. The idea is to use what we already have implemented on
views_data_export
as much as we can and to usedrush_backend_batch_process()
for the batched views instead of the Queue API originally proposed by VDE drush add-on.This will be the result when you are exporting in batches:
and this will be the result when you have the Standard export method:
Besides the drush command, the default export method should continue working as usual.
Thoughts?
Comment #58
_randy CreditAttribution: _randy commentedI agree with @hfernandes. The original approach uses gigs of RAM for exports of any real size.
Patch #57 works, however, it only dumps to the private files location.
I'm attaching a patch that uses @hfernandes' base approach and adds to it a command line parameter to copy the final file too. The original file is maintained in the private files location. This should probably be a command line option to remove the private files reference if a discrete file location is provided as is done in this patch.
EDIT:
command would look like:
Comment #59
sonfdTested patch in #58 and it does generate the file, but there are a couple issues worth noting:
Comment #60
eelkeblokI would like to second the above points, with one small nuance: I would suggest to introduce a "force" option for overwriting the file. I don't think it is a great idea to overwrite by default.
The second point may be the result of the exact interfacing to the module's core code, as it seems to lift the file name from the View's data.
Additionally, the patch suffers from some coding standard violations.
Comment #61
simgui8 CreditAttribution: simgui8 as a volunteer and commentedFor #57 and #58, being in a cli, I would expect the output file parameter to accept any accessible location. The default generated public/private file would require an extra step to delete and is not exactly "drush style".
Here is my result after testing #57 and #58
-vde exports were resulting in 0 byte files
-$result['drush_process_finished'] was returning TRUE and [0]['vde_file'] was created (not at the specified output param).
For those who need #55, this is a reroll (#55 failed to apply to latest dev)
Comment #62
Medha KumariRerolled patch #30 in 8.x-1.x .
Comment #63
simgui8 CreditAttribution: simgui8 as a volunteer and commented@Medha Kumari: #62 is already in #61
#61 is the latest patch "legacy style" (a follow-up of every patch since #27 except #57 and #58)
Comment #64
code_brown CreditAttribution: code_brown at Catalyst IT commentedI have found two other patches useful for views_data_export, however they do not apply when using patch #58:
I have attached a patch including the two above which applies to the 8.x-1.1 version of the module, but I get the feeling this isn't the right way of doing things.
What do people recommend I do when I want to include two or more patches for a module but the patches do not cleanly apply together?
Comment #65
code_brown CreditAttribution: code_brown at Catalyst IT commentedJust re-rolled the above #64 against 8.x-1.2
Comment #67
simgui8 CreditAttribution: simgui8 as a volunteer and commented@code_brown, other issues should stay in their respective thread most of the time.
Mixing other issues makes it harder for maintainers to understand what is going on and will slow down patch adoption.
In doubt, ask for best practices on drupal slack.
As for #64 and #65, I don't think they are duplicates or related to this issue.
And I don't think we should hide other contributors' files.
Only hide yours (maybe you should hide #65 or explain in detail how it's related to this issue).
I hope it helps!
Comment #68
simgui8 CreditAttribution: simgui8 as a volunteer and commentedUnhiding files that were previously hidden by mistake.
Comment #69
_randy CreditAttribution: _randy commentedUnfortunately this issue has diverged into two streams. One using what's been referred to as "Legacy" and another which uses the Drush back-end processing capabilities. #57 and #58 address the Drush back-end.
I've re-rolled #58 to work with v1.2.
Comment #70
GuillaumeDuveauPatch in #69 still fills the memory for me:
It seems that the batch API is used indeed, but with just 1 operation instead of many. Meaning that the process would run in batch only for the View but as a single PHP process only when called by Drush. Which would kind of defeat the initial goal of having a batch.
Comment #71
superlolo95 CreditAttribution: superlolo95 commented#69 works with
Drupal 9.5.0
PHP 8.1.13
Drush 10.6.2
Views data export 8.x-1.2
and run drush cr ==> otherwise the patch is not taken into account by drush
Comment #72
hfernandes CreditAttribution: hfernandes at ImageX commented@GuillaumeDuveau do you have the method "Batch" set in the "Export settings" of your views?
If it's set as "Standard", it'll try to load all data. Maybe that's why you are getting this memory issue.
Comment #73
GuillaumeDuveau@hfernandes Yes I have batch with a small limit (100).
On my dataset, calling the view through the web without this patch uses around 512M whereas calling it through Drush with this patch uses around 1.5G.
Comment #74
hfernandes CreditAttribution: hfernandes at ImageX commentedHello, I've fixed some code standards issues and improved the messaging/logs.
@GuillaumeDuveau could you test this latest one and let me know the result?
It shouldn't behave differently when running via drush or through the web, since it uses the same processBatch function.
Comment #75
hfernandes CreditAttribution: hfernandes at ImageX commentedComment #76
reszliThe above solution hijacks the views arguments for some internal flags, so it won't work for views that have contextual filters.
I re-wired the patch to move those flags into a different variable, and extended the drush command with a new parameter for specifying arguments.
So now it can be executed as follows:
drush vde [view_name] [display_name] [arg1/arg2/etc] [path/to/file.xml]
Comment #78
reszlifixing errors and coding standards
Comment #79
reszliComment #80
jaapjan CreditAttribution: jaapjan at 25Knots for iO commentedI've applied the idea suggested in #60 in a new patch based on patch #78. Adding a force option, so the export will always be downloadable on the same file path.
Drush command example:
drush vde view_name display_name '' sites/default/files/my-file-name.csv --force
If you don't provide the --force option it will just throw the exception if the file already exists:
In ViewsDataExportCommands.php line 91:
The desired output file already exists. Please remove the file and try again.
Comment #81
jaapjan CreditAttribution: jaapjan at 25Knots for iO commentedLast patch was patched against 8.x-1.x. Attached file is patched against 8.x-1.2.
Comment #82
sakonn CreditAttribution: sakonn as a volunteer commentedHello, I have tried this patch on my website and I am encountering mysql error. I am using module to export products from my drupal eshop and during the testing I have found out that using drush for small exports works fine. but with the larger exports I am always getting the error:
General error: 2006 MySQL server has gone away
I think it might be related to my hosting environment where is
wait_timeout
variable set only to 300 seconds. The thing is that this error is not showing up when running export through browser. In the browser I am redirected to batch page and export finish without problem. Throught browser I am able to export any number of products. Using the cli I get error with about 3000 products (I need to export about 50000 products). In addition to that other large batch processes, for example feeds for import of these products is working fine.Is there a way to resolve this error with cli command?
edit:
I have migrated to different server and mysql error is gone. But there is still issue with larger exports and drush. As it was already mentioned drush export use enourmos amount of RAM in comparison to browser export. Through browser I am able to make an export of any number of products.
Comment #83
sakonn CreditAttribution: sakonn as a volunteer commentedThe solution I have found was to set this in
settings.php
file.For some reason it seems that drush is ignoring settings of php memory limit (some discussion here: https://github.com/drush-ops/drush/issues/3294) and is trying to run whole export in one process. After settings this limit the process consumes less then 200Mb of RAM and is working fine even with large exports. During export these messages are logged:
Comment #84
sergiurThe last working patch for me is the one in comment #75. None of the patches after that seem to save the export at the custom path defined.
I am using version 1.3 of the module which seems to be up to date with the dev version. The command I'm running is
drush vde test_export data_export_1 "" private://test/test_export.csv
. I have tried the public file system too, but still not saving at the path I set. In the end I am deleting the old file as part of the cron job, before running the drush command using the patch in #75 which works.Comment #85
PierreEmmanuel CreditAttribution: PierreEmmanuel commentedPatch re-roll for D10 (v1.3)
! As per code review "ViewsDataExportCommands.php" is missing from this patch.
Comment #86
Giuseppe87 CreditAttribution: Giuseppe87 commented@PierreEmmanuel
The patch at #85 is missing the
ViewsDataExportCommands.php
class, is a mistake?Meanwhile, I've updated the patch from #81, as on D10 emptying the caches gives the error:
Comment #87
SachinT1996 CreditAttribution: SachinT1996 as a volunteer and at Innoraft commentedPatch at #86 does not apply correctly for 8.x-1.3.
Re-rolled for compatibility.
Comment #88
Giuseppe87 CreditAttribution: Giuseppe87 commentedYes, I forgot to mention that #86 is against the dev version
Comment #89
Giuseppe87 CreditAttribution: Giuseppe87 commentedThe #86 as well the #87 missed deprecated a
file_save_data
- or better, the Rector missed it.Attached a fixed version of #87
Comment #90
GuillaumePacilly CreditAttribution: GuillaumePacilly commentedHello,
First thanks for this patch, this feature is really much appreciated.
I could be wrong, but I think the parameters order is inversed between outputfile and arguments in the viewsDataExport function. At least I had to inverse them for it to work properly. Attached an updated patch with the right order.
Comment #91
_randy CreditAttribution: _randy commented@Giuseppe87 for #89:
Drupal 10.1.6, VDE 8.x-1.4
Patch applies however, in my testing it would appear that the copy of the file never happens to the final specified location.
It seems the $options array has a key in [1], however it is blank.
$args, however, does seem to hold the path in separate array indexes.
I've attached an updated full patch here.
~line 920 of src/Plugin/view/display/DataExport.php, I've altered my version to look like
Comment #92
IT-CruComment #93
IT-CruComment #95
IT-CruI've opened a MR based on patch from #90 and adjust this to recent supported Drush version 11/12 to make testing and reviewing easier for all of us.
Patch from #91 has strange format so it was not possible to apply for me.
I think we should also discuss with maintainers of https://www.drupal.org/project/vde_drush module to merge Drush commands back into views_data_export code base.