D7 had drush support for exports, are there plans to have the same for D8? Especially for batched exports.

CommentFileSizeAuthor
#91 vde-drush-with-output-location-2887450-91.patch26.21 KB_randy
#90 2887450-90.views_data_export.Support-views-data-export-using-drush_0.patch25.75 KBGuillaumePacilly
#89 interdiff.2887450.87-89.txt680 bytesGiuseppe87
#89 2887450-89.views_data_export.Support-views-data-export-using-drush.patch25.75 KBGiuseppe87
#87 vde-drush-with-output-location-2887450-v1.3-87.patch25.7 KBSachinT1996
#86 2887450-86.patch52.62 KBGiuseppe87
#85 vde-drush-with-output-location-2887450-85.patch20.27 KBPierreEmmanuel
#81 vde-drush-with-output-location-2887450-v1.2-81.patch26.08 KBjaapjan
#80 vde-drush-with-output-location-2887450-v1.2-80.patch32.57 KBjaapjan
#78 vde-drush-with-output-location-2887450-v1.2-78.patch25.89 KBreszli
#76 vde-drush-with-output-location-2887450-v1.2-76.patch25.81 KBreszli
#75 vde-drush-with-output-location-2887450-v1.2-74.patch23.45 KBhfernandes
#74 vde-drush-with-output-location-2887450-v1.2-74.patch23.43 KBhfernandes
#69 vde-drush-with-output-location-2887450-v1.2.patch20.99 KB_randy
#65 2887450-drush-custom-export-directory-permanent-file-8.x-1.2.patch25.4 KBcode_brown
#64 2887450-drush-custom-export-directory-permanent-file-8.x-1.1.patch25.34 KBcode_brown
#62 2887450-62.patch1.67 KBMedha Kumari
#61 interdiff_55-61.txt1.04 KBsimgui8
#61 2887450-61.patch29.98 KBsimgui8
#58 vde-drush-with-output-location-2887450.patch22.46 KB_randy
#57 newdrushcommand-2887450-56.patch21.41 KBhfernandes
#55 interdiff_51_55.txt392 bytesdoidd
#55 2887450-55.patch31.26 KBdoidd
#52 interdiff_40-51.txt1.16 KBBS Pavan
#52 2887450-51.patch29.33 KBBS Pavan
#50 interdiff_40-50.txt697 bytesBS Pavan
#50 2887450-50.patch28.85 KBBS Pavan
#45 2887450-45.patch1.65 KBMegha_kundar
#40 2887450-40.patch28.06 KBpivica
#40 interdiff-2887450-34-40.txt1.29 KBpivica
#34 2887450-34.patch28.07 KBpatrickfweston
#30 2887450-30.patch27.65 KBManuel Garcia
#30 interdiff-2887450-29-30.txt1.05 KBManuel Garcia
#29 2887450-29.patch27.61 KBManuel Garcia
#29 interdiff-2887450-27-29.txt20.9 KBManuel Garcia
#27 interdiff-2887450-19-27.txt1.2 KBManuel Garcia
#27 2887450-27.patch31.32 KBManuel Garcia
#26 drush-9-support-to-export-views-data-2887450-22.patch31.51 KBB-Prod
#19 interdiff-2887450-17-19.txt573 bytesp4trizio
#19 drush-9-support-to-export-views-data-2887450-19.patch31.02 KBp4trizio
#17 2887450-drush-9-support-to-export-views-data-02.patch30.94 KBfrantisekivanko
#13 2887450-drush-9-support-to-export-views-data.patch30.82 KBa.ilyin
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

killes@www.drop.org created an issue. See original summary.

mscalone’s picture

+1. great feature, extremely useful

alexdoma’s picture

Hi, you can use VDE drush add-on for drush support

mscalone’s picture

great, thank you for the contribution!

The problem is that I need to use batch exporting because the dataset is so big that I have memory problems in standar mode.

Do you plan to add support to batch processing ?

alexdoma’s picture

@mscalone Yes, we planned to add batch support soon.

a.ilyin’s picture

@mscalone Hi!, we've updated our module VDE drush add-on. In current version it supports batched export for views data export. Please read README file or find information on module page on how to use it. Also feel free to create issues and bug reports for it.
Hope this module will help you.

mscalone’s picture

thanks, @a.ilyin I tested last version with batch support but I'm still experimenting memory problems. I read the code and realise its like an emulation of batch processing, and I thinks that's not enough in this case. I think a version that uses queue/batch api its needed in this case.

thank you very much.

a.ilyin’s picture

Thanks for your response @mscalone.
Could you provide more information, please?
How many items do you have for export, how many fields and the number of rows set for Batch size.

Also can you create an issue for our module?

Thanks!

leymannx’s picture

It seems a little bit inefficient to create a new module for Drush 9 support. Instead you should provide a patch and help to maintain this module. What if tomorrow Views data export incorporates their own Drush 9 commands? VDE drush add-on would be useless then at least, or conflicting at worst.

Please provide a patch in favor of an already existing well-known and well-maintained module.

leymannx’s picture

leymannx’s picture

Title: Drush support? » Drush 9 support to export views data
a.ilyin’s picture

Hi @mscalone!
We've added Queue API support to our module. Please try to use it

a.ilyin’s picture

We've also created patch for views data export module to use our module as submodule of views data export.
Thanks to @leymannx for suggesting to do so here

a.ilyin’s picture

Status: Active » Needs review
mscalone’s picture

Thank you @a.ilyin

I'm having the following error with a view that exports a field that is a relationship to a taxonomy term:

[error] The "field_xxxxxxx" entity type does not exist. [0.66 sec, 36.14 MB]

If I remove the taxonomy field and the relationship the patch works correctly:

 ...
 [info] Adding data to queue... [1.22 sec, 51.65 MB]
 [info] Queue filled. Starting executing items [1.22 sec, 51.68 MB]
 [info] Exporting records 0 to 1000. [11.51 sec, 119.92 MB]
 [info] Exporting records 1000 to 2000. [23.21 sec, 123.97 MB]
 [info] Exporting records 2000 to 3000. [34.9 sec, 125.48 MB]
 [info] Exporting records 3000 to 4000. [47.82 sec, 125.99 MB]
 [info] Exporting records 4000 to 5000. [60.72 sec, 127.34 MB]
 [info] Exporting records 5000 to 6000. [73.97 sec, 127.8 MB]
 [info] Exporting records 6000 to 7000. [87.81 sec, 128.34 MB]
 [info] Exporting records 7000 to 8000. [101.47 sec, 129.88 MB]
 [info] Exporting records 8000 to 9000. [114.23 sec, 130.53 MB]
 [info] Exporting records 9000 to 10000. [127.75 sec, 131.67 MB]
 [info] Exporting records 10000 to 11000. [140.73 sec, 131.25 MB]
 [info] Exporting records 11000 to 12000. [153.68 sec, 132.07 MB]
 [info] Exporting records 12000 to 13000. [156.32 sec, 91.21 MB]
 [info] Executed [156.32 sec, 91.21 MB]
 [success] Data export saved to /home/XXX/XXXXXXX_export.csv [156.32 sec, 88.72 MB]

You can test it adding a simple relationship to a taxonomy and then adding some field of the taxonomy (ex. language) and do the export.
I din't test another kind of relationship but may be have the same issue.

thanks again!

frantisekivanko’s picture

Thank you @a.ilyin for the patch.

I'm having the following error with a view batch export method.

In EntityTypeManager.php line 133:

  The "uid__node_field_data" entity type does not exist.

Problem is in function entityCacheDisable

private function entityCacheDisable(ViewExecutable &$view) {
    $entity_types = $view->query->getEntityTableInfo();
    $entity_type_manager = \Drupal::entityTypeManager();

    foreach ($entity_types as $entity_type => $entity_description) {
      $entity_type_definition = $entity_type_manager->getDefinition($entity_type);

      // Set the static cache flag to false.
      $entity_type_definition->set('static_cache', FALSE);
    }
  ...

The standard export method is working correctly.
I do have a relationship to author in data export view.

Can you pls tell me if it is possible that batch export will work also with relationships?

Thanks !

frantisekivanko’s picture

Fixed entityCacheDisable function.
Added try / catch to handel PluginNotFoundException.

p4trizio’s picture

Hello, thank you for this submodule. In my case, I have almost 20k product to export in batch.
I applied patch in #2789531 to get batch feature and it works great if I use it in frontend.
Unfortunately, when I use drush, rows are repeated for every chunk as if it starts again and again with an offset 0 and a limit 100 (that's my view batch size).
Anybody has the same issue? thanks

p4trizio’s picture

Hi, small fix to address problem described in #18

leisurman’s picture

Views data export says to use drush views-data-export [view-name] [display-id] [output-file]. When I do I get the error Command "views-data-export" is not defined. . I am using drush version 9. This post is about a new module to provide drush 9 support. This post was closed. Can the Views data export module provide drush 9 support within its module?

p4trizio’s picture

@leisurman the command is vde_drush:views-data-export
drush vde_drush:views-data-export [view-name] [display-id] [output-file]

or use the alias
drush vde [view-name] [display-id] [output-file]

leisurman’s picture

@p4trizio. Thank you for your help. My View name is atest1 and my display is rest display with the machine name 'rest_export_1'
And I tried.
drush vde_drush:views-data-export atest1 rest-export-1 myfile.json
and
drush vde atest1 rest-export-1 myfile.json
I get error: The view atest1 does not have the rest-export-1 display.

I also tried using undercore in my display name. and then ran.
drush vde_drush:views-data-export atest1 rest_export_1 myfile.json
and
drush vde atest1 rest_export_1 myfile.json
I get error: Incorrect display_id provided, expected a views data export display, found rest_export instead.
My Views format is set to data export, api jsn, and its using fields. I am using Drupal v8.5.8 and Lightning 8.x-3.104.

My task was to create a table list that displays over 3000 nodes and loads all at once. The problem was the view query was slow at 50 seconds. I then used a View to output the 3000 nodes in a static json file. Now the page loads this static json file in 4 seconds. We want to automate and run the view to output the json file every hour. Then the page will load this static json file.

leisurman’s picture

@p4trizio, My fault. I was using a rest display. I created a data export display and then it worked using this drush vde atest1 data_export_1 myfile.json. Thank you for your help!

B-Prod’s picture

Hi have the following error when exporting a CSV from a view:
[error] Error: Call to a member function __toString() on string in Drupal\vde_drush\Commands\VdeDrushCommands->viewsDataExport() (line 201 of /app/web/modules/contrib/views_data_export/modules/vde_drush/src/Commands/VdeDrushCommands.php)>/code>

This may occur when there is no result.

longwave’s picture

#19 works well for me in conjunction with the batch operations patch, but not even sure this needs to be a separate submodule, this could just be merged into Views Data Export itself? It will then automatically work if Drush 9 is installed and do nothing if Drush 9 is not installed.

B-Prod’s picture

Here is a patch that fixes:

  • empty view result (currently this throws an error)
  • backward compatibility (#markup is not a MarkupInterface in previous versions)
Manuel Garcia’s picture

Patch on #26 included the .DS_Store binary file by mistake, which is why it fails to apply.
I manually removed this hunk from the patch file and it applied fine.

diff --git a/modules/vde_drush/src/.DS_Store b/modules/vde_drush/src/.DS_Store
new file mode 100644
index 0000000..240b2d9
Binary files /dev/null and b/modules/vde_drush/src/.DS_Store differ

So here is the regenerated patch. I thought it would be useful to have the interdiff between it and the previous patch #19 so I generated it as well.

Manuel Garcia’s picture

re #25 I totally agree, there is no need for a separate module imho.

Manuel Garcia’s picture

Thought I'd cook up the patch making the command be part of views_data_export itself and not an extra module, so here it is.

A brief summary of what I did:

  • Renamed the command to be views-data-export
  • Moved all the files to their new location.
  • Added the relevant information into the README file
  • Cleaned up a bit some docblocks, CS, etc.
  • Updated the composer.json file to integrate with Drush 9
Manuel Garcia’s picture

mellowtothemax’s picture

This does not run with Drush 10. Any suggestions?

Regards,

steinmb’s picture

Title: Drush 9 support to export views data » Support views data export using drush

Since Drupal 9 req. Drush 10 it think we should change the issue topic and make sure it works with Drush 10.

josephdpurcell’s picture

The documentation states that this command is available. So, I ran it and discovered the drush command is not available. This is a great feature! If I get a chance, I'll try to report back if I was able to use the patch. Thanks for working on this!

patrickfweston’s picture

Hi all,

I applied the patch in #30, but I was getting errors due to a missing drush.services.yml file and the dependency injection it provides:

ArgumentCountError: Too few arguments to function Drupal\views_data_export\Commands\ViewsDataExportCommands::__construct(), 0 passed in /app/docroot/core/lib/Drupal/Component/DependencyInjection/Container.php on line 265 and exactly 4 expected in Drupal\views_data_export\Commands\ViewsDataExportCommands->__construct() (line 68 of /app/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php)

I've added to the patch in #30 to include this file. I was able to get this running on a Drupal 8 site using Drush 9 as a result.

To use the command provided by the patches in this queue, ensure you've disabled and removed the vde_drush contrib module and have cleared cache. Then, run a Drush command like so:
drush views-data-export [view_id] [display_id] [path to export file]

lee.cocklin’s picture

Applied patch from #34 to a Drupal 8.8.10 site with Drush 10 and it runs just fine. No issues to report.

mellowtothemax’s picture

I also tried #34 and it works with Drush 10.

However the output is slightly different to the output of the view preview and the manual download. Not really sure why but for some reason when exporting to xml as in my case, the closing </response> tag is absent and the link to product field (in this case for commerce products) returns http://default/ instead of the site url.

longwave’s picture

Re #36 you have to pass the -l or --uri option to drush to tell it the base URL, it has no way of knowing otherwise.

mellowtothemax’s picture

Thanks longwave that that fixed the url issue.

chiefme’s picture

Drupal: 8.9.6
PHP 7.3.19
Tried both: 8.x-1.0 and 8.x-1.x-dev
Could not apply patch! Skipping. The error was: Cannot apply patch... (#34)

Updated:
Everything is ok in local machine and other places where I tested, can't find out why this only occurs on client's server.
Could this be related with PHP?

Solved:
Stupid problem.. problem was because client VPS wasn't installed or disabled "patch"

composer update -v
...
  - Applying patches for drupal/views_data_export
    https://www.drupal.org/files/issues/2021-06-30/2887450-51.patch (Support views data export using drush)
patch '-p1' --no-backup-if-mismatch -d 'public_html/modules/contrib/views_data_export' < '/tmp/612780e94d18b.patch'
sh: patch: command not found
...
pivica’s picture

Got the same error as in 36:

> Not really sure why but for some reason when exporting to xml as in my case, the closing tag is absent

Here is a patch that should fix this and also fix while loop for a queue - that if check was wrong and was missing the last queue item. Tested this against custom XML format implementation and seems it is working great now.

Status: Needs review » Needs work

The last submitted patch, 40: 2887450-40.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

error84’s picture

Just to confirm patch #34 works for:

Drupal: 9.1.4
Drush: 10.4.0
PHP: 7.4.12

Tested with batched export to CSV.
Had to do 'drush cr' after patch was applied via composer.

PhilippVerpoort’s picture

Just to say that I can also confirm that patch #34 works for me with:

Drupal: 8.9.11
Drush: 10.4.3
PHP: 7.2.24

Also had to clear caches but worked like a charm after that. Only used CSV exports though.

Hoping that this will get finalised and committed soon! :)

steinmb’s picture

Issue tags: +Needs reroll

#40 raises a concern but the patch fails. I would speed things up if this could be rerolled and tested by end users.

Megha_kundar’s picture

Status: Needs work » Needs review
FileSize
1.65 KB

Status: Needs review » Needs work

The last submitted patch, 45: 2887450-45.patch, failed testing. View results

tim-diels’s picture

Patch #40 not working as some files are not inside the patch

steinmb’s picture

@tim-diels Try the reroll in #45

tim-diels’s picture

Sorry I mean that patch #45 is not working. Going to hide this as the patch is completely wrong. Using patch #34 at this moment and works.

BS Pavan’s picture

Status: Needs work » Needs review
FileSize
28.85 KB
697 bytes

Hi @tim-diels

I have rerolled the patch as per #40 and tried to fix the issue. I think the issue is because of count which is mentioned in

https://www.drupal.org/project/views_data_export/issues/3172799#comment-...

Please check once.

Status: Needs review » Needs work

The last submitted patch, 50: 2887450-50.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

BS Pavan’s picture

simgui8’s picture

#52 applies correctly on 8.x-1.0 and fixes "views-data-export" is not defined.

Thanks

ivnish’s picture

Status: Needs review » Reviewed & tested by the community

#52 works

doidd’s picture

I add one row to file src/Plugin/QueueWorker/FileWriter.php to support cron queue.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 55: 2887450-55.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

hfernandes’s picture

Hello,

This patch was not working in my case - I have a views configured to export in batches, but I'm still facing PHP memory limit issues.

I decided to come up with a new proposal. The idea is to use what we already have implemented on views_data_export as much as we can and to use drush_backend_batch_process() for the batched views instead of the Queue API originally proposed by VDE drush add-on.

This will be the result when you are exporting in batches:

fin drush vde user_export data_export_1
 [notice] Starting data export..
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Data export saved to private://views_data_export/user_export_data_export_1/1-1642177429/user_data_export.csv
 [success] Data export finished.

and this will be the result when you have the Standard export method:

fin drush vde user_export data_export_1
 [notice] Starting data export..
 [success] Data export saved to private://views_data_export/user_export_data_export_1/1-1642177832/user_data_export.csv
 [success] Data export finished.

Besides the drush command, the default export method should continue working as usual.

Thoughts?

_randy’s picture

I agree with @hfernandes. The original approach uses gigs of RAM for exports of any real size.

Patch #57 works, however, it only dumps to the private files location.

I'm attaching a patch that uses @hfernandes' base approach and adds to it a command line parameter to copy the final file too. The original file is maintained in the private files location. This should probably be a command line option to remove the private files reference if a discrete file location is provided as is done in this patch.

EDIT:
command would look like:

drush vde view_machine_name data_export_display /path/to/export/myexport.csv
sonfd’s picture

Tested patch in #58 and it does generate the file, but there are a couple issues worth noting:

  1. The command fails if you pass an output file path and the file already exists. IMO - this is not the correct behavior, it should overwrite. Especially if the file is also being saved somewhere else.
  2. When passing a path, the command outputs that it saved the file to default views_data_export directory, not the passed path (though it is also saved there). This is confusing and caused me to think it wasn't working / was ignoring the path.
eelkeblok’s picture

I would like to second the above points, with one small nuance: I would suggest to introduce a "force" option for overwriting the file. I don't think it is a great idea to overwrite by default.

The second point may be the result of the exact interfacing to the module's core code, as it seems to lift the file name from the View's data.

Additionally, the patch suffers from some coding standard violations.

simgui8’s picture

FileSize
29.98 KB
1.04 KB

For #57 and #58, being in a cli, I would expect the output file parameter to accept any accessible location. The default generated public/private file would require an extra step to delete and is not exactly "drush style".

Here is my result after testing #57 and #58
-vde exports were resulting in 0 byte files
-$result['drush_process_finished'] was returning TRUE and [0]['vde_file'] was created (not at the specified output param).

For those who need #55, this is a reroll (#55 failed to apply to latest dev)

Medha Kumari’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
1.67 KB

Rerolled patch #30 in 8.x-1.x .

simgui8’s picture

@Medha Kumari: #62 is already in #61
#61 is the latest patch "legacy style" (a follow-up of every patch since #27 except #57 and #58)

code_brown’s picture

I have found two other patches useful for views_data_export, however they do not apply when using patch #58:



I have attached a patch including the two above which applies to the 8.x-1.1 version of the module, but I get the feeling this isn't the right way of doing things.

What do people recommend I do when I want to include two or more patches for a module but the patches do not cleanly apply together?

code_brown’s picture

Status: Needs review » Needs work

The last submitted patch, 65: 2887450-drush-custom-export-directory-permanent-file-8.x-1.2.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

simgui8’s picture

@code_brown, other issues should stay in their respective thread most of the time.
Mixing other issues makes it harder for maintainers to understand what is going on and will slow down patch adoption.
In doubt, ask for best practices on drupal slack.
As for #64 and #65, I don't think they are duplicates or related to this issue.

And I don't think we should hide other contributors' files.
Only hide yours (maybe you should hide #65 or explain in detail how it's related to this issue).

I hope it helps!

simgui8’s picture

Unhiding files that were previously hidden by mistake.

_randy’s picture

Unfortunately this issue has diverged into two streams. One using what's been referred to as "Legacy" and another which uses the Drush back-end processing capabilities. #57 and #58 address the Drush back-end.

I've re-rolled #58 to work with v1.2.

GuillaumeDuveau’s picture

Patch in #69 still fills the memory for me:

In Process.php line 441:
                                                  
  The process has been signaled with signal "9".

It seems that the batch API is used indeed, but with just 1 operation instead of many. Meaning that the process would run in batch only for the View but as a single PHP process only when called by Drush. Which would kind of defeat the initial goal of having a batch.

      $batch_definition = [
        'operations' => [
          [
            [static::class, 'processBatch'],
            [
              $view->id(),
              $view->current_display,
              $view->args,
              $view->getExposedInput(),
              $total_rows,
              $query_parameters,
              $redirect_url,
            ],
          ],
        ],
        'progressive' => FALSE,
        'finished' => [static::class, 'finishDrushBatch'],
      ];
superlolo95’s picture

#69 works with
Drupal 9.5.0
PHP 8.1.13
Drush 10.6.2
Views data export 8.x-1.2
and run drush cr ==> otherwise the patch is not taken into account by drush

hfernandes’s picture

@GuillaumeDuveau do you have the method "Batch" set in the "Export settings" of your views?
If it's set as "Standard", it'll try to load all data. Maybe that's why you are getting this memory issue.

GuillaumeDuveau’s picture

@hfernandes Yes I have batch with a small limit (100).

On my dataset, calling the view through the web without this patch uses around 512M whereas calling it through Drush with this patch uses around 1.5G.

hfernandes’s picture

Status: Needs work » Needs review
FileSize
23.43 KB

Hello, I've fixed some code standards issues and improved the messaging/logs.

@GuillaumeDuveau could you test this latest one and let me know the result?
It shouldn't behave differently when running via drush or through the web, since it uses the same processBatch function.

hfernandes’s picture

reszli’s picture

The above solution hijacks the views arguments for some internal flags, so it won't work for views that have contextual filters.
I re-wired the patch to move those flags into a different variable, and extended the drush command with a new parameter for specifying arguments.

So now it can be executed as follows:
drush vde [view_name] [display_name] [arg1/arg2/etc] [path/to/file.xml]

Status: Needs review » Needs work

The last submitted patch, 76: vde-drush-with-output-location-2887450-v1.2-76.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

reszli’s picture

jaapjan’s picture

I've applied the idea suggested in #60 in a new patch based on patch #78. Adding a force option, so the export will always be downloadable on the same file path.

Drush command example:
drush vde view_name display_name '' sites/default/files/my-file-name.csv --force

If you don't provide the --force option it will just throw the exception if the file already exists:

In ViewsDataExportCommands.php line 91:
The desired output file already exists. Please remove the file and try again.

jaapjan’s picture

Status: Needs work » Needs review
FileSize
26.08 KB

Last patch was patched against 8.x-1.x. Attached file is patched against 8.x-1.2.

sakonn’s picture

Hello, I have tried this patch on my website and I am encountering mysql error. I am using module to export products from my drupal eshop and during the testing I have found out that using drush for small exports works fine. but with the larger exports I am always getting the error:
General error: 2006 MySQL server has gone away
I think it might be related to my hosting environment where is wait_timeout variable set only to 300 seconds. The thing is that this error is not showing up when running export through browser. In the browser I am redirected to batch page and export finish without problem. Throught browser I am able to export any number of products. Using the cli I get error with about 3000 products (I need to export about 50000 products). In addition to that other large batch processes, for example feeds for import of these products is working fine.

Is there a way to resolve this error with cli command?

edit:
I have migrated to different server and mysql error is gone. But there is still issue with larger exports and drush. As it was already mentioned drush export use enourmos amount of RAM in comparison to browser export. Through browser I am able to make an export of any number of products.

sakonn’s picture

The solution I have found was to set this in settings.php file.

if (PHP_SAPI === 'cli') {
  ini_set('memory_limit', '200M');
}

For some reason it seems that drush is ignoring settings of php memory limit (some discussion here: https://github.com/drush-ops/drush/issues/3294) and is trying to run whole export in one process. After settings this limit the process consumes less then 200Mb of RAM and is working fine even with large exports. During export these messages are logged:

>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
sergiur’s picture

The last working patch for me is the one in comment #75. None of the patches after that seem to save the export at the custom path defined.

I am using version 1.3 of the module which seems to be up to date with the dev version. The command I'm running is drush vde test_export data_export_1 "" private://test/test_export.csv. I have tried the public file system too, but still not saving at the path I set. In the end I am deleting the old file as part of the cron job, before running the drush command using the patch in #75 which works.

PierreEmmanuel’s picture

Patch re-roll for D10 (v1.3)

! As per code review "ViewsDataExportCommands.php" is missing from this patch.

Giuseppe87’s picture

@PierreEmmanuel

The patch at #85 is missing the ViewsDataExportCommands.php class, is a mistake?

Meanwhile, I've updated the patch from #81, as on D10 emptying the caches gives the error:

PHP Fatal error:  Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Fatal error: Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20
SachinT1996’s picture

Patch at #86 does not apply correctly for 8.x-1.3.
Re-rolled for compatibility.

Giuseppe87’s picture

Yes, I forgot to mention that #86 is against the dev version

Giuseppe87’s picture

The #86 as well the #87 missed deprecated a file_save_data - or better, the Rector missed it.

Attached a fixed version of #87

GuillaumePacilly’s picture

Hello,

First thanks for this patch, this feature is really much appreciated.

I could be wrong, but I think the parameters order is inversed between outputfile and arguments in the viewsDataExport function. At least I had to inverse them for it to work properly. Attached an updated patch with the right order.

_randy’s picture

@Giuseppe87 for #89:

Drupal 10.1.6, VDE 8.x-1.4

Patch applies however, in my testing it would appear that the copy of the file never happens to the final specified location.

It seems the $options array has a key in [1], however it is blank.
$args, however, does seem to hold the path in separate array indexes.

I've attached an updated full patch here.

~line 920 of src/Plugin/view/display/DataExport.php, I've altered my version to look like

else {
      // Due to https://github.com/drush-ops/drush/issues/5009
      // we need to set the $context['message'] instead of $context['results'].
      if (in_array('vde_drush', $options)) {
        $file_location = implode('/' , $args);
        $vde_file = \Drupal::service('file_system')->realpath($context['sandbox']['vde_file']);
        // If set, copy the file to the specified directory.
        if (array_key_exists(1, $options) && isset($file_location)) {
          if (!copy($vde_file, $file_location)) {
            $error_message = t('Could not write to final file location (@file). Check permissions.', ['@file' => $file_location]);
            \Drupal::logger('views_data_export')->error($error_message);
          }
          else {
            $context['sandbox']['vde_file'] = $file_location;
          }
        }
        $message = dt('Data export saved to !download_url', ['!download_url' => $context['sandbox']['vde_file']]);
        $context['message'] = $message;
      }

      // We're finished processing, set progress bar to 100%.
      $context['finished'] = 1;
    }
IT-Cru’s picture

Title: Support views data export using drush » Re-add support for Drush commands
IT-Cru’s picture

Title: Re-add support for Drush commands » Re-add Drush commands

IT-Cru’s picture

I've opened a MR based on patch from #90 and adjust this to recent supported Drush version 11/12 to make testing and reviewing easier for all of us.

Patch from #91 has strange format so it was not possible to apply for me.

I think we should also discuss with maintainers of https://www.drupal.org/project/vde_drush module to merge Drush commands back into views_data_export code base.