Re-add Drush commands

+1. great feature, extremely useful

Hi, you can use VDE drush add-on for drush support

great, thank you for the contribution!

The problem is that I need to use batch exporting because the dataset is so big that I have memory problems in standar mode.

Do you plan to add support to batch processing ?

@mscalone Yes, we planned to add batch support soon.

@mscalone Hi!, we've updated our module VDE drush add-on. In current version it supports batched export for views data export. Please read README file or find information on module page on how to use it. Also feel free to create issues and bug reports for it.
Hope this module will help you.

thanks, @a.ilyin I tested last version with batch support but I'm still experimenting memory problems. I read the code and realise its like an emulation of batch processing, and I thinks that's not enough in this case. I think a version that uses queue/batch api its needed in this case.

thank you very much.

Thanks for your response @mscalone.
Could you provide more information, please?
How many items do you have for export, how many fields and the number of rows set for Batch size.

Also can you create an issue for our module?

Thanks!

It seems a little bit inefficient to create a new module for Drush 9 support. Instead you should provide a patch and help to maintain this module. What if tomorrow Views data export incorporates their own Drush 9 commands? VDE drush add-on would be useless then at least, or conflicting at worst.

Please provide a patch in favor of an already existing well-known and well-maintained module.

Hi @mscalone!
We've added Queue API support to our module. Please try to use it

We've also created patch for views data export module to use our module as submodule of views data export.
Thanks to @leymannx for suggesting to do so here

Thank you @a.ilyin

I'm having the following error with a view that exports a field that is a relationship to a taxonomy term:

[error] The "field_xxxxxxx" entity type does not exist. [0.66 sec, 36.14 MB]

If I remove the taxonomy field and the relationship the patch works correctly:

 ...
 [info] Adding data to queue... [1.22 sec, 51.65 MB]
 [info] Queue filled. Starting executing items [1.22 sec, 51.68 MB]
 [info] Exporting records 0 to 1000. [11.51 sec, 119.92 MB]
 [info] Exporting records 1000 to 2000. [23.21 sec, 123.97 MB]
 [info] Exporting records 2000 to 3000. [34.9 sec, 125.48 MB]
 [info] Exporting records 3000 to 4000. [47.82 sec, 125.99 MB]
 [info] Exporting records 4000 to 5000. [60.72 sec, 127.34 MB]
 [info] Exporting records 5000 to 6000. [73.97 sec, 127.8 MB]
 [info] Exporting records 6000 to 7000. [87.81 sec, 128.34 MB]
 [info] Exporting records 7000 to 8000. [101.47 sec, 129.88 MB]
 [info] Exporting records 8000 to 9000. [114.23 sec, 130.53 MB]
 [info] Exporting records 9000 to 10000. [127.75 sec, 131.67 MB]
 [info] Exporting records 10000 to 11000. [140.73 sec, 131.25 MB]
 [info] Exporting records 11000 to 12000. [153.68 sec, 132.07 MB]
 [info] Exporting records 12000 to 13000. [156.32 sec, 91.21 MB]
 [info] Executed [156.32 sec, 91.21 MB]
 [success] Data export saved to /home/XXX/XXXXXXX_export.csv [156.32 sec, 88.72 MB]

You can test it adding a simple relationship to a taxonomy and then adding some field of the taxonomy (ex. language) and do the export.
I din't test another kind of relationship but may be have the same issue.

thanks again!

Thank you @a.ilyin for the patch.

I'm having the following error with a view batch export method.

In EntityTypeManager.php line 133:

  The "uid__node_field_data" entity type does not exist.

Problem is in function entityCacheDisable

private function entityCacheDisable(ViewExecutable &$view) {
    $entity_types = $view->query->getEntityTableInfo();
    $entity_type_manager = \Drupal::entityTypeManager();

    foreach ($entity_types as $entity_type => $entity_description) {
      $entity_type_definition = $entity_type_manager->getDefinition($entity_type);

      // Set the static cache flag to false.
      $entity_type_definition->set('static_cache', FALSE);
    }
  ...

The standard export method is working correctly.
I do have a relationship to author in data export view.

Can you pls tell me if it is possible that batch export will work also with relationships?

Thanks !

Fixed entityCacheDisable function.
Added try / catch to handel PluginNotFoundException.

Hello, thank you for this submodule. In my case, I have almost 20k product to export in batch.
I applied patch in #2789531 to get batch feature and it works great if I use it in frontend.
Unfortunately, when I use drush, rows are repeated for every chunk as if it starts again and again with an offset 0 and a limit 100 (that's my view batch size).
Anybody has the same issue? thanks

Hi, small fix to address problem described in #18

Views data export says to use drush views-data-export [view-name] [display-id] [output-file]. When I do I get the error Command "views-data-export" is not defined. . I am using drush version 9. This post is about a new module to provide drush 9 support. This post was closed. Can the Views data export module provide drush 9 support within its module?

@leisurman the command is vde_drush:views-data-export
drush vde_drush:views-data-export [view-name] [display-id] [output-file]

or use the alias
drush vde [view-name] [display-id] [output-file]

@p4trizio. Thank you for your help. My View name is atest1 and my display is rest display with the machine name 'rest_export_1'
And I tried.
drush vde_drush:views-data-export atest1 rest-export-1 myfile.json
and
drush vde atest1 rest-export-1 myfile.json
I get error: The view atest1 does not have the rest-export-1 display.

I also tried using undercore in my display name. and then ran.
drush vde_drush:views-data-export atest1 rest_export_1 myfile.json
and
drush vde atest1 rest_export_1 myfile.json
I get error: Incorrect display_id provided, expected a views data export display, found rest_export instead.
My Views format is set to data export, api jsn, and its using fields. I am using Drupal v8.5.8 and Lightning 8.x-3.104.

My task was to create a table list that displays over 3000 nodes and loads all at once. The problem was the view query was slow at 50 seconds. I then used a View to output the 3000 nodes in a static json file. Now the page loads this static json file in 4 seconds. We want to automate and run the view to output the json file every hour. Then the page will load this static json file.

@p4trizio, My fault. I was using a rest display. I created a data export display and then it worked using this drush vde atest1 data_export_1 myfile.json. Thank you for your help!

Hi have the following error when exporting a CSV from a view:
[error] Error: Call to a member function __toString() on string in Drupal\vde_drush\Commands\VdeDrushCommands->viewsDataExport() (line 201 of /app/web/modules/contrib/views_data_export/modules/vde_drush/src/Commands/VdeDrushCommands.php)>/code>

This may occur when there is no result.

#19 works well for me in conjunction with the batch operations patch, but not even sure this needs to be a separate submodule, this could just be merged into Views Data Export itself? It will then automatically work if Drush 9 is installed and do nothing if Drush 9 is not installed.

Here is a patch that fixes:

• empty view result (currently this throws an error)
• backward compatibility (#markup is not a MarkupInterface in previous versions)

Patch on #26 included the .DS_Store binary file by mistake, which is why it fails to apply.
I manually removed this hunk from the patch file and it applied fine.

diff --git a/modules/vde_drush/src/.DS_Store b/modules/vde_drush/src/.DS_Store
new file mode 100644
index 0000000..240b2d9
Binary files /dev/null and b/modules/vde_drush/src/.DS_Store differ

So here is the regenerated patch. I thought it would be useful to have the interdiff between it and the previous patch #19 so I generated it as well.

re #25 I totally agree, there is no need for a separate module imho.

Thought I'd cook up the patch making the command be part of views_data_export itself and not an extra module, so here it is.

A brief summary of what I did:

• Renamed the command to be views-data-export
• Moved all the files to their new location.
• Added the relevant information into the README file
• Cleaned up a bit some docblocks, CS, etc.
• Updated the composer.json file to integrate with Drush 9

Missed one spot :)

This does not run with Drush 10. Any suggestions?

Regards,

Since Drupal 9 req. Drush 10 it think we should change the issue topic and make sure it works with Drush 10.

The documentation states that this command is available. So, I ran it and discovered the drush command is not available. This is a great feature! If I get a chance, I'll try to report back if I was able to use the patch. Thanks for working on this!

Hi all,

I applied the patch in #30, but I was getting errors due to a missing drush.services.yml file and the dependency injection it provides:

ArgumentCountError: Too few arguments to function Drupal\views_data_export\Commands\ViewsDataExportCommands::__construct(), 0 passed in /app/docroot/core/lib/Drupal/Component/DependencyInjection/Container.php on line 265 and exactly 4 expected in Drupal\views_data_export\Commands\ViewsDataExportCommands->__construct() (line 68 of /app/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php)

I've added to the patch in #30 to include this file. I was able to get this running on a Drupal 8 site using Drush 9 as a result.

To use the command provided by the patches in this queue, ensure you've disabled and removed the vde_drush contrib module and have cleared cache. Then, run a Drush command like so:
drush views-data-export [view_id] [display_id] [path to export file]

Applied patch from #34 to a Drupal 8.8.10 site with Drush 10 and it runs just fine. No issues to report.

I also tried #34 and it works with Drush 10.

However the output is slightly different to the output of the view preview and the manual download. Not really sure why but for some reason when exporting to xml as in my case, the closing </response> tag is absent and the link to product field (in this case for commerce products) returns http://default/ instead of the site url.

Re #36 you have to pass the -l or --uri option to drush to tell it the base URL, it has no way of knowing otherwise.

Thanks longwave that that fixed the url issue.

Drupal: 8.9.6
PHP 7.3.19
Tried both: 8.x-1.0 and 8.x-1.x-dev
Could not apply patch! Skipping. The error was: Cannot apply patch... (#34)

Updated:
Everything is ok in local machine and other places where I tested, can't find out why this only occurs on client's server.
Could this be related with PHP?

Solved:
Stupid problem.. problem was because client VPS wasn't installed or disabled "patch"

composer update -v
...
  - Applying patches for drupal/views_data_export
    https://www.drupal.org/files/issues/2021-06-30/2887450-51.patch (Support views data export using drush)
patch '-p1' --no-backup-if-mismatch -d 'public_html/modules/contrib/views_data_export' < '/tmp/612780e94d18b.patch'
sh: patch: command not found
...

Got the same error as in 36:

> Not really sure why but for some reason when exporting to xml as in my case, the closing tag is absent

Here is a patch that should fix this and also fix while loop for a queue - that if check was wrong and was missing the last queue item. Tested this against custom XML format implementation and seems it is working great now.

Just to confirm patch #34 works for:

Drupal: 9.1.4
Drush: 10.4.0
PHP: 7.4.12

Tested with batched export to CSV.
Had to do 'drush cr' after patch was applied via composer.

Just to say that I can also confirm that patch #34 works for me with:

Drupal: 8.9.11
Drush: 10.4.3
PHP: 7.2.24

Also had to clear caches but worked like a charm after that. Only used CSV exports though.

Hoping that this will get finalised and committed soon! :)

#40 raises a concern but the patch fails. I would speed things up if this could be rerolled and tested by end users.

Patch #40 not working as some files are not inside the patch

@tim-diels Try the reroll in #45

Sorry I mean that patch #45 is not working. Going to hide this as the patch is completely wrong. Using patch #34 at this moment and works.

Hi @tim-diels

I have rerolled the patch as per #40 and tried to fix the issue. I think the issue is because of count which is mentioned in

https://www.drupal.org/project/views_data_export/issues/3172799#comment-...

Please check once.

#52 applies correctly on 8.x-1.0 and fixes "views-data-export" is not defined.

Thanks

#52 works

I add one row to file src/Plugin/QueueWorker/FileWriter.php to support cron queue.

Hello,

This patch was not working in my case - I have a views configured to export in batches, but I'm still facing PHP memory limit issues.

I decided to come up with a new proposal. The idea is to use what we already have implemented on views_data_export as much as we can and to use drush_backend_batch_process() for the batched views instead of the Queue API originally proposed by VDE drush add-on.

This will be the result when you are exporting in batches:

fin drush vde user_export data_export_1
 [notice] Starting data export..
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Data export saved to private://views_data_export/user_export_data_export_1/1-1642177429/user_data_export.csv
 [success] Data export finished.

and this will be the result when you have the Standard export method:

fin drush vde user_export data_export_1
 [notice] Starting data export..
 [success] Data export saved to private://views_data_export/user_export_data_export_1/1-1642177832/user_data_export.csv
 [success] Data export finished.

Besides the drush command, the default export method should continue working as usual.

Thoughts?

I agree with @hfernandes. The original approach uses gigs of RAM for exports of any real size.

Patch #57 works, however, it only dumps to the private files location.

I'm attaching a patch that uses @hfernandes' base approach and adds to it a command line parameter to copy the final file too. The original file is maintained in the private files location. This should probably be a command line option to remove the private files reference if a discrete file location is provided as is done in this patch.

EDIT:
command would look like:

drush vde view_machine_name data_export_display /path/to/export/myexport.csv

Tested patch in #58 and it does generate the file, but there are a couple issues worth noting:

1. The command fails if you pass an output file path and the file already exists. IMO - this is not the correct behavior, it should overwrite. Especially if the file is also being saved somewhere else.
2. When passing a path, the command outputs that it saved the file to default views_data_export directory, not the passed path (though it is also saved there). This is confusing and caused me to think it wasn't working / was ignoring the path.

I would like to second the above points, with one small nuance: I would suggest to introduce a "force" option for overwriting the file. I don't think it is a great idea to overwrite by default.

The second point may be the result of the exact interfacing to the module's core code, as it seems to lift the file name from the View's data.

Additionally, the patch suffers from some coding standard violations.

For #57 and #58, being in a cli, I would expect the output file parameter to accept any accessible location. The default generated public/private file would require an extra step to delete and is not exactly "drush style".

Here is my result after testing #57 and #58
-vde exports were resulting in 0 byte files
-$result['drush_process_finished'] was returning TRUE and [0]['vde_file'] was created (not at the specified output param).

For those who need #55, this is a reroll (#55 failed to apply to latest dev)

Rerolled patch #30 in 8.x-1.x .

@Medha Kumari: #62 is already in #61
#61 is the latest patch "legacy style" (a follow-up of every patch since #27 except #57 and #58)

I have found two other patches useful for views_data_export, however they do not apply when using patch #58:

• Save export file permanently
• Configure export directory

I have attached a patch including the two above which applies to the 8.x-1.1 version of the module, but I get the feeling this isn't the right way of doing things.

What do people recommend I do when I want to include two or more patches for a module but the patches do not cleanly apply together?

Just re-rolled the above #64 against 8.x-1.2

@code_brown, other issues should stay in their respective thread most of the time.
Mixing other issues makes it harder for maintainers to understand what is going on and will slow down patch adoption.
In doubt, ask for best practices on drupal slack.
As for #64 and #65, I don't think they are duplicates or related to this issue.

And I don't think we should hide other contributors' files.
Only hide yours (maybe you should hide #65 or explain in detail how it's related to this issue).

I hope it helps!

Unhiding files that were previously hidden by mistake.

Unfortunately this issue has diverged into two streams. One using what's been referred to as "Legacy" and another which uses the Drush back-end processing capabilities. #57 and #58 address the Drush back-end.

I've re-rolled #58 to work with v1.2.

Patch in #69 still fills the memory for me:

In Process.php line 441:
                                                  
  The process has been signaled with signal "9".

It seems that the batch API is used indeed, but with just 1 operation instead of many. Meaning that the process would run in batch only for the View but as a single PHP process only when called by Drush. Which would kind of defeat the initial goal of having a batch.

      $batch_definition = [
        'operations' => [
          [
            [static::class, 'processBatch'],
            [
              $view->id(),
              $view->current_display,
              $view->args,
              $view->getExposedInput(),
              $total_rows,
              $query_parameters,
              $redirect_url,
            ],
          ],
        ],
        'progressive' => FALSE,
        'finished' => [static::class, 'finishDrushBatch'],
      ];

#69 works with
Drupal 9.5.0
PHP 8.1.13
Drush 10.6.2
Views data export 8.x-1.2
and run drush cr ==> otherwise the patch is not taken into account by drush

@GuillaumeDuveau do you have the method "Batch" set in the "Export settings" of your views?
If it's set as "Standard", it'll try to load all data. Maybe that's why you are getting this memory issue.

@hfernandes Yes I have batch with a small limit (100).

On my dataset, calling the view through the web without this patch uses around 512M whereas calling it through Drush with this patch uses around 1.5G.

Hello, I've fixed some code standards issues and improved the messaging/logs.

@GuillaumeDuveau could you test this latest one and let me know the result?
It shouldn't behave differently when running via drush or through the web, since it uses the same processBatch function.

The above solution hijacks the views arguments for some internal flags, so it won't work for views that have contextual filters.
I re-wired the patch to move those flags into a different variable, and extended the drush command with a new parameter for specifying arguments.

So now it can be executed as follows:
drush vde [view_name] [display_name] [arg1/arg2/etc] [path/to/file.xml]

fixing errors and coding standards

I've applied the idea suggested in #60 in a new patch based on patch #78. Adding a force option, so the export will always be downloadable on the same file path.

Drush command example:
drush vde view_name display_name '' sites/default/files/my-file-name.csv --force

If you don't provide the --force option it will just throw the exception if the file already exists:

In ViewsDataExportCommands.php line 91:
The desired output file already exists. Please remove the file and try again.

Last patch was patched against 8.x-1.x. Attached file is patched against 8.x-1.2.

Hello, I have tried this patch on my website and I am encountering mysql error. I am using module to export products from my drupal eshop and during the testing I have found out that using drush for small exports works fine. but with the larger exports I am always getting the error:
General error: 2006 MySQL server has gone away
I think it might be related to my hosting environment where is wait_timeout variable set only to 300 seconds. The thing is that this error is not showing up when running export through browser. In the browser I am redirected to batch page and export finish without problem. Throught browser I am able to export any number of products. Using the cli I get error with about 3000 products (I need to export about 50000 products). In addition to that other large batch processes, for example feeds for import of these products is working fine.

Is there a way to resolve this error with cli command?

edit:
I have migrated to different server and mysql error is gone. But there is still issue with larger exports and drush. As it was already mentioned drush export use enourmos amount of RAM in comparison to browser export. Through browser I am able to make an export of any number of products.

The solution I have found was to set this in settings.php file.

if (PHP_SAPI === 'cli') {
  ini_set('memory_limit', '200M');
}

For some reason it seems that drush is ignoring settings of php memory limit (some discussion here: https://github.com/drush-ops/drush/issues/3294) and is trying to run whole export in one process. After settings this limit the process consumes less then 200Mb of RAM and is working fine even with large exports. During export these messages are logged:

>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
>  [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread

The last working patch for me is the one in comment #75. None of the patches after that seem to save the export at the custom path defined.

I am using version 1.3 of the module which seems to be up to date with the dev version. The command I'm running is drush vde test_export data_export_1 "" private://test/test_export.csv. I have tried the public file system too, but still not saving at the path I set. In the end I am deleting the old file as part of the cron job, before running the drush command using the patch in #75 which works.

Patch re-roll for D10 (v1.3)

! As per code review "ViewsDataExportCommands.php" is missing from this patch.

@PierreEmmanuel

The patch at #85 is missing the ViewsDataExportCommands.php class, is a mistake?

Meanwhile, I've updated the patch from #81, as on D10 emptying the caches gives the error:

PHP Fatal error:  Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Fatal error: Type of Drupal\views_data_export\Commands\ViewsDataExportCommands::$logger must be ?Psr\Log\LoggerInterface (as in class Drush\Commands\DrushCommands) in /var/www/html/docroot/modules/contrib/views_data_export/src/Commands/ViewsDataExportCommands.php on line 20

Patch at #86 does not apply correctly for 8.x-1.3.
Re-rolled for compatibility.

Yes, I forgot to mention that #86 is against the dev version

The #86 as well the #87 missed deprecated a file_save_data - or better, the Rector missed it.

Attached a fixed version of #87

Hello,

First thanks for this patch, this feature is really much appreciated.

I could be wrong, but I think the parameters order is inversed between outputfile and arguments in the viewsDataExport function. At least I had to inverse them for it to work properly. Attached an updated patch with the right order.

@Giuseppe87 for #89:

Drupal 10.1.6, VDE 8.x-1.4

Patch applies however, in my testing it would appear that the copy of the file never happens to the final specified location.

It seems the $options array has a key in [1], however it is blank.
$args, however, does seem to hold the path in separate array indexes.

I've attached an updated full patch here.

~line 920 of src/Plugin/view/display/DataExport.php, I've altered my version to look like

else {
      // Due to https://github.com/drush-ops/drush/issues/5009
      // we need to set the $context['message'] instead of $context['results'].
      if (in_array('vde_drush', $options)) {
        $file_location = implode('/' , $args);
        $vde_file = \Drupal::service('file_system')->realpath($context['sandbox']['vde_file']);
        // If set, copy the file to the specified directory.
        if (array_key_exists(1, $options) && isset($file_location)) {
          if (!copy($vde_file, $file_location)) {
            $error_message = t('Could not write to final file location (@file). Check permissions.', ['@file' => $file_location]);
            \Drupal::logger('views_data_export')->error($error_message);
          }
          else {
            $context['sandbox']['vde_file'] = $file_location;
          }
        }
        $message = dt('Data export saved to !download_url', ['!download_url' => $context['sandbox']['vde_file']]);
        $context['message'] = $message;
      }

      // We're finished processing, set progress bar to 100%.
      $context['finished'] = 1;
    }

I've opened a MR based on patch from #90 and adjust this to recent supported Drush version 11/12 to make testing and reviewing easier for all of us.

Patch from #91 has strange format so it was not possible to apply for me.

I think we should also discuss with maintainers of https://www.drupal.org/project/vde_drush module to merge Drush commands back into views_data_export code base.

Thanks for your work on this.

The patch from #90 applied smoothly using the patch command (not git). Tested on Drupal 10.2.26 + VDE 8.x-1.4.

Hoping this merge request gets approved quickly.

The MR needs the latest 8.x-1.x merged in as there is a conflict.

merged 8.x-1.x into this branch, though I will admit I only addressed the merge conflicts and haven't checked the rest of the patch in detail

Updated the merge request:

1. Modernized the drush command with attributes
2. Removed the drush.services.yml as this is longer used in Drush 12+
3. Fixed compatibility with Drupal 11 so PHPUnit runs correctly
4. General code cleanup

Merged 8.x-1.x into this branch, and fixed the merge conflicts.

I re-added the Drupal 11 compatibility fixes that got lost in the merge.

Hi, it seems that since the last update the patch (#90) and MR no longer work. Could it be updated?

I tried to fix it but there are many conflicts I have no context of the feature.

Thanks in advance.

Kind regards.

Sorry, the comment has been duplicated

Or can the patch be merged? I am already successfully testing/running the patch on version 1.4 of Views Data Export on a production site.

I’ve merged the 8.x-1.x branch back into our MR.
I also fixed a PHPCS issue, but there are still other reported issues in the ViewsDataExportCommands.php file:
Namespaced classes/interfaces/traits should be referenced with use statements.

I left this issue unresolved, as it seems we don’t yet have a definitive decision on whether this is a concern. For reference, see #3408527: False Postive: Aliased PHP 8.0 Attributes on Class properties leading to "Namespaced classes/interfaces/traits should be referenced with use statements".

Has anyone here tested this with D11 ?

The drush argument output_file doesn't work since the drush $options are being overridden by some views options.

Patch in attachment.
Diff agains merge request 34 in attachment.

Here is a patch for the current 8.x-1.5 release. Hiding the file.

I'm going to put this back into Needs work, and I'm going to assign it to myself, I'll work on it and get it to a place where I'm confortable committing it, ideally to 1.x, but maybe it'll go in a 2.x version.

I'm not going to commit this as-is because I don't think that the data export class should be making calls like drush_backend_batch_process if it can help it. Like, it seems weird that we're setting this 'your running in Drush' option and then changing quite a bit of the exection if that's the case. I'd rather find a way to isolate all of that to the Drush command if possible, providing enough output/hooks/events where needed for the Drush command to do what it needs to do.

I don't anticipate the actual invocation of the Drush command or the output changing between now and the final code that gets committed, so everyone here who's using the patch should be able to keep on using it and then do a trivial removal of the patch when it lands in a stable release of VDE.

Thanks for the patience everyone, should only be a little longer.

Thank you for your work.

But I have error on using patch MR 34 on line

// @phpstan-ignore-next-line
return \Drupal::service('file.repository')->writeData('', $destination, FileExists::Replace);

Error:

> > [26-May-2025 13:06:31 UTC] Error: Class "Drupal\Core\File\FileExists" not found in /var/www/html/modules/contrib/views_data_export/src/Plugin/views/display/DataExport.php on line 1078 #0 /var/www/html/modules/contrib/views_data_export/src/Plugin/views/display/DataExport.php(800): Drupal\views_data_export\Plugin\views\display\DataExport::getTempFile(Object(Drupal\views\ViewExecutable), 'content_pages_f...', 'data_export')
> > #1 /var/www/html/vendor/drush/drush/includes/batch.inc(257): Drupal\views_data_export\Plugin\views\display\DataExport::processBatch('content_pages_f...', 'data_export', Array, Array, 112100, Array, '', Array, Array)
> > #2 /var/www/html/vendor/drush/drush/includes/batch.inc(204): _drush_batch_worker()
> > #3 /var/www/html/vendor/drush/drush/includes/batch.inc(75): _drush_batch_command('329028')
> > #4 /var/www/html/vendor/drush/drush/src/Commands/core/BatchCommands.php(25): drush_batch_command('329028')
> > #5 [internal function]: Drush\Commands\core\BatchCommands->process('329028', Array)
> > #6 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(276): call_user_func_array(Array, Array)
> > #7 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
> > #8 /var/www/html/vendor/consolidation/annotated-command/src/CommandProcessor.php(175): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
> > #9 /var/www/html/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(387): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
> > #10 /var/www/html/vendor/symfony/console/Command/Command.php(326): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #11 /var/www/html/vendor/symfony/console/Application.php(1096): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #12 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #13 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #14 /var/www/html/vendor/drush/drush/src/Runtime/Runtime.php(110): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #15 /var/www/html/vendor/drush/drush/src/Runtime/Runtime.php(40): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
> > #16 /var/www/html/vendor/drush/drush/drush.php(139): Drush\Runtime\Runtime->run(Array)
> > #17 /var/www/html/vendor/drush/drush/drush(4): require('/var/www/html/v...')
> > #18 /var/www/html/vendor/bin/drush(119): include('/var/www/html/v...')
> > #19 {main}

I have Drupal 10.2.8.

Maybe you mean FileSystemInterface::EXISTS_REPLACE ? And it's work now :)

And another issue (#2).

I execute command drush views_data_export:views-data-export content_pages_for_google_ads data_export and file
private://views_data_export\content_pages_for_google_ads_data_export\1-1748266716\filename.xml was created and never moved to the right place.

It's because in DataExport::processBatch() in line $options = $view->getStyle()->options; variable $options was overwritten.

Issue #3.
On line $uri = $file->getFileUri();:

/var/www/html $ drush @rn views_data_export:views-data-export content_pages_for_google_ads data_export 345.xml
[notice] Starting data export..
> [warning] Undefined variable $file DataExport.php:938
> [26-May-2025 14:22:22 UTC] Error: Call to a member function getFileUri() on null in /var/www/html/modules/contrib/views_data_export/src/Plugin/views/display/DataExport.php on line 938

@sstapleton please see my changes in patch.

Small fix.

BTW I think it's better to have 2 separate patches for versions 10.2 and 10.3+.

#34 (i.e. https://www.drupal.org/files/issues/2020-08-05/2887450-34_0.patch) is a very old patch... Are your patches based on the patch or the merge request?

@eelkeblok Sorry, I mean MR 34.

The following notices appears during the exporting to file:

/var/www/html/rn $ drush views_data_export:views-data-export view_id display_id file.xml
[notice] Starting data export ..
> [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
> [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
> [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
> [notice] Batch process has consumed in excess of 60% of available memory. Starting new thread
> [notice] Data export saved to private: //content_export/content-pages-for-google-ads/content-pages-for-google-ads.xml
[success] Data export finished.

Indeed memory consumption during batch calls grows. But this does not happen if you follow the link in the view and generate the file there. There are batch calls there too, but memory consumption does not grow at all.

Okay, I've done a bit of work on this one, removed the magic array of options and made it a proper classed object.

The major change that I've done here is to remove the account switching to user 1, which is kinda wrong for lots of reasons.

We could add an optional option to the Drush command to do this switching if that's what we want to do I think. Maybe one for a follow up?

Attached is a patch of the current MR!34 diff'd with 8.x-1.6 if people want to apply to it to that and see if it's still working for their use-case.

Correct patch name.

Hi Steven.

With views_data_export 1.6 and this patch on #128, I'm getting:

>  [warning] Undefined variable $file DataExport.php:927
>  [error]  Error: Call to a member function getFileUri() on null in Drupal\views_data_export\Plugin\views\display\DataExport::processBatch() (line 927 of /var/www/html/docroot/modules/contrib/views_data_export/src/Plugin/views/display/DataExport.php)

Line 797 in DataExport.php is probably not being run?

The command I'm using:

drush --root=/var/www/html/scripts/../docroot views_data_export:views-data-export doctor_data_export data_export_1 "/var/www/html/scripts/../docroot/sites/default/files/doc_exports/1749834394/doctor_export.csv"

I've done some further fixes in the MR, and attached a patch off of 1.6 for those who want to test like that.

FYI the code as it stands at the moment wants the output file to be a stream wrapper URI, which is different to the patches and code that has gone before, where it seemed like half the code wanted a stream wrapper and the other half wanted a plain file.

Given that the Drush code now does a bit of a weird thing where it optionally moves the file after generation to a user-specified location, I wonder if anyone could expand on their use-case for specifying the output file?
Are you using it to save it to locations outside of the Drupal private files directory?
Would that be a requirement?

I wonder about leaving the generation of the file alone, and then at the end of the Drush command, it can simply copy the file to the new location. That then seems more in line with my expectations of doing something random with the file at the end of the process tbh.

But, yeah, I'd like to hear the use-cases for this option.

Our use case is pretty simple: the file needs to be available at a specified location, under a specific, descriptive name. We are actually generating the file in the private file location, although I could imagine the need to put it somewhere else. I'm not sure what you mean by "I wonder about leaving the generation of the file alone, and then at the end of the Drush command, it can simply copy the file to the new location." Do you mean to keep the different name out of the scope of this Drush command and leave it to the caller to move the file afterwards? In our case, that would make things more complicated, as we currently have a single line in our crontab to generate the file in a specified location using this command. I suppose it is possible to do that all in a single line in a crontab, but I'd also say that allowing the user to specify a file name for the export file is a very reasonable feature to have.

@eelkeblok: thanks for letting me know your use-case.

I wonder about leaving the generation of the file alone, and then at the end of the Drush command, it can simply copy the file to the new location.

To hopefully clarify what I mean: in the MR there's some code that watches for the end of the batch processing, and at the end changes the location of the file that VDE generates during normal processing, and updates Drupal. So Drupal is fully aware of the file, it's in the manage file page, Drupal will delete the temporary file on cron etc.
This seems like a bit of a kludge to me.

I am wondering about leaving the location of the temporary file alone, so that we don't need to add more code to the regular display handler, but at the end of the Drush command code we could take a copy of the file from the Drupal managed filesystem and copy it somewhere else that Drush has access to, which doesn't need to then use stream wrappers etc. it would basically be a plain file-copy, with no associated clean-up semantics etc.

In our particular case, the output file does not need to be managed by Drupal because it's overwritten every night when a new version is generated so a plain PHP file-copy works for us. We generate the output file every night so that another process from another vendor can log into the server, get the file from a predefined location and process it.

Just sharing my setup in case it’s helpful to others.

In my case, I run the Drush export every night via crontab. The export generates a fresh XML file, which is saved under the same filename to a specific location on the server, overwriting the one from the night before. It works reliably, and I’m still using the patched VDE 1.5 version.

Here’s the cron job I use:

15 3 * * * cd /data/sites/web/www && /usr/local/bin/drush vde publication_export data_1 /data/sites/web/www/client/file.xml --force >> /data/sites/web/www/client/file_export.log 2>&1

@steven jones Thanks for that clarification. Our use case does not rely on the file being managed, so your proposal sounds fine.

Okay, so in summary I believe that the things left to do are:

• Re-work the Drush file saving stuff to make an unmanaged copy at the end of the export process.
• Add back in the weird 'run this as another user' stuff that was unconditionally running exports as UID1

Here's an updated patch, if people want to give the latest code a spin.

I've done those two items now, so we copy the file after the usual export process runs, and we allow for changing the account that's doing the export via Drush.

Okay, just pushed some semi-major changes to the Drush command, mostly internal, one big external change:

It'll now output the file on stdout if you don't specify an output file. That's the major difference.

I'm also tempted to change that as an option, rather than an argument, because that makes much more sense to me. That will introduce a B/C break with the previous patches, but not a crazy big one, so I'm might let myself get away with it.

Much happier with the code now. The changes to the Display class are largely cosmetic and it doesn't have any Drush specific code in it any more, which is also ace.

I think this is good to merge now.

Going to pop this into a 1.x branch and see what happens when I release, since it bumps the PHP version required to 8.1, which we should probably do in a major version, but hey ho, we're not Semver!

Thanks! I tested the command below form the dev version with the updated Drush integration and I’m happy to report it worked perfectly. Hopefully this can make it into a stable release soon. Many thanks to all contributors!

drush vde publication_export data_1 --output-file=/data/sites/web/export/file.xml --force