Problem/Motivation

This ticket is in response to #2888446: View too large for default php session timout. We need a better way to export large data sets. Currently, if the view is large you will likely hit a php timeout and even updating this value might not be enough as there are mysql timeouts as well.

Proposed resolution

While updating the php/mysql timeout value is a possible solution, a better solution would be to leverage the batch api to write to a file on the server.

Remaining tasks

- determine how to do this OOPHP.

I am trying to use this as opportunity to learn rather than have someone else go and do it. Some open questions are

- how best can we leverage the CsvEncoder class? Is this simply extending the class and overriding the encode method?
- CsvEncoder includes an event, should I create another custom event that calls batch api()?

User interface changes

API changes

Data model changes

Comments

robpowell created an issue. See original summary.

robpowell’s picture

robpowell
he/him
Location Boston
CreditAttribution: robpowell as a volunteer commented
Issue summary: View changes
robpowell’s picture

robpowell
he/him
Location Boston
CreditAttribution: robpowell as a volunteer commented
Title: Implement batchAPI so large views don't hit the php timeout or sql timeout » Implement batchAPI so large views don't hit the php/mysql timeout
robpowell’s picture

robpowell
he/him
Location Boston
CreditAttribution: robpowell as a volunteer commented

Glad to see #2888446: View too large for default php session timout was closed. Would love to start work on this, we discuss the approach for this?

grasmash’s picture

grasmash CreditAttribution: grasmash commented

I did a little performance testing. I found that very little time was spent actually serializing the data, most of the resource consumption is done by views attempting to render entities and fields. I managed to improve the performance dramatically by making these changes to my view:

  • Check "raw output" for all fields under the format section.

This allows me to render a view with 2000 rows in approximately 3 seconds.

grasmash’s picture

grasmash CreditAttribution: grasmash commented
Status: Active » Closed (won't fix)

Since this module is not responsible for the bottleneck, implementing the batch API isn't an effective solution. The performance issue needs to be resolved via views configuration or else some modification to the rendering process in core. Batching the serialization isn't going to help things.

FatherShawn’s picture

FatherShawn
Primary language English
Location New York
CreditAttribution: FatherShawn at Digital Pulp commented

I have a view of user_field_data that also joins with three other tables - about 3,000 rows. Setting all the fields to raw it still dies when I try to export via CSV. Any other suggestions before I begin to dig into why?

grasmash’s picture

grasmash CreditAttribution: grasmash commented

You might need to modify your PHP settings by increasing memory limits or execution time.

You might also consider generating two separate CSVs, one for rows zero through 1501 for rows 1500 and onward.