Problem/Motivation

The "zip" step in BatchOperations::copyEntityFilesAndZip() which appends new files to an existing zip can cause several problems and needs to be reworked.

  • On systems that are handling large files, an individual file can take a substantial amount of time to be added to a zip file, introducing max_execution_time timeouts.
  • The zipping of a file is an order of magnitude slower than the copy operation, causing each operation to be quite slow compared to how quickly they run if the "zip" step is skipped.

Proposed resolution

Reduce the zipping behavior to an optional feature or even consider removing it completely because command line or other non-PHP methods is significantly better and more reliable. The biggest benefit of this module is gathering all of the necessary files for a deployment in one place, and many developer workflows wouldn't benefit from the additional zip stage, either.

Remaining tasks

  • Root cause analysis.
  • Make zipping optional.
  • Provide guidance on rsync or scp commands if a batch has been completed; either in docs or on-screen (or both)

Comments

wwedding created an issue. See original summary.

wwedding’s picture

wwedding commented
Title: Something might cause entities to be skipped or files attached to those entities to be skipped » Zipping files can cause severe slowdown and batch operations that don't even complete

Problem/Motivation

The "zip" step in BatchOperations::copyEntityFilesAndZip() which appends new files to an existing zip can cause several problems and needs to be reworked.

Proposed resolution

Make zip take place in a part of the batch where a failure will not potentially ruin the whole process, while also offering guidance on what to do with configuration screen to attempt to work around problems.

Remaining tasks

  • Root cause analysis.
  • Make zipping optional.
  • Move the zip step to the final stage of the batch, so that a failure to zip doesn't leave the entire batch in a half-copied state.
  • Provide guidance on rsync or scp commands if a batch has been completed.
wwedding’s picture

wwedding commented
Related issues: +#3032270: If a file fails to copy to the packager directory, any subsequent URIs to process for the batch operation are ignored completely
wwedding’s picture

wwedding commented
Assigned: wwedding » Unassigned
wwedding’s picture

wwedding commented
Related issues: +#3047003: Batch size configuration setting isn't used in the zip behavior
wwedding’s picture

wwedding commented
Issue summary: View changes
wwedding’s picture

wwedding commented
Status: Active » Needs work
wwedding’s picture

wwedding commented
Issue summary: View changes

After having used the zip feature for several more deployments, I think it's not actually that bad. For the most part, it has worked out really well even with relatively short PHP timeouts specified; I think it still deserves a spot as a core (with the ability to be made optional) feature and not removed from the project page or anything else extreme I was considering when I first had this issue.

Improved documentation and troubleshooting are still certainly necessary, as well as allowing the zip step to be skipped.

wwedding’s picture

wwedding commented
Status: Needs work » Closed (outdated)

Documentation concerns are covered in #3050530: Write documentation that provides guidance about what to do with copied files