I will be submitting a patch against the latest dev version to support the "chunking" functionality of Dropzone (see https://www.dropzonejs.com/#config-chunking) to enable uploading large files on slow connections.

CommentFileSizeAuthor
#45 dropzonejs-support-chuked-uploads-2.10.0.patch28.76 KBNicasso
#40 lightning_media_bulk_chunk.patch1.83 KBNicasso
#39 dropzonejs-support-chuked-uploads-corrected.patch28.45 KBNicasso
#38 interdiff.txt5.35 KBNicasso
#38 dropzonejs-support-chuked-uploads.patch29.81 KBNicasso
#37 dropzonejs_chunk_upload.patch1.72 KBabautu
#36 dropzonejs_chunk_upload.patch1.85 KBabautu
#33 dropzonejs-support-chuked-uploads-3125682-33.patch27.81 KBdishabhadra
#31 input_element_value.png63.66 KBdishabhadra
#31 finalize_chunk_response.png80.31 KBdishabhadra
#31 last_upload_chunk_response.png81.69 KBdishabhadra
#31 select_entities_error.png43.58 KBdishabhadra
#30 dropzonejs-support-chuked-uploads-3125682-30.patch27.44 KBSpadXIII
#29 dropzonejs-support-chuked-uploads-3125682-29.patch27.49 KBSpadXIII
#28 dropzonejs-support-chuked-uploads-3125682-28.patch0 bytesSpadXIII
#27 dropzonejs-support-chuked-uploads-3125682-27.patch27.35 KBSpadXIII
#20 dropzone.png15.5 KBmacdev_drupal
#19 dropzonejs-support-chuked-uploads-3125682-19.patch27.38 KBsemiaddict
#13 dropzonejs-support-chuked-uploads-3125682-13.patch27.38 KBsemiaddict
#10 adding txt extention.png10.67 KBsteveoriol
#9 dropzonejs-support-chuked-uploads-3125682-9.patch27.8 KBsemiaddict
#8 dropzonejs-support-chuked-uploads-3125682-8.patch27.63 KBsemiaddict
#5 chuke_error.png16.63 KBsteveoriol
#4 dropzonejs-support-chuked-uploads-3125682-4.patch27.74 KBsemiaddict
#3 dropzonejs-support-chuked-uploads-3125682-2.patch18.55 KBsemiaddict
#2 dropzonejs-support-chuked-uploads-3125682-2.patch37.1 KBsemiaddict

Issue fork dropzonejs-3125682

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

semiaddict created an issue. See original summary.

semiaddict’s picture

The attached patch is my first take at supporting chunked uploads.

Note that one breaking change has been introduced:
The UploadHandlerInterface::getFilename no longer accepts a UploadedFile, but a string with the original filename.

semiaddict’s picture

FileSize
18.55 KB
semiaddict’s picture

Here's a new patch that also adds chunking options to the entity browser widget.

steveoriol’s picture

FileSize
16.63 KB

I have an error ta the end...
chuke_error

semiaddict’s picture

@steveoriol, what version of the DropzoneJs library are you using?
Can you try with the latest version (5.7.0)?

I believe older versions had issues with chunked uploads. See https://gitlab.com/meno/dropzone/-/commit/f0c93d34fda2a6ed11f1d5b8fe7c64...

steveoriol’s picture

YES ! ;-) ... It is working with v5.7.0
Thank you

But, the resulting file is always rename with ".txt" at the end.

semiaddict’s picture

Status: Active » Needs review
FileSize
27.63 KB

Super!

I'm attaching an updated patch to indicate the requirement of version 5.7.0+ for chunked uploads.

semiaddict’s picture

FileSize
27.8 KB

New patch with fixed white space issues.

steveoriol’s picture

Status: Needs review » Active
FileSize
10.67 KB

Hello semiaddict,

I still notice that an '.txt' extension is each time add to the file name.
adding txt extension

semiaddict’s picture

@steveoriol,
Does that also happen with chunking disabled ?
The ".txt" extension was already being added for security reasons by the module before I created the patch, but it should normally be removed once the uplaod is done.
I personally haven't experienced this issue, but I'm not using the eb_widget. Are you ?

steveoriol’s picture

If I uncheck "Enable chunking" in the "Widget Settings"(/admin/config/content/entity_browser/dropzone_test/widgets) the ".txt" extension is not added.

semiaddict’s picture

The issue seems to happen when the ChunkedUploadHandler is used even if the file is not chunked due to small filesize.
I have fixed this issue in the attached patch as well as a library dependency issue.

semiaddict’s picture

Status: Active » Needs review
steveoriol’s picture

Status: Needs review » Active

OK semiaddict, It is working for me with patch #13, I tryed with small et big file chunked and the problem is gone ;-)
Thank you

semiaddict’s picture

Status: Active » Needs review

Super.
I'm putting status back to "Needs review" to notify maintainers that the patch is ready for review.

jungle’s picture

Assigned: semiaddict » Unassigned
Issue tags: +Needs tests

Thanks all for working on this. It's a great feature. I would commit it as soon as possible as a new co-maintainer. But would be great to have tests coverage if possible. I guess @semiaddict forgot to unassign, so unassigning. What if others want to pick this up.

semiaddict’s picture

Title: Support chuked uploads » Support chunked uploads
semiaddict’s picture

Here's an updated patch that applies on 8.x-2.3 and the latest dev.
I unfortunately do not have the time to add tests for the moment.

macdev_drupal’s picture

FileSize
15.5 KB

I've applied the Patch but I get an error with the php filesize limit. Do I have to raise the php limits? Thougt this patch would help to avoid raising the limits
dropzone chunk upload fails

semiaddict’s picture

@macdev_drupal could you please provide more information such as:

  • the full error message (either from Drupal's logs or server logs)
  • PHP upload_max_filesize and post_max_size
  • the file's size you're uploading
  • whether using entity browser widget or a custom form element
  • settings of widget or form element
macdev_drupal’s picture

Error Message from Server:

2021/01/29 16:17:37 [warn] 1040#1040: *3175 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000508, client: 172.18.0.6, server: _, request: "POST /dropzonejs/upload?token=ptgfZzMr1dQ6JSB3-hX-m7WSgQ-OyxxbNwmOVswemE0 HTTP/1.1", host: "web-fresh.ddev.site", referrer: "https://web-fresh.ddev.site/media/bulk-upload/media_bulk_upload_file"

[29-Jan-2021 16:17:38] WARNING: [pool www] child 8539 said into stderr: "NOTICE: PHP message: PHP Warning: POST Content-Length of 257566690 bytes exceeds the limit of 104857600 bytes in Unknown on line 0"

2021/01/29 16:17:39 [error] 1040#1040: *3175 FastCGI sent in stderr: "PHP message: PHP Warning: POST Content-Length of 257566690 bytes exceeds the limit of 104857600 bytes in Unknown on line 0" while reading response header from upstream, client: 172.18.0.6, server: _, request: "POST /dropzonejs/upload?token=ptgfZzMr1dQ6JSB3-hX-m7WSgQ-OyxxbNwmOVswemE0 HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm.sock:", host: "web-fresh.ddev.site", referrer: "https://web-fresh.ddev.site/media/bulk-upload/media_bulk_upload_file"

Error Message from Drupal Log:
Pfad: /dropzonejs/upload?token=ptgfZzMr1dQ6JSB3-hX-m7WSgQ-OyxxbNwmOVswemE0. Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException: in Drupal\dropzonejs\Controller\UploadController->handleUploads() (Zeile 63 von /var/www/html/web/modules/contrib/dropzonejs/src/Controller/UploadController.php).

post_max_size = 100MB
upload_max_filesize = 100MB

File Size is 251MB

Used the Media Bulk Upload Modul https://www.drupal.org/project/media_bulk_upload
Filesize Limit is set to 2000MB

My Intention would be not to raise the PHP limits but let People Upload large files via chunks.

semiaddict’s picture

@macdev_drupal,
The media_bulk_upload uses a dropzonejs form element directly.
So its code would need to be changed to allow setting chunking options on the form element. But this will most likely not happen before this issue is closed and the patch merged with the dropzonejs module.

What you could do in the meantime is implement a hook_form_alter in one of your custom module to add the chunking options to the media_bulk_upload form.

This would look something like this:

function mycustommodule_form_media_bulk_upload_form_alter(&$form, $form_state, $form_id) {
    if (isset($form['dropzonejs'])) {
        $form['dropzonejs']['#force_chunking'] = FALSE; // Set to TRUE to force even for small files.
        $form['dropzonejs']['#chunk_size'] = 2000000; // The max chunk size in bytes
        $form['dropzonejs']['#parallel_chunk_uploads'] = FALSE; // Set to TRUE to allow uploading multiple chunks in parallel
        $form['dropzonejs']['#retry_chunks'] = FALSE; // Set to TRUE to retry failed chunks
        $form['dropzonejs']['#retry_chunks_limit'] = 3; // Number of times a chunk upload should be retried
    }
}

Don't forget to change 'mycustommodule' with your module's name!
I haven't tested this, but I believe it should work as long as the patch in this issue is applied.
If it doesn't, please create a separate "Support request" issue, as this is not exactly the right place for it.

lussoluca’s picture

I'm trying the patch in #19 and it works, but I've to comment out (or manually set to a big number) the line 'maxFilesize' => $max_size, in DropzoneJs::preRenderDropzoneJs, otherwise I got a File is too big error.

I'm missing something?

semiaddict’s picture

Hello lussoluca,
This is unrelated to the patch.
The limit is likely set by the file field settings.
Can you double check you settings or try to trace the source of the $max_size variable ?

lussoluca’s picture

Yes you're right,
I wasn't set the max upload size for the field because I thought that it cannot be bigger than the PHP's max upload size, but it could be.

Sorry for the noise and thanks for the patch

SpadXIII’s picture

Did a quick re-roll on latest dev

SpadXIII’s picture

FileSize
0 bytes

Did another quick re-roll on latest dev - and uploaded the wrong file :( retry in next comment

SpadXIII’s picture

FileSize
27.49 KB

Reroll, attempt 2

SpadXIII’s picture

FileSize
27.44 KB

The third time is the charm! :crossed_fingers:

dishabhadra’s picture

I applied the #30 patch in my application of Drupal 9, patch got applied successfully but it's not working.

After enabling the chunks upload, the file is getting uploaded in chunks but when we click on select entities getting the error.

error

I debugged the issue and my findings are:

  • For chunks upload there are 2 routes one for upload-path and one for finalize-path
  • After chunks upload last upload-path has the response with filename Eg, upload
  • Finalize path has the response with filename Eg, finalize
  • After chunks upload input element getting some value from js (/dropzonejs/js/dropzone.integration.js) that value is incorrect. value is the last chunk uploaded path filename instead of finalize path filename. input_value due to this I am facing above error.
dishabhadra’s picture

Status: Needs review » Needs work
dishabhadra’s picture

Status: Needs work » Needs review
FileSize
27.81 KB

To solve #31 error related to the filename, I added some changes in the js file and updated the patch.

Please review.

dishabhadra’s picture

Version: 8.x-2.x-dev » 8.x-2.6
stefan.korn’s picture

Version: 8.x-2.6 » 8.x-2.7

@dishabhadra: Not sure why you had these issues.

Code you added at:

          // Chunked uplaods get an empty response.
          // (see https://gitlab.com/meno/dropzone/-/blob/master/src/dropzone.js#L1978).
          if (response) {
            // For chunked upload replace chunk count from the file path.
            if (instanceConfig.chunking) {
              var lastChunkValue = '.part' + (file.upload.totalChunkCount - 1);
              if (response.result.indexOf(lastChunkValue) > -1) {
                response.result = response.result.replace(lastChunkValue, '');
              }
            }
            // The file is transliterated on upload. The element has to reflect
            // the real filename.
            file.processedName = response.result;
          }

does never get called if you use chunked upload imho (see comment 'Chunked uploads get an empty response').

Which version of dropzone JS are you using?

abautu’s picture

I have prepared and tested a smaller patch. It doesn't offer all the features from the previous patches, but I'm adding it here in case it helps anyone.

It does only these things:
- enables chunking on all dropzone elements (using default chunk size of 2M from DropzoneJS)
- fixes missing response in the succes callback (in 5.7, the response parameter of success callback is an empty string for chunked uploads; this has been changed in 5.9 to be consistent between chuncked and regular uploads)
- fixes the upload callback to correctly write/append each chunk.

I tested it by uploading files from 5 to 200MB on a PHP server with 20MB upload limit.

abautu’s picture

FileSize
1.72 KB

Patch from #36 failed to apply to dev. I updated it here.

Nicasso’s picture

I made some additional changes to patch #33, that @dishabhadra provided:

- I force the usage of chunking.
- I added a new settings value (max_filesize_default) to be able to set a new maximum file size, thus not depening on the PHP upload_max_filesize value
- Changed the getting of CSRF tokens, to make chunking work for the Media library widget

With these changes chunking now works for me in the following 2 places:

- Bulk upload (admin/content/media/bulk-upload)
- Media library widget

Patch applies on Dropzonejs version 2.8.0 and I used it with in combination with the Dropzone library version 5.9.3.

Complete patch + interdiff with #33 provided.

Nicasso’s picture

Patch file in #38 contained a mistake with info.yml files.

Correct it here in this patch.

Nicasso’s picture

When using lightning media, you might also want to include this patch in combination with the path from #39.

vincent signoret’s picture

I applied patch #39 with

  • - drupal 10.2.1
  • - dropzonejs v2.8
  • - entity_browser v2.10



I had a media 'document' containing a field of type File. This field has a maximum file size of 1000 MB. I added a widget to the display form of this field of type Entity Browser. This widget has the following configuration settings:

    settings:
      submit_text: 'Sélectionner le document'
      auto_select: false
      upload_location: 'public://[date:custom:Y]-[date:custom:m]'
      dropzone_description: 'Drop files here to upload them'
      max_filesize: 1000M
      extensions: 'pdf odt ods xls xlsx doc docx xhtml xbrl zip'
      clientside_resize: false
      resize_width: null
      resize_height: null
      resize_quality: 1.0
      resize_method: contain
      thumbnail_method: contain
      chunking: true
      force_chunking: true
      chunk_size: 2000000
      parallel_chunk_uploads: true
      retry_chunks: true
      retry_chunks_limit: 3

When I tried to upload a file (900MB) I always have the error "File is too big (XXXMiB). Max filesize: XXXMiB."

With some debuging in DropzoneJsEbWidget.php line 165 the $config = $this->getConfiguration(); returned the configuration I had set in the entity browser widget, except for the max_file_size. The max_file_size alwayss returned the value of my upload_max_filesize in my php.ini

I have tried previous patches and different versions of enyo/dropzone, but I always face the same issue.

uberengineer’s picture

Patch #39 does not apply to dropzonejs 2.10.0

semiaddict’s picture

Just created a fork and a merge request on the latest version: https://git.drupalcode.org/project/dropzonejs/-/merge_requests/14

Nicasso’s picture

@uberengineer, hereby an updated patch for 2.10.0