Support chunked uploads

I have an error ta the end...

YES ! ;-) ... It is working with v5.7.0
Thank you

But, the resulting file is always rename with ".txt" at the end.

Hello semiaddict,

I still notice that an '.txt' extension is each time add to the file name.

If I uncheck "Enable chunking" in the "Widget Settings"(/admin/config/content/entity_browser/dropzone_test/widgets) the ".txt" extension is not added.

OK semiaddict, It is working for me with patch #13, I tryed with small et big file chunked and the problem is gone ;-)
Thank you

Thanks all for working on this. It's a great feature. I would commit it as soon as possible as a new co-maintainer. But would be great to have tests coverage if possible. I guess @semiaddict forgot to unassign, so unassigning. What if others want to pick this up.

I've applied the Patch but I get an error with the php filesize limit. Do I have to raise the php limits? Thougt this patch would help to avoid raising the limits

Error Message from Server:

2021/01/29 16:17:37 [warn] 1040#1040: *3175 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000508, client: 172.18.0.6, server: _, request: "POST /dropzonejs/upload?token=ptgfZzMr1dQ6JSB3-hX-m7WSgQ-OyxxbNwmOVswemE0 HTTP/1.1", host: "web-fresh.ddev.site", referrer: "https://web-fresh.ddev.site/media/bulk-upload/media_bulk_upload_file"

[29-Jan-2021 16:17:38] WARNING: [pool www] child 8539 said into stderr: "NOTICE: PHP message: PHP Warning: POST Content-Length of 257566690 bytes exceeds the limit of 104857600 bytes in Unknown on line 0"

2021/01/29 16:17:39 [error] 1040#1040: *3175 FastCGI sent in stderr: "PHP message: PHP Warning: POST Content-Length of 257566690 bytes exceeds the limit of 104857600 bytes in Unknown on line 0" while reading response header from upstream, client: 172.18.0.6, server: _, request: "POST /dropzonejs/upload?token=ptgfZzMr1dQ6JSB3-hX-m7WSgQ-OyxxbNwmOVswemE0 HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm.sock:", host: "web-fresh.ddev.site", referrer: "https://web-fresh.ddev.site/media/bulk-upload/media_bulk_upload_file"

Error Message from Drupal Log:
Pfad: /dropzonejs/upload?token=ptgfZzMr1dQ6JSB3-hX-m7WSgQ-OyxxbNwmOVswemE0. Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException: in Drupal\dropzonejs\Controller\UploadController->handleUploads() (Zeile 63 von /var/www/html/web/modules/contrib/dropzonejs/src/Controller/UploadController.php).

post_max_size = 100MB
upload_max_filesize = 100MB

File Size is 251MB

Used the Media Bulk Upload Modul https://www.drupal.org/project/media_bulk_upload
Filesize Limit is set to 2000MB

My Intention would be not to raise the PHP limits but let People Upload large files via chunks.

I'm trying the patch in #19 and it works, but I've to comment out (or manually set to a big number) the line 'maxFilesize' => $max_size, in DropzoneJs::preRenderDropzoneJs, otherwise I got a File is too big error.

I'm missing something?

Yes you're right,
I wasn't set the max upload size for the field because I thought that it cannot be bigger than the PHP's max upload size, but it could be.

Sorry for the noise and thanks for the patch

Did a quick re-roll on latest dev

Did another quick re-roll on latest dev - and uploaded the wrong file :( retry in next comment

Reroll, attempt 2

The third time is the charm! :crossed_fingers:

I applied the #30 patch in my application of Drupal 9, patch got applied successfully but it's not working.

After enabling the chunks upload, the file is getting uploaded in chunks but when we click on select entities getting the error.

I debugged the issue and my findings are:

• For chunks upload there are 2 routes one for upload-path and one for finalize-path
• After chunks upload last upload-path has the response with filename Eg,
• Finalize path has the response with filename Eg,
• After chunks upload input element getting some value from js (/dropzonejs/js/dropzone.integration.js) that value is incorrect. value is the last chunk uploaded path filename instead of finalize path filename. due to this I am facing above error.

To solve #31 error related to the filename, I added some changes in the js file and updated the patch.

Please review.

@dishabhadra: Not sure why you had these issues.

Code you added at:

          // Chunked uplaods get an empty response.
          // (see https://gitlab.com/meno/dropzone/-/blob/master/src/dropzone.js#L1978).
          if (response) {
            // For chunked upload replace chunk count from the file path.
            if (instanceConfig.chunking) {
              var lastChunkValue = '.part' + (file.upload.totalChunkCount - 1);
              if (response.result.indexOf(lastChunkValue) > -1) {
                response.result = response.result.replace(lastChunkValue, '');
              }
            }
            // The file is transliterated on upload. The element has to reflect
            // the real filename.
            file.processedName = response.result;
          }

does never get called if you use chunked upload imho (see comment 'Chunked uploads get an empty response').

Which version of dropzone JS are you using?

I have prepared and tested a smaller patch. It doesn't offer all the features from the previous patches, but I'm adding it here in case it helps anyone.

It does only these things:
- enables chunking on all dropzone elements (using default chunk size of 2M from DropzoneJS)
- fixes missing response in the succes callback (in 5.7, the response parameter of success callback is an empty string for chunked uploads; this has been changed in 5.9 to be consistent between chuncked and regular uploads)
- fixes the upload callback to correctly write/append each chunk.

I tested it by uploading files from 5 to 200MB on a PHP server with 20MB upload limit.

Patch from #36 failed to apply to dev. I updated it here.

I made some additional changes to patch #33, that @dishabhadra provided:

- I force the usage of chunking.
- I added a new settings value (max_filesize_default) to be able to set a new maximum file size, thus not depening on the PHP upload_max_filesize value
- Changed the getting of CSRF tokens, to make chunking work for the Media library widget

With these changes chunking now works for me in the following 2 places:

- Bulk upload (admin/content/media/bulk-upload)
- Media library widget

Patch applies on Dropzonejs version 2.8.0 and I used it with in combination with the Dropzone library version 5.9.3.

Complete patch + interdiff with #33 provided.

Patch file in #38 contained a mistake with info.yml files.

Correct it here in this patch.

When using lightning media, you might also want to include this patch in combination with the path from #39.

I applied patch #39 with

• - drupal 10.2.1
• - dropzonejs v2.8
• - entity_browser v2.10

I had a media 'document' containing a field of type File. This field has a maximum file size of 1000 MB. I added a widget to the display form of this field of type Entity Browser. This widget has the following configuration settings:

    settings:
      submit_text: 'Sélectionner le document'
      auto_select: false
      upload_location: 'public://[date:custom:Y]-[date:custom:m]'
      dropzone_description: 'Drop files here to upload them'
      max_filesize: 1000M
      extensions: 'pdf odt ods xls xlsx doc docx xhtml xbrl zip'
      clientside_resize: false
      resize_width: null
      resize_height: null
      resize_quality: 1.0
      resize_method: contain
      thumbnail_method: contain
      chunking: true
      force_chunking: true
      chunk_size: 2000000
      parallel_chunk_uploads: true
      retry_chunks: true
      retry_chunks_limit: 3

When I tried to upload a file (900MB) I always have the error "File is too big (XXXMiB). Max filesize: XXXMiB."

With some debuging in DropzoneJsEbWidget.php line 165 the $config = $this->getConfiguration(); returned the configuration I had set in the entity browser widget, except for the max_file_size. The max_file_size alwayss returned the value of my upload_max_filesize in my php.ini

I have tried previous patches and different versions of enyo/dropzone, but I always face the same issue.

Patch #39 does not apply to dropzonejs 2.10.0

@uberengineer, hereby an updated patch for 2.10.0

Rerolled the lightning_media patch for version 5.0.0.

What’s the holdup on merging the PR? We use this in production across multiple websites. What can we do to get this moved into a new release?

Ah ha good to know semiaddict. As you are the creator of this wonderful gem will you be adding the testing or waiting on the maintainers?

I tested MR14 but have the same issue as #41

if php upload_max_filesize is smaller the file size I am trying to upload, the error that file is too big referring at the limit of the PHP Limit.

Testing this as we finally have a use case for this.

The chunked upload itself seems to work fine, however, then nothing happens.

Debugging a bit, I can see that despite the comment saying otherwise, dropzoneInstance.on('success' is in fact called with a response, on the last part, *after* Drupal.dropzonejs.chunksUploaded, and it has result filename with the last part. This overrides file.processedName, this is then submitted and obviously the server then doesn't find that anymore.

This is with dropzone 6.x beta2 so there might have been a change with that.

Commenting out that line as a quickfix solves that and the media can then be edited. But it needs a better check, maybe by checking if file.processedName is already set and then assuming that it was set by the chunk response so that it's compatible with different versions?

Also, if you have problems with max file size, then you might be on an old core version, #2867336: File size validator should only respect the explicitly configured maximum file size should have resolved that.

@bedir you're seeing a 500 error on finalize with MR14? If so, what line did you end up commenting out?

I can confirm the comment #51 with MR14 and dropzone 6.0.0-beta.2. The success event does have a respone for the last uploaded chunk (eg. myfile.mp4.txt.part9). This overwrites the uploadedFilesElement value. So the comment " Chunked uploads get an empty response." is not entirely correct. I am unsure if this is only in 6.0.0-beta.2.

I fixed it by checking if the instance uses chunking:

        dropzoneInstance.on('success', function (file, response) {
          var uploadedFilesElement = selector.siblings(':hidden');
          var currentValue = uploadedFilesElement.attr('value') || '';
          // The file is transliterated on upload. The element has to
          // reflect the real filename.
          // Skip processing for chunked uploads.
          var uploadIsChunked = instanceConfig.chunking && file.upload.totalChunkCount > 1;
          if (!uploadIsChunked) {
            // The name for chunked uploads was already updated in the
            // finalize callback. Only update the name for non-chunked
            // uploads.
            if (response && response.result) {
              file.processedName = response.result;
            }
          }
          uploadedFilesElement.attr('value', currentValue + file.processedName + ';');
        });

Ir works fine after this small change.

Tried testing the MR. it applied, but I was unable to get it to work.

The images appear to upload in dropzone. but when I click the submit button I get an error message "At least one valid file should be uploaded."

and in the js console i have

VM11500 drupal.js:64 Uncaught TypeError: Cannot read properties of undefined (reading 'element')
    at VM11522 dropzonejs_eb_widget.common.js:15:39
    at Array.forEach (<anonymous>)
    at Object.attach (VM11522 dropzonejs_eb_widget.common.js:14:60)
    at VM11500 drupal.js:166:24
    at Array.forEach (<anonymous>)
    at Drupal.attachBehaviors (VM11500 drupal.js:162:34)
    at HTMLDivElement.<anonymous> (VM11520 ajax.js:1404:18)
    at ce.each (VM11495 jquery.min.js:2:3129)
    at ce.fn.init.each (VM11495 jquery.min.js:2:1594)
    at Drupal.AjaxCommands.insert (VM11520 ajax.js:1396:19)

Both the EB Media Dropzone widget, and the EB Media Dropzone widget with edit widet.

I tried to reproduce the error, but it seems to work normally for me (inside EB or Webform element).
From the error it looks like the instance inst registered in drupalSettings.dropzonejs.instances. Dit you configur ethe EB widget as Dropzone plugin?

I can confirm comment #54. Uploading file(s) with the EB widget plugins 'dropzonejs_media_entity_inline_entity_form' and 'dropzonejs_media_entity' does not work correctly with chunked uploads.
The current logic determines whether an upload is chunked by inferring it indirectly from totalChunkCount while Dropzone already exposes the correct per-file flag on the client side: file.upload.chunked. This is also important for the filename handling.