Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
I will be submitting a patch against the latest dev version to support the "chunking" functionality of Dropzone (see https://www.dropzonejs.com/#config-chunking) to enable uploading large files on slow connections.
Comment | File | Size | Author |
---|---|---|---|
#45 | dropzonejs-support-chuked-uploads-2.10.0.patch | 28.76 KB | Nicasso |
#40 | lightning_media_bulk_chunk.patch | 1.83 KB | Nicasso |
#39 | dropzonejs-support-chuked-uploads-corrected.patch | 28.45 KB | Nicasso |
#38 | interdiff.txt | 5.35 KB | Nicasso |
#38 | dropzonejs-support-chuked-uploads.patch | 29.81 KB | Nicasso |
Issue fork dropzonejs-3125682
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
semiaddict CreditAttribution: semiaddict commentedThe attached patch is my first take at supporting chunked uploads.
Note that one breaking change has been introduced:
The UploadHandlerInterface::getFilename no longer accepts a UploadedFile, but a string with the original filename.
Comment #3
semiaddict CreditAttribution: semiaddict commentedComment #4
semiaddict CreditAttribution: semiaddict commentedHere's a new patch that also adds chunking options to the entity browser widget.
Comment #5
steveoriolI have an error ta the end...
Comment #6
semiaddict CreditAttribution: semiaddict commented@steveoriol, what version of the DropzoneJs library are you using?
Can you try with the latest version (5.7.0)?
I believe older versions had issues with chunked uploads. See https://gitlab.com/meno/dropzone/-/commit/f0c93d34fda2a6ed11f1d5b8fe7c64...
Comment #7
steveoriolYES ! ;-) ... It is working with v5.7.0
Thank you
But, the resulting file is always rename with ".txt" at the end.
Comment #8
semiaddict CreditAttribution: semiaddict commentedSuper!
I'm attaching an updated patch to indicate the requirement of version 5.7.0+ for chunked uploads.
Comment #9
semiaddict CreditAttribution: semiaddict commentedNew patch with fixed white space issues.
Comment #10
steveoriolHello semiaddict,
I still notice that an '.txt' extension is each time add to the file name.
Comment #11
semiaddict CreditAttribution: semiaddict commented@steveoriol,
Does that also happen with chunking disabled ?
The ".txt" extension was already being added for security reasons by the module before I created the patch, but it should normally be removed once the uplaod is done.
I personally haven't experienced this issue, but I'm not using the eb_widget. Are you ?
Comment #12
steveoriolIf I uncheck "Enable chunking" in the "Widget Settings"(/admin/config/content/entity_browser/dropzone_test/widgets) the ".txt" extension is not added.
Comment #13
semiaddict CreditAttribution: semiaddict commentedThe issue seems to happen when the ChunkedUploadHandler is used even if the file is not chunked due to small filesize.
I have fixed this issue in the attached patch as well as a library dependency issue.
Comment #14
semiaddict CreditAttribution: semiaddict commentedComment #15
steveoriolOK semiaddict, It is working for me with patch #13, I tryed with small et big file chunked and the problem is gone ;-)
Thank you
Comment #16
semiaddict CreditAttribution: semiaddict commentedSuper.
I'm putting status back to "Needs review" to notify maintainers that the patch is ready for review.
Comment #17
jungleThanks all for working on this. It's a great feature. I would commit it as soon as possible as a new co-maintainer. But would be great to have tests coverage if possible. I guess @semiaddict forgot to unassign, so unassigning. What if others want to pick this up.
Comment #18
semiaddict CreditAttribution: semiaddict commentedComment #19
semiaddict CreditAttribution: semiaddict commentedHere's an updated patch that applies on 8.x-2.3 and the latest dev.
I unfortunately do not have the time to add tests for the moment.
Comment #20
macdev_drupal CreditAttribution: macdev_drupal commentedI've applied the Patch but I get an error with the php filesize limit. Do I have to raise the php limits? Thougt this patch would help to avoid raising the limits
Comment #21
semiaddict CreditAttribution: semiaddict commented@macdev_drupal could you please provide more information such as:
Comment #22
macdev_drupal CreditAttribution: macdev_drupal commentedError Message from Server:
Error Message from Drupal Log:
Pfad: /dropzonejs/upload?token=ptgfZzMr1dQ6JSB3-hX-m7WSgQ-OyxxbNwmOVswemE0. Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException: in Drupal\dropzonejs\Controller\UploadController->handleUploads() (Zeile 63 von /var/www/html/web/modules/contrib/dropzonejs/src/Controller/UploadController.php).
post_max_size = 100MB
upload_max_filesize = 100MB
File Size is 251MB
Used the Media Bulk Upload Modul https://www.drupal.org/project/media_bulk_upload
Filesize Limit is set to 2000MB
My Intention would be not to raise the PHP limits but let People Upload large files via chunks.
Comment #23
semiaddict CreditAttribution: semiaddict commented@macdev_drupal,
The media_bulk_upload uses a dropzonejs form element directly.
So its code would need to be changed to allow setting chunking options on the form element. But this will most likely not happen before this issue is closed and the patch merged with the dropzonejs module.
What you could do in the meantime is implement a hook_form_alter in one of your custom module to add the chunking options to the media_bulk_upload form.
This would look something like this:
Don't forget to change 'mycustommodule' with your module's name!
I haven't tested this, but I believe it should work as long as the patch in this issue is applied.
If it doesn't, please create a separate "Support request" issue, as this is not exactly the right place for it.
Comment #24
lussolucaI'm trying the patch in #19 and it works, but I've to comment out (or manually set to a big number) the line
'maxFilesize' => $max_size,
in DropzoneJs::preRenderDropzoneJs, otherwise I got aFile is too big
error.I'm missing something?
Comment #25
semiaddict CreditAttribution: semiaddict commentedHello lussoluca,
This is unrelated to the patch.
The limit is likely set by the file field settings.
Can you double check you settings or try to trace the source of the $max_size variable ?
Comment #26
lussolucaYes you're right,
I wasn't set the max upload size for the field because I thought that it cannot be bigger than the PHP's max upload size, but it could be.
Sorry for the noise and thanks for the patch
Comment #27
SpadXIII CreditAttribution: SpadXIII at SIM commentedDid a quick re-roll on latest dev
Comment #28
SpadXIII CreditAttribution: SpadXIII at SIM commentedDid another quick re-roll on latest dev - and uploaded the wrong file :( retry in next comment
Comment #29
SpadXIII CreditAttribution: SpadXIII at SIM commentedReroll, attempt 2
Comment #30
SpadXIII CreditAttribution: SpadXIII at SIM commentedThe third time is the charm! :crossed_fingers:
Comment #31
dishabhadra CreditAttribution: dishabhadra at Axelerant commentedI applied the #30 patch in my application of Drupal 9, patch got applied successfully but it's not working.
After enabling the chunks upload, the file is getting uploaded in chunks but when we click on select entities getting the error.
I debugged the issue and my findings are:
Comment #32
dishabhadra CreditAttribution: dishabhadra at Axelerant commentedComment #33
dishabhadra CreditAttribution: dishabhadra at Axelerant commentedTo solve #31 error related to the filename, I added some changes in the js file and updated the patch.
Please review.
Comment #34
dishabhadra CreditAttribution: dishabhadra at Axelerant commentedComment #35
stefan.korn@dishabhadra: Not sure why you had these issues.
Code you added at:
does never get called if you use chunked upload imho (see comment 'Chunked uploads get an empty response').
Which version of dropzone JS are you using?
Comment #36
abautu CreditAttribution: abautu commentedI have prepared and tested a smaller patch. It doesn't offer all the features from the previous patches, but I'm adding it here in case it helps anyone.
It does only these things:
- enables chunking on all dropzone elements (using default chunk size of 2M from DropzoneJS)
- fixes missing response in the succes callback (in 5.7, the response parameter of success callback is an empty string for chunked uploads; this has been changed in 5.9 to be consistent between chuncked and regular uploads)
- fixes the upload callback to correctly write/append each chunk.
I tested it by uploading files from 5 to 200MB on a PHP server with 20MB upload limit.
Comment #37
abautu CreditAttribution: abautu commentedPatch from #36 failed to apply to dev. I updated it here.
Comment #38
Nicasso CreditAttribution: Nicasso at Unc Inc commentedI made some additional changes to patch #33, that @dishabhadra provided:
- I force the usage of chunking.
- I added a new settings value (max_filesize_default) to be able to set a new maximum file size, thus not depening on the PHP upload_max_filesize value
- Changed the getting of CSRF tokens, to make chunking work for the Media library widget
With these changes chunking now works for me in the following 2 places:
- Bulk upload (admin/content/media/bulk-upload)
- Media library widget
Patch applies on Dropzonejs version 2.8.0 and I used it with in combination with the Dropzone library version 5.9.3.
Complete patch + interdiff with #33 provided.
Comment #39
Nicasso CreditAttribution: Nicasso at Unc Inc commentedPatch file in #38 contained a mistake with info.yml files.
Correct it here in this patch.
Comment #40
Nicasso CreditAttribution: Nicasso at Unc Inc commentedWhen using lightning media, you might also want to include this patch in combination with the path from #39.
Comment #41
vincent signoret CreditAttribution: vincent signoret as a volunteer commentedI applied patch #39 with
I had a media 'document' containing a field of type File. This field has a maximum file size of 1000 MB. I added a widget to the display form of this field of type Entity Browser. This widget has the following configuration settings:
When I tried to upload a file (900MB) I always have the error "File is too big (XXXMiB). Max filesize: XXXMiB."
With some debuging in DropzoneJsEbWidget.php line 165 the
$config = $this->getConfiguration();
returned the configuration I had set in the entity browser widget, except for the max_file_size. The max_file_size alwayss returned the value of my upload_max_filesize in my php.iniI have tried previous patches and different versions of enyo/dropzone, but I always face the same issue.
Comment #42
uberengineer CreditAttribution: uberengineer at Unc Inc commentedPatch #39 does not apply to dropzonejs 2.10.0
Comment #44
semiaddict CreditAttribution: semiaddict as a volunteer commentedJust created a fork and a merge request on the latest version: https://git.drupalcode.org/project/dropzonejs/-/merge_requests/14
Comment #45
Nicasso CreditAttribution: Nicasso at Unc Inc commented@uberengineer, hereby an updated patch for 2.10.0