From #318960: Upload Progress Bar (rfc1867 support):
PECL Uploadprogress sounds good, but there is one problem that you have not thought.
What happens when PECL Uploadprogress is installed and it is not an apache webserver?
the upload fails!
An HTTP error 404 occurred. /filefield/progress/a8989ffa6226356644edc7246c704799I use lighttpd and the new dev of filefield breaks all filefields.
There must be a server check that the apache is used as webserver or checkbox to (de-)activate the uploadprogress.
Lighttp 1.4 has no support if pecl uploadprogress
lighttpd 1.5 has it, but it is not stable and a stable release is far away.I do not know the support of uploadprogress on other webserver like nginx or other, but you should be some checks, an advice and a help in the readme.txt or install.txt
Comment | File | Size | Author |
---|---|---|---|
#28 | uploadprogress.patch | 2.26 KB | smoothify |
Comments
Comment #1
quicksketchI'm wondering, why would you have uploadprogress installed on a server that doesn't support it? It seems like the solution is to uninstall the uploadprogress pecl extension if it doesn't work.
Comment #2
quicksketchThe more I think about this, the less appropriate it seems that FileField should need to accommodate it. Should FileField really ship with a comparison list of when it should or shouldn't use a library based on server? If the server is not configured correctly, it's a server problem, not something the software should need to account for. It's just using the library that's available.
Comment #3
quicksketchMarking postponed until someone can rationalize why FileField should account for a broken library.
Comment #4
quicksketchAlthough I still think it's silly to check not just if a library is available, but if it actually works, the PECL extension page is pretty clear about which platforms uploadprogress works on:
Comment #5
quicksketchYou can now disable the progress bar per-field by using the option added in #439694: Progress bar option.
Comment #6
quicksketchWell after no responses I'm marking won't fix. A simple work-around is to just not enable the progress bar if your server has the PECL extension installed but it doesn't work.
Comment #7
j0rd CreditAttribution: j0rd commentedIt also does not work in nginx. Uploading a file via filefield works, just not the progress bar. Works by default and no need to disable anything. So I'm happy.
Ideally I'd like the progress bar if it's possible, but I can't find any information on how I can even attempt on getting this working. It would be nice if someone who knows more about what's required for this could look into it and see if it's possible by say enabling a module for NGINX which I'm not aware of.
Any assistance or guidance is appreciated.
NGINX has this module http://wiki.nginx.org/NginxHttpUploadProgressModule which can be patched into ubuntu's nginx by downloading and extracting nginx_uploadprogress_module into /usr/src and doing.
cd /usr/src
apt-get source nginx
cd nginx-0.6.35
vim auto/options
EDIT THE LINE NGX_ADDONS= to be NGX_ADDONS=/usr/src/nginx_uploadprogress_module
dpkg-buildpackage -rfakeroot -uc -b
then you will have a nginx which supports uploadprogress module in debian/nginx/usr/sbin/nginx
Disable your currently running nginx, configure your site to enable upload progress as described here: http://wiki.nginx.org/NginxHttpUploadProgressModule#Configuration_Example
Then I renamed function filefield_requirements() in filefield.install to filefield_DONOTRUN() so that Drupal would enable the progress bar.
After all this it's still not working and I'm currently not interested in digging any deeper, but this should be a head start for people who are interested in getting this working.
If we could get this working, it would be great, as NGINX seems to be the flavour of the day for low resource VPS, which I feel most people will be running in the future since Dreamhost and the like really stink.
Cheers,
Jordan
Comment #8
j0rd CreditAttribution: j0rd commentedI don't know much about how this all works so please bare with me.
Upon further inspection and configuration, it appears nginx's upload progress module provides a json entry point and returns data in it's own format when queried with the upload ID. This JSON is incompatible with the JSON filefield returns from the filefield_progress function, so the only way to currently get it working with NGINX and their upload progress module would be to modify the JS which is polling for progress to identify NGINX and deal with the format it returns.
PECL uploadprogress and NGINX do not get along. While you can install the PECL module and enable it in PHP, the uploadprogress_get_info($key) simply returns no data (NULL). PECL's uploadprogress module or NGINX would have to get updated to work properly together.
I believe that the former solution of getting uploadprogress to work with nginx would be the best one. The NGINX uploadprogress module requires you to recompile nginx, it's only recommended for older versions of nginx and it requires some server side configuration as well. Installing PECL's upload progress is much easier.
Just figure I'd post my findings to assist others later. I've given up.
Comment #9
j0rd CreditAttribution: j0rd commentedI was installing APC on my NGINX server today for speed and I remembered that was the other option for the filefield progress bar, so I figure I'd give it another stab.
Step#1: Enable APC module in PHP.
Step#2: in php.ini set "apc.rfc1867 = 1"
Step#3: Restart your php-fastcgi process and check phpinfo() to make sure you APC is running with rfc1867 on.
Now on an upload with filefield you will get the progress bar, even though your status page will still complain that filefield will not work...and so far rightfully so...because it's still not working correctly.
I believe this to be a bug with APC and not filefield though, as it's been reported by some users from a quick google. Here's what I've found: http://pecl.php.net/bugs/bug.php?id=13583
Same issue for me. Progress bar doesn't work, until the file is completely uploaded and then jumps to 100%. I've tried fiddling with apc.rfc1867_freq as I thought having it set to 0 by default would not allow scripts to poll, but it appears this does not resolve the issue.
APC is definitely a better option for NGINX and filefield progress bars for sure, as the uploadprogress module issues I believe will take longer to resolve. APC also speeds up PHP significantly and is the most stable off all the opcode caches as per: http://2bits.com/articles/benchmarking-drupal-with-php-op-code-caches-ap... . I installed today and did some benches and the speed improvements with APC under FASTCGI have been wonderful.
I hope this helps someone, cheers,
Jordan
Comment #10
mrfelton CreditAttribution: mrfelton commentedNot sure of the relationship between APC and the progressbar?!... but I too would love to see this working. Many of our developers use Apache in their local environment, and so the progress bar works for them. However, on our Staging and Live environments, NginX is used, and the progress bar doesn't work.
Comment #11
dicreat CreditAttribution: dicreat commentedsubscribe
Comment #12
brianmercer CreditAttribution: brianmercer commentedAPC upload progress is unworkable with nginx for the same reason that PECL uploadprogress is unworkable; because nginx buffers the entire upload before passing it to the php backend.
The APC upload bar works, but not for the period of time while the file is uploading to the nginx buffer on the server. It only works for the period of time while the file is being copied from the nginx buffer on the server to the fastcgi backend. With a really large file (or a slow enough server) you can see the progress bar go from 0 to 100 in steps, but only after you've finished uploading the file to the server.
According to this thread, http://forum.nginx.org/read.php?2,5506,5507#msg-5507, there's no way to turn off that behavior, though there might be at some point.
Some kind of ajax calls to the nginx uploadprogress module looks like the best answer atm.
Comment #13
crea CreditAttribution: crea commentedSubscribing.
Comment #14
crea CreditAttribution: crea commentedJust found this one:
http://github.com/drogus/jquery-upload-progress
Looks promising! Can anyone with Jquery skills provide us a patch or a module please ? :)
Comment #15
MTecknology CreditAttribution: MTecknology commentedaptitude install php5-dev php-pear php-cgi
compile nginx with the upload_progress module (http://wiki.nginx.org/NginxHttpUploadProgressModule)
setup the configs as described
pecl install uploadprogress
This should be everything that's needed. The link above describes how the uploads are tracked.
I think the filefield module should be able to deal with uploads being tracked this way.
Comment #16
quicksketchI'll add this if a patch can be provided, but since I don't use nginx, I won't be writing this myself.
Comment #17
errand CreditAttribution: errand commentedFantastic! It still not working!
Comment #18
brianmercer CreditAttribution: brianmercer commentedquicksketch,
I'm not a programmer, but I've been looking at this nginx upload problem for a couple hours. If you have a moment, please offer some advice on the best way to do this. The nginx upload progress module doesn't use a hidden form field to set the key. It requires a query string be added to the post url and the progress call.
So after reading up my feeling is to use hook_form_alter to change these paths like so:
and
Except that 'path' urlencodes the extra stuff and the reserved characters ? and = get lost. What would be the best way to add the query string to those paths?
nginx can conform the path and json output to match one of the other implementations so this might be all we need to make this work.
Thanks much.
Comment #19
quicksketchUnfortunately it doesn't look like #ahah['path'] supports query strings: http://api.lullabot.com/form_expand_ahah. I'm not sure #ahah would is able to work with that approach.
Comment #20
brianmercer CreditAttribution: brianmercer commentedThanks. Will have to try something else.
Comment #21
MTecknology CreditAttribution: MTecknology commentedThis kinda blows. This is the only thing that using nginx has caused to be a pain for me. The rest has just been a very simple setup which proved to be much easier than apache. I'm still lost here.
Comment #22
j0rd CreditAttribution: j0rd commentedCould we not do a simple hack to "Get" the path from the arguments and put it into the #ahah['path']?
Could we use an nginx rewrite config (think mod_rewrite) to hack it in there on the server config side?
Could we create a simple patch to the ahah code for nginx users and later hope to get it committed. From me playing with the ahah code base, I've come to loathe it anyways.
Personally I'd like to see this hacked in by any means necessary. Any ahah code extensions would hopefully get adopted into core. Otherwise I'm happy to apply a patch for this funcitonality as I'm sure most other nginx users would be (we're asking for additional hassles when we choose nginx anyways).
Nginx, FastCGI + APC is all I use these days. It's been gaining more and more support because of the low memory environments in VPS solutions. I say, hack it in and then we'll get it into core after it's tested well.
Comment #23
smoothify CreditAttribution: smoothify commentedBrian,
I'm pretty new to the world of nginx, but i'm trying to set up the upload progress to work with it.
I wondered is there a way to rewrite urls in nginx that could get around this issue?
e.g. something like
$element['filefield_upload']['#ahah']['progress']['path'] = 'filefield/progress/x-progress-id:' . $upload_progress_key;
then in the nginx.conf a rule is created that redirects that url into the correct url with query?
Comment #24
quicksketchOr maybe the hidden field for upload IDs can be re-used. Both APC and the uploadprogress PECL extension work by reading a $_POST variable. If "X-Progress-ID" can be read from POST instead of GET, then nginx can use the same approach used by APC and uploadprogress.
Comment #25
brianmercer CreditAttribution: brianmercer commentedHmmm, this sounds like it should work. Gonna have to try it. The /progress rewrite should be easy, but the POST would require a bit of module hacking to append the key since I've never heard of nginx being able to break out POST variables.
Comment #26
smoothify CreditAttribution: smoothify commentedAs far as I can see the nginx upload progress module doesn't accept a POST variable.
The documentation states:
quicksketch, would setting a header on the upload be viable, if so where would be the best place to do it?
Comment #27
quicksketchIt's possible to set an HTTP header with jQuery, but as FileField uses the generic ahah.js file and this functionality is not implemented there, it doesn't seem to be any better an option than using the GET string. Working with what we have, I think the rewrite solution is the best we have to go on. Note that it would be a good idea to investigate Drupal 7's handling of the progress bar, since this same issue will likely exist there.
Comment #28
smoothify CreditAttribution: smoothify commentedI've been working on this and I now have a functioning upload progress bar on nginx using the redirect method.
It looks like this will have to go in a contrib module due to the complexity and brittleness of it and the fact that nginx upload progress can be configured in different ways.
However at the moment I have just hacked the filefield module - I will try to write this up better tomorrow, but here is a summary:
the rewrite in nginx.conf
There are then changes in filefield.module and filefield_widget.inc which i've attached a patch for as proof of concept.
Nginx requires you issue an http request to get the upload status, however while configurable in its output, it doesn't provide a percentage so I had to add a function call into filefield_progress that issues a drupal_http_request.
Comment #29
smoothify CreditAttribution: smoothify commentedI've now created a custom module which can be found in a github repository (along with a simple README)
http://github.com/smoothify/filefield_nginx_progress
Currently there is no configuration options for it, it just adds progress bars to all filefields (providing your nginx is configured correctly)
Comment #30
sunThe CVS applications team needs to know whether this really deserves its own module, due to the application in #832326: smoothify [smoothify]
Comment #31
brianmercer CreditAttribution: brianmercer commentedNice work, smoothify.
I installed the module today. It works perfectly and definitely fills a need for many of us. As stated at #21, this is perhaps the only common functionality that was missing under nginx.
It's up to quicksketch if he wants to integrate this with filefield, but if not then I vote in support of a module.
Comment #32
brianmercer CreditAttribution: brianmercer commentedSmoothify, please post something to http://groups.drupal.org/nginx when you can, thanks.
Comment #33
smoothify CreditAttribution: smoothify commentedBrian, thanks for the feedback and the suggestion - I have now created a post on g.d.o:
http://groups.drupal.org/node/76128
Comment #34
crea CreditAttribution: crea commentedThank you for your work Smoothify. Nice to see we have a progress here.
Comment #35
MTecknology CreditAttribution: MTecknology commentedYAY!
Considering how things have been going already - I think a separate module makes more sense. It doesn't seem as if the filefield maintainers have any interest in maintaining this piece even if they were to integrate the work.
Comment #36
quicksketchThat's true, I'm not going to set up a third development environment just for nginx (I've already got 2, on for pgsql and mysql, plus toggling between PHP 5.2 and 4.3). So perhaps separate would be the way to go.
Comment #37
j0rd CreditAttribution: j0rd commentedMuch appreciated Smoothify. I'll install and test this sometime in the future.
Thanks to quicksketch as well for providing insight to assist on moving this forward.
I'd recommend adding this Drupal module into drupal.org when you get a chance as well, so people will find it and once that's done, perhaps get it added into the "recommended modules" on the filefield module page, so people who need it will find it easily.
http://drupal.org/project/filefield
Here are some articles on syncing GIT code with Drupal.org CVS repository:
http://groups.drupal.org/node/26022
www.mig5.net/content/developing-git-and-working-drupalorgs-cvs-central-git-server
--
Drupal Commerce Development and Themes
Comment #38
smoothify CreditAttribution: smoothify commentedThanks for the feedback from those of you that have tested it or will be testing it.
I would like to add it to drupal.org if possible, but as of now I don't have cvs access. I do have an application in but I think its in the queue.
Comment #39
sunCould the people who are interested in this functionality, and ideally perhaps some of those more familiar with FileField module, review the CVS application in #832326: smoothify [smoothify] ?
Thanks in advance.
Comment #40
smoothify CreditAttribution: smoothify commentedThe filefield_nginx_progress module has now been added to drupal.org
http://drupal.org/project/filefield_nginx_progress
Comment #41
MTecknology CreditAttribution: MTecknology commentedMarking as closed because a module now exists to fix this. Any issues in the module should be placed in the module issues queue.
Comment #42
MTecknology CreditAttribution: MTecknology commentedoops - shouldn't have closed it
Comment #44
j0rd CreditAttribution: j0rd commentedIt would be nice to get this added onto the filefield module page, so that those who need this can find it.