From #318960: Upload Progress Bar (rfc1867 support):

PECL Uploadprogress sounds good, but there is one problem that you have not thought.

What happens when PECL Uploadprogress is installed and it is not an apache webserver?

the upload fails!
An HTTP error 404 occurred. /filefield/progress/a8989ffa6226356644edc7246c704799

I use lighttpd and the new dev of filefield breaks all filefields.

There must be a server check that the apache is used as webserver or checkbox to (de-)activate the uploadprogress.
Lighttp 1.4 has no support if pecl uploadprogress
lighttpd 1.5 has it, but it is not stable and a stable release is far away.

I do not know the support of uploadprogress on other webserver like nginx or other, but you should be some checks, an advice and a help in the readme.txt or install.txt

CommentFileSizeAuthor
#28 uploadprogress.patch2.26 KBsmoothify
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

quicksketch’s picture

I'm wondering, why would you have uploadprogress installed on a server that doesn't support it? It seems like the solution is to uninstall the uploadprogress pecl extension if it doesn't work.

quicksketch’s picture

The more I think about this, the less appropriate it seems that FileField should need to accommodate it. Should FileField really ship with a comparison list of when it should or shouldn't use a library based on server? If the server is not configured correctly, it's a server problem, not something the software should need to account for. It's just using the library that's available.

quicksketch’s picture

Status: Active » Postponed (maintainer needs more info)

Marking postponed until someone can rationalize why FileField should account for a broken library.

quicksketch’s picture

Although I still think it's silly to check not just if a library is available, but if it actually works, the PECL extension page is pretty clear about which platforms uploadprogress works on:

It is only known to work on Apache with mod_php, other SAPI implementations unfortunately still have issues.
At least PHP 5.2 is needed.

quicksketch’s picture

You can now disable the progress bar per-field by using the option added in #439694: Progress bar option.

quicksketch’s picture

Status: Postponed (maintainer needs more info) » Closed (won't fix)

Well after no responses I'm marking won't fix. A simple work-around is to just not enable the progress bar if your server has the PECL extension installed but it doesn't work.

j0rd’s picture

Title: Upload progress does not work in Lighttpd » Upload progress does not work in Lighttpd & nginx

It also does not work in nginx. Uploading a file via filefield works, just not the progress bar. Works by default and no need to disable anything. So I'm happy.

Ideally I'd like the progress bar if it's possible, but I can't find any information on how I can even attempt on getting this working. It would be nice if someone who knows more about what's required for this could look into it and see if it's possible by say enabling a module for NGINX which I'm not aware of.

Any assistance or guidance is appreciated.

NGINX has this module http://wiki.nginx.org/NginxHttpUploadProgressModule which can be patched into ubuntu's nginx by downloading and extracting nginx_uploadprogress_module into /usr/src and doing.

cd /usr/src
apt-get source nginx
cd nginx-0.6.35
vim auto/options
EDIT THE LINE NGX_ADDONS= to be NGX_ADDONS=/usr/src/nginx_uploadprogress_module
dpkg-buildpackage -rfakeroot -uc -b

then you will have a nginx which supports uploadprogress module in debian/nginx/usr/sbin/nginx

Disable your currently running nginx, configure your site to enable upload progress as described here: http://wiki.nginx.org/NginxHttpUploadProgressModule#Configuration_Example

Then I renamed function filefield_requirements() in filefield.install to filefield_DONOTRUN() so that Drupal would enable the progress bar.

After all this it's still not working and I'm currently not interested in digging any deeper, but this should be a head start for people who are interested in getting this working.

If we could get this working, it would be great, as NGINX seems to be the flavour of the day for low resource VPS, which I feel most people will be running in the future since Dreamhost and the like really stink.

Cheers,
Jordan

j0rd’s picture

I don't know much about how this all works so please bare with me.

Upon further inspection and configuration, it appears nginx's upload progress module provides a json entry point and returns data in it's own format when queried with the upload ID. This JSON is incompatible with the JSON filefield returns from the filefield_progress function, so the only way to currently get it working with NGINX and their upload progress module would be to modify the JS which is polling for progress to identify NGINX and deal with the format it returns.

PECL uploadprogress and NGINX do not get along. While you can install the PECL module and enable it in PHP, the uploadprogress_get_info($key) simply returns no data (NULL). PECL's uploadprogress module or NGINX would have to get updated to work properly together.

I believe that the former solution of getting uploadprogress to work with nginx would be the best one. The NGINX uploadprogress module requires you to recompile nginx, it's only recommended for older versions of nginx and it requires some server side configuration as well. Installing PECL's upload progress is much easier.

Just figure I'd post my findings to assist others later. I've given up.

j0rd’s picture

I was installing APC on my NGINX server today for speed and I remembered that was the other option for the filefield progress bar, so I figure I'd give it another stab.

Step#1: Enable APC module in PHP.
Step#2: in php.ini set "apc.rfc1867 = 1"
Step#3: Restart your php-fastcgi process and check phpinfo() to make sure you APC is running with rfc1867 on.

Now on an upload with filefield you will get the progress bar, even though your status page will still complain that filefield will not work...and so far rightfully so...because it's still not working correctly.

I believe this to be a bug with APC and not filefield though, as it's been reported by some users from a quick google. Here's what I've found: http://pecl.php.net/bugs/bug.php?id=13583

Same issue for me. Progress bar doesn't work, until the file is completely uploaded and then jumps to 100%. I've tried fiddling with apc.rfc1867_freq as I thought having it set to 0 by default would not allow scripts to poll, but it appears this does not resolve the issue.

APC is definitely a better option for NGINX and filefield progress bars for sure, as the uploadprogress module issues I believe will take longer to resolve. APC also speeds up PHP significantly and is the most stable off all the opcode caches as per: http://2bits.com/articles/benchmarking-drupal-with-php-op-code-caches-ap... . I installed today and did some benches and the speed improvements with APC under FASTCGI have been wonderful.

I hope this helps someone, cheers,
Jordan

mrfelton’s picture

Not sure of the relationship between APC and the progressbar?!... but I too would love to see this working. Many of our developers use Apache in their local environment, and so the progress bar works for them. However, on our Staging and Live environments, NginX is used, and the progress bar doesn't work.

dicreat’s picture

subscribe

brianmercer’s picture

APC upload progress is unworkable with nginx for the same reason that PECL uploadprogress is unworkable; because nginx buffers the entire upload before passing it to the php backend.

The APC upload bar works, but not for the period of time while the file is uploading to the nginx buffer on the server. It only works for the period of time while the file is being copied from the nginx buffer on the server to the fastcgi backend. With a really large file (or a slow enough server) you can see the progress bar go from 0 to 100 in steps, but only after you've finished uploading the file to the server.

According to this thread, http://forum.nginx.org/read.php?2,5506,5507#msg-5507, there's no way to turn off that behavior, though there might be at some point.

Some kind of ajax calls to the nginx uploadprogress module looks like the best answer atm.

crea’s picture

Subscribing.

crea’s picture

Just found this one:
http://github.com/drogus/jquery-upload-progress
Looks promising! Can anyone with Jquery skills provide us a patch or a module please ? :)

MTecknology’s picture

Status: Closed (won't fix) » Active

aptitude install php5-dev php-pear php-cgi
compile nginx with the upload_progress module (http://wiki.nginx.org/NginxHttpUploadProgressModule)
setup the configs as described
pecl install uploadprogress

This should be everything that's needed. The link above describes how the uploads are tracked.

I think the filefield module should be able to deal with uploads being tracked this way.

quicksketch’s picture

Category: bug » support
Status: Active » Postponed

I'll add this if a patch can be provided, but since I don't use nginx, I won't be writing this myself.

errand’s picture

Fantastic! It still not working!

brianmercer’s picture

quicksketch,

I'm not a programmer, but I've been looking at this nginx upload problem for a couple hours. If you have a moment, please offer some advice on the best way to do this. The nginx upload progress module doesn't use a hidden form field to set the key. It requires a query string be added to the post url and the progress call.

So after reading up my feeling is to use hook_form_alter to change these paths like so:

$element['filefield_upload']['#ahah']['path'] = 'filefield/ahah/'.   $element['#type_name'] .'/'. $element['#field_name'] .'/'. $element['#delta'] .  '?X-Progress-ID=' . $upload_progress_key,

and

$element['filefield_upload']['#ahah']['progress']['path'] = 'filefield/progress/' . '?X-Progress-ID=' . $upload_progress_key

Except that 'path' urlencodes the extra stuff and the reserved characters ? and = get lost. What would be the best way to add the query string to those paths?

nginx can conform the path and json output to match one of the other implementations so this might be all we need to make this work.

Thanks much.

quicksketch’s picture

Unfortunately it doesn't look like #ahah['path'] supports query strings: http://api.lullabot.com/form_expand_ahah. I'm not sure #ahah would is able to work with that approach.

brianmercer’s picture

Thanks. Will have to try something else.

MTecknology’s picture

This kinda blows. This is the only thing that using nginx has caused to be a pain for me. The rest has just been a very simple setup which proved to be much easier than apache. I'm still lost here.

j0rd’s picture

Could we not do a simple hack to "Get" the path from the arguments and put it into the #ahah['path']?

Could we use an nginx rewrite config (think mod_rewrite) to hack it in there on the server config side?

Could we create a simple patch to the ahah code for nginx users and later hope to get it committed. From me playing with the ahah code base, I've come to loathe it anyways.

Personally I'd like to see this hacked in by any means necessary. Any ahah code extensions would hopefully get adopted into core. Otherwise I'm happy to apply a patch for this funcitonality as I'm sure most other nginx users would be (we're asking for additional hassles when we choose nginx anyways).

Nginx, FastCGI + APC is all I use these days. It's been gaining more and more support because of the low memory environments in VPS solutions. I say, hack it in and then we'll get it into core after it's tested well.

smoothify’s picture

Brian,

I'm pretty new to the world of nginx, but i'm trying to set up the upload progress to work with it.

I wondered is there a way to rewrite urls in nginx that could get around this issue?

e.g. something like
$element['filefield_upload']['#ahah']['progress']['path'] = 'filefield/progress/x-progress-id:' . $upload_progress_key;

then in the nginx.conf a rule is created that redirects that url into the correct url with query?

quicksketch’s picture

Or maybe the hidden field for upload IDs can be re-used. Both APC and the uploadprogress PECL extension work by reading a $_POST variable. If "X-Progress-ID" can be read from POST instead of GET, then nginx can use the same approach used by APC and uploadprogress.

brianmercer’s picture

Hmmm, this sounds like it should work. Gonna have to try it. The /progress rewrite should be easy, but the POST would require a bit of module hacking to append the key since I've never heard of nginx being able to break out POST variables.

smoothify’s picture

As far as I can see the nginx upload progress module doesn't accept a POST variable.

The documentation states:

This identifier can be transmitted either as a GET argument or as an HTTP header whose name is X-Progress-ID.

quicksketch, would setting a header on the upload be viable, if so where would be the best place to do it?

quicksketch’s picture

It's possible to set an HTTP header with jQuery, but as FileField uses the generic ahah.js file and this functionality is not implemented there, it doesn't seem to be any better an option than using the GET string. Working with what we have, I think the rewrite solution is the best we have to go on. Note that it would be a good idea to investigate Drupal 7's handling of the progress bar, since this same issue will likely exist there.

smoothify’s picture

FileSize
2.26 KB

I've been working on this and I now have a functioning upload progress bar on nginx using the redirect method.

It looks like this will have to go in a contrib module due to the complexity and brittleness of it and the fact that nginx upload progress can be configured in different ways.

However at the moment I have just hacked the filefield module - I will try to write this up better tomorrow, but here is a summary:

the rewrite in nginx.conf

    location ~ (.*)/x-progress-id:(.*)$ {
       rewrite ^/(.*)/x-progress-id:(\w*)  /$1?X-Progress-ID=$2;
    }

There are then changes in filefield.module and filefield_widget.inc which i've attached a patch for as proof of concept.

Nginx requires you issue an http request to get the upload status, however while configurable in its output, it doesn't provide a percentage so I had to add a function call into filefield_progress that issues a drupal_http_request.

smoothify’s picture

I've now created a custom module which can be found in a github repository (along with a simple README)

http://github.com/smoothify/filefield_nginx_progress

Currently there is no configuration options for it, it just adds progress bars to all filefields (providing your nginx is configured correctly)

sun’s picture

The CVS applications team needs to know whether this really deserves its own module, due to the application in #832326: smoothify [smoothify]

brianmercer’s picture

Nice work, smoothify.

I installed the module today. It works perfectly and definitely fills a need for many of us. As stated at #21, this is perhaps the only common functionality that was missing under nginx.

It's up to quicksketch if he wants to integrate this with filefield, but if not then I vote in support of a module.

brianmercer’s picture

Smoothify, please post something to http://groups.drupal.org/nginx when you can, thanks.

smoothify’s picture

Brian, thanks for the feedback and the suggestion - I have now created a post on g.d.o:

http://groups.drupal.org/node/76128

crea’s picture

Thank you for your work Smoothify. Nice to see we have a progress here.

MTecknology’s picture

YAY!

Considering how things have been going already - I think a separate module makes more sense. It doesn't seem as if the filefield maintainers have any interest in maintaining this piece even if they were to integrate the work.

quicksketch’s picture

It doesn't seem as if the filefield maintainers have any interest in maintaining this piece even if they were to integrate the work.

That's true, I'm not going to set up a third development environment just for nginx (I've already got 2, on for pgsql and mysql, plus toggling between PHP 5.2 and 4.3). So perhaps separate would be the way to go.

j0rd’s picture

Much appreciated Smoothify. I'll install and test this sometime in the future.

Thanks to quicksketch as well for providing insight to assist on moving this forward.

I'd recommend adding this Drupal module into drupal.org when you get a chance as well, so people will find it and once that's done, perhaps get it added into the "recommended modules" on the filefield module page, so people who need it will find it easily.

http://drupal.org/project/filefield

Here are some articles on syncing GIT code with Drupal.org CVS repository:
http://groups.drupal.org/node/26022
www.mig5.net/content/developing-git-and-working-drupalorgs-cvs-central-git-server

--
Drupal Commerce Development and Themes

smoothify’s picture

Thanks for the feedback from those of you that have tested it or will be testing it.

I would like to add it to drupal.org if possible, but as of now I don't have cvs access. I do have an application in but I think its in the queue.

sun’s picture

Could the people who are interested in this functionality, and ideally perhaps some of those more familiar with FileField module, review the CVS application in #832326: smoothify [smoothify] ?

Thanks in advance.

smoothify’s picture

The filefield_nginx_progress module has now been added to drupal.org

http://drupal.org/project/filefield_nginx_progress

MTecknology’s picture

Status: Postponed » Closed (fixed)

Marking as closed because a module now exists to fix this. Any issues in the module should be placed in the module issues queue.

MTecknology’s picture

Status: Closed (fixed) » Fixed

oops - shouldn't have closed it

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

j0rd’s picture

It would be nice to get this added onto the filefield module page, so that those who need this can find it.