Upload progress module breaks uploads on all D7 sites

So far it is really weird. We have tested already many combinations of various versions of Drupal, Nginx, Nginx configuration, Nginx Upload Progress module, and BOA itself, disabling all caching, even APC, and we have found that even when we use the same versions working without issues on some old copy of demo.aegir.cc VM image (Drupal 7.10, Nginx 1.0.11, Nginx module v0.8.3, FileField Nginx Progress 7.x-1.x-dev) - the same versions on the latest BOA install are affected exactly by the same error reported in this issue: #1159400: Uploads fail with module enabled.

The extra mystery is that the issue above was reported when there were no such issues affecting BOA installs yet.

After adding simple, more verbose debugging it says something like:

form_build_id is x-progress-id:1768824202 and _POST is form-Mio98p4av0b3yPzlBDUw4A88l-RXCbuCCu1yvo0GdY8

While on the working old install it says, correctly, something like:

form_build_id is form-v8FOHfxmUV35-QfO9kQKfqJ44D9qLc6cBzUmk1kjLFs and _POST is form-v8FOHfxmUV35-QfO9kQKfqJ44D9qLc6cBzUmk1kjLFs

Note that the function file_ajax_upload() didn't change since Drupal 7.10, and the same Drupal 7.10 platform/site works fine on the old VM and doesn't work on the latest install. It is still the same check:

function file_ajax_upload() {
  $form_parents = func_get_args();
  $form_build_id = (string) array_pop($form_parents);
  if (empty($_POST['form_build_id']) || $form_build_id != $_POST['form_build_id']) {
    // Invalid request.
    drupal_set_message(t('An unrecoverable error occurred. The uploaded file likely exceeded the maximum file size (@size) that this server supports.', array('@size' => format_size(file_upload_max_size()))), 'error');
    $commands = array();
    $commands[] = ajax_command_replace(NULL, theme('status_messages'));
    return array('#type' => 'ajax', '#commands' => $commands);
  }

Of course its default error message is misleading, as it has nothing to do with uploads limits, but with $form_build_id taken from the wrong part of the $form_parents.

This suggests only one last possible source of the problem - the PHP version and its changes affecting array_pop() function between PHP 5.2 and PHP 5.3 - also because the working VM uses PHP 5.2 for D7 sites, while all newer BOA versions force PHP 5.3 for all D7 sites.

So we have tested it again, this time trying to reproduce the problem with PHP 5.2 on the same Drupal 7.10 based site. But this didn't help. The results were the same.

Next, we have modified function file_ajax_upload() a bit, to ger rid of that false error at least:

function file_ajax_upload() {
  $form_parents = func_get_args();
  $form_build_id = (string) array_pop($form_parents);
  if (preg_match("/x-progress-id/i", $form_build_id)) {
    $form_build_id = (string) array_pop($form_parents);
  }
  if (empty($_POST['form_build_id']) || $form_build_id != $_POST['form_build_id']) {
    // Invalid request.
    drupal_set_message(t('An unrecoverable error occurred. The uploaded file likely exceeded the maximum file size (@size) that this server supports.', array('@size' => format_size(file_upload_max_size()))), 'error');
    $commands = array();
    $commands[] = ajax_command_replace(NULL, theme('status_messages'));
    return array('#type' => 'ajax', '#commands' => $commands);
  }

This helped only to get the correct value from the array, and upload worked, but without the progress indicator, so it looks like the problem is elsewhere :/

It looks like something is still wrong in handling the array of values used in the form, but it is still a bit of mystery.

It was a cascade of small changes in *our* Nginx configuration what caused this issue - which is a good news, because we were able to fix this, finally.

The commit: http://drupalcode.org/project/octopus.git/commit/e20aa03

More related commits after further testing:

http://drupalcode.org/project/octopus.git/commit/f43f65c
http://drupalcode.org/project/octopus.git/commit/9da4a97

To sum this up - it was broken because we either had ajax|ahah used in some other regex based locations, plus (this was the most important) we didn't explicitly disabled Nginx cache (the Speed Booster) for all URLs used for upload progress GET requests. Furthermore, Redis cache active for POST requests could still cause issues like pseudo-hanging uploads (and who knows what else, honestly).

How have we discovered this finally? By examining closely the results of debug error log mode enabled in Nginx and finding that it lost node tracking:

2012/10/17 22:15:45 [debug] 10126#0: *6 upload-progress: get_tracking_id
2012/10/17 22:15:45 [debug] 10126#0: *6 upload-progress: get_tracking_id no header found
2012/10/17 22:15:45 [debug] 10126#0: *6 upload-progress: get_tracking_id no header found, args found
2012/10/17 22:15:45 [debug] 10126#0: *6 upload-progress: get_tracking_id found args: X-Progress-ID=660296031
2012/10/17 22:15:45 [debug] 10126#0: *6 malloc: 00000000014F0A40:16
2012/10/17 22:15:45 [debug] 10126#0: *6 upload-progress: get_tracking_id found args: 660296031
2012/10/17 22:15:45 [debug] 10126#0: *6 trackuploads id found: 660296031
2012/10/17 22:15:45 [debug] 10126#0: *6 trackuploads hash E1FE09E8 for id: 660296031
2012/10/17 22:15:45 [debug] 10126#0: *6 upload-progress: find_node 660296031
2012/10/17 22:15:45 [debug] 10126#0: *6 upload-progress: can't find node

Just for the record, after upgrade to 7.x-2.3 I have enabled Redis for all POST requests again. Disabling it helped during debugging, but it also broke almost all forms in Drupal (which was expected, probably). Now upload progress works properly also with Redis active on POST requests, probably thanks to latest improvements introduced by @perusio - thanks!