Browser decompresses database backup while keeping the .gz suffix

If i'm choise ani compressing type: Happend error '502 Bad Gateway'

My experience is that when you use *download* on a backup, it comes down with mysql.gz suffix but not gzipped. On the other hand, my backups that go to AWS bucket are in fact gzipped.

Sounds like maybe something is set up incorrectly on the server? I haven't been able to reproduce this, in fact I'm seeing the files being compressed correctly on both Linux and Windows server as expected.

Maybe we could add some extra logging around the compression functionality, maybe use hook_requirements() to identify requirements and whether things should work correctly

I can confirm this happens in Chrome version 72 under Windows 10 Home.

Commenting out the following code makes the problem go away:

// In some circumstances, web-servers will double compress gzipped files.
// This may help aleviate that issue by disabling mod-deflate.
if ($this->mimetype() == 'application/x-gzip') {
  $headers[] = array('key' => 'Content-Encoding', 'value' => 'gzip');
}

So does changing 'Content-Encoding' to 'Transfer-Encoding'.

I am not sure that this is a bug in the browser. From RFC 3721:

The "Content-Encoding" header field indicates what content codings have been applied to the representation, beyond those inherent in the media type, and thus what decoding mechanisms have to be applied in order to obtain data in the media type referenced by the Content-Type header field. Content-Encoding is primarily used to allow a representation's data to be compressed without losing the identity of its underlying media type.

In other words, Chrome could be argued to be doing what it is supposed to do.

The code above was introduced in #1564408 Gzip backups are compressed twice when downloaded. Disabling it might bring back the old problem.

Having had to support answers to this questions loads and loads of time for ddev I took a quick wander through and I think this patch fixes the issue. AFAICT this happens in every single environment with default settings. My first patch is oh-so-long.

The bug is easy to test, in my case I used ddev, which by default has nginx/php-fpm and a very vanilla config.

* Enable backup_migrate, don't even bother to configure anything.
* Visit backup_migrate
* Click the "Download" button.
* Inspect the file that results. `file `

We want to see `d7git.ddev.local-2019-03-12T15-34-26.mysql.gz: gzip compressed data, max compression, from Unix, original size 410736`

We do NOT want to see `d7git.ddev.local-2019-03-12T15-23-12.mysql.gz`

The attached 2-character patch seems to fix this problem.

Hmm, that's a fix, but it's not the fix, it's kind of accidental. I note that the incoming mimetype is application/x-gzip, so the change in #9 just causes us to skip that stanza

    if ($this->mimetype() == 'application/gzip') {
      $headers[] = array('key' => 'Content-Encoding', 'value' => 'gzip');
    }

doesn't get executed, and that solves the problem.

AFAICT, that whole stanza is the source of the problem. If I just remove the stanza, as with this patch, it works with nginx-fpm, apache-fpm, apache-cgi. If I check the original, it fails on all three.

But does your fix cause some web servers to "double compress gzipped files"? Because that is what the code you removed was introduced to prevent.

I am changing the title of this issue to more closely reflect the problem the reporter is having.

The original title, "compression still not happening on database backup", is how users might first perceive this problem.

I tested this for double-compression on nginx+fpm, apache+fpm, and apache+cgi and saw none of that. Those situations cover almost all the world's webservers.

I do note that either with or without this patch, Safari decompresses (and sort-of-apporopriately) removes the .gz suffix. So with or without the patch, with gzip compression configured, Safari presents a .mysql file.

Edit: Also tried with Firefox on macOS, and it always offers to decompress, but doesn't try to do it automatically. If you choose the default option, it downloads the (one-time) compressed archive, with or without this patch.

Bottom line is:
* Most users are using Chrome.
* Lots of unnecessary and frustrating support occurs as a result of this.
* No regression has been discovered in testing.
* D7 is old, not long for this world, and the D8 version doesn't have this problem.

So we should try to solve the problem for the majority of users.

Actually, the more I think about it, the idea in this code is not what it ever should have been. All download types should be set to mime-type `application/octet-stream`. None of the compressed types should be expecting browsers to decompress them. Changing to `application/octet-stream` make the download land predictably as a .mysql.gz file using both Firefox and Google Chrome. Safari still gunzips it and saves it as a .mysql file. I'll do a patch suggesting this.

This patch changes all of the compression mimetypes to application/octet-stream, which I think is correct in all cases. There's no reason I know of that the web client should take any action on the downloaded file (like uncompressing it).

I have looked at this patch (though not in great detail, see below) and found that it does what it sets out to do.

Quick Backup of Default Database to Download using Default Settings:

Server: Apache 2.4 on Windows 10 Home.

--- Without patch #17: ---

Firefox 65 / Win 10 Home: UA displays a dialogue with the options Open With [application] and Save File. Choosing Open, the archive is opened inside my application (in my case: 7zip).

Chrome 72 / Win 10 Home: UA shows a bar with a filename ending in .gz. When I click on this, my associated application (in my case 7zip) produces an error message "Cannot open file FILENAME.mysql.gz as archive" (translated by from Dutch).

Edge 42 / Win 10 Home: UA displays a dialogue with the question "What do you want to with FILENAME.mysql.gz" and the options Open, Save and Cancel. Clicking Open produces the aforementioned error message.

Fiddler 4 intercepts the response header as (truncated for brevity):

X-Content-Type-Options: nosniff
Content-Disposition: attachment; filename="Drupal7testsite-2019-03-18T12-43-20.mysql.gz"
Content-Encoding: gzip
Content-Type: application/x-gzip

--- After application of patch #17: ---

Firefox 65 / Win 10 Home: UA displays a dialogue with the options Open With [application] and Save File. Choosing Open, the archive is opened inside my application (in my case: 7zip).

Chrome 72 / Win 10 Home: UA shows a bar with a filename ending in .mysql.gz. When I click on this, the archive is opened inside my application (in my case: 7zip).

Edge 42 / Win 10 Home: UA displays a dialogue with the question "What do you want to with FILENAME.mysql.gz" and the options Open, Save and Cancel. Clicking Open displays the archive in my associated application.

Fiddler 4 intercepts the response header as (truncated for brevity):

X-Content-Type-Options: nosniff
Content-Disposition: attachment; filename="Drupal7testsite-2019-03-18T13-08-11.mysql.gz"
Content-Type: application/octet-stream

--- ---

What worries me a little is that #17 changes the content-type for all destinations.

Would it perhaps be better to change includes/destinations-browsers.inc instead? I.e. add the line

$file->type['filemime'] = 'application/octet-stream';

to its save_file() implementation?

I am having trouble imagining a destination that does anything meaningful with the mime-type, so I imagine your solution will probably work in most if not all cases anyway (and indeed the e-mail destination already completely ignores the backup_file object's mime-type settings and hard codes it as 'octet-stream'), but doing it in the right class seems cleaner.

Successful test of patch #17.
It produced a .mysql.gz file that was compressed. It uncompressed smoothly in Ddev with $ddev import-db --src=~/Downloads/D7PatchTesting-2019-03-20T07-46-46.mysql.gz

Drupal core 7.65-dev. Backup and Migrate 7.x-3.6.
Chrome Version 73.0.3683.75

@BrankoC I fussed a little at the time about setting it to "application/octet-stream" for everything. But the reality is that all download types that I know of from backup_migrate really *should* be treated as octet-stream. Can't figure out why they wouldn't be.

I'm pretty sure this patch is the right thing to fix a long-term problem with backup_migrate, sorry it hasn't gone yet.

I don't disagree as far as downloads are concerned, but destinations encompass much more than downloads. Currently B&M supports the following destinations: browsers (i.e. downloads), databases, e-mail attachments, files saved in a site's files directory, FTP, Nodesquirrel and Amazon S3. Other modules (including custom modules) may add more.

The official MIME type for ZIP files, for example, is application/zip, so changing that to application/octet-stream for all destinations seems overkill to me if all we want to do is fix the behaviour of one dissenting browser.

Yeah, you absolutely don't want application/zip as the mimetype for a zip download... as that encourages the browser to unzip it, which is just what we don't want to do in any of these cases. I'm not familiar with all the destinations, but for most, the mimetype is irrelevant. Just browser stuff AFAICT.

As you know, if you think you can improve the patch, you can post a followup patch. But I'm pretty sure this is doing what's needed for an aging module that is destined for obscurity. (D8 doesn't have this problem, didn't study why).

"D8 doesn't have this problem, didn't study why"

D8's B&M has "application/octet-stream" hard-coded in "lib/backup_migrate_core/src/Destination/BrowserDownloadDestination.php". There is a method for applying the default mime-type, which BAM 8.x-x defines as "application/zip" for ZIP files, et cetera, but that method does not appear to be used for destinations, only for sources.

(This is done through Traits - my knowledge of D8, Symphony and OOP in PHP is too limited to say much about it.)

I have yet to look into why it was done that way.

PHPMyAdmin had the same problem with double gzipping, applied the same solution, and then added browser sniffing when Google Chrome 43 came out (May 2015), which is apparently the version when the problem with decoding files started:

        // inform the server that compression has been done,
        // to avoid a double compression (for example with Apache + mod_deflate)
        $notChromeOrLessThan43 = PMA_USR_BROWSER_AGENT != 'CHROME' // see bug #4942
            || (PMA_USR_BROWSER_AGENT == 'CHROME' && PMA_USR_BROWSER_VER < 43);
        if (strpos($mimetype, 'gzip') !== false && $notChromeOrLessThan43) {
            header('Content-Encoding: gzip');
        }

PHPMyAdmin issue: https://github.com/phpmyadmin/phpmyadmin/pull/1714

Chrome issue: https://bugs.chromium.org/p/chromium/issues/detail?id=268085

It sounds like this could use a little further work to add some browser testing.

This seems to be still an issue for 5.1.x. But I couldn't test it on firefox. It might also be a problem related to #2946032: Gzip not compressing when using "Download" target, but .gz appended.

Seeing this in Firefox too.

Please do not change the version number when a) this is still very much a BAM for D7 problem en b) BAM 5 is a complete rewrite; even if it shares the same problem, it probably does so for completely different reasons.

@DamienMcKenna, do you have an approximate idea of what you would like browser testing to achieve?