Massive memory usage on backup to download with ver 7.x-3.5

Am hitting this problem myself. I bumped the backup_migrate_backup_memory_limit variable to 1gb, and then increased it 0.1gb each time until it finally worked without hitting an error, and it only worked when I got to 2gb.

It also didn't matter whether I set the file to be compressed or not - I started with the file being compressed with 'zip', then changed it to 'uncompressed', but it failed at 1.9gb in both scenarios while succeeded at 2.0gb.

Gah.

This site's backup resulted in a file which was 750mb. One of the tables has 150,000 rows, and after I reimport the backup into a fresh database my db tool (Navicat) says this one table uses 826mb.

The table configuration is not set to exclude any tables. However, it is set to exclude "accesslog", all "cache" tables, tables for Search API DB and core's search module, and "sessions".

I wonder if maybe something in how it handles compiling the data from the database is using up an endless amount of memory and not clearing the memory?

The database code works from the following:

    $data = $this->query("SELECT * FROM `" . $table['name'] . "`", array(), array('fetch' => PDO::FETCH_ASSOC));

This means it builds a large associated array of all of the data from the table, so if the table has hundreds of thousands, or even millions of records, it'll load the entire thing into memory as an array, and then process each row.

I wonder if we could do a ranged query instead?

Lets at least improve this for the next release.

The data is obtained using DatabaseConnection::query() inside _dump_table_data_sql_to_file(). Maybe if we change that entire function so that it does a queryRange() using the $rows_per_line variable to control the range, and then jumping straight to the output.

WIP to use a queryRange().

I set the memory limit to 512mb, set backup_migrate_data_rows_per_query to 1000, and the backup completed in just under 6 minutes and the peak memory usage was 126mb. However, I don't think the memory report is accurate..

Ok, I changed the memory size check to follow the same logic that Devel uses, it now reports 130mb creating a 742mb uncompressed backup in just over 5 minutes.

Please give it a try.

This updates the destination file too, and adds new items to hook_uninstall() to remove the three variables.

Rerolled.

Committed.