I, since December first 2010, I don't get any more a Full DB export, but only Partial.
I see that all my sites with DB over 200MB have partial DB dump.
Also, when Backup and Migrate compress dump file it got a file .gz with 0KB in volume.

I don't know what's happen.

Maybe Backup&Migrate have a limit for large DB?
If I use mysqldump I don't have this problem.

Three DBs of mine have 206.16MB, 375.03MB, 219.71MB.

What is the problem?
Have DBs lines with wrong mysql syntax, maybe?
is it a problem of memory limit, maybe?
is it a proble of Backup&Migrate, maybe?

Please, can you help me?

Comments

jscm’s picture

jscm commented

I've got the same problem also for DB of 182.14MB.
I don't know what's happen

ronan’s picture

ronan commented
Status: Active » Postponed (maintainer needs more info)

There's no hard backup limit in backup and migrate, but a 200mb+ db is probably pushing the limits of what your server can handle. The obvious candidates are exceeding your memory limit or time limit. B&M tries to mitigate both issues by not storing the entire db in memory during most backups (backing up to some destinations such as email or S3 do require that thoug) and by checking timeouts periodically during a backup to fail gracefully.

I have successfully backed up databases as large as yours but of course your mileage may vary.

Try increasing your php max_execution_time and memory_limit to see if that helps, and back up to a server directory rather than downloading or saving to a remote destination and see if that helps.

jscm’s picture

jscm commented

Hi,
I have 10 websites with large DBs. I think that the problem is that every sites runs backup&migrate at the same time, parallel. These Processes use too much resources.

I have created a bash script to export DB dumps individually. Using this scripts I solved the problem.
I think that Backup&Migrate should have a form to set a time to start backup.

Backup&Migrate how manage large dbs (over 500MB, over 1GB)?

giorgio79’s picture

giorgio79 commented

Not sure how BM does it currently, but could it perhaps do the backup per table? Large tables would not be a problem in that case.

jscm’s picture

jscm commented

I done a bash script that:

1- exports schema
2- backup x tables,
3- creates restore file (used with mysql to import),
4- after compress all .sql files using gzip
5- also remove backup files older x days.

I think that Backups & Migrate should manage Large DBs and add a form for scheduling by time to start backup.

jscm’s picture

jscm commented
Status: Postponed (maintainer needs more info) » Active

...

kriskhaira’s picture

kriskhaira commented

jscm, have you tried setting your Drupal cron jobs to start at different times? For example:

15      *       *       *       *       curl --silent --compressed http://www.site1.com/cron.php  > /dev/null 2>&1
20      *       *       *       *       curl --silent --compressed http://www.site2.com/cron.php  > /dev/null 2>&1
25      *       *       *       *       curl --silent --compressed http://www.site3.com/cron.php  > /dev/null 2>&1
30      *       *       *       *       curl --silent --compressed http://www.site4.com/cron.php  > /dev/null 2>&1
35      *       *       *       *       curl --silent --compressed http://www.site5.com/cron.php  > /dev/null 2>&1
jscm’s picture

jscm commented

yes, I do.
I have a script in cron that run drupal cron job for every websites sequentially..
and I have a backup scripts in cron that backups all dbs sequenzially.
all works properly.

ronan’s picture

ronan commented
Status: Active » Closed (works as designed)

Closing this as designed because B&M (nor any all-php script) will never be the best solution for very large db's especially when run through a web server.

dqd’s picture

dqd
Location London | N.Y.C | Paris | Hamburg | Berlin
commented

sry ronan, but I need to inform here for followers and readers to get not confused, that a DB with 150MB+ size is not large with 10 fields and revisions activated. Even if the issue is 4 years old. Post 100 nodes and use revision management and this size is easily reached.