ATTENTION: The 8.x branch is now stable. If anyone would like to open separate issues with some of the specific items mentioned below for feature requests, please feel free to do so. Otherwise, the maintainers for this module have extremely limited time, and we have failed to see a large influx of community involvement for help with programming or even testing for patches, so feature requests should be seriously considered as far as resources available to devoting to development.

Original issue text as follows:

Let's discuss:

  1. Current roadmap
  2. The shortest path to a stable release. Blockers, majors and minors
  3. Maintaining the 7.x branch, backporting opportunities and thoughts
  4. Community involvement, volunteering and co-maintaining
  5. Future versions
  6. Feature requests

What we want to achieve imminently:

What we want to achieve in the near future:

  • - Drush support for 8.x
  • - #2912459: Add an initial test (D7)
  • - MySQL autocommit support for faster restores of larger db (more about that here )
  • - Parallel (multi-threaded) compression support using either PIGZ, PLZIP or PBZIP2
  • - Any other proven and battle tested method for speeding up export and restores. ( MySQL’s LOAD DATA INFILE anyone ? )

    Feel free to share your war-stories!

What we want to achieve in the not so distant future:

We will be updating this list constantly with any noteworthy suggestions from the comments bellow.

Comments

Alex Andrascu created an issue. See original summary.

Alex Andrascu’s picture

Alex Andrascu’s picture

DamienMcKenna’s picture

Having gone through the port / rewrite process myself, here are some thoughts (aka, I'm on the internet and I have opinions ;-) ):

  • The first stable release of B&M for D8 does not need to be feature complete or on par with the D7 release. It just has to work.
  • I would recommend boiling down the functionality to a basic list of key features, make sure that it works, has some initial documentation, and release 8.x-4.0.
  • Everything else can be done as follow-ups issues.
  • On a different note, the 7.x-3.x is pretty stable but needs a few minor fixes for compatibility with PHP 7 and some other things; IMHO a new release could be done with little effort once 8.x-4.0 is out the door.
DamienMcKenna’s picture

Sorry about that.

DamienMcKenna’s picture

Also:

  • Using Plan issues can be an excellent way of helping you to keep focus on smaller goals, and help inform the community as to what's going on.
Alex Andrascu’s picture

Thank you for your input Damien and totally agree with everything you said. In fact this issue *is* a Plan issue.
Also please feel free to point out the patches for 7.x-3.x that you'd like to see in the next release.

Quick question for @ronan but I would welcome your thoughts on it too.

We have this little issue here #2826108: Use tokens to create file names for backups which we've already contributed and reviewed and I would like to see it in the next release.

However this requires a change to the vendor component (currently sitting in lib/backup_migrate_core)

Specifically it requires extending:

lib/backup_migrate_core/src/Filter/FileNamer.php

Given the original implemntation is externally hosted on github by ronan (https://github.com/backupmigrate/backup_migrate_core) how would we go about it ?

Would we fork it once and keep it as part of the module inside the lib folder or is it better keep it on github thus making it easier for @ronan to review any pull requests that may come out of any future developments ?

DamienMcKenna’s picture

I'd honestly suggest keeping B&M one self-contained module - maybe make the libraries also available via the github project, but in the interest of keeping everything easy for beginners there shouldn't be any separate dependencies, the library should be built into the module.

Remember that people who are more skilled can always use other means to do their backups, like running Drush commands. B&M's primary use (IMHO) is for people who are not experts, who just want a reliable way of making backups, and this should be the focus.

Alex Andrascu’s picture

Issue tags: +Roadmap
couturier’s picture

Can we publish a stable release now?

@DamienMcKenna in Comment #4:

The first stable release of B&M for D8 does not need to be feature complete or on par with the D7 release. It just has to work.

The currently recommended Alpha 2 works as far as I've heard back from people who have tested and restored from it, unlike Alpha 1 which produced corrupted backup files. The D8 dev released on May 9, 2017, which is now missing from the module's home page, should contain many updates that ronan marked as "fixed" in the queue through late April 2017. If someone could double-check that restoring a backup from the newest dev code works, I think we could publish a stable release now.

Documentation & Issues

@Alex Andrascu thanks for your comments at the Add a new maintainer. I have opened an issue where I will work as soon as possible to list recommended documentation updates: Update main page documentation. It has been a while since I have looked at the D7 and D8 issue queues, but I will revisit them to see if I can help there as well. Thanks for your comments regarding the future of D8 Backup and Migrate as non-Composer-dependent here.

DamienMcKenna’s picture

Thanks for the 7.2 release Alex! :)

Alex Andrascu’s picture

Issue summary: View changes
Alex Andrascu’s picture

Issue summary: View changes
Alex Andrascu’s picture

Issue summary: View changes
couturier’s picture

@Alex Andrascu, I have completed my recommendations for documentation changes here.

Several of the issues in the 8.x queue I have closed as being outdated, but a few deserve a closer look at present, as follows:

Several people would like to add a note field to backups, something that is available in 7.x

Some people are having trouble creating a backup destination other than the default

Cache table contents aren't being excluded in 8.x-4.0-alpha2, and this was a feature of 7.x

In addition, several patches are available and ready to be ported and a few other minor issues exist in the 8.x queue. Overall, the situation looks good for a stable release soon, especially since Alpha 2 seems to work for most people already. It might be safer to publish a release candidate next to get feedback before posting an official "stable" release.

Alex Andrascu’s picture

Alex Andrascu’s picture

Issue summary: View changes

  • Alex Andrascu committed 4868203 on 8.x-4.x
    Issue #2912153 by Alex Andrascu: Roadmap\n - MySQL autocommit support...

  • Alex Andrascu committed beac6f5 on 8.x-4.x
    - Issue #2912153 by Alex Andrascu: Roadmap
    
    Remove Upload and Debug...
ikit-claw’s picture

So want me to start anywhere specific? I could dig into the testing.

couturier’s picture

@ikit-claw If you have any time this weekend and haven't heard back from Alex Andrascu or DamienMcKenna, digging into the testing for 8.x-4.x is an excellent idea. DamienMcKenna marked this as a major issue, and Alex Andrascu added it as a primary goal for D8 Backup and Migrate in the Roadmap summary above.

Alex Andrascu commented today:

Just prepping up a 8.x rc and will revert to 7.x once that's out of the way. . . . Also I could really do with a list of outstanding issues ordered by priority for all branches.

DamienMcKenna responded that he would work on triaging issues over the next week. I've already updated the status of all issues in the 7.x branch where 31 issues need work and 12 issues need to be reviewed and tested by the community (which Alex Andrascu has abbreviated RTBC). The 8.x branch has about a dozen open issues in progress here.

Other ideas for helping include the following:

  1. Alex Andrascu has approved this feature request to limit UI functionality for 8.x-4.x-dev, and no one appears to be working on it yet.
  2. DamienMcKenna wants an upgrade script from 7.x-2.x to 7.x-3.2 for the 90,000 users still using v2, since we need to deprecate that branch. He says if no one gets to it he will try to find time, but he has already committed to working on a number of other issues already, so help would be appreciated. Read more about that here.
  3. Alex Andrascu has specifically requested RTBC (review and testing by the community) on 3 finished patches in the 7.x-3.x branch here: Backup to NodeSquirrel, Add support for the Swiftmailer module, and a directory configuration issue.

Thanks for your offer of help, @ikit-claw. I wanted to provide a list of the most current options so you could select something that is a good fit for the amount of time that you have, but if DamienMcKenna and Alex Andrascu respond they may have something they would ask you to do first. Keep in mind that Alex is based in London and Damien is in the United States, so the time frames they are available are slightly different.

Alex Andrascu’s picture

Daniel,

Either working on testing, implementing support for NodeSquirel, Drive, Dropbox or Rackspace, SFTP/SSH would be a massive help. Backup encryption would be nice to have too.

Though I think unit testing is top of the list now.

fgm’s picture

FWIW, for crontabs, until drush bam-backup reappears in a future version, a simple drush ev "backup_migrate_cron();" does the job too.

DamienMcKenna’s picture

Should we close this and just focus on separate issues?

couturier’s picture

Status: Active » Closed (outdated)

@DamienMcKenna, yes and a huge thanks to everyone who has gotten the 7.x branch up to speed and a stable release out for 8.x! This module's stability has been a huge issue for many of us non-Composer users needing to upgrade sites to D8, and it's impossible to say thanks enough for all your untold hours of work during the last 9 months or so!

gister’s picture

Is this 8.x release stable? If yes, could you please remove "Stable 8.x version" from under "What we want to achieve imminently" and maybe add a text that explicitly states the stability of the module?

gister’s picture

Status: Closed (outdated) » Active

trying to set to ACITIVE so that I get an answer to my previous question; apologies if not the appropriate procedure

couturier’s picture

Issue summary: View changes

We do have a few feature requests still left in this issue, but I really think we need to open new issues for each one specifically if we still have an interest in doing so. I've left the issue open for now and revised the issue summary with a new opening paragraph, as follows:

ATTENTION: The 8.x branch is now stable. If anyone would like to open separate issues with some of the specific items mentioned below for feature requests, please feel free to do so. Otherwise, the maintainers for this module have extremely limited time, and we have failed to see a large influx of community involvement for help with programming or even testing for patches, so feature requests should be seriously considered as far as resources available to devoting to development.

DamienMcKenna’s picture

Status: Active » Fixed

a) It is always up to the site maintainer to test any module, theme, etc they want to use to make sure it works for them.
b) There definitely are lots of features in 7.x-3.x that aren't in 8.x-4.x. This is on purpose. The D7 module's functionality was built up over years the D8 module will improve over time.
c) If you find a bug, and there isn't an issue for it already, please report it and we (community) can try to fix it.
d) Ditto for features.

Closing the issue for the above reasons (-:

Alex Andrascu’s picture

Issue summary: View changes
Alex Andrascu’s picture

There we go. I hope everyone's happy now (-:

gister’s picture

I am! Thank you all!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

bas123’s picture

I am trying to determine if this is a stable solution for me but the module page refers to this discussion for D8 and I am unclear based on the conversations above!

I have an up to date D8.5.5 distribution running on a shared server in a subdomain and feel it is ready to migrate to the production folder.

My hosting company describes my support requests as "over and above" and will not advise.

I have maintained drush archive-dumps along the way and was planning to use drush to restore to the new location along with a new database ie:
drush archive-restore ./example.tar.gz --db-url=mysql://root:pass@127.0.0.1/dbname

As I am on a shared server, I do not however have access to the root password (and they won't supply it), so I am looking for alternatives.

I would like to end up keeping my development copy in the subdomain and the production version in /public_html/site/...

Bottom line, my question is: Does this Module "Backup and Migrate" solve my problem?
if not, any guidance on making my move via drush (in part or whole) would be appreciated!

ps. For past updates, Composer has been used per instructions on update page: https://www.drupal.org/docs/8/update/update-core-via-drush

couturier’s picture

@bas123 Yes, I believe that you can move your D8 site via the Backup and Migrate copying of the database plus a copy of your site files. We've had a few issues filed against the current 8.x Backup and Migrate release, but it seems to be working fine for most people who are backing up and restoring. If you have any problems, just jump back in to the issues queue and leave us a message about it.

Once your site is in its new location, then you should still be able to use Drush as you have previously depending on the type of server access you have. If you're struggling against server issues, you might consider switching everything to a provider that optimizes for Drupal and will work with your specific needs. I've heard a lot about how fast Pantheon is for Drupal sites, like "order of magnitude" faster if you're looking for speed, and I personally have used Worx, a Drupal-only shop, to serve my sites since 2010, including a couple of D8 sites, and have been very happy with the service and pricing.

Your comment does bring up the fact that we're probably still needing to update the module's main documentation page to be more descriptive for people coming into this module for the first time with D8. An open issue is filed against that here.

@bas123 Also note that Backup and Migrate for Drupal 8 does not require Composer.

bas123’s picture

couturier, Thank you for your insights and indeed it does sound like this Module will be a valued addition to my production site but wonder of it I can resolve my current quandary prior to (and without it) actually getting my site working...

Here's where I am at:
Because of my lack of server access (Shared Hosting plan at A2 Hosting), I was successful at using the '# drush archive-restore' to build my file structure in a new subdomain with the idea of renaming that folder ultimately to the production folder 'public_html'.

Initially, of course the settings.php points to the original (development) database (call it mysite_dev'). This actually showed me that I can have two instances of the website sharing one database.

However the goal is to have the Production version run from its own distinct database, while keeping the original development version on the site in the subdomain!

Using phpMyAdmin, I attempted to populate a new database with a mysite.sql dump from the original, but found it's file size too large for the program to accept, so I simply "COPIED" the original to a new database (call it 'mysite_live').

Finally, I modified the database line in settings.php from:
'database' => 'mysite_dev',
to
'database' => 'mysite_live',

But when I do, I get an error:
"The website encountered an unexpected error. Please try again later."

When I revert back, the site runs fine but with the dev database.

So:
1) Can you assist with what I must do to get the new "copied" database to work with the file set?

2) Is Backup & Migrate my best and simplest method given my progress so far?

Thanks in advance!

couturier’s picture

[Note: I have edited this comment to change a reference to an alternate module, Backup Database backup_db, that is coded to work with Composer. Backup and Migrate 8.x is unsupported for Composer compatibility.]

@bas123 There are obviously other ways to duplicate your database, but one of the most simple is as follows: do a fresh Drupal 8 install and create a clean database, and then copy over the contents of your dev sites folder. Then you can install the Backup and Migrate module and restore the database from the UI. You should be able to use the database copy you already have, but if that doesn't work, then install Backup and Migrate on your dev site, make a backup via the UI, then restore that one through the UI of your new production site. There is a "Restore" tab in configuration options where you can simply upload your file to restore a site. Some people have issues with uploading and restoring based on what type of file was created for the database copy. In the Backup and Migrate module, you have a choice of file extensions. If you're having further trouble, it would be better to move the issue to a new post in the issues queue since this current issue is a Roadmap for future features of the module.

If you're needing a different type of server access that A2 Hosting can't provide for you, I'd really encourage you to continue looking around. There are plenty of other Drupal-only providers that can customize to your needs. I use Hosts of America via Worx Co which is owned by a husband-wife team and serves sites of all sizes up through large, international, commercial sites, and they have given me special server access to maintain my sites for a very affordable price. They've also done restores for me on the back end using files I created through the Backup and Migrate module when I needed extra help. Other companies may claim to be optimized for Drupal, but I've had the frustrating experience in the past of realizing that it's better to go with a company that exclusively specializes in Drupal so you never have to wonder if site failures you are experiencing may be related to some element of the server not being optimized for Drupal.

bas123’s picture

@couturier Thanks again for your continued support on this.

Regarding your hosting suggestions, I am committed to this Hosting Company for the time being but if they continue this level of "lack of support", when my term expires I most certainly will follow your suggestions including Pantheon (who's sandbox I have played around in) and Worx Co. - At this time, however it's a matter of $$$ (or lack thereof)!

Respecting my dilemma however, what I am trying to get a handle on is the VERY BASIC concept of copying a site on a shared server to a new location on the same server and having it run from a unique (but initially duplicate database) on that server…

In my case: from 'Development' in one Subdomain to another (temporary) Subdomain (with the ultimate intention of renaming the subdomain's folder to Live Production in '/user/public_html').

Please confirm that regardless of technique (drush, ftps, git, or otherwise), if I copy the exact set of files from '/site' down to this new location, then create a NEW database and populate it with all of the data from the original (development) database, the ONLY code changes to get the new site running is to modify the database name in the '/site/sites/default/settings.php to reflect the name of this new database!

If this is the case, then it seems to me that the only possible issue (since the file sets both run using the original database) is the manner in which the new database was created!

Again when the new settings.php points to the new database, I am getting:
"The website encountered an unexpected error. Please try again later.", but when I switch it back it runs fine!

In my case, (because of the file size limitation for importing a *.sql file is limited to Max: 105MiB), I used the "phpMyAdmin > Operations > Copy database to" to duplicate my database which required that I create a new name!

So if the issue IS the method I used to create the new DB, then I have learned I have several options including compressing the *.sql to below 105MIB. It seems like this can be accomplished EITHER with "Backup & Migrate" OR "Backup Database", or using a command line operation to zip it into *sql.zip!

(Now this next part does apply to or at least present a reference for this "Roadmap" Queue and the need for seperate specific Drupal 8 Documentation)

Otherwise, if additional coding is necessary to get the new site operation, I have seen references to changing the "site_name" in the settings.php... Well there is no reference to site_name in the D8 version of settings.php ...

I have also seen suggestions elsewhere about renaming the site in "/admin/config/system/site-information" which I would certainly do once I get a working copy running!

Perhaps there are other techniques I am simply not "getting" and would appreciate it if this could be explained somehow.

Also, if this discussion truly does belong somewhere else, then by all means, please let me know where, and I will try to start over with my request(s)

Thanks again in advance!

DamienMcKenna’s picture

It's also worth mentioning that Drush has its built-in "sql-dump" command in Drush 8 and "sql:dump" in Drush 9.

couturier’s picture

@bas123 I have read carefully through your latest post and unfortunately don't have any new tips for you. If you're paying USD $45 per month or more for up to 3 Drupal websites and would like to have easier server access, you would do well to look at an alternative provider.

bas123’s picture

@DamienMcKenna Thank, however I use Drush 8 all the time for archiving and used #drush archive-restore for the attempt I have described that is causing the error. Everything works out fine until I try to change the database (even though the new one is a duplicate). I will probably do the fresh install via Drush per @couturier's suggestions above. As that seems to be worth a try.

@couturier, I do not pay anywhere near $45/mo for my shared hosting account (which has quite generous options, just seemingly limited tech support), and if I could would probably go with Acquia or Pantheon...

It has been my plan that if I can generate some decent traffic and therefore some ad revenues to justify it, I would need a platform like one of those anyway!

So, later today I will be following your suggestions above in #37 and will report back

Thanks

bas123’s picture

@couturier, in your instructions above you describe making a clean install (which would obviously point to a new database which I will install via my cPanel but then you say:
"copy over the contents of your dev sites folder"

Are you referring only to /home/username/dev/site/sites/*.* ?

If that is the case what about my themes, modules etc.?

Or did you mean /home/username/dev/site/*.* (which would then require some editing of files such as the settings.php) ???

I suppose this is what has caused me to hesitate.

Thanks

bas123’s picture

@couturier, in your instructions above you describe making a clean install (which would obviously point to a new database which I will install via my cPanel but then you say:
"copy over the contents of your dev sites folder"

Are you referring only to /home/username/site/sites/*.* ?

If that is the case what about my themes, modules etc.?

Or did you mean /home/username/site/*.* (which would then require some editing of files such as the settings.php) ???

I suppose this is what has caused me to hesitate.

Thanks

And:

With respect to this "Roadmap":
@DamienMcKenna

Clearly, my dilemma, if I am to use Backup & Migrate involves the need to RESTORE, however your documentation for D8 says:

Check with an experienced admin before attempting to restore from backups..

Anyway to enlighten me on restoring, once I can figure out how to actually get the Backup & Restore module to work, I have setup a private directory as instructed, gotten past the scare of a huge red error block by clearing caches etc. but have only gotten ONE backup of my database through and NO Full Site Backups through by Download OR Private Files Directory!!!

couturier’s picture

@bas123 For a Drupal 8 site, yes, you would also need to copy over the contents of your Modules and Themes folders. Those folders resided in the Sites folder in Drupal 7 but are separated out into separate folders in Drupal 8, obviously. I'm not experienced too much with settings.php files since I've had help in getting those arranged, but you are at the very least going to need to change your URL to match your new website. You'll need to contact A2 Hosting, your host, to get help with establishing the new database if you're unable to create a clean one on your own with a fresh Drupal 8 install.

It sounds like you are making some progress. Go back to your dev site and do the "Quick Backup" option with Backup and Migrate. You should get a file to your Downloads folder on your local computer. Then go to your new website into the Backup and Migrate options and follow instructions under the "Restore" tab. If that doesn't work, try contacting someone at A2 Hosting and email them a copy of your backup file and see if they can restore that to your new database for you. If this doesn't work, I'd recommend giving things a second try with a different module, Backup Database (backup_db), that is a no-frills version of a backup module that does a very simple backup file and is in certain instances more performant than Backup and Migrate, but it is dependent upon the Composer module.

bas123’s picture

I would like to report to all concerned that I have accomplished my goal of:

Creating a copy of my Drupal 8 development site to a new location (subdomain) which I can now move into Production.

But first I will say that the "missing ingredient" to the above techniques was the need to "Clear ALL Caches" which I had to do via phpMyAdmin

Second, this is perhaps a here-to-fore not attempted technique so I will spell it out step-by-step for anyone interested.

1) Create a new Subdomain to temporarily house your files

2) Make a backup archive of your database and files.

a) I used the drush command:

#drush archive-dump

b) I assume one could use Backup & Migrate for this as well (I just haven't mastered the module enough yet to do so)

3) Using ftp, sftp, cPanel or otherwise, copy that archive dump file into the newly created directory

4) in phpMyAdmin go to your existing (Development) database and "Operations > Copy database to"
a) enter the name of the LIVE database you wish to use eg: 'username_website_live'

5) Using Drush

a) #drush archive-restore BACKUPNAME.tar.gz
b) This will recreate your files in the new directory

6) Edit /site/sites/default/settings.php to reflect the new database name:

  'database' => 'username_website_live',
  'username' => 'your_database_username',
  'password' => 'your_database_password',

7) In phpMyAdmin, go to your newly created database and "empty" (TRUNCATE) all tables beginning with "cache"
a) There are several methods for this operation, I simply checked those tables and selected "Empty"

Then go to the new subdomain in your browser and the "cloned" site SHOULD be working...

If you get the error, "The website encountered an unexpected error. Please try again later." you may need to retry to clear the cache tables again or using another method...

Possibly Use the command drush cache-rebuild to clear and rebuild all cached data for a site. After running this command, you will see the output message "Cache rebuild complete."

Perhaps one of you more advanced contributors might reword this as a "proper" tutorial!

Thanks again to @couturier and @DamienMcKenna for your help - it is most appreciated!

@DamienMcKenna, Perhaps my above input can be reinterpreted to be useful to the "Backup & Migrate" community and your continued quest to keep up with the documentation of this worthy module!

To be clear, @couturier, I have complete access to my shared hosting account including SSH and a fully functional cPanel, What is lacking there is assistance with Drupal specific Tech Support! They have techs who have bits and pieces of knowledge on Drupal (more-so with Wordpress), but their Support "OVERSEER'S seem to want to monetize the information they provide and shield support tickets that go beyond what they have defined as within their defined "boundries".

couturier’s picture

@bas123 I am glad you've succeeded. The reasons you mention about limited support are exactly why I'd recommend going with a host that is dedicated to Drupal. Some of the big companies advertise that they are Drupal-friendly, but they also try to serve a lot of other CMSs as well, so the service isn't always targeted. The hosts who are exclusive to Drupal really know what it takes.

While I'm glad you've found your solution, the steps you outline won't be searchable here because this topic is completely something different. If you like, you can post under a dedicated topic in one of the Drupal forums with a specific title that would help other people, too. The best forum is probably the Installing Drupal forum where you can copy this content to a new forum topic.

bas123’s picture