We migration working: good. But what if I want to test a migration? Sure, if the SQL or the update.php fails to upgrade, migrate will rollback, but it will not do functional testing (does the theme look right? does everything actually work?).

So we need clone support: this is basically backup + restore to any platform. Should be fairly simple to implement, based on migrate: just don't remove the old site. We need some knobs to choose the cloned site name, but that's about all.

CommentFileSizeAuthor
#10 448692_clone_provision.patch3.27 KByrocq
#10 448692_clone_hosting.patch3.89 KByrocq
#6 448692.patch1.37 KBmig5
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

adrian’s picture

adrian CreditAttribution: adrian commented

I do however think keeping a relationship to the old site might be useful.

ie: to clone a dev something site

anarcat’s picture

anarcat CreditAttribution: anarcat commented

Just to make things clear here: 0.2 already supports "deploy" in the backend, which is almost complete clone support (almost, because it doesn't backup the original site). I guess that a 'clone' command in provision would be in order. Something like:

provision clone site_A.example.com site_B.example.com

We should be able to choose a destination platform too:

provision clone site_A --destplatform=/var/aegir/drupal-6.12 site_B.example.com

destplatform could support ssh too. :)

anarcat’s picture

anarcat CreditAttribution: anarcat commented

This will require to solve the same issues as rename (which aren't currently solved by migrate or deploy): see http://drupal.org/node/275737#comment-1968674

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Version: 5.x-0.2-alpha1 »
Status: Active » Needs review

clone support has been implemented by koumbit and is available in the branch named "clone" on provision and hosting on git.koumbit.net. Official patch up for review here soonish.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

Cloning to other platforms works great :)

Cloning to the existing platform is (as far as I can see) currently impossible because the 'Current' radio box in the migrate form, which this feature inherits, is disabled, and the changes in #558628: Form to migrate from current platform to current platform submits, and shouldn't stopped the form being able to be submitted if no other target platform has been selected.

I'm trying to work on a way to make the form intelligently know what task it's being asked to render itself for (if migrate, disable the Current radio, if clone, enable it), which will fix this problem... any ideas on how to do this last part welcome

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
FileSize
448692.patch1.37 KB

Anarcat,

I've patched hosting_task_migrate_form() to check the argument handed to the form and if it's 'site_clone', the Current platform can be selected to clone a site to. If it isn't (i.e if this is a regular migration), the Current platform radio button remains disabled and thus a site cannot be migrated to its own platform.

Wanna apply my patch and confirm that this makes cloning to the current platform work as expected? (did you get this to work some other way already? I think this fix is the only way to allow it to work without duplicating the migrate form into clone though)

Edit: I see Yann's TODO comments in the commitdiff relate to this, maybe point him at this patch if he's still working on it

adrian’s picture

adrian CreditAttribution: adrian commented

You need to maintain the relationship of which site it was cloned from.

The best example of this being development -> qa -> live

We're essentially talking about the ability to have multiple instances of a site.

anarcat’s picture

anarcat CreditAttribution: anarcat commented

@mig5 - it's weird because cloning to the same platform worked here. The validation function is different (we fixed it) and the submit actually works. That's how we tested it. So i'm not sure how this patch is required in the first place. Furthermore, i'd rather keep clone stuff out of migrate for now. (I would think that for "rename" we would need the site name in the migrate form however...)

@adrian - which relationship do you want to have? where/how to store it? are you saying that we should allow overwriting existing sites too? (not that I oppose, i just wonder how you figured it would work)

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

My bad, sorry for the noise. Dunno what was wrong with my installation on that server: a fresh install and yes the form submits just fine.

While I'm here, we will need to introduce dependency checks somehow for the task modules (Clone can be enabled but not Migrate, which of course will throw undefined function exceptions when it tries to return the migrate form)

yrocq’s picture

yrocq CreditAttribution: yrocq commented
Version: » 6.x-0.3-rc4
Assigned: Unassigned » yrocq
FileSize
448692_clone_hosting.patch3.89 KB
448692_clone_provision.patch3.27 KB

Hi !

Here are the patches from the Koumbit Git server in a more traditionnal form (see #4).

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

Hi Yann, do you mind resubmitting the patches.. something weird happened and the links are to http://drupal.org/files/issues . This happened to me yesterday too, must be a drupal.org issue

Cheers for this terrific work!

asak’s picture

asak CreditAttribution: asak commented

Files are accessible at:
http://drupal.org/files/issues/448692_clone_provision.patch
http://drupal.org/files/issues/448692_clone_hosting.patch

asak’s picture

asak CreditAttribution: asak commented

Patches from #4 and #6 seem to work well for me.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

asak: my patch from #6 is probably not needed unless you also experienced the issue whereby you could not clone a site to its own existing platform (received form validation errors).

I tested on a fresh install in the end, without my own patch, and found it was working fine. See #9

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Status: Needs review » Needs work

So there was an interesting comment in the rename issue, see http://drupal.org/node/275737#comment-1986406.

In general, I think I agree that it would be nice to allow overwriting existing sites when cloning, and keep track of the relation, I'm just not sure how to do that exactly. Should we consider a site node as possibly representing multiple sites across different platform? That's a major paradigm shift that would involve changing the way we do tasks and a whole lot of other things. I think I would rather have relations between sites that would authorize overwriting sites (so if site nid 22 was cloned to site 34, then it's always allowed to overwrite 34 later). I guess a seperate hosting_site_hierarchy would be necessary to track those down.

Other comments?

univate’s picture

univate CreditAttribution: univate commented

As you pointed out what I described would be a bit paradigm shift to how a site is thought about.

But creating these commands like 'clone' should just be a frontend/interface change, as I think the key should be that at the lower levels in drush/provision is where all the standard CRUD functionality actually exist (e.g. create/backup/restore/edit/upgrade/verify/delete), that way anyone can come along and build all sorts of different workflows or even a completely different front end.

So the clone operation from SiteA -> SiteB could be:
1. Create new site - "SiteB"
2. Backup - "SiteA" to file "X"
3. Restore backup file "X" to "SiteB"

An override command from SiteA -> SiteB could be:
1. Create Backup "SiteB" (so we can roll back if need be)
2. Create Backup "SiteA to file "Y"
3. Restore backup file "Y" to "SiteB"

I haven't dug around in the code to understand how everything works, so please ignore if I am stating things that are already being done.

On the front end I think having a hierarchy or some type of site relationship as mentioned in #1 is required for commands where want to remember where cloned "siteB" from "siteA" - but it would also be good to have some history of the various actions (rename/clone/delete) with backups saved at each step so we can also rollback - when you give someone the power to easily do things like overriding an entire site you also really need to provide a way to undo that action in case the user makes a mistake (since we always tell people to make a backup before updating their sites).

I think there are two types of clone operations you may want to perform:
* clone for staging or testing a site, so you want a reference that this is testing version of some other site which you also may want to be able to override later to resync the database.
* clone for a new site, where you are taking a copy to build something new (which we would never want to override or provide the ability to override).

And this is where i feel there should be a way to group websites under the one site object. In reality you should only ever have one 'production' site, so maybe its as easy as having a parent id against a site object. If a site doesn't have a parent its assumed to be the 'production' site, but if it does have a parent site then its a staging/testing version of its parent.

This could then dictate the operation that you can perform on each type of site.
e.g.
* a staging site (one with a parent site) could be overridden by the production site (but not the other way around)
* staging sites would probably not need to be listed in the view all sites table (or maybe you just want a filter so you can enable/disable showing them)
* etc...

Anyway thats a few thoughts/ideas (I have lots of others...) and I hope to actually find some time to start contributing something to this project, as I think its great.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Status: Needs work » Needs review

So I welcome your input, but at this point, I'm not sure the whole paradigm shift is worth it. We can just have named relationships between sites that will accomplish exactly the same thing.

I'm rather fond of the current patch. It's simple and does work fine. It answers your first use case ("clone for a new site"). The second use case, with relationships between sites (dev/qa/prod) requires an extra table and UI work to show for it, but that's about it. No use in refactoring the whole thing.

So let's commit clone as is for the first part and throw this back to the frontend (hosting) for the second part.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
Status: Needs review » Fixed

Committed. Thanks Yann and those at Koumbit for the great work!

Matto’s picture

Matto CreditAttribution: Matto commented

I'm a relative Drupal newbie and I'm hugely impressed by what I've seen of Aegir, so thanks for a great tool.

I'm far less Drupal-capable than Univate, but I came up with exactly the same idea as he did (http://drupal.org/node/275737#comment-1979562) regarding the desire to have live/test/dev/whatever versions of a single site, with exactly the same name, existing on different aegir "platforms" on the same server simultaneously.

I understand that aegir doesn't currently allow this, and that this is a deliberate design decision, but I'm not smart enough to understand why (that's not sarcasm).

However, if it was possible, it would solve significant workflow issues for me (and, I suspect, many non-experts like me).

Firstly, the fact that the filepath in a Drupal site is relative to the Drupal core and not the specific site folder (e.g. siteone) means there is much pain if you need to change the name of siteone, such as when you copy siteone to do some offline development work and you still want the file system to work. In aegir you need to give the copy of siteone a different name, e.g. siteone_dev.

But if aegir enabled this on the same server...
aegir/platforms/live_drupal-6.13/siteone
aegir/platforms/dev_drupal-6.13/siteone
... it would (I think) solve that issue.

See Univate's comment here:
http://drupal.org/node/275737#comment-1986406

The aliases/links for the above might be www.siteone.com and dev.siteone.com, respectively (which I believe is possible - your aegir site name doesn't need to have any domain suffixes or prefixes - correct?)

And that's actually the second thing on my wishlist - I don't want to have to name my aegir sites (folders) as www.siteone.com, I want to be able to give it whatever name I want, like "gomerpyle" (without any ".com") and make it appear in a web browser at www.notgomerpyle.com, via the alias settings. One doesn't always know what the live url/s for a site will be ahead of time, and sometimes people re-brand later on. So I want the name of the site (as I set it in aegir) to be totally abtstracted from the url or urls at which a browser can see it in action, now and in the future.

I know the above is not necessarily new to these threads, but I was keen to express my support for Univate's initial outline of these features from the perspective of a non-expert user.

Anyway, go aegir, you're kicking arse already.

anarcat’s picture

anarcat CreditAttribution: anarcat commented

I would like to keep further comments out of this thread and into new, more well-targeted issues or feature requests. Feel free to refer back to this issue, but now things are just confused for me here and I would like to see more clearly what's missing and desired out of clone right now.

Clone works, it may be improved, but this particular feature request is done. Please open a separate one if required.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit 273aa65 on 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by mig5: 
    #448692 by yrocq et al - site clone support

  • Commit 273aa65 on 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by mig5: 
    #448692 by yrocq et al - site clone support