Drupal 10, the latest version of the open-source digital experience platform with even more features, is here.I really like how DevShop operates, but it seems to me that keeping each platform entirely in Git will be a maintenance and performance nightmare. Bear in mind that I'm coming from a mass hosting perspective, where we maintain platforms that can contain 100+ sites. I don't see how I could sustainably update 100+ projects/platforms/sites. Also, multiple drupal code-bases and contrib would kill performance at that scale.
Admittedly, I'm still very new to DevShop, so perhaps these problems have been solved in some other fashion. In that case, you can consider this a 'support request'.
At Koumbit, we have have a team that updates our platforms on a regular basis. We normally maintain 2 staging platforms (D6 & D7) and 2 production platforms (again D6 and D7). Staging and production platforms are built using the same makefiles, but isolate production sites from WSOD-typos and such.
Last week, @gboudrias and I brainstormed on the matter. Our objectives were to be able to:
- enable DevShop functionality during development
- allow Drush-based workflows (e.g., 'drush dl module')
- maintain our strict operational procedures in production
- be able to bring a site in production back into DevShop for any fixes or additional development.
At first, we tried to figure out a way to do this natively with DevShop, but nothing obvious came to mind. Then we thought about just generating a makefile based on the project. This would allow for (1) and (2), and parts of (3), but we didn't see any easy way to do (4). Also, generating makefiles was error-prone the last time either of us had tried it.
Finally, we came up with a hybrid that we think is the best of both worlds. Basically, the idea is to only keep the site under Git, along the lines of hosting_site_git. However, we'd also include a makefile in the site directory, that could be used to build the project platform in DevShop. While this could be a normal makefile, specifying any required modules, themes, etc., it could also just have includes pointing towards makefiles maintained by our ops team.
So that's the general idea. We've worked through a couple implementation workflows, such as running 'git pull' in the site prior to updatedb to keep automated rollbacks, should a change cause problems. Obviously, this will entail further analysis, and no small amount of development. However, I think the benefits will justify it.
Comments
Comment #1
ergonlogicWe've got the above mostly working, but we've had to override a number of things that are somewhat baked into DevShop. As such, I'm going to propose some changes that should affect DevShop minimally, but provide the flexibility we need.
The first of these is to add a couple optional parameters to drush_devshop_provision_provision_devshop_commit(). The first allows us to specify an alternative path to the root of the Git repo, and the second an alternate path to the features directory.
I've attached two patches. The first minimally implements the change described above. The second also implements those changes, but also renames some variables for clarity, simplifies loading context objects and cleans up some whitespace.
Comment #2
ergonlogicThe other major place we need to override stuff is in how we pass arguments and options to tasks in
drush_devshop_projects_pre_hosting_task(). The patch is a little messy, as it's mostly indentation changes. Really all we're doing is adding a conditional check on the type of project, that is whether the git repo is in the platform or the site:
Since 'project_type' isn't set in DevShop, this will always return true, and thus allow the normal passing of arguments and options to tasks. In our case, however, we have implemented a 'project_type' field for projects, platforms and sites. This allows us to wrap our comparable code in
if ($task->ref->project_type == 'site') {, which in turn allows others to define their own project_type(s).There may be other places where similar overrides are needed, but that we haven't come across yet. If you're amenable to this approach, we'll submit comparable patches if/when we encounter such situations.
Comment #3
ergonlogicThe attached patch does essentially the same for 'Pull Code' tasks as #1 does for 'Commit Features'. It adds a 'git_repos' option that allows a list of repo paths to be passed in. I had forgotten to add the documentation for the additional optional arguments in #1, so I added those here as well.
Comment #4
gboudrias CreditAttribution: gboudrias commentedSame patch as #2, rerolled for latest dev version (which, as far as I can tell, only adds 'force = FALSE' on the Pull operation).
Comment #5
gboudrias CreditAttribution: gboudrias commentedRerolled patch in #3. This one was quite a lot more complicated, so I dropped the whitespace and minor comment changes.
Comment #6
Jon PughI'm going to be back on devshop code next week to get as much done as I can before DrupalCon... is anyone else here going? I'm going to have a sprint to 1.0 release if there's interest.
Comment #7
Jon PughJust reviewed it without a test, it looks pretty good. I'm glad for the replacements of "platform" for "environment". I need to do this everywhere. This is all you need to allow using makefiles to build projects? Pretty cool...
Comment #8
gboudrias CreditAttribution: gboudrias commentedGood to know!
It's certainly the vital part, but there are a few place where hooks would help. We've either already submitted patches for those, or we will in the near future :-)
I personally am not going to DC, but I'll try to help with testing (and rerolling patches!)
Comment #9
ergonlogicHi all,
The patch in #4 wraps all the passing of arguments and options to tasks with:
The implementation of the project_type property is currently in Valkyrie. Hence the call to empty() which would be the default in DevShop. I think it'd be better to move that into DevShop too, and have 'platform' be the default project type. Then Valkyrie could just implement a 'site' type, and anyone wanting to do similarly wouldn't have to depend on Valkyrie.
Also, right now it's all or nothing. As soon as we implement a new project type, we end up having to handle all the different task_type's, even if they work just fine with the default arguments and options. Ideally, we'd be able to get more granular and override just the tasks that are relevant for a new project type.
Comment #10
helmo CreditAttribution: helmo commentedI'm not going to be in Portland, but I also plan to be 'available'.
I've created dev-makefiles branches in my devshop sandboxes, based on 6.x-2.x Both #4and #5 applied cleanly. I plan to use this in my dev/testing...
@gboudrias, @Jon Pugh : I've added you both as maintainer to:
Comment #11
Jon PughFantastic, I will check these sandboxes out this week.
Comment #12
Jon PughI want to revisit this. I believe some code is already in.
Comment #13
Jon PughBumping again. Let's revisit, lots of refactoring has happened.
Comment #14
Jon PughExpanding the scope of this to include building environments where the git repo is the installation profile.
We could mimic drupal.org's packaging and look for a {profile_name}.make file to build the platform.
If we did this we would have to clone the install profile somewhere, then build the platform from the cloned makefile.
I do want to start cloning a project's code to the server outside of the platforms, which would allow analysis in the create project wizard possible, and allow much faster "cloning" by just copying this folder instead of re-cloning everytime.
Comment #15
Jon PughI'm open to figuring out how to make this work but I can't do it myself.
I'm postponing this for 1.0.0, but would be happy to include it in the next minor release if someone would take the lead.
Comment #16
TechNikh CreditAttribution: TechNikh commented+1
I really need this feature for my project to be able to install Drupal from a git repository like https://github.com/commerceguys/commerce_kickstart
Comment #17
TechNikh CreditAttribution: TechNikh commentedIt's mostly running drush make commands if it detects a make file in the repository right?
Comment #18
Jon PughKicking this over to Hosting Git module.
I've been able to get a platform to deploy from a makefile in your git repo, once we added the "repo_path" field to hosting git platform.
See the branch '2838489-repo-platform-path' in hosting_git.
It required a tiny patch to Provision that I pushed in http://cgit.drupalcode.org/provision/commit/?id=c50a0c0
Now comes the hard part: what to do when there is new code to "pull", which is why I always stuck with the entire git codebase.
Comment #19
Jon PughTechNikh the tricky part isn't the initial build but updating the platform from the makefile in the git repo. drush make won't replace an existing directory, even with the --overwrite option. Even if it could, doing it that way isn't up to "strict operational procedures" of creating an entirely new platform. The hard part is that in order to push a release easily we have to dynamically create platforms with some kind of naming convention and then automatically migrate all the sites to that new platform.
With enough clever code, we can make a way to set a site's "mode", then if in production mode, on git release we create either a blue or green platforms then migrate to it.
...
Comment #20
Jon PughSee my comment at https://www.drupal.org/node/2838489#comment-11867831
Enabling a separate repo_path property opens up hosting_git to be able to handle makefiles in git repos, but we need a better strategy for deploying between platforms.
Comment #22
Jon PughComment #23
Jon PughThis code simply removes the validator that prevented makefile and git url from both being filled out. This allows the platform to be built from the makefile in a git repo.
Comment #24
Jon PughComment #25
helmo CreditAttribution: helmo at Initfour websolutions commented+1 for dropping the validation
Comment #26
Jon PughOk, sounds like RTBC then :D
Comment #28
Jon Pugh