[Meta] Drupal.org Development Environments

We also need a subset of Git repos, and maybe a r/o mount of all Git repos too.

Fixing url, also tag.

"How multiple people can work on one environment? Nothing from infra side, better project management needed"

Um. What? Sorry, but no. :)

From the infra side, we need the following things to support this:

1) Move code from Bzr to Git, since everyone in our community knows Git, and like 12 people know Bzr. (Already on the hit list. Yay.)

2) Allow for developers to use actual version control during their Drupal.org development, rather than leaving a bunch of modified files sitting around in a source tree they don't have commit rights to, and which get blown away when either another user edits the same file (oops!) or when we inevitably run out of space on the server and need to obliterate a dev environment to free up room.

Not sure how to solve this one, technically. One idea that's probably terrible is a drupal_dot_org_sandbox project that mirrors everything in the production git repo, but automatically grants everyone with SSH access to devwww with commit rights. Then, we document a process whereby when people want to work on something they create a branch 123456-feature-description and hack there. If volunteer X goes away because they get busy, volunteer Y can pick up where they left off, and have the full history of what got changed, when, and why. And since it's in version control (versus a patch file), it's easy to merge in with later changes that happened upstream. Speaking of which...

3) Also, provide a way for developers to self-update their code and DB without "ping drumm" being the only methodology available, and without destroying everything that's there in the process.

(Warning: Major hand-waving ahead!) Ideally, drupal_dot_org_sandbox would also have drupal_dot_org (or whatever the production repo is called) as a "remote" (I think) so you could "git pull" and get the latest code changes on prod without having to request a "please blow away all of my stuff; hope to $diety I remembered to save a copy of the diff in my home directory first!"

Then, I guess a drush command or similar that's like "drush sql-sync @prod" and could pull down the sanitized prod DB on-demand.

Once we get all of that, then yes, project management will be our biggest problem. I greatly look forward to that day. ;)

I may be too out of the loop to suggest this, but I noticed the BZR to Git migration wasn't listed in the related issues. "How multiple people can work on one environment? Nothing from infra side, better project management needed" was something that came out of the migration to Git solving the workflow issue and not requiring another git server or services for the dev vm's. I'm rewording this to "How multiple people can work on one environment? BZR to Git (#2099755: Migrate self-hosted BZR to Bitbucket Git Assigned to: drumm) should account for development workflow and allow for sane revision control of d.o dev work."

Lets discuss more on how the Git developer workflow should work in the #2099755 issue. I think we can make use of multiple remotes and use the dev vm as the origin to solve a lot of the workflow problems.

"Possibility to create/delete dev environments themselves (openstack snapshot, clone vm, quotas)" hints at the DevOps focus for the development environments, but it's not highlighted enough -- this is extremely important that updating and creating development environments can be handled in a way that makes it easy for developers (whether it's drush, git merge, openstack ui, or vagrant).

Is there an ideal workflow that we should strive for? Not just for Git, but the entire process of requesting a dev environment, provisioning development environments, getting access, writing code, adding people to your dev environment, syncing from production, and any other common operations?

Yes! I am the person for this. Thank you for taking the time to document the bottlenecks and help clear things up about what is needed. I will be diving in to this as soon as I have the hardware for openstack ready.

Is there an ideal workflow that we should strive for? Not just for Git, but the entire process of requesting a dev environment, provisioning development environments, getting access, writing code, adding people to your dev environment, syncing from production, and any other common operations?

+1 to everything Melissa said. In an ideal world, this is how it would work for me. (Also speaking as myself, not as the committee.)

1. Random-ass volunteer (RAV) says, "I have a great idea for Drupal.org (or I like someone else's idea). But going through a big community bikeshed about it and/or asking for access is intimidating, since no one really knows who I am in the community. I have a free evening, a case of Red Bull, and an IDE. Let me see if I can whip something up as a prototype, so it's easier to talk about my idea."

2. RAV goes to the prominently linked and promoted "How to contribute to Drupal.org" page, hits a big-ass button "Get your Drupal.org developer environment!"* which has a VirtualBox image / shell command to run / whatever so they can get a copy of the stuff on their local machine where all of their same tools they're used to using on other Drupal projects are located.

(Warning: That "local machine" part is contentious. It could also auto-spin up a dev environment on drupal.org's servers as it currently does, but that's been traditionally limited by space on the box. The bottom line is to remove the "ask someone permission and wait around for them to get to it" part, because by then RAV's free evening is gone.)

3. RAV hacks on the D.o code the way they hack on code in any other of their Drupal projects: open up their editor/IDE and debugger of choice, commit things to Git (remotely hosted on d.o so the work doesn't get lost if RAV decides to go play Zelda for the next 4 weeks instead) in small stages as they figure things out, etc. The full gamut of d.o's functionality is available to them, so they don't hit a wall when their idea touches e.g. Solr or Git or Testbot.

4. Once RAV gets something kinda working/worth looking at, they post an issue+patch to the d.o issue queue, along with a demo URL for the community to try. (Here's where d.o-hosted dev environments would be really helpful, so people don't have to punch holes in their firewall; maybe the dev sandboxes could optionally be initiated from $branch in $drupalorg_sandbox repository, which was committed to in step 3.) EDIT: http://simplytest.me/ does this, for inspiration.

5. The change looks pretty cool, and like it's on the right track, but there are numerous issues with it still, since it was just a prototype. For example, the theming looks crappy, it doesn't take into account this other area of the site, etc. Random Ass Front-End Dev (RAFED) and Random Ass D.o Contributor (RADC) both pull in the changes from RAV's branch into *their* local dev environments and start hacking, committing and pushing as they go. They'll probably also need to refresh their environment's database, cos the last time they did anything on the environment was a few months/weeks back, so they run a drush command (or whatever) for that.

(We could also use patches to the issue queue instead of Git commits to the same branch if that feels too ooky. The key thing here being, simulate the standard d.o development process where lots of people who may or may not know each other directly can hack on this together, but no one needs to ask permission/access from anyone else for anything. Cos hey, it's in Git. If someone commits something dumb, you just revert it. If they keep committing dumb things despite being asked to cease, we can revoke their Git access for awhile. :P~)

6. Once these folks all feel like we have something worth looking at again, the d.o-hosted demo URL pointing to $branch is refreshed so the community can see the fruits of the labour. Oooh! Ahhh! There is much applause and rejoicing. Now when the D.o teach team is looking through the list of things to see what needs review, they can review something far further along and can inspect the code easily using standard tools they use for inspecting any other code in Drupal to check for performance issues, security issues, and so on.

I might be forgetting some things, and I might also be answering the wrong question entirely :P, but that's sorta the "d.o contributor nirvana" vision in my head anyway. Emphasis on loose coordination of smart people with varying degrees of availability, rather than tight collaboration of the same, resulting in lots of little delays piling up and initiatives stalling out. EDIT: And also emphasis on "use the same tools / workflow you use on every other project / site" to greatly lower the barrier of entry to helping out Drupal.org.

umask should be set in user login shells so that new files, such as newly-installed modules and compass-generated files, are group writable. This should be set for devwww, this is happening on current dev sites.

From a "UX" POV, here are three issues I ran into tonight trying to test #2295411-35: Auto-generate Git attribution info / commit messages on Drupal.org. Fixing these would make testing changes in dev environments way easier for "non-Drupal.org insiders." (Who are basically exactly who we want to test proposed changes. :D)