Deploy cgit on new git servers

Here's an example so people can see what this looks like: http://cgit.freedesktop.org/

another https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/

Another example, a little more 'themed', from GNOME https://git.gnome.org/browse/

added parent issue

Added the parent issue as a proper parent issue.

Other than that, it would be great if we could get it styled in Bluecheese.

Test instance up: cgit.drupalcode.org

With cgit it is likely we will not support viewing all sandboxes, but instead a whitelist of 'important' sandboxes. We are still considering this.

I asked about motivation for not supporting sandboxes. There are two problems. One is that sandboxes are basically untrusted content and putting untrusted content onto the web is risky. We've had a few xss issues in this realm. I think that issue is moderately solved by putting it on drupalcode.org.

The second reason is that every single repository we add to cgit slows it down even for a single client browsing the content. Add to that the fact that more repositories means more pages for spiders to crawl and the effect is probably even worse. So...we'll need to figure out something to mitigate that issue.

Greggles pointed out that the sandbox view is used a lot for reviews. Due to this, even if cgit gets launched without sandbox support, it'll likely be added via another vhost or another solution to mitigate the performance issues of the sandbox repositories.

-N

Suppose spiders should not index this pages, maybe it makes sense to provide a robots.txt

I could see people using external search engines to find content on drupalcode.org. That seems like a pretty good idea to me, actually, so a robots.txt to disallow all doesn't seem like a great solution.

I think all sandboxes need to have some sort of repository viewing, and if deploying cgit means we don't have that any more, that seems like a regression of d.o functionality.

Indeed. :\

Sandbox repo viewing is pretty critical for use cases like:

- Approving new Drupal module/theme contributors from https://drupal.org/project/issues/projectapplications. Would-be contributors are instructed to put their code into a sandbox and both maintainers and other applicants use the repo viewer extensively to provide code reviews to help the process along.

- Rallying contributors around crowd-sourced efforts like Migrate in Core: http://drupalcode.org/sandbox/chx/2105305.git The repo viewer is basically a view into what's done / left to do for a given initiative. We generally don't want Drupal core forks to be first-class projects, as that's very confusing.

This was actually already semi-resolved via IRC.

@webchick: yes, those were the two issues pointed out. The core forks one is solvable via a whitelist, but the approval of new themes/modules is not easy to work around without sandbox viewing and is obviously very important.

@killes: In its easiest to roll out form, it simply doesn't work. The issue being how cgit scans for git repos. The load time for any page goes from seconds to 15-20 seconds. This was what lead to the discussion about just removing viewing and then delaying viewing (rolling it out without sandbox support and then adding support).

As it stands at the moment, I have delayed the roll out and am working on a script to fix this.

The sandbox issue is now fixed, as are most of the redirects. This is likely ready to go. Will coordinate testing.

-N

cgit is now active. Drupalcode is redirecting to it. Most common URI's _should_ redirect.

-

Awesome. I just tested a few urls and they redirected fine. It feels very fast and it seems great to me that it's got an overview page at drupalcode.org rather than the sparse page that was there before.

Are we purposefully not using google analytics on this site? It seems like a good idea to get at least some metrics about it.

In testing, it seemed like it actually has a much nicer interface. In particular for patches/diffs.

I don't think we have ever looked at GA on drupalcode. I don't really feel strongly either way. Could do awstats as well/instead of GA. (might be easier since its CGI based)

-N

Unfortunately, this appears to lack a git blame feature presently. The UI for git blame is pretty much the only reason I've ever used drupalcode.org over my local git repository (other than occasionally linking someone a particular diff), and it has been an important tool for helping contributors understand the development of Drupal core and how to navigate the core community and codebase. We have https://drupal.org/documentation/git/blame explaining how to use the feature that was provided by the previous drupalcode.org application (and probably other references in other documentation).

I'm concerned about this change being deployed right before DrupalCon Austin (with its enormous sprint for new contributors), and was wondering what the implications of rolling it back and deploying after Austin instead would be.

While its definitely possible to roll back. I am very against it. The version of gitweb we were running is actively dangerous and I really do no want to be running it during a drupalcon. This was part of the reason we have been trying to get this done before Austin.

In addition to being insecure, gitweb was broken on certain tasks and certain files. Viewing the source and blame seemed to be the 2 things that most frequently triggered errors for me.

https://dev.haiku-os.org/ticket/8510 seems to be the issue about adding annotate/blame to cgit and its stale since 2012 (but gitweb was stale since longer than that, afaik).

Yep, https://drupal.org/project/issues/infrastructure?text=server+error&statu... has a few of the issues with breaking on certain files. I'm sure there are more that weren't categorized. Gitweb also had a habit of using up all the resources on our util server frequently.

For more details on the problems with Gitweb and deciding on a replacement, see #1914722: Evaluate alternatives for gitweb, which is also linked from this issue's summary.

Just a heads up, the switch breaks any snapshots used in make files (e.g. http://drupalcode.org/project/contextual_view_modes.git/snapshot/d5d3ced...)

Is there a way to bring those back?

On the upside, cgit does feel a LOT faster.

Thanks

What were the snapshots? Is it a tarball from a specific commit hash?

I don't think that was really a supported feature. Can't you specify in the make file getting the files via git clone of a specific hash?

They were indeed tarballs from a specific commit hash.
I've since updated the make file(s) to clone the specific commit hash, thanks.

Though I'd bring that up in case people stumble upon it.

Cheers,
Dan

Snapshots are supported by cgit. I don't know what the impacts are on resources.

https://wiki.archlinux.org/index.php/Cgit#snapshots_does_not_show_properly

http://git.zx2c4.com/cgit/tree/cgitrc.5.txt#n91
http://git.zx2c4.com/cgit/tree/cgitrc.5.txt#n403

Actually it looks like snapshots are supported/switched on, it's just that the links are hidden in the UI. You just need to craft the right URL:

http://cgit.drupalcode.org/PROJECTNAME/snapshot/COMMITSHA1.tar.gz

For example, the very first commit in the master branch of the Brightcove module is downloadable here.

I am honestly unsure what I think about snapshots, but ya they are supported in cgit.

I wouldn't recommend using the snapshot tarballs from either Cgit, or previously Gitweb, in drush make files. While they may work, drush make isn't designed to support them well.

Syntax highlighting has been enabled with the use of a custom python+pygments script.

I've noticed that http://cgit.drupalcode.org/bootstrap doesn't work for https://drupal.org/project/bootstrap.

Talked with @basic and he did a search: http://cgit.drupalcode.org/?q=bootstrap

There's two "bootstrap" repos.

Fascinating. From a quick look I think there is something 'abnormal' with that repo. Will have to look in detail.

-N

Yes indeed there is something weird with this repo. It looks like a REF was either renamed or failed to save. It looks like what happens in this case is undocumented, which is great. We may have to call Mr. Boyer into this to look at this repo.

EDIT: Or perhaps not...looks like this is the case for some other repos that work....may still need Boyer.

-N

This project doesn't look to be working…

http://cgit.drupalcode.org/commerce

Search shows duplicate repositories…

http://cgit.drupalcode.org/?q=commerce

Just an FYI.

Thanks,

•• matt

I think snapshots should be reverted back because this is a only way to get tarball for sandbox project

@andypost see #26. What scenario do you need to download tarballs of sandbox?

Since its not really disabled, just hidden. I'm going to just turn snapshots on unless anyone thinks thats a really bad idea.

-N

@mattIt Thanks for the report, we are tracking this down. It is happening on what seems to be a very few repos and looks like it has to do with naming.

-N

re #36: Historically, we have purposefully not packaged sandboxes as tarballs to avoid a casual user who might think "tar.gz exists = stable/supported". Sandboxes are often not "production ready" code. In this case, making it "hard" to download (i.e. git only) is a feature.

@mattIt/Mark

This issue should be resolved. Thanks for the reports.

@greggles the scenario for tarballs is:
1) from sandbox project visit "Repository viewer" link
2) copy a link to tarball
3) paste the link into 'admin/modules/install'

Not packaging sandbox projects is by design. They are intended for collaboration among developers, not for site builders to easily install. See https://drupal.org/node/1294804.

FTR I have configured cgit versioncontrol webviewer url handler plugin at git7site for testing.

AFIAK there is not a cgit instance in git7site, so I pointed it to production cgit. Should be good enough to try ;-) i.e. see the links generated at https://git7site.devdrupal.org/node/148303/commits or https://git7site.devdrupal.org/commitlog/commit/3428/2b9b962066c6e9331b7...

Filed #2285061: [feature regression] Bring back git blame UI.

This also effected simplytest.me #2288983: Sandbox: The selected project has no available releases.

just adding related issue #2354221: Generate links from commit message to d.o issue