Problem/Motivation

Drupal 8 depends on many 3rd party libraries.

Composer uses GitHubs API to retrieve Repo data and Tarballs (in Composer prefere-dist mode). GitHub API Rate Limits for unauthenticated requests are very low. We will hit the limit in Drupal CI easily.

https://developer.github.com/v3/#rate-limiting

Proposed resolution

Generate OAuth token via GitHub and store it in ~/.composer/auth.json on Testbots.

Remaining tasks

User interface changes

API changes

Data model changes

Comments

webflo created an issue. See original summary.

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented
Project: DrupalCI: Drupal.org Testing Infrastructure » DrupalCI: Test Runner

Moving to the testrunner queue.

webflo’s picture

webflo commented

This is the actual error message from Drupal CI.

22:25:08   - Installing phpdocumentor/reflection-docblock (2.0.4)
22:25:08     Downloading: Connecting...
22:25:08 Could not fetch https://api.github.com/repos/phpDocumentor/ReflectionDocBlock/zipball/d68dbdc53dc358a816f00b300704702b2eaff7b8, please create a GitHub OAuth token to go over the API rate limit
22:25:08 Head to https://github.com/settings/tokens/new?scopes=repo&description=Composer+on+ffd2900619fc+2015-09-23+2208
22:25:08 to retrieve a token. It will be stored in "/root/.composer/auth.json" for future use by Composer.
jthorson’s picture

jthorson commented

Perhaps this could be an optional step in drupalci init, which could help step people through the process.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

(Commenting as a total noob WRT Composer, as just a mostly passive observer.)

Composer uses GitHubs API to retrieve Repo data and Tarballs (in Composer prefere-dist mode). GitHub API Rate Limits for unauthenticated requests are very low. We will hit the limit in Drupal CI easily.

But Drupal's canonical repo is not on GitHub. So… isn't the actual solution to not use GitHub? To make Composer hit d.o's infra?

webflo’s picture

webflo commented

The issue is about third party dependencies like symfony, phpunit etc. these libraries are hosted on GitHub.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Issue summary: View changes

Ahhh! Thanks for the clarification :) Clarified the IS too.

joshtaylor’s picture

joshtaylor commented

If you remove the vendor subfolders, you could then run something like:

composer install --prefer-source -n --verbose
mradcliffe’s picture

mradcliffe
he/him
Primary language English
commented

I did that on github until composer manager changed, and it worked, but it was also much slower.

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented
Title: Setup GitHub OAuth for Composer » Setup GitHub OAuth for Composer on Testbots
mradcliffe’s picture

mradcliffe
he/him
Primary language English
commented

Going to correct myself a bit. I think the slowness was to due to some complex dependency chains. With some refactoring, I'm not seeing that slowness anymore.

isntall’s picture

isntall commented
Status: Active » Reviewed & tested by the community

This has been accomplished, after many debates, by adding the token to the job before set -uex

This is not a great solution but it allows for the token to be set on a per job basis and allows the token to be rotated.

In the future we'd like to have the token set once the testrunner has been turned on.

isntall’s picture

isntall commented
Issue tags: +Needs deployment
isntall’s picture

isntall commented
Assigned: Unassigned » isntall
Status: Reviewed & tested by the community » Fixed
Issue tags: -Needs deployment

This has been merged into dev, production and is now live.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.