We should be writing unit-like tests to ensure our hook_project_src_info() and hook_project_src_releases() implementations return the information we expect. This could be difficult because we need to "inject" the GitHub API. We might be able to "mock" it by explicitly keeping GitHub API disabled in the test case and writing our own github_api_client() function in a test-only submodule to return our own mock GitHub API object.

Also, we may as well do end-to-end testing by doing the same above, but doing a series of xpath queries against the returned XML, checking the content type of the download files.

After this is complete, we should roll a beta.

Comments

iamEAP’s picture

Status: Active » Needs review
StatusFileSize
new12.77 KB

Adding tests for snapshot releases. Still much to do (see below). Still, marking needs review to ensure the actual testing methodology (which is pretty wonky) works with TestBot. Tests seem to be working on local.

Things this covers

  • Valid XML content type headers for a given project in 6.x and 7.x and 8.x (the last of which we intentionally leave undefined),
  • A branch name that doesn't adhere to standard release branch naming conventions is not included (e.g. master),
  • The "no release history" message is shown when no releases are defined for a given API version,
  • Project title, short name, and creator are properly pulled from the GitHub API,
  • The number of specified releases matches what's returned by the API,
  • That specified dev releases correspond to what's returned by the API (valid branch, valid name/tag values, major version, extra version, download link, and release date).

Things this doesn't cover, but that it should

  • Some project metadata is not validated (api version, recommended/supported/default major version, status, and link); see @todos,
  • Some release metadata is not validated (release link and status); see @todos,
  • No work at all has gone into verifying file XML structure or actual file content types/sizes,
  • No work at all has gone into verifying any non-dev (e.g. properly tagged) releases and associated release xml metadata validation,
  • No work at all has gone into validating that tagged releases are properly saved to the managed file system,
  • No work at all has gone into ensuring that .info metadata is added when tarballs are re-packaged.

Status: Needs review » Needs work

The last submitted patch, 1: project_src_github-add_automated_tests-2129369-1.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 1: project_src_github-add_automated_tests-2129369-1.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review
StatusFileSize
new12.43 KB

Was hoping a push to the 7.x-1.x branch (with a blank test file) would force a rebuild of this module's dependencies in TestBot, but that may have broken in upgrading to 7.x. I also can't manually trigger a branch-based test. Hopefully will take care of itself by tomorrow.

Patch is virtually identical to #1, just up to latest HEAD (which includes the blank test file I added + the .info changes required).

Status: Needs review » Needs work

The last submitted patch, 5: project_src_github-add_automated_tests-2129369-5.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 5: project_src_github-add_automated_tests-2129369-5.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 5: project_src_github-add_automated_tests-2129369-5.patch, failed testing.

iamEAP’s picture

Version: 7.x-1.0-alpha3 » 7.x-1.x-dev
iamEAP’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 5: project_src_github-add_automated_tests-2129369-5.patch, failed testing.

iamEAP’s picture

Quoting my list of todos and crossing off what's done in this patch. Still some to go.

Things left to do

  • Some project metadata is not validated (api version, recommended/supported/default major version, status, and link)
  • Some release metadata is not validated (release link and status)
  • No work at all has gone into verifying file XML structure or actual file content types/sizes,
  • No work at all has gone into verifying any non-dev (e.g. properly tagged) releases and associated release xml metadata validation,
  • No work at all has gone into validating that tagged releases are properly saved to the managed file system,
  • No work at all has gone into ensuring that .info metadata is added when tarballs are re-packaged.

The remaining tests are more directly file-related, so I may break them off into a separate class, or at least a separate testmethod.

Status: Needs review » Needs work

The last submitted patch, 14: project_src_github-add_automated_tests-2129369-14.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 14: project_src_github-add_automated_tests-2129369-14.patch, failed testing.

jtwalters’s picture

Wish I could help somehow but I've never written tests before. :)

iamEAP’s picture

iamEAP’s picture

Status: Needs work » Needs review
StatusFileSize
new17.92 KB

Here's #14 again with latest head.

The last submitted patch, 14: project_src_github-add_automated_tests-2129369-14.patch, failed testing.

iamEAP’s picture

Status: Needs review » Needs work

After playing around on Travis, I think I've determined the solution to the problems that occur here: https://travis-ci.org/iamEAP/project_src_github/builds/14124616

drush test-run 'Project Source GitHub' --uri=http://127.0.0.1:8080
Watchdog: sites/default/files/simpletest/803869/project-src-github/drewpaul_inc/drewpaul_inc_module/6.x-1.x/download.tar.gz | severity: 4 | type: page not found | uid: 0 | 127.0.0.1 | http://127.0.0.1:8080/sites/default/files/simpletest/803869/project-src-github/drewpaul_inc/drewpaul_inc_module/6.x-1.x/download.tar.gz |  | 

Apparently the actual public file system according to the test environment is at sites/default/files, while the parent environment is at /sites/default/files/simpletest/###. We can manually delete the "file_public_path" variable in the test set up to force it to properly handle tar files.

With that in mind, we just need to take care of the following remaining tasks:

  • No work at all has gone into validating that tagged releases are properly saved to the managed file system,
  • No work at all has gone into ensuring that .info metadata is added when tarballs are re-packaged.
iamEAP’s picture

Let's see how testbot likes these file tests. If it works, we'll just have to ensure that our .info file package changes are working as expected.

Status: Needs review » Needs work

The last submitted patch, 23: project_src_github-add_automated_tests-2129369-23.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review
StatusFileSize
new33.62 KB

Gotta add a --binary flag when generating the diff.

Status: Needs review » Needs work

The last submitted patch, 25: project_src_github-add_automated_tests-2129369-25.patch, failed testing.

iamEAP’s picture

Did not completely implement the "project_src_github_domain" variable across all tests. Fix here.

Next up, .info packaging verification.

iamEAP’s picture

And finally, adding checks for tagged release / branch release .info file packaging.

Status: Needs review » Needs work

The last submitted patch, 28: project_src_github-add_automated_tests-2129369-28.patch, failed testing.

iamEAP’s picture

Guessing TestBot doesn't like to file_get_contents() remote URLs. We'll try drupal_http_request().

Status: Needs review » Needs work

The last submitted patch, 30: project_src_github-add_automated_tests-2129369-30.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review
StatusFileSize
new37.56 KB

Status: Needs review » Needs work

The last submitted patch, 32: project_src_github-add_automated_tests-2129369-32.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review
StatusFileSize
new37.56 KB

Need to use the getAbsoluteUrl() method rather than appending the path to $GLOBALS['base_url'].

Should do the trick.

Status: Needs review » Needs work

The last submitted patch, 34: project_src_github-add_automated_tests-2129369-34.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review
StatusFileSize
new37.62 KB

May as well do some debug while the testbot queue is lean.

Status: Needs review » Needs work

The last submitted patch, 36: project_src_github-add_automated_tests-2129369-36.patch, failed testing.

iamEAP’s picture

Status: Needs work » Needs review
StatusFileSize
new37.65 KB

Issue appears to be that the .info file contains the drupal release history URL as if it clean URLs weren't enabled. Adding a manual variable set on that variable here.

iamEAP’s picture

Status: Needs review » Fixed

Good start for automated tests. We'll see if any additional need arises. Committed db1fbc7

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.