Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.Travis has new infrastructure that should allow for some performance gains if we can port our tests to it:
http://docs.travis-ci.com/user/migrating-from-legacy
http://docs.travis-ci.com/user/workers/container-based-infrastructure/
The most exciting thing to me is ability to have cache directories. This should allow us to cache some of the setup we do between builds so that getting from the start of the build to the part where the tests are actually running is much shorter. That could shave 10-15 minutes off each test!
Things to put into Travis' cache:
- The Drush download cache
- The composer vendor/ (and bin/) directories
- The Google chrome sandbox binary
- Maybe Selenium? (if we're sure to include the version in the filename)
This will get both easier and harder (depending on the thing) with #2529428: Use Phing build system to run tests on Travis-CI
| Comment | File | Size | Author |
|---|---|---|---|
| #25 | panopoly-drupal-ti-2532134-25.patch | 23.05 KB | dsnopek |
| #17 | panopoly_test-drupal-ti-2532134-17.patch | 1.83 KB | dsnopek |
Comments
Comment #1
dsnopekStarted a list of stuff to cache in the issue summary.
Comment #2
dsnopekRight now we're blocked on Google Chrome being added to the APT whitelist:
https://github.com/travis-ci/travis-ci/issues/3964
Comment #3
Fabianx CreditAttribution: Fabianx as a volunteer commentedI will try to get https://github.com/LionsAd/drupal_ti running for distributions and panopoly is a great example, that will give you chrome without whitelist and sudo: false for free.
I have Google Chrome working with behat and selenium on sudo: false containers :).
drupal_ti in dev-master also has advanced caching logic that allows to avoid re-packing archives again and again.
( compared to travis' caching logic it uses rsync to compare and also excludes always changing files )
With that I have pure phpunit builds running in 19 seconds :).
Ultimately it will be your choice to use or not use it (obviously), but I want to support distributions anyway, so I will use panopoly as an example to work on.
Comment #4
dsnopek@Fabianx: That looks really cool! Our current process is actually structured quite similar at the outer-most level: we call a travis-ci.sh script with the current Travis-CI stage. So, I'd love to see what you come up with!
Also, I don't know how this affects things, but we're slowly working towards replacing all our scripts and build processes with Phing: #2529428: Use Phing build system to run tests on Travis-CI
So, in the end we'll probably want to run setup and run Behat and Selenium via Phing. Would drupal_ti allow for that?
Comment #5
Fabianx CreditAttribution: Fabianx as a volunteer commented#4: Yes, you can very easily specify your own custom runners.
All functionality is encapsulated in bash functions and you can easily add your own or even replace them.
e.g. you would then call in panopoly-simpletest/before_script.sh
drupal_ti_ensure_drupal
drupal_ti_ensure_module
drupal_ti_ensure_webserver
as in runners/simpletest/before_script.sh
BUT
in runners/panopoly-simpletest/script.sh you might just call your phing test process.
Or a mixture of that.
So you might use the setup of the webserver (and that was really hard work for PHP 5.3 / hhvm) and e.g. selenium with chrome that drupal_ti provides, but delegate other tasks like downloading the right modules to phing.
Comment #6
dsnopekSuper cool, thanks for the explanation! I look forward to seeing what this looks like for Panopoly. :-)
Comment #7
Fabianx CreditAttribution: Fabianx as a volunteer commentedhttps://github.com/panopoly/panopoly/pull/2
is the current work-in-progress (closed only so that travis does not need to run it once for my fork and once for panopoly, saving resources).
Battling random test failures due to the dreaded
"Selenium::WebDriver::Error::MoveTargetOutOfBoundsError: Element cannot be scrolled into view"
happening randomly.
Mostly happens when trying to activate the panels IPE, but had also happened already on modals (e.g. popup then Save).
Likely container based is a little faster and hence Firefox tries to click the button while the javascript is still being loaded.
Will ask jhedstrom for advice.
Comment #8
dsnopekI started reviewing these changes and for the most part this is looking awesome! However, before I got too far I got caught on the fact that Behat and Selenium are failing with Chrome 44: #2537902: Chrome tests failing on Travis-CI
The current (hopefully temporary) solution is to pin on Chrome 43. We might end up having to do something similar with drupal_ti in order to move to it, depending on the timing of when that problem gets fixed, versus how quickly the other things on this issue get resolved.
Comment #9
Fabianx CreditAttribution: Fabianx as a volunteer commented#8: Atm my chrome tests are still passing, but I am very happy to pin to a version or we can override the drupal_ti_ensure_chrome function easily.
Comment #10
Fabianx CreditAttribution: Fabianx as a volunteer commentedActually I will push a commit and will compare if tests still pass.
Stay tuned :).
Comment #11
dsnopekI tried a commit based on your branch earlier today to see if it failed with Chrome too and it did:
https://travis-ci.org/dsnopek/panopoly/builds/72456595
Chrome 44 was just released 3 days ago, so I think it just missed it. :-)
Comment #12
Fabianx CreditAttribution: Fabianx as a volunteer commentedI released drupal_ti 1.4.1 (https://github.com/LionsAd/drupal_ti/releases/tag/1.4.1), which makes it possible to specify the DRUPAL_TI_BEHAT_CHROME_URL, but in my tests pcbeta.com was not always reliable, so I had to refrain to make it the default.
(https://github.com/LionsAd/drupal_ti/pull/28)
I will update the PR shortly to see how things pin out now and also do a test on legacy infra for checking if the random test fails occur there as well.
Comment #13
dsnopek@Fabianx: Awesome, thanks!
Comment #14
dsnopekAwesome, thanks! Yeah, I'm seeing it being unreliable in the Panopoly tests too, but at least we're able to continue working. Looking at the Chrome issue, it seems like this will be fixed upstream soon-ish:
https://code.google.com/p/chromium/issues/detail?id=513768
... but at least we have a recourse for if it happens again!
Great! Thanks again for your work on this! I'm looking forward to seeing the next iteration, and digging into it deeper myself. At this point, I'm definitely for using drupal_ti, so hopefully we'll be able to work out any kinks and get this merged soon. :-)
Comment #15
Fabianx CreditAttribution: Fabianx as a volunteer commentedHi,
I am sorry for the lack of updates, but I got busy elsewhere.
I have started several builds here https://travis-ci.org/LionsAd/panopoly/builds to see which combinations work and which do not.
Thanks,
Fabian
Comment #16
dsnopekI did some testing with the latest version of Fabian's code, and did a little clean-up. Here's everything reduced into two patches that have been working in my testing!
EDIT: Here's a build of this: https://travis-ci.org/dsnopek/panopoly/builds/78301710 - Strangely, my last test build before this one failed on some of the Chrome tests, even though the previous ones succeeded. Let's see what this one does!
Comment #17
dsnopekHm, maybe I reduced Fabian's hacks to panopoly_test too far because now I'm seeing some random failures on "I customize this page with the IPE". Here's a new version of the panopoly_test patch, which I'll try out in a moment!
EDIT: Here's the build: https://travis-ci.org/dsnopek/panopoly/builds/78305751
Comment #18
dsnopekI just noticed that this comments out our scripts/check-overridden.sh, which we'll need to reinstate! I haven't really had much chance to really dig into the code for drupal_ti, which I should probably do just to make sure we're still doing all the things we used to be doing.
Comment #19
Fabianx CreditAttribution: Fabianx as a volunteer commentedI just did that to ensure tests run faster - that was pure debug and for reverting,
Comment #20
dsnopek#19: No worries! I was just making a note so I'd remember to do it in the next patch. :-)
Also, looking at my builds from yesterday, we're getting some random failures in the Chrome tests. :-/ I've seen some similar random failures on the last build with the old scripts when switching back to latest Chrome, so that might be related? Trying another build with Chrome 43 pinned again:
https://travis-ci.org/dsnopek/panopoly/builds/78382054
Comment #21
dsnopekSo, downgrading to Chrome 43 again helps to fix the flakiness in the Chrome tests. However, drupal_ti is much flakier with Chrome 44 than the old Travis script, and I think the reason is that drupal_ti is using an older chromedriver (version 2.13), while the latest version is 2.19, which specifically mentions "Changes include many bug fixes that allow ChromeDriver to work more reliably with Chrome 44+":
https://sites.google.com/a/chromium.org/chromedriver/downloads
Here's a PR to drupal_ti to always download the latest version of chromedriver:
https://github.com/LionsAd/drupal_ti/pull/47
Hopefully that'll help but I haven't had a chance to test it with the Panopoly build yet (because I need to figure out the composer incantation to pull my fork/branch).
EDIT: Here's a build with my drupal_ti fork (just used Git to pull it in) and the latest Chrome: https://travis-ci.org/dsnopek/panopoly/jobs/78392567
EDIT-2: Gah! Forgot to select the correct branch. Here's a new build: https://travis-ci.org/dsnopek/panopoly/jobs/78397655
Comment #22
Fabianx CreditAttribution: Fabianx as a volunteer commentedOh, thanks so much for the PR. Some little styling nits, then it will directly go in.
I hopefully can then re-enable chrome tests for drupal_ti itself, which would rock!
Comment #23
dsnopekGreat! Thanks for the review on the PR - I've updated it.
Also, my last build passed using my fork of drupal_ti! So, here's a patch that represents the most recent code. I think once we have a new release of drupal_ti with the chromedriver changes, this will finally "committable" but I'll probably do a little more exploring in the code before do it.
Comment #24
Fabianx CreditAttribution: Fabianx as a volunteer commentedMerged the PR tp drupal_ti, the patch looks great (Well, I mean the changes since my last PR / patch test)!
Thanks for the idea to update chrome driver!
For now you can probably use dev-master, though I can probably tag a version of drupal_ti later that has that fix.
Comment #25
dsnopekTested with :dev-master and everything worked fine:
https://travis-ci.org/dsnopek/panopoly/builds/78462803
Here's an update patch that does that, and also re-rolls for the 1.27 release. I think I'm just going to merge this. :-)
Thanks again, Fabian, for the excellent work on drupal_ti and integrating it with Panopoly!
Comment #26
dsnopekAnd committed. :-)
Comment #28
dsnopekAnd I already have a follow-up: #2562105: Add back PHP 5.5 and 5.6 tests!
I just noticed that this commented out the PHP 5.5 and 5.6 tests - probably also for debugging, but we need to add them back!