Comment from SeanBannister on Google+ (limited access), but moving here to keep track and share: "Yeah, I was thinking about writing install scripts for each OS (XP, Vista, 7, 8, Ubuntu, Mac OS) that'd install a large set of browsers (to be determined). Plus when the VM was launched it'd launch all the browsers. I was also thinking about writing some code for the VM host that would watch for a new commit launch the VMs and then shut them down once the commit has been tested."

CommentFileSizeAuthor
#3 windows.vbs_.txt2.12 KBSeanBannister
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

attiks’s picture

attiks CreditAttribution: attiks commented

For the moment we have TestSwarm running on one of our servers, both Drupal and TestSwarm are updated every 30 minutes using the following script

cd /var/aegir/platforms/h001/testswarm003
git pull
LATESTHASH=`git log --pretty=format:'%H' -n 1`
cd /var/aegir/platforms/h001/testswarm003/sites/testswarm003.h001.attiks.com

drush updb -y
drush vset testswarm_githash $LATESTHASH -y
drush cc all

cd /var/aegir/platforms/h001/testswarm003/sites/all/modules/testswarm
git pull

cd /var/aegir/platforms/h001/testswarm003/sites/testswarm003.h001.attiks.com

drush updb -y
drush cc all
attiks’s picture

attiks CreditAttribution: attiks commented

Another possibility for the VM's is to have a look at http://www.browserstack.com/automated-browser-testing-api

SeanBannister’s picture

SeanBannister CreditAttribution: SeanBannister commented
FileSize
windows.vbs_.txt2.12 KB

BrowserStack looks interesting and the pricing is very reasonable. However it does concern me that their pricing isn't based on actual usage, could they really handle us testing every Drupal.org project on the $19 a month plan? Apart from that its certainly more affordable than buying a license of Mac OS X, Win XP, Vista, 7, 8. However on looking at the Operating Systems they support it doesn't list multiple versions of Windows so I'm not sure if they actually provide this. This is important as some browsers perform differently under different versions of Windows.

Either way I'm keen to run my own testing environment. I started writing an install script for Windows and realised it might be more effort than its worth and I'm probably better off writing an install guide. I ran into a number of issues, and while none of them were show stoppers they weren't ideal:

  • The only language that has a low level of entry (and doesn't need to be compiled) and runs across all versions of Windows is VBscript. And it's not ideal. We could move to a compiled language but I'm a web developer and exe's scare me :)
  • As a side note: VBScript is supported but no longer developed by Microsoft so it may disappear from Windows in a future version, the replacement is Power Shell.
  • The installs for some of the browsers can't be installed silently so the user still has to manually install the application, so it really isn't automating much or saving much time.
  • Its going to be hard getting people to maintain code in a language the Drupal community isn't familiar with and for code that most people will only use once every year or so.

So I really didn't get far into the VBScript but I've attached the basic code I came up with just in case someone wants to run with it. I just really felt the maintenance hassle was going to be higher than the convenience but if someone else feels differently please run with this.

So I decided to start writing an install guide, and work on some boot scripts that launch all the browsers when the VM launches.

So I've decided to start with IE, presuming it would be the hardest and it hasn't disappointed. I started researching and came to the conclusion along with this article that the best way to test multiple versions of IE under a single operating system is using Utilu IE Collection but the problem with it is it won't install IE7 and IE8 under Window 7 or Vista instead it tells you to use IE9 and switch rendering engines but this doesn't provide a 100% accurate representation of IE7 and IE8, plus I can't even figure out a way to launch IE with a different rendering engine you have to manually do it. Anyway I've shot of a question to the guy who wrote that article to see if he can give us a hand.

attiks’s picture

attiks CreditAttribution: attiks commented

I've sent an email to browserstack asking for details on the OS. Regarding the pricing, jQuery is being sponsored by them.

Using VBScript isn't really an option, AFAIK it isn't installed by default on Win7. The only way to test this is to have separate browsers, so no emulation, for IE this means separate installs or separate VM's. I have no idea how Safari, FF, Chrome behave on different installs of (Win) OS's, but assume there will be differences as well.

to be continued ....

SeanBannister’s picture

SeanBannister CreditAttribution: SeanBannister commented

Yeah I noticed jQuery uses them but I guess they're only testing their main project not contrib projects as well?? Whereas if Drupal takes this up we have 1000's of contrib projects to test which I presume would add significant load.

I'm really hoping they come back and say they support multiple versions of Windows.

Addy Osmani got back to me about running IE Collection and says they now run multiple VMs instead https://twitter.com/#!/addy_osmani/status/182915347102433280

That'd be a complete nightmare to handle all those VMs.

attiks’s picture

attiks CreditAttribution: attiks commented

I think we first have to sort out everything for core, in a second phase we can look at contrib.

BrowserStack runs the IE's on Windows Server 2003 & Windows Server 2008, and their API only supports Windows for the moment, I asked them to get access to the APi so we can run some tests.

Regarding the multiple VM's, in theory we only need to set them up once, open the browser and keep them pointed at http://testswarm003.h001.attiks.com/testswarm-browser-tests