Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
Problem/Motivation
When making changes to CSS files, it's very easy to introduce visual regression that are hard to pick up. CSS does not have an explicit scope, so unintentional changes can appear in unexpected places. Changes can also be very subtle, as opposed to functional tests, where they either work or they don't.
When a patch is manually tested, visual regressions can be easy to miss.
Proposed resolution
There are several command line tools such as PhamtomCSS and Wraith that can be used to automate visual regression testing and produce reports.
Comments
Comment #1
LewisNyman CreditAttribution: LewisNyman commentedComment #2
cosmicdreams CreditAttribution: cosmicdreams commentedI'm guessing that this would require setting up some tasks for our CI system. It may be nice to have the option to run these tests or not.
Comment #3
LewisNyman CreditAttribution: LewisNyman commentedYes, it would be ideal to request to have this suite run, here's how I imagine it working from a users perspective:
I've been playing PhantomCSS recently, I've written a basic test suite that logs in to Drupal as admin and snap a few pages. Automating a Drupal checkout and applying a patch might be a bit beyond me.
Comment #3.0
LewisNyman CreditAttribution: LewisNyman commentedTypos
Comment #4
YesCT CreditAttribution: YesCT commentedComment #5
ti2m CreditAttribution: ti2m commentedA bit of the general discussion about various tools has accidentally moved into #2229187: SiteEffect: Automated frontend regression testing
Tools that have been mentioned:
+ PhantpmCSS, resemble.js, which can all be found on the general overview on http://csste.st/tools/
Comment #6
Anonymous (not verified) CreditAttribution: Anonymous commentedThis could be really useful.
There are tons of tools for this now; I'm quite intrigued by PhantomFlow (nascent, but seriously cool) and I've played with DalekJS which has a really nice syntax for user interactions.
That said, I think the scope of this ought to be fairly limited to keep it simple. At a minimum, the tested sites need to be logged-in if requested, since most of the visual changes that need to be tested are for the admin interface.
I'm thinking about how to make a little proof of concept app (standalone rather than mucking about with Testbot)
User
- Upload a patch
– Select whether to log the admin user in
- Enter a list of URLs (e.g. /admin/reports)
App
– Spin up two locally-accessible sites with 8.x head
- Apply patch to the second site
- Log in the admin user
- Use PhantomCSS or Wraith (probably the former since npm)
- Destroy the sites
- Return a zip file with the output images
Perhaps I can borrow from https://github.com/patcon/simplytest/tree/travis-ci/scripts ?
Comment #7
attiks CreditAttribution: attiks commentedTwo other alternatives:
Comment #8
TechNikh CreditAttribution: TechNikh commentedPeople interested to discuss more on this, Consider coming to the "Automated frontend regression testing" BOF at Drupalcon https://austin2014.drupal.org/bof/automated-frontend-regression-testing
Room: 10B - Blink Reaction | 3rd floor
Time slot: Tuesday · 11:45-01:00
Comment #9
ti2m CreditAttribution: ti2m commentedI came across two major problems while testing D8 core with siteeffect, which aren't specific to siteeffect:
Would be really interested in ideas on how to approach (1) in an efficient way.
Comment #10
b2f CreditAttribution: b2f commentedHello,
I released an alternative CSS regression testing tool, it's called SUCCSS. You might give it a try.
It is easier to use than PhantomCSS and can use CasperJS interactions.
Comment #11
PieterDC@ti2m his presentation this morning at Drupal Dev Days Montpellier drew my attention to this topic.
He already did quite some work on SiteEffect, see #2229187: SiteEffect: Automated frontend regression testing.
It looks a bit like http://crossbrowsertesting.com/screenshot-comparisons, but that's not open source.
'Depicted' on the other hand, is open source; https://dpxdt-test.appspot.com/
And it looks quite cool: https://www.youtube.com/watch?v=1wHr-O6gEfc
Should we join forces with them? https://github.com/bslatkin/dpxdt
Comment #12
PieterDCA couple of months later, we (at Paddle) decided to give Shoov.io a go because it integrates with BrowserStack or Sauce Labs allowing you to test on more than only WebKit browsers through PhantomJS. But also test on Firefox and Internet Explorer.
In contradiction to:
Drulenium may have some good ideas (only show differences, in percentage and in GIFs), but it relies to much on Drupal for configuring. We need stuff (config, tests, baseline images) that live in version controlled repositories to plug into Continuous Integration.
Both BrowserStack and SauceLabs are free for Open Source projects like Drupal, so, that shouldn't hold the Drupal project back for using Shoov.io, AFAIK.
Shoov and all its sub-components are open source!
Comment #13
TechNikh CreditAttribution: TechNikh commentedUpdate on Drulenium module: https://www.drupal.org/project/drulenium
I have updated the Drulenium module to take snapshots using SauceLabs, BrowserStack (https://www.browserstack.com/) & local selenium server.
As a prototype, I have setup http://drulenium.org/ which contains visual testing report of every git commit made to Drupal 8.0.x branch.
This module has Drush support http://cgit.drupalcode.org/drulenium/tree/drulenium_visual_regression/dr...
and also integrates with Aegir through https://www.drupal.org/project/hosting_drulenium & https://www.drupal.org/project/provision_drulenium
I have given a talk about this at the NYC meetup http://drulenium.org/blog/drulenium-drupal-nyc-meetup-february-3rd-2016
I am also coming to the NYC camp this Saturday http://www.drupalcamp.nyc/ if we want to discuss more on this.
Comment #18
Ivan Berezhnov CreditAttribution: Ivan Berezhnov as a volunteer and at Drupal Ukraine Community for Levi9 commented