Problem/Motivation

When making changes to CSS files, it's very easy to introduce visual regression that are hard to pick up. CSS does not have an explicit scope, so unintentional changes can appear in unexpected places. Changes can also be very subtle, as opposed to functional tests, where they either work or they don't.

When a patch is manually tested, visual regressions can be easy to miss.

Proposed resolution

There are several command line tools such as PhamtomCSS and Wraith that can be used to automate visual regression testing and produce reports.

Comments

LewisNyman’s picture

Title: Discuss automatic visual regression testing » Discuss automated visual regression testing
Component: Seven theme » CSS
cosmicdreams’s picture

I'm guessing that this would require setting up some tasks for our CI system. It may be nice to have the option to run these tests or not.

LewisNyman’s picture

Issue tags: +Seven, +Bartik

Yes, it would be ideal to request to have this suite run, here's how I imagine it working from a users perspective:

  1. Upload a patch
  2. Request the visual regression test suite
  3. Maybe add to or modify the list of pages to compare?
  4. The results return a list of pages that have changed visually between HEAD and the patch.
  5. The user can drill into detail on the changes on the page

I've been playing PhantomCSS recently, I've written a basic test suite that logs in to Drupal as admin and snap a few pages. Automating a Drupal checkout and applying a patch might be a bit beyond me.

LewisNyman’s picture

Issue summary: View changes

Typos

YesCT’s picture

Issue summary: View changes
Issue tags: +Testing system
ti2m’s picture

A bit of the general discussion about various tools has accidentally moved into #2229187: SiteEffect: Automated frontend regression testing

Tools that have been mentioned:

+ PhantpmCSS, resemble.js, which can all be found on the general overview on http://csste.st/tools/

Anonymous’s picture

This could be really useful.

There are tons of tools for this now; I'm quite intrigued by PhantomFlow (nascent, but seriously cool) and I've played with DalekJS which has a really nice syntax for user interactions.

That said, I think the scope of this ought to be fairly limited to keep it simple. At a minimum, the tested sites need to be logged-in if requested, since most of the visual changes that need to be tested are for the admin interface.

I'm thinking about how to make a little proof of concept app (standalone rather than mucking about with Testbot)

User
- Upload a patch
– Select whether to log the admin user in
- Enter a list of URLs (e.g. /admin/reports)

App
– Spin up two locally-accessible sites with 8.x head
- Apply patch to the second site
- Log in the admin user
- Use PhantomCSS or Wraith (probably the former since npm)
- Destroy the sites
- Return a zip file with the output images

Perhaps I can borrow from https://github.com/patcon/simplytest/tree/travis-ci/scripts ?

attiks’s picture

Two other alternatives:

  1. grunt photobox, use phantomjs so the workflow i somerhing like: install drupal from HEAD, make first round of screenshots, apply patch and make second round of screenshot. It uses canvas or ImageMagick for the comparison.
  2. Since we are using behat, you could incorporate it. we use behat in combination with phantomjs and you can create screenshots, only step missing is a comparison of the before and after images.
TechNikh’s picture

People interested to discuss more on this, Consider coming to the "Automated frontend regression testing" BOF at Drupalcon https://austin2014.drupal.org/bof/automated-frontend-regression-testing
Room: 10B - Blink Reaction | 3rd floor
Time slot: Tuesday · 11:45-01:00

ti2m’s picture

I came across two major problems while testing D8 core with siteeffect, which aren't specific to siteeffect:

  1. On a plain install we e.g. don't have a node and therefore no node edit form. If something is off there we simply won't detect it. Same accounts for all kind of modules that need some extra config for more options to show up. catch referred me to #913086: Allow modules to provide default configuration for running tests, but as long as this is not usable we have to think about a default testing profile or something like that. It's core specific and doesn't apply to project sites in my opinion.
  2. On the other hand we have e.g. a tone of menu edit forms which will cause a huge test fail mess if anything changes in the menu form implementation. I have solved that for siteeffect with a build in crawler, which collects new urls on the fly. These are being send to Drupal in batch and a special module gets the menu router path for each url and returns it. That enables me to track all menu edit forms back to a single menu callback and only test one of the menu edit forms. For real world projects you probably want to test all nodes, but not all menu items, so you can decide per menu router path if you want to check all found urls of the menu router path or just one. If anyone is interested I can push the code somewhere, implemented the 'url to router path' function in D7 and D8. It's also language aware.

Would be really interested in ideas on how to approach (1) in an efficient way.

b2f’s picture

Hello,

I released an alternative CSS regression testing tool, it's called SUCCSS. You might give it a try.

It is easier to use than PhantomCSS and can use CasperJS interactions.

PieterDC’s picture

@ti2m his presentation this morning at Drupal Dev Days Montpellier drew my attention to this topic.
He already did quite some work on SiteEffect, see #2229187: SiteEffect: Automated frontend regression testing.
It looks a bit like http://crossbrowsertesting.com/screenshot-comparisons, but that's not open source.

'Depicted' on the other hand, is open source; https://dpxdt-test.appspot.com/
And it looks quite cool: https://www.youtube.com/watch?v=1wHr-O6gEfc
Should we join forces with them? https://github.com/bslatkin/dpxdt

PieterDC’s picture

A couple of months later, we (at Paddle) decided to give Shoov.io a go because it integrates with BrowserStack or Sauce Labs allowing you to test on more than only WebKit browsers through PhantomJS. But also test on Firefox and Internet Explorer.

In contradiction to:

Drulenium may have some good ideas (only show differences, in percentage and in GIFs), but it relies to much on Drupal for configuring. We need stuff (config, tests, baseline images) that live in version controlled repositories to plug into Continuous Integration.

Both BrowserStack and SauceLabs are free for Open Source projects like Drupal, so, that shouldn't hold the Drupal project back for using Shoov.io, AFAIK.
Shoov and all its sub-components are open source!

TechNikh’s picture

Update on Drulenium module: https://www.drupal.org/project/drulenium
I have updated the Drulenium module to take snapshots using SauceLabs, BrowserStack (https://www.browserstack.com/) & local selenium server.

As a prototype, I have setup http://drulenium.org/ which contains visual testing report of every git commit made to Drupal 8.0.x branch.

This module has Drush support http://cgit.drupalcode.org/drulenium/tree/drulenium_visual_regression/dr...
and also integrates with Aegir through https://www.drupal.org/project/hosting_drulenium & https://www.drupal.org/project/provision_drulenium

I have given a talk about this at the NYC meetup http://drulenium.org/blog/drulenium-drupal-nyc-meetup-february-3rd-2016
I am also coming to the NYC camp this Saturday http://www.drupalcamp.nyc/ if we want to discuss more on this.

  1. Drulenium module can take screenshots by logging in as administrator.
  2. The module can perform actions like filling textboxes, clicking buttons, hovering over elements.. before taking screenshots http://drulenium.org/reports/drupal8/release/36?threshold=-1
  3. Drulenium module can take screenshot excluding certain section of the page by filling the element with red color. http://drulenium.org/reports/drupal8/release/34 Snapshot with excluded element

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Ivan Berezhnov’s picture

Issue tags: +CSKyiv18

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.