Running Panopoly's Behat tests locally (and adding your own!)

Behat is a tool for running automated behavioral tests on a website. You provide Behat with short scenarios for using the website, and it essentially opens a web browser and executes each of the steps (ie. click this link, fill in this field, press submit, etc) to make sure the website works as expected.

Panopoly (panopoly_test) includes a suite of Behat tests to make sure Panopoly's features are working as expected. Everytime we commit a change to Panopoly, the tests are run (on Travis-CI), so we can be sure that we haven't broken any features.

Unfortunately, at the moment, our Behat test suite doesn't cover 100% of Panopoly's features. Improving our test suite is one of the best ways you can contribute to Panopoly! Once a feature is covered by a test, we won't ever break it. This makes life easier for maintainers and allows us to commit patches and make releases faster.

Also, Behat's tests are NOT written in code, but a simple human-readable text format that sounds like a story! So, even non-coders can help. :-)

This page is about how to run the Panopoly Behat tests and the basics of adding or changing a test.

Note: Panopoly behat tests have been moved from the distribution (v1.6) to the panopoly_test module.

Installing Panopoly

For best results, you should have a working local installation of the latest available 7.x-1.x branch of Panopoly. You should have only generic or demo content (from the Panopoly Demo module), and don't make changes to permissions or other configuration.

See https://drupal.org/project/panopoly for download and installation instructions. If you want to run the latest bleeding-edge development code, see https://drupal.org/node/1717546.

Note: The panopoly_test module needs to be enabled to properly run all of the tests. This module is hidden from the admin/modules page, but it can be enabled via drush.

drush en panopoly_test

Installing dev and testing tools

Install Composer

cd path/to/sitename/profiles/panopoly/modules/panopoly/panopoly_test/behat
php -r "eval('?>'.file_get_contents('https://getcomposer.org/installer'));"
php composer.phar install

If you have already installed Composer, skip Step 2 and change Step 3 into:

composer install

Install Drush

If you don't already have drush installed, see the installation and usage instructions.

Install Selenium

• Selenium requires Java
• Get the latest version of Selenium Server at http://docs.seleniumhq.org/download/ (currently 2.53.0)
• Start your Selenium server (change the path to reflect where you downloaded it):
  java -jar /path/to/selenium-server-standalone-2.{latestVersion}.0.jar

Selenium produces a ton of output in your terminal window. If you are developing or debugging tests, you might need to see that output, but if you are just running tests, you probably don't. You can run Selenium in the background and discard all of its output by using this command:
java -jar /usr/local/lib/selenium-server-standalone-2.{latestVersion}.0.jar &>/dev/null &

Configuring tests for your local environment

Panopoly comes with a configuration file template named behat.template.yml. To configure it for your own setup, first copy the file to behat.yml:
cp behat.template.yml behat.yml

Then edit the file and change any parameters as needed. At minimum, you will need to change:

• base_url
  (e.g. http://panopoly.local)
• drush.root
  The path to the drupal root folder
  (e.g. /var/www/panopoly)
• drupal.drupal_root
  The path to the drupal root folder
  (e.g. /var/www/panopoly)

default:
  extensions:
    Behat\MinkExtension:
      base_url: http://mydrupalsite.com
      files_path: './profiles/panopoly/modules/panopoly/panopoly_test/behat'
    Drupal\DrupalExtension:
      drush:
        root: '/path/to/drupal/root'
      drupal:
        drupal_root: '/path/to/drupal/root'

Running Behat tests

From the panopoly_test/behat directory, you can run all tests using the default profile (i.e. using behat.yml):
bin/behat

More custom execution options

Behat features & custom configurations

Run one test by name:
bin/behat features/featurename.feature

Run a separate configuration file (i.e. for a local install vs. Travis-CI, etc.)
bin/behat --config behat.local.yml

More custom execution options, including running single scenarios, tag options.

Behat tagging

Behat tests can be tagged into groups. Tags affect what options from the config file are in effect. You can also use tags to run only a subset of the available tests.

For example, to include only tests that are tagged as 'javascript':
bin/behat --tags '@javascript'

To exclude tests that are tagged as 'javascript':
bin/behat --tags '~@javascript'

Any test that requires anything more than blackbox access (for example, any test that starts by creating a user and logging them in) should be tagged with @api so it can use drush or get direct access to internal Drupal functions.

Any test that requires browser interaction (for example, WYSIWYG editing or other Javascript functions) should be tagged with @javascript.

Behat profiles

Panopoly includes several tests that fail intermittently on Firefox due to issues in the Selenium browser driver. These tests will not run under the default profile. To see results, you have to run them using the profile provided for Chrome-only tests:

bin/behat --profile chrome

Note: In order to run tests on Chrome, you need to additionally download chromedriver and tell Selenium where it is when starting it, for example:

java -jar selenium-server-standalone-2.{latestVersion}.0.jar -Dwebdriver.chrome.driver=/path/to/chromedriver

More Behat test output options

To send the full output to a file while also 1) displaying a shorthand, progress bar of pass/fails on rules and 2) returning all the failed rules at the end:

bin/behat --format pretty --out ~/.fullOutput.out --format progress --out std

More about Behat 3.x test output formatting.

Troubleshooting

No ability to ... in Drupal\Driver\DrushDriver

While \Drupal\DrupalExtension supports Drush aliases for remote Behat testing, the Drupal API Driver must be run locally.

The Drupal API Driver is the fastest and the most powerful of the three drivers. Its biggest limitation is that the tests must run on the same server as the Drupal site.

The team that maintains Drupal.org also uses Behat to do automated testing. Here is a one of their documents with solutions to troubleshooting common Behat issues:

https://drupal.org/node/1783000

Also, feel free to open an issue in the Panopoly issue queue or ask for help in the #panopoly channel on IRC.

PHP Fatal error: Cannot redeclare class TestSubContext in

./bin/behat features/text_widget.feature
PHP Fatal error:  Cannot redeclare class TestSubContext in /Users/bwood/code/drupal/bwood/ARCH/openberkeley-drops-7-deep/profiles/openberkeley/modules/panopoly/panopoly_test/behat/steps/panopoly_test.behat.inc on line 13
PHP Stack trace:
PHP   1. {main}() /Users/bwood/code/drupal/bwood/panopoly_test/behat/vendor/behat/behat/bin/behat:0
PHP   2. Symfony\Component\Console\Application->run()
/Users/bwood/code/drupal/bwood/panopoly_test/behat/vendor/behat/behat/bin/behat:32

If you have symlinked panopoly_test into your panopoly installation
e.g.

ll profiles/panopoly/modules/panopoly/panopoly_test
lrwxr-xr-x 1 bwood 44 May 15 09:49
profiles/panopoly/modules/panopoly/panopoly_test ->
/Users/bwood/code/drupal/bwood/panopoly_test/

You may need to change the subcontext paths in your behat.yml:

subcontexts:
  paths:
    - "./profiles/panopoly"

And then you might want to use an absolute path to the panopoly
profile directory of your drupal root. Here's my corrected
behat.yml

imports:
    - behat.common.yml

default:
  extensions:
    Behat\MinkExtension:
      base_url: http://panopoly-dev.localhost:8082
    Drupal\DrupalExtension:
      drush:
        root: '/Users/bwood/code/drupal/bwood/panopoly-dev'
      drupal:
        drupal_root: '/Users/bwood/code/drupal/bwood/panopoly-dev'
      subcontexts:
         paths:
           - "/Users/jsmith/code/drupal/jsmith/panopoly-dev/profiles/panopoly"

Behat unable to load database settings when using Acquia Dev Desktop

Acquia Dev Desktop includes database settings in settings.php dynamically. When trying to run behat tests against a site running on Acquia Dev desktop, these settings aren’t loaded correctly.

Example Behat error message

[Drupal\Exception\BootstrapException]
  Missing database setting, verify the database configuration in settings.php.

Mac OS X Workaround

A workaround for this is to force Dev Desktop’s additional settings to be included in settings.php. To do this, add the following line in your settings.php file (for the site you which to test) above the // <DDSETTINGS> block inserted by Dev Desktop, replacing “{username}” with your actual Mac OS X username and "{sitename}" with the name of your DevDesktop site (you may have to navigate to the '~/.acquia/DevDesktop/DrupalSettings' directory to find the correct site name):

require_once '/Users/{username}/.acquia/DevDesktop/DrupalSettings/loc_{sitename}_dd.inc';

Still stuck with various execution errors? Try updating the Behat dependencies

If the versions of various dependencies included here in the docs or in the corresponding version of Panopoly Test aren't working, try using the latest versions of the following:

• Selenium Standalone Server
• ChromeDriver
• Drupal Behat Extension - Update panopoly_test/behat/composer.json to latest version.

Basics of writing Behat tests

Test files

Behat tests are written in plain English (Gherkin is the technical term for the business language used). Here's an example of a test file:

Feature: Add text widget
  In order to put additional text on a page (beyond the main content)
  As a site administrator
  I need to be able to add a text widget

  @api @javascript @panopoly_widgets
  Scenario: Add text to a page
    Given I am logged in as a user with the "administrator" role
      And Panopoly magic live previews are disabled
      And I am viewing a landing page
    When I customize this page with the Panels IPE
      And I click "Add new pane"
      And I click "Add text" in the "CTools modal" region
    Then I should see "Configure new Add text"
    When I fill in the following:
      | Title   | Text widget title       |
      | Editor  | plain_text              |
      | Text    | Testing text body field |
      And I press "Save" in the "CTools modal" region
      And I press "Save"
      And I wait for the Panels IPE to deactivate
    Then I should see "Text widget title"
      And I should see "Testing text body field"

Anyone can write a Behat test using predefined steps. To see a list of the available steps:
bin/behat -dl

You can get pretty far with just these step definitions. Be sure to tag your tests with @api or @javascript as needed.

Custom step definitions

TODO: Add (very basic) instructions for writing custom step functions.