Comments

xjm’s picture

Status: Active » Needs review
Issue tags: +Needs backport to D7

So this a lot more than I thought it would be, but I couldn't think of another way to make it backportable.

I have not proofread this patch and I do not guarantee that it is not typo-ridden, incorrect, buggy, inconsistent, or on fire.

Help me testbot! You're my only hope.

xjm’s picture

StatusFileSize
new23.25 KB

NORLY

xjm’s picture

Status: Needs review » Needs work
+++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.phpundefined
@@ -2325,9 +2353,10 @@ abstract class WebTestBase extends TestBase {
+      'Pattern "@pattern" not found', array('@pattern' => $pattern),

Missing format_string() here. That should fail prettily.

+++ b/core/modules/simpletest/lib/Drupal/simpletest/WebTestBase.phpundefined
@@ -2498,19 +2529,18 @@ abstract class WebTestBase extends TestBase {
+      $default_message = t('Found field with name @name', array(
...
+      $default_message = t('Found field with name @name and value @value', array(

@@ -2580,7 +2625,11 @@ abstract class WebTestBase extends TestBase {
+      t('Checkbox field @id is checked.', array('@id' => $id)),

Missed a t() here, at least.

xjm’s picture

Status: Needs work » Needs review
StatusFileSize
new1.86 KB
new23.24 KB

With issues in #3 fixed. Still haven't proofread thoroughly.

Status: Needs review » Needs work

The last submitted patch, assert-1601202-4.patch, failed testing.

xjm’s picture

(Edited.)

So, this is interesting.

  1. SimpleTest inserts the assertion messages into a database table when it runs. The first fail above is on:
    $this->assertEqual($out, $expected, 'CJK tokenizer worked on all supplied CJK characters');
    

    where $out and $expected have (as one might infer) a bunch of CJK unicode. In current core, that data is eliminated from the message, but with the patch, the DB chokes on this:
    PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\xA0\x80\x80 \xF0...' for column 'message' at row 1: INSERT INTO {simpletest} (test_id, test_class, status, message, message_group, function, line, file) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7); Array ( [:db_insert_placeholder_0] => 7 [:db_insert_placeholder_1] => SearchTokenizerTestCase [:db_insert_placeholder_2] => pass [:db_insert_placeholder_3] => Value '一 盨 鿏 㐀 䃠 䶿 豈 切 﫿 ᄀ ᆀ ᇿ ꥠ ꥰ ꥿ ힰ ퟘ ퟿ ㄱ ㅠ ㆎ a n z a n z ヲ ᄀ ᅵ 가 쇘 힯 ぀ ば ゟ ァ バ ヿ ㇰ ㇸ ㇿ

  2. The failing database tests connection appear to be due to the fact that var_export() blows up (recursing somehow?) when used on a database connection.
  3. The CSS unit tests and line break filter test give a weird message that seems to imply the DB table is missing; I haven't debugged that further.
  4. The image field display tests, image style tests, and OpenID tests all appear to be throwing the same "invalid multibyte sequence" warnings when binary data is used as a test value in the assertion.
  5. The aggregator feed assertion warning is coming from:
    $this->assertResponse(array(200), t('URL !url is accessible', array('!url' => $edit['url']))); 
    
  6. The XMLRPC tests are failing because the message is too long to insert.

#1 and #7 could potentially be fixed in the assertionMessage() method, but the others are happening upstream in the assertions when trying to generate the default messages from the passed data. This would be easier to fix if assertions were objects.

xjm’s picture

Status: Needs work » Needs review
StatusFileSize
new668 bytes
new23.3 KB

So this should eliminate some but not all of the failures.

Status: Needs review » Needs work

The last submitted patch, assert-1601202-7.patch, failed testing.

xjm’s picture

Hmm. So #3 and #4 ran on 803 and got failures I can reproduce locally, but #7 ran on 654 and had different failures, including update path test stuff again. Retesting that.

Edit: So the "invalid multibyte sequence in argument" errors appear to happen on 803 but not 654.

xjm’s picture

Failure #6 in my list is actually a bug in assertResponse():

  protected function assertResponse($code, $message = '') {
    $curl_code = curl_getinfo($this->curlHandle, CURLINFO_HTTP_CODE);
    $match = is_array($code) ? in_array($curl_code, $code) : $curl_code == $code;
    $message = $this->assertionMessage(
      format_string(
	'HTTP response expected !code, actual !curl_code',
        array('!code' => $code, '!curl_code' => $curl_code)
      ),
      $message
    );
    return $this->assertTrue($match, $message, t('Browser'));
  }

The assertion accepts either an array or a string, but when passed an array, it tries to just print the array as a string argument.

xjm’s picture

StatusFileSize
new4.85 KB
new24.27 KB

The attached fixes #10 and also addresses the resource ID issue.

xjm’s picture

Status: Needs work » Needs review
xjm’s picture

Status: Needs review » Needs work
+++ b/core/includes/common.incundefined
@@ -5097,6 +5097,28 @@ function drupal_valid_token($token, $value = '', $skip_anonymous = FALSE) {
 
+
+/**

Extra blank line there.

xjm’s picture

StatusFileSize
new4.85 KB

Not sure where the interdiff went, but here it is.

xjm’s picture

xjm’s picture

StatusFileSize
new3.88 KB
new28.15 KB
new87.26 KB

With workarounds for the handful of assertions in core that are not compatible with var_export() because of circular references, plus the CJK thing which I couldn't figure out how to solve. Unfortunately I think the options are to either add a more sophisticated variable exporter, or not to backport this.

Screenshot showing what some test results look like with this patch:
assertions.png

xjm’s picture

Status: Needs work » Needs review
sun’s picture

Status: Needs review » Needs work

The Database Connection test failures pretty much clarify why this cannot fly.

It is simply not possible to dump each and everything you pass into an assertion method.

Furthermore, I'm strongly opposed to the fundamental idea here. Developers need debugging details in case things go wrong. But not a mixture of poorly written custom assertion message novels, followed by an enforced variable dump garbage.

Almost all of the custom assertion messages are way too long in the first place. At some point, we started to write entire paragraphs of messages that try to explain the whole world and context. But an assertion message should state the expectation only.

Most tests were written before #582364: Add sensible defaults to SimpleTest assert messages landed, and I can only guess that the guidelines for how to write tests have never been updated accordingly.

So what this change effectively will do is to encourage people to continue to write and use completely bogus custom assertion messages, instead of writing better tests.

What we actually need is people to write proper tests. And we need to provide methods that are smart enough to support people to write proper tests.

sun’s picture

That screenshot nicely shows what I mean: Duplicate assertion messages all over the place.

Printer friendly title found. ('0 - SimpleTest test node 3t8hoihg' found)

Duplicate every single assertion message? No thanks. Why didn't the author use a proper assertion message in the first place?

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mile23’s picture

Issue summary: View changes
Status: Needs work » Closed (works as designed)

PHPUnit-based tests show you the assertion result and also the optional message, but not when you run them from run-tests.sh with --browser.

Calling this works-as-designed, but obviously re-open as desired.