If you call $this->clickLink() and the link is not on the page, you'll get a PHP notice:
ERROR: Undefined variable: url_target on line 1857 in /opt/www/apisite/sites/all/modules/simpletest/drupal_web_test_case.php (Notice)
This is because $url_target is only defined here:
if (isset($urls[$index])) {
$url_target = $this->getAbsoluteUrl($urls[$index]['href']);
}
Simple fix is to say else { $url_target = '' } or something to that effect.
This is actually a pretty annoying bug because I have PHP notices printing and so if a clickLink fails, instead of the test finishing I get something like the attached screen shot.
I haven't checked to see if this is present in Drupal 7/8 core simpletest yet...
Comment | File | Size | Author |
---|---|---|---|
#97 | interdiff-d7-1452896_96.txt | 536 bytes | joshi.rohit100 |
#97 | d7-1452896_96.patch | 1.79 KB | joshi.rohit100 |
#95 | d7-1452896_95.patch | 1.79 KB | joshi.rohit100 |
#90 | interdiff.txt | 2.15 KB | Mile23 |
#90 | 1452896_90.patch | 6.53 KB | Mile23 |
Comments
Comment #1
jhodgdonI have just verified that this is still present in the Drupal 7 core version of Simpletest, and it's still just as annoying... so I'm moving this issue to core.
The code is the exactly the same in Drupal 8/7 and the contrib Simpletest module for D6. Here's a patch for D8 to start with.
Comment #2
jhodgdonJust as a note, this is primarily of interest to get fixed for contrib/core module developers who are writing and debugging tests, who have PHP notices turned on so they can make sure their code is clean. Or to people trying to port a well-tested contrib module from D6 to D7. :)
Instead of just having a one-line red test failure on the test results page, due to this link not being on the page where it's supposed to be (or you had a typo in your test and the link doesn't exist), you get a "TEST FAILED TO COMPLETE" failure.
Comment #3
tim.plunkettIn a case like this, I think a ternary is actually MORE readable.
Comment #4
mikran CreditAttribution: mikran commentedWouldn't it be better to have something like 'not found' as url_target so the message doesn't look like "Clicked link %label () ..."?
Comment #5
jhodgdonIt will already show up as a failed test. At this point I don't care a whole lot about the message, but perhaps 'not found' would be more sensible. I'm fine with the patch in #3...
Comment #6
mikran CreditAttribution: mikran commentedpatch #3 it is then
Comment #7
catchCan we add a test for this to simpletest.test?
Comment #8
jhodgdonGood idea to make a test for it. Seems like making the test might be a good Novice project? I might get to it if no one else takes it on, but not for a few days probably.
Comment #9
kid_icarus CreditAttribution: kid_icarus commentedSo this is my first attempt at writing a test!
After writing the test though, I was curious:
It felt strange writing this assertion. The behavior implemented in patch #3 is such that if clickLink() is passed a label that doesn't exist, it winds up returning a drupalGet() on an empty string, which is essentially $base_url. I'm not sure if this is the desired behavior, but my concern lies in tests that might assume clickLink() either returns drupalGet() on a link if it exists, or false if it doesn't.
I ran the test before and after applying patch #3 by using
git checkout
on drupal_web_test_case.php. The test (by itself) fails before patch #3, and passes afterwards.Review is greatly appreciated :)
Comment #10
kid_icarus CreditAttribution: kid_icarus commentedComment #12
jhodgdonI don't think this test is quite right (and of course it is also failing).
In the current (and patched) clickLink function, the code does this:
So if the link doesn't exist, it's not doing the drupalGet() at all. What I think you want to do is verify that the return value to clickLink is FALSE in the case of a nonexistent link, and perhaps also to verify that the URL of the page hasn't changed (i.e., use drupalGet to go to some page on the site, and then after clickLink verify the URL has not changed).
Comment #13
kid_icarus CreditAttribution: kid_icarus commented@jhodgdon, Thanks for the review.
Your assumption of how DrupalWebTestCase::clickLink works appears to me to be incorrect. I've used debug() to verify all values within the function when it is called with a bad link.
If DrupalWebTestCase::clickLink is passed a bad link, I've found the following sequence occurs:
DrupalWebTestCase::xpath() returns an empty array if the xpath query comes back with no results. Therefore $urls is assigned an empty array.
The ternary operation assigns an empty string, '', to $url_target
This assertion will always fail due to the fact that isset($urls[$index]) evaluates to FALSE because $urls is an empty array.
This condition will always evalutate to TRUE because according to php.net, isset($var):
Returns TRUE if var exists and has value other than NULL, FALSE otherwise.
An empty string is not NULL. Therefore DrupalWebTestCase::clickLink returns $this->drupalGet(''), which indeed is the HTML of the base URL. Using
if (!empty($url_target))
solves this last problem.There is a larger logic error which lies in the fact that DrupalWebTestCase::clickLink asserts that the link passed to it exists. One can test that DrupalWebTestCase::clickLink returns FALSE when given a bad link, however, in doing so the assetion within DrupalWebTestCase::clickLink itself will fail.
I'm assuming that one of the assertions must be removed. Attached is a new patch which addresses the issue of DrupalWebTestCase::clickLink not returning FALSE when passed a bad link. However, the aforementioned logic error still exists, and I'm not sure how to best handle it.
Comment #14
kid_icarus CreditAttribution: kid_icarus commentedComment #16
jhodgdonYeah, I guess due to (3) that probably a test to verify this will not ever pass, because if you pass a bad link in, the assertion will fail, thereby causing a failing test... But aren't there ways to test this? I mean, aren't there some tests for the simpletest module that test that assertions correctly fail?
One other note in your proposed patch: "existent" is misspelled. :)
Comment #17
Freso CreditAttribution: Freso commentedThis is a straight reroll (of #13) for the new directory/file structure - it doesn't work.
Comment #18
Freso CreditAttribution: Freso commentedAnd the actual reroll of the patch in #13... >_>
Comment #19
Freso CreditAttribution: Freso commentedAlright, so Cottser said this is likely no longer a "Novice" issue (I also removed "Quick fix" as it's turning out to be not so quick a fix anyway...). I'm not confident at all with the SimpleTest system to make any calls on this, so I'll just leave it here.
However, kid_icarus is right with regards to his point about the logic. If you run a test on clickLink'ing a bad link, you will get an error from clickLink(). However, removing that assertion in clickLink will also not reveal bad links in places where a bad link is not supposed to be.
Currently, I'm thinking to either have clickLink tage a paramenter, e.g.,
$badlink
, which could swap assertTrue with assertFalse if= TRUE
(defaulting to= FALSE
, to not break current tests (and also because that's the most likely scenario...)), or have the assertTrue check on something else; like $url_target or the $urls array - or perhaps just populate $urls with $index if it's empty? But then again, we do not want to open up for actually broken links... This is making my head hurt. :pThe two attached files are the same as the reroll from just before, only split up in the "fix" and its test. (No, they still don't work. :))
Comment #20
marthinal CreditAttribution: marthinal commentedWith this patch we could know if the link was correctly clicked.
Comment #21
jhodgdonI'm assuming that the test in this patch is going to fail, because it's explicitly trying to click a link that doesn't exist, and clickLink() is now testing for that.
Also, there's a coding standard problem in this patch: test assertion messages should not be wrapped in t() (they shouldn't be translated). Use format_string() instead.
I also don't think this is very good code:
The same assert is in both sides of this if() statement -- there's no reason for it to be within the if() at all, and it should be moved out.
Comment #22
marthinal CreditAttribution: marthinal commented@jhodgdon excuse about the if, I was trying different assertions and at the end I didn't verify very fine what I did. This is the same code but better.
I don't know if this test is the better option.Please let me know the best way to do this under your point of view.
upsss also this text is not correct in the patch 'Active URL persists after clicking non-existant link'
Comment #23
marthinal CreditAttribution: marthinal commentedLike Freso says in #19 'This is making my head hurt. :p' trying to understand the best way to test when clickLink fail correctly... :))
Comment #24
jhodgdonOK, I took a look at the tests that currently exist in the Simpletest module. In file core/modules/simpletest/lib/Drupal/simpletest/tests/SimpleTestTest.php, there *is* actually a way to test for a failing test. What it basically is doing in that test is that it runs a test called "Stub Test" in the testing UI, and then it verifies that the output is what it expects to see.
So I think we could:
- Add a line that would try a clickLink() on a non-existing link to the stubTest() method in that file.
- Add a line that verifies the correct assertion message in the confirmStubTestResults() method in that file.
Anyway... The patch in #22 looks like the right code fix to me. What I don't understand is how the added test passes, because it sure looks to me as though it's trying to click a non-existing link, and why is it passing? Maybe the added test isn't being run?
Comment #25
marthinal CreditAttribution: marthinal commentedHere we test if the link is correctly clicked (in this case we want that) and verify that the user login form isn't correctly validated so we are in the same url.
Also added the patch created in the previous comments.
Added a test to verify that a link that doesn't exist, is correctly no clicked.
Changed at assertTrue the t() with format_string.
Comment #27
jhodgdonMost of this patch looks great!
One thing I don't understand though. Can you explain this added test? It looks like the comments in the test are saying it's not a valid link, but you're expecting the test to pass... I just don't understand it. I think if you want to verify that clickLink works, you should test on some URL that would actually work?
Comment #28
marthinal CreditAttribution: marthinal commentedAt localhost the test pass correctly....
@jhodgdon about the test snippet ... What I want to do is to click correctly a link and check if the actual url is correct.
I really don't know the best possibility and also if we need it.
What do you think about it?
Comment #29
jhodgdonI'm inclined to say that we don't need a test to verify that clickLink() works correctly on an existing link, since there are thousands of correct uses of clickLink() in the body of tests. But testing that it fails correctly (without an exception), which is the subject of this issue, does seem like a good thing to test. :)
Speaking of which... The addition that you made should *not* add exceptions or debug output to the test output, should it? It seems like it ought to add exactly 1 test failure, because there is 1 bad clickLink() added to the tests... right?
Comment #30
marthinal CreditAttribution: marthinal commentedOk so i'll create another patch without clickLink() test ... let me time to check the fails and exceptions correctly I need to debug what's going on here :) It seems that clickLink is adding another fail more ...
Comment #31
jhodgdonI apparently changed the status incorrectly above... marthinal: thanks for working on this!
Comment #32
marthinal CreditAttribution: marthinal commentedComment #33
marthinal CreditAttribution: marthinal commented@jhodgdon:
We have more fails actually because when we try for example a drupalGet() from stubTest appear these fails:
Undefined index: backtrace
Notice errors.inc 188 _drupal_log_error() Exception
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'd8.simpletest211209semaphore' doesn't exist: SELECT expire, value FROM {semaphore} WHERE name = :name; Array ( [:name] => variable_init )
Debugging with debug() I can see why it appears and this is why:
'Tables_in_d8' => 'simpletest907255simpletest211209semaphore',
In the setup we add the first prefix and then another new (second prefix) + table name.
So the tables are created but when trying to use them it's not possible.
I fixed the problem in the this patch.The problem is .... I really don't know when exactly we could need to use the first prefix and then the second... like actually is in the code.
Actually seems with no side effects with this patch.
Also we have two debug messages more added when we go to 'node' and another when clicking a link.
Comment #34
marthinal CreditAttribution: marthinal commentedComment #35
jhodgdonHm. I don't think it's OK to change TestBase like this:
Maybe you were debugging and forgot to put this line back to its original? Hopefully...
Then in the added test code, this comment needs to be slightly reformatted:
Needs a space after // and I think I would say "Check that the clickLink failure was recorded" as the comment.
Other than that, looks great!
Comment #36
marthinal CreditAttribution: marthinal commentedPerfect. I will add the space in the comment.
About the first point :
Actually with $value['prefix']['default'] . $this->databasePrefix we have 2 fails at stubTest when trying to click a link or ... log a user or something. Exactly this error:
Undefined index: backtrace
Notice errors.inc 188 _drupal_log_error() Exception
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'd8.simpletest211209semaphore' doesn't exist: SELECT expire, value FROM {semaphore} WHERE name = :name; Array ( [:name] => variable_init )
Ok, so I can verify debugging that the semaphore table name is 'simpletest907255simpletest211209semaphore', while testing we ask for 'simpletest211209semaphore'. The method name is changeDatabasePrefix(). I see that it ' Changes the database connection to the prefixed one. '.
What I don't understand is why we need to have a table with this name 'simpletest907255simpletest211209semaphore' and when exactly we need to have it.
If I don't remove the first prefix '$value['prefix']['default'] . $this->databasePrefix' then I will always have the 2 unexpected fails. Also I am trying to do the same but without modifying TestBase ...
1) So I would like understand why and when we need two prefixes because at the moment I don't understand why...
2) Do you think that there's another way to do this ???
3) Maybe this is a bug because we are asking for a table that doesn't exist.
4) Modifying TestBase the test passes correctly.
Thanks for your help.
Comment #37
jhodgdonI don't know what's going on -- maybe one of the simpletest module maintainers can help us out?
Comment #38
marthinal CreditAttribution: marthinal commentedYes , I'll try in #drupal-contribute .
Comment #39
boombatower CreditAttribution: boombatower commentedSimpleTest tests itself using a browser test which means that it runs a test using the web runner and does so from a test. We must go deeper...etc etc, but yes it does.
The assertions for the inner test require a database even if the test itself does not, something clever perhaps, but I doubt that will be any cleaner then the way it works now. Although I do not see why we need to explicitly prepend both prefixes as it should just be recursive.
I have not read the issue in length or looked at patches, just responded to #36 as linked to from IRC. Let me know if this is helpful or need more details.
Comment #40
jhodgdonThe objective we really need help with here is how to add a test that verifies that clickLink() will fail (but without an exception as it is doing now) if a nonexistent link is clicked. The patch in #33 attempts to do this, but it made a change that I don't understand to how that test was running, and I'm not sure it's OK to change the test in that way (see #35). Apparently (see #36) the test doesn't pass without that change... so I'm still confused about why the test won't pass without that change, what the change really does, and whether it's OK to make that change.
Comment #41
webchickJust hit this myself while porting Pants module from D7 to D8. Marked #1089792: Undefined variable: url_target in clickLink as a duplicate of this bug.
Comment #42
star-szrJust unassigning to free up (and bump) this issue. @marthinal, if you still want to work on this feel free to reassign. Thanks!
Comment #43
martin107 CreditAttribution: martin107 commentedI am amazed by this discussion.
Can I convince people here that they have gone off track into a cul-de-sac?
I take a much simpler approach... if WebTestBase.php clickLink() is fed an invalid label it should abort with an exception to tell the test developer he/she is doing something wrong. Exceptions should be limited to when the routine cannot sensibly continue this is such a case.
That is what burn myself today and webchick a month ago and i guess jhodgdon ( OVER TWO YEARS AGO ) and nothing has been done!!!!!
Fundamentally this should be a ONE line fix...
I say again If clickLink() is given invalid input it should bark at the webtest creator appropriately.
Behind the scenes it is more than an one line fix if you include the @throw comment and a web test that feeds invalid input into clickLink() and check for the exception.
I will work on this further if I can get consensus... but given the history this is unlikely
sorry for the rant.
https://drupal.org/comment/8538789#comment-8538789
Comment #44
Mile23Well, no. clickLink() should report a failed test if the link label doesn't exist. It should also report a fail if the URL under that link is unreachable.
Patch this evening.
Comment #45
Mile23Basically, this is the only change to clickLink():
This just says 'none' for the path if xpath can't find anything.
Also added a unit test so we can check the behavior when the link exists and when it doesn't.
We can't unit test (or simpletest) the output of the assertion, because assertions are not injectable, and you can't assert that a failing assertion is a pass. :-)
Comment #46
Mile23Comment #47
sunI agree with @martin107 in #43:
Throwing an exception makes sense, because if a link cannot be clicked, then the entire rest of the test method will not succeed either way.
In almost all cases, failing to click a link is an unmet precondition, so proceeding with the test method only makes you wait unnecessarily long(er) for test results, whereas the results will only consist of failures after the failing
clickLink()
.I consider those needless wait times to be even more annoying than the faulty fail/error handling of
clickLink()
itself. — In 99% of all cases I experienced myself,clickLink()
failures are "D'oh!" situations. Instead of waiting minutes for the test to complete with tons of failures, I would have highly appreciated an early exception that halts the test (method), so that the brokenclickLink()
can be fixed first (and fast).Comment #48
Mile23Yes exactly. So FAIL and move on, but there is exactly zero need for an exception. Subsequent tests will fail also, but that's good, because this is, after all, a TEST. :-)
Here, let's get remedial:
Because clickLink() has an assertion in it, it is a test. It should really be called assertClickLink().
We're not trying to follow the link, we are trying to FIND OUT if we CAN follow that link, and then we do it if we can.
If the label doesn't exist, then clearly the test fails.
And in that circumstance, we're completely done. There is nothing else to say.
We are not in the midst of an error condition. We simply failed to find the link, click it, or follow it.
Again: Failing to find the label is not an error condition. Failing to follow the link is not an error condition. They are both FAIL conditions, which is different.
SimpleTest should hide errors related to discovering whether the test has passed or failed.
Because that's how testing works.
Comment #49
Mile23Also: My interest in this issue comes about from writing tests which are supposed to fail at this point, because that's what programmers do these days. You write tests which fail, and then make them pass.
Here's the failing test for a skeleton module I wrote: https://qa.drupal.org/pifr/test/764843
Note that it throws a bunch of errors which deliver zero value to the test author (me), or anyone else. These errors are fixed by the patch in #45, but the test still fails, because that's how testing works. :-)
Comment #50
Mile23Comment #51
Mile23Here's what you see if you get a failing test in Drupal due to clickLink().
It's completely unacceptable.
I want to suggest that there be a checkbox in the UI that says 'Report stack trace for failures,' which is turned off by default, but I doubt that if I do the work, anyone will want it.
If you want this, please ping on this issue.
Comment #52
jhodgdonI'm with Mile23 as well. If a test fails, it should fail, not throw an exception. If it throws an exception, the rest of the test you are trying to develop doesn't finish, and the rest of the tests you were trying to run don't run, and you lose the list of what you were even trying to run. It is definitely not what I want to happen -- it's totally annoying, which is why I filed this issue in the first place.
Normally when you are developing a test, you get a bunch of failures, and iterate on the code and test until everything is green, but when you get an exception you are just hit in the face and it derails the whole process.
So... As several of us had said, clickLink should just fail if the link doesn't exist, not completely derail your development process.
Comment #53
Mile23@jhodgdon: There's a patch in #45 that could use a review, which fixes this error. It unit-tests clickLink().
Comment #54
jhodgdonI don't see an actual code change in the patch in #45 -- it just looks like the formatting of the code was changed there, and you added a test? If there is a code change, can you make a patch that just changes the code and leaves the formatting the same, so it is more obvious what you are changing? Code formatting seems to me to be out of scope here.
The early patches on this issue that I made and others modified actually did fix the problem...
Comment #55
Mile23Formatting according to coding standards. :-)
Basically it checks for $url_target and if it's not set, then subtitutes 'none'. This way we get the message even if the target isn't found, and we also get no crash.
We have a number of things going on in this issue.
We have a problem where SimpleTest has an actual bug. (It tries to use an uninitialized variable.)
We have a problem where people have confused this bug for useful debugging information. (Other fails present a stacktrace and error codes, so it seems like this one should, too.)
We have two solutions to the problem: 1) Martin107 wants to throw an exception so that we can get the debugging information, and 2) others and myself want to just see whether the link exists and then just report a fail.
My solution is very simple, fails on fail rather than throwing an exception, and has a unit test for clickLink().
If we can get an RTBC, then a maintainer can decide, and we can move forward with something other than that screenshot in #51.
This is an almost-blocker for #2277599: [meta] Port tests from D7 to D8 Examples, Add Skeleton Module Structure because of the extreme tedium it creates.
Comment #56
jhodgdonI do not see a place in the coding standards that says function calls should ever be split up into multiple lines:
https://drupal.org/coding-standards#functcall
This was proposed several years ago, but people disagree so it was never adopted. You may like this format; I don't...
Anyway, that aside, as I said it makes it really hard to see what was changed in the patch when you mix unrelated changes like this with actual changes. Would you be willing to make a new patch that keeps the formatting as it was? It's not just me -- someone has to review this, then someone has to commit it, and if it's a pain to double-check that the changes are as intended, then it's a pain for several people.
But in any case, regarding reviewing... I took a look at the test code and it's about as clear as mud. :( providerTestClickLink makes a couple of cryptic arrays with no documentation on what is in them, and then ... well I don't know too much about unit testing, so ... it's not obvious to me what's going on or whether it's really testing that clickLink fails gracefully if it doesn't find the requested URL in the current page (which is the bug being reported here).
Maybe some code comments could be added to explain at least what the array values in providerTestClickLink are (it's not even an associative array where the array keys would provide a clue)? Or maybe you can get someone else to review it? Or maybe a WebTest that used an actual Drupal admin web page with a link that doesn't exist on the page would be more illuminating (since it would then be clear that it was testing the actual bug)?
Oh, and: 'an_url' ?!? What is this? I would personally always say "a URL", not "an URL'. :)
I am tempted to set this to Needs Work but maybe it is just my own deficiency of understanding that makes the code cryptic and hard to read?
Comment #57
Mile23Your suggestions are good. :-) I'll re-do the formatting and pad out the test docs.
Comment #58
Mile23Needed a re-roll anyway, so here it is.
Reverted the formatting, which is still just as unreadable in the patch. :-) Also added more docblock and inline commenting to the test, and added another piece of test data.
What this patch does: It introduces a check to make sure that $url_target exists, and if it does not, it substitutes '<none>' for the path in the report line. It also adds a heavily-mocked unit test with data provider, which verifies that if $url_target doesn't exist, clickLink() returns a fail. Were clickLink to throw the error for this issue, PHPUnit would fail this test.
Here's a reformatted version of the original function for readability's sake, and then the changed version.
Before:
After:
And here we see the test failing against the current code:
Comment #59
jhodgdonMuch better, thanks!
We normally upload test-only patches to verify that the test fails as expected without the code patch. :)
A few minor details on the docs:
a) List formatting -- in the last bullet item, indent the additional lines of text 2 spaces.
b) What is all the "as per" here? Maybe should just be "Expected return value of clickLink", "Parameter $label to clickLink", etc.? Same comments apply to the @param docs for the test method.
c) in "Test data which will be returned" ... which -> that [also in test method param]
d)
What does this mean? I shouldn't have to get out a dictionary to figure out what a code comment means. :)
Comment #60
Mile23You have to know what 'extant' means. :-)
Nevertheless... This is #58 with all the changes above, separate test patch to illustrate fail.
Check comment #58 to see a readable version of the changes.
Comment #62
Mile23I'd set this back to 'needs review' but then it would just test again and fail and be set at 'needs work.'
Comment #63
jhodgdonNo, setting to "needs review" does not cause a retest. :) Setting it to RTBC would... I guess we should probably upload just the actual patch on its own comment and set that to RTBC so that the test bot will not keep testing the "test-only-fail" patch.
Anyway... This looks good to me! Someone should probably manually test this in the Simpletest UI with a test trying to click on a nonexistent link.
Comment #64
jhodgdonforgot status. And by the way I looked at the test result for the failing test and it is failing where the test was designed to fail to test this issue.
Comment #65
Mile23Ok, here's a re-upload of the patch, without the test-only patch.
Note that the failing test-only patch appears in #60 and the run-down of what this patch does appears in #58.
Comment #66
Mile23And here's what it looks like in the UI:
Compare with: https://drupal.org/comment/8843605
Comment #67
jhodgdonLooks good, thanks!
Comment #68
alexpottI agree with @sun that a test method should be halted the moment you get an error - how every other test method in the same class should run - however that is not how simpletest currently works so the change here is in line. Changing this behaviour is a separate issue (and one that I would support).
The major issue I have with this patch is the complexity of understanding the test and the fact that the test is not really covering much of the clickLink functionality. Yes PHPUnit is great and fast and everything but when we're mocking 5 methods and using reflection to change whether or not something is protected alarm bells should be ringing. This test will be expensive to maintain because it is tightly coupled to the internals of the
clickLink()
method.That said if we are going to go down the PHPUnit route, why are we not just passing in a correctly structure return value for xpath? Or even better not actually mock xpath but mock
drupalGetContent()
since providing content would make the test much easier to understand.To improve the coverage of what is being tested we should be asserting if drupalGet() is called and what with.
Comment #69
Mile23@alexpott #68:
There's no error. It's a test. It fails. Not an error. The desired behavior is to report a fail, not crash in your face. The current behavior is a bug.
It exercises *every line* of clickLink() towards a number of desired results. Don't believe me? Here's a picture:
Note that a report like this is impossible using SimpleTest.
I'm using PHPunit because there is a veritable army of patches in this issue which failed to use SimpleTest to test this functionality. The unit test proves that clickLink() now does not do the thing it shouldn't do. Which is error out.
Either way works. Will change it.
drupalGetContent() is not a dependency of clickLink(), so it doesn't need to be mocked.
That's happening in the patch in #65 already.
Edit: Actually, yah, it's not, strictly speaking. :-) However, we are asserting that drupalGet() is called and returns a value. We don't care what it returns. We know what value is being sent to it, because we mocked getAbsoluteUrl().
Attached patch puts the 'href' in the xpath data, rather than adding it in the test.
Comment #70
Mile23Comment #71
alexpott@mile23 phpunit stops running a test method the moment an assertion fails :) It does not run any more assertions in the same method - I think simpletest (or any testing framework) should work the same way since after one assertion fails (especially one in click link) all bets are off as to what will happen next.
Exercising lines does not mean that all the logic in the method is tested - for example:
We should assert this is called once and with the correct url and if there is no target url we should assert that it is not called.
And people have got code coverage reports out of simpletest https://www.drupal.org/project/code_coverage
Comment #72
Mile23@alexpott #71
In an ideal world, yes, we'd short-circuit the whole method as failed on the first assertion fail. That's outside the scope of this issue though. I can't find an issue for such a change, but fire it up and let's go.
In this issue, we're going from buggy crash to consistent output.
That's exactly what we're testing with the unit test. In fact, it's the *only* thing we're testing with the unit test, because it's the only thing we can test. :-)
Our mocked drupalGet() can never return FALSE. We mock it because we're not testing the behavior of drupalGet(), and if we return a known value we know that it was called.
With all the mock scaffolding in place, clickLink() can only ever return 'A Web Page.' or FALSE. If we get 'A Web Page.' then we know drupalGet() was called. Thus, mission accomplished.
But... Here's a patch to make it more clear, substituting 'This Text Returned By drupalGet()' for 'A Web Page.'
Comment #73
jhodgdonReally? You want to make it so that Simpletest stops at the first failed test, or not fix this issue?
This would totally break my ability to develop tests and code in a timely manner. When I've made significant changes to the code in the API module, for instance, I usually need to run the entire suite of tests, which takes about an hour on my machine (there are a LOT of tests). So, I generally start the tests running, and go do something else, like eat lunch or get some exercise. When I get back to my machine, if all goes as planned, all the tests have run and I have a list of all the things I need to fix. I would then change code/tests and rerun either the failed tests or all tests, depending on what I had to change.
If the testing stopped at the first problem each time, or if the bug that this issue is trying to fix came up so that one of the tests aborted due to a PHP error that was really just a simple test failure, I would instead need to fix one error at a time and start the tests up again. This really would not work.
So... PLEASE can we just get this fixed? Simpletest does not work the way you're describing, and I hope it never does. It would really make things difficult for people without a really fast machine running long suites of tests to get anything done.
Meanwhile... Reviewing the current patch:
Two problems here:
a) I don't think this description was updated when the patch was updated? I think the first item in the array is the return value from drupalGet() mock, for instance?
b) The last bullet point is garbled... not sure what it is supposed to say really?
Until these two problems are fixed in the documentation, I'm unable to review the test properly, since I'm not sure what it is meant to be doing.
Comment #74
Mile23@jhodgson:
Agreed that halting on fail is out of scope for this issue. Everyone can argue about that in some other issue. :-)
The code snippet you quote above is kind of garbled... Here's what I see in the patch.
We're mocking xpath() so it returns known results for trying to find $label on the page. If we tell xpath() to return an empty array, then we're testing for the situation where no link was found.
Doing this, we can force clickLink() down different code paths based on the result of xpath(). The one we care about most is if there was no link found, because we want to be sure that our code catches any errors related to it.
The modification to clickLink() only catches an error condition. The test only proves that the error condition doesn't happen any more.
If someone feels a need for more testing of clickLink() beyond the scope of this issue, then open another one please.
Comment #75
jhodgdonOh, sorry about the garbled comment. Here is what I see in the latest patch:
So the first item in the array appears to be text returned by drupalGet(), but it is documented to be "Expected return value of clickLInk". That is not right?
And the part that I do not understand says;
"Test data which that be returned" ?!?!?!? What does that mean?
Comment #76
Mile23Ahh, structure error sentence in the latter. Will fix that. :-)
As for the former:
drupalGet() is mocked to return a known value. It's 'This text from drupalGet()' because alexpott seemed to be unclear on whether we were checking if drupalGet() was returning a value.
If you do a little analysis of clickLink(), you see that it returns either FALSE or the value of drupalGet().
Therefore, we know that if clickLink() returns 'This text returned from drupalGet()', from our mocked version of drupalGet(), then clickLink() found the link.
So it's both the return value of the mocked drupalGet() *and* the expected output of clickLink(). :-)
Comment #77
jhodgdonOK, so you just spent several paragraphs explaining this to me. How about documenting that the first value is used for, as you said "... both the return value of the mocked drupalGet() *and* the expected output of clickLink()" so that the test will be clear when the next maintainer comes around? :)
Comment #79
Mile23Added some more explanation.
I hope the seemingly unrelated test fail from the last patch goes away... I couldn't get Drupal to install to make a test work and see.
Comment #80
alexpott@jhodgdon whilst not wanting to derail this issue any further - my point is that once there is a failure in one test method that method should no longer continue to run - and this is especially important for clickLink since if the assertion fails the simpletest browser is not even on the expected page anymore. The other methods contained in that class should still run. But if you've read my comments you'll see I'm not blocking this patch and agree that it should be fixed. All I'm trying to do is point out that the test we're adding is not actually covering all the logic in the clickLink method and the test is extremely expensive to maintain. Personally I would have preferred if this test had gone down the same path as the
testAssertFieldByName()
contained in the same test class - because understanding html is far far simpler than understanding xpath output and the actual input to the method is output from the simpletest browser - but y'all seem hellbent on this approach so whatever.So here is what I'm on about - I've changed the test to now actually test all the logic contained within the clickLink() method. If we're testing that method it is important to assert that we don't ever try to use drupalGet() on a non existent link :) As we're adding a completely new test for clickLink() and using code coverage reports to say hey look we've covered everything it is important that we actually do cover everything.
Comment #81
alexpottNow for the patch
Comment #82
Mile23If someone doesn't understand what xpath() is doing, why are they writing unit tests for this class? :-)
Your version is much better. I see what you were trying to goad me into doing.
In the past I've gotten into trouble for writing tests beyond the scope of the thing I changed. If the issue was 'fix the php error and expand test coverage,' then yah.
Anyway: I give it the Mile23 Seal Of Approval. Let's see if jhodgdon has further criticism.
Comment #83
jhodgdonI like the new patch too. The docs are clear on what it's doing. Thanks!
Comment #84
jhodgdonShould we open another issue to discuss the other "when to stop" questions that have come up here, or is there already one?
Comment #85
Dries CreditAttribution: Dries commentedCommitted this patch to 8.x. I think the message could actually be made more clear when
$target_url
is not set but this is a lot better than what we have.Comment #86
jhodgdonI do not think the patch above was actually committed.
Comment #87
jhodgdonSo I was going to make a new patch with a better message... but since the above patch wasn't committed, that's a bit of a problem.
Here's the suggested code though:
Comment #88
Mile23Incorporated @jhodgdon's change, which is way better. :-)
Small tweak to method docblock.
Added pass()/fail() requirement to unit test, which is kind of the whole point, right?
If #81 was really added but not pushed or something, it might need reroll.
Comment #89
jhodgdonI'm not thrilled about this documentation line:
We normally don't put "We" in function header docs. :) And to me "assert a pass" (or fail) doesn't make sense. You don't "assert" a pass, you assert values and conditions. Maybe something like "The test passes if the link is discovered, and fails if it doesn't exist."?
The rest of the patch, I am of course happy with. ... A bit confused in the test though as to why 3 of the places in the interdiff have this line added and one doesn't:
I guess I don't understand what that does, but ... why add it mostly?
Comment #90
Mile23Yah, I think an extra returnValue(NULL) crept its way in there in a copypasta frenzy.
As far as 'assert'... If you look at pass() and fail(), they just map to asssert()... Also, I think clickLink() should really be called assertClickLink(), but that's another issue for another day.
So: Removed extra returnValue(NULL), changed docblock wording for clickLink(), removed some docblock stuff from the test method that no longer applies.
Comment #91
jhodgdonOK, assuming the test bot agrees, I think this is back to RTBC. Thanks!
Comment #92
catchCommitted/pushed to 8.x, thanks!
Moving to 7.x for backport.
Comment #94
jhodgdonSince we do not have the capability for the same kind of tests in D7/D6 as we have in D8, I think that the ported patch only needs to port the part in the clickLink() method, not the test. Should be a simple backport... any takers?
Comment #95
joshi.rohit100Patch for Drupal-7
Comment #96
jhodgdonThere is an extra space at the end of the middle line in:
You've been contributing a lot of patches (thanks!) -- please check your code editor settings. If you have a decent code editor, you should be able to set it to either remove or highlight end-of-line spaces, so that this doesn't happen.
Otherwise, the patch looks good.
Comment #97
joshi.rohit100Removed the extra space.
Comment #100
joshi.rohit100some testbot glitch
Comment #101
jhodgdonThat looks good, thanks!
Comment #104
joshi.rohit100Comment #105
jhodgdonComment #108
dcam CreditAttribution: dcam commentedComment #111
jhodgdonComment #113
jhodgdonErrors are unrelated to this patch...
Comment #114
David_Rothstein CreditAttribution: David_Rothstein commentedCommitted to 7.x - thanks!
Pretty sure this could be tested for Drupal 7 (the test would just look different than Drupal 8) but testing bugs in the test framework internals doesn't seem like it's super high-priority, so someone can do that as a followup if they want to.