Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
HEAD failed on DateUpgradePathTest this morning. It went away on a retest. I overheard someone mention this test the other day so I think it may be unstable:
I didn't find any existing core issues for this failure, just #1860778: Provide an upgrade path for date formats, so filing against testbot for now until someone can confirm.
Comment | File | Size | Author |
---|---|---|---|
#18 | use-fail-1893800-18.patch | 938 bytes | Berdir |
date_upgrade_path_test.png | 15.42 KB | xjm |
Comments
Comment #1
jthorson CreditAttribution: jthorson commentedConfirmed as a random fail ... also saw it yesterday morning on a test.
Comment #2
xjmIt's annoying, but I wouldn't call it critical since it only happens intermittently. :)
Comment #3
xjmHm, though looking at this test, it looks like the failure was at the beginning, during the upgrade. So there might actually be a bug in the upgrade path itself.
Comment #4
xjmI ran this test 200 times locally using
run-tests.sh
on the commandline, and didn't see a single fail, so whatever the problem is, it's probably something related to the environment or runtime conditions.Comment #5
jthorson CreditAttribution: jthorson commentedSo this was seen in DateUpgradePathTest, but also in FilledStandardUpgradePathTest in http://qa.drupal.org/pifr/test/426458. The failure points towards line 208 in UpgradePathTestBase, which means we may see this on any upgrade test leveraging it. Adjusting the title accordingly.
Comment #6
larowlanSee also http://qa.drupal.org/pifr/test/426518
Comment #7
xjmI'm currently assaulting the testbots to see if I can pinpoint a commit that caused this, since I can't reproduce it locally.
Comment #8
sunSomething is requesting a table, for which no schema definition exists:
That's based on #6. But I suspect #5 to be caused by the same code/error.
The failure clearly seems to be caused in upgrade path tests only.
Are we sure that we can limit the scope to the last ~36 hours? (Jan ~20nd?)
If so, #1848998: Problems with update_module_enable() would be one major upgrade path change that landed within this time-frame. Didn't check for others yet.
Comment #9
larowlanDuring work on #1887904: update_retrieve_dependencies() only considers enabled modules (which could also come in to play depending on how far we go back) I noticed that block_update_dependencies() states that block_update_8005() requires user_update_8011() but in block_update_8005() we reference the {users_data} table and the {_d7_users_data} table but these aren't created and populated until user_update_8015() has run.
Prior to #1887904: update_retrieve_dependencies() only considers enabled modules block_update_dependencies() didn't even run in the disabled modules upgrade test, now it does - perhaps that's a factor?
Clutching at straws here, regardless block_update_dependencies() is wrong, and the patch in #1871772: Convert custom blocks to content entities changes it, but yet that issue still gets random failures. So most likely this is unrelated.
But thought it was worth mentioning.
Comment #10
xjm@sun, yeah, I am currently running 10 patches that revert that commit (
273499059dab
).Comment #11
xjmOkay, reverting
273499059dab
does not resolve it: http://qa.drupal.org/pifr/test/426633Comment #12
xjmNext batch in #1635240-243: Disregard -- patch testing issue ☢. It reverts #1887904: update_retrieve_dependencies() only considers enabled modules, because that seems quite like something that could cause a schema definition to be gone walkies.
Comment #13
larowlanNor did reverting
31e0e1832
. #1635240-243: Disregard -- patch testing issue ☢Comment #14
xjmNext candidate is #1479454: Convert user roles to configurables. Edit: See also #1894286: System updates are executed without priority.
Comment #15
xjmHmm, reverting #1479454: Convert user roles to configurables still had a failure in the upgrade path, but not in the same way as the others: http://qa.drupal.org/pifr/test/426818
Comment #16
xjmAlright, unfortunately, it looks like these failures happen in HEAD as far back as at least Jan. 16:
http://qa.drupal.org/pifr/test/427028
Comment #17
xjmHEAD broke on this again, twice. I take back my previous statement; this is disruptive enough to be critical.
Comment #18
BerdirHow could that ever have worked? We have our own ContainerBuilder in that namespace.
This is not a fix for the random failures, there's nothing randomish about this. In fact, it should fail all the time?
Comment #19
alexpottI get 100% fails locally on PHP 5.3.15 when tried to run the
"Drupal\system\Tests\Upgrade\DateUpgradePathTest"
test - this patch fixes that...I'm now looking for the random failure but i consider #18 to be rtbc and it fixes something that is plainly broken.
Comment #20
webchickIt does, but I'm very nervous to commit it without test coverage, because something is clearly wackadoo with our update.php tests.
Comment #21
webchickOk, this needs a new title. :P
Here's what we know so far.
On most systems, #18 happens when you access update.php in HEAD right now. This makes total sense, because #18 is clearly fixing a "duh" bug.
On some systems, this does not happen, and update.php runs fine, and the upgrade path tests pass with flying colours.
We know it's not related to PHP version, because both chx and the testbots are running PHP 5.3.5-0-debian-something. chx's fails, the testbot's don't.
In my case, even on the same exact machine/*AMP stack, I was getting it consistently and now suddenly don't after testing to see if tests passed or failed in the interface (they passed, btw).
We suspect demons.
Comment #22
BerdirI was hoping it would be the PHP version but I had a smilar situation locally. First the tests were passing fine and suddenly, they failed consistently. No idea what's going on...
Comment #23
xjmSo, the upgrade path tests don't always fail right at the beginning. http://qa.drupal.org/pifr/test/428443 manages to start the upgrade before it chokes. Stochastic bugs, yay!
Comment #24
xjmI think I've finally found a known good commit; patches rolling HEAD back to
77f0384244
did not fail the upgrade path tests in 50 or so runs on the bots. So something between that andd037c55eb66
started causing the random failures. Now commences git bisect fun time!Comment #25
tim.plunkettOf that list, #1887904: update_retrieve_dependencies() only considers enabled modules and #1814496: Make queue a container service are the only ones that jump out at me.
Comment #26
xjmI already confirmed that reverting #1887904: update_retrieve_dependencies() only considers enabled modules does not resolve it, unfortunately. And if I'm reading the log correctly, it worked when rolled back to after #1814496: Make queue a container service, unless I'm really losing the lottery on my tests.
Comment #27
xjmSo, argh. Apparently the trick used in http://drupal.org/files/upgrade-tests-only_0.patch also reduces or eliminates the chance of failure... this patch failed once while it was run concurrently with other tests, but has not failed since (when run only with copies of itself in a mostly empty bot queue). So I have no way of telling if
77f0384244
was actually a good commit (since we can't prove a negative). :(@berdir's attempt to get some debugging information is also not yielding any of the expected fails. We know it's still broken, though, because head failed on the most recent core commit this afternoon. It's beyond me to debug this at this point.
Comment #28
webchickOk, well. As much as I'd love a test for #18, it doesn't look like this is possible, at least not in a reasonable timeframe. And in the meantime, this prevents the vast majority of sites from running update.php at all.
Therefore, committed and pushed #18 to 8.x.
Leaving this open though for the randomness, and hopefully to get to the bottom of this.
Comment #29
Wim LeersBlockUpgradePathTest
failed randomly at #1890502-62: WYSIWYG: Add CKEditor module to core today.Comment #30
webchickYeah I've actually seen this and similar failures in almost every issue I'm subscribed to. :( Even dumb stuff like "Add so and so to MAINTAINERS.txt."
Comment #31
Wim LeersAnd …
Drupal\openid\Tests\Upgrade\OpenIDAuthmapUpgradePathTest
today at #1890502-70: WYSIWYG: Add CKEditor module to core.(I'll shut up now — I just want to point out that this is reducing agility for some issues.)
Comment #32
Wim Leersplach pointed me to #1874562: Upgrade path broken and yet tests pass over at #1890502-73: WYSIWYG: Add CKEditor module to core due to ANOTHER failure.
Comment #33
tim.plunkettYes, Wim, tests are failing at an alarming rate, across all core issues. Just hit retest, or work with Berdir to fix this.
Comment #34
jthorson CreditAttribution: jthorson commentedFor others wanting to dive in, the problem is related to the update progressbar page occasionally returning a zero-byte response on the 'no_js' URL. The question is 'why'?
Comment #35
chx CreditAttribution: chx commentedthat could be timeout or memory limit -- but both, I believe, would leave a trace in Apache logs. Anything?
Comment #36
Wim Leers#33: My apologies. My productivity depends in significant part on d.o & testbot — hence my frustration became visible here on d.o. Won't happen again.
Comment #37
jthorson CreditAttribution: jthorson commented#35: Nope, nothing in the apache logs.
Comment #38
jthorson CreditAttribution: jthorson commentedHmmm ... any chance that this may be related? https://bugs.php.net/bug.php?id=55438
Comment #39
chx CreditAttribution: chx commentedUnlikely. That's a streams bug; we do not use the curl stream wrapper; we use cURL directly.
Comment #40
jbrown CreditAttribution: jbrown commented"Block upgrade test" and "Basic minimal profile upgrade path, populated database" fail almost every time on my laptop and also here: http://qa.drupal.org/pifr/test/334783
Comment #41
chx CreditAttribution: chx commentedIf it fails locally somewhat reproducible, is it possible to get shell access to that machine? Or care to ping me in #drupal-contribute, I will direct you on what to debug
Comment #42
jbrown CreditAttribution: jbrown commentedThis commit fixed it for me: http://drupalcode.org/project/drupal.git/commit/8c0f2e8
Comment #43
p5systems CreditAttribution: p5systems commentedI think it's probably something related to the environment or runtime conditions.
Comment #44
Wim LeersI haven't been seeing this anymore, so I suspect the commit mentioned in #42 fixed it? Or is this still happening?
Comment #45
jthorson CreditAttribution: jthorson commentedClosing based on #42 (and having not seen this myself in at least a few days).
We can re-open if it re-appears.
Comment #46
andypostNot sure it's fixed because I got random number of failed upgrade tests in http://qa.drupal.org/pifr/test/443928
Suppose this random failures are caused by undocumented usage of displays in upgrade from #1852966-126: Rework entity display settings around EntityDisplay config entity
Comment #47
Berdir@andypost: That is an error that I haven't seen before ("The string cannot be saved: %type: !message in %function (line %line of %file). Drupal\locale\StringStorageException"), so I'm not sure if it's the same thing.
Comment #48
jthorson CreditAttribution: jthorson commentedYeah ... this is a different symptom than we were chasing on this particular issue .... I'd suggest a new ticket for it.
Comment #49
andypostThanx, I filed #1916556: Random test failures - The string cannot be saved