Something is very, very wrong with update.php / upgrade tests... demons suspected

Confirmed as a random fail ... also saw it yesterday morning on a test.

So this was seen in DateUpgradePathTest, but also in FilledStandardUpgradePathTest in http://qa.drupal.org/pifr/test/426458. The failure points towards line 208 in UpgradePathTestBase, which means we may see this on any upgrade test leveraging it. Adjusting the title accordingly.

See also http://qa.drupal.org/pifr/test/426518

Something is requesting a table, for which no schema definition exists:

function drupal_schema_fields_sql($table, $prefix = NULL) {
  $schema = drupal_get_schema($table);
  $fields = array_keys($schema['fields']);

That's based on #6. But I suspect #5 to be caused by the same code/error.

The failure clearly seems to be caused in upgrade path tests only.

Are we sure that we can limit the scope to the last ~36 hours? (Jan ~20nd?)

If so, #1848998: Problems with update_module_enable() would be one major upgrade path change that landed within this time-frame. Didn't check for others yet.

During work on #1887904: update_retrieve_dependencies() only considers enabled modules (which could also come in to play depending on how far we go back) I noticed that block_update_dependencies() states that block_update_8005() requires user_update_8011() but in block_update_8005() we reference the {users_data} table and the {_d7_users_data} table but these aren't created and populated until user_update_8015() has run.

Prior to #1887904: update_retrieve_dependencies() only considers enabled modules block_update_dependencies() didn't even run in the disabled modules upgrade test, now it does - perhaps that's a factor?

Clutching at straws here, regardless block_update_dependencies() is wrong, and the patch in #1871772: Convert custom blocks to content entities changes it, but yet that issue still gets random failures. So most likely this is unrelated.

But thought it was worth mentioning.

Nor did reverting 31e0e1832. #1635240-243: Disregard -- patch testing issue ☢

PHP Fatal error:  Cannot use Symfony\\Component\\DependencyInjection\\ContainerBuilder as ContainerBuilder because the name is already in use in core/lib/Drupal/Core/DependencyInjection/UpdateBundle.php on line 10

How could that ever have worked? We have our own ContainerBuilder in that namespace.

This is not a fix for the random failures, there's nothing randomish about this. In fact, it should fail all the time?

I get 100% fails locally on PHP 5.3.15 when tried to run the "Drupal\system\Tests\Upgrade\DateUpgradePathTest" test - this patch fixes that...

I'm now looking for the random failure but i consider #18 to be rtbc and it fixes something that is plainly broken.

It does, but I'm very nervous to commit it without test coverage, because something is clearly wackadoo with our update.php tests.

Ok, this needs a new title. :P

Here's what we know so far.

On most systems, #18 happens when you access update.php in HEAD right now. This makes total sense, because #18 is clearly fixing a "duh" bug.

On some systems, this does not happen, and update.php runs fine, and the upgrade path tests pass with flying colours.

We know it's not related to PHP version, because both chx and the testbots are running PHP 5.3.5-0-debian-something. chx's fails, the testbot's don't.
In my case, even on the same exact machine/*AMP stack, I was getting it consistently and now suddenly don't after testing to see if tests passed or failed in the interface (they passed, btw).

We suspect demons.

We know it's not related to PHP version, because both chx and the testbots are running PHP 5.3.5-0-debian-something. chx's fails, the testbot's don't.
In my case, even on the same exact machine/*AMP stack, I was getting it consistently and now suddenly don't after testing to see if tests passed or failed in the interface.

I was hoping it would be the PHP version but I had a smilar situation locally. First the tests were passing fine and suddenly, they failed consistently. No idea what's going on...

* d037c55 Issue #1875818 by Berdir: Update composer.json to use the latest stable 3.0.x version of Guzzle.
* 8da4965 Issue #1885420 by kim.pepper, xjm: Document that node access queries on other tables need metadata
* 62c93ff Issue #1253820 by yched, zserno, tim.plunkett, xjm, effulgentsia, plach: Fixed It's impossible to submit no value for a field that has a default value.
* dc8fcba Issue #1875504 by fabpot: Add the possibility to use services as controllers.
* fd838b4 Issue #1874584 by kim.pepper: 'Block Library' link label is unclear.
* 83193f1 Issue #1833716 by quicksketch, Wim Leers, sun, TwoD: Added WYSIWYG: Introduce 'Text editors' as part of filter format configuration.
* b6ac1b8 Revert "Issue #1833716 by quicksketch, Wim Leers, effulgentsia: added WYSIWYG: Introduce 'Text editors' as part of filter format configuration."
* d2bfb66 Issue #1411276 by swentel, anrikun, quicksketch: Fixed image.admin.inc: new effect options are double escaped (check_plain()).
* 248284d Issue #1889826 by tim.plunkett: DefaultFactory::getPluginClass() is very useful and should be public and static.
* 4c19bdc Issue #1869328 by vijaycs85, Gábor Hojtsy, manningpete, YesCT: Restore simplicity of language list.
* 32e23b5 Issue #1888942 by dawehner: Rename viewStorageController::attachDisplays().
* ee1b6de Issue #1833716 by quicksketch, Wim Leers, effulgentsia: added WYSIWYG: Introduce 'Text editors' as part of filter format configuration.
* a2ae42b Issue #1853720 by vijaycs85, sun, Gábor Hojtsy: Fixed Hide language selection option is backwards.
* 6a5461c Issue #1884180 by dawehner: Cleanup base.js for real.
* 31e0e18 Issue #1887904 by larowlan: Fixed update_retrieve_dependencies() only considers enabled modules.
* d9666b2 Issue #1887244 by beejeebus: Use Config->save() and Config->delete() on import.
* d049222 Issue #1876942 by damiankloip, dawehner, tim.plunkett: Use direct methods instead of arrayAccess for display handlers.
* c1a7254 Issue #1886876 by IRuslan, swentel: Fixed Useless assignment in node_feed().
* ad551b6 Issue #1887692 by xjm: Clean up unused {block_node_type} table and references.
* 289e12f Issue #1874470 by kim.pepper: Having two RouterTest classes in the same module is confusing.
* 7be75b4 Issue #1814496 by Berdir, deviance, Lars Toomre: Make queue a container service.
* 65e54ed Issue #1874936 by frega, Wim Leers: Pluggable in-place editors (allow modules to define new Create.js PropertyEditor widgets).
* 189108a Issue #1833334 by das-peter, plach, Berdir, fago: EntityNG: integrate a dynamic property data table handling.
* db1122d Issue #1778178 by fago, Berdir, effulgentsia, mradcliffe, das-peter: Convert comments to the new Entity Field API.

Of that list, #1887904: update_retrieve_dependencies() only considers enabled modules and #1814496: Make queue a container service are the only ones that jump out at me.

Ok, well. As much as I'd love a test for #18, it doesn't look like this is possible, at least not in a reasonable timeframe. And in the meantime, this prevents the vast majority of sites from running update.php at all.

Therefore, committed and pushed #18 to 8.x.

Leaving this open though for the randomness, and hopefully to get to the bottom of this.

BlockUpgradePathTest failed randomly at #1890502-62: WYSIWYG: Add CKEditor module to core today.

Yeah I've actually seen this and similar failures in almost every issue I'm subscribed to. :( Even dumb stuff like "Add so and so to MAINTAINERS.txt."

And … Drupal\openid\Tests\Upgrade\OpenIDAuthmapUpgradePathTest today at #1890502-70: WYSIWYG: Add CKEditor module to core.

(I'll shut up now — I just want to point out that this is reducing agility for some issues.)

plach pointed me to #1874562: Upgrade path broken and yet tests pass over at #1890502-73: WYSIWYG: Add CKEditor module to core due to ANOTHER failure.

Yes, Wim, tests are failing at an alarming rate, across all core issues. Just hit retest, or work with Berdir to fix this.

For others wanting to dive in, the problem is related to the update progressbar page occasionally returning a zero-byte response on the 'no_js' URL. The question is 'why'?

that could be timeout or memory limit -- but both, I believe, would leave a trace in Apache logs. Anything?

#33: My apologies. My productivity depends in significant part on d.o & testbot — hence my frustration became visible here on d.o. Won't happen again.

#35: Nope, nothing in the apache logs.

Hmmm ... any chance that this may be related? https://bugs.php.net/bug.php?id=55438

Unlikely. That's a streams bug; we do not use the curl stream wrapper; we use cURL directly.

"Block upgrade test" and "Basic minimal profile upgrade path, populated database" fail almost every time on my laptop and also here: http://qa.drupal.org/pifr/test/334783

If it fails locally somewhat reproducible, is it possible to get shell access to that machine? Or care to ping me in #drupal-contribute, I will direct you on what to debug

This commit fixed it for me: http://drupalcode.org/project/drupal.git/commit/8c0f2e8

I think it's probably something related to the environment or runtime conditions.

I haven't been seeing this anymore, so I suspect the commit mentioned in #42 fixed it? Or is this still happening?

Closing based on #42 (and having not seen this myself in at least a few days).

We can re-open if it re-appears.

Not sure it's fixed because I got random number of failed upgrade tests in http://qa.drupal.org/pifr/test/443928
Suppose this random failures are caused by undocumented usage of displays in upgrade from #1852966-126: Rework entity display settings around EntityDisplay config entity

@andypost: That is an error that I haven't seen before ("The string cannot be saved: %type: !message in %function (line %line of %file). Drupal\locale\StringStorageException"), so I'm not sure if it's the same thing.

Yeah ... this is a different symptom than we were chasing on this particular issue .... I'd suggest a new ticket for it.

Thanx, I filed #1916556: Random test failures - The string cannot be saved