I've seen in a couple of issues that tests seem to fail w/ an out of memory error. Let's see if that also happens with an empty patch.

CommentFileSizeAuthor
#18 2784849-18--debug.patch422 bytesdrunken monkey
#7 patch_136.patch248 bytesmpp
#2 patch.patch262 bytesborisson_

Comments

borisson_ created an issue. See original summary.

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented
Status: Active » Needs review
StatusFileSize
new patch.patch262 bytes

Status: Needs review » Needs work

The last submitted patch, 2: patch.patch, failed testing.

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented
Related issues: +#2777483: Unmet dependencies
drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

Sorry, but I can't reproduce this problem. As said, I normally use PHP 7, so it's expected that it works on my installation.

However, I also tried it with XAMPP (horrible experience under Linux, don't touch it!) and even though I used its default memory limit of 128 MB the tests ran fine. (The test bot uses 320 MB, apparently.)
I couldn't run them from the command line, though.

Then I tried running my own local test bot instance, but couldn't get that to work either.

Consequently, I'm completely at a loss how I could debug this. My best suggestion for now is to just switch the default test environment to PHP 7 and hope we don't introduce any real incompatibilities with PHP 5.

You said you managed to reproduce this, right? Would it be possible for you to do a git bisect to find out which Core commit broke this? After all, it still worked fine on August 1, and then broke with no commits in between. So it's more or less certain a Core commit is the cause. Maybe finding out which commit that was will give us a hint how we can fix this.

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented

I can't actually reproduce this, not on php5 either. I'll try to have a go at this with a local testbot tomorrow

mpp’s picture

mpp commented
Status: Needs work » Needs review
StatusFileSize
new patch_136.patch248 bytes
mpp’s picture

mpp commented

The default branch for testing is set to D8.3 but this should be D8.1.

Status: Needs review » Needs work

The last submitted patch, 7: patch_136.patch, failed testing.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

@ borisson_: Thanks!
For now, I've really just switched the default test environment to PHP 7 – don't want to hold up all other work because of this.

The default branch for testing is set to D8.3 but this should be D8.1.

See #2733311: Contrib module tests should be able to configure which drupal 8 branch to use as a default.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Issue tags: +Release blocker
drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

One new idea for tackling this problem, if we're not able to reproduce it: look at what all the failing tests have in common and try to narrow done the potential causes via that. Might be an idea.

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented

I ran: $ php core/scripts/run-tests.sh --verbose --browser --php /Applications/MAMP/bin/php/php5.5.26/bin/php --url "http://search.dev" --class "Drupal\search_api\Tests\CacheabilityTest" locally, as that is one of the failing tests on the testbot, my php5.5 instance has a 128M memory limit and this test was green.

So I ran all tests: $ php core/scripts/run-tests.sh --verbose --browser --php /Applications/MAMP/bin/php/php5.5.26/bin/php --concurrency 4 --url "http://search.dev" --directory modules/search_api. (That was slow though: Test run duration: 53 min 35 sec)

Those didn't all pass: 1932 passes, 1 fail, 0 exceptions, 450 debug messages
The Drupal\search_api\Tests\ViewsTest failed

So I reran just the views test:

php core/scripts/run-tests.sh --verbose --browser --php /Applications/MAMP/bin/php/php5.5.26/bin/php --url "http://search.dev" --class "Drupal\search_api\Tests\ViewsTest"

Drupal test run
---------------

Tests to be run:
  - Drupal\search_api\Tests\ViewsTest

Test run started:
  Monday, September 5, 2016 - 12:02

Test summary
------------

Drupal\search_api\Tests\ViewsTest                            312 passes   1 fails                 62 messages

Test run duration: 2 min 47 sec

Looks like that one has an actual problem right now on php5. I don't see any of the OOM errors though.

andypost’s picture

andypost
he/him
Primary language Russian
commented

Issue with memory could be very similar to #2688297: File migration slows down and eats more and more memory, eventually stops
That means that loading entities make static cache populated (and never cleaned) so once memory ends no more entities could be loaded

PS one more core issue #2558857: Migrations invalidate entity caches when trying to reclaim memory, should flush

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

Thanks a lot for your work, Joris!
The fact that none of use can reproduce this problem, even with lower memory limits than the test bot has, seems to point to a problem specifically with the test bot code.
Has anyone managed to run a test bot instance locally? That might be our only change to reproduce, debug and fix this. (I've tried setting this up twice already, but it always kept failing for me.)

Or maybe we should report a bug for the test bot? I've had a quick look, though, and no-one seems to have reported a similar issue yet.

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented

Or maybe we should report a bug for the test bot? I've had a quick look, though, and no-one seems to have reported a similar issue yet.

Not sure if we should post a bug to core or to the test-bot, since it does seem to work against d8.1/8.2 but doesn't seem to work against 8.3? Can you post an issue against core? We can always move it to the testbots if it looks like this is a bot-issue.

As a sidenote, I did my tests with a d8.2, I'll try running tests again with 8.3

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented

I ran tests again w/ 8.3 locally and those seem to pass as well. I still think we should file this against core though.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Status: Needs work » Active
StatusFileSize
new 2784849-18--debug.patch422 bytes

Does it really work with 8.2? I would've thought that already broke it. Attaching a dummy patch to check this.

Regarding the Core issue: I wouldn't really know what to report there, or where. "Test bot reports out of memory errors for our contrib module in some Core versions, but we don't know why or how and can't reproduce." Doesn't seem like we'd get any help that way.
If you look at the lines at which the error is reported, it kinda seems like it could have something to do with plugin handling. I guess we could try it with that. I just don't want to annoy Core developers with a vague issue they can't do anything about.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Status: Active » Needs review
borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented
Status: Needs review » Active

Looks like you're right, there's a difference between 8.1 and 8.2+ where this broke. That's also why I said that this might be related to something in core, because it does work with 8.1

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

Looks like you're right, there's a difference between 8.1 and 8.2+ where this broke. That's also why I said that this might be related to something in core, because it does work with 8.1

Yes, it's definitely related to a Core change – see #5. Specifically, a change on the 8.2.x branch between August 1 and August 16 (or whenever you first saw those fails). (Actually, that's not even too many – about 55, it seems.)
However, that doesn't mean it's a bug in Core, just that something changed in a way that makes the test bot run out of memory for some of our tests. And since we can't reproduce this locally, any of us, it does seem that it's some specific issue of the test bot that gets triggered. And specific to PHP 5 (or, possibly, to the test bot's configuration for PHP 5 – maybe there's some other changes in there as well).

In any case, I created #2795335: Plugin manager runs out of memory in test bot. I guess it can't hurt.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Status: Active » Fixed

I'm so good!
My brilliant "Just wait and see." tactic worked flawlessly!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.