Drupal Association members fund grants that make connections all over the world.
Posting here from.
Large testbot result sets are not successfully communicated back to qa.d.o; which causes that test to continuously cycle on the testbot and eventually cause an apache segfault. Originally this was *all* D8 tests, until we removing a debug() in core which was responsible for about 10k exceptions in the result set. This was a band-aid fix which works for 'clean' D8 tests, but does not address the root problem ... tests for broken patches with alot of exceptions are still tripping the HTTP 413 response, and I've had to clean up at least 2 segfaults per bot in the last 16 hours.
I've managed to isolate the headers from a failed results post from a testbot to qa.d.o. The HTTP 413 is being returned by nginx. This may explain why it was not showing up in the qa.d.o apache logs, if the POST is being blocked by the proxy before it ever gets to qa.d.o.
The result set in question was approx 4MB in size, with only 5k exceptions (and I've seen tests with 3-4 times that in the past); so we're going to need to set a fairly high value in order to prevent this from biting us again in the future. However, nnewton said he had already increased the client_max_body_size; so I'm not sure if this change isn't taking effect, or nginx is proxying the response from something further downstream.