Testbots possibly crash because sqlite isn't properly cleaned up when OOM'ing

This particular testbot problem is actually pretty simple - when there is so much ooming, we end up with very large sqlite database. When the testbots go to clean up the database, it uses run-tests.sh --clean to 'clean up after itself', which makes sqlite do a 'drop table'. This for some reason needs a bunch *more* disk space to drop the table, but we keep the sqlite db on a tmpfs so that its all in memory, so we dont have enough space to remove the old test data.
Perhaps we could patch run-tests.sh to not just 'drop table' but to delete the sqlite file and start over or something: http://cgit.drupalcode.org/drupal/tree/core/scripts/run-tests.sh#n499.

So my 2 cents on this is that we need to reduce the php memory limit to 128mb on the bots since this is what tests need so we'll crash sooner and perhaps with less bad effects. See #2495411: Make simpletest fail a test when it detects pages that need more than 64MB for memory usage during tests.

The current memory limit:

php -i | grep memory
memory_limit => 320M => 320M

Can;t see how this is critical or even major. New uncommitted code introducing an infinite loop?

Lol core committer is only thinking about the core queue... this is not that.

Im updating the issue title, because this isn't restricted to just memory limits. That was symptomatic, but anytime a test run produces reams and reams of output, causing the sqlite testresult database to get too big, it will cause all further testing to bail on that bot. It could also be issues with php doing a fatal error and not cleaning up properly as well. In any case we can solve this with a patch to run-tests.sh that removes the file vs doing a drop table. So no matter what class of problem would cause the sqlite database to get huge, we could at least know it wouldnt crush the bot for further testing.