This issue is similar to https://www.drupal.org/node/1884182
and it affects the core testing.

The connection is here
https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Database!Driver!s...

But because the default limit is 999, the tests fail to get written in results/ with the error bellow:

Drupal test run
---------------

All tests will run.

Test run started:
  Tuesday, September 23, 2014 - 11:18

Test summary
------------

 [32mDrupal\system\Tests\Action\ActionUnitTest                     15 passes                                      
 [0m [32mDrupal\action\Tests\ConfigurationTest                         51 passes                                      
 [0m [32mDrupal\action\Tests\ActionUninstallTest                       13 passes                                      
 [0m [32mDrupal\action\Tests\BulkFormTest                              52 passes                                      
 [0m [32mDrupal\aggregator\Tests\AggregatorConfigurationTest           43 passes                                      
 [0m [32mDrupal\aggregator\Tests\AggregatorCronTest                    55 passes                                      
 [0m [32mDrupal\aggregator\Tests\DeleteFeedTest                        34 passes                                      
 [0m [32mDrupal\aggregator\Tests\AddFeedTest                           67 passes                                      
 [0m [32mDrupal\aggregator\Tests\DeleteFeedItemTest                   270 passes                                      
 [0m [32mDrupal\aggregator\Tests\FeedCacheTagsTest                    227 passes                                      
 [0m [32mDrupal\aggregator\Tests\FeedFetcherPluginTest                 63 passes                                      
 [0m [32mDrupal\aggregator\Tests\FeedLanguageTest                      57 passes                                      
 [0m [32mDrupal\filter\Tests\FilterFormatAccessTest                   292 passes                                      
 [0m [32mDrupal\aggregator\Tests\Views\IntegrationTest                 38 passes                                      
 [0m [32mDrupal\system\Tests\Ajax\CommandsTest                         77 passes                                      
 [0m [32mDrupal\aggregator\Tests\AggregatorRenderingTest              236 passes                                      
 [0m [32mDrupal\aggregator\Tests\ImportOpmlTest                        66 passes                                      
 [0m [32mDrupal\aggregator\Tests\UpdateFeedTest                        76 passes                                      
 [0m [32mDrupal\aggregator\Tests\ItemCacheTagsTest                    176 passes                                      
 [0m [32mDrupal\system\Tests\Ajax\ElementValidationTest                13 passes                                      
(... reduced output)                              
 [0m [32mDrupal\views_ui\Tests\HandlerTest                            311 passes                                      
 [0m [32mDrupal\views_ui\Tests\StyleTableTest                          17 passes                                      
 [0m [32mDrupal\views_ui\Tests\PreviewTest                            135 passes                                      
 [0m [32mDrupal\views_ui\Tests\StyleUITest                             39 passes                                      
 [0m [32mDrupal\views_ui\Tests\ViewsUITourTest                         23 passes                                      
 [0m [32mDrupal\views_ui\Tests\OverrideDisplaysTest                   150 passes                                      
 [0m [32mDrupal\views_ui\Tests\WizardTest                              26 passes                                      
 [0m [32mDrupal\views_ui\Tests\DisplayTest                            214 passes                                      
 [0m [32mDrupal\views_ui\Tests\ViewEditTest                           159 passes                                      
 [0m
Test run duration: 39 min 37 sec


Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables' in /var/www/core/lib/Drupal/Core/Database/Connection.php:336
Stack trace:
#0 /var/www/core/lib/Drupal/Core/Database/Connection.php(336): PDO->prepare('SELECT * FROM s...')
#1 /var/www/core/lib/Drupal/Core/Database/Connection.php(536): Drupal\Core\Database\Connection->prepareQuery('SELECT * FROM {...')
#2 /var/www/core/scripts/run-tests.sh(913): Drupal\Core\Database\Connection->query('SELECT * FROM {...', Array)
#3 /var/www/core/scripts/run-tests.sh(92): simpletest_script_reporter_write_xml_results()
#4 {main}

Next exception 'Drupal\Core\Database\DatabaseExceptionWrapper' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables: SELECT * FROM {simpletest} WHERE test_id IN (:test_ids_0, :test_ids_1, :test_ids_2, :test_ids_3, :test_ids_4, :test_ids_5, :test_ids_6, :test_ids_7, :test_ids_8, :test_ids_9, :test_ids_10, :test_ids_11, :test_ids_12, :test_ids_13, :test_ids_14, :test_ids_15, :test_id in /var/www/core/lib/Drupal/Core/Database/Connection.php on line 569

Actual testbot command:

/usr/bin/php ./core/scripts/run-tests.sh --concurrency 8 --php /usr/bin/php --url http://ec2-54-198-133-147.compute-1.amazonaws.com/checkout --keep-results --all --clean

New testbot command:

php ./core/scripts/run-tests.sh --php /usr/bin/php --url 'http://localhost' --color --concurrency 6 --verbose --xml '/var/workspace/results' --sqlite /var/www/test.sqlite --dburl pgsql://drupaltestbot:drupaltestbotpw@172.17.1.83/build_2014_09_12_190001 --keep-results --all
CommentFileSizeAuthor
#25 drupal-test-script-2343389-25.patch2.96 KBSutharsan
#25 interdiff-2343389-24-25.txt1.64 KBSutharsan
#24 drupal-test-script-2343389-24.patch3.18 KBSutharsan
#24 interdiff-2343389-21-24.txt681 bytesSutharsan
#21 drupal-test-script-2343389-21.patch3.18 KBSutharsan
#21 interdiff-2343389-18-21.txt626 bytesSutharsan
#18 drupal-test-script-2343389-18.patch3.16 KBSutharsan
#18 interdiff-2343389-17-18.txt1.45 KBSutharsan
#17 interdiff-2343389-14-17.patch3.38 KBSutharsan
#17 drupal-test-script-2343389-17.patch2.94 KBSutharsan
#15 drupal-test-script-2343389-15.patch2.18 KBSutharsan
#14 drupal-test-script-2343389-14.patch1.85 KBSutharsan
#14 interdiff-2343389-13-14.patch2.29 KBSutharsan
#13 drupal-test-script-2343389-6.patch1.71 KBricardoamaro
#12 drupal-test-script-2343389-5.patch1.71 KBricardoamaro
#10 drupal-test-script-2343389-4.patch1.71 KBricardoamaro
drupal-test-script-2343389-4.patch1.71 KBricardoamaro
#4 drupal-test-script-2343389-3.patch1.61 KBSutharsan
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

@sun, @jthorson, ping

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented
Issue summary: View changes
ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

@Sutharsan, thanks for having a look

Therefore, to fix this we need to fix the "--sqlite" option to divide the query in parts.

Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented
Status: Active » Needs review
FileSize
drupal-test-script-2343389-3.patch1.61 KB

This should fix the problem. If required, it performs multiple select queries to load the test results.

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

Woot!

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

Running:

~/drupalci_testbot# sudo ./scripts/build_all.sh refresh mysql-5.5
~/drupalci_testbot# sudo \
DCI_PATCH="https://www.drupal.org/files/issues/drupal-test-script-2343389-3.patch,." \
DCI_DBTYPE='mysql' \
DCI_DBVER='5.5'  \
DCI_TESTGROUPS='--all' \
DCI_DRUPALBRANCH='8.0.x' \
DCI_VERBOSE="true" \
DCI_CONCURRENCY="8" \
./containers/web/run.sh
ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

Running...

------------------------ ENVIRONMENT VARIABLES IN USE ------------------------
DCI_IDENTIFIER="build_2014_09_27_154526"
DCI_DRUPALBRANCH="8.0.x"
DCI_DRUPALVERSION="8"
DCI_UPDATEREPO="false"
DCI_REPODIR="/home/ricardo/testbotdata"
DCI_DRUPALREPO="http://git.drupal.org/project/drupal.git"
DCI_DRUSHREPO="https://github.com/drush-ops/drush.git"
DCI_BUILDSDIR="/home/ricardo/testbotdata"
DCI_WORKSPACE="/home/ricardo/testbotdata/build_2014_09_27_154526/"
DCI_DEPENDENCIES=""
DCI_DEPENDENCIES_GIT=""
DCI_DEPENDENCIES_TGZ=""
DCI_MODULESPATH="./modules"
DCI_PATCH="https://www.drupal.org/files/issues/drupal-test-script-2343389-3.patch,."
DCI_DBUSER="drupaltestbot"
DCI_DBPASS="drupaltestbotpw"
DCI_DBTYPE="mysql"
DCI_DBVER="5.5"
DCI_DBCONTAINER="drupaltestbot-db-mysql-5.5"
DCI_DBLINK="--link=drupaltestbot-db-mysql-5.5:db"
DCI_CMD=""
DCI_INTERACTIVE=""
DCI_INSTALLER="none"
DCI_VERBOSE="true"
DCI_PHPVERSION="5.4"
DCI_CONCURRENCY="8"
DCI_RUNSCRIPT="php ./core/scripts/run-tests.sh --php /usr/bin/php --url 'http://localhost' --color --concurrency 8 --verbose --xml '/var/workspace/results'"
DCI_TESTGROUPS="--all"
VERBO="--verbose"

------------------------ STARTING DOCKER CONTAINER ---------------------------
Operation [start]...
Starting apache2 service
ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

Still go the same error... :(

Detailed test results
---------------------

Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables' in /var/www/core/lib/Drupal/Core/Database/Connection.php:336
Stack trace:
#0 /var/www/core/lib/Drupal/Core/Database/Connection.php(336): PDO->prepare('SELECT * FROM s...')
#1 /var/www/core/lib/Drupal/Core/Database/Connection.php(536): Drupal\Core\Database\Connection->prepareQuery('SELECT * FROM {...')
#2 /var/www/core/scripts/run-tests.sh(1014): Drupal\Core\Database\Connection->query('SELECT * FROM {...', Array)
#3 /var/www/core/scripts/run-tests.sh(90): simpletest_script_reporter_display_results()
#4 {main}

Next exception 'Drupal\Core\Database\DatabaseExceptionWrapper' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables: SELECT * FROM {simpletest} WHERE test_id IN (:test_ids_0, :test_ids_1, :test_ids_2, :test_ids_3, :test_ids_4, :test_ids_5, :test_ids_6, :test_ids_7, :test_ids_8, :test_ids_9, :test_ids_10, :test_ids_11, :test_ids_12, :test_ids_13, :test_ids_14, :test_ids_15, :test_ids in /var/www/core/lib/Drupal/Core/Database/Connection.php on line 569
ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented
ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented
FileSize
drupal-test-script-2343389-4.patch1.71 KB

new test with count and more output

The last submitted patch, drupal-test-script-2343389-4.patch, failed testing.

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented
FileSize
drupal-test-script-2343389-5.patch1.71 KB

Trying again with chunks with 50

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented
FileSize
drupal-test-script-2343389-6.patch1.71 KB

another patch

Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented
Status: Needs review » Needs work
FileSize
interdiff-2343389-13-14.patch2.29 KB
drupal-test-script-2343389-14.patch1.85 KB
  • Fixed a problem with array_merge().
  • Changed the max. number query variables back to 500.
  • Corrected spelling of sqlite.

I will put the status to 'Needs work' until we have completed successful tests with the new testbot command.

Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented
FileSize
drupal-test-script-2343389-15.patch2.18 KB

For testing purpose:
* reduced the chunk size to 1
* added debug messages

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

Error:

[29-Sep-2014 11:48:20 UTC] PHP Fatal error:  Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1 too many SQL vari
ables' in /var/www/core/lib/Drupal/Core/Database/Connection.php:336                                                                             
Stack trace:                                                                                                                                    
#0 /var/www/core/lib/Drupal/Core/Database/Connection.php(336): PDO->prepare('SELECT * FROM s...')                                               
#1 /var/www/core/lib/Drupal/Core/Database/Connection.php(536): Drupal\Core\Database\Connection->prepareQuery('SELECT * FROM {...')              
#2 /var/www/core/scripts/run-tests.sh(1019): Drupal\Core\Database\Connection->query('SELECT * FROM {...', Array)                                
#3 /var/www/core/scripts/run-tests.sh(89): simpletest_script_reporter_display_results()                                                         
#4 {main}                                                                                                                                       
                                                                                                                                                
Next exception 'Drupal\Core\Database\DatabaseExceptionWrapper' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables: SELECT * 
FROM {simpletest} WHERE test_id IN (:test_ids_0, :test_ids_1, :test_ids_2, :test_ids_3, :test_ids_4, :test_ids_5, :test_ids_6, :test_ids_7, :tes
t_ids_8, :test_ids_9, :test_ids_10, :test_ids_11, :test_ids_12, :test_ids_13, :test_ids_14, :test_ids_15, :test_ids in /var/www/core/lib/Drupal/
Core/Database/Connection.php on line 569
Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented
FileSize
drupal-test-script-2343389-17.patch2.94 KB
interdiff-2343389-14-17.patch3.38 KB

We overlooked another query that queries _all_ test results. Made a separate function that does the (chunked) loading.
Still at a variable limit of 1. Only if this passes on sqlite, we can start increasing the limit.

Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented
FileSize
interdiff-2343389-17-18.txt1.45 KB
drupal-test-script-2343389-18.patch3.16 KB

Still failing ... Adding debug messages.

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented 
Detailed test results
---------------------
DEBUG: test count 1455
DEBUG: chunk count 1
DEBUG: chunk size 1455

Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables' in /var/www/core/lib/Drupal/Core/Database/Connection.php:336
Stack trace:
#0 /var/www/core/lib/Drupal/Core/Database/Connection.php(336): PDO->prepare('SELECT * FROM s...')
#1 /var/www/core/lib/Drupal/Core/Database/Connection.php(536): Drupal\Core\Database\Connection->prepareQuery('SELECT * FROM {...')
#2 /var/www/core/scripts/run-tests.sh(1144): Drupal\Core\Database\Connection->query('SELECT * FROM {...', Array)
#3 /var/www/core/scripts/run-tests.sh(995): simpletest_script_load_messages_by_test_id(Array)
#4 /var/www/core/scripts/run-tests.sh(89): simpletest_script_reporter_display_results()
#5 {main}

Next exception 'Drupal\Core\Database\DatabaseExceptionWrapper' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables: SELECT * FROM {simpletest} WHERE test_id IN (:test_ids_0, :test_ids_1, :test_ids_2, :test_ids_3, :test_ids_4, :test_ids_5, :test_ids_6, :test_ids_7, :test_ids_8, :test_ids_9, in /var/www/core/lib/Drupal/Core/Database/Connection.php on line 569


Saving image build_2014_09_29_143211 at Mon Sep 29 15:01:07 UTC 2014:
Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented

[edit: duplicate comment removed]

Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented
FileSize
interdiff-2343389-18-21.txt626 bytes
drupal-test-script-2343389-21.patch3.18 KB

Added missing global global $args;.

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

Test is running

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

This seems to be working now:
http://testbot-pgsql.drupal-pt.org/sites/default/files/build_2014_09_29_...

Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented
FileSize
interdiff-2343389-21-24.txt681 bytes
drupal-test-script-2343389-24.patch3.18 KB
Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented
Status: Needs work » Needs review
FileSize
interdiff-2343389-24-25.txt1.64 KB
drupal-test-script-2343389-25.patch2.96 KB

Previous patch finally worked with sqlite.
Patch cleaned.

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented
Category: Task » Bug report
Status: Patch (to be ported) » Reviewed & tested by the community
ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented
Priority: Major » Critical
ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

This is a blocker for the testbot, please commit when possible
Thank you

bzrudi71’s picture

bzrudi71 CreditAttribution: bzrudi71 commented

Just startet local test run (just to make sure). Results will follow...

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

@bzrudi71 is your test done? ;)
This patch passed on all of my local tests

bzrudi71’s picture

bzrudi71 CreditAttribution: bzrudi71 commented

@ricardoamaro sorry no. Running --all on mysql seems to take ages ;-) I ran an 'aggregator' only test in front of full testing and that worked. However, I have seen a FATAL because of memory limit on http://testbot.drupal-pt.org earlier when trying to store the results. Let's see if it will happen to me also, and if, we need to higher the PHP memory_limit in web container I guess, but thats outside the scope of this patch...

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

OK, Thanks!

bzrudi71’s picture

bzrudi71 CreditAttribution: bzrudi71 commented

Finally ;-)

I get the mentioned memory_limit FATAL, but that's unrelated to this patch. I tried some test runs with some thousands of tests to make sure the patch is working properly, in terms of chunks - and it did. So from my side RTBC.

About this error:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 4194304 bytes) in /var/www/core/scripts/run-tests.sh on line 1142

We need to file an issue in drupalci_testbot repo.

ricardoamaro’s picture

ricardoamaro CreditAttribution: ricardoamaro commented

RTBC for me also!
@Sutharsan please give your comment

Thanks all

Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented

I've investigated the patch for memory consumption but found no possibility to significantly reduce the memory usage.

bzrudi71’s picture

bzrudi71 CreditAttribution: bzrudi71 commented

@Sutharsan Not that I have really investigated this patch, or looked deeper at the source, but can' we not just write the results chunk by chunk? I mean, instead of collection all results by array_merge(), why not write results if a chunk is processed/completed. This way we will free memory...

Sutharsan’s picture

Sutharsan CreditAttribution: Sutharsan commented

@bzrudi71, sure we can. But currently all messages are loaded first and then processes. So I suggest to open a new issue for your optimisation suggestion.

  • webchick committed 2634b43 on 8.0.x 
    Issue #2343389 by ricardoamaro, Sutharsan: Fixed Drupal core testing...
webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
CreditAttribution: webchick commented
Status: Reviewed & tested by the community » Fixed

Agreed that sounds like an optimization we could pursue in another issue.

As for this one, looks good.

Committed and pushed to 8.x. Thanks! Onward, testbotNG! :)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.