[plan] Remove blockers to the "minimum viable" state of DrupalCI to ship Drupal 8 RC1

Thinking about:

What level of testing on each? (per-patch; per-commit; periodically, and if so, what period?)

It would be nice if all patches had PHPUnit tests run on all supported PHP minor versions - it only takes a minute to do a full PHPUnit run.

I do not think that it is a requirement to test the full matrix of all databases vs. PHP versions etc. as the chance of having a failure that only occurs on PHP 5.5. with SQLite is low. I think the minimum requirement is to test core's supported databases in the minimum supported version, and core's supported PHP versions. We should test combinations that make it easiest to quickly identify the potential cause of the failure:

1. PHP 5.4 & MySQL/Maria
2. PHP 5.5 & MySQL/Maria
3. PHP 5.6 & MySQL/Maria
4. PHP 7 & MySQL/Maria
5. PHP 5.4 & SQLite
6. PHP 5.4 & PostgreSQL

I do not consider different cache backends, PHP configurations, etc. a release blocker; all that should wait until much later.

In terms of timing, I think the minimum requirement that we have is:

• Nightly testing with emails upon failure including a diff against the previous night's result for environments that do not currently pass the test suite in HEAD. (AFAIK in my list above this would be #3-6.) The committers adopt a policy of reverting things that introduce a new failure.
• Commit-level testing of combinations that do pass in HEAD (currently #1 and I think #2).
• Patch-level testing of #1 with an option of on-demand patch-level testing for other combinations. We need this so that we can test issues that are likely to regress alternate databases against those databases, and to confirm that fixes for a specific environment actually... fix.
• All of this easily visible in the UI and issue queue.

Then we can decide whether to increase the test rate for #3-#6 once they pass, weighing the benefit against the extra resource requirements.

I like @alexpott's suggestion of running PHPunit tests on all of #1-6 on the patch level as well since the resource overhead is minimal. We could also test simply installation and the database component tests per patch as well to save resources while getting most of the benefit.

+1 to the 6 combinations and 4 bullet points in #4 as being a good viable state. If any of those items would take long to implement though, please say so, because maybe we can get away with less than that, or substitute one of those capabilities for something easier to implement. Having even one of those bullet points up and running sooner would be better than additional weeks/months of not having any of them.

@webchick, yeah, I think it should be just the minimum supported version for each.

+1 to #4, we might want to tweak things later but this would be a great point to start from.

The association team met up to review these MVP requirements this morning and wanted make some clarifications and ask a couple questions.

With respect to MySQL/MariaDB - for now, wherever the requirements list above says "MySQL/MariaDB" we'll be opting for MySQL. At the same time we want to note that in the long run we may not want to treat these as interchangeable, as MariaDB may begin to diverge in some future versions.

Regarding Postgres - During the Portland sprint we came to conclusion that we'd prefer to support Postgres 9.1.2 as our minimum version. We know that 8.3 is still considered an official minimum, per: https://www.drupal.org/requirements but PHP 8 is well past end of life.

That decision isn't really ours to make of course - but this issue makes the case for 9.1.2 as well: #2464481: PostgreSQL: deal with case insensitivity

So is it satisfactory to use MySQL where ever MySQL/MariaDB has been suggested above and to build Maria containers sometime post MVP?

And can we at least strongly consider 9.1.2 as minimum version for Postgres on the new testbots?

PostgreSQL 8.3 is EOL on Centos 5 and 6 as well. 9.1.2 probably should be the earliest version to support.

On Ubuntu 14.04 (Trusty), which the testbots run on, the earliest version is PostgreSQL 8.4 though earlier versions are still supported on older Ubuntu versions.

Thanks @hestenet and @mradcliffe. I filed #2477413: Increase minimum version requirement for Postgres to 9.1.2 to discuss increasing the supported minimum version for core for 8.x. (I think one dedicated PHP 5.3/MySQL bot is sufficient for D7 testing, so we don't need to worry about supporting testing on the versions of Postgres supported by D7.)

Doesn't the current infrastructure run MariaDB? Either way I think it's OK to treat them as interchangeable until we don't.

I'd personally tend towards testing later versions of postgres (i.e. not the minimum, but the current stable). At least this way if core raises its requirements, then no need to update the bot at the same time.

Also in terms of postgres/sqlite testing if we have one bot, I'd probably tend towards current PHP stable on those too (i.e. 5.5) - since again if we drop support for PHP 5.4 it would just mean stopping sending 8.x patches to that bot, not 'HEAD broken' on postgres/sqlite until an upgrade of PHP on those.

Don't have a strong preference, just seems like it could be less disruptive when core requirements change.

If PHP 5.5 is what we are going to list as the recommended version, I'm fine with having testbots run that. However, we should at least still run a linting pass on 5.4 to catch accidental use of newer language features as 5.5 does have a few convenience shortcuts that we don't want people to use accidentally (yet). Functionality-wise, the 5.4->5.5 upgrade is, as I understand it, smooth enough that we likely don't need the full test suite run on both.

However, we should at least still run a linting pass on 5.4

That would be covered by the MySQL + PHP 5.4 bot. MySQL we'll have tests with every major PHP version.

For 5.5 testing I was specifically talking about the postgres and sqlite bots which at least to start with will only test on one PHP version each.

We'll get commit-level (not patch-level) testing on everything that's expected to pass if I understand things correctly. And a test run every 24 hours for environments with known failures.

Doesn't the current infrastructure run MariaDB? Either way I think it's OK to treat them as interchangeable until we don't.

Yes, for Drupal.org sites. The DrupalCI containers do not necessarily need to follow.

So is it satisfactory to use MySQL where ever MySQL/MariaDB has been suggested above and to build Maria containers sometime post MVP?

Yep. This is inline with using whatever would be the most common platform, and MariaDB definitely isn't it. My understanding is also that they're more or less inline until a later version of 5.5.

Updating the issue summary for this.

As we drill down into this a bit further we realized we could use a bit more clarification around this requirement:

Nightly testing with emails upon failure including a diff against the previous night's result for environments that do not currently pass the test suite in HEAD. (probably #3-6.) The committers adopt a policy of reverting things that introduce a new failure.

Could someone unpack what they'd like to see for the diff against the previous night's result a bit further?

Is this diff a diff of which tests were passing/failing the previous night vs. which passed/failed in the latest run? Or is there something more specific needed?

For the "ideal state" side of things, I think that sometime in future when everything works properly, patch-level testing for core should be done on the fastest possible PHP / DB combination, which probably means PHP 7 & SQLite in memory/tmpfs.

As further discussed in: [2477413] - For the DrupalCI MVP we'll be able to provide Postgres 9.1.18 and 9.4.4 (rather than Postgres 8.3). At this time that issue is awaiting RTBC, but there seems to be consensus. Updating the issue summary here accordingly.

Dries has approved #2296557: [policy] Require PHP 5.5. So the lowest PHP version we are going to use is 5.5. Updated the issue summary to reflect this.

Adding some much needed detail into the DA's work on deploying the DrupalCI MVP. My apologies that more of these updates weren't included here in real-time!

DrupalCI MVP Deployment to Drupal.org

On June 11th the DA demoed the DrupalCI integration with Drupal.org with some of the core maintainers. We collected some initial feedback, asked some questions, and set out the path to bring DrupalCI into production over the course of the next two weeks. The main architectural shift with this MVP proposal was to remove the results server component of DrupalCI in favor of directly parsing test results on the Jenkins/Dispatcher instance and exposing those results on Drupal.org.

On June 17th the DA enabled DrupalCI in production for branch and commit testing for Drupal Core. The automated testing tab now shows the DrupalCI tests and test configuration matrix below the PIFT/PIFR tests. At the same time we enabled some initial jobs for the environments that were stable. (In particular, holding off on the Postgres testing as we worked to make the results we were getting from these bots more consistent). We emailed a number of the core maintainers to let them know that these tests were now available.

Shortly after these initial tests were enabled the minimum version of php for Drupal 8 was increased to php 5.5, so we updated the tests accordingly.

On June 23rd the DA enabled testing of Postgres on Drupal.org after better stablizing the Postgres bots. However - we are still seeing some inconsistencies in test results from one run to the next on the Postgres bots - and could still use the assistance of core devs working on Postgres support (particularly those who have stable containers running locally) in fixing up the stability of our Postgres environment.

@Drumm also created test history pages on Drupal.org which can be seen by clicking through any of the test result summaries on the Automated Testing tab of the project, (and will be available wherever else those summaries appear, such as next to a patch in an issue). The test history page provides a summary and meta data about the most current test, as well as a link to the results in Jenkins. It also provides a history of the past tests run in the same environment.

Since that time we have 1) begun rearchitecting the jobs in Jenkins to better allow us to provide a diff against the previous results for each environment - a precursor to enabling the email notifications to core maintainers, and 2) begun improving the Junit/XML output of the test runner to provide a better organization of the test results (by group) and expose additional detail, such as debug messages that are available in the qa.drupal.org version of the tests results.

I'll update the results summary with the current hit list, link in or create issues for those where appropriate, and also list some items that are out of scope for initial deployment, but that will need to be done to retire PIFT/PIFR.

(As a sidenote - we had some breakthroughs that enabled us to dramatically improve the stability of the old testbots, so that maintaining those bots has required much less labor, saving us from being pulled away from this work. Thank you @mixologic!)

Support for PHP 5.5 was mostly complete for the PIFR/PIFT bots already, and @isntall finished the pieces missing there. In short:

• We have bots ready to run PHP 5.5 on pifr/pift
• @drumm was waiting to update d.o to request the PHP 5.5 bots based core devs need (we NEED to have it on the pifr/pift ones because we need DCI patch testing along side pifr/pift patch testing to verify we are regression free)
• Since we want to raise that minimum requirement to 5.5 on the 15th (middle of next week) we’re out of time to turn on DCI patch testing and have any reasonable length of time to do side by side testing
• We need to schedule this change in a change record/change notice for next week and coordinate with core devs the time we switch pifr/pift to 5.5

Let me ask (this is vital for one of my patches): will these bots have opcache on? I presume yes otherwise they'd be very slow but I need to ask.

chx: For DrupalCI Opcache has been enabled by default . So will APCu. And for the old testbots on qa.d.o we will also have Opcache and APCu enabled by default.

We need to schedule this change in a change record/change notice for next week and coordinate with core devs the time we switch pifr/pift to 5.5

Sorry if this is a stupid question, but why not just do it? Like as soon as someone with the necessary creds is able to perform the switch? I'm assuming we're only talking about making this switch for testing 8.x patches, in which case, how would this impact core devs in a way that requires coordination? As far as I'm concerned, the sooner it happens the better, no need to wait until the 15th.

Drupal 8 is already running on 5.4 while 7 and 6 still uses 5.3. So switching 5.4 to 5.5 should not affect them?

#38: I am pretty happy they want to coordinate with core-devs.

i.e. if they had "just" done the testbot migration in the middle of "Lemon", that would have severely impacted productivity.

In that regard (and because there could be problems), I think it is good to warn in advance.

+1 to warn one or two days in advance (more is not necessary though).

Today is the last day of the sprint. If infra is ready, I'd recommend announcing the 5.5 change for perhaps Friday July 10, or if weekend deployment is an option, over the weekend? I agree on "sooner the better" in general.

The current plan is to build a php5.5.9 testbot today, and manually force it to grab the patch from https://www.drupal.org/node/2508231#comment-10096748, though that currently doesnt appear to apply, may need a reroll? (https://dispatcher.drupalci.org/job/php5.5_mysql5.5/19/console)

We're running into difficulties on getting the legacy testbots running 5.5.9 (they are on debian wheezy), so in the interests of time we're going to go ahead with legacy testbots on 5.5.26.

Currently, the DCI testbots are running on 5.5.23. We're working on downgrading the container to 5.5.9.

If our smoke test succeeds and we proceed with the legacy testbot upgrade, we will have 5.5.26 in parallel with 5.5.23.

During that time, green patches will be "greenish" - meaning that there is a slim chance that any patch going in could be impacted by whatever php fixes were put into place between 5.5.9 and 5.5.26, and that there may be one or more 'new fails' revealed once we start using the drupalCI containers at 5.5.9.

Hopefully thats acceptable risk, otherwise we can spend a lot more time trying to figure out how to get the legacy testbots on 5.5.9.

I don't think 5.5.26/23 is a problem at all tbh.

Wow, it's really cool the multi-environment testing is live!

Why isn't it turned on for Drupal 7, though? I seem to have the power to turn it on via the UI (and would like to) but don't want to if it will break something. Will it?

In the short term, this is actually more important for Drupal 7 than Drupal 8, since there are hundreds of thousands of live Drupal 7 sites actually running with these various combinations of PHP/database versions...

More specifically I'm asking about doing this:

D7 has some differences we haven't been looking for yet. runtests.sh is in a different location, which we have set up: http://cgit.drupalcode.org/project_issue_file_test/tree/includes/pift_ci.... The best way to help is to run the test runner containers locally against D7 to see if tests pass, and if other environment variables need to be set.

Within 8.x, core maintainers should go ahead and re-configure as necessary.

By the way, if any of those select boxes don't make sense to run, maybe 8.x with PHP 5.4, some of those select boxes can be removed to declutter things. This should help out contrib maintainers, so the test combinations are helpful.

Drupal 8 on PHP 5.4 (or below) definitely doesn't make sense anymore, so if removing that possibility is possible (heh), that sounds great.

Done.

I don't know where to post this but can we test patches in #2530640: PHP 7 testing issue on PHP 7 & MySQL 5.5?

We may need to consider testing on both MariaDB and MySQL.

https://www.drupal.org/node/2228217#comment-10134064 (via @alexpott) shows a local fail on MySQL (without long index support) whereas MariaDB passed.

Also Drupal\system\Tests\Entity\EntityDefinitionUpdateTest fails locally for long indexes.

@catch, @alexpott: this is a false pass due to configuration (strict sql mode is now enabled by newer versions of MySQL; it is not in MariaDB, where this is a warning instead). The same issue is discussed in https://www.drupal.org/node/1314214#comment-9984519

Summary of discussion:
- Truncation is unsafe on unique indexes because a constraint can not safely be applied.
- It is mostly safe on selective non-unique indexes.
- Thus "not-safe" to rely on database truncation by disabling strict sql mode to create tables.
- A more complete fix is available in 5.7, where the barracuda file format + innodb_large_prefix is enabled by default. This allows indexes up to 3072 bytes.

There are a number of features which have been removed or deprecated in MySQL 5.7. May I suggest adding it to the test-matrix as well, so that no features are introduced which specifically make use of what is intended to be removed?

See: http://mysqlserverteam.com/removal-and-deprecation-in-mysql-5-7/

Updating the summary to reflect that:

• Contrib testing is now enabled
• PHP 7 testing is enabled

Also - this has been called out through some back channels, but please update: #2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure if you see any regressions between old bots to new, issues with the test output, etc -- basically any feedback you have not covered by existing issues that would prevent turning off the old testbots.

Awesome!

It looks like PHP 7 testing is only an option for Drupal 8 though (not Drupal 7) - any reason?

I'm also wondering about the fact that PHP 5.3 is not an option, even though the legacy testbots run on that (but will bring that up in more detail at #2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure). If a project decides to enable the new options now (and e.g. selects "PHP 5.4 & MySQL 5.5" as the one to run on "Issues & commit") would that mean they lose the ability to test patches on PHP 5.3 with the old testbot?

Getting 5.3 running in DrupalCI is definitely part of the plan (it's hiding in the issue summary up there under "Not in the scope of MVP, but required so that we can retire PIFT/PIFR" probably needs its own child issue at this point). Getting those bots working and getting CI working for D7 in general is definitely a next major priority.

Ah thanks, sorry I missed that. Well it's now mentioned explicitly at #2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure too, which is probably a good thing :)

Definitely a good thing. Added a new child issue(#2538042: DrupalCI PHP 5.3 containers need to be stable and available for testing ) and updated the summary

It looks like PHP 7 testing is only an option for Drupal 8 though (not Drupal 7) - any reason?

I assumed Drupal 7 wasn't going to support PHP 7. I can add that as an option if testing there is useful.

Following up to #59 for both the short and long term, I think we should change the default test runner to a MySQL or MariaDB configuration that doesn't support long indexes - then patches added indexes that are too long will fail and we won't break HEAD. afaik that's the only issue we have where MySQL version/variant matters at the moment.

Multiple MySQL/MariaDB testing would be great to have down the line for other things that might come up, but even if we have that, the default should be as strict as we support.

On #2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure @David_Rothstein pointed out that we probably need a 5.3 environment for D7 testing only.

Based on a test run I did, it looked like D7 actually had fewer issues with PHP 7, at least the sort that requires API changes (like renaming String/Null/... classes). So enabling certainly doesn't hurt.

Yay, PHP 7 testing.

Would be awesome to include a php --version into the build output so it's clear against what we are testing exactly. For example, a lot of those reported exceptions have been fixed a while back, see https://bugs.php.net/bug.php?id=69996.

If you look at my test results in #2454439: [META] Support PHP 7, there were new PHP 7 changes and bugs every other week that broke stuff again. So it would be pretty important to have an up to date PHP7 build that is updated from time to time, e.g. with the latest beta/alpha version or just build it again every few weeks until we have a stable release. Is that possible? That would allow us to quickly identify regressions/changes in PHP7.

@catch, what did you think about my comment re: 5.7 deprecation? ( http://mysqlserverteam.com/removal-and-deprecation-in-mysql-5-7/ ) I ran the d8 test-suite some time back and it didn't appear to be using any features scheduled to be removed, but it's possible they could be introduced. This is a similar-ish problem to PHP7.

@morgantocker that's good to add to 'ideal state', I don't think it's 'minimal state' at all. Changing the default to not support long indexes feels like 'minimal state' though.

I assumed Drupal 7 wasn't going to support PHP 7. I can add that as an option if testing there is useful.

Please do - thanks! It's not something that's badly needed right away, but as others pointed out above it will be needed eventually.

The initial tests Berdir did with PHP 7 on Drupal 7 are at #2454439-31: [META] Support PHP 7 - so even before people are ready to use Drupal 7 on PHP 7 in practice, it's still useful to have a testbot keep track of any progress on that issue (rather than having to run the tests manually).

7 for 7 is enabled now.

Some contrib projects have proactively enabled DrupalCI testing for Drupal 7.x modules, and you can see how it fails, https://dispatcher.drupalci.org/job/php5.6_mysql5.5/38/console. I'm not the expert on this, I think the SQLite DB used by SimpleTest for results is set up a bit differently in D7.

For contrib, and even core, do go ahead and remove the legacy testing when DrupalCI is meeting your needs.

@Berdir in #72

include a php --version into the build output

I guess that would be #2204253: Output PHP version of testbot environment in review log

#2 is my fault. I had success running a certain configuration and asked @isntall to try it to see if it would fix random exceptions, but it ended up not being successfully on AWS. The commit has been reverted, but the container may not have been rebuilt yet.

Going to do some reorganization on the issue summary here so that we're only tracking the D8 blockers in this issue and that we use #2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure for the disable legacy testbots hit list. Also going to add some more insight into our priority order and who on the DA team is working on these.

Branched feature requests for the Drupal 8.1.x to a new issue #2553289: [meta] Feature requests for DrupalCI to support Drupal 8.1.x