Base DCI containers off official containers

I'm curious ... define 'official' ...

Not maintaining our own containers makes it much more difficult to 'tweak' those containers to meet the changing needs of our testing system. One of the advantages of this system has been our ability to easily modify our containers as new needs are met ... 'official' containers, to me, infers the use of someone else's builds. Admittedly, I haven't been around for the discussion, and might be misinterpreting ... but if not, then I have some concern with this direction.

Unless we're talking about the 'FROM' line in the 'base' dockerfiles ... in which case, are these not 'official' already?

Some restructuring will be needed for php official images to be used.

Since we have made changes to php-base for several issues to make varying types of tests available.

A good example is #2580007: Add phantomjs2 to the testrunners.

It might be a bit harder to maintain if we had to add all those changes to each web container.

I guess a start is getting a list of dependencies that aren't provided by the official php container.

A suggested work around so we don't have to add repeated code in each web container is to use #include and use make.

What do the maintainers think about this? Its going to affect you the most.

Docker isn't going to change the syntax any time soon to add an 'INCLUDE' call.

In the github discussion people suggested using MAKE. Which would work.

It is just that you as a maintainer would have another step as apart of deployment.

I am happy with what ever you suggest.

I would say based on your comments in #2609562: mysql-5.5 official container that it might be easier to start with the web containers.

Linked some issues that will need to be re-implemented after the changes to the web-containers.

Updated Summary

Pretty much all the web containers are ready to be tested. 5.6 and 5.3 still need changes.

All web containers are ready to be tested. 5.3 will work but needs re-factoring.

A shell script could be written to add all the packages we need.

In the dockerfile ADD can then be used to add it to the image.
http://docs.docker.com/engine/reference/builder/#add

The script can then be executed using normal methods in the dockerfile.

Do you think this is reasonable?

Are we able to switch each individual container over? Or do they all have to be switched over at the same time?

Any decisions here?

I'd review the child issues if I knew how.

Basically all the containers will build in docker.

The problem is then running them using DrupalCI and having the test results remain the same.

You can get information about running it locally at https://www.drupal.org/node/2487065.

An easy issue to start with would be #2620688: Create script to generate dockerfiles for containers, you just need to go through and find all the packages that are used by all the containers.

I have to admit my Docker-fu is no good, so I'm only guessing here...

But doesn't it seem like the point of having containers is that you can build them once according to spec, and then just use them in your infrastructure without having to rebuild? As your specs change you just poke the build process and it makes fresh new images?

So it seems like the thing to do is host our own images built by our own infrastructure.

Creating an install script for each container seems like the same thing as not using Docker.

Mile23, I think you misunderstand docker a little bit.

Basically you write a docker file which is run to generate an image.

When you have to maintain multiple images which can't be based off of the same base image (This is the case when we use official base images for php, mysql etc.), but you want the same packages. It become tedious to update each docker file.

So we looked into it and the easiest way is to write a script which is run when the Docker generates the image.

You still make changes to individual docker files for packages only required by that container. But that is rare.

The script is not used when the container is running.

Let me chime in here with a bit of history ... I'm not remotely sold on changing over to 'official' containers, based on past experience with the testing infrastructure.

One of the prime requirements and key characteristics of our testing infrastructure is that the environments provided by our containers need to be stable, consistent, and repeatable. In the past (on PIFR), making seemingly minor and trivial changes within the testbot environment would invariably result in some false failure within the core codebase test suite ... and there were more than a few occasions where these failures would lead to hours of debugging and troubleshooting by folks whose time was taken away from the development and progress of core. Stability and consistency are two of the primary tenants that drove development decisions, including the decision to build our own containers.

The issue with using one of the 'official' containers is that we can not guarantee the level of consistency which is desired (nay, required!) within our testing infrastructure. Even simple minor release upgrades of php or database packages have been enough to derail our entire testing infrastructure in the past, sometimes for days ... and so, in order to ensure that we have an opportunity to fully test these changes, and do not get caught unawares by an upstream package update in the base container, the decision was made to build and maintain our own container stack.

The other factor that went into this decision was a historical pattern of drupal-specific parameter tweaks and configuration/settings changes having to be made within the database and php environments on the old PIFR testbots ... while not all of these are necessary to install and use drupal, there were cases where we needed to tweak settings in order to be able to provide test coverage for a very specific configuration - one example which comes to mind is needing specific APC settings in order to trigger the running of environment-specific tests within the overall test suite, which otherwise were not executed during test runs. While this consideration could be handled in a container layer on top of the base provider containers, it means that we still need to build and manage our own drupalci-specific layer on top of each individual 'official' base container ... which results in one extra layer in the container stack compared to the current model (where these tweaks are made directly in the same base container that sets up that particular database provider).

I would also question the statement that using the base containers would speed up the building process ... each of the containers in the DrupalCI stack is built and published on docker hub, so there should never be a need for anyone to need to build the container stack from scratch (unless doing so to test an environment-specific change) ... all that is needed is to download the compiled container from hub directly. Because we have custom-built containers which can be optimized to our own usage needs, as opposed to a generic 'official' container image geared to the general use case, I'd suggest (speculatively, I haven't actually checked!) that we may actually end up with smaller container images right now than what we would have with the 'official' images as a base ... and as the majority of use cases would simply require a download instead of a build, a smaller custom image size might actually be the "faster" container.

(Note: edited for spelling and grammer)

I guess, to be fair, I should temper my comments above somewhat ... because at the time these decisions were made (i.e. Docker 0.6), I don't think that many of these 'official' images even existed.

I see now that the php containers are locked down to a particular php point release, in which case this does provide us a benefit over the manual building of containers ... as we have also been bitten (albeit rarely) in the past by changes to the apt dependency trees introduced after a broken release, requiring manual troubleshooting and pinning to specific versions in order to get our PIFR bot build process working again.

So this would be a good argument in favor of using the base containers.

Given that work and personal life has kept me out of the loop for the last number of months, what is our current status on this particular issue? Has any of it been rolled forward into the production stream?

Its been pretty much at a stand still since efforts have been focused in other areas.

There are still a heap of problems with getting them to work, so the switch over won't be easy.

Also it is good to note that using the official containers make including extensions easier. Instead of our current situation which recompiles php every time.

Again, I disagree that the current situation re-compiles PHP every time ... it only re-compiles PHP once for every change we make to the container stack.

The reason for this was to support php-env, and be able to load multiple minor versions of PHP onto a single container image, and switch between PHP versions on the fly. One of the requirements we have been asked to support from a DrupalCI perspective is the ability to test patches against any particular minor version of PHP (which I have dialed back to 'ability to test patches against any of the last three/five minor versions of PHP'). This is a legitimate requirement; as we have often found ourselves troubleshooting a random failure that has creeped in during a recent php minor release change and needing to test against multiple previous versions to determine exactly which PHP version introduced the fault. Because most of these have been 'random' failures, they don't always appear within the space of a single php minor release.

In order to support this requirement while using the 'official' php base containers as proposed here, we actually end up significantly multiplying the number of containers that we need to support ... because we now have a situation where introduction of a container layer on top of the php layer needs to be duplicated for every single php minor release version container that we support. Instead of starting from a single base container, and diverging as we move up the container stack, we start from a point where we have dozens of base containers ... and I'm concerned about just how complex administration of those multiple container versions will become (though I acknowledge that we also offload some of that administration to the 'official' container maintainers).

So, while I seem to be a distinct and lonely minority in wanting to continue to support the php-env model, there were some very very good reasons for us choosing that direction ... I believe that the lack of support for the model stems primarily from the fact that most folks only evaluate the option from the perspective of "today's implementation" (which, admittedly, is one where php-env introduces more complexity than value), as opposed to evaluating the overall flexibility of the the system for new and innovative job types in the future (such as a dedicated job which could directly run a specific test against multiple php versions all within the same container, and tell us directly which php version introduced the regression, instead of having to run multiple individual test runs on a variety of different containers and manually analyze the results).

This is blocking #2600626: [PP-1] Ensure availability of node/npm in the testrunners, which is major. So bumping this to major too.

Could we get an update on what still needs to be done? Is it blocked by the MySQL issues still?

This is not the only thing blocking eslint. There's actually several things in the pipeline in front of adding additional jobtypes to drupalci.

The primary one is functional regression testing, followed by some major refactoring to allow for multiple job types to exist. Finally, basing off of official containers, and potentially re-doing the way we manage the environments (using docker compose instead of managing from within drupalci).

The testing is almost there. And now theres actually resources to move it forward. So I'm going to go ahead and mark this as postponed until such time as we've got the other work done.

Thanks Mixologic, if you have time could you please link those issues as parent issues so we can track them as blockers?

If you'd like to learn some testbot code and write tests, you're welcome: #2680565: [meta] Create functional tests of run command output to test production essentials

Writing the tests is the first step in this meta: #2811773: [meta] Test Runner Re-Architecting

#2531564: Fix leaky and brittle container serialization solution is running into spl_object_hash() being broken on the php 5.4 build specifically. I have been unable to recreate this on the 5.4 version bundled with Ubuntu so this seems to be specific to bugs patched by distributions but broken in our build so this should unblock that issue.

Now that #2811773: [meta] Test Runner Re-Architecting is done, should we start work on this? Or are there other issues blocking this?

This is the next logical step.

Part of the refactoring was to remove all container building/docker files etc and move it to its own repository : https://www.drupal.org/project/drupalci_environments

As far as making environments work better, we need a better build process.

The current one was totally broken, as it relied on dockerhub to build our images, and had a chain reaction that caused image builds to take upwards of 48 hours.

Instead we need to use our internal jenkins and build machine to build the images and push them to docker hub.

We also need to move all of the issues currently marked "environments" over to that other issue queue.

I'll put together a mini plan issue in environments tomorrow.

Moved to DrupalCI Environments.

I moved them all for you. Sorry for the email spam.

Oh, no worries. I should have mentioned that I have a way to do that programatically, but hey, this is cleaner and its done so thanks!

So do we create a task to remove the containers from drupalci_testbot?

Also should we merge the changes from the branch in drupalci_testbot related to this issue?

Woops, just checked and see this was already done! Good stuff.

Okay, so we have a new container build process, and it turns out that using official containers, particularly for php, will not work for our needs. We need specific things done at compile time, like preserving the debug symbols without removing the optimization flags, that the official containers cannot provide.

The official php containers are pretty simple, and I used them as a basis for what we have. We still need to explore using somebody elses mysql/pgsql containers to see if they can fit what we require.

Wow this is impressive.

We had issues with mysql 5.5, but mysql 5.7 worked. isntall was the last one working on that as far as I remember. He would be better placed to summarise the issues with it.

I might be missing something but can't we just create our own Dockerfile extending from official php images and then perform our additional logic?

We should also be looking at the alpine images but that is a later issue.

The additional logic we need to perform is changes to the way that php is compiled, so no, we cant really use their images because have to recompile php. The only thing their containers do is setup apache, download php src, and compile php, and then delete the src. So we'd have to download the src again, and compile php again, so there isnt really that much saved in using FROM their containers.

And using alpine is okay, but is main goal is reduction of container size, which we're not really at all concerned with because we bake the containers into an aws AMI whenever we build new ones. Also, alpine introduces certain classes of bugs that nobody using real world drupal is facing, so its better to be on a more common distro for the testing infra.

Ah I assumed we could leverage one of their variants and that anything custom was just an ext that could be enabled with docker-php-ext etc. This is unfortunate as we would lose out on the vetted updates and docker layers security scan. Ideally there is some way to work around this like: https://github.com/docker-library/php/issues/308#issuecomment-252199440 but I appreciate the clarification. Definitely understand the issue now. ^_^

As for alpine my main concern indeed was the reduction of container size by about 70% and the reduced surface area. However your correct this is largely moot as is all baked into the AMI. This was more coming from a development view point where testing on public infrastructure like gitlab or Travis where it is important to have reduced layering and small quick builds. Thanks for the correction and taking the time to respond :)

Just for anyone reading as I didn't want anyone to walk with the conception that there are many bugs in alpine. In particular the musl -> glibc bug has been reduced as most top tier projects now support it but definitely does still pop up once in a while. I mostly choose Alpine since it has pretty great memory efficiency and good DX for docker layering thanks to some tricks within apk.

Yeah it was essentially the second comment on this issue from one of the maintainers: https://github.com/docker-library/php/issues/111 That made me realize we had to fork, if it were just extensions or modules thatd definitely be desireable to keep on their vetted images.

The alpine bugs revolved around compiling additional extentions and not necessarily having all the right libs. Those might have been solved, but I saw enough of them that it was worth sticking on a common distro. i.e. I didnt try cause it looked scary, not because of any real issues encountered.

"... it turns out that using official containers, particularly for php, will not work for our needs."

Good to see folks finally coming around to this realization ... this was a very early conclusion reached by the original DrupalCI team prior to handing off to DA staff. Over the last number of years, the Drupal test suite has managed to uncover some incredibly obscure quirks and a few bizarre PHP bugs - managing our own php container build is certainly an administrative pain, but was viewed as the only approach that gave us the flexibility to rapidly adapt to these as they were uncovered.

Unrelated (but not worth a separate issue) I'm also happy to see that these projects have finally been moved over to an official DrupalCI namespace, as I requested back at DC New Orleans ... though a fork might have been more appropriate given the functional differences of opinion between Ryan and the original developers, rather than a firing/hijacking. ;)