Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.Now that it's easier to add custom build steps within the DrupalCI build flow, there's a big demand in other tests that need to be executed. Some of these tests are normally displayed using HTML or on a 3rd party website.
As we don't want to be reliable on the SLA's of many (or even one) 3rd party websites. There are several services that offer a free service for open source projects. But as soon as we start sending them a million lines of code to analyze 50 times a day, and start accumulating a pile of results that we want to keep in perpetuity, they may ask to renegotiate, *or* we don't get put on hardware that can really handle our behemoth and it takes 16 hours to run one analysis.
This issue is dedicated to discuss how we should move forward.
Topics to discuss
Should we;
- run our own self static analysis tools as part of drupalci?
- partner with a SaaS project like Code Climate or Scrutinizer to do the processing?
- host our own static analysis suite, like http://www.sonarqube.org/ (which has drupal coding standards etc) open source, so not as pretty, but also not code we have to maintain.
- create a custom lay-out for drupal.org so that we can integrate the results
Comments
Comment #2
legovaerI've been thinking quite a lot about this issue the last several months. I see a possible solution where we will host our test results ourselves on a sub-site of drupal.org. So in general this is how the flow would be;
Pitfalls
Comment #3
MixologicThere's a couple of options for publishing results from these various tool options:
Running the tests ourselves, with custom UI for viewing the results on drupal.org
Right now, we are using jenkins to kick off builds, and gather artifacts from those builds. Many of these code quality/analysis/performance/coverage tools offer an output format that is consumable and parseable by jenkins, and is therefore available via jenkins json api. So tasks like code coverage, phpmd, phpcs etc all have a format we can feed to jenkins.
This is what we are doing now with the junit xml that gets sent to jenkins. We use drupal.org as the publishing site, it gathers the output data and formats it for consumption on d.o.
With many of these tasks (phpcs/code coverage/phpmd/etc) we can add a BuildTask plugin, and have jenkins pull in the results and have a rudimentary UI to display the results as an MVP.
We can then, down the road, add UI to d.o. via the api where it makes sense.
Pros: Tailor made to our situation we might add functionality like 'create an issue if errors found', short path to mvp
Cons: Development time for a UI on d.o., new analysis features are not automatic, code that we have to maintain
Running the tests ourselves, with custom UI for viewing the results on a subsite of drupal.org
We've been working toward *not* having as many subsites for drupal.org because the security updates and maintenance burden ends up being significant over time. This option is pretty much off the table from an infrastructure/maintainability standpoint. Theres not a whole lot that can be accomplished on a subsite that cant just live on the main d.o. site feature-wise, so theres not much win here to make it worth the tradeoff.
Finding a 3rd party Service Provider
We can look to external service providers to provide this functionality. Service providers are typically engaged in giving their end users a good experience, so as new things come along, they might add new features and we dont really have to do anything at all. Additionally oftentimes they cover multiple tools, like eslint and phpcs and phpmd, so we get all of them in one fell swoop. Another advantage is that some of these external tools are familiar to developers on other projects, so the potential learning curve may be lower and easier to compare (say codeclimate drupal vs codeclimate joomla). The key is if they can do it for a project our size. Running d8's codebase through many of these tools ends up crashing or running out of memory on a typical box, so we might need the horsepower that we have now to make them happen. That can be costly, and might not align with the interests of the external providers.
Pros: Bulk Functionality, Somebody else's problem to maintain, new features happen without intervention, developer familiarity
Cons: Relationships may need to be negotiated/renegotiated, might not be free, and may not last forever. Companies go out of business, servers go down etc. Might need a combination of providers (ie. Blackfire/Scrutinizer/Coveralls) to get all the functionality we want.
Self hosting third party application
Hosting something like sonarqube internally could also satisfy some feature needs, depending on whether or not we want to support something like that. It would add a java application into the mix for us to maintain, it'd be another, different thing we need to pay attention to for security updates etc.
Pros: similar OOTB functionality as a service provider, we control the host size
Cons: server maint/security, may be unfamiliar like java hosting.
Conclusion
We've discussed this internally at the DA, and for us, the ideal scenario is if we can find a 3rd party service provider that can handle our needs with a reasonable agreement, then that's where we would lean. If we want to MVP stuff in the interim, then hosting it on jenkins would be adequate while we find out if a 3rd party would work. If we can't find a 3rd party provider for something that we've got mvp'd then we'd probably work extending the UI to drupal.org itself to create a better user experience, time/priorities permitting.
Comment #4
MixologicRetitling because I cant ever remember what this title meant.