Problem/Motivation

#3324984: Create test that reports % of config entity types (and config schema types) that is validatable is a nice start, but:

  1. it's a core patch, and requires constant re-rolling
  2. … which also means I cannot apply that core patch to any version of Drupal core and get some numbers out
  3. it focuses on config types alone, whereas config objects are more impactful: it may be better to reach only 30% of types if it gets us to 90% of config objects in core to be validatable
  4. it produces complete output, for every config type property path in Drupal core, which means it produces a LOT of output

👆 For this to be widely adopted by the community, and to be able to answer the question "so how much more work is there/how far are we?", we need ONE metric, which we can then also visualize.

Steps to reproduce

N/A

Proposed resolution

  1. Add drush config:inspect --statistics
  2. Write script that, when run from a Drupal core clone, will run drush config:inspect --statistics on every commit

Bonus: visualize this data. But for now … this metric's evolution per day in Drupal core will suffice 👍

Remaining tasks

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#27 validatability.png218.07 KBwim leers
#22 validatability.png429.49 KBwim leers
#6 Screenshot 2023-10-05 at 6.37.28 PM.png2.49 MBwim leers
#2 Screenshot 2023-10-05 at 2.19.52 PM.png661.63 KBwim leers

Issue fork config_inspector-3391990

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

Wim Leers created an issue. See original summary.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
StatusFileSize
new Screenshot 2023-10-05 at 2.19.52 PM.png661.63 KB

This is what it looked like a few hours ago:

That only worked going back to August, for July I had to do some cache clearing, and for Jan 1, I had to do crazy composer work-arounds 😬

But it’s working! Still chasing one more bug…

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Turns out that this is a reproducible bug in Drush 12:

$ vendor/bin/drush updatedb --yes
 -------- ----------- --------------- ---------------------------------------- 
  Module   Update ID   Type            Description                             
 -------- ----------- --------------- ---------------------------------------- 
  system   10101       hook_update_n   10101 - Change the {batch} table [bid]  
                                       field to serial.                        
 -------- ----------- --------------- ---------------------------------------- 


 // Do you wish to run the specified pending updates?: yes.                                                             

>  [error]  TypeError: ArrayObject::__construct(): Argument #1 ($array) must be of type array, bool given in ArrayObject->__construct() (line 15 of /Users/wim.leers/core/vendor/consolidation/output-formatters/src/StructuredData/AbstractListData.php) #0 /Users/wim.leers/core/vendor/consolidation/output-formatters/src/StructuredData/AbstractListData.php(15): ArrayObject->__construct(false)
> #1 /Users/wim.leers/core/vendor/consolidation/output-formatters/src/StructuredData/UnstructuredListData.php(21): Consolidation\OutputFormatters\StructuredData\AbstractListData->__construct(false)
> #2 /Users/wim.leers/core/vendor/drush/drush/src/Commands/core/UpdateDBCommands.php(137): Consolidation\OutputFormatters\StructuredData\UnstructuredListData->__construct(false)
> #3 [internal function]: Drush\Commands\core\UpdateDBCommands->process('10', Array)
> #4 /Users/wim.leers/core/vendor/consolidation/annotated-command/src/CommandProcessor.php(276): call_user_func_array(Array, Array)
> #5 /Users/wim.leers/core/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
> #6 /Users/wim.leers/core/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
> #7 /Users/wim.leers/core/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(391): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
> #8 /Users/wim.leers/core/vendor/symfony/console/Command/Command.php(312): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #9 /Users/wim.leers/core/vendor/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #10 /Users/wim.leers/core/vendor/symfony/console/Application.php(314): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #11 /Users/wim.leers/core/vendor/symfony/console/Application.php(168): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #12 /Users/wim.leers/core/vendor/drush/drush/src/Runtime/Runtime.php(110): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #13 /Users/wim.leers/core/vendor/drush/drush/src/Runtime/Runtime.php(40): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
> #14 /Users/wim.leers/core/vendor/drush/drush/drush.php(139): Drush\Runtime\Runtime->run(Array)
> #15 /Users/wim.leers/core/vendor/drush/drush/drush(4): require('/Users/wim.leer...')
> #16 /Users/wim.leers/core/vendor/bin/drush(119): include('/Users/wim.leer...')
> #17 {main}. 
> TypeError: ArrayObject::__construct(): Argument #1 ($array) must be of type array, bool given in /Users/wim.leers/core/vendor/consolidation/output-formatters/src/StructuredData/AbstractListData.php on line 15 #0 /Users/wim.leers/core/vendor/consolidation/output-formatters/src/StructuredData/AbstractListData.php(15): ArrayObject->__construct(false)
> #1 /Users/wim.leers/core/vendor/consolidation/output-formatters/src/StructuredData/UnstructuredListData.php(21): Consolidation\OutputFormatters\StructuredData\AbstractListData->__construct(false)
> #2 /Users/wim.leers/core/vendor/drush/drush/src/Commands/core/UpdateDBCommands.php(137): Consolidation\OutputFormatters\StructuredData\UnstructuredListData->__construct(false)
> #3 [internal function]: Drush\Commands\core\UpdateDBCommands->process('10', Array)
> #4 /Users/wim.leers/core/vendor/consolidation/annotated-command/src/CommandProcessor.php(276): call_user_func_array(Array, Array)
> #5 /Users/wim.leers/core/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
> #6 /Users/wim.leers/core/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
> #7 /Users/wim.leers/core/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(391): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
> #8 /Users/wim.leers/core/vendor/symfony/console/Command/Command.php(312): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #9 /Users/wim.leers/core/vendor/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #10 /Users/wim.leers/core/vendor/symfony/console/Application.php(314): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #11 /Users/wim.leers/core/vendor/symfony/console/Application.php(168): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #12 /Users/wim.leers/core/vendor/drush/drush/src/Runtime/Runtime.php(110): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
> #13 /Users/wim.leers/core/vendor/drush/drush/src/Runtime/Runtime.php(40): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
> #14 /Users/wim.leers/core/vendor/drush/drush/drush.php(139): Drush\Runtime\Runtime->run(Array)
> #15 /Users/wim.leers/core/vendor/drush/drush/drush(4): require('/Users/wim.leer...')
> #16 /Users/wim.leers/core/vendor/bin/drush(119): include('/Users/wim.leer...')
> #17 {main}
>  [warning] Drush command terminated abnormally.

… happens 100% of the time when running drush updatedb --yes against core commit fca42b9d59da506d9c9aac1c3be018dc04205d67.

Root cause appears to be:

    public function process(string $batch_id, $options = ['format' => 'json']): UnstructuredListData
    {
        $result = drush_batch_command($batch_id);
        return new UnstructuredListData($result);
    }

… where $result is FALSE, UnstructuredListData's processing chokes on it.

Perhaps because drush_batch_command() is itself using the batch system? 🤔

For now, ignoring this, because no precise data for a few commits is not a big deal. The trend should still be clear, and my script SHOULD be robust enough to handle this.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

To debug that, I added this to \Consolidation\SiteProcess\ProcessBase::getOutputAsJson():

$msg .= "\n" . print_r(debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS, 20), TRUE). "\n";

(just before throwing the exception).

Wim Leers opened merge request !6

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Active » Needs review
StatusFileSize
new Screenshot 2023-10-05 at 6.37.28 PM.png2.49 MB

This is what that should look like:

🤓

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented

You are adding a gitlab CI yml here :) It would also be important to port the regular test to gitlab CI. I think it would be easier to rely on the standard templates like I did in upgrade_rector recently for easier contributor involvement, rather than making it custom. For the custom job I agree custom config is needed :) Do you think the GitLab CI config added here could still accommodate porting the drupalci tests over without much rework?

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented

FYI this is the matrix I have there, although with Drupal 9 going EOL in a few weeks, we may not need to test that anymore? https://git.drupalcode.org/project/upgrade_rector/-/blob/2.x/.gitlab-ci....

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Porting the existing test suite to GitLab CI is an independent task/concern. They can keep running just fine the way they are for now. I'm sure I'll get to that some time in the next few months though!

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Needs review » Needs work

GitLab CI adoption by this project now is a fact: #3396042: Adopt GitLab CI.

I'll need to modify this MR.

Wim Leers credited diqidoq.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Per @diqidoq in Slack:

#3 seem to be gone and faded away like in miracles after updating Drupal core to 10.2 beta ...

— https://drupal.slack.com/archives/C1BMUQ9U6/p1701264422604679

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Picking this back up where I left it off. I want to make the data much easier to consume too, and thanks to #3384688: Automatically publish a GitLab pages site (and https://timvink.github.io/mkdocs-charts-plugin/usage/, maybe https://squidfunk.github.io/mkdocs-material/reference/diagrams/ and if really necessary https://docs.gitlab.com/ee/user/project/pages/getting_started/pages_from...) that should become feasible 🤓

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Issue tags: +gitlabci-contrib
Related issues: +#3384688: Automatically publish a GitLab pages site
wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Related issues: +#3396042: Adopt GitLab CI

Merged in upstream, which means this now no longer conflicts with #3396042: Adopt GitLab CI, but extends it :)

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Well 🤣

…
xhprof
xhprof support => enabled
…
wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented 
 [success] Successfully enabled: config_inspector
real	0m 3.97s
user	0m 2.41s

4 seconds for installing a single module. Yeah. That's gonna make things mind-bogglingly slow 😬

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Now down to ~8 seconds/commit. With a 1-hour time limit and a few minutes for setup, that should mean roughly 55 minutes gives us 55*60/8 = ~410 days that can be analyzed. That's good enough for now! (It took 49 minutes to process the 355 days of 2023, including all overhead.)

And … the countless "permission denied" problems are now also gone! 👍

That means that next week, I'll be able to continue this for real 😄

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented

Just looked at the latest output, super amazed that we are already at +50%. This is already very cool to see this output. Looking forward to see this in graphical form.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

There are a lot of type: labels in Drupal core 🤣

Note that this does not mean 53% of all core config is validatable. Only 53% of core config property paths is validatable. We'll eventually need to clarify that, but right now the key goal is to get some visualization going on…

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

It is MUCH faster now! 14 minutes for analyzing all of 2023.

Next up: using R to generate charts.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Needs work » Reviewed & tested by the community
StatusFileSize
new validatability.png429.49 KB

This now is generated automatically:

🥳

  • Wim Leers committed 603a7fbf on 2.1.x 
    Issue #3391990 by Wim Leers, diqidoq: Automated report on core config...
wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Reviewed & tested by the community » Fixed

Wim Leers opened merge request !11

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Fixed » Needs work

A specific job called pages in the configuration file makes GitLab aware that you’re deploying a GitLab Pages website.

— https://docs.gitlab.com/ee/user/project/pages/getting_started_part_one.h...

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Assigned: wim leers » Unassigned
Status: Needs work » Reviewed & tested by the community
StatusFileSize
new validatability.png218.07 KB

Now starting 1 month earlier (Dec 1, 2022) and actually publishing to https://project.pages.drupalcode.org/config_inspector/ 👍

  • Wim Leers committed 683bba26 on 2.1.x 
    Issue #3391990 by Wim Leers, diqidoq: Automated report on core config...
wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented
Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.