Problem/Motivation

From time to time scripts operating in .gitlab-ci need to make changes to JSON, YAML, and XML files based on current

Steps to reproduce

https://git.drupalcode.org/project/quasar_build/-/blob/7fc317ea6df9072e5...
https://git.drupalcode.org/project/gitlab_templates/-/commit/61563ab8de2...
(this could be simplified to a single command cat input.xml| xq-python -x 'del(.phpunit.source)' > output.xml)

Proposed resolution

add jq(provides jq) and kislyuk/yq(provides yq and xq-python, depends upon jq) to base images to allow manipulating files as part of build process.

Remaining tasks

User interface changes

API changes

jq, yq and xq-python will be available on the images.

Data model changes

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

cmlara created an issue. See original summary.

cmlara’s picture

Issue summary: View changes

Clarifying: this originally was intended to ask for kislyuk/yq

There is mikefarah/yq however it notes it supports a smaller syntax than kislyuk/yq

ksislyuk/yq works by converting XML to JSON allowing jq to use its natural language support.

andypost’s picture

idea looks good, personally I think jq is the most common and it should be enough

who can review dependencies and security consequences?

cmlara’s picture

I think jq is the most common and it should be enough

The yq package adds support for XML which is likley just as important given we now have been modifying the phpunit config to support differences in core versions. As noted above yq uses jq for the parsing engine and is primarily a converter.

who can review dependencies and security consequences

A good question, to my knowledge this has not been requested before in this queue and there is no documented review policy. Given the lack of policy it would likley need to be done by a queue admin (yourself?) as none of us can speak to an undefined standard.

andypost’s picture

I mean adding yq requires to add python interpreter with few dependencies, IMO it's useless for CI image

cmlara’s picture

I mean adding yq requires to add python interpreter

The python interpreter is already on the images and is used by gitlab_templates to provision the mkdocs stage (in other words: we are not likely to accept the removal of the python interpreter anytime soon) .

with few dependencies,

Fair it may add more dependencies into python.

If there is a binary XML->JSON->XML and YAML->JSON->YAML converter that could be a reasonable alternative to the yq package.

IMO it's useless for CI image

https://git.drupalcode.org/project/gitlab_templates/-/blob/main/scripts/... is an example of where we are creating a 51 line script for what could likely be done more efficiently and reliably a 1 line yq command. I've had other discussions in Slack where we are making the jobs harder to maintain, more fragile (dependent upon another HTTP GET, and can be broken by mismatch in versions downloaded vs version of the gitlab template) (these are incidents that have occurred in the past) by using external PHP Scripts.

Downloading extra scripts is somewhat of an anti-pattern in GitLab, and the tricks that gitlab_templates team currently use do not currently exist for GitLab Components meaning the ability to inline commands can become even more important long term.

The script alone proves that we need to make changes to XML files as part of the CI process which renders the argument 'useless' as moot and only leaves 'is it worth the cost vs savings'.

andypost’s picture

Looks it's only few packages python3-argcomplete python3-toml python3-xmltodict python3-yaml

root@821a35c6e52f:/var/www/html# apt install jq yq
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libjq1 python3-argcomplete python3-toml python3-xmltodict python3-yaml
The following NEW packages will be installed:
  jq libjq1 python3-argcomplete python3-toml python3-xmltodict python3-yaml yq
0 upgraded, 7 newly installed, 0 to remove and 30 not upgraded.
Need to get 409 kB of archives.
After this operation, 1357 kB of additional disk space will be used.
Do you want to continue? [Y/n] ^C
root@821a35c6e52f:/var/www/html# apt install jq   
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libjq1
The following NEW packages will be installed:
  jq libjq1
0 upgraded, 2 newly installed, 0 to remove and 30 not upgraded.
Need to get 206 kB of archives.
After this operation, 481 kB of additional disk space will be used.
Do you want to continue? [Y/n] 

andypost’s picture

Status: Active » Needs review

As jq is instalkled as dependency here's MR

  • andypost committed eda60e51 on dev
    Issue #3487621 by andypost, cmlara: Add jq and yq packages to images
    
andypost’s picture

cmlara’s picture

As jq is instalkled as dependency here's MR

I generally recommend to be explicit.

For all we know next year yq could change and not use the jq package (highly unlikely however if they do we loose jq and don’t realize it)

  • andypost committed 5d185374 on dev
    Issue #3487621 by andypost, cmlara: Add jq and yq packages to images
    
andypost’s picture

@cmlara thanks updated commit to both yq jq https://git.drupalcode.org/project/drupalci_environments/-/commit/5d1853...

PS: this week new PHP releases will out, meantime I did update prod images with fresher PECL

  • andypost committed 79218893 on production
    Issue #3487621 by andypost, cmlara: Add jq and yq packages to images
    
    
    (...

  • andypost committed d645787a on dev
    Issue #3487621 by andypost, cmlara: Add jq and yq packages to PHP images
    

  • andypost committed 1d436071 on dev
    Issue #3487621 by andypost, cmlara: Add jq and yq packages to PHP images
    

  • andypost committed c3c6ae5e on production
    Issue #3487621 by andypost, cmlara: Add jq and yq packages to PHP images...
andypost’s picture

Status: Needs review » Reviewed & tested by the community
andypost’s picture

Status: Reviewed & tested by the community » Fixed

no regressions

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.