Problem/Motivation
From time to time scripts operating in .gitlab-ci need to make changes to JSON, YAML, and XML files based on current
Steps to reproduce
https://git.drupalcode.org/project/quasar_build/-/blob/7fc317ea6df9072e5...
https://git.drupalcode.org/project/gitlab_templates/-/commit/61563ab8de2...
(this could be simplified to a single command cat input.xml| xq-python -x 'del(.phpunit.source)' > output.xml)
Proposed resolution
add jq(provides jq) and kislyuk/yq(provides yq and xq-python, depends upon jq) to base images to allow manipulating files as part of build process.
Remaining tasks
User interface changes
API changes
jq, yq and xq-python will be available on the images.
Data model changes
Issue fork drupalci_environments-3487621
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
cmlaraClarifying: this originally was intended to ask for kislyuk/yq
There is mikefarah/yq however it notes it supports a smaller syntax than kislyuk/yq
ksislyuk/yq works by converting XML to JSON allowing jq to use its natural language support.
Comment #3
andypostidea looks good, personally I think
jqis the most common and it should be enoughwho can review dependencies and security consequences?
Comment #4
cmlaraThe yq package adds support for XML which is likley just as important given we now have been modifying the phpunit config to support differences in core versions. As noted above yq uses jq for the parsing engine and is primarily a converter.
A good question, to my knowledge this has not been requested before in this queue and there is no documented review policy. Given the lack of policy it would likley need to be done by a queue admin (yourself?) as none of us can speak to an undefined standard.
Comment #5
andypostI mean adding
yqrequires to add python interpreter with few dependencies, IMO it's useless for CI imageComment #6
cmlaraThe python interpreter is already on the images and is used by gitlab_templates to provision the mkdocs stage (in other words: we are not likely to accept the removal of the python interpreter anytime soon) .
Fair it may add more dependencies into python.
If there is a binary XML->JSON->XML and YAML->JSON->YAML converter that could be a reasonable alternative to the yq package.
https://git.drupalcode.org/project/gitlab_templates/-/blob/main/scripts/... is an example of where we are creating a 51 line script for what could likely be done more efficiently and reliably a 1 line yq command. I've had other discussions in Slack where we are making the jobs harder to maintain, more fragile (dependent upon another HTTP GET, and can be broken by mismatch in versions downloaded vs version of the gitlab template) (these are incidents that have occurred in the past) by using external PHP Scripts.
Downloading extra scripts is somewhat of an anti-pattern in GitLab, and the tricks that gitlab_templates team currently use do not currently exist for GitLab Components meaning the ability to inline commands can become even more important long term.
The script alone proves that we need to make changes to XML files as part of the CI process which renders the argument 'useless' as moot and only leaves 'is it worth the cost vs savings'.
Comment #7
andypostLooks it's only few packages
python3-argcomplete python3-toml python3-xmltodict python3-yamlComment #9
andypostAs
jqis instalkled as dependency here's MRComment #11
andypostmerged to dev so it can be tested https://git.drupalcode.org/project/drupalci_environments/-/jobs/4611960
Comment #12
cmlaraI generally recommend to be explicit.
For all we know next year yq could change and not use the jq package (highly unlikely however if they do we loose jq and don’t realize it)
Comment #14
andypost@cmlara thanks updated commit to both
yq jqhttps://git.drupalcode.org/project/drupalci_environments/-/commit/5d1853...PS: this week new PHP releases will out, meantime I did update prod images with fresher PECL
Comment #19
andypostadded both to 8.1-8.3 https://git.drupalcode.org/project/drupalci_environments/-/jobs/5283944
Comment #20
andypostno regressions