Convert .info file rewriting in packaging plugins to deal with D8 .info.yml files

Without proper version in the .info.yml file, there is no version information the translation import/update feature can work with. Without this issue up to date translations will not be imported. See #2030537: Translations not downloaded when adding a new language.

If anyone is able to start of this, start with http://drupalcode.org/project/drupalorg.git/blob/refs/heads/7.x-3.x:/dru... and be sure to take a look through the rest of that file.

do all the stuff we used to do to .info files to .yml files but with proper syntax (e.g. # instead of ; for comments, etc).

needs to be expanded to an example of what needs to be added.

What's the best way to read/write YML in D7?

#2229641: D8 usage statistics not shown on d.o closed as duplicate.

See https://drupal.org/node/1935708 for the change notice explaining the differences between D7 and D8 syntax-wise.

Drupal 8 is using Symfony's YAML parser https://github.com/symfony/Yaml to read these; probably makes sense for D.o to do the same.

It'd be great if this could be prioritized soon. It's causing metrics (including board metrics) around Drupal 8 usage, contribution, etc. to be useless.

Are there other tag(s) we should be using for being a general 8.x blocker, or would that mess up the core workflow?

Apologies, I provided invalid advice on the topic of this issue to @drumm some time ago.

I was misguided, because literally all YAML implementations for PHP do not follow the YAML specification, which clearly states:

JSON's RFC4627 requires that mappings keys merely “SHOULD” be unique, while YAML insists they “MUST” be.

The outdated YAML 1.1 spec contained a crystal clear note on how the error of duplicate keys is to be handled by parsers, which seems to have vanished from the latest 1.2 spec somehow (only leaving the requirement):

It is an error for two equal keys to appear in the same mapping node. In such a case the YAML processor may continue, ignoring the second key: value pair and issuing an appropriate warning. This strategy preserves a consistent information model for one-pass and random access applications.

A quick'n'dirty test with Symfony YAML (1.2) as well as the PECL YAML (1.1) extension reveals that all of them are broken:

foo: first
foo: second

$yaml = file_get_contents(__DIR__ . '/debug.yml');
var_dump(
  \Symfony\Component\Yaml\Yaml::parse($yaml),
  yaml_parse($yaml)
);

…yields:

array(1) {
  'foo' =>
  string(6) "second"
}
array(1) {
  'foo' =>
  string(6) "second"
}

Python's PyYAML ran into the same trap as PHP parsers, equally unresolved — most likely, because both parser implementations are sequentially processing the document + simply eat up later values in a mapping.

drupal.org should not rely on this misbehavior. It's a clear bug in all of the libraries, which violates the YAML specification, and the bug can be fixed at any time without prior notice.

My original advice was to simply append the necessary values to the .info.yml file, in the same way we do for the .info files already. Please disregard that.

This correction/clarification makes the situation substantially harder, because:

1. We don't know whether the .info.yml file contains the to-be-added properties already.
2. We do NOT want to kill any comments in the YAML document by parsing + dumping it.

(2) entirely eliminates every PHP YAML parser from the available choices, because they do not retain comments (because the data is parsed into native PHP data types and PHP has no "comment" data type).

Since we are only dealing with top-level keys in .info.yml files, the best recommendation I can give at this point is to operate on the raw (unparsed) document itself, like so:

$filename = 'some.info.yml';
$doc = file_get_contents($filename);
// Comment out the keys we're going to add.
$doc = preg_replace('/^((?:project|timestamp|...)\s*:.*)$/m', '#$1', $doc);
// Add drupal.org packaging keys.
$doc .= "
project: $project
timestamp: $now
";
file_put_contents($filename);

"Are there other tag(s) we should be using for being a general 8.x blocker, or would that mess up the core workflow?"

Hm. Not aware of a tag. We track release-blockers in the Drupal core queue via "critical" tasks/bugs. I cannot seem to search for https://drupal.org/project/issues/drupal?text=drupal.org+usage without a WSOD though, so not sure if one already exists. :( Created one here #2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1 just in case.

Instead of removing the duplicate entries, let's comment them out and add a quick note about what happened. That transparency will point anyone looking in the right direction, instead of letting them think some packaging gremlin is stealing random lines.

In addition to getting sun's code integrated into http://drupalcode.org/project/drupalorg.git/blob/refs/heads/7.x-3.x:/dru..., it would be good to have some before/after samples uploaded here, to get on the way to having test coverage for this change.

There's already an extensive before/after example in the change notice at https://drupal.org/node/1935708 (though it doesn't cover sun's edge case).

The D8 change notice for .info.yml files is mostly irrelevant for this issue here. All it does is to explain the basics of how the YAML serialization format works, which I assume to be given knowledge for the participants of this issue here.

Good point with regard to commenting out instead of deleting. I've updated my comment to prevent confusion.

I'm not able to help with the question of before/after examples, because I'm not familiar with the d.o packaging code and which exact keys it adds under which circumstances. That code should essentially add the same properties to .info.yml files that it adds to .info files already — with the exception that possibly preexisting keys need to be commented out first.

But aside from that, .info and YAML parsers are fairly similar, and given that we're only dealing with primitive string and integer values, no further data value conversions should be required. The only diff between .info and YAML is the syntax for mappings:

-project = drupal
-timestamp = 123456789
+project: drupal
+timestamp: 123456789

Unlike .info, the integer will be parsed into an actual integer. Strings technically need to be wrapped into single-quotes or double-quotes, but as long as project shortnames only support alphanumeric + underscore characters, that's not required (although it might be wise to wrap them nevertheless).

But yeah, these aspects should be obvious and given knowledge for participants here. — If someone is not familiar with the YAML standard, then I don't think this is a good issue to work on.

Yep, the before/after I meant was foo.info.yml as checked into Git vs. as packaged in tarballs.

FWIW, filed a PR to fix at least Symfony's YAML implementation:

https://github.com/symfony/symfony/pull/10902

FWIW, this issue also holding up a multilingual major/critical in core: #2030537: Translations not downloaded when adding a new language Any ETA?

I have this working on dev. http://cgit.drupalcode.org/page_manager/tree/page_manager.info.yml becomes:

type: module
name: Page Manager
description: 'Provides a way to place blocks on a custom page.'
package: Layout
# version: VERSION
# core: 8.x
dependencies:
  - block

# Information added by drumm2 drupal dev packaging script on 2014-06-20
version: '8.x-1.x-dev'
core: '8.x'
project: 'page_manager'
datestamp: 1403230061

Does that look okay?

Yes, that looks correct.

Deploying & re-building page_manager's dev release.

Now deployed. I repackaged page_manager and it looks as expected.

Removing the ftp from the download URL, http://drupal.org/files/projects/page_manager-8.x-1.x-dev.tar.gz, will shortcut a syncing step if you want to try it in the next 30 minutes. Otherwise, the regular download is good, https://www.drupal.org/node/2272257.

Great news! Thanks, drumm. So the usage stats should reflect this change on Monday or so?

Probably not. People would need to run the tarballs of newly-packaged releases and have them run update status by the cutoff time, I don't know the specific time offhand.

I could probably trigger re-packaging of all 8.x dev releases with ~30min of work. Same for tagged releases, but I'm not sure how much the md5s changing would affect drush, etc.

I tested this manually for about 2 months and stats have not updated, but I'm not sure the datestamp is a requirement. see #2229641: D8 usage statistics not shown on d.o

Ok, so basically the only 8.x usage stats that'll count is:

1) Tarball downloads (not git clones — we'd need a 8.x version of https://www.drupal.org/project/git_deploy for that)
2) That were packaged from today, onwards (because they'll then have the necessary "version/core/project/datestamp" keys) — meaning that module devs should make new tags / do additional commits to 8.x branches in order to trigger the packaging script?
3) Running on a site with Update Status enabled (and presumably with cron running)

So while we could/should see a somewhat inflated count on Monday, the clearest picture will only emerge over time as these modules get updated and newer releases are tagged. Is that the gist?

No, I don't think we'll see much today. The change to packaging went in late last week, and I don't think there will be many, if any, sites that are running the updated tarballs in time.

I think there is likely more work to be done at #2229641: D8 usage statistics not shown on d.o too.