Working towards providing what the client expects, https://github.com/php-tuf/composer-integration/wiki

Comments

drumm created an issue. See original summary.

  • drumm committed 56e776e1 on 7.x-3.x 
    Issue #3341719: Add tuf_ready_… directory
wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

🤓 Interesting:

-  drush_shell_exec('rsync -v -r --delay-updates %s %s', $workspace . '/' . $vendor, $destination);
+  drush_shell_exec('rsync -v -r --delay-updates %s %s', $workspace . '/' . $vendor, $destination . '/tuf_ready_' . REQUEST_TIME . '_' . getmypid());

… but I'm not sure I understand what this does? 😅

drumm’s picture

drumm
he/him
Location NY, US
commented
Issue summary: View changes

That’s how https://gitlab.com/rugged/rugged/-/issues/120 landed. A directory named tuf_ready_* signals to rugged that it can batch process the targets within. rsync --delay-updates allows us to populate that directory atomically.

  • drumm committed db990620 on 7.x-3.x 
    Issue #3341719: Scaffold files, then move; rsync creates the directory...
wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

👍 Thanks for the context!

  • drumm committed aaceff04 on 7.x-3.x 
    Issue #3341719: Set permissions so rugged can remove incoming targets

  • drumm committed 03401dd4 on 7.x-3.x 
    Issue #3341719: Send project’s composer.json
drumm’s picture

drumm
he/him
Location NY, US
commented

We can now send targets as if they were at:

drupal/google_analytics_counter/3.0.0.0-alpha21
p2/drupal/google_analytics_counter.json

Now we need the root packages.json. That needs to be a file on the filesystem first: #3342516: Write out packages.json instead of serving it dynamically

  • drumm committed 33004015 on 7.x-3.x 
    Issue #3341719: Send packages.json
drumm’s picture

drumm
he/him
Location NY, US
commented
Status: Active » Needs review

The basics here are done. We now have:
https://drupal:drupal@packages.staging.devdrupal.org/metadata/1.root.json
https://drupal:drupal@packages.staging.devdrupal.org/metadata/targets.json
https://drupal:drupal@packages.staging.devdrupal.org/metadata/snapshot.json
https://drupal:drupal@packages.staging.devdrupal.org/metadata/timestamp....

We'll have a few followups:

  • More-efficient backfilling - ability to send a whole project at a time, ability to skip redoing packages.json.
  • TUF is not hosting the targets. Can we remove them from rugged’s files so we aren’t keeping a second copy of everything?
  • Caching - watch for signing to complete, purge CDN cache for them.
phenaproxima’s picture

phenaproxima
he/him
Primary language English
Location Massachusetts
commented

TUF is not hosting the targets. Can we remove them from rugged’s files so we aren’t keeping a second copy of everything?

You should be able to remove the targets. As long as the hashes have been generated for the zip files that Composer will actually download -- that is, whatever is in the dist for any given version of any given package -- I should be able to safely download those zip files from anywhere.

drumm’s picture

drumm
he/him
Location NY, US
commented

We’re not routing /targets to be downloadable, but they are still on disk for rugged. Rugged might not like its copies of the targets files being taken away or set up in a way that they are never saved.

  • drumm committed 16e6e240 on 7.x-3.x 
    Issue #3341719: Allow sending dev releases to rugged

  • drumm committed eeff7516 on 7.x-3.x 
    Issue #3341719: Allow skipping packages.json

  • drumm committed b0adc64f on 7.x-3.x 
    Issue #3341719: Allow sending all versions of a project to rugged at...
drumm’s picture

drumm
he/him
Location NY, US
commented
Status: Needs review » Fixed

I believe just about everything is done for this issue, we can tackle getting this to production in the parent issue and more child issues from that.

The one loose end is

TUF is not hosting the targets. Can we remove them from rugged’s files so we aren’t keeping a second copy of everything?

My initial testing says we can. So, in a followup, we can either:

  • Live with the extra space used
  • Not have rugged save targets
  • Empty targets on cron
  • Or set up some filesystem trick so files placed in targets go nowhere
wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Does that mean #3343490: Deploy rugged for TUF signing to production will happen next? 🤞

drumm’s picture

drumm
he/him
Location NY, US
commented

It will be a few issues to round up, since we took a few shortcuts to get to where we are. #3349408: Decide on & implement targets management for Rugged instance covers my last comment here, since it isn’t really related to what the drush command to load the targets for signing does.

  • drumm committed b0adc64f on reg-prot 
    Issue #3341719: Allow sending all versions of a project to rugged at...

  • drumm committed eeff7516 on reg-prot 
    Issue #3341719: Allow skipping packages.json

  • drumm committed 16e6e240 on reg-prot 
    Issue #3341719: Allow sending dev releases to rugged

  • drumm committed 33004015 on reg-prot 
    Issue #3341719: Send packages.json

  • drumm committed 03401dd4 on reg-prot 
    Issue #3341719: Send project’s composer.json

  • drumm committed aaceff04 on reg-prot 
    Issue #3341719: Set permissions so rugged can remove incoming targets

  • drumm committed db990620 on reg-prot 
    Issue #3341719: Scaffold files, then move; rsync creates the directory...

  • drumm committed 56e776e1 on reg-prot 
    Issue #3341719: Add tuf_ready_… directory

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.