[META] Replace update_manager with a more powerful solution

I'd suggest that there be a console component to this process, so it doesn't have the same fate as the update manager itself.

That is: People who don't use the web-based UI can (and should, and will want to) use the CLI part. Then it gets more attention and maintenance.

I'd also suggest that the CLI part be developed first, so we can discover the real sticky parts of the problem without having to re-work forms all day long.

a) the site’s composer.json (possibly reduced) with the addition of the name/version of the new module.

Still a bit of a problem to guarantee the site's composer.json, since composer can build a codebase from an arbitrary composer.json file. Either this process will have to demand that composer.json files be located in certain places, or else have a way to specify the file used (that is, something like a form where you paste in your composer.json file). If we have a CLI this isn't such a big deal and we can account for it, because COMPOSER=../../special/composer.json ./bin/console update.

It's possible that we could make a composer plugin that stores the composer build info in a known location. This would get back to @Xano's idea from #2494073: Prevent modules which have unmet Composer dependencies from being installed (it's a looooong comment thread) of keeping that information during composer install phase. (@Mixologic solved that issue in scope by suggesting we have a build_dependencies key in info.yml instead. Much easier set to solve for.)

<cents amount="2">
I agree with comment 3 - doing it at API layer with console commands first.
UI later.
I also think we should try and get funding for this initiative like we did for d8 accelerate.
The issue queue is full of people for whom composer is causing a lot of pain and that would rather avoid it. How much pain is it causing? Enough for you to fund a team?
Because the issue is that most people are not willing to work on something that they won't use.
Providing funding will change that view, but also would allow us to contact experts like Paragonie (their blog post is in the issue summary) who already have solved this for airship in a secure manner
</cents>

So once the changed are downloaded and applied, Can there be an option to commit to a new branch with a timestamp or some such? There needs to be a way to get this stuff into Git (or some other VCS). Even if the server doesn't have push access (which it shouldn't), someone can run a diff, download it, and then push it from his/her local.

Wow, I've been working on this for over a year on my own, and feeling like I was the only person who cared. Dries calling out automatic updates as a priority is consequently pretty exciting for me :)

I also love what I'm seeing in the initial implementation suggestions here. In particular, yes yes yes it should be a logically separate, standalone application -- shipped with Drupal, sure, but loading no scripts used by core or the user-facing site. This is the approach I have taken, for reasons including

• The update of the site can go completely south, but you can at least be sure you'll still have some kind of administrative UI on that server that you can go to.
• You don't have to think about any potential complexities from updating components that are currently being used to install the update.
• Doing automatic updates of php web applications is a complex problem, but it is substantially the same problem for all application frameworks. If we bake it into Drupal 8 core, you could still find yourself at square 1 with Drupal 7 even, and why leave that million sites behind if we don't have to? Having a separate application whose sole purpose is securely applying updates to other applications would allow the hard problems to be solved once, by the best people, with many php web app frameworks immediately benefiting.

Its also worth noting that php interpreters at least back to 5.4 will, regardless of configuration, serve entire web applications out of one .phar file and one .php file that simply includes the .phar. So you can manage a modern, symfony-like app in a neat little package. This is probably the appropriate distribution format.

I also wholeheartedly agree on the order of priority. Number 1 is done, if you allow for there not being a UI :p. The code is here. The CI environment is among many other things performing a complete Drupal 7 core update over ftp. In time, we can think about adding cool additional features like a composer UI, or update rollback support -- also things which would be beneficial for Drupal and beyond too.

I'd of course love to see this ultimately solved through direct contributions to the work I've already started, but even if we wanted to start the base app over, some potentially useful components that are already developed and tested are:

1. Since performing core updates may require production website downtime, I think performance should be a priority. To help with that I've spent quite a bit of time on a batch framework, conceptually similar to what you have in Drupal core where a bunch of little tasks are getting executed, driven by repeated ajax calls from the client. Except, mine is designed to not get befuddled, fouled up, or otherwise hosed if the browser is sending several ajax requests simultaneously and several little tasks are running simultaneously. Since the main job an updater would do is rewrite lots of files, I think we can get speedup from that kind of parallelization, especially when the underlying file access is ftp etc.
2. Since webservers must have read access to the main web application's files but may not have write access, I developed a unique filesystem abstraction that differentiates between reads and writes, performing reads through the standard mounted filesystem as this will always be fastest, but can perform writes through ftp etc. There are many filesystem abstraction libraries for php, but they all direct both reads and writes through the same filesystem access mechanism.
   This one is optimized for this task.
3. An update package specification designed for packaging and distributing updates, and a processor for packages in this format. This is advantageous over, for example, downloading the entire release .tgz just to get the drupalgeddon patch, because it reduces the number of moving parts. The updater can know it doesn't have to rewrite the entire Drupal 8 core tree, and instead only makes the modifications and deletions necessary to transform the installed version to the updated version.

Very happy to see Dries recognizing that it's time for Drupal to take this seriously.

Regarding Paragon IE's work on securing automatic updates: I believe their most important contribution is sodium-compat. It is ostensibly the first native-php library that would allow you to perform and verify code signing, but Paragon themselves believes it should be validated by security experts besides themselves. I was contacted a few weeks ago by the Mozilla Foundation, asking me whether they should fund such a security audit. I told them that imho yeah of course they should, and yeah of course I'll use sodium-compat in my work discussed in #7. But what Mozilla is really looking for is a commitment from Drupal's decision makers that Drupal would incorporate sodium-compat into automatic updates as well. I can't offer that assurance, although I see absolutely zero reasons why we would not want to use a properly audited and highly backwards compatible code signature verification library.

I realize we still have decisions to make and planning to do. But with this currently getting so much attention in Drupal, and Mozilla having potential funding now, who can help me to get an email from the right person sent over to Mozilla?

@mbaynton as you've spent a lot of time investigating this, would be interested in your opinion of the feasibility of this concept, basically keeping things as close to the current state, then look into improvements like it's own separate UI -- https://www.drupal.org/node/2908394#comment-12274918

This "remote composer resolution" service could be reused by non Drupal projects and even potentially for CLI in cases where the network latency is extremely poor -- https://www.drupal.org/node/2845379#comment-12068852

@8thom, I've not put much thought into the problems of #2908394: Use Composer to build sites without forcing users to learn Composer, actually.

I wish we there there already, but I think the issue summary says it best

Figuring out how to structure the updater, and how to update core will be a big task, so I'd like to avoid discussing the Composer part for now.

Of course, it could be that's shortsighted. If anyone can make a concise argument about that, it could be quite constructive. That's just how I see it right now.

100% agree with #3 / @Mile23.

Also, this sounds extremely sensible:

I'd also suggest that the CLI part be developed first, so we can discover the real sticky parts of the problem without having to re-work forms all day long.

I mean, like, the Drush pm-* commands that are going away with Drush 9 (and that are on shaky ground for later versions of Drupal 8.x) -- if update manager changes, it might make it easier to keep/maintain the Drush pm-* commands that so many site builders know and love? I know this isn't a Drush dev thread, but it's still relevant, since we're talking about CLI things for site builders (AFAIK). (Related Drush PR: #2787)

Also related to this issue, maybe? (I know I can add to the "related issues" list, but I'm not positive, so I'm going to leave it to y'all this time.)
#2242947: Integrate Symfony Console component to natively support command line operations

(And, related conversation over here -- yes the issue is closed, but a good bit of the conversation is relevant: #2906637: [META] Drush and core compatibility is fragile)

There's some question about how much formal Composer-based infrastructure to introduce in #2830880-47: Warn site admins when composer dev dependencies are installed inside of docroot.

Should we introduce a child issue here to define some Composer-oriented services to core, such as discovering the project root?

@alisonjo2786, thanks for referencing the problems Drush is running into with Composer. I read the PR description you linked to and the first few comments. Interesting reading. As I understand it, the problem they're running into is more related to just keeping Drush from experiencing its own errors, and less related to adding new automatic composer integrations as part of module installations. Drush installs an autoloader built with its own composer.json, and then, because its raison d'être is to do complex things within Drupal, it needs to bootstrap Drupal, which means loading an additional autoloader built by a different (Drupal's) composer.json. Version conflicts ensue.

This is something it's good for me to be thinking about for sure. I'm all advocating for an implementation of this "more powerful tool" that, like Drush, is a separate project with its own autoloader. It _mostly_ runs independently from and without bootstrapping Drupal, but the way I'm currently doing it there's one point in the Drupal / Curator handoff where both application kernels, or at least their dependencies, both end up being simultaneously loaded in the interpreter. Basically I can't do that because of the risk of encountering this same problem.

@mbaynton Cool, welcome, thanks for reviewing and considering!

I don't know but I feel like the other genre of issues is the same dependency management stuff that has been problematic for update manger -- like, I think that some of the pm-* commands use the Drupal core "update manager" system, right? (but I'm not positive!)

Still looking for answers on #12, about #2494073: Prevent modules which have unmet Composer dependencies from being installed and #2830880: Warn site admins when composer dev dependencies are installed inside of docroot

We should move forward on those, regardless of what super-duper dependency machine is being built in the cloud. Then we can modify or remove them as needed.

Agree, these short term solutions would be good to put in place.

If the reference to "super-duper dependency machine" is this project -- https://github.com/thom8/faas-composer

There's 34 lines of code required to make this work...

As a site builder with limited coding/console skills and/or drive, the proposal to let drupal.org do Composer-As-A-Service sounds really clever to me.

Am I correct in thinking that this would allow Drupal to once again be easy to install and update on shared hosting?

@groovedork out of curiosity, are you talking about Drupal 7 or 8, and what makes it not easy?

@mbaynton I've outlined just a few of the issues in this post:
https://www.drupal.org/forum/support/post-installation/2017-03-24/how-do...

I work with a lot of non-profits who use Drupal on shared hosting, often building the sites themselves. They don't have SSH skills or access and, well, never had to. One of Drupal's core features was that you could build amazing things through WYSIWYG. But that idea seems to have quietly died? #sad

@groovedork, thanks for sharing. I think it's fair to summarize the problems you encountered fell into two categories: the tooling provided to do an update was not simple and the update from an earlier version to 8.4 was disruptive. The scope of this issue is improving the update tooling so there's a thing just built into the site, and you don't need Acquia Dev Desktop or Drush or Composer. But no tooling will provide a non-disruptive update experience if the update path itself requires as much manual intervention / fixing as people are reporting with recent 8.x minor releases post-update, so fixing the tooling issue alone would basically just give your site owners a way to shoot themselves in the foot (look! I can update my site! Now it's borked!)

It's these reports that I believe really legitimize the other project I'm trying to get off the ground, iglue, which basically aims to allow you to stick with the Drupal 8 version you started with while keeping your site secure.

But no tooling will provide a non-disruptive update experience if the update path itself requires as much manual intervention / fixing as people are reporting with recent 8.x minor releases post-update

@mbaynton was discussing this the other day, are you or anyone aware of an initiative to provide an LTS Drupal 8 release?

It could sit outside the normal 6 monthly release cycle and maybe catch up every 4th release (2yrs), whilst backporting any security issues.

Otherwise, we just need to get better at not introducing any BC breaks in minor releases..

This work is now in #3319030: Drupal Core Roadmap for Package Manager and Update Manager.

Saving credits for what began with a Hard Problems discussion that was one of the first steps toward the Autoupdates Initiative.