[meta] Make Drupal compatible with persistent app servers like ReactPHP, PHP-PM, PHPFastCGI, FrankenPHP, Swoole

ReactPHP take a very different approach to PHP's performance problem, it's more memory efficient and can be used in combination with HHVM.
It's i believe how it should be, Drupal as we all know has a very expensive bootstrapping specially when many modules are installed so the performance boost we get from using ReactPHP is amazingly high! it's not comparable to any other solution available right now 1000x - 2000x boost and even more if the application is written in an asynchronous way.
Sine Drupal 8 is service based and using symphony components as core it seems possible and within reach
Also event driven design is not only for ReactPHP, there are other solutions and more will come in the future
It would be great if we could have opinion of some of core developer regarding how difficult this task is and how we can help to make it happen

Some efforts have been made in the past, might be useful
https://drupal.org/project/daemoncli
https://drupal.org/project/daemon
https://drupal.org/project/drushd

There are also libraries for supporting push technology (web socks) using reactphp which eliminate the need for nodejs. Since it's already in PHP this support can be added to Drupal out of the box.
https://groups.drupal.org/node/192173
http://socketo.me/

A symfony bundle to run symfony under reactphp https://github.com/Blackshawk/SymfonyReactorBundle
Realtime chat using Laravel4 and Reactphp https://medium.com/laravel-4/eaa550829538

I created a wiki page : https://groups.drupal.org/node/417723

Correcting the title
This is still the best way to significantly improve Drupal's performance.
New attempt on this https://github.com/php-pm/php-pm-drupal

Note that this is not possible without patching the core https://github.com/php-pm/php-pm-drupal/tree/master/patches

Targeting against 8.3.x

Wrong update

Updating the summary

Is there an issue for https://github.com/php-pm/php-pm-drupal/blob/master/patches/kentr-allow-... out there?

Not that i know of, so lets ask https://github.com/php-pm/php-pm-drupal/issues/14

I added the related issues for patches

@dawehner, @sinasalek I'll add my patch.

Glad to see that support for the PHP-PM integration is growing.

Tried it today, got a bunch of problems:

* the second patch in kentr/stop_using-2505339-24.patch no longer applies on 8.3.x HEAD
* php-pm/React/RequestParser uses non-existent React\Http\RequestParser , which seems to have been replaced by RequestHeaderParser
* php-pm/httpKernel-adapter/Bridges/HttpKernel::mapRequest() uses non existent methods getFiles() and getPost() on React\Http\Request

The two last ones look like depending on an incorrect version of React.

Created #2829346: Allow repeated calls to DrupalKernel::setSitePath()

2 related issues are not done
- #2613044: Requests are pushed onto the request stack twice, popped once is very close, just needs to explain chosen approach
- second one needs tests #2708827: Allows DrupalKernel::handle to handle more than one requests (rough support for PHP-PM)

Also it's not clear how to handle static vars that still used by core & sessions

RIght, we should basically try to identify every place where we have leaking state in our system. It would be great to collect them all. #2760167: Add \Drupal\Core\Messenger\Messenger solves one of those cases ...

Pager no longer using globals, added related "polishing" issue

This would be great. What do you think about using Swoole that is based on C and supposedly is very fast. There is a discussion here:

React PHP vs Swoole PHP

@miksha nowadays reactphp/swoole are mostly the same - php itself manages promises/generators
Comparisons are mostly synthetic and real cases to compare are php-fpm, nginx-unit, roadrunner - personally (today) I prefer reactphp as it foes not need any extensions installed (there's new extension pecl parallel but it needs php compiled with zts)

@andypost Thank you. ReactPHP doesn't support coroutines by default but it can be combined with RecoilPHP.

Here goes swoole and WP https://github.com/swoole/library/pull/17
So core could redo its quickstart command

Meantime Laravel started swoole / roadrunner 2 custom web-server https://github.com/laravel/octane

This will have the biggest impact on the future of Drupal as a platform and we already have a solid feature-rich, production-ready, high performance async and multi-core library for it: Swoole

I think this issue is very important,Drupal cannot compete with Node.js regarding to speed,so a lot of people are leaving drupal for node.js.

Meh. Node is great, but I'd like to see anyone build something as feature-rich as you could in Drupal in less than ten times the time it would take in Drupal.

Use the tool that is right for the job. You want a microservice or a small web application that is *lightning fast*, please use Node, I would and I do. Want a backend application that can handle thousands of user sessions, present a world-class content-editing experience, manage the most granular permissions you can imagine with a few lines of code, automatically build your database, plug in to hundreds of third-party applications with ongoing security support, etc. etc. etc?... I wouldn't want to build that by myself.

What about if we propose this as a drupal core idea?

@pinkonomy

What about if we propose this as a drupal core idea?

This ticket is already in the Drupal core project. What else is required to propose it as a drupal core idea?

This ticket is already in the Drupal core project. What else is required to propose it as a drupal core idea?

It needs to change the "Project" from "Drupal core" to "Drupal core ideas" and it needs to follow the template that the Drupal core ideas page proposes https://www.drupal.org/project/ideas

Maybe a new ticket needs to be created that is more up to date than current ticket.

I'm not sure if there's any benefit to having this in the ideas issue queue.

That's about actual _features_ on a product/end user level. This is a technical task on a very different level.

https://github.com/php-pm/php-pm-drupal seems to be the only project mentioned in the issue summary to be even remotely up to datele and that wasn't updated in 5 years.

I don't think core itself necessarily needs to support this out of the box, this is IMHO just a meta issue to track improvements that are necessary for something like php-pm-drupal to work. So if you want this to proceed, then have a look at the referenced issues and help push them forward.

@berdir

I don't think core itself necessarily needs to support this out of the box, this is IMHO just a meta issue to track improvements that are necessary for something like php-pm-drupal to work.

At least w/ PHP-PM, it would not work without changes to core. The issues I encountered w/ PHP-PM required patches to core.

IMO, they are not unreasonable changes, and they clean up the architecture.

Agree with @Berdir there is nothing directly actionable here, but there are probably lots of technical subtasks that can be parented under this issue to help improve this, so repurposing this as a meta issue.

@kentr if those patches to core exist as issues already, feel free to relate them or set them as children of this issue so they can be collected together and anyone who wants to work on this can find them more easily. Alternatively if there are specific tasks that need to be done that don't exist yet, create them as children of this issue.

#3108687: Compatibility with/take advantage of code preloading in PHP 7.4 could be a good first step towards something like this, I suspect the improvements required for that issue will be quite similar to those required for this issue.

This was brought up in Slack. Big fan of this idea. Thought I'd chime in with some recent experiences.

At Open Social we're currently running ReactPHP to handle Websockets: https://platform.sh/blog/2021/how-to-build-a-real-time-saas-business-wit...

Most comments in this thread seem to be focused on the processing of the HTTP requests. Which is probably relatively easy to tackle since it's a boundary concern.

When looking into bootstrapping Drupal within ReactPHP for our websocket server, the biggest hurdle that I encountered is that Drupal's database layer. Drupal's database calls are synchronous and loaded entities are all statically stored in memory.

The synchronous nature of the database calls prevents Drupal from actually running in an asynchronous context because it means that querying the data blocks the entire process. This is not an easy thing to solve because it means that the entity layer will need to move from returning entities directly to returning promises. This would have to happen also when running outside of an event loop: the code would still use promises but be executed synchronously.

The shared state is great for performance improvements but for a long running process it will probably cause all of a site's data to be held in memory which will likely exceed memory limits for the application and may cause the server process to be terminated.

Those issues are why eventually we decided on an approach where we use GraphQL (and some RPC-like endpoints) to fetch data from Drupal, rather than bootstrapping Drupal within our websocket server. This allows the websocket server to process other tasks while waiting for Drupal to process our HTTP request.

Even though those are quite fundamental challenges, I still think it would be very cool to tackle. In most of my hobby NodeJS projects I end up rebuilding Drupal in some form or another because its module and data system are just so powerful. Being able to put that into long running real-time processes would be awesome.

In terms of choosing what async event loop/promise library to pick, my vote would go to Revolt PHP which is where AmPHP and ReactPHP (the two largest event loop implementations and ecosystems) are bundling their efforts in anticipation of PHP Fiber support.

There is a good chance that PHP 8.1 will be the minimum required version for Drupal 10 and PHP 8.1 will have the Fiber support. See: #3223435: Track PHP 8.1 support in hosting and distributions.

@Kingdutch How Swoole? it's the most performance and feature rich async solution for PHP. it even out performs nginx! for static content. it's a solution that's already here in production use

@sinasalek I have not tried Swoole. It was disqualified by us for requiring a non-standard PHP extension to be installed. I suspect it's not an option for Drupal for the same reason.

ReactPHP can use a PHP-based implementation of the event loop, so it can work without extensions. Alternatively it can use the "event" extension for native support to speed things up.

From ReactPHP.org:

• Requires no extensions and runs on any platform - no excuses!
• Takes advantage of optional extensions to get better performance when available.

IIRC, the biggest hurdle I encountered with PHP-PM was also regarding global state. AFAIK, it was not possible to keep session state separate between requests. IOW, different requests commingled session state which should not be commingled.

That made the bootstrap-once-and-share-it strategy ineffective, b/c the wrongly-commingled state was in the persistent bootstrap.

I would guess that solving these issues is necessary regardless of which persistent app server framework is preferred.

I believe marcjschmidt brought some of the issues over from GitHub to here.

This isn't a priority for me anymore, but for anyone who's interested there may be other issues in the GitHub queue that are still relevant.

This should be a guideline for drupal core developments. For now, I think Drupal is kilometers behind being compatible with long these technologies.
Recently I had a Drupal-WTF issue with drush: https://www.drupal.org/project/drupal/issues/3215035
It would reappear between requests. But I'm pretty sure Drupal is lot of these bad implementations (like previously said by other comments).

More optimism:
- Laravel Octane will boost the interest in those technologies, swoole and roadrunner
- Roadrunner as already symfony implementations: https://roadrunner.dev/docs/integration-symfony

Final thoughts:
Drupal bootstrap is an obviously huge weight in performance optimization. If drupal survives, it should modernize and get ready for it.
For the roadmap, maybe we should first add tests where requests are reused.

Yes this is sad that Drupal is this far behind when it comes to performance. And it bugs me why Drupal lead Developers never discuss this seriously so far. I believe Drupal has a much better chance in becoming compatible with these libraries comparing to for example WordPress. The performance boost is game changing specially for headless Drupal. I wonder when we will see an Initiative for this. Laravel Octane is a great example of how it can be done. I don't think the amount of effort needed for this compatibility is more than the other successful Drupal initiatives. So may be we should just propose a new initiative so anyone can gather and focus on the problem. I'm pretty much sure that many companies who have Drupal based product would be interested given how much money they can save down the road.

Any suggestion for this initiative's name? we have to start somewhere :) we can put together the details and a roadmap as a starting point and start the practical discussion on how to move forward from there.

Meantime Nginx Unit 1.26.0 fixed opcache (finally shared) so it can have the same perf as reactphp

There is a new project,Revolt PHP,where React PHP and Amp PHP combine forces.
https://revolt.run/
This is important, since PHP 8.1 comes with Fibers support AND Drupal 10 will require PHP 8.1.
Thoughts?

Benchmarking Laravel Octane vs. Express JS
https://lukecurtis.me/benchmarking-laravel-octane-vs-express-js/

I have created the Swoole module for Drupal. It is based on Laravels Octane. The Swoole module for Drupal supercharges your website’s performance by serving it via the Swoole or the OpenSwoole PHP server. Both are PHP extensions.

The (Open)Swoole PHP server boots Drupal once, keeps it in memory and then feeds it requests at supersonic speeds.

For instance on a new and empty site a page request with NginX takes on average 0.060 seconds. The same request on the Swoole server takes on average 0.025 seconds. The request time for the Swoole server should go down even more with the use of Swoole memory tables.

Drupal core needs to be patched to make it all work. Apply the patch located in the Swoole module: patch/drupal-core.patch. There are 2 Drupal issues to have Drupal core updated, so that the patch is no longer needed:
- #3279191: Add the container parameter asynchronous
- #3279192: Change the method Drupal\Core\DrupalKernel::handle() to make it work for the Swoole module

The module is not yet feature complete. It is only for other developers/devops persons to play/test it.

The module is NOT ready for production!

Developement is done: https://gitlab.com/daffie/swoole

Documentation can be found: https://daffie.gitlab.io/swoole/

@daffie this sounds intriguing. did you tackle session handling as well or is that if no concern using swoole?

@attisan: The module does do the session handling, only it could use a review from a security expert.

The module does do the session handling, only it could use a review from a security expert.

So I came to this issue by doing some sleuthing around on session handling for #3283058: Incompatibility with SameSite=Lax or stricter session cookie and I think I've stumbled upon an elegant solution to improve Drupal's session handling writ large, and also avoid (most?) of the core service overrides in the Swoole module.

It all boils down to what I describe in #3283288: Implement Symfony SessionListener - basically, if we stick closer to the Symfony trunk, they are solving this upstream for us. These changes are more or less what is included in the referenced Swoole module, plus some other goodness that would fix issues like #3228658: Session cookie is re-set on all Big Pipe'd responses.

I think Drupal could implement Symfony's SessionListener and reduce our own boilerplate, and reduce the Drupal-specific compatibility layers for things like Swoole. Thoughts?

A list of currently active "persistent app servers"

1. https://driftphp.io/
   PHP Framework on top of Symfony and ReactPHP components
2. https://roadrunner.dev/
   RoadRunner is a high-performance PHP application server, load-balancer, and process manager written in Golang
3. https://appserver.io/
   appserver.io is a multithreaded application server for PHP, written in PHP
4. https://amphp.org/
   Amp is an event-driven concurrency framework for PHP
5. https://reactphp.org/
   Event-driven, non-blocking I/O with PHP
6. https://github.com/php-pm/php-pm
   PHP-PM is a process manager, supercharger and load balancer for PHP applications
7. https://openswoole.com/
   Powering the next-generation microservices and application
8. https://www.swoole.com/
   High Performance Programmatic Server for PHP with Async IO, Coroutines and Fibers

The order is random (chose ol over ul due to formatting). Some of them depend or build on top of others but all seem to tackling with the very same topic. I think we should try to find an agnostic solution to provide support - one that would enable the use of any of these (probably) incompletely listed app servers.

As for compatibility (after fiddling around with a custom react/http setup and swoole (thx @daffie)) I see the session handling as the common denominator of trouble - specifically the fact that session handling is crippled by checks against PHP_SAPI being 'cli'.

Perhaps we could introduce (as @daffie already proposed in #3279191: Add the container parameter asynchronous) an additional global drupal parameter like persistent that - whoever implements a persistent drupal app-server with whichever available variant - can be set and is checked wherever PHP_SAPI is checked.

The asynchronous parameter is only a workaround to allow app-server control container

The real solution is #2371629: [meta] Finalize Session and User Authentication API which moving slowly but more then half of way passed atm

IIRC the main blocker nowadays is #2484991: Add the session to the request in KernelTestBase, BrowserTestBase, and drush

Will depend on this issue:
https://www.drupal.org/project/drupal/issues/3228629

So core could redo its quickstart command

Thanks for sharing @andypost, Drupal's quick-start often freeze when I use it, #3271178: Quick start command installation stops responding.

See also Drupal Quick Start Command > Troubleshooting added by @louis-cuny.

@sinasalek
Can you give more details about how you're using swool with drupal?

Could also be interesting to move to/implement symfony/runtime, which advertises:

The Runtime Component decouples the bootstrapping logic from any global state to make sure the application can run with runtimes like PHP-PM, ReactPHP, Swoole, etc. without any changes.

Used to give a test run of
- swoole - it needs version<5.0, works with some hacks - env is not passed when executed via cli, localhost:8080 fauled to build some links
- roadrunner, works only first request, needs workaround for sessions, failed to deliver static assets ... except front-page
- unit - works with default config, out of box same speed as php-fpm useful for docker as running as one process

I just found this new php server:
"Boot your application once and keep it in memory! It is ready to handle incoming requests in a few milliseconds. FrankenPHP relies on Go's iconic feature: goroutines!
https://frankenphp.dev/

Could this be a solution to persistent app server?Could it be used with Drupal?

Sorry if this is not related to this issue

I'm excited to use nginx-unit with PHP - it just a bit slower then php-fpm but it highly customisable and allows to replace nginx+fpm with single binary (near 1MB in size)

About FrankenPHP, there is a repo for Drupal but worker mode does not work with sessions as per this issue here.

https://github.com/dunglas/frankenphp-drupal/issues/1

Commenting mostly more for my own benefit next time I read this issue; nginx-unit looks like it will be nice, but isn't at feature parity with nginx, yet.

I’m also very interested in Nginx-unit, planning to experiment with it soon. Once I’ve had a chance to properly use it, I’ll share my experience.

Per #24 and #51, these concerns regarding static caching (I guess not to be confused with concerns over static assets) came to mind today when I was implementing some static result caching inside of a service.

I'm not sure how many examples there are in Core; a quick search turned up at least FileCache, and of course there are static collections of loaded entities. And probably most important on the security side, EntityAccessControlHandler::$accessCache.

This seems to me to be a major blocker and even trickier because it's not something that you can control for at runtime.

Am I missing something key here? Cache invalidation on data stored with Cache API would work just fine, I think, but what about all these pesky static caching properties? Seems to me that they must be sandboxed to the currently-handled request? And what's to stop sites from using this pattern and having nasty side-effects when they start using one of these servers? Sorry if I'm repeating others here.

Are services like current_user also an issue, or is that handled in a way via moving this kind of data to the request object? Regardless, there's tons of code (e.g., context system) that rely on the current user in the container.

I think we end up with a handful of patterns, and maybe a few dozen services to touch.
Some static caches can be moved to MemoryCache backends, some others and some services need to be moved to the request object.
This can be done with thin wrapper services or methods.
As of now, it looks it a bit of work, but no big complexity.
(If no new dragons appear...)

I bet it mostly about priority of related issues

I opened #3324241: Provide DIC-friendly, concurrency-safe alternative(s) to `drupal_static()` to address the broader static caching question. TIL that drupal_static() is still the official way to do this, but it's not broadly applied in D10 times.

There is an integration of Laravel Octane with FrankenPHP.
The results show that FrankenPHP is even quicker than Swoole.

https://blog.laravel.com/octane-frankenphp

FrankenPHP has a known issue about crashes when certain PHP constructs and functions are used within Fibers. I think it would be great to open a separate Drupal core issue that tries to find out if Drupal makes any of those calls within fibers. That could happen independently of this issue, which affects all persistent app servers, not just FrankenPHP. However, I don't have any ideas at the moment on what the best way would be to determine if Drupal triggers any of those FrankenPHP crashes without first solving this issue.

The only child issue and possible way forward is #3313404: Use symfony/runtime for less bespoke bootstrap/compatibility with varied runtime environments

FrankenPHP official support in Drupal would be awesome, I'll give it a try if I find the time !

Nice write-up and tool to catch shared data https://dev.to/sergiid/getting-symfony-app-ready-for-swoole-roadrunner-a...

Re: #87, that is the issue we need to untangle at #3324241: Provide DIC-friendly, concurrency-safe alternative(s) to `drupal_static()`. Should we change the parent of that issue to this one? Runtime is one of a few items that need to be addressed. The reference to the ResetInterface in Symfony 7 is excellent and provides us a model to build on instead of rolling our own.

For searchability

Let's add Swoole then also for searchability.

Added an issue to add a Caddyfile to core, used by frankenphp (which is based on caddy), with an rough implementation of the early hints feature that's available thanks to frankenphp (and symfony). We can start by using that in the non-worker mode and once all the issues from this meta are dealt with we can turn on the worker mode of frankenphp to make everything faster without changing much on the infra side.