Automated Drupal 11 compatibility fixes for flysystem_s3

I'm unable to install this fork in D11 with composer due to its dependency on flysystem v2 which is not D11 compatible. It needs to be dependent on version 3, or at least this MR: https://git.drupalcode.org/issue/flysystem-3451092/-/tree/fix-merge-error

MR!21 Makes this module compatible with Drupal 11. Functionality is working as expected.

RTBC +1

@shashi_shekhar_18oct as I said above, the dependency on version 2.0 of flysystem module, which does not support Drupal 11, means that this version of flysystem_s3 cannot be installed with composer into a Drupal 11 site. This module needs to be made dependent on a version of flysystem that supports D11.

Looking at https://www.drupal.org/project/flysystem/issues/3378738 , it looks like there's no chance Flysystem v3 will be ready anytime soon, so I drop my objection. I'll see if I can make this work with the 2.2.x-dev MR at https://www.drupal.org/project/flysystem/issues/3433330 .

I can confirm that MR!21 can now be installed with composer and enabled in Drupal 11.

Hey all,

Any chance of getting this merged in and a release?

Cheers.

I'm now seeing,

Call to deprecated method generateImageStyle() of class Drupal\flysystem_s3\Flysystem\S3. Deprecated in flysystem:2.2.0 and is removed from flysystem:3.0.0. Adapters should use generateImageUrl() to enable non-blocking image uploads.

on line 227 of S3.php. I'll see if I can fix it.

Following up on this. I'm looking to try and use flysystem for S3 in a new D11 project and avoiding drupal-lenient unless absolutely necessary =).

On second thoughts, using 2.1.x for D10 and 2.2.x for D11 prevents us from ever having another D10 minor release, which is something we may possibly want to do. So let's use 3.0.x for D11.

I've tried to push this along as far as I can today.

The only issue I'm currently aware of stopping this being merged and having a D11 release is that the PHPUnit job in CI is currently failing.

For example https://git.drupalcode.org/project/flysystem_s3/-/jobs/5464727

All the tests themselves pass, so I presume the job failing is due to the deprecations. I can't see the specifics of those in CI, but running locally I can see

❰elliot.ward❙~/code/d11test/d11/web/modules/contrib/flysystem_s3(git:3430648-d11_ready)❱✔≻ ddev exec phpunit -c web/core/phpunit.xml.dist web/modules/contrib/flysystem_s3
HTML output directory sites/simpletest/browser_output is not a writable directory.

PHPUnit 10.5.36 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.21
Configuration: /var/www/html/web/core/phpunit.xml.dist

DD.D......                                                        10 / 10 (100%)

Time: 00:02.496, Memory: 20.00 MB

3 tests triggered 5 deprecations:

1) /var/www/html/vendor/symfony/error-handler/DebugClassLoader.php:341
Method "Symfony\Component\EventDispatcher\EventSubscriberInterface::getSubscribedEvents()" might add "array" as a native return type declaration in the future. Do the same in implementation "Drupal\flysystem\ImageStyleCopier" now to avoid errors or add an explicit @return annotation to suppress this message.

Triggered by:

* Drupal\Tests\flysystem_s3\Functional\ModuleInstallUninstallWebTest::testInstallationAndUninstallation
  /var/www/html/web/modules/contrib/flysystem/tests/src/Functional/ModuleInstallUninstallWebTest.php:27

2) /var/www/html/vendor/symfony/error-handler/DebugClassLoader.php:341
Method "Aws\CacheInterface::get()" might add "mixed" as a native return type declaration in the future. Do the same in implementation "Drupal\flysystem_s3\AwsCacheAdapter" now to avoid errors or add an explicit @return annotation to suppress this message.

Triggered by:

* Drupal\Tests\flysystem_s3\Unit\AwsCacheAdapterTest::testBasicGetSetDelete
  /var/www/html/web/modules/contrib/flysystem_s3/tests/src/Unit/AwsCacheAdapterTest.php:20

3) /var/www/html/vendor/symfony/error-handler/DebugClassLoader.php:341
Method "League\Flysystem\AwsS3v3\AwsS3Adapter::has()" might add "bool" as a native return type declaration in the future. Do the same in child class "Drupal\flysystem_s3\Flysystem\Adapter\S3Adapter" now to avoid errors or add an explicit @return annotation to suppress this message.

Triggered by:

* Drupal\Tests\flysystem_s3\Unit\Flysystem\S3Test::testGetExternalUrl
  /var/www/html/web/modules/contrib/flysystem_s3/tests/src/Unit/Flysystem/S3Test.php:61

4) /var/www/html/vendor/symfony/error-handler/DebugClassLoader.php:341
Method "League\Flysystem\AwsS3v3\AwsS3Adapter::getMetadata()" might add "false|array" as a native return type declaration in the future. Do the same in child class "Drupal\flysystem_s3\Flysystem\Adapter\S3Adapter" now to avoid errors or add an explicit @return annotation to suppress this message.

Triggered by:

* Drupal\Tests\flysystem_s3\Unit\Flysystem\S3Test::testGetExternalUrl
  /var/www/html/web/modules/contrib/flysystem_s3/tests/src/Unit/Flysystem/S3Test.php:61

5) /var/www/html/vendor/symfony/error-handler/DebugClassLoader.php:341
Method "League\Flysystem\AwsS3v3\AwsS3Adapter::upload()" might add "array|bool" as a native return type declaration in the future. Do the same in child class "Drupal\flysystem_s3\Flysystem\Adapter\S3Adapter" now to avoid errors or add an explicit @return annotation to suppress this message.

Triggered by:

* Drupal\Tests\flysystem_s3\Unit\Flysystem\S3Test::testGetExternalUrl
  /var/www/html/web/modules/contrib/flysystem_s3/tests/src/Unit/Flysystem/S3Test.php:61

OK, but there were issues!
Tests: 10, Assertions: 27, Deprecations: 5.

These deprecations do not seem to be raised on D10 / flysystem_s3 2.1.x / phpunit 9.

I probably won't get time to look at this again in the nearish future, so if anyone is requiring a D11 release, the best thing you can do to move this issue forward is to look at why PHPunit is failing in CI.

Here's a pass at a patch for the 2 deprecation issues I see in flysystem_s3, for AwsCacheAdapter.php and S3Adapter.php - the remaining deprecation for the ImageStyleCopier.php file in the core flysystem module.

When I run phpunit on my local, it complains about a bunch of missing composer dependencies. One at a time, of course. If I require each of them, I eventually get

PHP Fatal error: Uncaught ArgumentCountError: Too few arguments to function Drupal\Tests\flysystem_s3\Unit\Flysystem\S3Test::__construct(), 0 passed in /var/www/html/vendor/phpunit/phpunit/src/Framework/TestBuilder.php on line 138 and exactly 1 expected in /var/www/html/web/modules/composer/flysystem_s3/tests/src/Unit/Flysystem/S3Test.php:48

Now I'm getting,

1) Drupal\Tests\flysystem_s3\Unit\AwsCacheAdapterTest::testBasicGetSetDelete
Failed asserting that false is null.

Swapping false for null should resolve that, ensuring AwsCacheAdapter::get() returns null (instead of false) when an item is not found. This should make the “assert null on a cache‐miss” pass again.

--- a/src/AwsCacheAdapter.php
+++ b/src/AwsCacheAdapter.php
@@ -38,7 +38,7 @@ class AwsCacheAdapter implements CacheInterface {
public function get(string $key): mixed {
if ($item = $this->cache->get($this->prefix . $key)) {
return $item->data;
}
- return false;
+ return null;
}

public function set(string $key, mixed $value): void {

Thanks for all the ongoing work on this.

Just want to remind folks we need to use FALSE and NULL rather than false and null for coding standards!

@benstallings

I think @eli-t's comment means it should be:

--- a/src/AwsCacheAdapter.php
+++ b/src/AwsCacheAdapter.php
@@ -38,7 +38,7 @@ class AwsCacheAdapter implements CacheInterface {
public function get(string $key): mixed {
if ($item = $this->cache->get($this->prefix . $key)) {
return $item->data;
}
- return false;
+ return NULL;
}

public function set(string $key, mixed $value): void {

OK, but I had already made that change in the MR before either of you commented about it. So the error message must be due to something else.

So I capitalized null in the test as well, and now I don't understand what phpunit is saying about why it is failing.

Posted a related deprecation patch for core flysystem's ImageStyleCopier.php file in the 3.0 D11-compatibility issue - I think that's the correct place.

OK, but I had already made that change in the MR before either of you commented about it. So the error message must be due to something else.

Apologies - I didn't mean to be confusing.

The case change was unrelated to the PHPUnit failure - it was to do with failed PHPCS run in the validate stage of the pipeline after commit b19950b8 : https://git.drupalcode.org/issue/flysystem_s3-3430648/-/jobs/5477376

The validate stage is now passing again following commit a7d997cb .

Looks like the deprecations about which phpunit was complaining were in the custom phpunit.xml.dist file supplied with this project.

The merge pipeline now passes with that file removed.

PHPunit says we have 10 tests and 26 assertions all passing with this removed.
These are the same totals as before the phpunit.xml.dist was removed, so it looks like this file was unnecessary.

I am not aware of any more issues with this merge request.

Testing with flysystem 2.3.x and this MR, I can successfully upload image field images to S3.

However when trying to view such images I get error below.

Not sure if this is just a config issue on my test setup or a genuine D11 issue.

The website encountered an unexpected error. Try again later.

Symfony\Component\Routing\Exception\RouteNotFoundException: Route "flysystem.image_style_redirect.serve" does not exist. in Drupal\Core\Routing\RouteProvider->getRouteByName() (line 208 of core/lib/Drupal/Core/Routing/RouteProvider.php).
Drupal\Core\Routing\UrlGenerator->getRoute() (Line: 269)
Drupal\Core\Routing\UrlGenerator->generateFromRoute() (Line: 105)
Drupal\Core\Render\MetadataBubblingUrlGenerator->generateFromRoute() (Line: 96)
Drupal\Core\Render\MetadataBubblingUrlGenerator->generate() (Line: 94)
Drupal\flysystem_s3\Flysystem\S3->generateImageUrl() (Line: 227)
Drupal\flysystem_s3\Flysystem\S3->getExternalUrl() (Line: 75)
Drupal\flysystem\FlysystemBridge->getExternalUrl() (Line: 103)
Drupal\Core\File\FileUrlGenerator->doGenerateString() (Line: 66)
Drupal\Core\File\FileUrlGenerator->generateAbsoluteString() (Line: 255)
Drupal\image\Entity\ImageStyle->buildUrl() (Line: 302)
.
.
.

Setting up with Drupal 10, flysystem 2.2.0-beta2 and flysystem_s3 2.1.3 and exactly the same settings.php configuration for flysystem works without the issue in comment #30 so this doesn't look like config, and does look like an outstanding D11 issue in either flysystem or flysystem_s3.

This issue is due to flysystem_s3 swapping out the deprecated \Drupal\flysystem\Plugin\ImageStyleGenerationTrait::generateImageStyle() for its nominated replacement generateImageUrl().

generateImageUrl() relies on the presence of the flysystem.image_style_redirect.serve route, which doesn't appear to always be dynamically generated:

return \Drupal::urlGenerator()->generate('flysystem.image_style_redirect.serve', $args, UrlGeneratorInterface::ABSOLUTE_URL);

flysystem_s3 will have to revert to the old function, and inform phpstan to ignore this deprecation.

This will stop flysystem_s3 from being compatible with flysystem 3.0.x until it is resolved.

Adding credits

Well done everyone! We got there in the end.

New release 3.0.0 coming imminently.

The issue in #33 is, AFAICT, actually a regression in Flysystem relating to image style generation with non-local/remote adapters. So this was early smoke from that regression. See #3513799: Regression: Image style generation no longer works for non-local drivers.