Problem/Motivation

On Drupal sites using Flysystem S3 with multiple schemes, the S3 integration condition does not fire if the scheme has a name other than S3. Additionally, the TrustedRedirectReponse triggers a leaked caching metadata error.

Error message:
The controller result claims to be providing relevant cache metadata, but leaked metadata was detected.

The error happens when flysystem returns a local URL for the TrustedRedirectResponse.

Proposed Solution

Patch is attached to check for scheme names containing 's3' in the string, and replaces TrustedRedirectResponse with RedirectResponse. For the scheme checking, it would be a better solution to access the scheme settings and access the type property, which would always be 's3' for those file systems.

Example Flysystem S3 settings:

Note the key 's3' is default in the example, but that can be any string for the scheme. In this example 'mybucket' is the $scheme name.

$schemes = [
  'mybucket' => [
    'driver' => 's3',
    'config' => [
      'key'    => '[your key]',      // 'key' and 'secret' do not need to be
      'secret' => '[your secret]',   // provided if using IAM roles.
      'region' => '[aws-region-id]',
      'bucket' => '[bucket-name]',
      // Optional configuration settings.
      // ...
      // 'public' => TRUE, 
      // ...
    ],

    'cache' => TRUE, // Creates a metadata cache to speed up lookups.
  ],
];

$settings['flysystem'] = $schemes;

It is also important to make sure the current functionality is unchanged for the s3fs module.

Comments

JProffitt created an issue. See original summary.

JProffitt’s picture

Issue summary: View changes
JProffitt’s picture

StatusFileSize
new1.4 KB

Uploaded patch containing proposed updates to scheme condition check and RedirectResponse object

nathaniel’s picture

Status: Active » Needs review
StatusFileSize
new1.8 KB
new1.1 KB

Thanks for reporting the issue and the patch! Attached one that checks the flysystem driver settings.

jsacksick’s picture

$flysystemSettings casing is incorrect.

Also, what module defines the following setting?

Settings::get('flysystem', []);
nathaniel’s picture

StatusFileSize
new3.46 KB
new2.07 KB

Let's try this one. Updated dependency injection for settings. The flysystem & flysystem_s3 module. Custom settings are added to settings.php. Here is an example from the readme:

Note the key 's3' is default in the example, but that can be any string for the scheme.

$schemes = [
  's3' => [
    'driver' => 's3',
    'config' => [
      'key'    => '[your key]',      // 'key' and 'secret' do not need to be
      'secret' => '[your secret]',   // provided if using IAM roles.
      'region' => '[aws-region-id]',
      'bucket' => '[bucket-name]',
      // Optional configuration settings.
      // ...
    ],

    'cache' => TRUE, // Creates a metadata cache to speed up lookups.
  ],
];

$settings['flysystem'] = $schemes;
nathaniel’s picture

StatusFileSize
new3.56 KB
new721 bytes

Added a code comment to help describe what the settings are for.
// Check if Flysystem settings exist and if there are any S3 scheme's.

I don't agree with the case change since the module and library are both spelled "Flysystem", but I'm happy to change that if needed.

jsacksick’s picture

I don't agree with the case change since the module and library are both spelled "Flysystem", but I'm happy to change that if needed.

Well the issue here is Drupal coding standards, only class variables should be cased like this. Otherwise from within methods, the variable should be named like this (snake case):
$flysystem_settings

nathaniel’s picture

StatusFileSize
new3.57 KB
new910 bytes

Snake case! I misunderstood the note.

JProffitt’s picture

While this patch worked on my local docker environment running PHP7.3 and D8.8.12 (We're in the process of moving to D9, but had to bring up even older stuff in an intermediate step before moving up to PHP7.4), we noticed that we were getting a consistent 404 in the k8s cluster environments. Something in the S3 block is causing the private file link to fail without hitting any of the read methods in the league libraries. I've not been able to trace the exact location, but it seems to be something in the stream wrapper code. The prefix is generated appropriately for the private S3 bucket, but it never actually makes the call to S3, instead returning 404.

Removing the entire S3 condition block temporarily solves this issue on D8.8.12, but I'm hoping whatever is in the core file and stream wrapper libraries causing that issue is a non-issue for D9. I'll be able to get back to testing that now that I've unbroken private file downloads in our production instance.

nathaniel’s picture

Status: Needs review » Needs work

I ran a few manual tests on a fresh install. I think it has to be a TrustedRedirectResponse if an external URL is used. The 'public' => TRUE option for flysystem_s3 uses the external S3 bucket URL. The patch with TrustedRedirectResponse worked for me. With 'public' => FALSE it uses an internal URL and I get the error:

The controller result claims to be providing relevant cache metadata, but leaked metadata was detected.

Additional notes:

Removing the entire S3 condition block temporarily solves this issue

I'll try to test a bit more, but I can confirm that both seem to work without the if 's3' code.

If we still want to support the external S3 URL this patch needs to check if the public option is also TRUE before returning the TrustedRedirectResponse.

Note that I am testing on 8.9.20 with everything as up to date as it can be.

Someone should probably test the s3fs module (I'll try to sometime soon).

Side note: the download counter is not updating with S3.

nathaniel’s picture

Status: Needs work » Needs review
StatusFileSize
new2.03 KB
new3.02 KB

Same results on D9 - everything up to date.

I'm not familiar with all of the s3fs settings, but initial testing with a basic setup works as expected ($scheme is 's3' and redirects to the S3 URL).

Updated the patch to keep the TrustedRedirectResponse and check the flysystem config public setting.

nathaniel’s picture

Title: S3 Support leaks metadata, does not cover schemes with names other than s3 » Add support for Flysystem S3
Category: Bug report » Feature request
Issue summary: View changes

Changing to a feature request to add support for Flysystem S3.

Updated issues summary to include example settings and the full error message.

nathaniel’s picture

Status: Needs review » Needs work

If flysytem settings uses the default 's3' scheme the same error can occur if public is not true...

This should work:
if (($scheme === 's3' && !isset($flysystem_settings[$scheme])) || ...

nathaniel’s picture

Status: Needs work » Needs review
StatusFileSize
new941 bytes
new3.06 KB

Okay, final patch from me for now!

Updated the if statement to handle flysystem with 's3' scheme name.

  • jsacksick committed 78558b2 on 8.x-2.x
    Issue #3263735 by Nathaniel, JProffitt, jsacksick: Add support for...
jsacksick’s picture

Status: Needs review » Fixed

Committed!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.