Problem/Motivation
On Drupal sites using Flysystem S3 with multiple schemes, the S3 integration condition does not fire if the scheme has a name other than S3. Additionally, the TrustedRedirectReponse triggers a leaked caching metadata error.
Error message:
The controller result claims to be providing relevant cache metadata, but leaked metadata was detected.
The error happens when flysystem returns a local URL for the TrustedRedirectResponse.
Proposed Solution
Patch is attached to check for scheme names containing 's3' in the string, and replaces TrustedRedirectResponse with RedirectResponse. For the scheme checking, it would be a better solution to access the scheme settings and access the type property, which would always be 's3' for those file systems.
Example Flysystem S3 settings:
Note the key 's3' is default in the example, but that can be any string for the scheme. In this example 'mybucket' is the $scheme name.
$schemes = [
'mybucket' => [
'driver' => 's3',
'config' => [
'key' => '[your key]', // 'key' and 'secret' do not need to be
'secret' => '[your secret]', // provided if using IAM roles.
'region' => '[aws-region-id]',
'bucket' => '[bucket-name]',
// Optional configuration settings.
// ...
// 'public' => TRUE,
// ...
],
'cache' => TRUE, // Creates a metadata cache to speed up lookups.
],
];
$settings['flysystem'] = $schemes;
It is also important to make sure the current functionality is unchanged for the s3fs module.
| Comment | File | Size | Author |
|---|---|---|---|
| #15 | commerce_file-s3-3263735-15.patch | 3.06 KB | nathaniel |
| #15 | interdiff_12-15.txt | 941 bytes | nathaniel |
Comments
Comment #2
JProffitt commentedComment #3
JProffitt commentedUploaded patch containing proposed updates to scheme condition check and RedirectResponse object
Comment #4
nathaniel commentedThanks for reporting the issue and the patch! Attached one that checks the flysystem driver settings.
Comment #5
jsacksick commented$flysystemSettingscasing is incorrect.Also, what module defines the following setting?
Comment #6
nathaniel commentedLet's try this one. Updated dependency injection for settings. The flysystem & flysystem_s3 module. Custom settings are added to settings.php. Here is an example from the readme:
Note the key 's3' is default in the example, but that can be any string for the scheme.
Comment #7
nathaniel commentedAdded a code comment to help describe what the settings are for.
// Check if Flysystem settings exist and if there are any S3 scheme's.I don't agree with the case change since the module and library are both spelled "Flysystem", but I'm happy to change that if needed.
Comment #8
jsacksick commentedWell the issue here is Drupal coding standards, only class variables should be cased like this. Otherwise from within methods, the variable should be named like this (snake case):
$flysystem_settingsComment #9
nathaniel commentedSnake case! I misunderstood the note.
Comment #10
JProffitt commentedWhile this patch worked on my local docker environment running PHP7.3 and D8.8.12 (We're in the process of moving to D9, but had to bring up even older stuff in an intermediate step before moving up to PHP7.4), we noticed that we were getting a consistent 404 in the k8s cluster environments. Something in the S3 block is causing the private file link to fail without hitting any of the read methods in the league libraries. I've not been able to trace the exact location, but it seems to be something in the stream wrapper code. The prefix is generated appropriately for the private S3 bucket, but it never actually makes the call to S3, instead returning 404.
Removing the entire S3 condition block temporarily solves this issue on D8.8.12, but I'm hoping whatever is in the core file and stream wrapper libraries causing that issue is a non-issue for D9. I'll be able to get back to testing that now that I've unbroken private file downloads in our production instance.
Comment #11
nathaniel commentedI ran a few manual tests on a fresh install. I think it has to be a TrustedRedirectResponse if an external URL is used. The
'public' => TRUEoption for flysystem_s3 uses the external S3 bucket URL. The patch with TrustedRedirectResponse worked for me. With'public' => FALSEit uses an internal URL and I get the error:Additional notes:
I'll try to test a bit more, but I can confirm that both seem to work without the if 's3' code.
If we still want to support the external S3 URL this patch needs to check if the public option is also TRUE before returning the TrustedRedirectResponse.
Note that I am testing on 8.9.20 with everything as up to date as it can be.
Someone should probably test the s3fs module (I'll try to sometime soon).
Side note: the download counter is not updating with S3.Comment #12
nathaniel commentedSame results on D9 - everything up to date.
I'm not familiar with all of the s3fs settings, but initial testing with a basic setup works as expected ($scheme is 's3' and redirects to the S3 URL).
Updated the patch to keep the TrustedRedirectResponse and check the flysystem config public setting.
Comment #13
nathaniel commentedChanging to a feature request to add support for Flysystem S3.
Updated issues summary to include example settings and the full error message.
Comment #14
nathaniel commentedIf flysytem settings uses the default 's3' scheme the same error can occur if public is not true...
This should work:
if (($scheme === 's3' && !isset($flysystem_settings[$scheme])) || ...Comment #15
nathaniel commentedOkay, final patch from me for now!
Updated the if statement to handle flysystem with 's3' scheme name.
Comment #17
jsacksick commentedCommitted!