Add Drupal 10.6/11.3 support

Per #3324737: Do not support core versions newer than current published beta or higher we wait for Core to release a beta before adding support into the module.

Core has released a beta.

We do have #3559132: FileSystem::deleteRecursive() leaves files/directories when realpath() returns FALSE that is known breaking of proper operations with S3FS. That isn't an S3FS bug and current releases of S3FS running on supported versions of core are also impacted, however I'm not sure we should re-enforce that we support the latest versions of core when we know there are issues until either that issue is fixed or until we provide our own resolution for it.

Addtionaly adding to complexity (this has only been lightly mentioned in Slack) there are concerns about retaining D.O. as the home for development given issues with the D.A. I'm torn between rushing a migration to a new location in order to be sure the word gets out (rather than being mid-release and development just stops) and holding off to allow one more release through D.O.

On a positive side: I did a very quick test a few weeks ago and no obvious faults observed, this might be an easy minor to support.

For those of us out of the loop: what concerns are there with the Drupal Association and the continued use of Drupal.org for development?

Separate from that topic, I've done some reading about the issues with public:// and private:// takeover, and I can understand why you'd want to lock to specific minor versions of Drupal to ensure downstream stability. That being said, is there an argument against splitting the core functionality of this module into a separate module project? (My organization uses this module on a handful of our sites, but we don't use the takeover functionality. The tighter version constraint that this module has on Drupal core has come up as a factor that we now have to consider when coordinating updates to our sites.)

I could imagine the following migration path for users:

1. A new module project is created (e.g., s3fs_core) less any code with BC concerns.
2. Next major version of this project depends on the new s3fs_core, and extends it to provide the takeover functionality.
3. Users who do not use the takeover functionality can now upgrade to the new major of this project, then safely uninstall it and swap to a dependency on s3fs_core instead.

That being said, is there an argument against splitting the core functionality of this module into a separate module project?

The 4.x branch does somewhat set the groundwork for the code being significantly more modular as it has the client, the bucket access and the streamWrappers split into individual dedicated modules. However I don't believe this current layout would allow us to be fully decoupled from core.

I've done some reading about the issues with public:// and private:// takeover, and I can understand why you'd want to lock to specific minor versions of Drupal to ensure downstream stability.

The concerns are unfortunately not limited to just public/private takeover. Takeover issues have been a significant portion of our concerns in the past however not all of them.

Core's BC policy is quite restrictive, s3fs having just the following line of code
class S3fsFileService implements FileSystemInterface { causes us to be no longer protected by a BC gurantee with Core (BC for interfaces is for type-hinting not for implementing). This is our FileSystem Decorator, the primary purposes of this class is to protect against silent data loss in core, (see #3057565: copy(), move(), and move_uploaded_file() ignores return value of stream_flush() ), however it also solved bug reports due to disk space consumption with large files. Changes in these classes have/would have caused WSOD's on sites. Technically speaking Core could change this interface a PATCH release and we should be restricting tighter. Other solutions to avoid this entanglement are to open an issue asking core to make this interface @api.

Some of the faults we have encountered in recent release cycles:

D11.2: WSOD, caused by a circular reference fault #3528246: [META] Add support for Drupal 10.5/11.2. In defense of core, I hit that once before with 3rd party code and in hindsight I should have raised an issue to avoid a possible repeat. Core has committed to remove the root cause in Drupal Core 13.0.0. This impacted all schemes not just public/private takeover.

D10.5/D11.2 Likely should have been blocked on core breaking FileSystem:recursiveDelete(). Strictly speaking this isn't an S3FS bug, however equally in the S3FS project we know that Core's development team routinely fails to consider non local streamWrappers leaving room to debate how much we should be validating and restricting usage of S3FS to avoid unexpected experiences for site owners. This impacts all schemes not just public/private takeover.

D10.3: Changes to core interface specifications impacting all schemes and changes to internal classes extended for the s3:// Image Style Generation. We have decoupled Image Style Generation from the core controller class to reduce future risks and move us one step closer to having no further non-api code.

D10.1 was public:// takeover related to CSS/JS aggregation With the new assets:// scheme in core we no longer interact and have been able to avoid a significant portion of Core non-api dependency.

We have 40% of Core minor releases in the 10.x series to date wold have broken the s3:// scheme in some manner.

For those of us out of the loop: what concerns are there with the Drupal Association and the continued use of Drupal.org for development?

Acknowledging I have seen this question. I'm going to break the response out into a separate message when I have a bit more time to respond.

For those of us out of the loop: what concerns are there with the Drupal Association and the continued use of Drupal.org for development?

First off, let me start by acknowledging that I am a strong willed and opinionated Security Engineer, I am one to fight for what I believe and prefer to prevent incidents before they can occur. I am not one to shy away from questioning community public figures, individuals who have willfully and knowingly put themselves in positions of enhanced scrutiny. I have ruffled my share of feathers over the years. As a Maintainer who posts code to the Drupal.Org (D.O.) marketplace I am also subject to enhanced scrutiny for the design decisions I make on projects I manage.

Unfortunately the Drupal Association(DA) has created a situation where it is unclear how much longer I will have access to D.O. or git.drupal.org(G.D.O) to manage issues, commit code, tag releases, or to access security.drupal.org(S.D.O) to file vulnerability reports.

For context I am the only active maintainer on S3FS and it has been that way since ~2021.

In September the DA sent notice that they intended to terminate my accounts access to D.O. They have (obviously) not yet done so, however to my knowledge they have not yet closed out their proceedings. The information the DA has given me is that this originated from the Drupal Security Team(DST).

The September incident is the final trigger, Development on projects I maintain has essentially stalled due to a loss of confidence in the ecosystem.

I am in a spot where I have absolutely zero faith in the DA or any of its representatives. The DA have had several incidents over the years where good developers, those who were far more valuable to the Drupal ecosystem than I will likely ever be, have been pushed out and I have no reason to believe that I'm not next to join that lineage.

I am lucky that Drupal is not my livelihood, a luxury many other developers here may not have which likely contributes to why time a threat has been lodged my way it just deepens my resolve to raise concerns.

Other related incidents causing this lack of confidence in the DA include:

• May 2022: In response to discussing DST members being non responsive to S.D.O reports I was direct messaged on Slack with threats of potential termination of ability report/respond to security issues and revocation of maintainer permissions on projects enrolled in the Drupal Security Opt In program (S3FS would be one). See www.drupal.org/project/securitydrupalorg/issues/3280775#comment-14521904 for the post that triggered this threat.
• Additional occasions that appear to be indirect threats to terminate access spread out throughout the years (discussion of changes to D.O. Terms of Service to make it easier to revoke access to S.D.O and G.D.O that could reasonably be linked to interactions I had with the DST through S.D.O. or other public calls for accountability).
• In May the DST opened an issue #3522339: Clarify policy for revoking security advisory coverage to create a policy allowing them to terminate security opt in privileges for users, 2 weeks later, without any policy having been agreed to (and to date no policy has yet been created) the DST terminated privileges to a user and removed security support from all the projects they maintained. To my knowledge this issue was not targeted at my account, however I was the reporter/reviewer on several of the S.D.O issues and observed no actions I would classify as reasonable grounds to terminate a maintainer raising further questions to will the DA terminate users without due process.

Any chance Drupal 11.30 support will be released by December 19th?

ny chance Drupal 11.30 support will be released by December 19th?

We still need discussion around #3559132: FileSystem::deleteRecursive() leaves files/directories when realpath() returns FALSE impact on S3FS and best way to handle as its the primarily known holdup at this time.

Speaking as a maintainer: Ideally core fixes the issue, I really don't like having to clean up their mistakes in contrib (though I'll do so if left with no other choice). I'm not fond of (even though it would be the lowest effort path to do so) releasing an update to support a version of Core where I know at least some reasonably significant and not straightforward part of the normal usage is broken (and that is ignoring the possible GDPR concerns involved).

I would imagine the thing holding up 11.3 is also holding up 10.6?

IIRC i did run a test on 10.6 and didn't see any obvious issues.

Correct, the flaw is present in Core 10.5 and 11.2 as well, however we already released compatibility with those versions and can't undo that.

Most likely location for the flaw to be encountered will be broken Image Styles Generation or image style flushing, make a style change and it doesn't apply as the old style is never removed from disk. Credit to @oli4 for that, I had them (on Slack) running down as far as looking at S3FS database entries trying to track what turned out to be a core bug. That is the insidious part of that bug, its faults, unless you know about them in advance, can create some very unexpected situations.

I use S3FS on several production sites and wanted to say I appreciate the work that’s gone into maintaining it.

For context, I’ve ported and maintain S3FS for Backdrop CMS, where core moves more slowly, so I haven’t been following the Drupal 10/11-specific implementation as closely and would need some time to get up to speed. That said, I’m happy to help in a scoped way if it’s useful—testing against newer core versions, reviewing patches, or helping with targeted fixes—without stepping into governance or security processes.

Totally understand if now isn’t the right time, but wanted to offer.

We are mostly only using this to store documents attached to webforms via private file on s3. Doesn't seem like we'd run into much issue by moving to 10.6.x.

That is fair for your deployment, as a maintainer I unfortunately have to work against 'worst case' scenario assumptions. If someone can give arguments for general reasons that are not workflow specific that would be helpful.

Ideally the core team fixes #3559132: FileSystem::deleteRecursive() leaves files/directories when realpath() returns FALSE however seeing no motion from them we may just be forced to fix it ourselves. Opened #3564611: Core assumes deleteRecursive() will only be called for files on local disk Bonus if we fix it in S3FS is that it backports to older core versions, negative is it further re-enforces S3FS being 'off api'. Longer term, opens up the potential for more API optimized deletion of files.

drupal/core-recommended 11.3.1
I have the issue that drupal/s3fs version 3.9 requires Drupal core versions >=8.8 < 10.6 || >=11.0 < 11.2 || ~11.2.3, but you're trying to install drupal/core-recommended 11.3.1, which is outside the supported range. The drupal/s3fs module doesn't support Drupal 11.3.x yet (except for 11.2.3).

Committed to dev and 8.x-3.10 published.