Problem/Motivation

Flysystem 3.x is now stable (current stable release version is 3.19), and is receiving all feature enhancements and support. The new version has a much simpler and straightforward API that is more closely aligned with the StreamWrapper interface that is part of Drupal Core.

Proposed resolution

Let's refactor the Flysystem main and supporting module ecosystem to update the modules to be based on Flysystem 3.x.

Remaining tasks

* Create Drupal Flysystem 3.x branch.
* Assess Flysystem 3.0+ to determine the best path forward to integrate with Drupal 10:
* Develop a Roadmap for implementation, and break down tasks into an orderly implementation.

Initial assessment of the Flysystem 3.x library indicates that a roadmap that includes the following is probably the most straightforward implementation with the current Flysystem library:
- Leverage the existing Flysystem Filesystem Class to create an integration point that implements Drupal's StreamWrapper interface (Flysystem's Filesystem class has a 90% compliant implementation with StreamWrapper interface currently)
- Create an Adapter interface within the Drupal Flysystem module that adopts the Flysystem Adapter Interface, and implements any additional Drupal intergration points needed for implementation
- Write an Adapter for a Local filesystem stream wrapper, which is part of the base Flysystem 3.x library
- Develop an Adapter implementation plan for supplemental Drupal Flysystem modules, such as Flysystem S3, Flysystem FTP, etc.

User interface changes

* It appears that these would be minimal; however implementation of an admin UI to allow for configuration options to be managed through the UI would be a nice feature enhancement. Currently, most configuration for Flysystem stream wrappers takes place in the settings.php file directly.

API changes

* Since the API for Flysystem 3.x has significant changes from the 1.x version, documentation of API changes (which would be significant breaking changes) would be in order.
* It's worth noting that, while there are significant breaking changes it is likely that a "migration" would not be in order for content (file assets), but a minimal migration of configuration may be needed.

Data model changes

* Data model changes are not in order here; storage of data related to assets managed with the stream wrappers would leverage existing Drupal data structures, as does the current module versions.

League\Flysystem 3.x Overview

External Dependencies

league/mime-type-detection
Used to detect nimetypes:
* based on file names and file contents (MimetypeDetection\MimeTypeDetector)
* based on file extension (MimetypeDetection\ExtensionLookup)

Interfaces in League/Flysystem v3

The two key interfaces here are FilesystemOperator and FilesystemAdapter.

FilesystemWriter

Contract interface for CUD (Create / Update / Delete) methods

FilesystemReader

Contract interface for R (Read) methods

FilesystemOperator extends FilesystemReader, FilesystemWriter

Main interface for the library. Implemented by Filesystem, which is the primary class through which interaction happens, including managing the adapters for different supported file systems.

FilesystemException extends Throwable

Main interface for the libraries Exception identification.

FilesystemOperationFailed extends FilesystemException

FilesystemAdapter

Contract interface for supported adapters. Class instances are injected into Filesystem instances to register various file systems interaction support.

PathNormalizer

Contract for path normalization methods. Class instances implementing this interface are injected into Filesystem instances.

ChecksumProvider

Contract for methods to provide ad-hoc chucksum calculations (defined as using MD5 hash for checksums)

StorageAttributes extends JsonSerializable, ArrayAccess

Contract for methods performing operations on File, Directory, and Mount Attributes

UnixVisibility/VisibilityConverter

Contract for methods managing permissions for Files and Directories.

UrlGeneration/TemporaryUrlGenerator

Contract for methods related to temporary file URLs

UrlGeneration/PublicUrlGenerator

Contract for methods related to public file URLs

Comments

lhridley created an issue. See original summary.

lisa.rae’s picture

Version: 2.0.x-dev » 3.x-dev
lisa.rae’s picture

Issue summary: View changes
Related issues: +#3196056: Create Flysystem 3.0.x branch
lisa.rae’s picture

Category: Feature request » Plan
BioALIEN’s picture

I recommend you use the 2.0.x branch rather than 3.0.x as no stable release has been cut on the 2.x branch.

bradjones1’s picture

@mglaman is working on this on a few related issues.

mxr576’s picture

bradjones1’s picture

No real significant changes in 3.x, but yeah, this is mostly blocked on bandwidth.

jan.stoeckler’s picture

Status: Active » Needs review
StatusFileSize
new35.1 KB
new4.15 KB

Hey folks,

this is an attempt to port the flysystem.module to Flysystem v2/3 API with as little architectural changes to the module as possible, so basically foregoing the refactoring mentioned in the issue summary as a first step to get something working. Local and FTP adapters are working on my end, as well as the DrupalCacheAdapter. I have not touched any tests yet, so those are very likely to fail.

In case anyone wants to try this out, and/or needs a working S3 integration, there's also an example provider module using the AsyncAws S3 Adapter attached: https://www.drupal.org/files/issues/2023-06-22/flysystem_asyncawss3.zip.

Would appreciate any feedback and/or guidance on if and how to proceed further. Thanks!

(Patch is rolled against 2.1.x, can reroll/change that if appropriate.)

jan.stoeckler’s picture

StatusFileSize
new36.07 KB
new1.02 KB

Added a small fix to __flysystem_check_dependencies to account for updated dependencies.

  • lhridley committed c8a54ba5 on 3.0.x
    Issue #3195832: Converting existing Bridge and Factory to StreamWrapper...

  • lhridley committed 8ed5887c on 3.0.x
    Issue #3195832: Applying partial patch contributed by jan.stoeckler
    
lisa.rae’s picture

Version: 3.x-dev » 3.0.x-dev

  • lhridley committed 3c14b0c4 on 3.0.x
    Issue #3195832: Fixing composer.json typo in package dependency name
    

  • lhridley committed 3fd2715a on 3.0.x
    Issue #3195832: Fix static analysis issues detected with phpstan, remove...
lisa.rae’s picture

Assigned: Unassigned » lisa.rae
Status: Needs review » Needs work
lisa.rae’s picture

Title: Implement 3.x version based on Flysystem v2 API changes » Implement 3.x version based on Flysystem v2/v3 API changes

  • lhridley committed 2cc47f73 on 3.0.x
    Issue #3195832: Removing Ftp Adapter and related code from module, this...

  • lhridley committed 880b9508 on 3.0.x
    Issue #3195832: Convert tests from FlysystemFactory and FlysystemBridge...

  • lhridley committed 53231a2d on 3.0.x
    Issue #3195832: Convert FlysystemFactory to...

  • lhridley committed ff3775f2 on 3.0.x
    Issue #3195832: Fixing static analysis issues with AssetDumper
    
lisa.rae’s picture

Issue summary: View changes
lisa.rae’s picture

Issue summary: View changes
lisa.rae’s picture

Issue summary: View changes
lisa.rae’s picture

Issue summary: View changes
lisa.rae’s picture

Issue summary: View changes
lisa.rae’s picture

Issue summary: View changes

  • lhridley committed b4b642fa on 3.0.x
    Issue #3195832: Add a FileSystem class for Flysystem that extends Drupal...

  • lhridley committed d3c525d6 on 3.0.x
    Issue #3195832: Stub out FlysystemStreamManager until we determine if we...

  • lhridley committed 9d657f6f on 3.0.x
    Issue #3195832: Adding inline comments for in progress work
    

lisa.rae’s picture

Title: Implement 3.x version based on Flysystem v2/v3 API changes » Implement 3.x version based on Flysystem v3 API changes
lisa.rae’s picture

Title: Implement 3.x version based on Flysystem v3 API changes » Implement 3.0.x version based on Flysystem v3 API changes
lisa.rae’s picture

Category: Plan » Feature request

  • lhridley committed f7c534d9 on 3.0.x
    Issue #3195832: Relocated all Adapter related code to the src/Adapter...

  • lhridley committed 05cdc706 on 3.0.x
    Issue #3195832: Converting existing Bridge and Factory to StreamWrapper...

  • lhridley committed 923e372b on 3.0.x
    Isssue #3195832: Correcting coding standards issues easily fixable with...

  • lhridley committed 1d5ff156 on 3.0.x
    Issue #3195832: Typecast  to reduce code complexity in...

  • lhridley committed dbaed501 on 3.0.x
    #3195832: Massive refactoring, including the implementation of Drupal...

  • 526df8f7 committed on 3.0.x
    Issue #3195832: Removed deleted trait from service provider
    

  • 712b1bee committed on 3.0.x
    Issue #3195832: Moved schema for flysystem_local into the module...

  • lhridley committed 49224de8 on 3.0.x
    Issue #3195832: Continues major refactoring, including introduction of...

  • lhridley committed a3d62868 on 3.0.x
    Issue #3195832: Moving compiler into src directory
    

  • lhridley committed 705ff2b4 on 3.0.x
    Issue #3195832: Refatored Adapter Utility Factory for Local Adapter
    

  • lhridley committed 898f0c7f on 3.0.x
    Issue #3195832: Refactored Service Provider to eliminate Compilers
    

  • lhridley committed 3f854206 on 3.0.x
    Issue #3195832: Finalized Service Provider logic, tweaked to use id as...

  • lhridley committed e75d2ffb on 3.0.x
    Issue #3195832: Fixed issues with form rendering and saving
    

  • lhridley committed c38664f5 on 3.0.x
    Issue #3195832: Fixed coding standards and static analysis issues with...

  • lhridley committed d382f0b8 on 3.0.x
    Issue #3195832: Added Flysystem Filesystem Trait class, complete with...

  • lhridley committed 34182053 on 3.0.x
    Issue #3195832: added dependency on flysystem module to flysystem_local...

  • lhridley committed 42758773 on 3.0.x
    Issue #3195832: adding dependency on field module for flysystem
    

  • lhridley committed 28614558 on 3.0.x
    Issue #3195832: Switched service provider to use config.storage.active...