Files uploaded via Drupal to remote storage (S3 in my case) appear in the list of files admin/content/files immediately.

However, all the other already existing files and folders on S3 are not listed in Drupal.

How to make sure all remote files and folders show on admin/content/files?

Comments

Vacilando created an issue. See original summary.

esolitos’s picture

esolitos
he/him
Primary language Italian
Location Trondheim
commented
Status: Active » Closed (works as designed)

This is to be expected, because what you see in admin/content/files is what is in the Drupal database, not what is in the filesystem, the same exact behaviour can be observed in a standard Drupal installation if you manually add files in the public directory (Usually sites/default/files), no file will be shown unless you upload them via Drupal itself.

vacilando’s picture

vacilando commented
Status: Closed (works as designed) » Active

It is attractive to use Flysystem to attach a pre-existing file repository (say a file bank or an image gallery) so that those files can become available to Drupal.

Because of this issue, however, it does not work. It is not practical to have to upload each and every image to be able to browse them through Flysystem.

If this does not work out of the box (as explained by @esolitos, thanks for that), would it make sense for this module to provide a way to synchronize the Flysystem-attached file repository to the Drupal database?

joum’s picture

joum
he/him
Location Porto
commented

I feel that this would be the killer feature of this module.

As it stands, there is no way of letting Drupal know of/manage existing assets in the remote file system. This makes (ie) migrations very hard to accomplish, as it requires two steps: a first step to let Drupal "import" remote files (no import actually needs to happen, Drupal just needs info about them in the DB), and second, a step to create/update the content using those file references.

If there was a way to create/sync those references to remote files automatically, the import of content would be more or less trivial.

Not sure how this could be implemented to work consistently accross different remote file systems, though. Even so, I propose changing the category of this ticket to feature request.

benjifisher’s picture

benjifisher
he/him or they/them
Primary language English
Location Boston area
commented

I did this on a recent migration project. After copying the files from old cloud storage (S3) to new storage (GCS) I generated a list of missing files and saved it as a CSV. I then used this migration to track the files in the file_managed table (i.e., to create File entities):

langcode: en
status: true
dependencies: {  }
id: missing_file
class: Drupal\migrate\Plugin\Migration
migration_tags:
  - 'Drupal 7'
  - Content
migration_group: mymodule
label: 'Files in cloud storage'
source:
  plugin: csv
  path: modules/custom/mymodule_migration/sources/missing-files.csv
  ids:
    - ID
process:
  filename:
    -
      plugin: callback
      callable: basename
      source: URI
  uri: URI
  status:
    -
      plugin: default_value
      default_value: 1
  uid:
    -
      plugin: default_value
      default_value: 1
destination:
  plugin: 'entity:file'
migration_dependencies: {  }

If you want Flysystem to do the work for you, then that would be a feature request.

lisa.rae’s picture

lisa.rae commented

Another potential solution might be the External Entities module.

lisa.rae’s picture

lisa.rae commented
Version: 8.x-1.x-dev » 2.1.x-dev