Is there a way to add this to an existing site and sync the local file system to the containers?

I tried setting the file system to where the current files are and adding a CDN container, however the Filesystem container does not register the files?

On an audit I can see the files outlined in the extras list, howver on repair it just removes them. Is there any way to add these into the Filesystem container?



jbrown’s picture

Title: Implementation on existing site » Create a module that imports files into a Storage API scheme
Version: 7.x-1.1 » 7.x-1.x-dev
Component: Miscellaneous » Code
Category: support » task

The idea with the class selectors is that the class that a group of files is in can be changed at any time. The files will be migrated with eventual consistency.

However, files that are not yet managed by Storage API cannot currently be migrated to Storage API, i.e. there is no way to change files' scheme from 'public' to a Storage API scheme.

This could be implemented quite simply as an additional module, but it hasn't been written yet.

paul.linney’s picture

Would the best practice for starting a site be to enable StorageAPI with only the Filesystem container in a class. So that once the site is moved to production you can simply add your CDN container?

I think a module that could move all the files into the filesystem container would be really handy to retrospectively introduce it to existing sites.

I'll have look through the module and have a go at coding up a 'add existing files to a container' option.


larsdesigns’s picture

This proposed feature would remove a barrier to entry for adopting CDN use with Drupal and Storage API.

Island Usurper’s picture

What would be involved to make this happen? I'm sure most people will have a lot of files directly under public://, and you can't specify sites/default/files as the root of a filesystem container. So there'd be a lot of file moving. And then you'd have to update file paths in {file_managed}. Is that enough? There are a lot of variables to take care of.

mc0e’s picture

I would *really* like to see some tools for importing files into storage_api after they're uploaded.
I'd like to be able to auto-generate nodes for files found in a given directory on the local system, or perhaps even on S3, with some control over things like taxonomy terms so I can make the newly imported items listable.

I'm scoping out a project which would need to deal with very large files (>2GB), which PHP doesn't do well, so I'm looking at various independent mechanisms for doing the uploads. Trouble is such approaches don't appear to play nice with storage_api.

Island Usurper’s picture

Figured out that you can set sites/default/files as the filesystem container path. I just didn't have write permissions there, somehow. Still not sure that's a good idea, though. Too many unmanaged files would show up in the audit, and then maybe get deleted.

mc0e’s picture

I'm interested to hear if anyone has made any progress on this, or started to map out what's required.

For my purpose, what I want is a tool to do:

Identify any files in S3 bucket which are not currently managed.
For each such file {
   set up the data for managing the file
   create a node associated with the file(
      owned by a specified user.
      initially the node should not be 'published'.

If anyone has pointers on data structures and functions that are needed or useful for this, I'd like to hear about them.

Presumably others will want to do this for their local file system. I see two use cases there, which seem useful, but are not my current concern:

  • Importing files which drupal doesn't otherwise know about from a given directory - similar to above
  • Importing files which Drupal already knows about into storage_api, changing the storage type
jbrown’s picture

note that the audit does invoke a hook when it is dealing with an unmanaged file - see storage_audit.api.php

paul.linney’s picture

Finally I got time to put a module together to move files that reside in an existing site into Storage API.

You can find the module at https://drupal.org/project/storage_api_populate

Still to do a lot of testing on this, but its a start - only took over a year of pondering!


jlscott’s picture

Re #8: The return from the hook call is "ID of selector extra files should become owned by".

How can I find out what selector ID's already exist, so I can choose the correct one, or how can I create a selector if a suitable one does not exist?

Is it possible to specify that only some of the extra files in the container should be imported for management by Storage API?


jonhattan’s picture

Issue summary: View changes
Status: Active » Fixed

I think this issue can be closed because https://drupal.org/project/storage_api_populate solves it.

Perignon’s picture


Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.