Integration with Filefield Sources

Yes, exactly. The idea is to get the Amazon S3 filenames from the Commerce File bucket into a table that could then be used by filefield_sources, or something similar.

Ah, you're right. I hadn't thought to check since that is a completely different title. I'd suggest marking the other one as a duplicate, simply because it started out as a somewhat different issue. (Either way is fine, but if we mark this one as the dupe, we should fix the title of the other one to be more accurate.)

EDIT: Upon further review, I think that issue should possibly be kept separate. It sounds like the OP was asking for a batch creation of nodes based on files uploaded to S3, i.e., with no human intervention. For example I could see this use case:

1) Upload a new photo album.
2) Run a batch script in Drupal.
3) A bunch of nodes are created, possibly based on folder structure.

What they're asking for in that case is much more complex than what I've proposed (and what you asked for in that issue) which is a small module that simply populates an autocomplete field with a single filename.

I'm working on this again for our own site. Currently finishing up a "Rebuild S3 Cache" function.

Related issue: #1651096: Adding file metadata to {amazons3_file} fails with filesize > 2GB, use BIGINT instead

Also would like to know! I'm finally restarting work on this need, to try and create a new Filefield Reference widget (as well as modifications to the AmazonS3 module that allows cached files to be referenced by the widget). I'm still kind of learning the Field API so it's a bit slower for me. Things that look like they should work tend not to :/

I'd be happy to check out the code in #19 though!

The thing with Commerce File is it just uses a generic File Widget, really, that's called Commerce File. Since it uses the Field API you can also use Filefield Sources with it. That's what I am basing my modifications on; integrating the AmazonS3 module with File field Sources, and extending it so that the cached files from the S3 bucket are available as a source.

Just FYI I almost have this working. The process will be:

1) Rebuild the AmazonS3 cache. Maybe we can do this on Cron? ¹
2) Make sure "S3 reference" is selected in your Product Type's File Field configuration.
3) Choose the file from the autocomplete field, hit select, the file is migrated into the Commerce File data tables (and attached to the product).

I hope to post a patch (or a tarball, but probably a patch) in the next day or two.

¹ We'll want to see if there's a way to keep the list of files in sync, but I'm not sure how best to handle that. For instance if a file is deleted from the S3 bucket, that may negatively affect files and products, but that may be out of the scope of this first attempt at a feature addition. In any event it's a toss-up between constantly re-syncing during Cron, or making sure this only re-syncs on demand. I mention this because it could be a lengthy process, depending on how many files you have in your bucket.

Okay, here is my first run-through. The patch does a lot, so I will lay it out:

1. Does a bit of whitespace fixing. This was from my editor, so I've left it in.

2. Adds a new file "amazons3_reference.inc" which contains all of the filefield_sources functionality.

3. Adds Batch API hooks (for rebuilding the S3 cache). (These could probably also be split into their own .inc file.)

4. Includes the $e (error code from catch()) in some of the Amazon S3 errors, for verbosity.

5. (And this is major) It changes a bunch of "protected" functions in AmazonS3StreamWrapper.inc into public functions. I had to do this in order to be able to access metadata-fetching functions, for example. I will go through in a future patch, since I'm obviously not using ALL of the functions, and only make the ones that I need as public. Unless no one objects to keeping them all public; I'm not really sure why you'd want reusable functions such as these to be protected, but I'm still a bit of an OOP novice. (I have a @todo in there which states that we could use reflection, basically extending the wrapper class and using some trickery to make those protected functions accessible. We could do it this way as another solution, but I don't think it's quite as nice as simply reusing what's there. Why reinvent the wheel?)

This code is kind of messy as I haven't proofed it yet, and there is at least one section I think needs to be improved (grabbing the filename from the widget field uses a nested array that's all sorts of fugly) but I am short on time so I'm posting this here in the hopes that others can see it and improve it.

Let me know how it works for you; I've tested it on my local stack and it seems to work pretty seamlessly.

Note that this patch requires you to install and enable File Field Sources. (Perhaps this should eventually become its own module, so we can use File Field Sources only as a requirement for it, instead of amazons3?)

Setup:
1. Patch your module (naturally). If you can't, let me know and I'll tar up my module and post it.
2. Clear your caches.
3. Under admin/config/media/amazons3, scroll down and click "Rebuild file metadata cache." Note: at the moment this also truncates your {amazons3_file} table, but I don't suppose that is a huge issue. If it is, you should comment it out in the module; future versions of this patch should try to only add newly-uploaded files.
4. Wait for the Batch process to finish. It will cache all of your S3's files in increments of 100 during the batch process. (I will eventually make this value configurable.)
5. Now in your Product Type's file field setup, add the Amazon S3 Reference as a source.
6. Edit your product, and start typing a filename into the S3 Reference autocomplete textfield.
7. Highlight your file and click Select. Continue uploading or Save your product as necessary.

Let me know if you run into any issues.

EDIT: Use the patch in #31, as it is much cleaner.

That patch was pretty ugly with the AmazonS3StreamWrapper.inc stuff. Let's try a new one.

EDIT: Yikes, this one is even worse. I've tried fixing it in a few different progs to no avail. Sorry :/ For some reason my local install of Git just really does not like that include file. I'll try again tonight from my other machine (that isn't Windows!)

I'll set this as "needs review" just so people can have eyes on it :)

Okay this patch does look better. I've tested and it seems to work, but as always, let me know if there are any problems.

Great, thanks! That's awesome news.

If I understand correctly from what you said above, it will delete the cache and then run a batch of up to 100 files. Am I wrong?

Nope, you're wrong! :) It will cache every file in your bucket, but with a Batch process that only grabs 100 objects at a time. (I want to make this value configurable, too.) So as you watch the cache get rebuilt, you'll see the "currently processing..." number jump by 100 every time the progress bar is updated. I did this because too much higher and you would see a long delay between batches, which looked like an unresponsive website but is really just PHP connecting to the S3 bucket and then looping through its objects.

To test, I used our backup S3 bucket which has ~5000 objects in it, and all were imported with ease.

One issue I did notice, though, that I will fix in another patch, is that files with "S3://" will show up in the autocomplete. A minor thing, but I don't want users to have to see that!

EDIT: I edited my earlier comment for clarity :)

Is the patch the only thing that you changed? Or did you update other modules as well (such as Commerce File)?

If you add another product with a file that you upload, does the license still get granted?

I'm currently looking at #1568240: Entity API Update Breaks License Creation which might be related to this, and which mostly seems to indicate that the latest Commerce File handles its entities and Commerce File fields differently.

Can you tell me what versions of the modules you're using? (Entity API and Commerce File in particular?) Also, if you revert the patch, does the problem go away? I'm currently experiencing "licenses not being granted" with files that have been uploaded through the Upload interface as well as files referenced or imported via Migrate.

It seems that my issue is relating to changes in how Commerce File handles entities. When I upload a file through the Upload interface, I can create a license through the Order Edit page (clicking the "Create" button). But with orders that have Commerce File fields that were imported (and which are hosted on S3) I get the error "No licenses were created."

In your case it sounds like the licenses are being created but you are not getting access to your files. Can you tell me if the access denied error happens at your site, or at Amazon? If it's on the Amazon side, perhaps the file is not being created with proper permissions? Also, are you able to download the file from the Product Edit page? (In other words, once you've attached the file, and the link to the file appears, if you click on it, is that also resulting in an error?)

That's definitely an AmazonS3 error, then, not Drupal, although it could be that you haven't configured your site for private URLs. Try adding an asterisk (*) to the "presigned URLs" at admin/config/media/amazons3. However I have tested it and it does not seem to actually presign URLs, so I think something might be borked in the module itself.

For your testing purposes, login to your S3 bucket with something like Cloudberry or S3Fox, or even the AWS admin, and make the file you'd just uploaded "Public". That way we can make sure that the issue is with your S3 permissions, and in fact requires that the Presigned URLs functionality to work, but isn't.

Back to the helper module situation:

I have the "Reference" field working pretty well, the only issue is it does seem like old files that I have migrated in (via Commerce Migrate) are failing to be passed as real entities, for some reason. New files that I've added via Commerce File to a product seem to work fine and are granted license access, so I'm continuing to see exactly what I'd missed that could be causing such a discrepancy.

Hmm, interesting. Actually I think the patch posted here may be behind my local working version. For now, can you tell me if any errors were returned? (Especially in Watchdog...) if not perhaps we can communicate directly to try and nail this issue down.

Can you tell me anything more about your bucket? Is there anything in the main root of the bucket, or are they all in folders? Etc.

Awesome! I'd be happy to share it. I'm still in the process of working on our major D6->D7 migration, and so the "amazons3_reference.inc" I've created for the Filefield Sources module may continue to evolve, but it seems to work perfectly for our needs. (Though I haven't yet fully tested whether these files are able to be purchased, as is our initial usage with Drupal Commerce.. that's where the testing comes in, I suppose.)

Give me a little bit to post an update!

It sounds like you need to use the -p1 or -p0 option, since patch simply can't find the correct files.

was trying to find a solution where I can upload them directly to S3 and then reference them from the Drupal site.

That's exactly how my solution works. Download and enable Filefield Sources, and then for your content type's file fields you can enable S3 Autocomplete for each. (But first you'd need to rebuild the cache in the AmazonS3 settings page. Once that's built, you'll have a list of files for the autocomplete widget to grab.)

--

I know this patch is needed for a few other people, so I will try to post mine here soon once I've tested it a little bit more. It will most definitely need some extra help, though, as my Filefield Source integration may be a bit janky. Also it still needs the ability to periodically update the Amazon S3 File List (Cache) without user intervention - right now an administrator has to go to the Settings page and click a "Rebuild cache" button, but it'd be more elegant if Cron handled this as well.

Additionally I'd love to have support for not JUST autocomplete widgets (as it stands now) but, if a user only has a few S3 files, a select list as well. Of course I could see a bucket with many "directories" being an issue here, but that could be fodder for some follow-up feature requests.

Is there a way to refresh the list of S3 files being referenced using cron? At the moment when new files are uploaded to S3, to refresh the list in Drupal, I need to rebuild the file metadata cache on the AmazonS3 config page.

At the moment, no - I mentioned this in my comment in #47. This is a feature that needs to come later as it's mandatory, IMO.

Same goes for the other issue regarding subfolders. I'd love to allow more granular control - perhaps even only allowing certain subfolders on a per-field basis, and stripping the folder names out. But this was just a first-draft approach, and as I mentioned, my Field API-foo is not very strong.

Looks great Dave!

Thanks for getting more work done on this. I've been dealing with Commerce issues at the moment and meant to get back to these modules at my next opportunity. Glad to know things are working well - I've had no problems either, in fact it's one of the few parts of our D7 development site that are working without my intervention :)

Nice work, Dave! I also just noticed that patch doesn't alter the .info file to include the Filefield References module as a dependency, when it probably should.

I had thought about it but that would take some refactoring. The ideal solution would not require the filefield reference module, my goal was to keep things in line with how I thought it should be done "the Drupal way." But I admit installing 2 modules when you only really need the one is not great housekeeping.

If you have an idea how to best achieve it, let me know. The filefield reference module does a lot of things under the hood that I was able to leverage (various hooks that allow the widget and things like that) but I don't think it'd be impossible to decouple them.

Well, I think I understand what you're saying but you may be confusing two issues. I like the idea of this functionality living in its own submodule, but as it stands now, that submodule would still require Filefield Sources, as it uses several hook_filefield_sources hooks to create its main functionality.

So the patch could be re-rolled so that instead of creating the amazons3_reference.inc file, it just creates a new sub-module, but as I mentioned that sub-module would still require filefield_sources. Does that make sense?

@destinationsound: You should probably open a feature request that is separate from this issue for that particular functionality.

@deggertson: Does it actually honor those access limits? AmazonS3 hasn't, at least in my testing, which I'll admit hasn't been extensive. We also use "unlimited" for most things so haven't had the need to yet.

Similar, yeah, although I think that issue was more for performance. fsockopen() being called basically runs the download through the browser (your machine acting like a middleman to "fetch" the download from s3 and serve it to the end user) so that's why we wanted to stop going that route. However, it's still perfectly possible to create an authenticated URL after access has been verified - either with the help of a url_alter function or some other kind of redirection that takes place. (For example, clicking a link brings you to example.com/download/filename which checks access and upon success just redirects you with a drupal_goto() to an authenticated S3 url).

Both options I described above seem feasible but we should probably keep the discussion over at the newly-created Issue, #2021997: Honor access limits when serving files through AmazonS3

Yes, in fact I'm working on a way to implement this now, with a custom module and a file widget specifically geared to pulling from the Amazon cache. Unfortunately the Field Widget API is new to me, and file uploads make it particularly tricky. I'll be working on it over the next few weeks.

I'm not doing that because, in our case, we have 5000+ (fairly large) files in our bucket, and querying the S3 API for all of them every time is very slow.

Indeed! I had it working with Filefield Sources and the Reference field there, but that's a couple additional dependencies that we'd (ideally) like to avoid.

Just a little while longer and I'll be able to focus on this again.

I'm finally committing some time to this. Right now I have a widget created with autocomplete lookup, and a "rebuild cache" mechanism that pre-populates the amazons3_file table. I'm right about to dive into the "attach a file object to the node" part. If anyone has tips or a detailed, advanced tutorial, this might take me a bit. I might go into IRC for some pointers as well, because the documentation for this type of thing is kind of sparse.

Working on this again, I will provide a patch ASAP.

I am almost done with integration with Filefield Sources. I had to add an additional hook implementation (hook_file_download) to get around a Commerce File limitation (Related: #2360383: commerce_file_is_licensable() uses wrong approach, breaks compatibility with Filefield Sources). (An added bonus is I was able to figure out how to do what I needed without having to patch the AmazonS3StreamWrapper class and making protected functions public, which is a plus; however I'm interested to know if there is a faster alternative to stream_stat() to get metadata from an S3 object.)

I was originally working on a new plan that was a new "Autocomplete" file widget for File fields, but eventually decided that Filefield Sources is the correct way to go - at least for now, because it abstracts a lot of the File API out and does the majority of the heavy lifting. This means AmazonS3.module doesn't have to reinvent the wheel just to add a file to a product; that being said, I may try to take another stab at a new widget just for AmazonS3 so we don't have to require Filefield Sources as a dependency.

Though, it would probably be a good idea to decouple the "populate the AmazonS3 cache" batch functionality from this patch as that is a function that could be used even if not using S3 as a file field source. In fact I may do that before I post it, because we also need to get Cron integration with that and find a cleaner way to update that table. (To give you an idea, even taking 500 objects at a time, our bucket has 10k+ items in it and it takes some time to populate that table. Either we need a faster solution or a new process that finds only the NEW files in S3 and adds just those to the cache table.)

Will post the patch as soon soon.

Okay, attached are two patches:

1. Is for the "rebuild cache" mechanism. I realized with AmazonS3 the cache table {amazons3_file} includes the s3:// prefix in the uri field. My code reflects that.
CAVEAT: I have included a call to stream_stat() with each object that is fetched from the bucket. This is slow and we need to find a higher-performing method of retrieving file metadata, such as filesize, during this process. (I also wanted to try and NOT hack the stream wrapper file..)

2. This one is for the rest of the Filefield Sources integration. I put it in a new "includes/" folder for better separation, and called it filefield_source.inc just to give it a little more distinction. You'll probably want to delete the old amazons3_reference.inc file from previous patches.

Also I would like to make the theme a bit better as we probably don't need the "s3://" prefix to show in the textfield when you type into it or choose a file.

Please test this by rebuilding your AmazonS3 cache and re-selecting the AmazonS3 source in your file field settings.

Setting to Needs Review.

Changing the title since the issue now has a clear focus. Since I've gotten relatively comfortable with how the API works, if I end up writing an autocomplete field widget that does not require Filefield Sources I will make a separate issue for that.

No, it won't mess anything up. The only thing that may have changed is the name of the Amazons3 reference field, because I tried following the patterns of Filefield Sources more closely (with regard to naming conventions, etc.) It won't mess up any existing entity relationships, because at the end of the day, file fields are just an array attached to an entity, and since those exist, nothing will break in that sense. The only thing changing is the process (behind the scenes) for adding new files to entities going forward.

The only things you'll need to do if you're coming from an old version are:

1. Rebuild your AmazonS3 cache. (You want to make sure they all have an "s3://" prefix, since that is how AmazonS3's own "use database caching" mechanism stores them in {amazons3_file}, and I wanted to make sure we didn't have dupes in the table.)

2. You will probably need to re-add AmazonS3 as a source in the file field for your entity.

Of course if you are going to try this on a production site I would back up your databases before testing; that being said, I'm going to be using these patches with AmazonS3 and Filefield Sources on our live production site immediately because we need it to work correctly. :) See my related issue in comment #92 - without the hook_file_downloads() implementation, a "new" file can never be added to a Commerce File field. With this patch it now works perfectly.

I also took the time to fix some mb_strlen errors that were popping up when Drupal was attempting to run translations on arrays. I pretty much did a huge cleanup of this patch and reworked it to be what I consider relatively solid. Please test it on a dev site and like I said, backup your databases!

Apparently I'm wrong about one thing: the {amazons3_file} table does not include the s3:// prefix. I'm rewriting my code to reflect this. The confusion came from the fact that I was doing a db_merge as well as a stream_stat(). Running stream_stat() does place the file into the table if you have database caching enabled. Enabling the db caching in AmazonS3 is a requirement for this integration with Filefield Source to work, but I'm not sure how obvious that is.

Anyhoo - I will post an updated patch asap.

Finding some issues with the patches I generated... using wrong naming scheme and stuff. Not sure how since the local copy I was working from was the most recent... I may have accidentally applied an old patch to mine and that got included in the new patch. Sorry about that !Setting this back to Needs work while I clean it up and will re-post it later.

Yeah, see my comment above. Patch might not be 100% so I'm rewriting it a little bit. Hang tight.

I'm much more confident about this patch! I decided to combine them both again for ease of application. To test:

1. Backup your database!
2. Apply the patch.
3. Rebuild your AmazonS3 cache @ admin/config/media/amazons3*
4. Configure the "File source" settings for your content types.

Now when you edit a node with this field you can type the filename (or subfolder) and click "Select". Then you can add the description (if configured) for the file. We do this on our Commerce File products and use the description as the link text along with the filesize.

Please test it out and let me know if you run into issues applying it. I've been testing it like crazy and I'm pretty confident with it. Got rid of a bunch of cruft, too.

* NOTE: This might take awhile, as we run a stat() on each file. Buckets with tens of thousands of files might take a while - we need to find a faster solution for this step.

Ah, so for some reason my patch didn't include the new file. Weird. Had to use the --staged flag which seems to help.

Sorry about that! Try this patch instead.

Not a bad idea. I'll move it to its own project and either re-roll the patch or link to it here.

Ah so it seems to me an issue with creating a new product. Let me test that and work on a solution. I'm going to be traveling today too so that might make things a bit difficult - and we have to release a new product today. Yay! The timing couldn't be worse! :)

Thanks for testing. I'll be back with you shortly (with a new module!).

Very very soon! I promise! I'm working on making the file discovery faster. Had to read a bunch of S3 documentation, and then I was traveling on business this week. I'll pick it back up and try to put it to bed over the weekend. I just used it to release a few products today so it is working pretty well :)

Yes! It's still being worked out. I'm almost done (I managed to get the "rebuild cache table" stuff working and much faster than it was before) but I still need to fix the error @deggertson was facing.

Sorry for the delay, my day job has been stretching me thin lately. I will post it soon, I promise.

I'm going to post it to drupal.org as a sandbox project in the next day or so.

Created a new module and project page here: https://www.drupal.org/project/amazons3_ffs

I went with "amazons3_" for the module prefix because some of the naming conventions used by Filefield Sources caused a minor conflict. This way seemed easier (plus I had anticipated making this module a sub-module of AmazonS3).

Anyways please feel free to test and make note of my TODOs. I will be creating some tasks so I can handle adding the few features that are currently not very robust (for instance S3 prefixes and the settings for Filefield sources are not yet worked out).

I'll just set this to Fixed so people see the green :)

Please submit any issues to the issue queue over at the new module's page. The -dev gz file should be ready soon.

Yes, I should probably mention that the latest -dev of AmazonS3 is recommended.

Glad to hear you're having good results so far. I put a ton of work into this, even though it's a relatively small module! Now that it's out in the wild hopefully we can get more hands on deck to help continue development. Thanks Dave!