Amazon supports publishing S3 events to an SNS topic, which can then push to an arbitrary HTTP(S) endpoint. In our case, we have objects set to expire after 2 weeks, and we want to be able to clean up our Drupal file entities when they are deleted. However, this could also support uploading an asset through an S3 client, and then pushing the file entity to Drupal automatically.

This could be a separate module, with plugins in flysystem_s3 to react to the events, but I'm going to start this as a pure flysystem_s3 patch for now just so I can get it working.

CommentFileSizeAuthor
#2 2779231.2-amazon-sns.patch4.02 KBdeviantintegral

Comments

deviantintegral created an issue. See original summary.

deviantintegral’s picture

deviantintegral commented
StatusFileSize
new 2779231.2-amazon-sns.patch4.02 KB

Here is a proof-of-concept patch that works, cleaning up S3 files when they are deleted by another client. For example, I can delete an image, and then the site will delete the file entity and image styles that have been generated.

I also realized SNS support could be a nice performance improvement with caching. The big issue with caching is that it doesn't support multiple clients well. If SNS sends all notifications to a site, then you could enable caching and be super aggressive about what you store.

deviantintegral’s picture

deviantintegral commented

For an entirely different reason, we ended up creating an SNS module: https://www.drupal.org/project/amazon_sns

If this gets looked at again, it should use the events fired by that module instead of implementing this all from scratch.