Unpublished files should be in private storage

In theory I agree that this would make lots of sense. The problem is that idea of "published"/"unpublished" is not common among all entity types but specific to nodes. File handling on nodes is managed by the file field, though, which is entity-type agnostic.

Now node.module *could* alter the file field form and hook into the field behavior, but...

This idea comes up a lot in our D7 projects, but I'm not sure how feasible it actually is. Some of the moving parts are:

1. What do we do to the current (arguably broken) garbage collection that removes files on unpublished nodes?
2. Is it possible to move a file from private to public? (Let's assume that is technically feasible.)
3. How do field settings (e.g. default file field as public/private) affect this behavior?
4. What UI signals do we give to editors about file status?
5. Does that work with remote files (and do we care)?
6. In a scenario where a file can be attached to multiple entities, how do you enforce privacy?
7. When using a WYSIWYG, how is file usage tracked, especially if we have WYSIWYG in core?
8. How do we allow preview of files to the "right" users?

Given the state of D8 feature freeze, I would suggest that the best we can do is allow this behavior to be defined in contrib, which means attacking only the garbage collection and possibly the WYSIWYG in this issue.

Right, and I think a contrib workflow module can handle that if we solve problem 1 (bad garbage collection) and problem 2 (actually moving files from private:// to public://) and possibly problem 7 (WYSIWYG).

Any solution for this? I am looking for a solution in my D8 site.

Our working D8 solution for the mentioned problem:
1) Move public files to private - to be able to manage file access.
2) Add entity_usage module with "Track file/image fields" patch to track images, files too. It works for WYSIWYG files too.
3) Implement your custom file access management based on entity_usage data with hooks: hook_file_access() (hook_ENTITY_TYPE_access) for referenced files and hook_file_download() for WYSIWYG files.

In 3) we are checking entity (node) view access to nodes from which is the file referenced. If at least one node view access is granted, we return AccessResult::allowed(), otherwise AccessResult::forbidden().

If you need tracking non-linked images added with CKEditor Image plugin, add "Add plugin for tracking (non linked) images inserted with WYSIWYG Image" patch.

@szato, your solution works but files in the private file system are expensive. Each file download requires PHP processing. A public file is just served by the webserver or frontend cache and that's a relief for Drupal/PHP

You can combine that with moving files between private and public, so you only need to do access checks as long as there is no public content using them. Not saying it's trivial to do that, at least not in a generic way, but it's... possible.

@Berdir, yes, I was thinking on a solution which combines both. I'm wondering whether such a feature could be made generic enough to create a contrib module.

Maybe, if you check for access on any referenced entity for an anonymous user? And you probably also want to specifically handle media entities, so check the nodes/entities using the media, not just the media entity. Which means you need to depend on the entity_usage module as mentioned above (although possible a combination of both file usage and entity usage, to not require that patch).

@Berdir, it would be great if you have some time to review a first approach at https://www.drupal.org/project/unpublished_file

Not yet done: "check the nodes/entities using the media, not just the media entity" not implemented yet.

EDIT: See #26

OK, I have a complete solution but in an early stage of development, the File Visibility module.

Solution from #24 is abandoned.

Coming from #3087660: Automatically move files to public / private storage, depending on referencing entities' access and #2904842: Make private file access handling respect the full entity reference chain, which lived for quite some time as a dup of this issue.
Maybe the title of this issue is not really perfect. Updating it accordingly (ymmv).

@claudiu, kudos for the file_visibility module!
It looks far more feature-rich and elaborated than my file_access_fix that i created some time ago and roughly corresponds to your unpublished_file POC.

So i am deprecating my module in favor of yours.

Copying some info from #3087660: Automatically move files to public / private storage, depending on referencing entities' access.

I think the new issue title is a little bit blurry now. From a UX point of view a "file" can not be set "(un)published" in Drupal core yet in the same sense like content, but an entity holding a file, like media or nodes with image fields, or nodes with body embedded images can.

There is another issue with unpublished (media) entities holding files not being 100% unpublished in references. Which is another issue in core, but out of the scope of this issue.

From my understanding this issue here is about the file accessibility in connection with unpublished entities referencing the file, where files (not media) keep being accessible even if unused or being part of unpublished content (references). The title should follow this a little bit clearer to distinguish between these both issues. The later isn't probably even not documented yet (I will take a look next days).

Another contrib project not mentioned yet in this scope: https://www.drupal.org/project/file_unpublish

We're dealing with this issue related to meeting digital accessibility requirements related to the DOJ rule. Even though a media entity may be unpublished, or even deleted, the file remains on the server. So if we have for example a lot of non-WCAG 2.1 AA compliant PDFs, and someone goes in there and deletes them from the Media Library, they assume that those files are gone and we no longer have to worry about them. Nope! They're still there on the server. I'm not sure why you would ever want to delete a Media entity but still leave the file on the server? I'm sure there must be a reason for it to be built that way. I'm currently going to be testing the Media File Delete contrib module to see if it will solve part of the problem, but it won't solve the issue of us temporarily wanting to unpublish Media that we may be working to remediate and then later replace it with the remediated file. This would help with us not having to go in and update links with a new file if that file may have references throughout the site.

That's not related to this issue. You're looking for https://www.drupal.org/node/2891902. Note that this will not delete any files that are already unreferenced, but it will delete any files when you delete the medias going forward after switching that setting. And there are modules mentioned in the comments there that can help with existing leftovers.

Is this issue not about having files that are initially in the public directory moved to the private one when the related Media entity is unpublished? That's how I was reading it. I kind of interjected/lead with a separate issue we're also having (which your link seems to probably fix, thanks, I wasn't aware of that setting).

The part that I felt was relevant here was: "... won't solve the issue of us temporarily wanting to unpublish Media that we may be working to remediate and then later replace it with the remediated file. This would help with us not having to go in and update links with a new file if that file may have references throughout the site."

Basically, we want those files to remain on the site, but when we unpublish them they get shifted to the private directory and are inaccessible for anon users, until we can fix them. All the Entity References to them throughout the site would remain (but only work for signed in admins or those w/ permission to see unpublished media).

> Is this issue not about having files that are initially in the public directory moved to the private one when the related Media entity is unpublished?

Fair enough, I overlooked unpublished in the "*unpublished* or even deleted" part. The issue/topic I referenced is deleted.

This kind of makes sense, but there are many really complicated edge cases. For example revisions, if a file is no longer used on the default revision, it would need to be moved to private as well, what if a file is used in more than one place and only some of those are not public? what does public even mean, you can have access control on published media entities. What about direct links to files, files indexed by google and so on.

If you primarily care about documents, the easy solution is to just make the file field on your document media type always private. If you use something like https://drupal.org/project/media_entity_download to avoid direct links to files then requests all go through Drupal/PHP anyway. Of course, you kind of have to do that from the start to avoid some of the issues and some kind of conversion of existing files.