Follow-up of #2801777: Give users the option to prevent drupal from automatically marking unused files as temporary .

We identified a problem there that can result in users now no longer being able to save nodes if they trigger one of the file usage bugs that result in file usage incorrectly reaching 0.

Comments

Berdir created an issue. See original summary.

berdir’s picture

Status: Active » Needs review
StatusFileSize
new810 bytes

This is where it gets fun. The validation error is \Drupal\file\Element\ManagedFile::validateManagedFile(). We are not allowed to reference permanent files without usage, as it was assumed that those have been created by non-file-usage managed things like custom uploads. If we'd use them and then no longer use, we'd end up deleting that file which has an unknown, unmanaged origin.

The problem is that our test somehow triggers one of those weird cases where we lose file usages, probably due to the repeated translations being added, removed and language changed. Then we are now longer allowed to save the node because our own file is no longer being used.

Before this patch, this made the file temporary and then probably on save it made it permanent again, so it kind of worked. Now with this committed, there can be cases where it is no longer possible to save an entity if the contained file has no usages anymore due to one of the known or unknown bugs around file usage.

I think as a minimal fix, we should wrap the references check in \Drupal\file\Element\ManagedFile::validateManagedFile() in a check whether this setting is enabled or not, and only check that if unused files are still marked temporary.

dawehner’s picture

Nice find!

xjm’s picture

Priority: Major » Critical

Users not being able to save nodes sounds critical to me, even if it only happens with certain modules or on certain sites, so promoting this to critical.

berdir’s picture

@xjm: You don't even need special modules for this, according to #2787187: Data loss: Deleting a translation of an entity deletes all file_usage entries for files referenced by the entity , it should be enough to create a node, add a translation, remove that translation again and then trying to edit the original node again. (Assuming no revisions are used)

berdir’s picture

Was looking for test coverage for this, but didn't find anything. The test form has support for existing files, so it was fairly easy to add.

dawehner’s picture

StatusFileSize
new1.79 KB
new2.58 KB

I'll work on the test coverage

Consider the difference between

a) An image disappears on the frontpage, without any user interaction before
b) Someone cannot save a node with a node, which had an file deleted.

Unlike a), b) at least doesn't leave a utterly broken site. B) is also an active process, and as such MUCH easier to figure out. B) also already existed before the other issue.

Given that I still don't understand why this patch was reverted.

dawehner’s picture

Damnit crosspost.

dawehner’s picture

Berdir's test coverage is much better!

+++ b/core/modules/file/src/Element/ManagedFile.php
@@ -406,7 +406,7 @@ public static function validateManagedFile(&$element, FormStateInterface $form_s
-          if ($file->isPermanent()) {
+          if ($file->isPermanent() && \Drupal::config('file.settings')->get('make_unused_managed_files_temporary')) {

We need some form of documentation here ...

The last submitted patch, , failed testing. View results

The last submitted patch, , failed testing. View results

The last submitted patch, 7: 2896480-test.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 7: 2896480-6.patch, failed testing. View results

martin107’s picture

Referring to #6

I have the smallest of changes .. which can wait until the next iteration.

Some of the new test code is using t('Save') - I think we should simplify those to 'Save'

There is no need to invoke translation in tests unless directly testing translation.

mpdonadio’s picture

Version: 8.3.x-dev » 8.4.x-dev
Status: Needs work » Needs review
StatusFileSize
new2.47 KB
new3.51 KB
new3.02 KB

Should be back to green w/ feedback addressed.

The last submitted patch, 15: 2896480-test-only.patch, failed testing. View results

berdir’s picture

Status: Needs review » Needs work
+++ b/core/modules/file/src/Element/ManagedFile.php
@@ -406,6 +406,10 @@ public static function validateManagedFile(&$element, FormStateInterface $form_s
         if ($file = File::load($fid)) {
+          // Issue @link https://www.drupal.org/node/2896480 @endlink disabled
+          // unused, but permanent, files from being garbage collected. We
+          // need to check the status here to see if we should check the
+          // reference count.
           if ($file->isPermanent() && \Drupal::config('file.settings')->get('make_unused_managed_files_temporary')) {

"need to check the status" is IMHO unclear, we're not really checking the status of anything, we're checking a configuration setting. Also, permanent files are never gargabe collected, only temporary files, the point is that we no longer mark them as temporary.

We also have a comment at the top of the method that explains this part, but it's further up and seems to apply to the whole thing when it actually doesn't, it is the explanation what this block is doing, not the part about a file existing.

My suggestion would to move the existing comment to this position and and update/extend it:

When files that are no longer in use are automatically marked as temporary (now disabled by default), it is not safe to reference a permanent file without usage. Adding a usage and then later on removing it again would delete the file, but it is unknown if and where it is currently referenced. However, when files are not marked temporary (and then removed) automatically, it is save to add and remove usages, as it would simply return to the current state.

xjm’s picture

Issue tags: +8.4.0 release notes

We need to mention this issue in the release notes alongside #2801777: Give users the option to prevent drupal from automatically marking unused files as temporary until a fix is committed here.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mpdonadio’s picture

Version: 8.5.x-dev » 8.4.x-dev
Assigned: Unassigned » mpdonadio

Will try to post an update tonight.

mpdonadio’s picture

Assigned: mpdonadio » Unassigned
Status: Needs work » Needs review
StatusFileSize
new4.51 KB
new2.18 KB

I read this several times, and think @Berdir's explanation makes sense after the moved comment; just fixed a type in his text. Also added a @see to the CR (which is still unpublished).

berdir’s picture

Status: Needs review » Needs work
+++ b/core/modules/file/src/Element/ManagedFile.php
@@ -398,15 +398,23 @@ public static function preRenderManagedFile($element) {
+          // it is not safe to reference a permanent file without usage. Adding
+          // removing it again would delete the file, but it is unknown if and
+          // where it is currently referenced. However, when files are not

I think you lost a part of my suggestion here between "Adding removing .."

mpdonadio’s picture

Status: Needs work » Needs review
StatusFileSize
new4.54 KB
new1.48 KB

Yup. Must have chopped it off when flowing to 80 cols.

berdir’s picture

Looks good to me, but not sure if I can RTBC as I wrote most of the patch myself. @dawehner? :)

dawehner’s picture

Status: Needs review » Reviewed & tested by the community
+++ b/core/modules/file/src/Element/ManagedFile.php
@@ -398,15 +398,23 @@ public static function preRenderManagedFile($element) {
+          // If referencing an existing file, only allow if there are existing
+          // references. This prevents unmanaged files from being deleted if
+          // this item were to be deleted. When files that are no longer in use
+          // are automatically marked as temporary (now disabled by default),
+          // it is not safe to reference a permanent file without usage. Adding
+          // a usage and then later on removing it again would delete the file,
+          // but it is unknown if and where it is currently referenced. However,
+          // when files are not marked temporary (and then removed)
+          // automatically, it is safe to add and remove usages, as it would
+          // simply return to the current state.

Just some random rambling: Sometimes it feels like its easier to express these kind of technical problems using some kind of ascii graph / description of possible states instead of a long text like this. Anyway, this is just a sitenote. This text is totally understandable

xjm’s picture

  1. Sometimes it feels like its easier to express these kind of technical problems using some kind of ascii graph / description of possible states instead of a long text like this.

    Well, this assumes that the person reading the code has the facility to interpret the ascii spatially (sight, spatial reasoning, possibly LTR native language...). For something like a list, the API module can render it semantically, but not if the spaces are a spatial map. In those cases we would at least want the information to be available as parseable text as well.

    In this particular case a list or numbered list could do the trick? The text is good but walls-of-text are best broken up into smaller chunks for readability, generally.

  2. +++ b/core/modules/file/src/Element/ManagedFile.php
    @@ -398,22 +398,23 @@ public static function preRenderManagedFile($element) {
    +          if ($file->isPermanent() && \Drupal::config('file.settings')->get('make_unused_managed_files_temporary')) {
    

    \Drupal::config() here makes it backport-safe. Do we have a followup to add the service?

  3. +++ b/core/modules/file/src/Element/ManagedFile.php
    @@ -398,15 +398,23 @@ public static function preRenderManagedFile($element) {
    +          // @see https://www.drupal.org/node/2891902
    

    Thank you for referencing a CR instead of an issue ID.

Neither of 1-2 is particularly worth blocking commit on given the severity of this issue, so leaving RTBC for the moment.

  • xjm committed fdef855 on 8.5.x
    Issue #2896480 by mpdonadio, dawehner, Berdir, martin107: Allow to...
xjm’s picture

Status: Reviewed & tested by the community » Fixed
Issue tags: +Needs followup

Normally I might not credit the reasonably small review in #14 but this is definitely something that we want to fix in the long term. Good spot, thanks @martin107!

Going ahead and committing this with the "honor system" for followup(s), given that it would be good to have this in the beta for testing with the file usage change.

We might want to backport this to 8.3.x along with #2801777: Give users the option to prevent drupal from automatically marking unused files as temporary -- the two together might be worth a final 8.3.x patch release. However, that patch should be combined with this one if so.

  • xjm committed 8bee048 on 8.4.x
    Issue #2896480 by mpdonadio, dawehner, Berdir, martin107: Allow to...
mpdonadio’s picture

Is anything really needed for an 8.3.x version? It applies cleanly (I checked), so we can just cherry-pick, depending on what we do with #2801777: Give users the option to prevent drupal from automatically marking unused files as temporary .

mpdonadio’s picture

And re #26-2, since preRenderManagedFile() is a static method, we can't use DI. Or did you have a different idea? (and thanks to @Sam152 and @ larowlan for reminding me about this).

xjm’s picture

@mpdonadio, we might as well include it WITH the 8.3.x version (see the other issue for details). They are for release together and the other isn't backported yet.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.