Change record status: 
Project: 
Introduced in branch: 
8.4.x
Introduced in version: 
8.4.0
Description: 

Before this change

In the following scenario:

  1. upload file
  2. use file, for example in an image field, or embedded in a text field
  3. stop using this file (for example uploading a different file in an image field, or removing the <img> tag in the text field)

… the file was uploaded as temporary (1), used/referenced (2), then no longer used/referenced (3). Because it no longer was in use, Drupal 8 settings would have downgraded the file from "permanent" to "temporary" after. The file is now "orphaned". It then depended on your Drupal 8 site's Delete orphaned files after (temporary_maximum_age) setting, which is set to 6 hours by default.

This is a problem for:

  1. sites that want to maintain a media library
  2. sites that are bitten by one of the several critical bugs wrt "file usage" tracking: there are several scenarios where files do have >0 uses, but due to bugs Drupal is told that there are 0 uses, hence deletes files actively in use. See #2708411: [PP-1] editor.module's editor_file_reference filter not tracking file usage correctly for translations + #2810355: $entity->isDefaultTranslation() behaves incorrectly when changing default translation, causing file/image field usage to be set to zero, causing files to be deleted + #2821423: Dealing with unexpected file deletion due to incorrect file usage

(In code terms: step 3 in the scenario was performed by \Drupal\file\Plugin\Field\FieldType\FileFieldItemList::postSave() for file & image fields, and by editor_entity_update() + editor_entity_delete() + editor_entity_revision_delete() for text fields. They don't actually delete files, they call \Drupal\file\FileUsage\FileUsageBase::delete() which only marks the files as "temporary" when zero references/uses remain. The actual deletion then happens in file_cron(), which deletes all temporary files after 6 hours — or whatever it's configured to.)

After this change

A new make_unused_managed_files_temporary setting exists in file.settings which is set to false by default, for both existing and new sites.

Any file that is "permanent" is no longer downgraded to "temporary" automatically. Therefore it is also never deleted. This change in default behavior is necessary to prevent data loss due to the other bugs (again see #2708411: [PP-1] editor.module's editor_file_reference filter not tracking file usage correctly for translations + #2810355: $entity->isDefaultTranslation() behaves incorrectly when changing default translation, causing file/image field usage to be set to zero, causing files to be deleted + #2821423: Dealing with unexpected file deletion due to incorrect file usage). You can still opt out from this new default behavior, to go back to the old behavior (there is no user interface for going back, but it can be done using Drush or by making a site-specific code change to change the above configuration setting to true).

Impacts: 
Module developers

Comments

rewted’s picture

I've been wondering why deleted images were accumulating on the file system, only to discover this is the intended behavior in Drupal 8 now.

It appears all images have STATUS Permanent, but those USED IN 0 places are not removed. Is the solution to modify file.settings.yml, setting make_unused_managed_files_temporary to true?

Also, moving forward, how is the novice Drupal user expected to remove image from their site. This needs some TLC, no?

imclean’s picture

The module Audit Files helps manage orphaned files.

dmezquia’s picture

Yes, set $config['file.settings']['make_unused_managed_files_temporary'] = TRUE; in your settings.php and that's is it, this for me has worked!

This will remove the permanent files when they are not in use, depending on the time set in /admin/config/media/file-system

For example when you remove a node that has a image, the image will not remove at once, the image will be removed then when cron execute whether make_unused_managed_files_temporary = TRUE.

permanaj’s picture

Hi @diosbelmezquia,

When I set `make_unused_managed_files_temporary` to TRUE, only 1 file is changed to temporary,
the other 0 usage permanent file still not changed to temporary.

What could be the reason?

rewted’s picture

Comments in this thread will help you. https://www.drupal.org/project/drupal/issues/2821423

permanaj’s picture

Hi @Rewted,

Thanks for pointing me to the comment.
Yes, apparently existing Permanent file with already 0 usage won't be changed to Temporary.
That files need manually removed.

awm’s picture

if one decide to opt out from this new default behavior and set `make_unused_managed_files_temporary ` = TRUE. We are still left with old files that have 0 usage that are still market permanent and may not change their status to Temporary. Are they supposed to change their status? Is this something drupal would take care of? Based on my research I don't see anything that look for files that were marked permenant and set them to temporary based on the value of "make_unused_managed_files_temporary"
For now, do we have to do in a hook update once? or should this be a cron function?

smustgrave’s picture

@awm I believe setting $config['file.settings']['make_unused_managed_files_temporary'] = TRUE; only applies to files going forward. I did see where someone added a query to address previous files (https://www.drupal.org/project/drupal/issues/2821423#comment-13094361).

Have a question if we add that line to our settings.php file what level of concern should there be with files being deleted accidentally. There's so many tickets floating out there about this issue don't want to introduce a new issue.

imclean’s picture

See my comment above. The auditfiles module allows you to remove unused files.

jnicola’s picture

Here's some PHP you can toss in an update hook in favor of the linked css:

/**
 * Move all managed unused files to temporary status.
 *
 * NOTE: Add your module name below, and change "N" to 8001 or above as needed.
 *
 * Implements hook_update_n().
 */
function YOUR_MODULE_NAME_update_N() {
  $query_force_temp_on_unused_managed_files = "
  UPDATE
    {file_managed} AS b1,
    (
      SELECT
        DISTINCT {file_managed}.fid AS file_managed_fid
      FROM
        {file_managed}
      LEFT JOIN
        {file_usage} file_usage_file_managed
        ON {file_managed}.fid = {file_usage_file_managed}.fid
      GROUP BY
        {file_managed}.fid
      HAVING
        (COUNT(file_usage_file_managed.count) = 0)
    ) AS b2
  SET
    b1.status = 0
  WHERE
    b1.fid = b2.file_managed_fid;";

  $database = \Drupal::database();
  $query = $database->query($query_force_temp_on_unused_managed_files);

  // Attempt to execute query, throw error if it failed.
  if ($query->execute()) {
    return t('Successfully flagged all file entities with no value in table file_usage for removal after 6 hours on cron run.');
  }
  else {
    throw new UpdateException('FUNCTION_NAME_HERE Query failed to execute.');
  }
}

matiasmiranda’s picture

the HAVING clause must be SUM instead of COUNT, otherwise the files with usage >0 are marked as status=0.

/**
 * Move all managed unused files to temporary status.
 *
 * NOTE: Add your module name below, and change "N" to 8001 or above as needed.
 *
 * Implements hook_update_n().
 */
function YOUR_MODULE_NAME_update_N() {
  $query_force_temp_on_unused_managed_files = "
  UPDATE
    {file_managed} AS b1,
    (
      SELECT
        DISTINCT {file_managed}.fid AS file_managed_fid
      FROM
        {file_managed}
      LEFT JOIN
        {file_usage} file_usage_file_managed
        ON {file_managed}.fid = {file_usage_file_managed}.fid
      GROUP BY
        {file_managed}.fid
      HAVING
        (SUM(file_usage_file_managed.count) = 0)
    ) AS b2
  SET
    b1.status = 0
  WHERE
    b1.fid = b2.file_managed_fid;";

  $database = \Drupal::database();
  $query = $database->query($query_force_temp_on_unused_managed_files);

  // Attempt to execute query, throw error if it failed.
  if ($query->execute()) {
    return t('Successfully flagged all file entities with no value in table file_usage for removal after 6 hours on cron run.');
  }
  else {
    throw new UpdateException('FUNCTION_NAME_HERE Query failed to execute.');
  }
}
liquidcms’s picture

Do the unused (removed/deleted) files ever get deleted? After setting make_unused_managed_files_temporary I see that files i have removed through the UI are now marked as status=0 in file_managed (even though Audit Files still states the status as permanent). Does this file ever get actually deleted; or is that up to me to write code to do? I would have expected file_cron to clean these up. Possibly only after temporary_maximum_age has expired? As minimum for this in the UI has a minimum of 6 hours (why?); i may have set it to 20 seconds by setting: $settings['temporary_maximum_age'] = 20; in my settings.php (even though there is no evidence this is overriden on the File Settings page (even with Override Warning module enabled).

I don't recall any of these issues in Drupal 7.

donquixote’s picture

This is a problem for:
sites that want to maintain a media library

I don't quite get why this would be a problem.
A media is a separate entity, and a media referencing a file counts as a usage.

raynaldmo’s picture

By default make_unused_managed_files_temporary in file.settings configuration is set to false

drush  cget file.settings
_core:
  default_config_hash: 0aMkoXYnax5_tHI9C9zHs-K48KJ6K75PHtD9x-0nbgM
...
make_unused_managed_files_temporary: false

As discussed in this thread, a file's status will remain "Permanent" even when its usage count is 0.

To change this and set the file's status to "Temporary" when its usage count is 0 and thus get deleted by file_cron (next time it runs, execute the following drush command

drush cset file.settings make_unused_managed_files_temporary true