Problem/Motivation

Drupal heavily relies on file_scan_directory to discover modules and other stuff.

On Windows systems this discovery is slower when compared to *nix because PHP does not directly consume the native windows API.

The Wincache team at MS has worked on a "reroute_enabled" setting for the wincache php extension that routes PHP's to an internal faster and cached implementation of:

  • file_exists
  • file_get_contents
  • readfile
  • is_readable
  • is_writable
  • is_writeable
  • is_file
  • is_dir

After they fixed this issue, heavy operations suchs as registry rebuilding, cache clearing, enabling/disabling a module or simply accesing the modules UI listing experienced nice performance gains.

But still, file_scan_directory is very easy to reliable cache based on the last modified timestamp. So I spent a few minutes modifying file_scan_directory to implement some caching over it with good results such as the module's page loading inmediately (it can get slow on sites with > 500 modules - even if not enabled) and other operations such as enabling/disabling modules or clearing the cache have had good performance boosts.

Obviously, the performance gains depends on the number of files in the filesystem. In D8 this has grown, and in D7 this can be very big if you have many contrib modules. This cache just makes file_scan_directory more scalable.

Proposed resolution

Implement a cached version of file_scan_directory based on the directory last modified timestamp.

Remaining tasks

User interface changes

None

API changes

The original file_scan_directory looses it's depth parameter, but it was only used for the internal looping.

CommentFileSizeAuthor
cache-file-scan-dir-d7.patch2.23 KBdavid_garcia
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Status: Needs review » Needs work

The last submitted patch, cache-file-scan-dir-d7.patch, failed testing.

david_garcia’s picture

Issue summary: View changes
cilefen’s picture

@david_garcia This is certainly important on sites with lots of modules. Before you proceed, please check to see if there is not already an issue open.

This could be a duplicate of #546584: Modules or themes with too many files kill drupal_system_listing performances, and also note that issues should be opened in the highest version of Drupal on which they apply. If this is actually an 8.x issue, it should be opened on that version. If it is fixed in 8.x, check for issues needing back-porting (possibly #1081266: Avoid re-scanning module directory when a filename or a module is missing).

david_garcia’s picture

Status: Needs work » Closed (duplicate)

Duplicate of #546584: Modules or themes with too many files kill drupal_system_listing performances.

The errors of the testbot are due cache backend being called before DB is even available. I've only tested this on memory cache backend (APC/Wincache).

And I also found that the patch needs some modification because filemtime is not reliable (at least on windows) because it does not detect changes in all the directory hierarchy.