module_load_include() currently uses the following code to include php files:

  if (is_file($file)) {
      require_once $file;
      return $file;
  }
  return FALSE;

This is redundant, as the only difference between require_once() and include_once() is the behavior if the file cannot be included. Using @include_once will suppress any php warning. The return value of include_once() can be used to determine if the file was included successfully.

  if (@include_once($file)) {
      return $file;
  }
  return FALSE;

In addition, two things are bad about the current solution:

1.) In a production environment, Drupal will commonly be run using a php opcode cache like apc. Using the "stat = 0" configuration, apc will no longer check whether a known php file still exists or has been changed, giving a good speed boost, especially if the code files are located on a shared network drive. Using is_file(), however, forces php to do a file system check, negating this effect.

2) is_file() does not necessarily catch all situations in which require_once() will fail. Some php configurations will not execute files, for instance, if the execute right bit of the file is not set.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

bleen’s picture

Status: Active » Needs review
FileSize
491 bytes
bleen’s picture

Issue tags: +Performance
RobLoach’s picture

#1: 1524630.patch queued for re-testing.

RobLoach’s picture

Status: Needs review » Needs work

Yup! I grepped through and found one more instance where this pattern could be replaced....

In form_get_cache(), you can see:

          elseif (file_exists($file)) {
            require_once DRUPAL_ROOT . '/' . $file;
          }
RobLoach’s picture

Status: Needs work » Needs review
FileSize
1.06 KB
ralf.strobel’s picture

Can we flag this for a quick backport to D7, once all tests are passed? I could really use this in a productive system.

Also, here is another quick fun fact about include_once(), in case someone sees use for it:
The first time, when a file is actually included, it returns int(1). When called a second time it returns bool(true).

RobLoach’s picture

Issue tags: +Needs backport to D7

First step is to get it into Drupal 8. So, test it out, and RTBC it up! Then we can D7 backport it.

Martin Ringehahn’s picture

Please note that @include/@include_once also suppresses any notice generated within the included file.

ralf.strobel’s picture

I did not know that and it does sound like a problem...

Is the @ even necessary in the first place? How typical is it for module_load_include() to be called on a non-existing file? If it should never happen, maybe having the php warning thrown is in fact the desired behavior.

ralf.strobel’s picture

Title: replace is_file() + require_once() with @include_once() for apc performance » replace is_file() + require_once() with include_once() for apc performance
Category: task » bug
Status: Needs review » Active

Ok, maybe I can restart the conversation here...

I would still argue that a warning makes sense in case module_load_include() is invoked on a non-existing file.

If that is not the case and these errors should be ignored silently, then we should intercept the warning and react to it. Since Drupal already registers a custom error handler during bootstrap, this could happen through a modification of _drupal_error_handler_real()...

class IncludeException extends ErrorException {}

function _drupal_error_handler_real($error_level, $message, $filename, $line, $context) {
    //...
    if ($error_level == 2 && substr($message, 0, 7) == 'include'){
        throw new IncludeException($message, 0, $error_level, $filename, $line);
    }
    //...
}

Then module_load_include() could catch this custom exception class. Of course this would also possibly impact the behaviour of module code if they use include() directly.

Another alternative, without modification of the Drupal error handler, would be to use a temporary override:

function module_load_include($type, $module, $name = NULL) {
  if (!isset($name)) {
    $name = $module;
  }

  if (function_exists('drupal_get_path')) {
    $file = DRUPAL_ROOT . '/' . drupal_get_path('module', $module) . "/$name.$type";
    $file_exists = true;
    
    //PHP 5.3 closure makes this easier, but could be written backward compatible
    set_error_handler(function ($error_level, $message, $filename, $line, $context) use (&$file_exists){
        if ($error_level == 2 && substr($message, 0, 12) == 'include_once'){
            $file_exists = false;
            return true;
        }
        return _drupal_error_handler_real($error_level, $message, $filename, $line, $context);
    });
    
    include_once $file;
    
    restore_error_handler();
    
    if ($file_exists) {
      return $file;
    }
  }
  return FALSE;
}
casey’s picture

Totally agree on dropping is_file(). What about try/catch?

ralf.strobel’s picture

Try/Catch only works for exceptions, not errors. These are separate concepts in PHP (although unifying them is being discussed). Errors can only be intercepted through set_error_handler().

include() triggers an E_WARNING type error, while require() produces an E_COMPILE_ERROR which is fatal by definition and ends script execution immediately.

Like I said, Drupal already registers an error handler which reroutes all warnings to the dblog. So the worst outcome of using include_once() without @ would be that more warnings would end up in the log, whenever a module tries to include a non-existing file.

As long as we still evaluate the return value of include_once(), the return behaviour of module_load_include() should remain exactly the same.

ralf.strobel’s picture

Just did a little test with some Drupal installations that have a lot of third party modules installed...

Sadly, it looks like it is not a rare event that modules try to include non-existent files.

So, yeah, that basically leaves me with my solutions proposed in #10.

almaudoh’s picture

Just asking...
What's the performance impact of #10? Doesn't it negate whatever gains we obtain from dropping is_file()?

ralf.strobel’s picture

Everything in #10 is happening within PHP, so the impact should be negligible.

You are probably right though, that on a regular installation within a local file system, the performance gain will also be minimal, if the underlying operating system has the requested file system entries cached.

The real reason why I opened this issue in the first place was to avoid file system checks in case of a remote shared file system installation, because these are very costly.

ralf.strobel’s picture

Status: Active » Closed (won't fix)

PHP 5.5 now includes the new Zend OPcache by default, while APC is being discontinued. It's also available for earlier versions of PHP.

There is now a config option "opcache.enable_file_override", which makes functions such as is_file() skip the file system access if the requested file is a script which has been cached.

Since this is a suitable solution for the future and a legacy workaround within Drupal seems to be too complicated, I am now closing the issue. Thank you all for your input anyway!

pounard’s picture

#5 patch is OK but the @ operator should be removed from calls, a non existing file is synonym of a real user error and should not be silent IMO.

pounard’s picture

Status: Closed (won't fix) » Needs work

I don't agree this should be closed, most environments today are NOT 5.5.

catch’s picture

Also this is still a valid fix for sites where "opcache.enable_file_override isn't set. I agree with removing the @ as well - the PHP warning is a good thing.

RobLoach’s picture

Priority: Normal » Minor
Status: Needs work » Needs review
FileSize
1.05 KB

Status: Needs review » Needs work

The last submitted patch, 1524630.patch, failed testing.

pounard’s picture

There is an interesting measure in #1978592: Free Drupal 7's module_load_include() from is_file() calls (comment #15) (see #2077375: Performance issues) that states that using a static cache is actually better than leaving APC do the job. I'm not convinced since I did have surprises when using a profiler (some operations are slower due to observation) but it might be interesting to do raw benchmarks with and without a static arround module_load_include() not using such profiler.

Not using the @ operator needs further code fixes, since it will raise notices when file is not found: this why the tests will fail and site won't install with D7 version. It needs the upper layer to be fixed to never try to load a non existing file, which will make the patch much more extensive (hence the @ I used in #1978592 in order to keep the patch smaller for D7).

Nitesh Pawar’s picture

Issue summary: View changes
FileSize
534 bytes

Hello,

I have tried to solved it. Hope it will works as expected.

Nitesh Pawar’s picture

Status: Needs work » Needs review
pounard’s picture

if (is_file($file) && is_readable($file)) {

Won't fix any performance problem, it does an explicit stat operation on the file system due to is_file(), and maybe one more if not cached due to is_readable().

RobLoach’s picture

Another question is why is Drupal trying to include files that don't exist in the first place? If we get to the root of the problem, we wouldn't be getting the exceptions and fails happening in #20.

joelpittet’s picture

I agree with @pounard it's no longer a performance issue but it is a fix for error catching and likely more

So this needs an issue summary update.

joelpittet’s picture

Status: Needs review » Closed (duplicate)

Maybe a better solution, closing as a duplicate of #1443308: Add static cache to module_load_include().

If you still feel this is the right approach feel free to re-open.