Drupal 7 abysmally slow: file_scan_directory called > 1000 times per page load

I can't offer anything technical (above my head at this point), but I can confirm the drastic speed difference between 6 and 7. I would love to see whatever you come up with - Drupal 7 is far too slow for me to recommend on simple shared hosting environments (my typical small client).

Nice research, Chris. I am new to Drupal and can't offer any advice, but would encourage you to file a bug/issue ticket (I guess it is done at http://drupal.org/project/issues).
It's frustrating that in the various Drupal 7 Speed posts I've seen, nobody with knowledge of the project has been able to say whether this issue is being looked at, or recommend any settings that a host provider could change to help with the speed problem, or provided any type of feedback at all.

I have the same issue with my site with durpal 7, and I'm still investigating.
In your case you are you using big images and that make the browsing slower.

Trying to figure out this one too. Some detailed discussion in another post (I'm not technical enough to understand whether these are related).

Here's the other thread: http://drupal.org/node/1018126

My Drupal 7 install is on Bluehost (shared hosting), which I understand isn't optimal. And I can confirm that it gets throttled. What scares me is that: I am the only one who even knows it exists. The thread above points to index.php demanding lots of resources.

Just posting this to subscribe - thanks for pushing the work to figure out what's going on.

I don't know if you noticed, but your site doesn't even come up - just a 505 error page. Anyway, my drupal 7 install is so slow, that basic admin requests die due to time-out and my pages won't even come up half the time!

-Otto

I have run into the exact same problem.
I nearly sent on the login details to my client for proofing when my site stated to take 60 seconds to load. Then it timed out at 120 second and threw the following error:
Fatal error: Maximum execution time of 180 seconds exceeded in E:\inetpub\wwwroot\XXX\includes\file.inc on line 1992

After pulling my hair out all afternoon i decided to poke around in that core file to see what was going on.

This is the function that was causing the error:

function file_scan_directory($dir, $mask, $options = array(), $depth = 0) {
  // Merge in defaults.
  $options += array(
    'nomask' => '/(\.\.?|CVS)$/',
    'callback' => 0,
    'recurse' => TRUE,
    'key' => 'uri',
    'min_depth' => 0,
  );

  $options['key'] = in_array($options['key'], array('uri', 'filename', 'name')) ? $options['key'] : 'uri';
  $files = array();
  if (is_dir($dir) && $handle = opendir($dir)) {
    while (FALSE !== ($filename = readdir($handle))) {
      if (!preg_match($options['nomask'], $filename) && $filename[0] != '.') {
        $uri = "$dir/$filename";
        $uri = file_stream_wrapper_uri_normalize($uri);
        if (is_dir($uri) && $options['recurse']) {
          // Give priority to files in this folder by merging them in after any subdirectory files.
          $files = array_merge(file_scan_directory($uri, $mask, $options, $depth + 1), $files);
        }
        elseif ($depth >= $options['min_depth'] && preg_match($mask, $filename)) {
          // Always use this match over anything already set in $files with the
          // same $$options['key'].
          $file = new stdClass();
          $file->uri = $uri;
          $file->filename = $filename;
          $file->name = pathinfo($filename, PATHINFO_FILENAME);
          $key = $options['key'];
          $files[$file->$key] = $file;
          if ($options['callback']) {
            $options['callback']($uri);
          }
        }
      }
    }

    closedir($handle);
  }

  return $files;
}

the if statement was causing me the problems. I have commented it out for the time being so I can use my site. The function now looks like:

function file_scan_directory($dir, $mask, $options = array(), $depth = 0) {
  // Merge in defaults.
  $options += array(
    'nomask' => '/(\.\.?|CVS)$/',
    'callback' => 0,
    'recurse' => TRUE,
    'key' => 'uri',
    'min_depth' => 0,
  );

  $options['key'] = in_array($options['key'], array('uri', 'filename', 'name')) ? $options['key'] : 'uri';
  $files = array();
  if (is_dir($dir) && $handle = opendir($dir)) {
    while (FALSE !== ($filename = readdir($handle))) {
     if ($depth >= $options['min_depth'] && preg_match($mask, $filename)) {
          // Always use this match over anything already set in $files with the
          // same $$options['key'].
          $file = new stdClass();
          $file->uri = $uri;
          $file->filename = $filename;
          $file->name = pathinfo($filename, PATHINFO_FILENAME);
          $key = $options['key'];
          $files[$file->$key] = $file;
          if ($options['callback']) {
            $options['callback']($uri);
          }
        }
      }
    }

    closedir($handle);
  }

  return $files;
}

removing this snippet of code

 if (!preg_match($options['nomask'], $filename) && $filename[0] != '.') {
        $uri = "$dir/$filename";
        $uri = file_stream_wrapper_uri_normalize($uri);
        if (is_dir($uri) && $options['recurse']) {
          // Give priority to files in this folder by merging them in after any subdirectory files.
          $files = array_merge(file_scan_directory($uri, $mask, $options, $depth + 1), $files);
        }
        else

Anyone know what this function is doing and a better solution for me to get my site running again?

This script is realy helpfull to fix the performance issue!

for me none of the solutions above worked, but then clearing my site cache magically fixed the problem! try it out!

Walker,

Newbie here using the forum to try and get my own help with installation of Commerce Kickstart 2 demo store. Saw an update to your posting. Got curious and followed the link.

Your site popped up amazingly fast ... and the photography is refreshingly breathtaking and incredibly beautiful. Whatever web problems you think you are having... I don't experience them. But you are a very fine photographer...that is for sure!

"Tropfen"

We had the same problem, and it turned out to be caused by the open_basedir php ini directive. After disabling open_basedir the website was fast again.

Has anyone found out a solution for this slowness? It really makes the service unusable after spending so much time and money on contractors. Can anyone recommend a comparable system?

We've already re-installed it once thinking something is missing in our installation or configuration.

Since this still comes up in search results: A change introduced in Drupal 7.33 may expose fragility and errors in other modules, even if those other modules were not updated. For example, the current stable release (1.4) of imagecache_actions will cause this effect after Drupal core is updated to 7.33+. Please see #2381357: Wrong module_load_include path slows down whole site.

for the sake of documenting, note that this patch should mitigate/solve this issue: https://www.drupal.org/node/1081266