We ran into an issue today on our university Drupal instance where one site enabled Devel query logging to debug some slow queries in the morning. By the afternoon, /tmp was full (2GB) and other sites could no longer upload images because of /tmp/devel_querylog. I know Devel query logging is not supposed to be used for long, but it didn't seem to clean up after itself and it was necessary to delete files in /tmp/devel_querylog to regain space.

Is Devel supposed to clean up after itself? Thanks in advance.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

ottawadeveloper’s picture

We just encountered the same problem - I think it's simply that the log files are growing too fast (it's only programmed to delete 1 time in 1000 at random... which means there's only a 62.2% chance of it actually wiping the file within a given 1000 requests). I have some really big query log files (>50MB) so it's destroying the disk space really fast. It's essential to store these briefly because of the AJAX requests to them so we can't just delete them as soon as they're requested.

My proposal to fix this would be to add some settings to configure the wiping of the query logs. Personally, I'd like to do this by filesize and I think it makes the most sense for most people (that we limit the filesize available to the logging profile).

So something like building the size of the query log directory, then seeing if it's over 1GB (or pick your value perhaps?), then, if it is, order the files by creation time and delete files older than an hour. Letting them configuring the 1GB and one hour value might be useful (or I guess they can strongarm it as long as you use variables). Alternatively, we could delete files until we're below a certain threshold which would guarantee a certain amount of disk space free.

To lower run time, we can do that at random, but much more frequently (and ideally let them control the frequency as well).

Anyone else have any thoughts? I have to fix this in my project anyways (probably on init() or something in a custom module), but I'm perfectly happy to do a fix to patch into devel.

ottawadeveloper’s picture

If anyone else is looking for a way to handle this in the mean time, I put the following into a module:



/**
 * Implements hook_init().
 */
function mymodule_init() {
  $cap_range = variable_get('mymodule_max_cap_range', 15);
  if (rand(1, $cap_range) === 1) {
    $path = drupal_realpath("temporary://devel_querylog");
    $max_size = variable_get('mymodule_max_temp_size', 1024 * 1024);
    $current_size = _mymodule_get_dir_size($path);
    if ($current_size >= $max_size) {
      $mintime = variable_get('mymodule_min_file_age', 60 * 60);
      _mymodule_delete_old_files($path, $mintime);
    }
  }
}

/**
 * Deletes files older than a certain age. Does nothing to directories.
 * 
 * @param string $dir Deletes files from this directory
 * @param int $minage Files must be at least $minage seconds old to 
 *   be deleted.
 * @return boolean FALSE if we were unable to delete from the directory.
 */
function _mymodule_delete_old_files($dir, $minage = NULL) {
  // Protection against stupidly deleting the root directory.
  if (empty($dir) || $dir == '/' || !is_dir($dir)) {
    return FALSE;
  }
  $before = time() - $minage;
  $handle = dir($dir);
  while (($file = readdir($handle->handle)) !== FALSE) {
    if ($file === NULL) {
      break;
    }
    if ($file != '.' && $file != '..') {
      $full = $dir . '/' . $file;
      if (is_dir($full)) {
        _mymodule_delete_old_files($dir, $minage);
      }
      else {
        $mtime = filemtime($full);
        if ($mtime < $before) {
          unlink($full);
        }
      }
    }
  }
  return TRUE;
}

/**
 * Determines the size of all the files contained within $dir in kibibytes.
 * 
 * @param string $dir The directory to determine file size for.
 * @return int|boolean The size, in KiB, of all files within the directory or
 *   FALSE if unable to read the directory.
 */
function _mymodule_get_dir_size($dir) {
  if (!is_dir($dir)) {
    return FALSE;
  }
  $handle = dir($dir);
  $size = 0;
  while (($file = readdir($handle->handle)) !== FALSE) {
    if ($file === NULL) {
      break;
    }
    if ($file != '.' && $file != '..') {
      $full = $dir . '/' . $file;
      if (is_dir($full)) {
        $size += _mymodule_get_dir_size($full);
      }
      else {
        // Convert to KiB to avoid some problems with int32 systems.
        $size += filesize($full) / 1024.0;
      }
    }
  }
  return $size;
}

moshe weitzman’s picture

Checking filesize on every request is too expensive. We could simply reduce the current check to once every 200 requests, or so.

okeedoak’s picture

Same problem here.

dehacker’s picture

Issue summary: View changes

Devel querylog filled 1GB of server tmp partition on Ubuntu 12.04 and crippled the system. Was unable to run 'drush cc all' afterwards. How do we flush or limit the log?

annya’s picture

Version: 7.x-1.3 » 7.x-1.x-dev
Status: Active » Needs review
FileSize
1.83 KB

Checking filesize on every request is too expensive. We could simply reduce the current check to once every 200 requests, or so.

This patch add possibility to change this value via administrative interface.

Cogax’s picture

Status: Needs review » Reviewed & tested by the community

I had the same problem. My tmp/devel_querylog/ dir was 18GB(!) big before i deleted all. I applied patch #6 and decreased the value in the backend to 100. Then the dir size goes up to 1.9GB and now its on 1.2 GB, so i think it works :)

helmo’s picture

+++ b/devel.module
@@ -1205,7 +1205,9 @@ function devel_query_put_contents($queries) {
+  if (mt_rand(1, $range) == intval($range/2)) {

Assuming the random function works ok, this could als compare to 1 instead of $range/2 ... saving a little bit of calculation time.

salvis’s picture

Version: 7.x-1.x-dev » 8.x-1.x-dev
Status: Reviewed & tested by the community » Active

Could this be an issue that should be fixed in D8?

willzyx’s picture

Version: 8.x-1.x-dev » 7.x-1.x-dev

8.x use webprofiler for the query logging nowadays. Back to 7.x

salvis’s picture

Status: Active » Needs review
FileSize
1.89 KB

Re-rolled, integrated #8 (no need to make this overly complicated), adjusted the #description.

  • salvis committed 941389d on 7.x-1.x
    Issue #2091535 by annya, salvis: Devel query log fills up /tmp
    
salvis’s picture

Status: Needs review » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.