On 9/3/08, Drupal (according to GoDaddy) started depositing thousands of 75-meg files within my hosting account at /sites/default/files/tmp/. After not knowing this for a month (because they had no smart way of proactively notifying me of the alarming situation), it totaled 250 gigs in total. GoDaddy sent me a HUGE bill for disk space overage charges. I have been fighting with them tooth and nail about it for the past three days. They have discounted what they say I owe from that huge bill, but I have just had to pay a portion of it.

They say it was my fault (because Drupal did it and not them or some other sort of compromise) and I did have to pay some of it. It feels like robbery.

I have just deleted all the files in that directory. Once I did, more files slowly started appearing again. I then prevented write access to that directory using GoDaddy’s file manager tool. I believe that should disallow files from being written there, but I don’t know if they’ll appear somewhere else, why this is happening in the first place or what other adverse impacts there might be from this. Can you help me indentify why this is happening and how to stop it? GoDaddy says it’s a module in Drupal that keeps creating these 75-meg temp files, but I have no clue.

This was the final text from their tech support about this issue:

After further review it was found that the user application of Drupal itself is creating the files located in the /sites/default/files/tmp. This appears to be related to a third party module component of Drupal itself and it is recommended that you change the permissions for this from the Administrator > Site Configuration > File System for this application that is creating the excessive files.

Comments

styro’s picture

That has a large bearing on figuring out what is happening.

If you don't allow 75MB uploads to your site, then the only thing I can think of is that they are core files left over from Apache crashes - eg some Apache processes hit a 75MB memory limit and crash leaving behind a disk copy of their memory contents. Do the filenames have the word "core" in them?

If that is the case, then it (in my opinion) is your hosts responsibility to configure their set up to not to save them to disk. And they should have been able to tell you what the files were.

You probably have a buggy module installed that causes these problems.

Note: core files are only useful for C programmers to debug crashes in Apache or mod_php. They are of no benefit to the average PHP programmer.

But all that is guesswork, unless you know what the files actually are.

--
Anton

My current Drupal site: a software product site with a knowledge base and stuff.

WorldFallz’s picture

another option is to turn off all contrib modules and start adding them back one by until you find out the culprit. If you're on a production site, make a test copy of it and try it there.

And yes, without knowing what the files are or what they contain, there's not much anyone can do to help you.

===
"Give a man a fish and you feed him for a day.
Teach a man to fish and you feed him for a lifetime."
-- Lao Tzu
"God helps those who help themselves." -- Benjamin Franklin
"Search is your best friend." -- Worldfallz

_
Don't be a Help Vampire - read and abide the forum guidelines.
If you find my assistance useful, please pay it forward to your fellow drupalers.

HollywoodChicago.com’s picture

Full disclosure: I'm a Chicago journalist. While I do feel I have been very wronged by GoDaddy here, I decided this is an important story to tell. This isn't a personal axe to grind. This is a legitimate concern that was caused (or not prevented) either by GoDaddy or Drupal. I don't know which.

So, I just wrote this story and published it on the Huffington Post. You can read the full story here. I'm getting a lot of angry e-mails from people who are siding with me. Another Drupal discussion thread was created here by someone else, too.

To get back to the core of your question, I honestly don't remember the structure of the temp file names. In order to get them, I just reenabled the read/write permissions in that file directory. I just did, and sure enough, files started showing up there again. They trickle in one at a time every minute or so. Here are a couple of their file names (they're around 75 megs each):

tmp_L3PZjP
tmp_Ydlt8L

Any thoughts now that we know that?

Adam Fendelman
Publisher, HollywoodChicago.com

WorldFallz’s picture

i did a code search on the common part of the name ("tmp_") and it pops up multiple times so there's really no way to trace it (i don't supposed there was any recognizable text in them?).

I would turn off all the contribs and see if they stop. if they do you know it's a contributed module and you'll need to enable them one at a time to find the guilty party (and start with any recently added). If it doesn't stop i would reinstall the core drupal files-- something must have gotten corrupted or hijacked.

===
"Give a man a fish and you feed him for a day.
Teach a man to fish and you feed him for a lifetime."
-- Lao Tzu
"God helps those who help themselves." -- Benjamin Franklin
"Search is your best friend." -- Worldfallz

_
Don't be a Help Vampire - read and abide the forum guidelines.
If you find my assistance useful, please pay it forward to your fellow drupalers.

Genbushi’s picture

Are you or someone able to either copy off some of these tmp files, any access logs, or get someone ssh access to take a look? Disable the write access so you don't get hammered again of course, but preserve what you can so someone can take a look at what's going on.

HollywoodChicago.com’s picture

I've deleted all the files, but if I disable write access into that directory, more would start appearing. Would it be helpful if I snagged one of the 75-meg files and put one online?

By the way, shortly after my Huffington Post article went live describing this story, I got a call from the office of the president of GoDaddy. I'm sure you can guess what ensued. I'll be writing a follow up there to describe the whole resolution to the resolution.

Adam Fendelman
Publisher, HollywoodChicago.com

HollywoodChicago.com’s picture

Is it something to do with this at admin/settings/file-system?

Temporary directory:
sites/default/files/tmp
Location where uploaded files will be kept during previews. Relative paths will be resolved relative to the Drupal installation directory.

I now can't upload images to my site because I disabled the write permissions in sites/default/files/tmp. This is a problem. I made a new directory (sites/default/files/previewimages) and set that in the tool above. It started creating the same kinds of big files there, so disabled it. Could this have to do with the image system?

WorldFallz’s picture

are you using any image handling modules? If so, those would seem to be the likely place to start disabling and see if the problem stops. The easiest way to narrow this down is really to selectively disable modules until you find out which one is creating the massive files.

===
"Give a man a fish and you feed him for a day.
Teach a man to fish and you feed him for a lifetime."
-- Lao Tzu
"God helps those who help themselves." -- Benjamin Franklin
"Search is your best friend." -- Worldfallz

_
Don't be a Help Vampire - read and abide the forum guidelines.
If you find my assistance useful, please pay it forward to your fellow drupalers.

Genbushi’s picture

Sounds like it could be anything making use of uploading or writing files to that path; errant module, or possibly someone\something taking advantage of a function to write to that path.
As someone mentioned, try selectively disabling modules one-by-one to see which one causes the behavior to cease. That'll narrow it down.
If you have access to the access logs, or could get those from GoDaddy it'd be helpful.
If you could put one of the files somewhere accessible, someone could wget or scp it down to see if it's anything useful.
Also a breakdown of drupal, modules, apache, php version will be useful at some point if it looks more than just an errant module issue.

HollywoodChicago.com’s picture

I think it's the image system. When I go to /admin/settings/file-system and tell it to use sites/default/files/tmp for the "temporary directory" and enable the write permissions in that directory, my image system works fine but those 75-meg temp files keep appearing.

When I disable write permissions into that directory, my whole image system doesn't work but the 75-meg temp files stop appearing. So, right now, I need to enable write permissions there in order to get images working but then disable the write permissions right after so the temp files can't keep appearing.

The question now is why the temp files are appearing and how to fix that...

HollywoodChicago.com’s picture

Also, I have a dev site that's mostly a duplicate of my live site. What's especially interesting to me about that is the dev site is using a similar temp directory for its images (different from the directory for the live site), but I see something different there.

I see some "tmp_213jWo" files there, but not too many. Many of those files have no size. One is 11.5 megs, but that's it. So, why does that temp directory massively grow on the live site but not really on the dev site?

Then again, I have installed many modules to the live site that haven't been installed to the dev site.

HollywoodChicago.com’s picture

I installed the Drupal module "Meta tags" (for meta tags) and "Meta Tags Node Type" on 8/27/08. The disk space usage started spiking on 9/3/08, according to GoDaddy. That seems like an interesting coincidence since that's the only module I've installed in a while.

As a test, I just tried to disable both modules. Then I enabled the write permissions in that directory. My image system was again working. Then I waited for a while to see if new 75-meg temp files started appearing, and they are. So, that wasn't it. :-(

socialtalker’s picture

but why dont you try doing the same things on another hosting service to test and see if you get the same results.

jwuk’s picture

That sounds like a lot more work than doing what others have suggested, either disabling all modules and slowly adding them back one by one (checking thoroughly at each step) or else successively disabling modules one at a time.

Another reason not to try and reproduce it on a different hosting service is because it may be... different (and hence not reproduce the problem).

styro’s picture

They could be core dumps. eg if your PHP memory limit is around 64MB and Apache uses around 11MB for each child process (which is a plausible size), then core dumps from an Apache/mod_php crash will be around 75MB.

But there are a few things that don't fit. eg the filenames will usually have the word "core" in them, and the way they seem to end up anywhere writable (or where Drupals temp dir is) rather than a specific directory seems a bit odd.

http://httpd.apache.org/docs/1.3/mod/core.html#coredumpdirectory
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#coredumpdirectory

Another (I suspect more likely) possibility is that they are actual failed uploads. PHP will store an upload in the temp dir, and Drupal picks it up from there and moves it to the "files" directory. eg if your upload limits are higher than the combined 75MB Apache/PHP memory limit, an upload of bigger than that could hit that limit and fail causing a HTTP 500 error leaving the first 75MB of temp file in place with Drupal being unaware of the file due to PHP failing and not being able clean it up.

Unless they are actually core dumps, I don't think it is a plain old PHP memory limit issue with a hungry module. That shouldn't create a temporary file anywhere.

To help diagnose, we'll need some more info:

* What is your PHP memory limit? php_info() should tell you this.

* What are the PHP settings for things like max upload size, max post size etc? again this is from php_info().

* What are your upload filesize limits set at inside Drupal?

* Are there any 500 errors logged in your Apache server error logs?

* Does your host use suexec? eg are the uploaded files (ie through Drupal not FTP) in your files directory owned by your user account or a web server user account?

--
Anton

My current Drupal site: a software product site with a knowledge base and stuff.

HollywoodChicago.com’s picture

Anton,

Thank you for much for the helpful feedback. I'm seeking answers to your questions. I'll get back with you about what I found out. I'm not sure how to answer all those questions. I now have a very, very kind and helpful tech at GoDaddy helping me with this.

Because disabling write permissions to the file directory in question makes it so my whole image system doesn't work, it's obviously necessary to keep write permissions enabled there. The problem, though, is that these big files keep getting created there.

So, this GoDaddy tech has created a cron job that automatically deletes all tmp_* files from that directory every 10 minutes. This allows me to keep my image system working while no more than 10 minutes of dumps there can build up. He also has found some other interesting discoveries. He said.

I wasn't able to create the cronjob to delete those files underneath your account so you could see/manage it, but I have created it. It is running every 10 minutes currently. Here is a text file with the outputs we spoke about showing which exact URL was being accessed, including all of the Apache log file entries going along with this access.

It's important to note that the URL does change, but always appears to be a redirect off of hollywoodchicago.com/index.php. The IP addresses are different every time as well. Additionally, I've included a copy of the database dump text file for your review (zipped, to make it much smaller). Please let me know if I can be anymore help to you on this issue.

It does appear to be an issue with Drupal and/or the active module list. Hopefully the folks that wrote Drupal can help find a cause for this activity. Thanks Adam!

He also gave me tmp_MbOYhE.zip as well if that would help anyone. It's about 9 megs.

styro’s picture

On Drupal 5 you can get a simplified php_info() page here:

/admin/logs/status/php

Things to look for (in the local value column):

memory_limit
post_max_size
upload_max_filesize

I see from those logs that your PHP is running as a CGI program rather than via mod_php in Apache, and that it is probably using suexec (it is running under you user account). So it won't be Apache core dumps.

But thinking about it some more, it probably isn't uploads either - those would be accumulating in a different temp dir. The fact they move to whereever Drupals temp dir is, means that it is something Drupal is running that is creating them. And nothing I know if in core acts that way, so it will be a contrib module.

I think the comment suggesting the backup module (I've never used it), is probably on the money.

--
Anton

My current Drupal site: a software product site with a knowledge base and stuff.

HollywoodChicago.com’s picture

Here is some of that information:

memory_limit = not listed there
post_max_size = 8M
upload_max_filesize = 2M (I remember this causing problems in the past and GoDaddy won't change it)

styro’s picture

But it sounds like you've pinpointed the culprit now anyway :)

--
Anton

My current Drupal site: a software product site with a knowledge base and stuff.

awpti’s picture

They are dumps of your database.

I had the same issue in the past, but never got around to investigating it because my client stopped using drupal soon after the issue presented itself.

Check your Backup module - it's either configured wrong or running wild.

Go and download one of the files and open it. I bet you it's a backup of your DB.

Not a host issue, just a drupal/module problem.

HollywoodChicago.com’s picture

The GoDaddy tech confirmed with 100 percent certainty that the content of the big data is definitely MySQL database content. I do know I've had problems with two separate Drupal backup modules in the past. I've never got them working properly.

One is at admin/content/backup and the other is at admin/content/backup_migrate. The admin/content/backup doesn't appear to have been running or have a schedule to run.

That Backup module does have a link to my phpinfo() data. There's a ton of information there, though.

Now that Backup and Migrate module was set to backup every 96 hours. I just turned that off. Do you think I should try to disable both of these modules? I'm not using them because they're not working right any way.

HollywoodChicago.com’s picture

I've just disabled both backup modules (so I'm not using them any way because they never worked correctly). I'm now asking the GoDaddy tech to tell me if any more files are created there. I haven't seen any more there since he started the cron job...

HollywoodChicago.com’s picture

Thank you, everyone! I believe we've found the culprit here (though I'm not *exactly* sure because the cron is doing an effective job). I am still waiting to hear back from GoDaddy to see if they can see any more files being created, but either way, I am now functional again.

If anyone else has this problem, hopefully this thread should help you out! I might help to report this within those two Drupal backup modules so they are aware of it over there, too. Cheers!

avskip’s picture

I'm using the backup-migrate module on Drupal 5.10 with no problem. Do you have the latest version of your modules?

I did notice one thing in common in a LOT of what the text file from GoDaddy contained. Each of the lines contains a reference to
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org). The ip address shown on the HTTP logs for this is x.x.111.243, whois returns their IP as x.x.111.242 for their website. Pretty close.

I do remember quite some time ago that a search engine nearly took my site down with it's incessant slurping, once told of it they toned it down. I don't remember getting the large dump files you have though.

I'm really interested in what you find the problem to be on this!

Skip

HollywoodChicago.com’s picture

Skip,

I'm running Drupal 5.5 with Backup module version 5.x-4.x-dev and Backup and Migrate module version 5.x-1.x-dev. At this time, I have them both disabled. As such, no new temp files are appearing in the directory in question.

I'm not sure if that's because disabling those modules has fixed the problem or if it's because the GoDaddy cron job is deleting them effectively (every 10 minutes) so I can't see them. I'm waiting to hear back from the GoDaddy tech about that. I last heard from him late on Friday.

I just checked and the recommended Backup and Migrate module is now on 5.x-1.1. Mine is 5.x-1.x-dev, which is a development snapshot. As for the Backup module, the recommended version is now 5.x-3.0. Mine is 5.x-4.x-dev, which is a development snapshot. Either one of those modules could have been the problem because they're not the current recommended version.

~ Adam

minesota’s picture

Can the cron job be put off temporarily to see if it was indeed the Backup module ?

HollywoodChicago.com’s picture

My story has now been picked up by The Consumerist (http://consumerist.com/5056063/perhaps-you-dont-owe-godaddy-6579). There are lots of interesting comments there from their readers.

rszrama’s picture

Any chance you can get them to post a correction on the article? : P

Drupal being a cancerous tumor is a far cry from "user using and misconfiguring development versions of code contributed by unmoderated third parties."

----------------------
Drupal by Wombats | Current Drupal project: http://www.ubercart.org

------------------------------------------
Read more: http://ryanszrama.com/topics/drupal

Kevin Hankens’s picture

I just wrote a letter to the editor of The Consumerist, Ben Popkin. His email address is listed in the left-column of the original article. I don't think that we should go nuts, but it would help to send him a kind note asking for a correction.

Kevin
--
Kevin Hankens
VeloNews.com

timmillwood’s picture

Sorry but if it was me I would of done a "sudo rm -R /var/www/*" pretty quick, and then restored from a backup.

But GoDaddy should of enforced the 150gb limit and it is their fault if the user uses more than that.

kjv1611’s picture

Please forgive me if I missed it in the thread. I just was hoping someone would sort of put the whole thing into one post that would explain the whole situation. Basically what caused this issue, precisely - one certain module or option? And what can each of us do to make sure we don't run into this same nightmare?

HollywoodChicago.com’s picture

Aside from the GoDaddy issue being discussed in the press, a lot of people have been asking me for the definitive culprit for this issue so we can all prevent it from happening. So, I've been researching that with the help of a very helpful GoDaddy tech. We just got out answer. This is new text from him that also answers several questions asked by someone else earlier:

Sorry for the wait Adam, I wasn't checking my email over the weekend.

I just disabled the cron job, and I am no longer seeing the files being created. I'd say it's a pretty safe assumption to be the backup
module(s) at this point. I'll leave the cron job disabled for the time being. Please let me know if you need to turn it back on for further troubleshooting and I'll help get it sorted out.

I could modify the job to only delete these files if they exceed a certain age (more than 24 hours, for example) so as to leave a certain number of files for troubleshooting, but without leaving the directory to grow large enough to affect your bill. Here are the answers to the questions posted from "styro":

* What is your PHP memory limit? php_info() should tell you this. - 8M
* What are the PHP settings for things like max upload size, max post size etc? again this is from php_info(). - Max upload = 2M, Max post size = 8M
* What are your upload filesize limits set at inside Drupal? - This is handled within PHP, not Drupal specific
* Are there any 500 errors logged in your Apache server error logs? - A few have been logged, but none that appear to correlate with this issue when the files are created.
* Does your host use suexec? eg are the uploaded files (ie through Drupal not FTP) in your files directory owned by your user account or a web server user account? - suexec is used, all files are owned by your user account.

kbahey’s picture

Looking at the code for both the back and the backup_migrate modules, it seems the latter is the one responsible for this, since it is the one that creates files called tmp_XXXX (see the function _backup_migrate_temp_file()).

You should not just disable cron though, what you need to do is re-enable cron again, but disable this module and file an issue for this so it gets fixed in a future release.
--
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc.
Personal blog: Baheyeldin.com.

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

kbahey’s picture

Adam

Looking at the configuration for the module, I found the following:

1. One cannot make backups run any more frequently than once an hour. So at most, you would have 24 attempts per day. How many of those are needed to blow out the 969GB limit?

2. By default, the module will not delete any old backups.

But, the fact that they are tmp_XXXX files, and not renamed to the proper name you specified in the settings tells me that this problem is probably due to the backup not completing.

And it can be caused by an environmental factor: Perhaps it is maximum execution time has expired, perhaps it is a cap on CPU seconds per process, or memory usage.

If Godaddy is capping these, and killing the process because it used too much of a certain resource, the temp file would be left there, and would not be cleaned by anything (until they wrote the cron to clean them up).
--
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc.
Personal blog: Baheyeldin.com.

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

kbahey’s picture

Here is a theory of why it can consume multi gigabytes. If the temp files are initially say 50MB, and they stay there. Then the following attempt will backup the failed backup. We now have two failed. Then the next one will backup 3 failed backups, ...etc.

So it can grow fast ...
--
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc.
Personal blog: Baheyeldin.com.

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

ronan’s picture

This does sound like a timeout issue. The deleting of temp files in backup and migrate happens at the end of the run, so a timeout could cause temp files to pile up. This theory is backed up by the original poster saying that the backup modules never worked properly for him. If the cron is timing out, however, the watchdog log should indicate that.

As for the backup of a backup theory, backup and migrate does not backup files, just the database so the files would not be growing in size exponentially. Some quick back-of-the-envelope calculations indicate that it would take 140ish days to reach 250gigs at 75mb a pop once an hour so I have to assume that the poster has cron running more than once an hour.

The backup module does backup files so it's possible that the combination of the two was causing issues.

------------------------------------
Ronan - Gorton Studios - http://www.gortonstudios.com/

------------------------------------
Ronan
Founder - NodeSquirrel - http://www.nodesquirrel.com/
Lead Developer - Gorton Studios - http://www.gortonstudios.com/

Amazon’s picture

We've been scrambling with GoDaddy for few hours chasing this down. I've had three calls with GoDaddy this morning and it sounds like we identified the confluence of configurations and modules that has led to this.

I am going to reach out to the module maintainers and ensure they are aware of this possible configuration problem.

Kieran Lal

Kieran Lal

ronan’s picture

Adam,

I'm the author of the Backup and Migrate module and if you have the time and inclination, I'd love talk to you about what happened here and maybe try and figure out exactly what went wrong. If my module did indeed go rogue, I want to figure out why so that nobody else runs into this.

If you are willing to take a little time for a post-mortem, I'd really appreciate it. Drop me a line on my contact form (http://drupal.org/user/72815) so we can go over it.

Thanks
------------------------------------
Ronan - Gorton Studios - http://www.gortonstudios.com/

------------------------------------
Ronan
Founder - NodeSquirrel - http://www.nodesquirrel.com/
Lead Developer - Gorton Studios - http://www.gortonstudios.com/

kbahey’s picture

I think it is the hosting environment that is the culprit here. The fact that the files are still in the tmp_XXXX stage says that the module did not get a change to complete the backup and the process was killed for some reason.

The only thing the module can do better is to:

1. Name the files "backup_migrate_XXXX", so their source is better known, rather than the generic sounding "tmp_XXXX".

2. When cron is run, clean any such temp files that are more than (say) a day old.

This makes sure that a restrictive host that limits resources would not cause the damage described by Adam.
--
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc.
Personal blog: Baheyeldin.com.

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

HollywoodChicago.com’s picture

I'd definitely concur that the shared GoDaddy hosting environment is restrictive. Notice the phpinfo limits mentioned in this thread.

While I'd say my site for the most part is healthy on Drupal, to this day I still have certain functions blow up (often due to timeouts) because of GoDaddy hosting restrictions (i.e. when forwarding a page to several people, etc.). The speed of the site is decent and can definitely be better, too. There is no doubt in my mind the site would perform better in a more robust hosting environment.

But you get what you pay for.

ronan’s picture

Those are exactly the steps I was thinking and I'm going to try and get them into all branches of the modules as soon as I can.

I should also be able to manage the timeouts a little better by monitoring the time taken by the module against the server timeout limit and have it die gracefully with a warning (and cleanup) if there is not enough time to complete. I think if users know the module is timing out (or running out of memory) they'll be able to head off problems before they get out of hand.

I can also try and implement a semaphore/mutex so that two backups aren't running at the same time, that way the module will know that it can safely delete any errant temp files.

The more people popular this module becomes the more I realize that I need to really nail down these edge cases and make sure the module fails gracefully (or at least not silently) to avoid problems like this one.

The very nature of the module (making backups easy) means that it's likely to be run in restrictive environments and by users who are not professional server admins. People put a lot of trust in this module and I need to take that responsibility very seriously.
------------------------------------
Ronan - Gorton Studios - http://www.gortonstudios.com/

------------------------------------
Ronan
Founder - NodeSquirrel - http://www.nodesquirrel.com/
Lead Developer - Gorton Studios - http://www.gortonstudios.com/

HollywoodChicago.com’s picture

Certainly. I'm all for helping people. I just e-mailed you from your contact form.

Publisher, HollywoodChicago.com