That's the question:

Is it at all possible to tweak drupal or the environment in which a drupal site functions, in a way that memory usage problems could be overcome within a shared cpanel environment? (searching the drupal site will show many others who encountered the same or similar issues)

This is a question i ask myself and a log as to what i have found so far in tweaking my sites, that are currently seeing a lot of 500error page not found because of premature end of file errors, as a direct result of too much memory usage (more then 16mb).

The hosting company only gives 16mb of RAM per account within packages as normal hosting, reseller hosting and fail-over hosting. They would be between $5 - $30 a month. To increase RAM i would have to go with a form of dedicated hosting with dollar amounts that are beyond that mentioned earlier.

The site causing the problem has around 150 members with maybe 40 daily active members logging in and reading pages and posting comments and some nodes (around 1 blog and maybe 4 comments a day).
The extra modules are maybe around 20, including pathauto, within drupal 5+.

I've been searching drupal.org and found the following posts concerning memory usage problems:
- fatal memory Memory Problem
- Views + CCK (fields) use a lot of memory
- .htaccess overloads my servers memory
- memory allocation error
- and many more...

My quest led me (duhh) to the drupal handbook and i discovered the technical specifications :).
One of those pages is:
- Server tuning considerations

Others are:
- target=_>Optimize and Tweak High-Traffic Servers

Then the following page:
- System requirements
mentions the following:

"PHP memory of approximately 8MB for a Drupal core installation. In reality you will probably need to use a higher setting depending on your site and contributed modules you are using. A good starting point is 16-24 MB."

Conclusion:
Doing the reading i did above leads me to understand that to run a drupal site with an average amount of members i should forget a regular hosting account of not more then $10 (or even $20) a month, because a decent functioning drupal member site is going to cost me big bucks.

If i am wrong and missed some good articles on how to tweak my site or the cpanel environment, please tell me so i can implement them, for i need it bad.

Comments

inforeto’s picture

Yes, when there's no cushion you need to fine tune your sites to save RAM.
Go through the apache and mysql tuning articles, altough tweaking these may need the host's intervention.

In cpanel, KeepAliveTimeout is 15 seconds by default, so get it lowered to 3-5 seconds.
Then turn on and configure Query Cache in mysql, which is also off in cpanel by default.

If you can, try to run the script mentioned in http://drupal.org/node/85768 to see if things like open tables are under control.

Now, php may take 16+ MB RAM, but the whole of an apache process usually grows to 25 MB or more.
On top of that, mysql allocates 1-5 MB more of RAM per connection to handle sorts, joins and reads.
So RAM usage per page can be an issue, but only if the site is highly dynamic like a forum.
For regular sites, cache modules and software can keep traffic under control.

In drupal, see that you don't use unnecessary functionality, like collecting statistics that could be taken by google or the server.
If you have views + cck, the performance techniques do help, cutting page generation time.

Eagle-i’s picture

I 'd like to try your tips, but have no idea where to look for them. I open my cpanel and from i work. or ftp.
Where would i find "KeepAliveTimeout" and be able to change it?
Or "Then turn on and configure Query Cache in mysql"?

Any help is appreciated.

inforeto’s picture

Unfortunately, those settings aren't accesible in cpanel, not even in whm.
Those are configurations on apache and mysql, so you need root to change them.

But you can check with your host to know what the values are (all of them, found in the tuning tutorials). This way you know if the shared host is suitable for drupal performance, request changes or plan on moving.

php can show some of those values. Just upload a file with the following line:
<?php phpinfo(); ?>

Eagle-i’s picture

This is exactly my point:
A growing or average drupal site can't function within a shared hosting cpanel / whm account or environment.
That's almost the conclusion i have to come to.

Btw:
the value for KeepAliveTimeout is 300 (not sure if that is miliseconds).

NancyDru’s picture

But it depends a lot on the host. Some hosts will cram as many users as they can onto a server. Other hosts (the better ones) limit the number of accounts (not quite the same as sites) on a server. The ones with limits will accommodate a growing site longer.

Nancy W.
Drupal Cookbook (for New Drupallers)
Adding Hidden Design or How To notes in your database

NancyDru’s picture

Are the values of those settings available by "SELECT VARIABLES"?

Nancy W.
Drupal Cookbook (for New Drupallers)
Adding Hidden Design or How To notes in your database

NancyDru’s picture

Actually, very few hosts that I've looked at limit your storage usage. A few more block your usage of php.ini or settings.php to override memory.

So tell your host that if they don't remove your storage limit, or at least raise it to a more sane level, that you will go elsewhere - and be prepared to do so. You don't need a server of your own for the traffic you're getting (it's low). There are plenty of budget hosts that will let you do what you want.

I would suggest you avoid the hosts that have a separate MySql server (such as Dreamhost) because these tend to be problematic, especially for lower traffic sites (can't keep stuff in cache long enough). There are many recommendations out here on DO.

Nancy W.
Drupal Cookbook (for New Drupallers)
Adding Hidden Design or How To notes in your database

Eagle-i’s picture

You probably mean RAM and not storage limit :).

Is there a thread where i can find a list of hosts that will accommodate me with the RAM i need?
Otherwise it's going to a VPS or something else, and i do not have the experience or knowledge (yet) to work with those.

But as i mentioned earlier:
I love drupal and choose this above the other CMS's around, but if it can't function within a regular shared hosting cpanel account, things will have to be reconsidered. I will have to see what CMS can do all this within a cpanel.

My thoughts would be:
isn't there some memory usage module or hook that can be implemented that will somehow allow drupal and or contributed modules to work within a certain allocation of RAM, which could also be adjustable? This way sites on hosts with more RAM could be configured to use more RAM and therefore perform better.
Or somehow re-evaluate the use of public var's and private var's and the amount of queries of the Mysql database? I am not a techie, just talking from an old Clipper database programming background :) (php and Mysql looks very much like a sophisticated version of it).

VM’s picture

Is there a thread where i can find a list of hosts that will accommodate me with the RAM i need?

an entire forum related to hosting Drupal commercially see: http://drupal.org/hosting. My advice is that if you are expecting your site to be busy, seek out a VPS or Dedicated Hosting. Or at the very least a hosting package where you can easily bounce from shared to VPS, or Dedicated when necessary.

isn't there some memory usage module or hook that can be implemented that will somehow allow drupal and or contributed modules to work within a certain allocation of RAM, which could also be adjustable?

no. Drupal runs fine under 16MB's contributed modules on the other hand, depending on how well they are written and what they are designed to do can consume RAM faster than an overweight family at an all you can eat buffet.

Eagle-i’s picture

here is another example of what seems to be a drupal issue. or maybe not:

VPS runing out of memory

"Drupal continuously calls to the database which is in a different server. For that, the server allocates some memory until the request is processed.
The problem in this case would be that since if the request are not being completed so the memory is never freed. This builds up until the server resources are gone!"

VM’s picture

There are many sites running on a VPS, if this was a widespread issue, there would be widespread posts about it. My money is on incorrect server configuration or possibly not having their VPS tuned properly.

Eagle-i’s picture

Reading some more today at drupal, i discovered an excellent post (where you commented on as well) discussing this problem, maybe in different setting.

The post is Digg Effect" and discusses high volume traffic taking down a drupal site.

There is also a comment in there talking about comparing joomla and drupal in performance.

A point made in this whole discussion there was the need to somehow optimize drupal's code so it would not cause a problem within a SHARED environment, such (i think) as in cpanel accounts.

inforeto’s picture

Not really, you can use cpanel in any server setting, not only shared.
In fact, cpanel does take a chunk of the server's RAM.
(say, 40 MB with httpd off, unsuitable for a 128 MB account)
This cost is not significant in shared accounts, because it's divided among multiple users.

However, in vps and up it must be included when calculating costs, because it isn't factored in the pricing.
Some of the cheaper offers out there are even priced without control panel software at all.
On the good side, drupal benefits from vps accounts because sites don't compete for server resources.
The process queues are inherently slower when shared, so the results are better and under control.

For more on this and the digg effects, see webhostingtalk.com, or google for cpanel vs. plesk comparisons.

inforeto’s picture

You may also find many search result speaking of "drupal making too many queries", but most of the time it turns out that the host had the database configured to work with few queries per page.
Queries are fought with cache, but you may just need to find a host with a good database setup.

Eagle-i’s picture

I did a search with your sentence and found a number of interesting articles. It also seems that the drupal site itself had issues in May this year (not sure if they still have it):

Status update on Drupal.org performance issues and how you can help

There is also a similar site mentioned in the next post that has problems:

Site down! - too many connections?

It seems that the issues that play are:
1. MySql database connection and optimization issues
2. RAM usage by drupal's 3th party modules and caching

So, this to me looks like a bigger, meaning structural, problem, that may not have been a problem with previous (meaning first) versions of drupal, but now leads me to think that ... maybe we've come to a phase in the growth of drupal that some rethinking has to be done, from ground up. At least, from a cpanel user point of view :).

inforeto’s picture

Well, this isn't limited to drupal. There's too many factors in play that add up.
The shared hosting model is a structural problem by itself.

But in drupal, contributed modules interact with each other, which is not seen in many CMS.
This makes them hard to optimize and update, leads to tons of support requests, less time for replies, and at the same time get extremely invaluable.

Eagle-i’s picture

Well, i have been doing some searching for VPS packages and discovered that there are some that might be not too expensive for now. They offer between 5-15GB, 100-150gb bw and 128 - 250RAM. These are in the price range of $12-$33.

As i am coming from a cPANEL user standpoint and have no experience what so ever with VPS, what are the requirements for me to run a good drupal site(s), with some extra sites i host for friends?

Is there a thread on requirements?

Things like what:
- OS
- admin panel for me as admin and panel for a thirdperson user (friends, family, etc)
- etc

Appreciate the help.

inforeto’s picture

With VPS note that they may offer burstable RAM but likely won't be available at peak hours.
But if you have only a forum and few things else you should have no problem.

It is still a good idea to investigate where is your current site getting slower.
Ask your host how many RAM it is using at peak and for how long.
Otherwise, if your users grow again you will outgrow the VPS very quickly.

As for recommendations:
OS -> CentOS
CP -> Cpanel with WHM. (plesk/virtuozzo is good too)

But note that cpanel takes its own RAM that wasn't in your expenses while sharing an account.
Then there's the expense of configuring and managing it.
You have stated you have only used cpanel as an user so you'll likely need it managed.

Eagle-i’s picture

Thanks for tips about VPS.
I am trying to tune my install as much as possible but without success. I also understand that pathauto is also heavy on the CPU / memory. Maybe i find an alternative.

inforeto’s picture

See if the http://drupal.org/project/advcache
can help with the pathauto problem (by reducing the path queries).

Eagle-i’s picture

Thanks. Well, i can't run those needed patches since i don't have a server but a shared cpanel account. Is there another way to patch drupal?

inforeto’s picture

Your concerns are right, but configuring and managing a server is the core of the hosting business.
What happens is that there's a grey zone between the small sites and the large sites where the usage won't fit the price.

Out of this zone the scalability works as normal, where shared hostings are the bottom of the ladder.
VPS is one step up, in which you get isolated resources, but usage must be calculated carefully.
Only the entry level VPS have a price advantage, as entry level dedicated servers cost the same as a large VPS.

At this point, you just have to weight the cost of tuning a system against spending a few dollars in a better hosting.
For example, cpanel license cost monthly fees, which are supposed to lower the cost of managing a server.
Of course, non-commercial sites are bound by the dollars, so tuning must be made.

Back to drupal, limiting the RAM only limits the site to work with less modules.
See if you can benefit from cache, so check drupal cache, mysql query cache and cpanel also comes
with zend php optimizer, which is a php accelerator.

There's many things that can be done and must be done, but the scope spans across several issues.
Usually, drupal is already optimized for regular cms usage, only special cases must be tweaked.
One needs to search why and where a small site is taking too much generation time and resources.
A forum is not really a heavy module by itself, but it all kind of forums are known to grow rapidly.

Beyond this, you'll need to choose between using modules out of the box or working with php to use programming techniques.
This is the case with Views and CCK, where the initial load can be lessened by using the API, etc.
In most cases, using the advanced functions is still better than creating your own custom module.

kus’s picture