Issue
Since Friday 6th October, it is slow to log into our internal site. This could be 12-30 minutes or not being able to log in at all. prior to this there have been no issues, it has been working fine.

This is making our internal live site unusable.

This site is our document repository recently we have added another 20 users and approx 10k documents, 100gb.

Our test site is also experiencing a similar problem, development sites are ok.

Would be very grateful for any support in diagnosing this issue.

symptons
slow log in for both AD authenticated users and non authenticated users
Slow performance once/if logged in.
When a user logs in the mysql and acpache processes reach 50% cpu, any additional log ins create a further apache process that uses high cpu.
From Friday the following error messages are appearing regularly.
mysql error.log
2017-10-09T09:56:51.105440Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4554ms. The settings might not be optimal. (flushed=52 and evicted=0, during the time.)
2017-10-09T10:07:25.915712Z 0 [Note] Inn
Apache error.log
[Mon Oct 09 09:26:28.190036 2017] [:error] [pid 28790] [client 10.10.20.63:50198] PHP Fatal error: Maximum execution time of 240 seconds exceeded in /home/devadmin/eim/webroot/includes/database/database.inc on line 448, referer: http://

cron job scheduled to run every 3 hours has not run for 2 days 18 hours, i.e. last run Saturday afternoon.

Site details
Drupal 7.41
Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-96-generic x86_64)
I would estimate that we have no more than 100 concurrent sessions.
We have between 500k 1000k documents as content items.
we use the ldap module to automatically give users access to drupal roles and organic groups as I identified by active directory groups.

site modules
core
administration
chaos tool suite
date time
development
EIM (our custom development)
features
fields
lightweight directory access protocol
mail
media
Multilingual - Internationalization
Organic groups
Other
rules
services
services authentication
services servers
taxomony
views
workflow

Comments

cfox612’s picture

Since this started on a particular date have you looked at the logs? Any recent updates before the crash? Maybe a new theme hook?

aburrows’s picture

What’s the resource usage on it, eg memory_limit

Is it a self hosted box or a VM elsewhere?
Any updates to apache, php, etc

pietrosantoro’s picture

Hi all,

I have the same problem too. Since a month, MySQL's error.log file is full of:

InnoDB: page_cleaner: 1000ms intended loop took xxxxms. The settings might not be optimal. (flushed=yyy and evicted=0, during the time.)

I'm logging also MySQL slow queries and, for every "page_cleaner" error, there is a slow query (as obvious). The most frequent queries are:

UPDATE cache SET

INSERT INTO accesslog

I tried to tuning MySQL but errors remain.

Some solution?

Thanks