Simple views edit triggers too many server processes & site becomes unresponsive [#2511914]

This website (www.aspergillus.org.uk) is large - 85000 pages of data, 370 installed modules but all works well. I had the greatest confidence in Drupal (despite drupalgeddon last year) but this has shaken it.

A week ago I upgraded from 7.37 to 7.38 by copy/paste using Windows interface. All went well and I checked reports and had 7.38 confirmed. I also updated a few of the other requested module updates. Started to add in new content as normal. A few hours later (around 4pm) the site become unresponsive and page requests timed out. Looking at the server I could see that resources were at their limits - entry processes (20), physical (1GB) and virtual memory (2GB).

After some dicussion with the server hosts we rolled back our install to the previous day and all was well.

The following day all had been stable for 30 hours and I again updated to 7.38 as it is a security update. All seemed fine until 24 hours later (no modules updated this time but had spent some time adding in new content) when at 11am pages again became unresponsive and server resources were max'd out again. Had run update and cleared caches. Again we had to roll back 5 days.

This time left all alone for 2 days and there were no problems - server resources all normal. I assumed something had gone wrong with earlier upgrades and upgraded again. All was well for several hours so started adding content and once again at around 4pm things ground to a halt. We have rolled back this time to the previous day when all seemed stable and some content had been added when site was still 7.37.

I have looked for cron jobs, other automated services (feeds) that take place at around 4pm with no luck. No sign of unusual amount of traffic (in fact one crash took place on a sunday).

I do no custom coding, no changes to htaccess - just a straightforward core+modules install.

There is another problem with the theme (AT) in this install dating back 6 months see https://www.drupal.org/node/2405739 - which I have yet to resolve.

I have updated four other websites to 7.38 with no problems so the problem is unlikely to be my incorrect procedure!

So I am stuck with 7.37 for the time being until I can work out what is causing this problem. Could people more used to troubleshooting this kind of thing give me some more steps I can take to work this out?

Error log for yesterday prior to rollback contained the following. Nothing recorded around the critical time of 4pm but that might be due to lack of resources to write to the log? Bootstrap.inc seems prominent!:

[23-Jun-2015 17:59:38 Europe/London] PHP Parse error: syntax error, unexpected '=', expecting ')' in /home2/asperweb/public_html/modules/php/php.module(80) : eval()'d code on line 4
[24-Jun-2015 04:11:19 Europe/London] PHP Parse error: syntax error, unexpected '=', expecting ')' in /home2/asperweb/public_html/modules/php/php.module(80) : eval()'d code on line 4
[24-Jun-2015 12:48:31 Europe/London] PHP Warning: require_once(/home2/asperweb/public_html/includes/bootstrap.inc): failed to open stream: No such file or directory in /home2/asperweb/public_html/index.php on line 19
[24-Jun-2015 12:48:31 Europe/London] PHP Fatal error: require_once(): Failed opening required '/home2/asperweb/public_html/includes/bootstrap.inc' (include_path='.:/opt/php54/lib/php') in /home2/asperweb/public_html/index.php on line 19
[24-Jun-2015 12:48:36 Europe/London] PHP Warning: require_once(/home2/asperweb/public_html/includes/bootstrap.inc): failed to open stream: No such file or directory in /home2/asperweb/public_html/index.php on line 19
[24-Jun-2015 12:48:36 Europe/London] PHP Fatal error: require_once(): Failed opening required '/home2/asperweb/public_html/includes/bootstrap.inc' (include_path='.:/opt/php54/lib/php') in /home2/asperweb/public_html/index.php on line 19
[24-Jun-2015 12:48:39 Europe/London] PHP Warning: require_once(/home2/asperweb/public_html/includes/bootstrap.inc): failed to open stream: No such file or directory in /home2/asperweb/public_html/index.php on line 19
[24-Jun-2015 12:48:39 Europe/London] PHP Fatal error: require_once(): Failed opening required '/home2/asperweb/public_html/includes/bootstrap.inc' (include_path='.:/opt/php54/lib/php') in /home2/asperweb/public_html/index.php on line 19

UPDATE

This morning I attempted a simple add Global:text box to a view header while in version 7.37 and have recreated the same unresponsive website. It seems that this is NOT to do with 7.38 on its own

Error_log contains

[25-Jun-2015 10:07:20 Europe/London] PHP Parse error: syntax error, unexpected '=', expecting ')' in /home2/asperweb/public_html/modules/php/php.module(80) : eval()'d code on line 4
[25-Jun-2015 10:16:57 Europe/London] PHP Parse error: syntax error, unexpected '=', expecting ')' in /home2/asperweb/public_html/modules/php/php.module(80) : eval()'d code on line 4

How do I troubleshoot this??

Many thanks

Graham

Comments

Comment #1

GAtherton CreditAttribution: GAtherton commented 25 June 2015 at 11:43

This morning I attempted to carry out a simple add in a view while the site was still at version 7.37. It created the same issue so it seems that update 7.38 is not causing this problem. I had done this several times before I wrote all this and it did not cause the problem so I apologise for this being posted in the wrong place.

Incidentally error_log this morning only reports

So few error reports to help me - how on earth do I get to the bottom of this - and where should this question go?

Comment #2

GAtherton CreditAttribution: GAtherton commented 25 June 2015 at 11:47

Title:	Update to 7.38 triggers too many server processes & site becomes unresponsive	» Simple views edit triggers too many server processes & site becomes unresponsive
Project:	Drupal core	» Views (for Drupal 7)
Version:	7.38	» 7.x-3.7
Component:	base system	» Code
Issue summary:	View changes

Comment #3

cilefen CreditAttribution: cilefen commented 25 June 2015 at 13:18

It looks like the PHP module can't parse some code you have entered in the UI.

Comment #4

GAtherton CreditAttribution: GAtherton commented 27 July 2015 at 13:29

Finally engaged a consultant and the solution followed quite rapidly:

One thing that leapt out was a huge volume of error messages in the Log Messages page - all the same thing being repeatedly triggered. Here’s an example one:
http://test.asperweb.co.uk/users/gatherton#overlay=admin/reports/event/7...

I did some research and this looked to be being caused by the Media Browser Plus module. I disabled that module on the test site and the errors have now stopped. More importantly, the test site now appears to be loading at normal speed so I think the combination of this error, the size of it and the frequency with which it was being written to the error logs (causing an insert into the SQL database) is what was overwhelming SQL on the server and causing the slow-down.

I’ve now disabled the module on the live site too and cleared the caches (had to do both directly in the database via PHPMyAdmin) and the live site also now looks to be back working as expected. I’ve been able to log in without problems and click around numerous pages on the site which have all loaded in standard times.