Problem/Motivation

PHP notices are a sign for bad code, they fill the logs and are usually very easy to fix.

Proposed resolution

Add a sensor similar to 404, count of PHP notices, with output of most frequent php notice, group on variables.

Debug output with php notices.

Warning > 10 by default, no error threshold.

Remaining tasks

User interface changes

API changes

CommentFileSizeAuthor
#54 interdiff-2495613-52.54.txt5.13 KBjohnchque
#54 sensor_for_php_notices-2495613-54.patch5.31 KBjohnchque
#52 interdiff-2495613-50-52.txt2.78 KBjohnchque
#52 sensor_for_php_notices-2495613-52.patch7.24 KBjohnchque
#50 sensor_for_php_notices-2495613-50.patch6.46 KBjohnchque
#46 interdiff_44-46.txt3.24 KBLKS90
#46 sensor_for_php_notices-2495613-46.patch24.13 KBLKS90
#44 interdiff_39-44.txt833 bytesLKS90
#44 sensor_for_php_notices-2495613-44.patch23.93 KBLKS90
#41 sensor_for_php_notices-2495613-39.patch23.84 KBLKS90
#39 interdiff_36-39.txt3.8 KBLKS90
#36 interdiff_34-36.txt3 KBLKS90
#36 sensor_for_php_notices-2495613-36.patch23.79 KBLKS90
#39 separate_text_fields-2525432-39.patch26.63 KBLKS90
#34 sensor_for_php_notices-2495613-34.patch22.14 KBLKS90
#32 sensor_for_php_notices-2495613-32-interdiff.txt8.45 KBberdir
#32 sensor_for_php_notices-2495613-32.patch16.92 KBberdir
#27 sensor_for_php_notices-2495613-27.patch15.77 KBLKS90
#27 interdiff_23-27.txt1.18 KBLKS90
#23 interdiff_20-23.txt2.32 KBLKS90
#23 sensor_for_php_notices-2495613-23.patch15.76 KBLKS90
#20 sensor_for_php_notices-2495613-20.patch15.75 KBLKS90
#20 interdiff_17-20.txt11.71 KBLKS90
#20 sensor_for_php_notices-2495613-20.patch15.75 KBLKS90
#17 interdiff_15-17.txt2.69 KBLKS90
#17 sensor_for_php_notices-2495613-17.patch12.42 KBLKS90
#15 interdiff_12-15.txt1.78 KBLKS90
#15 sensor_for_php_notices-2495613-15.patch12.21 KBLKS90
#12 sensor_for_php_notices-2495613-12.patch12.19 KBLKS90
#12 Screen Shot 2015-07-15 at 17.27.14.png134.96 KBLKS90
#8 interdiff_6-7.txt1.01 KBLKS90
#7 sensor_for_php_notices-2495613-7.patch9.53 KBLKS90
#6 sensor_for_php_notices-2495613-6.patch10.09 KBLKS90
#1 sensor_for_php_notices-2495613-1.patch4.17 KBLKS90

Comments

LKS90’s picture

Status: Active » Needs review
StatusFileSize
new4.17 KB

Here is a patch that provides the sensor. Ordering is broken, but that's a general DatabaseAggregatorSensorPlugin problem.

Status: Needs review » Needs work

The last submitted patch, 1: sensor_for_php_notices-2495613-1.patch, failed testing.

miro_dietiker’s picture

This is mostly duplicate code from Watchdog issue.

Either postpone this issue immediately and wait for the Watchdog plugin... Or just reduce this issue to just provide default sensor configuration for the database aggregator sensor.. (and later switch to the Watchdog sensor.)

LKS90’s picture

Status: Needs work » Postponed

Postponed until watchdog sensor is committed (some code from here should be implemented over there, the replacement is more functional in this version). #2525426: Separate unaggregated results method

miro_dietiker’s picture

Status: Postponed » Active

Unpostponing since we have now the Watchdog aggregator.

LKS90’s picture

Status: Active » Needs review
StatusFileSize
new10.09 KB

Here is the sensor. It still has a custom SensorPlugin (because the sensor message is kind of custom) but it extends the watchdog sensor. Also has a test.

LKS90’s picture

StatusFileSize
new9.53 KB

Somehow added the runSesnsor() to the watchdog aggregator as well, unneeded.

LKS90’s picture

StatusFileSize
new1.01 KB

Interdiff for the removal.

Status: Needs review » Needs work

The last submitted patch, 7: sensor_for_php_notices-2495613-7.patch, failed testing.

The last submitted patch, 6: sensor_for_php_notices-2495613-6.patch, failed testing.

berdir’s picture

Can you post a screenshot of the verbose output?

I don't think that already looks like I'd expect it to

  1. +++ b/config/install/monitoring.sensor_config.php_notices.yml
    @@ -0,0 +1,31 @@
    +  conditions:
    +    -
    +      field: type
    +      value: php
    

    This should be hardcoded, the sensors makes no sense for anything else.

  2. +++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
    @@ -0,0 +1,50 @@
    +/**
    + * Monitors php notices logged in the watchdog.
    + *
    + * @SensorPlugin(
    + *   id = "php_notices",
    + *   provider = "dblog",
    + *   label = @Translation("PHP notices (database log)"),
    + *   description = @Translation("Monitors PHP notices from database log."),
    + *   addable = FALSE
    + * )
    + *
    + * Displays PHP notice with highest occurrence as message.
    

    one class description is enough ;)

  3. +++ b/src/Tests/MonitoringCoreKernelTest.php
    @@ -125,6 +125,54 @@ class MonitoringCoreKernelTest extends MonitoringUnitTestBase {
    +    // Prepare a fake PHP notice.
    +    $error = [
    +      '%type' => 'Recoverable fatal error',
    +      '!message' => 'Argument 1 passed to Drupal\Core\Form\ConfigFormBase::buildForm() must be of the type array, null given, called in /usr/local/var/www/d8/www/core/modules/system/src/Form/CronForm.php on line 127 and defined',
    

    this is not a notice ;)

    Maybe the sensor should be called PhpErrors, not sure.

LKS90’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new134.96 KB
new12.19 KB

Alright, here is the latest version. Looks rather good, but actually doesn't work as intended yet.

Problems:

  • The results are still not sorted by record_count (the entry with the most occurrences should be at the top)
  • The width of each column seems rather arbitrary (important columns squeezed together, 'useless' column takes up most space)

I had to go through some hoops to get it working (making slight changes to the parent class WatchdogAggregatorSensorPlugin), so some review before continuing would be useful. The default value for table and aggregation are definitely missing at the moment (creating a new watchdog sensor via UI is impossible).
By the way, I haven't changed the test yet, I don't expect it to pass (if it does, yay! :D).

Screenshot of the verbose result (and the query):

Status: Needs review » Needs work

The last submitted patch, 12: sensor_for_php_notices-2495613-12.patch, failed testing.

miro_dietiker’s picture

Yeah that's a common problem with tables and fine as a starting point.

We can consider an alternative table display (like responsive table that does a vertical display) but that's a different issue.

+++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
@@ -0,0 +1,105 @@
+  public function verboseResultUnaggregated(array &$output) {
...
+  public function resultVerbose(SensorResultInterface $result) {
...
+    $this->verboseResultUnaggregated($output);

Huh... No the Unaggregated is already called in DatabaseAggregatorSensorPlugin::verboseResult().
Don't call it a second time.

Other than tht, didn't check in much detail.

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new12.21 KB
new1.78 KB

Ordering works now, removed the unnecessary call to unaggregatedResultVerbose.

Status: Needs review » Needs work

The last submitted patch, 15: sensor_for_php_notices-2495613-15.patch, failed testing.

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new12.42 KB
new2.69 KB

Fixed the test fails.

(Moved the sensor to optional because of dblog dependency)
(updated tests with new sensor message)

berdir’s picture

Status: Needs review » Needs work

Looks pretty nice :)

  1. +++ b/config/optional/monitoring.sensor_config.php_notices.yml
    @@ -0,0 +1,22 @@
    +id: php_notices
    

    We usually prefix sensors with a module name.. probably dblog_ in this case.

  2. +++ b/config/optional/monitoring.sensor_config.php_notices.yml
    @@ -0,0 +1,22 @@
    +label: 'PHP notices'
    +description: 'Php notices logged by watchdog'
    

    Since we already have watchdog in the category, what about adding something more useful as the description?

    What about "Monitors and displays the most frequent PHP notices and errors"?

  3. +++ b/src/Plugin/monitoring/SensorPlugin/DatabaseAggregatorSensorPlugin.php
    @@ -295,7 +295,7 @@ class DatabaseAggregatorSensorPlugin extends DatabaseAggregatorSensorPluginBase
           '#tree' => FALSE,
    -      '#default_value' => implode("\n", $this->sensorConfig->getSetting('keys')),
    +      '#default_value' => implode("\n", $this->sensorConfig->getSetting('keys', ['variables'])),
           '#maxlength' => 255,
    

    Why hardcode this as the default here?

  4. +++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
    @@ -0,0 +1,105 @@
    +/**
    + * Monitors php notices logged in the watchdog.
    + *
    + * @SensorPlugin(
    + *   id = "php_notices",
    + *   provider = "dblog",
    + *   label = @Translation("PHP notices (database log)"),
    + *   description = @Translation("Monitors PHP notices from database log."),
    + *   addable = FALSE
    + * )
    + *
    + * Displays PHP notice with highest occurrence as message.
    + */
    

    Still that double-description thing, one (above the annotation) is enough.

  5. +++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
    @@ -0,0 +1,105 @@
    +      $result->setMessage('@count times: @error', ['@count' => (int) $this->fetchedObject->records_count, '@error' => SafeMarkup::xssFilter(SafeMarkup::format('%type: !message in %function (line %line of %file).', unserialize($this->fetchedObject->variables)), Xss::getAdminTagList())]);
    

    SafeMarkup() already marks a string as safe, xssFilter() should not be needed?

    Maybe what you want to do instead is filter the message?

  6. +++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
    @@ -0,0 +1,105 @@
    +    $query->orderBy('records_count', 'DESC');
    +    $query->range(0, 10);
    

    Why range(0, 10) if we only display the first?

  7. +++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
    @@ -0,0 +1,105 @@
    +    $query->groupBy('variables');
    +    $query->orderBy('records_count', 'DESC');
    +    return $query;
    

    This is the one that should have a range.. maybe 20? A site could have a hundred or more different php notices.

  8. +++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
    @@ -0,0 +1,105 @@
    +      $output['result']['#header'] = ['Type', 'Message', 'Function', 'File', 'Line'];
    

    Do we translate the headers elsewhere? we don't translate messages, so actually not sure if we should here or not.. but it should be consistent.. looks like no other table has anything that *could* be translated? but entity aggregate I guess will at it and that will be translated.

  9. +++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
    @@ -0,0 +1,105 @@
    +  /**
    +   * {@inheritdoc}
    +   */
    +  public function resultVerbose(SensorResultInterface $result) {
    +    $output = parent::resultVerbose($result);
    +
    +    return $output;
    +  }
    

    This is not doing anything anymore?

  10. +++ b/src/Plugin/monitoring/SensorPlugin/WatchdogAggregatorSensorPlugin.php
    @@ -52,7 +47,9 @@ class WatchdogAggregatorSensorPlugin extends DatabaseAggregatorSensorPlugin impl
    +    $form['aggregation']['time_interval_field']['#default_value'] = 'timestamp';
         $form['aggregation']['time_interval_field']['#disabled'] = TRUE;
    

    Wondering if there is a use case for not having a time interval field. I guess this is fine, you can always set the time to a high value.

  11. +++ b/src/Tests/MonitoringCoreKernelTest.php
    @@ -125,6 +125,55 @@ class MonitoringCoreKernelTest extends MonitoringUnitTestBase {
    +    $result = $this->runSensor('php_notices');
    +    $message = $result->getMessage();
    +    // Assert the new message is returned as a message.
    +    $this->assertEqual($message, SafeMarkup::format('3 times: %type: !message in %function (line %line of %file).', $new_error), 'The new message is now the sensor message.');
    +    $this->assertNotEqual($message, SafeMarkup::format('2 times: %type: !message in %function (line %line of %file).', $error), 'The old message is not the sensor message anymore.');
    +  }
    

    Verbose output test would be nice, since that's about the most complicated thing the sensor does now ;)

miro_dietiker’s picture

Berdir 2.) "Monitors and displays the " sounds like a lot redundancy?

5. XssFilter is exactly what we needed to add since the usa if "!" makes it mark the contant as unsafe and everything is escaped afterwards...

10. Monitoring has this problem with providing more specific sensors that things are configurable (from a base sensor) but should not be editable anymore... That's only one particular case. This problem is mostly unresolved in the UI yet. In this particular case i would expect that a time interval of 0 is like not applying a time interval...

11. I would prefer to have a table with individual columns:
- Type
- Date
- File
- Line
- Message
And yeah, please check the verbose output based on some real data, including the double encoding problem above (5).

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new15.75 KB
new11.71 KB
new15.75 KB

Here is the latest version, now with a count for each row and a generally better table building structure in DatabaseAggregatorSensorPlugin. The watchdog aggregator still works as usual.

Status: Needs review » Needs work

The last submitted patch, 20: sensor_for_php_notices-2495613-20.patch, failed testing.

The last submitted patch, 20: sensor_for_php_notices-2495613-20.patch, failed testing.

LKS90’s picture

StatusFileSize
new15.76 KB
new2.32 KB

The test for the dblog_php_notices sensor was not updated, the other exception will be gone when issue #2525432: Separate text fields per verbose extra field is committed.

LKS90’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 23: sensor_for_php_notices-2495613-23.patch, failed testing.

mbovan’s picture

When I apply the patch (#23) I get:
"Warning: implode(): Invalid arguments passed in Drupal\monitoring\Plugin\monitoring\SensorPlugin\DatabaseAggregatorSensorPlugin->buildConfigurationForm() (line 309 of /usr/local/var/www/d8.dev/www/modules/monitoring/src/Plugin/monitoring/SensorPlugin/DatabaseAggregatorSensorPlugin.php).".

Is it because #2525432: Separate text fields per verbose extra field is not committed yet?

  1. +++ b/src/Plugin/monitoring/SensorPlugin/DatabaseAggregatorSensorPlugin.php
    @@ -171,40 +171,51 @@ class DatabaseAggregatorSensorPlugin extends DatabaseAggregatorSensorPluginBase
    +   *   The $rows for which a header should be built.
    

    $rows => rows

  2. +++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
    @@ -0,0 +1,118 @@
    +class PhpNoticesSensorPlugin extends WatchdogAggregatorSensorPlugin {
    +  /**
    

    A blank line here?

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new1.18 KB
new15.77 KB

I amended the comment so the $ is gone and it's more explanatory.

Also added the new line, didn't know that, thanks!

Yes, the exception will be gone when #2525432: Separate text fields per verbose extra field is committed, the imploding/exploding is only done because of the text area to enter keys/verbose_fields (the new name) and causes some problems if the input NULL.

mbovan’s picture

+++ b/src/Plugin/monitoring/SensorPlugin/PhpNoticesSensorPlugin.php
@@ -0,0 +1,119 @@
+ */
+
+class PhpNoticesSensorPlugin extends WatchdogAggregatorSensorPlugin {
+  /**

Oh, I thought a blank line between class and method doc comment.

Status: Needs review » Needs work

The last submitted patch, 27: sensor_for_php_notices-2495613-27.patch, failed testing.

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 27: sensor_for_php_notices-2495613-27.patch, failed testing.

berdir’s picture

Status: Needs work » Needs review
Issue tags: +Needs tests
StatusFileSize
new16.92 KB
new8.45 KB

Took this on a bit.

The split for the verbose stuff was pretty good, but you still executed the query in there, so a subclass couldn't get to the values. I moved the query out and instead I'm passing the results in.

Also a number of small improvements, like shortening the file name.

Seems like we have no test coverage for the verbose output.

Let's add a new test class for that that looks at the verbose output with those notices.

Status: Needs review » Needs work

The last submitted patch, 32: sensor_for_php_notices-2495613-32.patch, failed testing.

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new22.14 KB

Fixed some test fails for the Kernel test, moved the $this->verboseResultUnaggregated($output); call. (The output was completely missing, this fixes that) and added test coverage for the verbose output in the WebTest.

Status: Needs review » Needs work

The last submitted patch, 34: sensor_for_php_notices-2495613-34.patch, failed testing.

LKS90’s picture

Couldn't reproduce the error after moving the unaggregatedResultVerbose method call anymore...

The foreach exception we saw once is easily fixed by removing the fields validation from the validation method. Since we can't configure the fields for the output, validating them is wrong anyway. The previous test failed with my xpath queries, so I left the debug in this path to take a look.

LKS90’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 36: sensor_for_php_notices-2495613-36.patch, failed testing.

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new26.63 KB
new3.8 KB

Readded validation + a stub method for PHP sensor.

Last version, if the xpath still fails, I'll have to find another solution.

Also fixed some more exceptions/error which resulted from parent methods validating/expecting fields/conditions which aren't used for the php sensor.

Status: Needs review » Needs work

The last submitted patch, 39: separate_text_fields-2525432-39.patch, failed testing.

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new23.84 KB

Uhhm, ignore the patch above.

LKS90’s picture

Status: Needs review » Needs work

The last submitted patch, 41: sensor_for_php_notices-2495613-39.patch, failed testing.

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new23.93 KB
new833 bytes

Added debugs for the two arrays that are compared, locally the test passes, so 'm kind of debugging the testbot.

Status: Needs review » Needs work

The last submitted patch, 44: sensor_for_php_notices-2495613-44.patch, failed testing.

LKS90’s picture

Status: Needs work » Needs review
StatusFileSize
new24.13 KB
new3.24 KB

I think this one should pass local tests and the testbot's test, I changed the logged error message's file path to use DRUPAL_ROOT as well, so the test works locally and on the testbot and the filename shortening is tested. I also changed one test to only assert the existence of a variable, instead of checking the value as well. The count changed with every minor change and the assert was not meant for that anyway.

berdir’s picture

Version: 8.x-1.x-dev » 7.x-1.x-dev
Status: Needs review » Patch (to be ported)
Issue tags: -Needs tests

Committed, working on a follow-up to clean some things up; #2543886: Allow database sensor plugins to control which configuration elements are shown.

  • Berdir committed bd64af2 on 8.x-1.x authored by LKS90
    Issue #2495613 by LKS90, Berdir: Sensor for php notices
    
johnchque’s picture

Assigned: Unassigned » johnchque

Let's backport this.

johnchque’s picture

Status: Patch (to be ported) » Needs review
StatusFileSize
new6.46 KB

Early patch, still didn't find the FormatableMarkup d7 replace. Also mixed the verboseResultUnaggregated method appending the other ones in one single method.

Status: Needs review » Needs work

The last submitted patch, 50: sensor_for_php_notices-2495613-50.patch, failed testing.

johnchque’s picture

Status: Needs work » Needs review
StatusFileSize
new7.24 KB
new2.78 KB

Changed a bit, fixed those FormattableMarkup thingys. Also added it to be listed on the sensors page.

Status: Needs review » Needs work

The last submitted patch, 52: sensor_for_php_notices-2495613-52.patch, failed testing.

johnchque’s picture

Status: Needs work » Needs review
StatusFileSize
new5.31 KB
new5.13 KB

New patch, now should work better, still working on tests.

Status: Needs review » Needs work

The last submitted patch, 54: sensor_for_php_notices-2495613-54.patch, failed testing.

The last submitted patch, 54: sensor_for_php_notices-2495613-54.patch, failed testing.