Problem/Motivation

Together with the duration of a database query, it would be great to make available the start time of the query when getting the database logs. This is already available in the query statement that performs the logging action so it doesn't add any overhead.

This is useful, for example, when logging the query in APM tools such as with the Elastic APM module. With start time + duration we have the full information about the query that otherwise is incomplete.

Proposed resolution

Add the start time of the query to the database log.

Remaining tasks

Commit

User interface changes

None

API changes

Added a new parameter start to the method Drupal\Core\Database\Log::log().

Data model changes

None

Release notes snippet

TBD
Together with the duration of a database query, it would be great to make available the start time of the query when getting the database logs. This is already available in the query statement that performs the logging action so it doesn't add any overhead.

This is useful, for example, when logging the query in APM tools such as with the Elastic APM module. With start time + duration we have the full information about the query that otherwise is incomplete.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

krystalcode created an issue. See original summary.

krystalcode’s picture

krystalcode’s picture

Status: Active » Needs review

Please review.
Not sure if this function is called anywhere else, couldn't find.
Also, not sure if that's an API change and it would need to go to a different release.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.9 was released on November 6 and is the final full bugfix release for the Drupal 8.7.x series. Drupal 8.7.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.8.0 on December 4, 2019. (Drupal 8.8.0-beta1 is available for testing.)

Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

daffie’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests
  1. +++ b/core/lib/Drupal/Core/Database/Log.php
    @@ -106,16 +106,21 @@ public function end($logging_key) {
    -  public function log(StatementInterface $statement, $args, $time) {
    +  public function log(StatementInterface $statement, $args, $start, $time) {
    

    New parameters should be added after all the existing ones. Also it must have a default value. Not doing so will result in a BC break.

  2. We need testing for this to make sure it works and that it keep working in the future.
ravi.shankar’s picture

Here I have re-rolled the patch #2.

agrochal’s picture

FileSize
1.76 KB
467 bytes

I just set parameters in correct order, because as mentioned in #5 we should change the order but not only in Log.php but Statement.php also. Attaching patch and interdiff as well.

daffie’s picture

@agrochal: Good fix, thanks.

Now we just need those tests to be added.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

mfb’s picture

Version: 8.9.x-dev » 9.2.x-dev
Status: Needs work » Needs review
FileSize
3.91 KB
3.26 KB

Rerolled patch, added some missing changes that were needed, added a default value of the new parameter for backward compatibility, and added a basic test.

(I'd love to get this in so I can add some performance tracing UI improvements elsewhere... :)

mfb’s picture

[deleted - wrong thread]

mfb’s picture

daffie’s picture

Status: Needs review » Needs work
Issue tags: -Needs tests +Needs change record, +Needs issue summary update

@mfb: Thank you for working on this patch. Just 2 points left for me:

  1. +++ b/core/tests/Drupal/KernelTests/Core/Database/LoggingTest.php
    @@ -32,6 +33,9 @@ public function testEnableLogging() {
    +      $start = $query['start'];
    

    I do not understand why this line is added. Can it be removed?

  2. In the file core/lib/Drupal/Core/Database/Log.php, we have the following docblock:
      /**
       * Cache of logged queries. This will only be used if the query logger is enabled.
       *
       * The structure for the logging array is as follows:
       *
       * array(
       *   $logging_key = array(
       *     array('query' => '', 'args' => array(), 'caller' => '', 'target' => '', 'time' => 0),
       *     array('query' => '', 'args' => array(), 'caller' => '', 'target' => '', 'time' => 0),
       *   ),
       * );
       *
       * @var array
       */
      protected $queryLog = [];
    

    The new key "start" needs to be added here.

mfb’s picture

Status: Needs work » Needs review
FileSize
1.2 KB
4.39 KB

I do not understand why this line is added. Can it be removed?

I was using it to test that each start time in the array is greater than (or equal to) the previous start time, but I removed it as it's not necessary in my opinion.

The new key "start" needs to be added here.

Ok, good catch

daffie’s picture

Status: Needs review » Needs work
+++ b/core/lib/Drupal/Core/Database/Log.php
@@ -22,8 +22,8 @@ class Log {
+   *     array('query' => '', 'args' => array(), 'caller' => '', 'target' => '', 'time' => 0, 'start' => 0),
+   *     array('query' => '', 'args' => array(), 'caller' => '', 'target' => '', 'time' => 0, 'start' => 0),

@@ -108,9 +108,13 @@ public function end($logging_key) {
+  public function log(StatementInterface $statement, $args, $time, $start = NULL) {

What is now the default value? NULL or 0?

mfb’s picture

Status: Needs work » Needs review

Not sure what you're saying needs work. Maybe you see an inconsistency here?

I made the default value NULL - i.e. this parameter is nullable - because 0 means midnight 1/1/1970 so isn't really suitable as a default value. There should always be a start time, but if somehow there is not, it's best to leave it unset.

The docblock is providing the "structure" of the array, not default values (some params do not have default values). The time and start elements do not default to zero; presumably the zero just indicated numeric.

daffie’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community
Issue tags: -Needs change record, -Needs issue summary update

@mfb: Good point.

All code changes look good to me.
I have added a CR and updated the IS.
For me it is RTBC.

mfb’s picture

👍 CR looks good, thanks daffie

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/lib/Drupal/Core/Database/Log.php
@@ -108,9 +108,13 @@ public function end($logging_key) {
+   * @param $start
+   *   The time the query started as a float (in seconds since the Unix epoch
+   *   with microsecond precision).
    */
-  public function log(StatementInterface $statement, $args, $time) {
+  public function log(StatementInterface $statement, $args, $time, $start = NULL) {

This is missing a typehint. Since we're adding a parameter we can take the opportunity to properly typehint it - also the @param should have a float. Yes the rest of the documentation could do with being fixed but that's for another issue.

ankithashetty’s picture

Status: Needs work » Needs review
FileSize
4.63 KB
806 bytes

Updated the patch in #14 addressing #19. Thank you!

ankithashetty’s picture

Updated patch.

daffie’s picture

Status: Needs review » Reviewed & tested by the community

The point of @alexpott has been addressed.
Back to RTBC.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/lib/Drupal/Core/Database/Log.php
@@ -114,7 +114,7 @@
-  public function log(StatementInterface $statement, $args, $time, $start = NULL) {
+  public function log(StatementInterface $statement, array $args, float $time, float $start = NULL) {

This should be:
StatementInterface $statement, $args, $time, float $start = NULL

We can't add the other typehints because that's a BC break.

I think it is fine to fix the @param typehints for the other parameters here although it is strictly out-of-scope.

ayushmishra206’s picture

Made the change suggested in #24. Please review.

ayushmishra206’s picture

Status: Needs work » Needs review
daffie’s picture

Status: Needs review » Reviewed & tested by the community

The point of @alexpott has been addressed.
Back to RTBC.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed c377d26 and pushed to 9.2.x. Thanks!

  • alexpott committed c377d26 on 9.2.x
    Issue #3089326 by mfb, ankithashetty, ayushmishra206, ravi.shankar,...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

mfb’s picture

FileSize
379.63 KB

Here's a sample display of database query trace now that we are logging query start time:

SQL tracing