Problem/Motivation
Dblog module hasn't really changed since Drupal 6. The upgrade to Drupal 8 left the dblog Controller with a lot of SQL queries inside. People looking for inspiration in Drupal core, should not use dblog as an example of how to organize the code of a Drupal 9+ project.
There is no way to access dblog entries without directly querying the database, and dblog is usually found in tests where modules want to ensure a given log is present after a certain action. There is no Dblog Entry API either, so everything dblog related is coupled to the table structure.
Proposed resolution
- Keep using original watchdog table schema, this guarantees sames indexes are created, no performance is list in terms of writing logs into the database.
- Keep using the Dblog Logger service to insert logs, so no performance is lost when creating logs. Also make the DblogEntry class final, so no contrib module can extend it. This will reduce the need of potential breaking changes.
- Convert Dblog Entries into content entities. But make these entities read only in the sense they cannot be updated after being created. This is what someone would expects from immutable log entries.
- Drupal core assumes all content entities can be saved, and jsonapi module has tests for this. This is incompatible with the point mentioned before, so we make the entity type internal to make it invisible to jsonapi. This also means unfortunatelly no jsonapi integration out of the box.
- Provide an EntityListBuilder to reduce the code of the DblogController. Will also simplify the views integration. and potentially expose the entity via jsonapi or rest modules. Provide a ViewBuilder to render the log details.
- When uninstalling dblog, dblog entities will be automatically deleted. This is required to be somehow compatible with how dblog worked before the entities conversion, but also to allow uninstall dblog in a production site where logs are created automatically. As this entities are content, a small change to the ContentUninstallValidator is needed.
Regarding BC compatibility:
- Leave DblogController untouched. Only change references in the the dblog.routing.yml to load the entity collection. By providing a ListBuilder and a ViewBuilder we should be able to replicate the same functionality as currently provided by the dblog module. The current set of automated tests should pass for the new implementation using ListBuilder and ViewBuilder.
- Direct queries to the watchdog table are still valid, but not recommended anymore.
Remaining tasks
Reviewers needed! All code is done and test are in place.
- Add the new DblogEntry class [✅ Done]
- Add the new DblogEntryStorage class [✅Done]
- Add the View Builder [✅ Done]
- Add the List Entity Builder. [✅ Done]
- Add the remaining methods the DblogEntryStorageInterface. [✅ Done]
- Add new tests (without deleting the existing ones). [✅ Done]
- Add Upgrade path for watchdog view [✅ Done]
- Get some in depth code reviews. [❌ in progress]
User interface changes
None. Some internal classes will change, instead of having dblog-error
to render the icons, we will rely on numeric values provided by RfcLogLevel. For example severity-4
.
API changes
- New DblogEntry class. and DblogEntryInterface interface.
- New DblogEntryStorage class and DblogEntryStorageInteface interface.
- New DblogEntryAccessControllerHandler class.
- New DblogViewsData class.
- New DblogEntryListBuilder class.
- New FormatMessage service.
- New Drupal\Core\Entity\ContentEntityType::requireDeleteContentBeforeUninstall() method
Data model changes
None. The watchdog table will remain intact. Column indexes are preserved.
Original report
By @chx: We could make dblog entities with a separate service writing the base data tables for fast writing.
Comment | File | Size | Author |
---|---|---|---|
#84 | 2401463-nr-bot.txt | 90 bytes | needs-review-queue-bot |
#79 | 2401463-nr-bot.txt | 90 bytes | needs-review-queue-bot |
Issue fork drupal-2401463
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #7
candelas CreditAttribution: candelas as a volunteer commentedThis is needed to be able to integrate with Views Bulk Operations and other modules. I hope someone with knowledge get the time to give us this present :)
At the moment no modules help admin Watchdog entries...
Comment #15
dagmarI'm working on this.
Comment #16
daffie CreditAttribution: daffie commentedA couple remarks on the MR and the testbot is not happy.
Comment #17
candelas CreditAttribution: candelas as a volunteer commentedThanks @dagmar :)
Comment #18
dagmarThanks for the initial review @daffie I will move this back to Needs Review when I consider is worth a review. The DblogStorage is incomplete and will have some methods to justify the existence of the class.
Comment #19
dagmarUpdated issue summary
Comment #20
dagmarComment #21
volegerReview comments need to be addressed.
Comment #22
dagmarComment #24
Gauravvvv CreditAttribution: Gauravvvv at OpenSense Labs commentedComment #25
dagmar@voleger Thanks for your help. I'm still working on this, will you let me know when you finish with your modifications?
Comment #26
volegerSorry, no changes planned.
Comment #27
volegerset proper status
Comment #28
fgmDid you have a look at the way other db loggers like mongodb_watchdog and redis_watchdog do it ? It could be good to have a shared common mechanism for that feature, so that these alternate modules can be smaller and implement only a storage layer.
Comment #29
dagmar@fgm not yet. This is mostly exploratory work at this moment since we do not have any 'non content' and 'non config' entity storage in core right now.
I'm trying to make the tests pass using the current D9 API. But I will take a look later for sure.
Comment #30
dagmarSomething we will need fixed before implement this. #3233143: DefaultTableMapping expects ContentEntityTypeInterface to work, but this may not be necesary ready for a review.
Comment #31
dagmarMost of the failing tests are due #3042467: Support entities that are neither content nor config entities for now I patched jsonapi to skip unsupported entities.
Comment #32
dagmarThere are a few pending things that I want to address as part of this patch:
But for now I think this is ready for a review @daffie @fgm
Comment #33
dagmarComment #34
dagmar@fgm I checked the code of
redis_watchdog
. With the last set of commits I pushed, this module will have automatic UI if they provide a custom implementation ofDblogEntryStorageInterface
.Comment #35
dagmarLinked issues resolved by this issue.
Comment #36
dagmarComment #37
dagmarComment #38
dagmarComment #39
dagmarComment #40
dagmarComment #41
dagmarComment #42
dagmarComment #43
dagmarComment #44
dagmarI think it is ready for some reviews.
Comment #45
dagmarComment #46
dagmarComment #47
dagmarComment #49
smustgrave CreditAttribution: smustgrave at Mobomo commentedTested this out on 9.5 and it still applies cleanly
Ran drush updb and the hooks worked created
Code looked clean to me
Only question on the db log page there is an operations column but it's empty.
Not sure how to get a framework manager to review and can't speak on it well enough for a change record.
Don't mind retesting it as this seems like a great feature so will be keeping an eye out! Thanks
Comment #50
dagmarThanks @smustgrave
Please consider the operations column for logs only are available for a very specific set of logs entries, like when creating roles.
Comment #51
smustgrave CreditAttribution: smustgrave at Mobomo commentedwhat's your thoughts on a default value like "NA"
Comment #52
dagmarFor other listings core does not use NA when no operations are there. I think it should remain empty.
Comment #53
dagmar@smustgrave a review on #3233143: DefaultTableMapping expects ContentEntityTypeInterface to work, but this may not be necesary would be useful to unblock some of the dependencies of this issue.
Comment #54
smustgrave CreditAttribution: smustgrave at Mobomo commentedTaking a look now. I can mark this RTBC or should I wait for the change record?
Comment #55
dagmarI just updated the change record. I cannot say if you can mark this as RTBC I coded the feature and I'm the dblog subsystem maintainer. I think we need an in depth code review, this is a pretty big change.
Comment #56
smustgrave CreditAttribution: smustgrave at Mobomo commentedMakes sense I took a look and tested locally and nothing seemed to break so I will least give it a vote of RTBC+1
Will post in slack and see if anyone else is interested
Comment #57
BerdirI didn't have a very close look, but my gut reaction is that I'm pretty conflicted on this.
The diff here is +2378 -284, its _a lot_ of new code to maintain (and initially review), there's performance to consider, some sites write _a lot_ of log entries (mostly not for good reasons, but it happens, like tons of php notices or so), making it an entity requires a ton of code (as it's a fieldable entity not a content entity, so it gets to reuse very little code but almost all the new code is custom for dblog).
The problem/motivation doesn't seem convincing enough for me to maintain that amount of code. Abstracting the storage could be also done with a regular service, similar to how things like key value store work.
Comment #58
dagmarThe diff here is +2378 -284, its _a lot_ of new code to maintain (and initially review), there's performance to consider, some sites write _a lot_ of log entries (mostly not for good reasons, but it happens, like tons of php notices or so), making it an entity requires a ton of code (as it's a fieldable entity not a content entity, so it gets to reuse very little code but almost all the new code is custom for dblog).
Glad to see you here @berdir, I was expecting a review from you.
I agree it is a lot of code to review.
Please note the logger system is still just an
INSERT INTO
, see item one of proposed resolution.Keep using the Dblog Logger service to insert logs, so no performance is lost when creating logs
Could you elaborate this a bit more?
Comment #59
smustgrave CreditAttribution: smustgrave at Mobomo commentedJust following up on this one.
Comment #60
dagmarI did some profiling. Test scenarios:
1. Install the site using drush.
2. Load a page not found as soon the site is installed.
3. Load a second page different page not found.
So I think we need to dig a bit deeper on this massive performance on uncached pages.
I will keep the needs profiling tag to see if I can fix this issue and provide new data.
Comment #61
dagmarI ran the profiling tests again with more detail this time. It seems I was comparing incorrect data. Here are my findings. In all cases Left is drupal 9.4 using regular watchdog entries. Right dblog entities. All test were made using the Standard profile.
Visit homepage after clearing the cache with drush cr
A page not found after visiting the homepage (cache warm scenario)
Another page not found just after cleaning cache
Visit the logs overview page - (cache warm scenario)
Visit the logs overview page just after cleaning cache
Display an individual log event - (cache warm scenario)
Comment #62
dagmarComment #63
smustgrave CreditAttribution: smustgrave at Mobomo commentedThat is very neat! If I'm understanding the numbers correctly it's slightly better in a lot of scenarios.
May be out of my league to fully mark it but +1 for RTBC
Comment #64
Gauravvvv CreditAttribution: Gauravvvv at Srijan | A Material+ Company for Drupal India Association commentedComment #65
tstoecklerI really like the idea of this issue but was surprised to see that dblog entries are not content entities here. Did not really find an explanation for that here, is
from the issue summary it?
If so, I'm not sure I'm conviced to be honest. I would agree that it does not make sense for dblog entries to follow the lead of nodes, terms, etc. by extending
EditorialContentEntityBase
but in my opinion it would still make sense for it to be a content entity. It's perfectly fine to not opt-in to revisions or translations and by not declaring afield_ui_base_route
, you are really disabling dynamic fields for all intents and purposes. That would achieve the "simplified" part as far as I can tell but allow a much greater amount of code re-use.Or are there other reasons that aren't explained here (or that are and I have overlooked, in which case: sorry!).
Comment #66
dagmar@tstoeckler thanks for your feedback (and @berdir too). I had to do some tests to remember why I took the decision of not going with regular content entities.
It seems all boils down to the plan to make the new entity type compatible with the legacy approach of using
$log->severity
as a StdClass object.Today I reflected about this architecture decision and I think it was incorrect. This is a new API, we have the chance to define the interface and let others use it properly.
I will take some days to refactor the codebase. I did some quick tests today and indeed it is possible to extend ContentEntityBase. Initial refactor work attached.
Comment #67
dagmarComment #68
dagmarComment #70
dagmarAlright, commit 12a875fb has a working patch for making dblog content entities, against Drupal 9.4.x. I have to say I'm much more comfortable with the shape of the code right now. Much less code duplication. And half of the lines of this patch are tests. Thanks @Berdir and @tstoeckler for pushing in that direction.
Some pending follow ups:
I won't be able to work on any Drupal issues anymore for the next 4~6 months (new baby incoming) so I'm leaving this for someone else to continue and do the follow ups. 👋
Comment #71
smustgrave CreditAttribution: smustgrave at Mobomo commentedStill very interested in this.
Wonder if you could provide a good way for one to test this?
Comment #72
volegerAdded review comments
Comment #74
jungleComment #75
mmbkThe failing test with postgres is not generated by this patch, it already fails with the 10.1.x-dex branch https://www.drupal.org/node/3060/qa
Comment #76
jungleNW for #75
No opinion.
Any reason for this? -1.
Make sense.
Comment #77
Gauravvvv CreditAttribution: Gauravvvv at Axelerant for Drupal India Association commentedComment #78
junglePer the test result of DbLogEntryTest::testSerialization() #75 may not be an issue anymore. Probably, it's not an issue in newer versions of PostgreSQL.
Comment #79
needs-review-queue-bot CreditAttribution: needs-review-queue-bot as a volunteer commentedThe Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".
Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #80
catchLeft some comments on the MR. More than profiling I am wondering if the indexes are correct (i.e. an EXPLAIN on some common queries would be useful) and whether the schema matches between existing sites and a new install.
Comment #81
dagmarI'm back to work on this.
Comment #82
dagmarI updated the issue summary to answer @catch concerns regarding indexes. I ran a `DESCRIBE watchdog;` command before and after installing the site using this patch, and the table structure remains the same.
I guess this also is valid for the issue #3081144: Database primary keys can exceed maximum integer storeable (has actually occurred, in watchdog), if we fix this in the dblog.install module, the changes will also apply to dblog as entities.
I also addressed the comments about dblog exception in ContentUninstallValidator. Based on my current understanding of the module_installer logic, there is no way for dblog to by pass this content validator exception, so I had to modify it any way. I tried to make it as simple and clear possible, and not only for dblog.
Comment #83
dagmarComment #84
needs-review-queue-bot CreditAttribution: needs-review-queue-bot as a volunteer commentedThe Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".
This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #85
dagmarComment #86
dagmarI updated the issue summary based on the latest reviews assuming the proposed method names are ok. Removing needs profiling as I did the profiling tests in #61.
Comment #88
dagmarI went over all the pending open threads of the merge request and closed most of them. I left a few open to make sure the provided answer is enough for the original reviewers.
I also checked some concerns I had related caching. It seems entities are not cached but routes are (example
route:[language]=en:/admin/reports/dblog/event/10
) in cache_data table. This is also happening for the current non entity implementation so I will not attempt to fix this as part of this issue. Maybe is not even an issue.Most of the concerns regarding performance, indexing and future compatibility with #3081144: Database primary keys can exceed maximum integer storeable (has actually occurred, in watchdog) are addressed as the underlying system still uses the same database scheme and insert queries.
My general feeling about the code (as the current dblog maintainer) is: this is pretty much ready to be in. There is some room of improvement in the
DblogEntryViewBuilder
, but probably qualifies for a follow up. And comparing this with the current code of the dblog module I would say this is a big improvement in code quality and alignment with Drupal standards. And will also allow to close several postponed issues for more than a few years now.The remaining steps in my opinion are:
as this is calling a new API method:
ContentEntityTypeInterface;:requireDeleteContentBeforeUninstall
which needs some grammar and common sense checking.Comment #89
dagmarI had more time to think about item 1. I think we should not make an exception for dblog entries. After researching a bit why the content validator was created it seems a bit dangerous to bypass the checks.
Concerns about disabling the dblog module in a production environment are still valid though. I created this to take care of this functionality: #3354081: Provide a way to disable creation of database log entries
Comment #91
solideogloria CreditAttribution: solideogloria commentedComment no longer applies