The 2026 Drupal Business Survey is open. Your response will help shape the future of the Drupal business ecosystem.Problem/Motivation
We noticed the log can become unreasonably large if a lot of files are being purged.
Large logs by itself are not the problem but the fact that this is serialized and written to a single row in
the key_value table causes some issues. The 3 main issues we experienced are.
1) Purge jobs become slower as the serialized string get larger (in our case about 100mb in serialized string)
2) Cannot visit the log page anymore because the string is to large for memory
3) If you have slow query logs turned on this can grow exponentially (350gb log in one day in our case)
This can be partially mitigated by setting the retention if the logs to a much smaller time at the cost of having less log history
Proposed resolution
My proposed solution is to not use the key_value table / Drupal::state to log purges but to add its own logging table so updates
won't have to be done in one single large serialized string.
Remaining tasks
Refactor code to use own logging table/ logging entity
Data model changes
Remove the akamai.purge_status state and move logs to own table.
Old data will need to be migrated to this new table
| Comment | File | Size | Author |
|---|---|---|---|
| #5 | akamai-status-table-3001392-5.patch | 22.42 KB | smustgrave |
Comments
Comment #2
smustgrave commentedGiving this a shot. So far I've
Where I'm at so far just wanted to share.
Comment #3
smustgrave commentedOk this is my first crack at it. Think I'm going down the right track but any feedback would be greatly appreciated. Somewhat hard to test so I was relying on the prewritten tests.
I still left the old PurgeState and StatusStorage files as I'm not sure what to do with those.
Comment #4
smustgrave commentedFixed a few things.
Comment #5
smustgrave commentedWas able to replace the old status and statusStorage files while still being able to move over the old records to the new table.
Comment #6
smustgrave commentedComment #7
smustgrave commentedI've neglected this ticket some what sorry about that. Has anyone had a chance to review?
Comment #8
WidgetsBurritos commented@smustgrave, Sorry for the silence on this. I meant to get to this a while ago, but my priorities have been a bit all over the place.
I can confirm this is a big issue. In fact, I'm labeling this as "critical", as I think it's a blocker for a stable release. We have plans to try to fully evaluate the solution and see if we can't finalize something here within the next few weeks.
Comment #9
smustgrave commentedI'd be happy to help where I can. Is there a way to setup a fake akamai account for testing?
Comment #10
WidgetsBurritos commentedTo be honest, I'm unsure. Might be worth reaching out to akamai to find out. If you do, let me know what they say. I can do it too, but can't right at the moment.
Comment #11
smustgrave commentedMaybe this could be used? https://developer.akamai.com/tools/akamai-sandbox
Comment #12
pobster commentedTry this...
Comment #13
pobster commentedComment #15
pobster commentedHmmm I was worried about this, the collating of results by purge IDs no longer works. Needs more thinking about...
Then again, maybe the test is wrong;
https://developer.akamai.com/api/core_features/fast_purge/v3.html
Now I've no idea what to do...
Comment #16
pobster commentedhttps://community.akamai.com/customers/s/question/0D50f00005RtqL4CAJ/how...
Okay so ... what I might do then, seeing as how this purely exists for checking the status of purges and as https://www.drupal.org/project/akamai/issues/3160840 exists ... I might just chop out status logging entirely? There's no need to store them, Akamai themselves are saying so. This has ceased to be a log for status checking, and now it's just a watchdog for purges.
This only existed previously in the D7 version as a means to "force a status check" to see if something has purged. But ... that code doesn't even exist anymore, previously it used to check on a cron (!) https://git.drupalcode.org/project/akamai/-/blob/7.x-3.x/akamai.cron.inc...
Comment #17
pobster commentedFor test... (and I'll refactor in the "remove CCUv2" issue).
Comment #19
smustgrave commentedSo if you're getting rid of statues all together. Should this ticket be repurposed to delete any code centered around statues and clear out the key_value entries.
Comment #20
pobster commentedI'll need to check what needs to be removed/ changed (mainly to avoid confusion). And obviously for debug it'll be best to leave in the the purge logs and still have cron expire them - I'll do it today, it's now part of my sprint work.
Comment #21
pobster commentedOkay ... I've gone over the Remove CCUv2 Support ticket, and can confirm that this ticket is now a duplicate. The status log only existed for the sake of CCUv2 - and in removing it, I've also removed all the status log code as well.
Note, I've also realised that there's no need to implement a purge log (which I'd considered might be useful), as there's already the facility to turn on;
DEVELOPMENT OPTIONS
Log requests
Log all requests and responses.
...which logs to watchdog anyway.
Once that ticket is reviewed/ merged, I'll close this one too.
Comment #22
pobster commentedCurrently testing in production, will merge ... next week? Then we can close these two tickets.
Comment #23
pobster commentedOkay, we can consider the CCUv2 patch very well tested, and as such I've merged it and am closing this issue.
Thank you!