--- AI TRACKER METADATA ---
Update Summary: Add AI context item usage data and page
Additional Collaborators: @kristen_pol
AI Tracker found here: https://www.drupalstarforge.ai/
--- END METADATA ---

Problem/Motivation

In order to update entities later, we need to know what context was used when touching those entities.

Steps to reproduce

Proposed resolution

Create usage data: id, date, context item id, agent machine name, entity id (if relevant)

  • If the context is used for the agent and entity id pair, update the date
  • Add a Usage page that shows up to the left of the Settings page and has its own permission

This mapping will be used when updating entities later when context changes

Remaining tasks

  • Add schema and code
  • Test and review
  • Commit :)

User interface changes

See example images added in comments

API changes

Data model changes

Has own database schema

AI usage

The code associated with this issue was developed using AI tools.

A human worked with the AI tool to:

  • Clarify the requirements
  • Spec out the plan and architecture
  • Create the code

The human:

  • Reviewed the code
  • Manually tested the functionality
  • Requested adjustments from AI
  • Adjust the code themselves when necessary

Issue credit is still relevant for humans working with AI as they are using AI as a tool and not to do everything for them. It requires human expertise and oversight to utilize AI effectively and efficiently in addressing these issues.

Issue fork ai_context-3550034

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

kristen pol created an issue. See original summary.

kristen pol’s picture

Issue summary: View changes
kristen pol’s picture

kristen pol’s picture

Status: Active » Postponed

Postponing until we've prioritized in the roadmap

kristen pol’s picture

kristen pol’s picture

Assigned: Unassigned » kristen pol
Status: Postponed » Active
Related issues: -#3559379: [META] CCC rearchitecture and roadmap

I'll take a look at this one now.

kristen pol’s picture

deleted... wrong issue

kristen pol’s picture

StatusFileSize
new637.51 KB

Here's an example of a usage table:

views plugin usage table with type, name, provided by, and used in columns

kristen pol’s picture

StatusFileSize
new370.46 KB

Another example:

field list usage table with field name, entity type, field type, used in, and summary columns

kristen pol’s picture

Title: Add AI context tracking for entities » Add AI context item usage data and page
Assigned: kristen pol » Unassigned
Issue summary: View changes
kristen pol’s picture

Issue summary: View changes
kristen pol’s picture

Issue summary: View changes
kristen pol’s picture

Issue summary: View changes
marcus_johansson’s picture

Issue tags: +priority
kristen pol’s picture

Issue tags: +mvp
kristen pol’s picture

Assigned: Unassigned » kristen pol
kristen pol’s picture

Assigned: kristen pol » Unassigned

Been focused on other things, so unassigning this one for now

marcus_johansson’s picture

Issue tags: +AI Initiative Sprint
kristen pol’s picture

Assigned: Unassigned » kristen pol
Priority: Normal » Major

I'm going to take a look at this now for the sprint and decide if I want to hand it off.

kristen pol’s picture

Issue summary: View changes
kristen pol’s picture

Assigned: kristen pol » Unassigned
Status: Active » Needs review

Been planning with Cursor.

The tricky bit is how do we know what entities have been touched?

Here's an implementation plan from Cursor (lightly updated by me):

https://docs.google.com/document/d/19GUjbEmIgxy-dGlKgeozTiRqZWU30NjfnVOU...

Are we going to be able to get the entity ids from AgentToolFinishedExecutionEvent?

kristen pol’s picture

Pushed up some starting code, which I've reviewed, but I need to test

But see the Google Doc above first for some questions

New files:

  • src/Entity/AiContextUsage.php - Content entity for tracking usage
  • src/Entity/AiContextUsageAccessControlHandler.php - Access control
  • src/AiContextUsageListBuilder.php - Entity list builder
  • src/Service/AiContextUsageTracker.php - Service for recording/querying usage
  • src/EventSubscriber/AiContextAgentToolSubscriber.php - Tracks tools and entity modifications
  • src/Controller/AiContextUsageController.php - Usage admin page

Modified files:

  • src/EventSubscriber/SystemPromptSubscriber.php - Records context usage when selected
  • ai_context.services.yml - Service registrations
  • ai_context.permissions.yml - Added view ai context usage permission
  • ai_context.routing.yml - Added /admin/ai/context/usage route
  • ai_context.links.task.yml - Added Usage tab

Data tracked per record:

  • Context item ID
  • Agent ID
  • Target entity type/ID (when entities are modified)
  • Runner ID (correlates a single agent run)
  • Tools used (JSON array)
  • Timestamps

kristen pol’s picture

Assigned: Unassigned » kristen pol

I'm still testing and fixing.

kristen pol’s picture

StatusFileSize
new731.51 KB

Fixed some bugs and updated the functionality. Here's what it looks like so far.

context control center usage page with columns context id, agent id, target entity id and type, tools, and routes

kristen pol’s picture

StatusFileSize
new735.73 KB

Switched to using a view

context control center usage page with columns context id, agent id, target entity id and type, tools, and routes

kristen pol’s picture

Assigned: kristen pol » Unassigned

Done for now.

Main question: How to get the entity ID if it's not part of the route?

rakhimandhania’s picture

Issue tags: +AI Innovation
kristen pol’s picture

Issue tags: +Needs UX review
kristen pol’s picture

From meeting with Aidan+Emma:

At minimum, limit the the number of records so database isn't flooded

Need to discuss the difference between deep logging for debugging vs usage for entity association.

Follow up issue on discussion issue for using the observability API.

kristen pol’s picture

Status: Needs review » Needs work
Related issues: +#3567803: [Discuss] Observability support for CCC

Needs work for: At minimum, limit the the number of records so database isn't flooded

Here's the observability issue: #3567803: [Discuss] Observability support for CCC

I posted that in a few channels to see if people will chime in.

kristen pol’s picture

Issue tags: -Needs UX review
kristen pol’s picture

Assigned: Unassigned » kristen pol
kristen pol’s picture

Assigned: kristen pol » Unassigned
Status: Needs work » Needs review
Issue tags: +Needs UX review
StatusFileSize
new244.17 KB
new258.73 KB

Added a settings page and code to purge records.

Usage page

context control center usage page with columns context id, agent id, target entity id and type, tools, and routes

Settings page

ccc usage settings page with settings like enable usage tracking, max records, max age of records, and clear records

bbruno’s picture

Assigned: Unassigned » bbruno
bbruno’s picture

Assigned: bbruno » Unassigned

I'll leave the code review for someone more on the BE-side, but on the UX/frontend side everything is looking good to me. Seems like a real nice addition to the module and everything is clear on how to use it.

kristen pol’s picture

Issue tags: -Needs UX review

Thanks, Bruno!

kristen pol’s picture

Assigned: Unassigned » kristen pol
kristen pol’s picture

Assigned: kristen pol » Unassigned

This can be used as a starting point if we don't do anything with the observability API before 1.0.

emma horrell’s picture

Issue tags: +Needs UX review
afoster’s picture

Emma and reviewed and #34 looks good. We're assuming the main use case we're using these logs for is:

"why does my page suck"... Hmm is it the context or the agent prompt or what...?

Table Update suggestions:

1. Agent should link to the Agent config page
2. Entity should be a link to the rendered page / entity.
3. Add a view link to view single log record.

Create a "entity view" for each log item record. (Similar to looking at a single Drupal watchdog log)

1. Move Route info from the table into single log item view
2. Print the actual Prompt / Context info here (so I can see if something unexpected was attached)
3. Maybe a history of calls, not the full logs but if it ran 50 times on a single call I would like know if it's running
4. Token usage

kristen pol’s picture

Assigned: Unassigned » kristen pol
Status: Needs review » Needs work
Issue tags: -Needs UX review

Thanks!

kristen pol’s picture

Assigned: kristen pol » Unassigned

Unassigning for now in case someone else wants to pick this up

scott falconer’s picture

Assigned: Unassigned » scott falconer

Taking a look at this one.

kristen pol’s picture

Great! Merge in the latest from 1.0.x and note the new installation instructions on the project page don’t have the steps for setting up the agent side of things. I was testing with the v2025 demo when creating this

scott falconer’s picture

Status: Needs work » Needs review
StatusFileSize
new63.05 KB
new58.15 KB

Updated MR !35:
https://git.drupalcode.org/project/ai_context/-/merge_requests/35

This adds the remaining UX items discussed in the thread (agent/entity links + single record view).

Changes
- Usage view: Agent column links to the agent config form.
- Usage view: Entity column links to the target entity (when available).
- Usage view: Added a "View" link column to open a single usage record page (watchdog-style).
- Single record page (/admin/ai/context/usage/): shows Tools used, Routes, and renders the current Context item content.
- Moved Routes out of the table into the single record page (kept the Route filter exposed).
- Merged latest from 1.0.x.
- Update hook: updates the ai_context_usage View display config on existing sites.
- Update hook: installs the new ai_context_item created field storage on existing sites (prevents missing column errors).
- Composer: ensure dynamic_entity_reference is required (module still uses this field type).
- Config schema: added scheduler third_party_settings schema for ai_context_item_type.
- Added a Kernel test for the usage tracker service.
- Docs: clarified AI Agent setup steps in README (create agents at /admin/config/ai/agents).

Quick verification
- Visit /admin/ai/context/usage and confirm Agent/Entity link behavior and the new "View" link column.
- Click "View" for a record and confirm the single record page renders Tools used, Routes, and Context item content.

Testing (local DDEV)
- PHPCS (Drupal, DrupalPractice): PASS
- PHP lint (changed files): PASS
- PHPUnit (ai_context module tests): PASS (49 deprecations on Drupal 11.3.x)
- drush updb / cr: PASS
- drupal-intent-testing compare run: IDENTICAL (0 JS errors, 0 Drupal alerts); screenshots attached

Related
- Installation error "The state 'draft' already exists in workflow" is tracked in https://www.drupal.org/project/ai_context/issues/3571188 (MR !51). Tested: enabling ai_context fails without MR !51 and succeeds with it on a fresh Drupal 11 install. This MR intentionally does not include that fix.

AI disclosure
This work was AI-assisted (analysis/drafting), with human review + local testing.

robloach’s picture

Assigned: scott falconer » robloach

I can do a review of this one.

robloach’s picture

Assigned: robloach » Unassigned
Status: Needs review » Needs work

When visiting admin/ai/context/usage, I'm getting a database error of the missing ai_context_usage table...

Drupal\Core\Database\DatabaseExceptionWrapper: Exception in AI Context Usage[ai_context_usage]: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'db.ai_context_usage' doesn't exist: SELECT COUNT(*) AS "expression" FROM (SELECT 1 AS "expression" FROM "ai_context_usage" "ai_context_usage") "subquery"; Array ( ) in main() (line 19 of index.php).

This is after I ran the database updates...

  1. Install ai_context@1.x-dev
  2. Switch to the MR!35 branch
  3. Run database updates ddev drush updatedb
  4. Cleared the caches ddev drush cr
  5. Visit admin/ai/context/usage: ddev launch admin/ai/context/usage

Need the schema definition?

scott falconer’s picture

Status: Needs work » Needs review

Updated MR !35:

Addressed the upgrade regression where visiting /admin/ai/context/usage failed with a missing ai_context_usage table after running updates.

Changes in this update

  • Added ai_context_update_8004() in ai_context.install to install ai_context_usage entity schema on existing sites when missing.
  • This is separate from the install-time workflow bug in #3571188 / MR !51; that fix remains tracked there.

Reproduction and verification

  • Reproduced the issue on a fresh Drupal 11 site:
  • Install drupal/ai_context:dev-1.0.x
  • Switch module code to MR !35 branch
  • Run drush updb -y and drush cr
  • Querying ai_context_usage failed with SQLSTATE[42S02] (table missing)
  • Applied this update-hook fix and reran the same flow:
  • drush updb -y ran ai_context_update_8004
  • SHOW TABLES LIKE 'ai_context_usage' returned the table
  • Entity query on ai_context_usage succeeded (count returned 0)
  • Testing (local DDEV)
  • ddev exec php -l web/modules/contrib/ai_context/ai_context.install : PASS
  • ddev exec vendor/bin/phpcs -s --standard=Drupal,DrupalPractice web/modules/contrib/ai_context/ai_context.install : PASS
  • ddev exec drush updb -y : PASS
  • ddev exec drush cr : PASS
  • ddev exec bash -lc 'SIMPLETEST_DB=mysql://db:db@db/db SIMPLETEST_BASE_URL=https://canvas-dev.ddev.site:8443 BROWSERTEST_OUTPUT_DIRECTORY=/var/www/html/web/sites/simpletest/browser_output vendor/bin/phpunit -c web/core/phpunit.xml.dist web/modules/contrib/ai_context/tests' : PASS (49 deprecations on Drupal 11.3.x)

AI disclosure
This work was AI-assisted (analysis/drafting), with human review + local testing.

kristen pol’s picture

Assigned: Unassigned » kristen pol

Thanks! I'll take a look.

Please note that this module is NOT used other than for testing, so I have intentionally NOT been including update hooks. We are on the "innovation" side right now and not doing all the standard things we'd do if this were already a used module. The "alpha release" was simply a tag to let people try the Vienna DrupalCon version with the v2025 repo. The code has been pretty much rebuilt since that time.

kristen pol’s picture

Assigned: kristen pol » Unassigned
Status: Needs review » Fixed
StatusFileSize
new160.5 KB
new232.7 KB

Merged! Works great, and love the new usage record page. Thanks!

context control center usage list with one record listed

context control center usage record with agents, tools, and context

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

kristen pol’s picture

Status: Fixed » Closed (fixed)
kristen pol’s picture

Issue tags: -priority