Problem/Motivation

Token count calculation is a common feature which could be included in ai_utilities.

I calculate basic token count of code here: https://git.drupalcode.org/project/aidev/-/blob/1.0.x/src/Source.php?ref...
And also have a feature request to calculate prompt token count here #3440408: Calculate prompt token count
It would be nice to improve prompt count calculation to make it more accurate, because basic calculation averages are different for text/code/etc.
Token calculation could also take into account nuances of different LLMs.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Comments

mindaugasd created an issue.