New token types require a schema migration each time; add a token_details column for extensible capture

Problem/Motivation

ai_metering_usage has three dedicated integer columns for token counts:
input_tokens, output_tokens, and cached_tokens.
Each time a provider introduces a new token type — reasoning tokens for extended
thinking, audio tokens, image tokens — adding it to the schema requires a new
hook_update_N(), a new column in hook_schema(), and
changes to every place that reads or writes usage records.

This pattern will not scale. drupal/ai is adding providers and capabilities quickly
and token type proliferation is expected. Raising the schema version for each new
type creates upgrade friction for site owners and makes the data model increasingly
rigid. Marcus Johansson noted this explicitly in his review of the module.

Steps to reproduce

1. Enable extended thinking on an Anthropic call that supports it.
2. Make an AI call and let it complete.
3. Inspect the ai_metering_usage row — there is no reasoning_tokens column; the reasoning token count returned by the provider is silently discarded.

Environment

• Drupal version: 11.3.11
• ai_metering version: 1.0.0-alpha1
• AI provider module + version: ai_provider_anthropic 1.2.2 (extended thinking enabled)
• AI feature module tested: N/A

Proposed resolution

Add a token_details column of type text (JSON serialised) to
ai_metering_usage. The three existing columns
(input_tokens, output_tokens, cached_tokens)
remain unchanged for backward compatibility and existing dashboard queries.
Any additional token type reported by the provider — for example
reasoning_tokens or audio_tokens — is written into
token_details without a schema change.

AiPostGenerateSubscriber reads the full usage object from the
provider and writes any key it does not recognise into token_details.
A helper method returns all token types merged from both the columns and
token_details, so the dashboard and Drush commands can surface extras
without knowing column names in advance.

Remaining tasks

• Add token_details text/JSON column to ai_metering_usage via hook_update_N() and hook_schema().
• Update AiPostGenerateSubscriber to write unrecognised token types into token_details.
• Add a helper method that returns all token types merged from columns and token_details.
• Update dashboard and Drush output to surface extra token types when present.
• Add a unit test covering at least one extra token type round-trip.
• Review.

User interface changes

Yes — the dashboard and AI Usage Log surface additional token types (e.g. reasoning_tokens) when they are present in the data.

API changes

N/A

Data model changes

Yes, a token_details text column (JSON) is added to ai_metering_usage via hook_update_N(). Existing rows have NULL; no data is lost.

AI assistance

N/A