Problem/Motivation

#3582953: Document how to run evals in various scenarios introduces an evals/compare.py script that compares before/after on various agent guidance rules.

However, the current implementation is locked to Claude Code. Given Anthropic is currently making a bunch of changes to pricing and usage limits in real-time to deal with third-party tool usage from e.g. OpenClaw: https://x.com/bcherny/status/2040206440556826908 it is probably better for our tools to remain provider-agnostic so developers have the freedom to switch from Claude to Codex for example.

(See also this Slack discussion where Drupal folk are discussing swapping back and forth based on capabilities / pricing / usage limits https://drupal.slack.com/archives/C0APH70JV18/p1775230385265339?thread_t... — this is a common pattern around developers using AI for development.)

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Issue fork ai_best_practices-3583202

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

webchick created an issue. See original summary.

zorz’s picture

zorz commented
Assigned: Unassigned » zorz
zorz’s picture

zorz commented

As soon as https://git.drupalcode.org/project/ai_best_practices/-/merge_requests/8 gets merged I have a ready MR for this.

zorz opened merge request !12

zorz’s picture

zorz commented
Status: Active » Needs review
ronaldtebrake’s picture

ronaldtebrake commented
Status: Needs review » Needs work

Oops was a bit too eager with the approval earlier was just starting the review, left some comments, could you take a look @zorz?
Sorry if my python is bit too rusty and the questions don't make sense.

For what it's worth I think the value is going to be great to have in especially what it opens up for DevDays and other providers! :D

zorz’s picture

zorz commented
Status: Needs work » Needs review

Ronald thank you for the great review! I resolved the issues. Maybe I should only comment on them? Please re-review

ronaldtebrake’s picture

ronaldtebrake commented
Status: Needs review » Fixed

Thanks George amazing, I've merged it. Think it will be very valuable to have before the DrupalDevDays and looks like you've addressed all my concerns

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.