Migrate the inner workings of Surge into AI Best Practices

I'm happy to mark this as fixed, as surge is now unsupported and I'll be focussing my efforts here.

The individual parts of Surge I can bring over I'll address in different issues where they might become a solution.

Surge was setup as an umbrella, a framework to deal with all of the scattered resources and being able to orchestrate that to a good experience. Where we're focussing on a strong default setup, and I think the vision as outlined really perfectly matches with the experience I've gained in Surge.

Which means that the bigger parts in Surge might not apply for this project.

It might be good if we could have a summary of those things here? Or in Surge? If you need help with that I could ask Catia to follow you and do it manually.

I found the Surge install process and default skills to be better than AI Best Practises right now and so there is a lot of good stuff there. I still wonder if we want to ressurect Surge and have AI Best Practises rely on it, keeping Surge for Devs and AI Best practises becoming for the OOB newbie experience.

Thanks Jamie!

I'm actually bringing in parts of Surge in here already. A little bit of a lightweight version of it, that fits the personas and scenario's were looking for.

For example: https://www.drupal.org/project/ai_best_practices/issues/3585618
Planning to do the same for Skills.

Surge was really an orchestration platform, but the more we're discussing the vision and requirements the more we need some of the parts in Surge, to ensure we bring everything together, move it to the right place, get it installed but also allow for a level of personalization. We could keep Surge for that, but I think we'll move towards a place where we will need a lot more in our harness whether that is a CLI, MCP tools or whatever, to get everything in `drupal/ai_best_practices` to work. So this will become bigger at one point before we might be able to move things out again and let that stabilize in the ecosystem.

Just thinking out loud here, but it might also be that we can just bring in the Surge features. If that makes the story and set up easier to get started, and trim it back whilst we move along and gather that feedback.

Re-opening for the sake of it, so I can attach some more documentation to help with that discussion.

[outdated]

Skills Module

Aggregates Agent Skills from multiple sources into .agents/skills/ by default for use with Claude Code, Cursor, OpenCode, and other AI tools.

Default output_directory follows agentskills#15; you can still set output_directory (for example .claude/skills) in surge.yaml.

How It Works

1. Discovery: Scans configured sources for .agents/skills/, .claude/skills/, skills/, and other configured layouts containing skill folders
2. Validation: Validates each skill's SKILL.md per Agent Skills spec
3. Aggregation: Copies configured and validated skills to your output_directory (default .agents/skills/)

Skill Sources (Priority Order)

Skills are discovered from multiple sources with priority handling:

1. Project-level (.agents/skills/, .claude/skills/, or skills/) - Highest priority, can override all others
2. Drupal modules (same directory names under modules/* or web/modules/*) - Custom module skills (enabled by default)
3. Composer packages (same under vendor/*) - Third-party package skills (enabled by default)

Skill Filtering

All skills are discovered by default. You can exclude specific skills:

• Exclude: Exclude specific skills (format: 'source-name:skill-name')

This is useful when a module ships multiple skills but you only want some of them, or have your own more specific Skills.

Configuration

Configure skill sources in surge.yaml (created automatically with sensible defaults):

skills:
  # Output directory where aggregated skills are installed (default: .agents/skills)
  output_directory: .agents/skills
  
  # Discovery sources (both enabled by default)
  discover_composer: true
  discover_modules: true
  
  # Directories to exclude from scanning (in addition to defaults: .git, node_modules, .idea)
  excluded_directories:
    - dist
    - tmp
    - build
  
  # Exclude specific skills (format: 'vendor/package:skill-name' or 'module-name:skill-name')
  exclude_skills:
    - drupal/views:views-legacy
    - my_module:deprecated-api

Most users won't need a config file at all since defaults are sensible. Everything is discovered by default; use exclude_skills to opt out of specific skills.

Adding Git Repositories Without composer.json

For Git repositories that don't have a composer.json file or tagged versions, you can use Composer's package repository type to add them directly to your composer.json. Surge will automatically discover skills from any Composer package.

Example: Adding drupal-claude-skills repository:

{
    "repositories": [
        {
            "type": "package",
            "package": {
                "name": "grasmash/drupal-claude-skills",
                "version": "1.0.0",
                "type": "library",
                "source": {
                    "url": "https://github.com/grasmash/drupal-claude-skills.git",
                    "type": "git",
                    "reference": "main"
                }
            }
        }
    ],
    "require": {
        "grasmash/drupal-claude-skills": "^1"
    }
}

Auto install / update Skills

Surge will automatically discover and update skills from packages you install or update. Once you require it, it will add the following to your composer to make that work.

{
  "scripts": {
    "post-install-cmd": ["@composer surge skills-install"],
    "post-update-cmd": ["@composer surge skills-install"]
  }
}

Commands

• composer surge skills-install - Install/update skills from all configured sources
• composer surge skills-list - List available skills with validation status
• composer surge skills-list --format=xml - List available skills, formats them as XML for file based tooling.

<available_skills>
  <skill>
    <name>drupal-agents-generator</name>
    <description>Generates a composable AGENTS.md file for Drupal projects by detecting the development environment...</description>
    <location>/surge/surge/skills/drupal-agents-generator/SKILL.md</location>
  </skill>
</available_skills>

Skill Format

Each skill is a directory containing SKILL.md with YAML frontmatter:

skills/
└── my-skill/
    ├── SKILL.md          # Required: name, description, instructions
    ├── scripts/          # Optional: executable code
    ├── references/       # Optional: documentation
    └── assets/           # Optional: templates, resources

See Agent Skills Specification for details.

Validation

Skills are validated per the Agent Skills spec validator over at ronaldtebrake/agent-skills-validator

After a back and forth on this, it seems it's better to just bring in Surge. (background in Slack)

This will make it easier to collect feedback and have an installable module, whilst improving that developer tooling as well.
Jamie & team already have some amazing insights that could improve what Surge has. So having the code shipped in here will make it easier to create specific issues and modify it and move faster.

I'll use this issue to do the migration.

Coming over from @webchick's mention in #3584914. I shared my approach there in case it helps as prior art: https://www.drupal.org/project/ai_best_practices/issues/3584914#comment-...

Reading this thread, the direction is interesting and there's a lot of overlap with what I've built. One thing I want to push back on gently though: I'm not fully convinced this should install via Composer. To me this isn't really a concern of an individual project, it's something common to any Drupal project and lives much closer to how the user is running their CLI (Claude Code, OpenCode, Codex, whatever).

The reason I say this is that skills and agents are inherently tied to how the developer works locally. A skill for clearing caches or importing config isn't the same command if you're inside a DDEV container, inside a Docker Compose setup, or running straight on the host. So before building out the skills and agents system, I think the first question is: which local environment are we assuming? DDEV, Docker Compose, something else? That decision shapes everything downstream.

Related to this, another thing to keep in mind: agents and tool calls are specified differently across CLIs. Claude Code and OpenCode don't use the same frontmatter or the same tool names, and I haven't checked Codex myself but I'd expect it to be its own thing too. You can't use the same agent prompt file across all of them and expect it to work. If the plan is for Composer to scan project directories and aggregate everything into a single skills folder, you're going to hit the problem that the same prompt isn't actually reusable across tools. I've solved this with a simple fat frontmatter + envsubst approach (explained in the comment linked above), but it's worth factoring in before the skills aggregation model gets set in stone.

My vote is DDEV, which is what I use and why I went with a DDEV add-on approach rather than a Composer plugin. Surge's autodetection is nice, but I'd argue picking one baseline and designing the skills around it gives a much cleaner result than trying to stay environment-agnostic with conditionals everywhere.

Happy to dig into any specific piece if useful.

Thanks @trebormc, appreciate the feedback!

I think those are very valid points. Personally, I do not think we need to decide that right now for this issue to move forward.

As you can see, we are also still very much discussing the extent of those boundaries: https://www.drupal.org/project/ai_best_practices/issues/3584914#comment-...

The goal of merging this is to move fast and give ourselves something concrete to work with for the MVP, and for that we chose Surge. But we are very open to changing this later, even at a larger architectural level.

I am not bound to Composer, even if it is part of Drupal's toolstack. I am also not bound to the current architecture. That is also why I sunsetted Surge in the first place: I came to the conclusion that auto-detection is not what we need.

For now, it is simply easier to give feedback on an existing system than to stay stuck in decision paralysis with all the great tools out there. At least this gives us something people can install, test, and give feedback on, while already benefiting from things like Agents and Skills.

We can and will follow up on this with refactors to the tooling side of the system.

Does that work for you?

Thanks @ronaldtebrake, that works for me. Shipping an MVP to iterate on is more valuable than architectural paralysis.
The only thing I'd flag is to avoid building twice when we already have some clarity. If we know users will want to pick between Claude Code, OpenCode, Codex, etc., the agent format shouldn't assume a single CLI from day one. And if skills and agents really belong to the user's environment rather than to the project, that points away from a Composer-installed model fairly early on. Worth keeping in mind so MVP choices don't lock us in.

Thanks @trebormc!

I think it's fair that a decision like that will be a blocker for a proper release.
For now, since it's composer based it also means it is relatively easy to remove again.
Based on that I've added additional test coverage to make sure the traces to the composer part of this tooling will be removed on

composer remove drupal/ai_best_practices

https://git.drupalcode.org/project/ai_best_practices/-/merge_requests/17...

Think we can get this in, will be able to update the documentation and perhaps make a demo videobut for the sake of dev days contribution I hope the above is enough to get up and running and start playing around with it and get working on those follow ups

Thanks for the feedback :D !

(this may be out of scope for this issue) I expected some changes to README.md installation instructions out of this to account for "new site" install (MVP use case #2 & #3), but it remains running some composer commands against an existing site. That still leaves quite a lot up to someone new/returning to a) figure out that Drupal and Drupal CMS are two different things, b) run across the fact that DDEV exists and is the thing to use now, c) install all of that correctly, d) also to have composer installed.

That's definitely something we should add, it's not what the inner workings of Surge ever did, but we should create a feature request out of it for sure. It's connected to what @trebormc mentioned above. I expect it'll be DDEV and will likely mean we'll throw things around from what we're merging here. But the work that we're doing here means we can (for now) easily install the AGENTS.md and Skills on existing websites with one composer command, which is far from what we're trying to achieve with the project but hopefully adds enough value to start testing the Skills and AGENTS.md work we're doing already. I'll work on the documentation a bit more for what the project is with this MR in it.

^ Anyway, so ALL OF THAT lol is what I'd love to eliminate in favour of one command (eventually :)).

FWIW I think this line of you're lovely journey :D sums up where we want to take the project!

I was also surprised in scanning through to see the very many .agents/templates/environments — ddev, lando, docker-compose, ... my expectation had been we're picking _one_ of those (most likely DDEV) to recommend for _all_ users (perhaps overridable in the YAML config file). But perhaps this is naive. :)
Another surprise was that things in the .agents/templates/ folder seem to be things that the current code base would keep in /skills? Or are they different in your mind? And if so, could we add some guidance to CONTRIBUTING.md so folks know where to add what in the future?

Good point, and you're right I think we'll ship one recommended environment. Part of why I sunsetted Surge is that it wasn't picking anything and trying to work with all flavors out there. I'll actually remove that AGENTS.md generation based on those templates from the MR, it moves the discussion in the wrong direction and away from our vision, and in turn keeps it light :D

This is a complete nit, but could we make apb something like drupal-ai or ai-start or anything with "ai" in the name? :)

On it :D

I've added https://www.drupal.org/project/ai_best_practices/issues/3586440 so we can start creating issues to move towards our vision and get the above steps done in one command :D

Tested MR17 on a fresh Drupal CMS project on DDEV. One finding I do not see in the thread.

AGENTS.md does not enumerate module-discovered skills. The plugin correctly synced create-functional-js-test from web/modules/contrib/ai/.agents/skills into .agents/skills/. The skill file is on disk. But AGENTS.md only lists the two skills shipped by ai_best_practices itself. Agents reading AGENTS.md will not learn that a module has contributed a skill. The module discovery feature half-works: it copies the file but does not surface it where the agent actually looks. This seems like a bug rather than a doc gap.

One thing that may help debug @webchick's "Command 'abp' is not defined" in #19. My install completed with abp registered. Two variables differ between our setups:

• Source type: VCS source against the issue branch on git.drupalcode.org (mine) vs local path repo (hers). Path repos go through a different composer code path with symlinks and a "dev source" flag. Post-merge users will install from drupal.org's composer endpoint, which is closer to the VCS source path than to the path repo path. So if the path repo is the trigger, real users will not hit this after merge.
• Composer execution context: ddev composer in the container (mine) vs composer on the host (hers). Drupal devs split roughly evenly between these. If host-vs-container is the trigger, real users will hit it.

Worth checking which axis matters before merge, since the user impact differs sharply between them.

Thanks @zorz!

AGENTS.md does not enumerate module-discovered skills. The plugin correctly synced create-functional-js-test from web/modules/contrib/ai/.agents/skills into .agents/skills/. The skill file is on disk. But AGENTS.md only lists the two skills shipped by ai_best_practices itself. Agents reading AGENTS.md will not learn that a module has contributed a skill. The module discovery feature half-works: it copies the file but does not surface it where the agent actually looks. This seems like a bug rather than a doc gap.

That's correct, by syncing it to `.agents/skills` we're matching the specification https://github.com/agentskills/agentskills/issues/15
Which means it should be discoverable by (and progressively disclosed to) almost all tools, without the need for adding it to AGENTS.md and costing extra tokens.

I'm happy to fix that as a bug after this gets in if we want all the skills to also end up in there, but perhaps we could also just make sure that the default output directory (where the skills land) changes for those tools that don't follow the proposed specification yet.

It also seems to match a direction that scott is outlining https://www.drupal.org/project/ai_best_practices/issues/3585585#comment-16564365, even if on the specifics we need to iterate, I think it indicates we're adding some good value to iterate on.

Worth checking which axis matters before merge, since the user impact differs sharply between them.

If I'm not mistaken once this is merged it shouldn't matter, perhaps it's just best to merge it in and create follow ups from there so it's easier to test and use. For what it's worth this is how I was testing it locally:

# Make sure you're on the MR branch checked out.
AI_BP_ROOT="$(pwd)"   # must be absolute; run from the package root
mkdir -p tests/smoke-workspaces
SMOKE="$(pwd)/tests/smoke-workspaces/smoke-$(date +%s)-$(openssl rand -hex 4)"
mkdir -p "$SMOKE"

cd "$SMOKE"
composer init --name=example/agents-smoke --type=project --no-interaction -n
composer config allow-plugins.drupal/ai_best_practices true
REPO_JSON=$(printf '{"type":"path","url":"%s","options":{"symlink":false}}' "$AI_BP_ROOT")
composer config repositories.ai_bp "$REPO_JSON"
composer require --no-interaction -n --dev 'drupal/ai_best_practices:@dev'
composer drupal-ai skills-list

Once this is in though, that symlinking and juggling won't be needed we could just use any way of composer requiring this.

Thanks @webcheck

In line with:

- I’m cool with anyone in leadership merging in their own code as long as they talk out big changes like we’ve done here… I want to keep momentum up, and we can continue to iterate.

I've merged it, to make sure I could double check,
I've verified locally that for me the install instructions now do the trick:

composer config allow-plugins.drupal/ai_best_practices true
composer require --dev drupal/ai_best_practices:@dev

I can now see the generated files immediately after running the above:

	.agents/
	AGENTS.md
	CLAUDE.md
	GEMINI.md
	ai_best_practices.yaml

and the contents match what we ship with the 1.0.x branch