How to make your project AI-driven

The thinking tool

Five levers, and how to choose between them

Almost every question of the form “how do I make it stop doing X?” is really the question “which lever does X belong to?” Getting that choice right matters more than how well you write the thing itself. If you take one idea from this page, take this one.

Ask these in order, and stop at the first yes:

1. 1.Can a script check it? → hook. Zero context, cannot be forgotten at turn two hundred, applies to every sub-agent automatically. This is the most under-used lever, because prose is easier to write — and teams respond to prose not working by writing more prose.
2. 2.Only needed sometimes? → skill. Forty skills cost about what one paragraph of AGENTS.md costs, because you pay for descriptions and nothing else until they fire.
3. 3.Needs its own context window or a different model? → agent. Not merely different instructions — if that is the only difference, you wanted a skill.
4. 4.Always true, non-obvious, and something goes wrong if unsaid? → instructions. Everything else is sentiment; delete it.

There is a fifth lever most people never set deliberately, and it is free. Once something is a skill, it is model-invoked by default — its description sits in the context window on every turn so the agent can fire it autonomously. Setting disable-model-invocation: true strips that description from the agent's reach entirely: zero context cost, and only a human typing its name can invoke it.

Stage 0

Install the workflow. Build only the enforcement.

The instinct is to build your own pipeline: a spec stage, a planning stage, an implement loop, a reviewer. Resist it for one round. Most of that already exists, is maintained by someone else, and is better than your first attempt — including ours.

Matt Pocock's skills repo is the current best set for real engineering work: wayfinder for charting work too big for one session, to-spec, to-tickets, tdd, implement, code-review, grilling. Install it and start from there.

npx skills@latest add mattpocock/skills
# choose your agent (Copilot CLI, Claude Code, Codex…)
# keep setup-matt-pocock-skills — everything else depends on it

# then, in a session:
/setup-matt-pocock-skills

What that set does *not* have is enforcement. Every one of its skills is advice, because that is all a skill can be. Nothing stops a destructive command, a stale clone, a source-only change, or a wall of tool output consuming the context window. That gap is what you build. The reusable part is in the starter kit; the rules unique to your team still belong to you.

Stage 1

Define “done” before you automate anything

If your definition of done is tribal, agents produce plausible garbage at speed, and review becomes the bottleneck that eats the entire gain. Write the rules down first, in the file your agents already read.

• —Blast radius: a change touches only files the current task declared. No drive-by refactors, no opportunistic formatting.
• —Small PRs: one ticket equals one branch equals one pull request. Over ~400 changed lines outside generated paths, it splits.
• —GitHub Flow: main is always deployable. Short-lived branches, merged via PR, deleted after.
• —Tests first: a failing test exists before implementation, visible in the commits.
• —Always shippable: incomplete work merges behind a flag, defaulted off — never behind a long-lived branch.
• —Comments are a last resort. Name things properly instead. A comment is allowed only where the code cannot express why — a non-obvious constraint, a workaround with a ticket reference.
• —Conventional Commits, with the work item id in the footer so traceability survives the squash.

Stage 2

Build a workspace, not another repository

When a feature spans an internal WSO2 service, an external one, a BFF, a React frontend and a Kubernetes repo, no single repo holds the truth. The fix is a thin workspace repo containing no product code — only the coordination layer — with the others cloned into a gitignored directory.

ai-workspace/
├── .github/
│   ├── skills/          # installed set + the few you write
│   ├── hooks/*.json     # the part that is yours
│   └── mcp/.mcp.json    # Azure DevOps, split by domain
├── docs/agents/
│   └── issue-tracker.md # the tracker contract every skill reads
├── repo-map.yaml        # area path + label -> repos, owners, pipelines
├── scripts/sync.sh      # fast-forward every clone, in parallel
├── AGENTS.md
└── repos/               # gitignored clones
    ├── wso2-internal/  AGENTS.md
    ├── bff/            AGENTS.md
    ├── frontend/       AGENTS.md
    └── k8s/            AGENTS.md

repo-map.yaml is the file you must write yourself, and nothing works without it. It turns “work on #4821” from a guessing game into a lookup: which repos are in scope, who owns them, which pipeline validates them, which contracts cross between them. An agent could grep across repos to work this out — it would usually be right, occasionally confidently wrong, and would burn a lot of context getting there.

Stage 3

An AGENTS.md at every level, and the gotcha under it

Resolution is nearest-wins: the file closest to the code being edited takes precedence. In a multi-repo workspace that is exactly right — NestJS conventions apply in the BFF, React conventions in the frontend, and neither pollutes the other's context.

Keep it brutally short — this file is in context on every request, so every line is a tax paid forever. Under 40 lines. Write only what is true, non-obvious, and would cause a wrong change if unstated.

# BFF (NestJS)

- Domain folder per module: controller, service, dto/, *.spec.ts beside source.
- Controllers validate and map only. No business logic.
- Outbound calls go through src/clients/*, never HttpService directly.
- openapi/bff.yaml is the contract. Changing a DTO without it is a bug.
- Test: npm run test -- --findRelatedTests <changed files>
- Never edit generated files. Never add a dependency without an ADR.
- Never touch src/migrations/ unless the task says so.

Stage 4

Give agents your systems, not your shell

If agents reach Azure DevOps by shelling out to the az CLI, every work item read dumps raw JSON into context. The Azure DevOps MCP server exposes the same data as structured tools, grouped into domains: core, work, work-items, search, repositories, wiki, pipelines, test-plans, advanced-security.

The optimisation that matters is scoping. MCP tool definitions are themselves context — every tool you expose is paid for on every turn, used or not. So do not load the whole server globally. Split it by domain and give each agent only what it needs: a planner gets work-items, a pipeline-doctor gets pipelines and cannot accidentally close a ticket.

Separately, the skills you installed need to know how your tracker works. Good skill sets abstract this into a single contract file — one document describing how to create, read, label, link and query. Porting to a new tracker means writing that one file, not editing every skill.

# 1. System.Tags REPLACES. A naive write drops every existing tag.
existing=$(az boards work-item show --id 4821 \
             --query "fields.\"System.Tags\"" -o tsv)
az boards work-item update --id 4821 \
  --fields "System.Tags=${existing:+$existing; }ready-for-agent"

# 2. There is no blocked-by summary field. Unlike GitHub, you must resolve
#    each predecessor and check its state yourself.
az boards work-item show --id 4823 --expand all -o json \
  | jq -r '.relations[]?
           | select(.rel=="System.LinkTypes.Dependency-Reverse")
           | .url | split("/") | last'

Stage 5

Refinement: one question at a time, with your answer attached

“Let's work on #4821” is where AI-assisted work usually goes wrong. The agent reads a two-line ticket, infers the rest, and starts editing. Everything downstream inherits invented assumptions.

The fix is an interview — and two rules make it work. Ask one question per turn. Several at once is bewildering, and it destroys the ordering between decisions: the answer to question two often determines whether question five is worth asking at all. Attach your recommended answer to every question. That is what makes one-at-a-time cheap, because the human is confirming rather than composing.

One more rule: if a *fact* can be found in the environment — the filesystem, the board, the contracts — look it up. Never ask what you can read. Only *decisions* go to the human. An interview that asks what the model could have discovered is an interview people learn to skip.

Stage 6

Tracer bullets, not layers

Small pull requests are a decomposition problem, not a discipline problem. By the time anyone is writing code the size is already decided. The only place PR size is controllable is here.

Slice vertically. Each ticket cuts a narrow but complete path through every layer it touches — contract, service, UI, tests — and is demoable on its own. The tempting alternative is to order by layer: contract first, then the BFF, then the frontend. That is a horizontal slice, and every intermediate ticket delivers nothing anyone can see.

HORIZONTAL — tidy PRs, nothing demoable until the last one
  1. extend contract          -> nobody can see anything
  2. BFF accepts payload      -> nobody can see anything
  3. FE regenerates client    -> nobody can see anything
  4. FE form                  -> feature appears, all risk lands at once

VERTICAL — awkward PRs, every ticket verifiable
  1. submit a claim with one deferred field, end to end   [demoable]
  2. add remaining deferred fields                        [demoable]
  3. validation on completion step                        [demoable]

Two more rules earn their place. Every ticket declares the files it may modify — that list is a review checklist, a branch boundary, something a hook can enforce, and a decomposition test all at once. A ticket that cannot name its files is not one ticket. And every ticket declares its blocking edges, so the pipeline can work the frontier — any ticket whose blockers are done — instead of marching down a list.

Stage 7

Hooks: the part nobody can give you

This is what separates a team with an AI tool from an AI-driven project. Hooks run at lifecycle points and can refuse: preToolUse returns a deny decision and agentStop can block a turn from finishing. Your standards stop being a document the agent is asked to remember and become a property of the runtime.

Five carry almost all the value:

• —Session baseline — snapshots the current commit and inherited dirty files, so later checks judge the agent's work without claiming the developer's unfinished changes.
• —Workspace sync — finds the shared repo map, then fast-forwards only clean clones and refuses paths outside the workspace.
• —Test coverage gate — on agentStop, blocks a source change when no behavioral test changed in the same session. A continuation can explain a genuinely untestable change.
• —Dangerous commands — a short denylist: force pushes, branch deletion, destructive az and kubectl verbs, production namespaces. Keep it short; a long denylist means you wanted sandbox mode.
• —Output trimming — rewrite tool output before it reaches the context window. Pipeline logs and kubectl JSON are mostly noise, and this applies to every agent and sub-agent automatically, which no amount of instruction-writing achieves.

Stage 8

Model diversity beats model size

The pipeline generates an artifact and then checks it, repeatedly: plan then analyse, implement then review. If the same model does both, the check carries almost no information — a model asked to find fault in its own reasoning re-derives that reasoning and concludes it is sound. The error and the audit share a blind spot.

So if you have Claude Sonnet 5 and the GPT-5.6 family but not the largest tiers, that is not the constraint it looks like. Cross the families at every verification boundary and your ceiling is set by diversity, not size.

Stage 9

Feature flags are what make small PRs possible

Small pull requests and a deployable main look contradictory: half a feature cannot ship. The usual resolution is a long-lived branch, which reintroduces everything GitHub Flow exists to avoid. Flags dissolve the tension — ticket-sized work merges the day it is written and sits dark until a human turns it on.

For agents there is a second benefit: the flag bounds the damage. Autonomous work that merges behind a default-off flag has a blast radius of zero until someone decides otherwise. That property is what lets a risk-averse organisation say yes to any of this.

The production-readiness gate is a fixed, mechanical checklist — not a judgement call — run before any PR opens:

• —Flag exists, defaults off in every environment, key recorded in a flags document.
• —Tests cover both sides of the flag. A suite that only exercises the on-path means the rollback is untested and the flag was decoration.
• —Migrations reversible, rollback independent of the code change.
• —The new path emits at least one log line and one metric. An unobservable feature cannot be safely enabled.
• —Contract updated and downstream client regenerated in the same PR.
• —No secrets, no new dependency without an ADR, no TODO introduced, no comment restating its own code.
• —A removal task exists on the board for the flag.

Stage 10

Autonomous loops, and the one thing Ralph gets right

By now the flags mean autonomous work has a blast radius of zero until a human turns it on. That is the precondition for the question everyone asks early and should ask here: can we just let it run? The best-known answer is the Ralph technique — Geoffrey Huntley, May 2025 — and it is one line of bash.

while :; do cat PROMPT.md | agent; done

It looks like a joke, and the interesting part is not the loop. It is that every iteration is a fresh context window loading the same files. Progress accumulates in the filesystem, in git, and in one plan file — never in a conversation. So the agent always works from the top of its window instead of the diluted middle of a long session.

The second thing it gets right is why it converges at all, and it is not the prompt. Tests, typecheck, lint and build reject wrong work without a human reading it. The loop generates; the checks refuse. Which means the loop is worth exactly as much as your enforcement layer — and if you built stages 7 and 9, you already own the expensive part.

What does not transfer is running it as Huntley does. Vanilla Ralph is unbounded, picks its own next task from a plan file, and runs with tool approval disabled because an interactive prompt stalls the loop. Each of those collides with something earlier on this page: small reviewable PRs, the tracker as the single authority on what is in flight, and a hook layer whose entire value is refusing. So bound it on three axes, and keep the loop's plan file as scratch rather than letting it become a second place work state lives.

./loop.sh plan          # gap analysis into .ralph/, no source edits
./loop.sh build 5       # at most 5 tasks, one commit each
touch .ralph/STOP       # stop after the current iteration, not mid-commit

# and it refuses to start at all on: a dirty worktree, main,
# a missing plan, or a machine that is not a container

• —Iteration cap — bounds the diff to something a human will actually review. If runs stop at the cap rather than an empty plan, the ticket was too large: fix the slicing, not the cap.
• —Wall-clock cap — so nothing is still running when you get back.
• —A stop file — touch .ralph/STOP finishes the current iteration and exits. Ctrl-C can leave a half-written commit.
• —A human starts it. Expose it as a user-invoked skill so no agent can decide to put itself in a loop.
• —One repo, one ticket. A loop spanning clones cannot produce the coordinated PRs a vertical slice needs.

One brownfield trap. Point a loop at legacy code with no spec and it will infer one from the implementation — bugs included, promoted to requirements. Reverse-engineer first, in a planning pass that writes specs and edits nothing, and read what it wrote before you build on it. Document reality, not intent.

Stage 11

When the team is backend-only and the frontend is React

This is the most common shape of the problem and the one teams handle worst. The instinct is to ask backend engineers to become adequate frontend engineers with an agent's help. That fails, because the missing thing is not typing speed — it is judgement. Knowing that a form needs an error summary, that a modal needs focus management, that this spacing is wrong. An agent will happily produce plausible React that a backend reviewer cannot evaluate.

1. 1.Contract-first, so most frontend work is mechanical. NestJS emits the OpenAPI document; the React client and its types are generated from it. Generated files are never hand-edited and are excluded from diff-size checks. A large share of frontend changes become regenerations a backend engineer can review with confidence.
2. 2.Encode taste as a skill. Carry what the team does not have in its head: the component inventory and when to reach for each, spacing and token rules, the accessibility checklist, and an explicit list of things never to hand-roll — modals, date pickers, toasts, form validation.
3. 3.Replace judgement with verification. An agent that runs the app, drives the new flow, screenshots it, and checks it against the list — including keyboard-only traversal and an automated accessibility pass. Backend engineers cannot reliably eyeball a React diff, but they can absolutely read a failed check.

Stage 12

Let it improve itself — without growing

A system that does not learn is just faster at the same mistakes. But the obvious implementation — an agent appending to a lessons file — is a trap, and it is exactly the trap that produces the thousand-line instruction files nobody wants. Every appended line is paid for on every future request, most stop being true within a month, and nothing ever removes one.

The discipline is promote or discard. A lesson is not a note; it is a proposal to change one thing, in strict order of preference:

Harvest from what you already produce: human comments on merged PRs, escalations, reverts, and reviewer findings that recurred. Only things that happened at least twice are eligible — one-offs are sampling noise, and encoding them fills your instructions with superstition.

The metric that tells you learning is real is recurrence: how often does a reviewer raise a finding it already raised last month? If that number is not falling, lessons are being recorded rather than absorbed — which usually means they went into prose when they should have become hooks.

Stage 13

Roll out narrow, and measure things that would embarrass you

Do not build every agent before shipping anything. Pick one ticket archetype that genuinely crosses your repos — “add a field end to end, from the external contract through the BFF to the React form” — and build the complete chain for only that. It will be unglamorous and it will expose every wrong assumption in your repo map and your instruction files. Fix those, then generalise.

• —Median PR size in changed lines — should fall, not rise. If it rises, ticket decomposition and review are not working.
• —Time from ticket to first review-ready PR.
• —Share of PRs merged without a scope-creep comment.
• —Revert and hotfix rate — whether the speed is real or borrowed.
• —Recurrence: how often a reviewer finding repeats month to month.
• —Open flags older than 30 days — the debt this whole model runs on.

When it works, package it. A plugin bundles skills, hooks and MCP configuration behind one manifest and installs declaratively, so every team gets the same guardrails without reading a setup document nobody reads. That is what turns one team's setup into a platform capability.

Reference

Failure modes