ctx-budget

Readme

# ctx-budget

**What will your agent's context cost before it runs — and what is eating it?**

It finds the text your agent loads, measures it, attributes it per source, and
separates what you pay on **every request** from what you only pay when
something is loaded. Then it gives you an exit code you can put in CI.

```bash
npx --yes github:liyixuan201211/ctx-budget --help
```

As a DSH plugin (installs the skill, not just the CLI):

```bash
dsh plugin --profile web add github:liyixuan201211/ctx-budget
```

中文：**agent 跑起来之前，它的上下文要花多少钱？** 它找出 agent 会读的东西——
指令文件、skill、MCP 工具定义、memory——逐项测量，并且分清**每次请求都要付的**
和**只有加载时才付的**。然后给你一个可以卡在 CI 里的退出码。

关于中文，它做了一件大多数同类工具不做的事：**「4 字符 1 token」是英文散文的
经验法则，用在中文上会低估约 2.4 倍**。所以它把 ASCII / CJK / 其他字符分开数，
各自用各自的比率，并且把原始字符数一起打出来，让你可以用真正的 tokenizer 复算。

---

## The problem

Every agent has a context budget, and almost nobody measures it. Three things
make it worth measuring:

1. **Two different costs get added together.** An instruction file is in every
   request. A skill body is only there when the skill is loaded. A 12,000-token
   skill body is fine; a 12,000-token `AGENTS.md` is a crisis — and a tool that
   reports one number cannot tell you which you have.
2. **Tool definitions are a permanent tax.** The DSH MCP client's own
   documentation says it plainly: *"Tool definitions add tokens to every model
   request."* Wire up four servers with fifty tools between them and you have
   bought a cost on every turn, whether or not the model ever calls one.
3. **Over-long descriptions are silently truncated.** The skill provider
   "renders this provider's invocable names and **capped descriptions** into the
   initial or replacement catalog". Write a 1,200-character description and the
   tail never reaches the model — so the part you carefully put last is the part
   that is not there.

`ctx-budget` measures all three, and it does it by reading files. No process, no
socket, no write.

## In use

```
$ ctx-budget --mcp-tools tools.json
ctx-budget  /home/me/project

  always-on        808 tokens  (646–1,076)    paid on every request
  on-demand        161 tokens  (129–215)      paid when a skill loads
  counted        3,873 characters                 across 15 sources (12 always-on, 3 on-demand)

  MCP tools are 221 of that (27%) — tool definitions you may never call

always-on, biggest first
  258  32%  skill  skill verbose — catalog
  136  17%  file   AGENTS.md
   68   8%  mcp    github / create_issue
   63   8%  mcp    github / list_pull_requests
   57   7%  memory memory/old-notes.md
   49   6%  skill  skill docs-writer — catalog
   46   6%  mcp    filesystem / read_file
   44   5%  mcp    filesystem / write_file
   27   3%  file   CLAUDE.md
   24   3%  skill  skill quick-notes — catalog
   23   3%  memory MEMORY.md
   13   2%  memory .dsh/memory/facts.md

on-demand, biggest first
  127  79%  body   skill docs-writer — body
   22  14%  body   skill verbose — body
   12   7%  body   skill quick-notes — body

MCP servers
       131  2 tools   github
        90  2 tools   filesystem

findings
  ! 2 copies of a 190-character block — about 48 tokens on every request
      AGENTS.md:13, memory/old-notes.md:3
      Before running anything that deletes or overwrites, work out what it would destroy and wh…
  ! 48 tokens per request are spent on text that appears more than once
  ! skill verbose — catalog: description is 1205 characters and the catalog keeps 1024: the last 181 never reach the model — put the useful part first, or shorten it
  ! skills/README.md: no YAML frontmatter: the agent's catalog cannot list this skill, so it is unreachable

measurement
  counted exactly: 3,873 characters, 3.8 KiB, 0 of them CJK
  estimated tokens at 4 chars/token (ASCII), 1.5 (CJK), 2.5 (other non-ASCII). Set them with --chars-per-token and friends.
  the rankings, the shares and the duplication above are computed from the character counts, so they hold even if these ratios are wrong.
```

Look at what that found: **the most expensive single thing in this project's
context is a skill description** — 258 tokens, more than `AGENTS.md` — of which
181 characters never even reach the model. That is not a number anyone would
guess, and it is a two-line fix.

## The two-cost model

A skill is not one blob that may or may not be loaded. It is two:

| | What it is | When you pay |
|---|---|---|
| **catalog** | `name` + `description` (capped) + `whenToUse` | every request, whether or not the skill is used |
| **body** | the text after the frontmatter | only when the skill is loaded |

This is not an assumption about how agents work; it is read off the DSH filesystem
provider's implementation, which parses frontmatter into a catalog entry and loads
the body on demand. Getting this wrong misreads the decision in both directions,
which is why the report never mixes the two tiers into one number.

An **invalid** skill — no frontmatter, no `name`, no `description`, or a
non-kebab-case name — is skipped by the provider entirely. So it costs nothing,
and counting it would over-report. It is reported as a finding instead: *"the
agent's catalog cannot list this skill, so it is unreachable"*.

## Measured versus estimated

This is the part to read before quoting a number.

**Counted exactly:** characters, bytes, lines, and the split into ASCII / CJK /
other non-ASCII. Also every character count behind the rankings and the
duplication.

**Estimated:** tokens. `chars/4.0 + cjkChars/1.5 + otherChars/2.5` by default.
An exact count would mean shipping a tokenizer for one specific model, which is
the wrong dependency for something that runs on every commit.

**Why the split is not decoration.** "Four characters per token" is a rule about
English prose. Applied to Chinese it under-reports by more than a factor of two:

```
$ ctx-budget --root ./chinese-project --json     # a 208-character AGENTS.md
  chars 208 (ASCII 22, CJK 178)
  CJK-aware estimate : 127 tokens
  naive 4.0 estimate :  52 tokens   <- what a tool that ignores CJK would say
  under-report factor: 2.44x
```

Under-reporting is the dangerous direction: it tells you your context is smaller
than it is. That is why the classes are counted separately, why the raw counts are
printed next to the estimate, and why the ratio question never touches the
rankings — those come from the character counts and hold even if every ratio is
wrong.

## Duplication, attributed to the tier it costs you in

The same block of instructions pasted into `AGENTS.md` and into a skill body is
the most common way a budget leaks. `ctx-budget` finds blocks that appear more
than once (after normalising whitespace and case, so re-indenting does not hide
them) and reports them as `file:line`.

Crucially, it attributes the waste by tier:

- **Two always-on copies** → one of them is wasted **on every request**.
- **Two on-demand copies** → wasted only when both are loaded. Real, but not the
  same number, and treating it as one would make the wrong fix look urgent.

## Exit codes are the contract

| Code | Meaning |
|---|---|
| `0` | measured, and within budget if one was given |
| `1` | unexpected error |
| `2` | usage |
| `3` | over budget |
| `5` | **could not determine** — something was found but not measured, so "it fits" is not a claim this run can make |
| `6` | nothing to measure |

**5 is the one that keeps a budget honest.** If an MCP server is declared in a
config but its tools were never dumped, the always-on figure is a lower bound —
and a lower bound under the limit says nothing at all. So an unmeasured source
makes `--max` exit 5 rather than quietly passing. Without a budget it is a
finding, not a failure, because no claim was being made.

```bash
ctx-budget --max 40000 || echo "context budget exceeded, or could not be verified"
```

## Commands

```bash
ctx-budget                                  # audit the current directory
ctx-budget --max 40000                      # the same, with a budget
ctx-budget --mcp-tools tools.json           # include MCP tool definitions
ctx-budget --mcp-config .mcp.json           # report declared servers as NOT measured
ctx-budget list                             # what was found, without the analysis
ctx-budget explain docs-writer              # one source in full
ctx-budget --json                           # machine-readable
```

What it looks at, project-relative:

```
AGENTS.md  CLAUDE.md  CONTEXT.md  GEMINI.md  .cursorrules  .windsurfrules
.github/copilot-instructions.md  SYSTEM.md  PERSONA.md  MEMORY.md
.dsh/memory/  memory/  skills/  .dsh/skills/  .claude/skills/  .codex/skills/
```

`--include <glob>`, `--skill-root`, `--memory` and `--system` add more;
`--user` also scans the global skill roots under your home directory (off by
default — reading `~` on every run would be a surprise, and a number nobody
asked for).

**MCP cost needs a dump**, because a config holds no schemas. Produce one with
the sibling tool, or with any MCP client:

```bash
mcp-cap inspect --json -- npx -y @modelcontextprotocol/server-filesystem /srv > tools.json
ctx-budget --mcp-tools tools.json
```

An `mcp-cap` lock file also works, and is reported as a lower bound: it stores a
schema *hash*, not the schema, and truncates descriptions.

## Honest positioning

The idea of auditing agent context is not virgin territory, and two of the
existing attempts are close enough to name:

| | What it does | Where this differs |
|---|---|---|
| [`jamespheffernan/agent-context-audit`](https://github.com/jamespheffernan/agent-context-audit) | Python; inventories `AGENTS.md`/`CLAUDE.md`, reports where instruction files duplicate or diverge and how large the skill/memory surfaces are | closest in spirit. This adds the **always-on vs on-demand split**, **MCP tool schemas**, **per-source token attribution with a stated ratio**, and a **budget with an exit code** |
| [`Ismail-2001/mcp-token-auditor`](https://github.com/Ismail-2001/mcp-token-auditor) | a **proxy** between MCP clients and servers doing real-time token counting and alerting | different in kind: that observes live traffic; this is a pre-flight static audit that needs no proxy, no traffic and no server started |
| [`toumai266/Vibe-Audit`](https://github.com/toumai266/Vibe-Audit) | a FastAPI + React console for agent intent alignment | an app for a different question |

A GitHub search for a pre-flight context-budget CLI of this shape returns
essentially nothing, which is either an opportunity or a warning. The honest
answer is that most people currently estimate this by pasting their files into a
token counter, one at a time, and never compare the result to anything.

## Things it is honest about not knowing

- **Tokens are estimated, never counted.** The ratios are configurable and the
  raw counts are printed so you can recompute. The rankings and the duplication
  do not depend on them.
- **It cannot see your system prompt.** Whatever your harness adds before your
  files is not measurable from the filesystem, so the numbers here are a floor
  for the real request size.
- **What your client actually renders is your client's business.** The catalog is
  modelled as name + capped description + whenToUse; the cap defaults to 1024 and
  is a documented assumption (`--description-cap`), not a measurement. What a
  provider caps at is its own implementation detail.
- **`--json` never contains file contents.** Nor does the text output, beyond a
  90-character preview of a *duplicated* block, with anything credential-shaped
  masked. An audit runs in CI, and CI logs are public more often than people
  expect — so a token pasted into `AGENTS.md` must not be echoed by the tool that
  noticed it.
- **Files over 8 MiB are reported as unmeasured, not read.** With a budget, that
  is exit 5.
- **A symlinked skill directory is not followed**, so a link out of the project
  is not pulled in. `--user` is the deliberate way to look further afield.
- **It does not tell you what to cut.** It ranks and attributes; the decision
  about which instructions earn their tokens is a judgement about your project.

## Development

Requires Node >= 20. Plain ESM JavaScript with JSDoc types: no build step, no
install-time scripts, and the published `bin` actually runs when installed — CI
asserts that by packing the tarball and running it from a real `node_modules`.

```bash
npm test            # 101 tests
npm run typecheck   # tsc --noEmit over the JSDoc types
npm run check       # both
./examples/demo.sh  # end to end, asserting every exit code
```

```
src/
  cli.js          the exit-code contract and argument parsing
  discover.js     what an agent will read, and where it lives
  frontmatter.js  a lenient SKILL.md frontmatter reader
  skills.js       the two-cost model: catalog vs body, and validity
  mcp.js          tool definitions from a tools/list dump, never from a server
  sources.js      a file becomes a measurable source
  estimate.js     the exactness boundary: what is counted and what is assumed
  dupes.js        duplication, attributed by tier
  report.js       human output, and the measurement footer
  audit.js        assembling it, with no path that can drop a source
```

The structural claim — *reads files and nothing else* — is asserted in
`test/safety.test.js` by checking that no file in `src/` mentions a process, a
socket, `eval`, or a write, and that `package.json` defines no lifecycle script.
CI runs the suite on Node 20/22/24, installs the packed tarball into a real
`node_modules` and audits a project with it, runs the safety invariants on their
own, runs the demo, and re-checks the no-network and no-lifecycle-script
properties from the outside.

## License

MIT.