dsh-auto-review

Readme

# dsh-auto-review

**English** | [简体中文](README.zh.md)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

LLM auto-review approval answerer for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): when a session's approval policy is `'auto'`, sandbox escalations beyond workspace-write (`sandbox_permissions: "danger-full-access"` + justification) are decided **without a human prompt** — a human is asked only when the reviewer is unsure, and every error path fails closed.

Two decision layers, in order:

1. **Deterministic filter** — hardline (always-deny) + deny/allow regex rules over a normalized command (`~`/`$HOME` expansion, comment stripping, whitespace collapse, macOS private-path folding). A hardline or deny match rejects immediately; an allow match grants immediately; neither runs the LLM. Deterministic denials are FINAL — operator policy is never appealed.
2. **Clean-context LLM safety review** — the gray zone goes to a small reviewer model with a fixed constant system prompt, `temperature: 0`, a single-word verdict (`ALLOW` / `DENY` / `UNSURE`), and a tight output cap. The user message carries ONLY the tool name, the justification, and the XML-wrapped command — never the conversation.

**Appeal mode** (`context: 'appeal-user-prompts'`, the default): an LLM-voiced `DENY` is re-judged ONCE with the session's genuine user prompts attached as intent evidence — extracted deterministically (never LLM-summarized) from `user/message` events whose source kind is exactly `'user'`. Appeals can only RELAX a denial, never tighten an allowance: they reduce false positives; they are not a detection upgrade.

The answerer registers with `{ prepend: true }` on `approval/request`, so it runs before any human/machine channel, and delegates with `next()` whenever the effective policy is not `'auto'` or the request carries no action. The reviewer is defense-in-depth against an honest-but-wrong model, never a sandbox-widening mechanism: workspace-write remains the boundary until a grant lands. An optional consecutive-denial **circuit breaker** (`denialBreakerThreshold`) short-circuits retry loops before any further LLM cost.

## ⚠ Requires a patched harness core

This plugin **cannot run on stock upstream deepseek-harness**. It needs three patched-core surfaces, shipped in [core-patches/](core-patches/) as two git patches with a rebase guide (authored against upstream base commit `47f943859b`, merge PR #2519):

- `0001-approval-auto-policy-and-escalation-action.patch` — the `'auto'` approval policy, `ApprovalRequest.action`, `ApprovalService.effectivePolicyFor`, and the tool/sandbox/permission-presets wiring that carries the escalation action.
- `0002-llm-auto-review-request-purpose.patch` — the `'auto-review'` LLM request purpose.

Apply them with `git am` (or `git apply`) **in order** after syncing upstream; if upstream conflicts, resolve and regenerate — see [core-patches/README.md](core-patches/README.md).

**On vanilla upstream the plugin is deliberately inert:** the policy gate never resolves to `'auto'`, so every escalation lazily delegates (`next()`) to the normal human answerers. Nothing breaks, nothing is auto-decided — but you get no auto-review until the patched core is in place.

## Install

Prerequisites: a patched harness build (above), and the profile's permission presets must include the `auto-review` preset (`sandbox: workspace-write` + `approval: auto`) — patch 0001 adds it to the harness's `permission-presets` package.

From GitHub (builds on install via its `prepare` script; pnpm will ask you to allow the build once):

```sh
dsh plugin --profile web add github:accpowered/dsh-auto-review
```

If pnpm prints an `allowBuilds` prompt, add the printed key under `allowBuilds` in the profile's `pnpm-workspace.yaml` and re-run the `add` — this is install-time code execution permission, so only allow sources you trust (pinning a commit, `github:accpowered/dsh-auto-review#<sha>`, is recommended).

From a local checkout:

```sh
dsh plugin --profile web add ./dsh-auto-review
```

Restart `dsh web` (or your profile). The bundle patch ([cordis.patch.yml](cordis.patch.yml)) inserts one host row with a default reviewer route (`deepseek-official` / `deepseek-v4-flash`). To use a different reviewer, restate the row in your profile's `cordis.patch.yml` — `provider` and `model` are required and a patch replaces the row's whole config:

```yaml
- id: auto-review
  name: dsh-auto-review
  config:
    provider: my-provider      # any registered LLM route
    model: my-small-model
```

## Configuration

All knobs (composition = the cordis.patch.yml row config; user = live override through the `auto-review` settings namespace):

| Key | Default | Composition | User-overridable | Purpose |
|---|---|---|---|---|
| `provider` | — (required) | ✓ | ✓ | Reviewer route (a registered provider) |
| `model` | — (required) | ✓ | ✓ | Reviewer model id on that route |
| `hardline` | shipped list | ✓ | — | Always-denied regex patterns; never bypassed while policy is `auto` |
| `deny` | shipped list | ✓ | ✓ | Configurable deny regex patterns |
| `allow` | shipped list | ✓ | ✓ | Configurable allow regex patterns (conservative, side-effect-free only) |
| `maxTokens` | `512` | ✓ | — | Reviewer output-token cap (single-word verdict + headroom) |
| `timeoutMs` | `15000` | ✓ | — | End-to-end reviewer deadline (ms) |
| `onError` | `deny` | ✓ | ✓ | Reviewer-failure fallback: `deny` (fail closed) or `ask` (human) |
| `onUnsure` | `ask` | ✓ | ✓ | UNSURE-verdict fallback: `ask` (human) or `deny` |
| `denialBreakerThreshold` | `0` (off) | ✓ | ✓ | Consecutive-denial circuit breaker |
| `context` | `appeal-user-prompts` | ✓ | ✓ | `appeal-user-prompts` or `minimal` (command alone, v1 behavior) |
| `appealMaxPrompts` | `0` (unbounded) | ✓ | ✓ | Appeal budget: newest user prompts attached (first prompt exempt) |
| `appealMaxChars` | `0` (unbounded) | ✓ | ✓ | Appeal budget: total user-prompt characters attached |

`hardline`, `maxTokens`, and `timeoutMs` are composition-only by design. The shipped filter defaults are documented in `src/index.ts` (`DEFAULT_HARDLINE_PATTERNS` / `DEFAULT_DENY_PATTERNS` / `DEFAULT_ALLOW_PATTERNS`).

## Live user-layer overrides

User-tunable keys live in the **`auto-review` settings namespace** (registered through `installSettingsSection`; the namespace name stays `auto-review` so existing user settings documents keep applying). The composition entry is the base layer; a committed user-layer change re-resolves and recompiles the rule patterns on the very next decision — no restart. A write carrying an invalid regex is refused at the settings seam, so a bad pattern can never reach the compiled filter.

## Uninstall

```sh
dsh plugin --profile web remove dsh-auto-review
```

The answerer and the settings section detach with the plugin fiber. Log-only `auto-review/decision` events already on session logs are left untouched, and sessions on the `auto-review` preset fall back to the human answerers.

## Development

```sh
pnpm install
pnpm build       # tsc declarations (lib/types) + tsdown (lib/index.js)
pnpm test        # vitest: filter/verdict/appeal flows, settings lifecycle (local approval stub)
```

Layout: `src/index.ts` (the whole plugin — answerer, config schema, settings section, exported pure functions), `tests/auto-review.spec.ts` + `tests/stubs.ts` (the stub replicates the patched ApprovalService's dispatch semantics; the npm rc.1 package predates the `'auto'` policy), `core-patches/` (the two required core patches + rebase guide).

## License

MIT