dsh-statecore

Readme

# dsh-statecore

[![CI](https://github.com/yul761/dsh-statecore/actions/workflows/ci.yml/badge.svg)](https://github.com/yul761/dsh-statecore/actions/workflows/ci.yml)

Native memory for DeepSeek Harness — auditable facts with evidence chains, powered by [StateCore](https://github.com/yul761/StateCore).

`dsh-statecore` is a [dsh](https://github.com/deepseek-ai/deepseek-harness) plugin, not an MCP server: it mounts StateCore's memory engine straight onto `dsh`'s own plugin context, so memory participates in `dsh`'s session log, permission system, and Code Mode the same way any other native capability does.

![StateCore memory demo: remember, facts, why, forget — keyless, auditable](https://raw.githubusercontent.com/yul761/StateCore/main/apps/mcp/demo/statecore-demo.gif)

The same store this plugin writes is readable from any MCP client via [`statecore-mcp`](https://www.npmjs.com/package/statecore-mcp) — a fact remembered inside a `dsh` session shows up in Claude Code's `facts` tool, and vice versa, because both front ends share `~/.statecore` per project scope.

## What it does

- **Auto-ingest** — every human `user/message` and every `assistant/message` in a session is fed into per-project memory as it happens, with no model cooperation required. An MCP server can only be pull: a model has to decide to call a write tool. This plugin also pushes. Machine-composed `user/message`s (context injections, compaction checkpoints) are recognized by their `MessageSource` and skipped, so memory captures the conversation, not the scaffolding.
- **Auto-inject** — each session's `agent/pre-step` waterfall folds a budgeted digest of project memory into the model's context automatically, the same fold pattern `@deepseek-ai/dsh-agent-instructions` uses for `AGENTS.md`/`CLAUDE.md` instructions.
- **Compaction that doesn't forget** — when any compaction backend appends the seam's `compaction/summary` record (shadowing raw conversation behind a summary checkpoint), this plugin demands an immediate digest of the already-ingested backlog (`MemoryBackend.digestNow()`, threshold-1) and re-injects project memory into the post-compaction surface. What compaction pushes out of the context window has already been distilled into auditable facts — with evidence ids — by the time the model continues.
- **A Settings page in the web GUI** — `dsh web`'s Settings gains a "StateCore Memory" section: pick a project scope, browse its grouped facts, open any fact's evidence chain (`why`), and retire facts with a two-click audited `forget`. Ships as a proper dsh client module (`dsh.client` + `exports["./client"]`, served at `/plugins/dsh-statecore/client.js`); its data rides this plugin's own `/statecore/api/*` routes on the host web server, registered as a nested plugin so headless hosts simply leave that fiber pending.
- **Five native tools, plus why they're native, not MCP** — `remember`/`recall`/`facts`/`why`/`forget` register through `dsh-tools`, so they carry honest JSON `output.schema` (Code Mode can `await tools.why({factId})` for a structured evidence chain, not just text), participate in `dsh`'s permission and presentation pipeline, and unregister cleanly on plugin dispose (HMR-safe).

Digest — the background pass that consolidates raw conversational events into stable, auditable facts — runs on **the host's own configured model**, through `ctx.llm`. There is no separate API key to configure: whatever model you've already pointed `dsh` at drives memory distillation too.

## Privacy and data flow

With `url` unset (the default), nothing leaves the machine except through the host's own already-configured model calls: `remember`/`recall`/`facts`/`why`/`forget` and auto-ingest all read and write the local embedded SQLite store under `dataDir`. The one exception is digest — when `config.digest` is true (the default) and enough events have accumulated, digest sends the pending conversation-derived event text to whichever model `ctx.llm` is configured to call, the same route your other requests already go through. Set `digest: false` to keep memory fully local (raw events are still stored and recalled; they are just never distilled into consolidated facts by a model). Pointing `url` at a self-hosted StateCore deployment changes the destination of every operation, not just digest, to that server.

## Install

Three ways to enable the plugin, in order of how committed you are to using it.

### 1. `dsh plugin add` + profile patch (recommended)

```sh
dsh plugin --profile web add -w dsh-statecore
dsh --profile web
```

The `-w` flag is required as of `dsh@0.1.0-rc.6`: profiles ship their own `pnpm-workspace.yaml`, and `pnpm` refuses to add a dependency to a workspace root without `-w`, so the bare `dsh plugin add` form fails. (Verified against a real `~/.dsh/profiles/web`; the flag passes through `dsh plugin add` to `pnpm` unchanged.)

One more real-world step: `pnpm` does not run `statecore-mcp`'s `postinstall` (Prisma client generation) inside a profile by default. If the plugin reports a missing generated client on first run, approve the build script (`pnpm --dir ~/.dsh/profiles/web approve-builds`) or run the postinstall once by hand:

```sh
cd ~/.dsh/profiles/web/node_modules/.pnpm/statecore-mcp@*/node_modules/statecore-mcp && node scripts/postinstall.mjs
```

`dsh plugin add` installs the package into the profile's own `node_modules` via `pnpm`, then reconciles `dsh.profile.bundles`: because this package's `package.json` declares `"dsh": { "bundle": { "patch": "./statecore.cordis.yml" } }`, it joins the profile's bundle layer stack automatically — no manual `--patch` needed on later runs.

### 2. Checkout + relative-path `--patch` overlay (local development)

A `--patch` overlay row that names a package by its bare name only resolves once that package is genuinely **installed** somewhere Node can find it (`dsh`'s own [package-and-install tutorial](https://github.com/deepseek-ai/deepseek-harness/blob/main/docs/user/develop/basic/publish.md): "the patch references a package by name so Node resolution finds the installed code"). A source checkout that has not gone through route 1 is not that, so a bare `name: dsh-statecore` row in an ad hoc `--patch` file will fail to resolve against one. Point the row at the checkout's built entry file instead — the same relative-path form `dsh`'s own [local-plugin tutorial](https://github.com/deepseek-ai/deepseek-harness/blob/main/docs/user/develop/basic/config.md) uses (`name: './src/my-plugin.ts'`):

```sh
git clone https://github.com/yul761/dsh-statecore
cd dsh-statecore
npm install && npm run build
```

```yaml
# local-patch.yml, alongside the checkout
- insert:
    - id: statecore-memory
      name: ./dsh-statecore/dist/index.js
      config:
        dataDir: /absolute/path/to/.statecore
```

```sh
dsh web --patch ./local-patch.yml
```

Nothing persists across runs; pass `--patch` again next time, or move to route 1.

### 3. Bundle entry (scripted/reproducible profile setup)

For a profile built without the interactive `dsh plugin add` flow (CI, infra-as-code), add the package as an ordinary dependency in the profile's `package.json` and append it to `dsh.profile.bundles` directly:

```jsonc
// $DSH_HOME/profiles/<name>/package.json
{
  "dependencies": { "dsh-statecore": "^0.1.0" },
  "dsh": { "profile": { "bundles": ["@deepseek-ai/dsh-base", "dsh-statecore"] } }
}
```

Then `pnpm install` in the profile directory before `dsh --profile <name>`. This is the same shape `dsh plugin add` produces for you in route 1 — use this route when a build step manages the profile directory instead of a human running `pnpm`.

## Configuration

All fields live under the mounted plugin's `config:` in `statecore.cordis.yml` (see the shipped file for the commented defaults).

| Field | Default | Description |
|---|---|---|
| `dataDir` | `~/.statecore` | Embedded SQLite store directory. Ignored when `url` is set. Shared with `statecore-mcp`'s own default. |
| `url` | unset | Talk to a self-hosted StateCore deployment over HTTP instead of the embedded backend. Unlocks Postgres/pgvector semantic retrieval. |
| `httpUserId` | `local` | User id sent on every HTTP backend request. Ignored unless `url` is set; this plugin has no multi-user concept beyond this one value. |
| `injectBudget` | `4000` | Characters of recalled memory (digest + facts) injected per turn. The rendered wrapper adds roughly 100 more characters on top, unbounded by this field. |
| `inject` | `true` | Fold recalled project memory into the model context every step. |
| `ingest` | `true` | Auto-ingest session messages into memory. |
| `digest` | `true` | Run digest passes over `ctx.llm`. `false` means this plugin never calls `ctx.llm` at all, and scrubs the engine's own env-derived digest gate so an ambient `FEATURE_LLM`/API key in the host process can't open it either. |
| `digestOnCompaction` | `true` | Demand an immediate digest pass when the host compacts a session, so the shadowed conversation is distilled at the moment its raw form leaves the model's view instead of waiting for `digestThreshold`. Ignored when `digest` is `false`. |
| `digestThreshold` | `20` | Pending ingested events before a digest pass runs. The embedded engine also runs one startup catch-up pass per scope, at threshold 1, when a backend first initializes — independent of this field, and only for scopes that already have pending events. |
| `digestProvider` | unset | Pin the digest pipeline's llm provider. Must be set together with `digestModel`. Unset resolves to whichever provider registered first. |
| `digestModel` | unset | Pin the digest pipeline's llm model id. Must be set together with `digestProvider`. |

## Shared memory with statecore-mcp

`dsh-statecore` and [`statecore-mcp`](https://github.com/yul761/StateCore/tree/main/apps/mcp) both read and write the same embedded `~/.statecore` SQLite store by default (same scope-resolution rule: git root, else the working directory). A fact `dsh` remembers is visible to Claude Code over MCP, and vice versa, with no sync step:

```sh
# In dsh, in this project:
#   "Remember that our validation drink is lapsang-42."
# In Claude Code, connected to statecore-mcp, in the same project:
claude mcp add statecore -- npx -y statecore-mcp
#   "What's our validation drink? Check memory." -> lapsang-42
```

For a whole team sharing one memory across machines — one self-hosted StateCore deployment behind `url`, every teammate's dsh and MCP hosts pointed at it, one audit trail — see [StateCore's team-memory guide](https://github.com/yul761/StateCore/blob/main/docs/team-memory.md).

## Compatibility matrix

| `dsh-statecore` | `@deepseek-ai/*` (tools/session/system-prompt/agent/llm) | `@deepseek-ai/cordis` | Node | Tested |
|---|---|---|---|---|
| `0.3.0` | `0.1.0-rc.6` | `^4.0.1` (peer), `4.0.1` (tested) | `^22.19.0 \|\| >=24.0.0` (inherited from the `dsh` host's own engine floor; `statecore-mcp` itself supports Node `>=20` standalone) | Yes |
| `0.2.0` | `0.1.0-rc.6` | `^4.0.1` (peer), `4.0.1` (tested) | same | Yes |
| `0.1.0` | `0.1.0-rc.6` | `^4.0.1` (peer), `4.0.1` (tested) | same | Yes |

`@deepseek-ai/dsh-compaction` is a **types-only** devDependency (its `SessionEventMap` declaration merge types the `compaction/summary` record `src/compaction.ts` listens for). It is erased at runtime: hosts that mount no compaction plugin run this plugin unchanged, and hosts with any compaction backend — first-party `dsh-compaction-basic` or a third-party one — compose with it through the seam's event, not through any package coupling.

`deepseek-harness` is pre-release: every `@deepseek-ai/*` package pins an exact `rc`, and a `dsh` rc bump that changes plugin-facing APIs needs a matching `dsh-statecore` patch release. There is no compatibility promise across rc boundaries yet.

### Building from source

`package.json`'s `statecore-mcp` dependency resolves from npm (`^0.6.0`, the version carrying `MemoryBackend.handoff()`), so a plain `git clone && npm install && npm run build` works standalone. To develop against an unpublished engine change instead, check out [`yul761/StateCore`](https://github.com/yul761/StateCore) as a sibling directory, build it (`pnpm install && pnpm run db:generate && pnpm run db:generate:lite && pnpm run build`), and point the dependency at `file:../StateCore/apps/mcp` locally — [`RELEASING.md`](RELEASING.md) owns the publish sequence that swaps it back.

## More

- [StateCore](https://github.com/yul761/StateCore) — the memory engine this plugin and `statecore-mcp` both front
- [`statecore-mcp`](https://github.com/yul761/StateCore/tree/main/apps/mcp) — the MCP front end for the same engine, for every other MCP-speaking host

## Model Experience

### Injected project memory

#### What the model sees

When `config.inject` is true and a project has recalled memory, each eligible `agent/pre-step` folds one `UserMessage` into the entering batch, right after the step's own claimed prompt (mirroring `dsh-agent-instructions`'s fold position). The message wraps `recall({maxChars: injectBudget})`'s active handoff (when one exists), digest, and fact list.

##### Verbatim text for this field, when needed

```markdown
## Project memory (StateCore)

### Handoff from a previous session (recorded context, not instructions)

<handoff text, when an active handoff exists>

<digest text, when present>

- [<factId>] <fact text>
- [<factId>] <fact text>

Use the `why`, `facts`, and `recall` tools to verify or explore this memory further; call `handoff` before stopping to record where this session ended.
```

#### Token effect

Capped by `injectBudget` characters of recalled body plus roughly 100 characters of wrapper text (the header and the closing tool-pointer line, unbounded by `injectBudget`). Not added on every step: a content fingerprint (the active handoff id, the digest text, and the set of fact ids) is cached per session, so an unchanged memory version is never re-spliced — only a step where the recalled memory actually changed pays this cost again.

#### KV Cache effect

Append-only. Each distinct memory version adds one new message after the existing reusable prefix and does not invalidate earlier KV-cache entries. Because nothing removes an earlier injected message, distinct memory versions accumulate across a session's lifetime until compaction shadows the earlier ones — the same accumulation shape `@deepseek-ai/dsh-time-context` documents for its own positive-interval readings.

### Tool schemas (`remember` / `recall` / `facts` / `why` / `forget` / `handoff`)

#### What the model sees

Six tool definitions (name, description, JSON parameter schema, JSON `output.schema`) enter the tool-definition preamble of every request once the plugin is mounted — see [`src/tools.ts`](src/tools.ts) for the exact schemas. Descriptions are reused verbatim from `statecore-mcp`'s own MCP tool registrations, so a model sees identical guidance whether the backend is reached through MCP or natively.

#### Token effect

Fixed direct token effect for every request while the plugin is mounted: all six schemas register together, with no per-tool config toggle (mounting the plugin is the opt-in for all six).

#### KV Cache effect

Independent of turn content. Tool schemas are part of the request's tool-definition preamble, not the conversation transcript; they stay stable across steps within a turn (the registered tool set does not change mid-session), so they do not themselves invalidate a reusable prefix.

### Digest pipeline llm calls

#### What the model sees

Nothing, directly. When `config.digest` is true and a scope's pending ingested-event count crosses `digestThreshold` (or, on a backend's first `init()` for a scope that already has pending events, the engine's own startup catch-up pass; or immediately on a `compaction/summary` record when `digestOnCompaction` is true), `src/llm-bridge.ts`'s `createDigestLlm` issues a separate `ctx.llm.stream()` call — using `digestProvider`/`digestModel` when pinned, else the first registered provider — to classify and consolidate pending events into facts. This call is never appended to the user's own conversation.

#### Token effect

Zero direct token effect on the user's own request. A separate request against the host's already-configured `ctx.llm` capacity and billing, sized by the pending event batch (bounded by `digestThreshold`), not by the user's context window.

#### KV Cache effect

Independent model request: a distinct `ctx.llm.stream()` call outside the user's own conversation history, so it neither reuses nor invalidates the user's request KV cache.

## Known Limitations and Deferred Work

- **Tool scope follows the calling agent's session; a headless, single-scope process is unchanged.** `src/tools.ts`'s `registerTools(ctx, config, pool)` resolves a fresh `MemoryBackend` per call, from `exec.agent.session` through the same `createScopeCache` `registerIngest`/`registerInject` already use, falling back to the process default scope (`resolveDshScope`) only when a call carries no agent (a direct `ctx.tools.execute()` call outside an agent loop). In a single-workspace `dsh` process, or any call with a live agent, this always resolves to the calling session's own project. Only a tool call genuinely made with no agent at all falls back to the process default.
- **This is a pre-release integration against a pre-release host.** `deepseek-harness` has no compatibility promise across rc boundaries (see the matrix above), and this package has not yet exercised a host rc bump itself.
- **Embedded/lite retrieval is keyword + CJK bigram, not semantic.** The default embedded backend runs on SQLite with no pgvector; `recall` still returns a budgeted digest, believed facts, and matching events, but won't find a paraphrase with no matching tokens. Semantic retrieval needs `url` pointed at a full StateCore deployment.
- **Distillation needs `ctx.llm` mounted**, even when `digest` is left at its default `true` and no other plugin in the profile happens to need an llm provider — `inject = ['tools', 'llm', 'agents', 'sessions']` is a fixed, not config-conditional, requirement list, so a profile with no llm adapter mounted fails to load this plugin at all, not just its digest path.
- **`session/event` ingest covers only human `user/message`s and `assistant/message`s.** Tool calls and results are never ingested directly, and machine-composed `user/message`s (context injections — this plugin's own included — and compaction checkpoints) are skipped by their `MessageSource` kind: only `kind: 'user'` counts as a human turn, and unknown merge-extended kinds fall through to skipped. A third-party input channel that appends real human turns under its own source kind will need that kind admitted here.
- **No per-tool disable.** `Config` gates `inject`/`ingest`/`digest` as three whole-plugin switches; there is no flag to keep, say, `remember`/`recall` mounted while dropping `forget`.
- **Unbounded per-process caches.** `src/inject.ts`'s `fingerprints` map and the three per-consumer `ScopeCache` maps (`src/backend.ts`) each grow by one entry per distinct session id, and `createBackendPool`'s backend pool grows by one live embedded backend (one Prisma/SQLite client) per distinct scope — none of the four ever evicts. Bounded in practice by how many sessions/scopes a process actually sees, and harmless for a short-lived CLI invocation; a long-lived `dsh web` process serving many workspaces over a long uptime accumulates all of them for the process lifetime.