dsh-knowledge-forge

Readme

<div align="center">

# DSH Knowledge Forge

### A local-first automatic Wiki and Skill memory plugin for DeepSeek Harness (DSH).

[![DeepSeek Harness](https://img.shields.io/badge/DeepSeek%20Harness-0.1.0--rc.6-4f46e5)](https://github.com/deepseek-ai/deepseek-harness)
[![Node.js](https://img.shields.io/badge/Node.js-%5E22.19%20%7C%7C%20%3E%3D24-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-16a34a.svg)](./LICENSE)
[![Tests](https://img.shields.io/badge/tests-21%20passing-2563eb)](#development)
[![Local-first](https://img.shields.io/badge/storage-local--first-7c3aed)](#storage-layout)

[Highlights](#-highlights) · [How it works](#how-it-works) · [Quick start](#quick-start) · [Dashboard](#dashboard-and-operations) · [Architecture](#architecture) · [Security](#security-and-privacy)

[**English**](./README.md) · [简体中文](./README.zh.md)

</div>

---

## ✨ Highlights

> **Built for DeepSeek Harness, DSH Knowledge Forge turns Agent conversation traces into progressively reusable knowledge.**
>
> - A first durable task case becomes a human-readable **Wiki page**.
> - A later independent case that repeats the same stable workflow can promote that Wiki into a native **DSH Skill**.
> - The Wiki then becomes compact metadata pointing to the Skill, so the Agent discovers procedures progressively instead of placing every Skill in the prompt.

| Capability | Behavior |
| :--- | :--- |
| **Local-first capture** | Redacts and bounds completed top-level turns before saving them to project-local SQLite |
| **Session-aware batching** | Processes one source Session at a time after 5 pending turns or 60 minutes |
| **One-shot distillation** | Uses a non-interactive, tool-free background Agent with strict structured output |
| **Wiki → Skill progression** | Stores the first valuable case as Wiki; promotes repeated executable workflows to native Skills |
| **Progressive disclosure** | Injects a compact Wiki index and loads full Wiki pages or Skills only when needed |
| **White-box observability** | Keeps readable Markdown artifacts, durable batch receipts, retry state, and a DSH Web Dashboard |

---

## Overview

[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) is a plugin-first open-source Agent Harness. DSH Knowledge Forge is a native DeepSeek Harness plugin that adds automatic local knowledge distillation without requiring an external memory service.

Agent sessions contain valuable engineering memory: what was attempted, what users corrected, what evidence mattered, which decision worked, and where a blocked task should resume. Raw transcripts are too large and noisy to become long-term context, while a single lossy summary is too hard to audit.

Knowledge Forge separates **operational evidence** from **reusable knowledge**:

- SQLite keeps queue state, scheduling, retries, receipts, Case relationships, and full-text indexes.
- Markdown keeps the durable Wiki and Skill artifacts readable and editable.
- A compact Wiki index guides recall without injecting every knowledge page or Skill into the prompt.

The result is a local knowledge loop for DeepSeek Harness:

```text
work happens -> valuable Case becomes Wiki -> repeated workflow becomes Skill -> future sessions reuse it
```

## How it works

<p align="center">
  <img src="./assets/architecture.svg" alt="DSH Knowledge Forge architecture and Wiki-to-Skill lifecycle" width="100%" />
</p>

### 1. Capture completed top-level turns

The plugin listens to DSH's top-level turn lifecycle. It extracts user-visible messages, bounded assistant output, reduced tool evidence, and relevant paths. Reasoning traces and full tool output are not copied.

Before persistence, candidate text is:

- redacted for credential-like content;
- truncated to configured limits;
- isolated by Git workspace and source Session;
- assigned an idempotent event identifier.

### 2. Schedule each Session independently

A source Session becomes due when either condition is met:

- it has **5 pending turns**; or
- its oldest pending turn has waited **60 minutes**.

Due Sessions are evaluated separately. Turns from different Sessions are never combined into one model request.

### 3. Run a one-shot background curator

Knowledge Forge starts a DSH subagent with these constraints:

- exactly one non-interactive response;
- no tools;
- strict JSON output schema;
- every input event must receive one result;
- all candidate and knowledge text is treated as untrusted data.

The Host validates event coverage, Case membership, safe identifiers, promotion evidence, and credential-like output before writing any artifact.

### 4. Distill Wiki first, Skill when repetition proves reuse

The knowledge policy is deliberately progressive:

1. **Discard** lookups, casual questions, transient chatter, and unchanged reuse of an existing Skill.
2. **Reuse an existing Skill** when it already covers the task.
3. **Write or update a Wiki** for a durable task Case: objective, user feedback, evidence, decisions, outcome, blockers, and continuation points.
4. **Promote to a Skill** only when a new independent Case repeats the same stable, executable workflow with clear inputs, steps, and verification.
5. **Replace the Wiki body with lightweight Meta** that points to the generated Skill.

This design keeps memory useful without treating every conversation as knowledge or every repeated phrase as automation.

## Quick start

### Requirements

- Node.js `^22.19.0 || >=24.0.0`
- DeepSeek Harness `0.1.0-rc.6`

Knowledge Forge uses Node's built-in `node:sqlite`; no separate SQLite installation is required.

### Install from this repository

Clone and build a prebuilt tarball:

```bash
git clone https://github.com/bill084153-cell/dsh-knowledge-forge.git
cd dsh-knowledge-forge
npm install
npm run build
npm pack
```

Install it into the DSH Web profile and start DSH:

```bash
dsh plugin --profile web add ./dsh-knowledge-forge-0.1.0.tgz
dsh web
```

The tarball path can be absolute. Install the bundle into every DSH profile where automatic knowledge capture should run. The Web profile additionally exposes the Dashboard.

> Direct GitHub installation of a TypeScript DSH plugin requires an install-time build allowance. Building a tarball first keeps installation explicit and avoids running repository build scripts inside the user's DSH profile.

### Verify installation

```bash
dsh web --dump-config | grep -A 8 dsh-knowledge-forge

dsh plugin --profile web exec \
  dsh-knowledge-forge doctor --workspace .
```

Expected doctor output includes the workspace root, local database path, queue counts, Wiki count, Skill count, and `status: ok`.

## Storage layout

Knowledge Forge writes only inside the current project:

```text
<project>/.dsh/
├── knowledge-forge/
│   ├── knowledge-forge.sqlite     # queue, batches, receipts, retries, FTS index
│   └── wiki/
│       ├── index.md               # compact recall map injected into Agent context
│       └── pages/
│           └── <wiki-id>.md       # durable task Cases or Skill metadata
└── skills/
    └── <skill-id>/
        └── SKILL.md               # native DSH Skill discovered by the Skill catalog
```

SQLite is operational state. Markdown Wiki pages and `SKILL.md` files are the readable long-term artifacts.

## Trigger, retry, and retention

| Policy | Default |
| :--- | :--- |
| Turn threshold | 5 pending top-level turns per Session |
| Age threshold | Oldest pending turn waits 60 minutes |
| Session mixing | Disabled; every due Session gets its own evaluation |
| Retry delays | 30 seconds, 2 minutes, 10 minutes |
| Dead letter | Fourth failed execution |
| Processing lease | 30 minutes before crash recovery |
| Operational retention | 90 days |
| Wiki / Skill expiry | Never automatically expired |

A long-running `dsh web` process can honor the age timer precisely. If DSH is stopped, pending work remains in SQLite and catches up when the workspace is active again.

The scheduler is process-level, not one permanent process per Session. If the exact source Agent is no longer live, another live top-level Agent in the same workspace may host the one-shot worker. Otherwise the batch waits for the next workspace activity.

## Dashboard and operations

Install the plugin in the Web profile, run `dsh web`, then open **Settings → Knowledge Forge**.

The Dashboard shows:

- Workspace and source Session filters;
- pending, running, waiting, completed, and dead-letter counts;
- source Session and worker Session navigation;
- batch trigger reason and per-turn action;
- generated Cases, Wiki pages, and Skills;
- validation or execution errors.

### In-session commands

```text
/knowledge-forge
/knowledge-forge run
/knowledge-forge retry <batch-id|all>
```

### Offline diagnostics

These commands read local state and do not start a model:

```bash
dsh plugin --profile web exec \
  dsh-knowledge-forge status --workspace .

dsh plugin --profile web exec \
  dsh-knowledge-forge doctor --workspace .
```

## Knowledge recall

Knowledge Forge adds two read-only tools:

| Tool | Purpose |
| :--- | :--- |
| `knowledge_search` | Search Wiki and generated Skill metadata through local SQLite FTS5 |
| `knowledge_read` | Read one indexed Wiki page or generated Skill after search |

At Agent creation, the plugin injects only the compact Wiki index. Full pages and Skill bodies are loaded on demand. This follows DSH's native Skill model and preserves context space.

## Architecture

| Layer | Implementation |
| :--- | :--- |
| DSH lifecycle | `agent/turn-stopping` and `agent/created` extension points |
| Candidate extraction | Bounded user/assistant text, reduced tool trace, path evidence, redaction |
| Operational store | Node 22 built-in `node:sqlite`, WAL when available, FTS5 search |
| Scheduler | One process-level scheduler across workspace databases |
| Semantic worker | DSH `subagents.start('spawn', ...)`, tool-free one-shot execution |
| Output contract | Strict JSON Schema plus Host semantic validation |
| Durable artifacts | Atomic Markdown Wiki and native `.dsh/skills/*/SKILL.md` writes |
| Recall | Compact Wiki context plus `knowledge_search` / `knowledge_read` |
| Observability | Read-only HTTP endpoint and DSH settings page |

The plugin has no external service dependency. Its runtime stays inside the DSH process and the current project directory.

## Security and privacy

- Raw Session logs, model reasoning, full tool output, and secrets are not copied into the plugin database.
- Candidate text is redacted and bounded before persistence.
- Candidate, Wiki, Skill, path, tool argument, and tool output text is untrusted input.
- The background Agent receives no tools and cannot execute candidate instructions.
- Host validation runs before every Wiki or Skill write.
- Artifact identifiers are restricted to safe kebab-case paths.
- Writes use atomic file replacement inside declared project roots.
- The Dashboard endpoint is read-only and follows the existing DSH Web exposure boundary.
- Operational rows expire after 90 days; curated Wiki and Skill artifacts remain until a user edits or removes them.

## Configuration

The shipped bundle defaults are defined in [`cordis.patch.yml`](./cordis.patch.yml):

```yaml
- insert:
    - id: knowledge-forge
      name: dsh-knowledge-forge
      config:
        turnThreshold: 5
        maxAgeMinutes: 60
        retentionDays: 90
        maxRetries: 3
```

A later DSH profile patch can override the plugin row. Available options also include `processingLeaseMinutes`, candidate and tool preview limits, Wiki context size, and `enabled`.

## Development

```bash
npm install
npm run typecheck
npm test
npm run build
npm run pack:check
```

The current suite covers candidate extraction, redaction, scheduling and Session isolation, retry/dead-letter behavior, structured-output validation, atomic Wiki/Skill writing, native DSH Skill discovery, bundle composition, and Dashboard transport behavior.

## Project status

Knowledge Forge currently targets DeepSeek Harness `0.1.0-rc.6`, which is a developer-preview release. DSH APIs may change before a stable release; compatibility updates may be required.

## Contributing

Issues, documentation improvements, compatibility fixes, and new tests are welcome. Please keep contributions scoped to the automatic local knowledge-distillation plugin and avoid introducing framework-specific business integrations into the core.

If this project is useful, consider starring the repository and sharing concrete Wiki/Skill promotion cases through GitHub Issues.

## License

[MIT](./LICENSE) © 2026 Zhongxibiao