dsh-encoding

Readme

# dsh-encoding

> Correct-encoding command runner and mojibake recovery for DeepSeek Harness.
>
> 让 DeepSeek Harness 在 Windows 上不再乱码。

[English](#english) / [中文](#中文)

---

## English

`dsh-encoding` is a **thin tools plugin** for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (dsh). On Chinese Windows, legacy CLIs, Git Bash/MSYS2 and PowerShell 5.1 often write GBK (cp936) or UTF-16LE bytes, which dsh's built-in `bash`/`pwsh` tools decode as UTF-8 and show as mojibake. The decoding happens inside the non-pluggable core service (`dsh-subprocess-local`), so this plugin does **not** try to patch it. Instead it adds two complementary tools:

| Tool | What it does |
| --- | --- |
| `encoding_run` | Spawns its own child process, captures **raw bytes** of stdout/stderr, auto-detects UTF-8 / UTF-16LE / GBK and decodes them correctly. |
| `encoding_decode` | Recovers already-garbled text (e.g. GBK bytes misread as latin1) back to correct Chinese. |

> **0.2.0 breaking change:** tool names were renamed from `run`/`decode_text` to `encoding_run`/`encoding_decode` (collision-safe prefix). 0.1.0 is deprecated on npm.

### Relation to official built-ins (verified on dsh 0.2.0-rc.2)

- dsh 0.2.0 does **not** ship built-in `run` / `decode_text` tools. Verified with a clean, plugin-free profile (`dsh --profile pmaudit --dump-config`): the base config contains no such tool rows (built-in tools are `tool-bash`, `tool-pwsh`, `tool-fs`, …), and the official tools reference documents none.
- The built-in `bash`/`pwsh` tools still have **no encoding configuration** — the shell seam's `ShellExecRequest` has no encoding field — and a GBK-producing command through the built-in `pwsh` tool still returns mojibake on this environment. This plugin's premise therefore holds.
- Tool names are prefixed so they cannot collide with built-in or future dsh tools: the registry rejects duplicates within one layer and the reserved `run_code` name (official `tools.md`).

### Install

From npm:

```sh
dsh plugin --profile <name> add dsh-encoding
```

From a local checkout (this repo):

```sh
dsh plugin --profile <name> add ./dsh-encoding
```

From GitHub (works with no build authorization because the package is pure ESM JavaScript with **no build step**):

```sh
dsh plugin --profile <name> add github:Lostforest7/dsh-encoding
```

Verify the layer is active, then start:

```sh
dsh --profile <name> --dump-config   # shows a "# == dsh-encoding" layer
dsh --profile <name>
```

### Tool: `encoding_run`

| Parameter | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `command` | string | yes | — | Command line to execute (Windows: `cmd /d /s /c`, POSIX: `/bin/bash -c`). |
| `workdir` | string | no | — | Working directory for the child process. |
| `timeoutMs` | number | no | `120000` | Timeout; the whole process tree is killed on expiry. |
| `encoding` | string | no | `auto` | `auto` (detect) or force `utf-8` / `utf-16le` / `gbk`. |

Example output (GBK output on Windows):

```
中文
[stderr]
... (if any stderr)
[encoding: gbk/utf-8 (auto-detected)]
[exit code: 0]
```

The last line is always `[exit code: N]`. When the output contains U+FFFD (�), an explicit warning line is appended instead of silently pretending the decode was right.

> **Positioning, stated plainly:** `encoding_run` is an *independent new tool*. It does **not** have the built-in `bash` tool's sandbox, approvals, or background-task support, and it executes with the same permissions as the harness process.

### Tool: `encoding_decode`

| Parameter | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `text` | string | yes | — | The garbled text to recover. |
| `from` | string | no | `auto` | How to turn characters back into bytes: `latin1` / `utf-16le` / `auto`. |
| `to` | string | no | `auto` | Target decoding: `gbk` / `utf-16le` / `auto`. |

The classic mojibake path is `from=latin1, to=gbk`: GBK bytes were misread as latin1 (one character per byte), so the tool re-encodes each character to one byte and decodes the bytes as GBK.

With `auto`, the direction is guessed: embedded NULs → `latin1 → utf-16le`; wide characters → `utf-16le → gbk`; everything else → the classic `latin1 → gbk`.

**Irreversible cases are reported, never faked:** if the input already contains U+FFFD (�) — i.e. the shell already destroyed the bytes with an earlier mis-decode — the result is `ok: false` with an explicit note.

> Note: `from=gbk` is rejected with a clear explanation. Node.js ships a GBK **decoder** (TextDecoder) but no GBK **encoder**, and this plugin is zero-dependency, so re-encoding characters back to GBK bytes is not possible. `to=gbk` is fully supported.

### 30-second example

Built-in `bash` shows `ÖÐÎÄ` where `中文` was expected (GBK bytes read as latin1):

1. Ask the agent to use `encoding_decode` with `{"text": "ÖÐÎÄ"}` → `中文` — done.
2. Or re-run the command with correct decoding: use `encoding_run` with `{"command": "chcp 936 >nul & echo 中文"}` → output `中文` plus `[encoding: gbk (auto-detected)] [exit code: 0]`.

### CLI smoke test (no DSH required)

The CLI subcommands keep their short names (`run` / `decode_text`) — they are a dev smoke runner, not the registered tool names:

```sh
node src/cli.js decode_text "ÖÐÎÄ"                  # -> 中文
node src/cli.js run "chcp 936 >nul & echo 中文"      # GBK output, auto-decoded
node src/cli.js run "echo hello"
node src/cli.js detect D6D0CEC4                      # -> detected: gbk / decoded: 中文
```

### Development

Zero dependencies — none at runtime, none for development; tests use only Node built-ins (`node:test`, `node:assert`). No `npm install`, no DSH, no build step needed:

```sh
node --test test/
```

Architecture:

```
dsh-encoding/
├── package.json          # type: module, dsh.bundle.patch -> cordis.patch.yml
├── cordis.patch.yml      # inserts one loader row: id "encoding", name "dsh-encoding"
├── src/index.js          # Cordis entry (dependency-free), registers encoding_run + encoding_decode
├── src/encoding.js       # pure functions: detect / decode / mojibake recovery
├── src/exec.js           # spawn wrapper + run formatting (spawnImpl injectable)
├── src/cli.js            # manual smoke runner (no DSH required)
└── test/                 # node:test suites, mock child_process.spawn
```

`src/encoding.js` and `src/exec.js` import nothing from `@deepseek-ai/*`. `src/index.js` is dependency-free too: it emits the registry-ready tool shape directly (the same `{type:'object', properties, required}` + `output.schema/render` form that the official `defineTool` compiles to — locked by literal-schema assertions in `test/index.test.js`, and accepted by the live 0.2.0-rc.2 registry). This is deliberate — when a plugin is installed as a **local checkout** (`dsh plugin add ./path`, a pnpm link), the package lives outside the profile tree, and Node's ESM resolver cannot reach the host's `@deepseek-ai/*` packages from the checkout directory (the import dies with `ERR_MODULE_NOT_FOUND`, verified on 0.2.0-rc.2). Emitting the registry shape directly keeps the plugin working under npm, git, and local-link installs alike.

### Known limitations

- **Does not replace or fix** the built-in `bash`/`pwsh` tools, `ctx.shell`, or `dsh-subprocess-local` — those are out of scope by design (version-fragile core work). This plugin is a *supplement*, and says so.
- No sandbox / approval / background-task integration for `encoding_run`.
- Encoding detection is heuristic (UTF-8 validity, UTF-16LE NUL pattern, GBK high-byte ratio, BOM). Ambiguous cases exist — e.g. BOM-less pure-CJK UTF-16LE vs GBK — so force `encoding` when you know the answer.
- U+FFFD in the input is information already destroyed; recovery is impossible and is reported as such.
- Output is capped at 8 MiB per stream; on timeout the process tree is killed (Windows: `taskkill /T /F`, POSIX: process-group SIGKILL).

### Publishing checklist

- [x] GitHub repository with topic `dsh-plugin` — [Lostforest7/dsh-encoding](https://github.com/Lostforest7/dsh-encoding)
- [x] npm name `dsh-encoding` (was unoccupied; published `0.1.0` → `0.2.0`)
- [x] `npm publish --registry=https://registry.npmjs.org` (China mirrors are read-only; always publish against the official registry)
- [ ] Submit to the `awesome-dsh-plugin` list and the dsh-plugin.org market (optional, not yet done)

## 中文

`dsh-encoding` 是 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)（dsh）的一个**薄工具插件**。在中文 Windows 上，老式 CLI、Git Bash/MSYS2、PowerShell 5.1 经常输出 GBK（cp936）或 UTF-16LE 字节，被 dsh 内置的 `bash`/`pwsh` 工具按 UTF-8 解码后变成乱码；而解码发生在不可插拔的核心服务（`dsh-subprocess-local`）里。本插件**不去修补内置 bash**，而是新增两个补充工具：

| 工具 | 作用 |
| --- | --- |
| `encoding_run` | 自己 spawn 子进程、按**原始字节**收 stdout/stderr，自动识别 UTF-8 / UTF-16LE / GBK 并正确解码。 |
| `encoding_decode` | 把已经乱码的文本（如 GBK 字节被当 latin1 误读）还原成正确中文。 |

> **0.2.0 破坏性变更**：工具名由 `run`/`decode_text` 改为 `encoding_run`/`encoding_decode`（防撞名前缀）。0.1.0 已在 npm 标记废弃。

### 与官方内置工具的关系（在 dsh 0.2.0-rc.2 上实测）

- dsh 0.2.0 **没有**内置 `run` / `decode_text` 工具。用干净的、未装插件的临时 profile（`dsh --profile pmaudit --dump-config`）验证：base 配置中没有任何同名工具行（内置工具是 `tool-bash`、`tool-pwsh`、`tool-fs` 等），官方 tools 参考文档也没有这两个工具。
- 内置 `bash`/`pwsh` 依然**没有编码配置**（shell 接口的 `ShellExecRequest` 无编码字段）；在本环境用内置 `pwsh` 工具跑 GBK 输出命令，结果仍是乱码。本插件的立论前提成立。
- 工具名加了 `encoding_` 前缀，避免与内置或未来 dsh 工具撞名——注册表会拒绝同层重复名与保留名 `run_code`（官方 tools.md）。

### 安装

从 npm：

```sh
dsh plugin --profile <name> add dsh-encoding
```

从本地目录（本仓库）：

```sh
dsh plugin --profile <name> add ./dsh-encoding
```

从 GitHub（纯 ESM JavaScript、**无构建步骤**，不需要 `prepare` 构建授权）：

```sh
dsh plugin --profile <name> add github:Lostforest7/dsh-encoding
```

验证配置层已生效，然后启动：

```sh
dsh --profile <name> --dump-config   # 能看到 "# == dsh-encoding" 层
dsh --profile <name>
```

### 工具 `encoding_run`

| 参数 | 类型 | 必填 | 默认 | 说明 |
| --- | --- | --- | --- | --- |
| `command` | string | 是 | — | 要执行的命令（Windows：`cmd /d /s /c`；POSIX：`/bin/bash -c`）。 |
| `workdir` | string | 否 | — | 子进程工作目录。 |
| `timeoutMs` | number | 否 | `120000` | 超时毫秒数，到期后杀掉整个进程树。 |
| `encoding` | string | 否 | `auto` | `auto` 自动检测，或强制 `utf-8` / `utf-16le` / `gbk`。 |

输出示例（Windows 上的 GBK 输出）：

```
中文
[stderr]
...（如有 stderr）
[encoding: gbk/utf-8 (auto-detected)]
[exit code: 0]
```

最后一行固定是 `[exit code: N]`。当输出含 U+FFFD（�）时，会附加明确的警告行，而不是假装解码正确。

> **定位如实说明**：`encoding_run` 是一个**独立的新工具**，不带内置 `bash` 工具的沙箱、审批与后台任务能力，并以 dsh 进程自身的权限执行命令。

### 工具 `encoding_decode`

| 参数 | 类型 | 必填 | 默认 | 说明 |
| --- | --- | --- | --- | --- |
| `text` | string | 是 | — | 要还原的乱码文本。 |
| `from` | string | 否 | `auto` | 字符还原成字节的方式：`latin1` / `utf-16le` / `auto`。 |
| `to` | string | 否 | `auto` | 目标解码：`gbk` / `utf-16le` / `auto`。 |

经典"乱码还原"路径是 `from=latin1, to=gbk`：GBK 字节被当 latin1 误读（一个字符一个字节），工具把每个字符还原成一个字节，再按 GBK 解码。

`auto` 模式下自动判断误读方向：文本含 NUL → `latin1 → utf-16le`；含宽字符 → `utf-16le → gbk`；其余 → 经典 `latin1 → gbk`。

**不可逆情况明确说明、不伪造结果**：若输入里已经含 U+FFFD（�）——即字节早已在上游误解码中被毁掉——返回 `ok: false` 并附说明。

> 注意：`from=gbk` 会被明确拒绝并解释原因——Node.js 内置了 GBK **解码器**（TextDecoder）但没有 GBK **编码器**，本插件又坚持零依赖，因此无法把字符重新编码回 GBK 字节。`to=gbk` 完全支持。

### 30 秒示例

内置 `bash` 把 `中文` 显示成了 `ÖÐÎÄ`（GBK 字节被当 latin1 读）：

1. 让 agent 调用 `encoding_decode`，参数 `{"text": "ÖÐÎÄ"}` → 得到 `中文`，完成。
2. 或者重新执行命令并按正确编码解码：调用 `encoding_run`，参数 `{"command": "chcp 936 >nul & echo 中文"}` → 输出 `中文`，并附 `[encoding: gbk (auto-detected)] [exit code: 0]`。

### CLI 手动冒烟（无需 DSH）

CLI 子命令保留短名（`run` / `decode_text`）——它们只是开发冒烟入口，不是注册的工具名：

```sh
node src/cli.js decode_text "ÖÐÎÄ"                  # -> 中文
node src/cli.js run "chcp 936 >nul & echo 中文"      # GBK 输出自动解码
node src/cli.js run "echo hello"
node src/cli.js detect D6D0CEC4                      # -> detected: gbk / decoded: 中文
```

### 本地测试

零依赖——运行时与开发时都没有任何依赖；测试只用 Node 内置的 `node:test` / `node:assert`，无需 `npm install`、无需 DSH、无需构建：

```sh
node --test test/
```

结构：

```
dsh-encoding/
├── package.json          # type: module，dsh.bundle.patch -> cordis.patch.yml
├── cordis.patch.yml      # 插入一行 loader：id "encoding"，name "dsh-encoding"
├── src/index.js          # Cordis 入口（零依赖），注册 encoding_run 与 encoding_decode
├── src/encoding.js       # 纯函数：检测 / 解码 / 乱码还原
├── src/exec.js           # spawn 封装 + run 输出组装（spawnImpl 可注入）
├── src/cli.js            # 手动冒烟（无需 DSH）
└── test/                 # node:test 套件，mock child_process.spawn
```

`src/encoding.js` 与 `src/exec.js` 不 import 任何 `@deepseek-ai/*`；`src/index.js` 同样零依赖：它直接输出注册表形态的工具定义（与官方 `defineTool` 编译产物一致的 `{type:'object', properties, required}` + `output.schema/render`，由 `test/index.test.js` 的字面 schema 断言锁定，并已被 0.2.0-rc.2 真实注册表接受）。这是刻意的——当插件以**本地目录**方式安装（`dsh plugin add ./path`，即 pnpm link）时，包位于 profile 树之外，Node 的 ESM 解析无法从插件目录找到宿主的 `@deepseek-ai/*` 包（会以 `ERR_MODULE_NOT_FOUND` 失败，已在 0.2.0-rc.2 实测）。直接输出注册形态让插件在 npm、git、本地 link 三种安装方式下都能工作。

### 已知限制

- **不替换、不修复**内置 `bash`/`pwsh`、`ctx.shell`、`dsh-subprocess-local`——刻意超出范围（核心层改动版本脆弱）。本插件定位为"补充工具"。
- `encoding_run` 无沙箱集成、无审批、无后台任务。
- 编码检测是启发式的（UTF-8 合法性、UTF-16LE NUL 模式、GBK 高位字节占比、BOM）。存在歧义场景——例如无 BOM 的纯中文 UTF-16LE 与 GBK 难以区分——已知编码时请用 `encoding` 参数强制指定。
- 输入中已有的 U+FFFD 属于已丢失的信息，无法还原，并会如实报告。
- 每路输出上限 8 MiB；超时杀进程树（Windows：`taskkill /T /F`；POSIX：进程组 SIGKILL）。

### 发布清单

- [x] GitHub 仓库已建并带 topic `dsh-plugin`：[Lostforest7/dsh-encoding](https://github.com/Lostforest7/dsh-encoding)
- [x] npm 名 `dsh-encoding`（确认未占用；已发布 `0.1.0` → `0.2.0`）
- [x] `npm publish --registry=https://registry.npmjs.org`（国内镜像只读，务必用官方 registry 发布）
- [ ] 提交到 `awesome-dsh-plugin` 列表与 dsh-plugin.org 市场（可选，尚未完成）

## License

MIT — see [LICENSE](LICENSE).