dsh-webfile

Readme

# dsh-webfile

[English](README.md) | [中文](README.zh.md)

S3 & FTP file tools for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness): 8 agent tools (list, stat, mkdir, delete, move, copy, download, upload) — approval-gated mutations, transfer jobs with progress & cancel.

Let your agent browse and manage files on S3 (including MinIO and compatible object stores), Xiaomi Galaxy FDS, and FTP/FTPS directly in conversation:

- **Read-only pass-through**: `webfile_list` / `webfile_stat` never require approval;
- **Per-call approval for mutations**: the other 6 tools ask the user before every call — a rejection means zero side effects;
- **Transfers run as background jobs**: download/upload stream progress into the Jobs panel and can be cancelled at any time;
- **Credentials by reference only**: secret values never enter config or the session log; rotations take effect on the next call.

## Install

```sh
dsh plugin --profile <name> add dsh-webfile
```

The package declares `dsh.bundle`, so installing **auto-mounts** it into the profile's layer stack — no manual rows needed. To upgrade:

```sh
dsh plugin --profile <name> update
```

## Configuration

Override the plugin config by id in the profile's `cordis.patch.yml` (`$DSH_HOME/profiles/<name>/cordis.patch.yml`):

```yaml
- id: dsh-webfile
  config:
    connections:
      prod-logs:
        protocol: s3
        endpoint: https://oss.example.com   # omit for AWS public cloud
        region: cn-north-1
        bucket: prod-logs
        pathStyle: true                     # required for MinIO
        accessKeyRef: OSS_ACCESS_KEY        # credential reference name, see below
        secretKeyRef: OSS_SECRET_KEY
      legacy-ftp:
        protocol: ftp
        host: ftp.example.com
        port: 21
        userRef: FTP_USER
        passwordRef: FTP_PASSWORD
        tls: explicit                       # none | explicit | implicit
      mi-fds:
        protocol: fds
        endpoint: https://cnbj2.fds.api.xiaomi.com   # minimal config: endpoint + bucket + ak/sk
        bucket: mi-bucket
        accessKeyRef: FDS_ACCESS_KEY        # required: FDS has no ambient credential chain
        secretKeyRef: FDS_SECRET_KEY
        # region: cnbj2                     # or set region instead and let it derive the host
        # https: false                      # default true; set false for plain HTTP
    maxTransferBytes: 2147483648            # per-file transfer cap, default 2 GiB
    multipartThresholdBytes: 67108864       # S3/FDS multipart threshold, default 64 MiB
```

### Connection fields

| Field | Protocol | Notes |
|---|---|---|
| `protocol` | all three | `s3`, `fds`, or `ftp` (required) |
| `endpoint` | s3 | Custom endpoint (MinIO etc.); omit for AWS public cloud |
| `region` | s3 | Region (required) |
| `bucket` | s3 | Default bucket |
| `pathStyle` | s3 | Path-style addressing; MinIO needs `true` |
| `accessKeyRef` / `secretKeyRef` / `sessionTokenRef` | s3 | Credential reference names, see below |
| `region` | fds | FDS region (`cnbj2` / `awsbj0` / `awsusor0` / `awssgp0` / `awsde0` / `ksyru0-eco` / `awsind0-eco`); **either region or endpoint** |
| `endpoint` | fds | Full host with or without scheme; takes precedence over region — when set, region may be omitted |
| `bucket` | fds | Default bucket |
| `https` | fds | Default `true`; FDS also serves plain HTTP |
| `accessKeyRef` / `secretKeyRef` | fds | Credential reference names (**required** — FDS has no ambient credential chain), see below |
| `host` | ftp | Server address (required) |
| `port` | ftp | Default 21; 990 for implicit FTPS |
| `userRef` / `passwordRef` | ftp | Credential reference names; omit for anonymous login |
| `tls` | ftp | `none` / `explicit` / `implicit` (required) |
| `passive` | ftp | Default `true`; only passive mode is supported |

### Patch semantics

- **Target by id**: patch rows without an `id` are skipped with a warning;
- **Whole-value replace**: `config` shallow-replaces the entire value; omitted fields fall back to schema defaults;
- **Do not re-mount**: never `insert` another row with the same name in the user layer — configure through id overrides only;
- **Disable**: `- id: dsh-webfile / disabled: true` turns the plugin off for that profile;
- **Hot reload**: long-lived surfaces watch this file — saving replays the patch transactionally via HMR, **no restart needed**; a parse failure fails loud and keeps the last good tree.

## Credentials

The config only holds **credential reference names** (environment-variable names, POSIX identifiers). Secret values live in `$DSH_HOME/.credentials.yaml`:

```yaml
OSS_ACCESS_KEY: AKIAxxxxxxxx
OSS_SECRET_KEY: xxxxxxxxxxxx
FDS_ACCESS_KEY: your FDS access key   # created at dev.mi.com / the Mi Fusion Cloud console
FDS_SECRET_KEY: your FDS secret key
FTP_USER: logbot
FTP_PASSWORD: xxxxxxxx
```

- **Four-layer precedence**: process environment > credentials file > project `.env` > user `.env`; the process environment is read-only and shadows same-named file entries;
- **Hot publish**: the file is watched (100 ms debounce) — an external edit takes effect on the **next tool call**, zero-restart rotation;
- **Format**: strictly a `reference: string value` map (not dotenv syntax); values must be non-empty; file permissions must be `0600`;
- Secret values never enter the config surface or the session log.

## Tools

| Tool | Parameters | Approval | Background job |
|---|---|---|---|
| `webfile_list` | `connection` · `remotePath` · `maxEntries` (default 200, cap 1000) · `nextToken` | none | no |
| `webfile_stat` | `connection` · `remotePath` | none | no |
| `webfile_mkdir` | `connection` · `remotePath` | **required** | no |
| `webfile_delete` | `connection` · `remotePath` · `recursive` (default false) | **required** | no |
| `webfile_move` | `connection` · `sourcePath` · `targetPath` · `overwrite` (default false) | **required** | no |
| `webfile_copy` | `connection` · `sourcePath` · `targetPath` · `overwrite` (default false) | **required** | no |
| `webfile_download` | `connection` · `remotePath` · `localPath` (optional; defaults to mirroring under the workspace root) | **required** | **yes** |
| `webfile_upload` | `connection` · `localPath` · `remotePath` · `overwrite` (default false) | **required** | **yes** |

## Semantics

- **S3 has no real directories**: directories are key-prefix aggregations; `mkdir` writes a zero-byte `key/` marker; `stat` synthesizes a directory for prefixes that have children but no object of their own.
- **FDS shares the same model**: FDS also simulates folders with `/`; the protocol differences stay inside the provider (see the FDS section below).
- **List pagination**: truncated results return `truncated: true` plus a `nextToken` (S3 `ContinuationToken` / FDS `marker` passthrough) — pass it back to continue; FTP has no server-side continuation, so retry with a larger `maxEntries`.
- **Overwrite protection**: `overwrite` defaults to `false`; an existing target raises `WEBBUF_EXISTS` with a hint to enable it.
- **Recursion protection**: `delete` refuses non-empty directories with `WEBBUF_DIR_NOT_EMPTY`; `recursive: true` deletes the whole subtree.
- **Transfer cap**: `maxTransferBytes` (default 2 GiB) is checked before the operation starts; oversized transfers raise `WEBBUF_TOO_LARGE` with zero remote side effects.
- **S3/FDS large uploads**: at or above `multipartThresholdBytes` (default 64 MiB) uploads use multipart with monotonic progress.
- **Cancellation**: killing a transfer job aborts the stream, removes the local half-file, and settles the job as `killed`.
- **S3 directory move/copy**: per-object copy (+delete) — large directories are slow and an interruption can leave a half-moved state, which the tool description and approval reason state explicitly.
- **FDS directory move/copy**: also per-object copy + batch delete (see below); single-file moves use FDS's native rename, so they cost no copy at all.
- **FTP symlinks**: surfaced as `type=link`; recursive delete removes the link itself and never follows it; copy refuses symlinks outright.

### FDS details

- **Protocol & signature**: FDS is its own REST API, not S3 on the wire. Requests carry `Authorization: Galaxy-V2 {AK}:{Sig}` with `Sig = Base64(Hmac-SHA1(SK, StringToSign))`; the string covers Method / Content-MD5 / Content-Type / **Date** (required — keep the system clock in sync with FDS) plus canonicalized `x-xiaomi-*` headers and subresources (`acl/quota/uploads/partNumber/uploadId/storageAccessToken/metadata`).
- **Endpoint**: defaults to `{region}.fds.api.xiaomi.com`; `endpoint` overrides it (internal `xxx-fds.api.xiaomi.net`); `https: false` switches to plain HTTP.
- **Native capabilities**: single-file move uses `renameTo` (server-side rename, no copy+delete round-trip); recursive directory delete uses the `deleteObjects` batch endpoint.
- **Multipart uploads**: parts are fixed at 16 MiB (FDS requires 5–50 MiB per part, consecutively numbered); cancelling best-effort aborts the upload so stray parts stop billing.
- **Platform limits**: objects up to 100 GiB (cnbj2 and newer regions; the retired cnbj0 allowed only 2 GiB); bucket names are 3–63 bytes with DNS rules; FDS offers no S3-style atomicity guarantees for large directory trees.

## Web transfer card

In the Web GUI, every `download` / `upload` opens a **transfer card in the chat flow** (`webfile-transfer` conversation node): direction, label, a live status dot (animated while transferring), elapsed time, source/target paths, and the terminal result. Data comes in two layers:

- **Durable layer**: the host half records `webfile/transfer-start` / `webfile/transfer-end` session events around each transfer, which place the card in the message flow and let it replay the terminal state even after the registry drops the job.
- **Live layer**: the client renderer reads the job's live status, timestamps, and terminal detail from the `jobsBySession` mirror, falling back to the durable events once the registry drops it.

The card ships through the `dsh.client` declaration (`package.json` → `dsh.client` + `exports["./client"]`); `pnpm build` emits `lib/client.js`. It only appears when the composing profile includes this package and the Web server has restarted and the page refreshed (the client plugin table is scanned at boot). Outside a Web composition it costs nothing: events are still recorded and the model-facing Jobs panel is unchanged.

## Security notes

- **Per-call authorization**: every mutation asks independently — no session-wide exemptions; `allowed-once` is the only grant.
- **Fail closed**: with no approval channel (e.g. pure headless without an answerer) or with session policy `approval/policy: never`, mutation tools are always denied — never silently allowed.
- **Session log**: tool arguments (connection id, remotePath, …) enter the session log with the call — mind paths that are sensitive; secret values never do.
- **S3 `CopyObject` cannot be aborted**: server-side copies, once issued, cannot be cancelled — relevant for large directory move/copy.
- **Downloads land through `ctx.fs`** into the workspace, governed by the DSH file sandbox policy.

## Error codes

| Code | Meaning |
|---|---|
| `WEBBUF_UNKNOWN_CONNECTION` | Connection id not configured (message lists the available ids) |
| `WEBBUF_CREDENTIAL_MISSING` | A declared credential reference has no value (message names the variable) |
| `WEBBUF_NOT_FOUND` | Path does not exist |
| `WEBBUF_EXISTS` | Target exists and `overwrite: false` |
| `WEBBUF_DIR_NOT_EMPTY` | Non-empty directory and `recursive: false` |
| `WEBBUF_PATH_TRAVERSAL` | Path contains `..` or is absolute |
| `WEBBUF_TOO_LARGE` | Exceeds `maxTransferBytes` |
| `WEBBUF_PROTOCOL_ERROR` | Protocol-level error (e.g. FTP refusing to follow a symlink) |

## Compatibility

- **Node**: `^22.19.0 || >=24.0.0` (aligned with DeepSeek Harness).
- **S3**: any object store implementing ListObjectsV2 / HeadObject / GetObject / PutObject / DeleteObject(s) / CopyObject / multipart (AWS, MinIO, and S3-compatible services).
- **FDS**: Xiaomi Galaxy FDS regions cnbj2 / awsbj0 / awsusor0 / awssgp0 / awsde0 / ksyru0-eco / awsind0-eco (native REST + Galaxy-V2 signing; keys from dev.mi.com).
- **FTP/FTPS**: MLSD-capable servers yield the fullest listings (precise timestamps); older LIST-only servers work but lack time fields and infer directory types from permissions; both explicit and implicit TLS are supported.
- One short-lived connection per operation in v1 (no pooling) — extremely high-frequency workloads are a later milestone.

## Development

The committed manifest and lockfile use registry packages so local development and CI resolve the same dependency graph:

```sh
pnpm install
pnpm check   # build + lint + tests
```

For cross-repository development, use `pnpm link <package-dir>...` to replace selected packages in `node_modules` without editing `package.json` or `pnpm-lock.yaml`.

Commit convention: `<type>(<scope>): <summary>`. The lefthook commit-msg hook enforces local commits after `pnpm exec lefthook install`; CI applies the same rule to pull request titles.

**Releasing**: fully automated via [semantic-release](https://semantic-release.gitbook.io/) — every merge to `main` analyzes the conventional commit history, bumps the version (fix → patch, feat → minor, e.g. 0.1.1 → 0.2.0), publishes to npm, pushes a version tag, and files a GitHub Release whose notes are generated from the commits. The source manifest intentionally remains at `0.0.0`; the Git tag is the version record.

The first publish needs a granular automation token in the `NPM_TOKEN` GitHub Actions secret. Once the package exists, configure npm Trusted Publishing for `modestoma/dsh-webfile` and workflow `ci.yml`, then delete the secret and its workflow environment entry. Later publishes use short-lived OIDC credentials and carry npm provenance attestations.

## License

[MIT](LICENSE) © 2026 modesto