dsh-agent-evaluator

Readme

# dsh-agent-evaluator

An agent evaluation plugin for DeepSeek Harness: import test sets, run agents, score results, and generate reports.

[Chinese](README.zh.md)

## Install

```sh
cd /path/to/deepseek-harness
pnpm dsh plugin --profile web add ./dsh-agent-evaluator-0.10.0.tgz
pnpm dsh web
```

If `dsh` is installed globally, use it instead of `pnpm dsh`. Cloning the repository or using a git URL also works.

## Usage

Web UI: Settings → Agent Evaluation, or run `/eval` commands:

```text
/eval import /path/to/suite.jsonl
/eval model provider/model
/eval judge provider/model
/eval run suite --parallel 4 --retries 1
/eval progress <jobId>
/eval report <jobId>
```

Subcommands: `import`, `list`, `drop`, `model`, `models`, `judge`, `run`, `resume`, `kill`, `runs`, `progress`, `report`.

Headless / CI:

```sh
cd /path/to/deepseek-harness
pnpm dsh --profile eval eval import suite.jsonl
pnpm dsh --profile eval eval model provider/model
pnpm dsh --profile eval eval run suite --parallel 4
```

## Features

- Imports JSONL / JSON / CSV / TSV and directories, adapting common benchmark field names.
- Runs each case in an isolated agent with concurrency, retries, and timeouts.
- Scorers: `exact`, `contains`, `llm`.
- Persists reports under `$DSH_EVAL_HOME/eval/reports`, with checkpoint/resume support.
- Web panel provides model pickers, live progress bars, and pass/fail charts.

## Documentation

- [INSTALL.md](INSTALL.md) / [INSTALL.zh.md](INSTALL.zh.md) — installation
- [USAGE.md](USAGE.md) / [USAGE.zh.md](USAGE.zh.md) — full usage
- [AGENTS.md](AGENTS.md) / [AGENTS.zh.md](AGENTS.zh.md) — architecture and maintenance

## Development

```sh
node --test test/*.test.js
```

Integration tests and client builds require a deepseek-harness checkout; see [AGENTS.md](AGENTS.md).

MIT License, see [LICENSE](LICENSE).