@hackerfish/dsh-video-studio

Readme

# 🐳 dsh-video-studio (Whale) — AI Video & Motion-Comic Studio for DeepSeek Harness

**A DeepSeek Harness native plugin for AI video and motion-comic (漫剧) production: seven-stage director pipeline × multi-account quota pool with failure fallback × credential vault × four-layer prompt engineering with a score-feedback loop. Quality first, cost second.**

[中文](README.zh.md) · Siblings: [dsh-lab](https://github.com/hackerFish/dsh-lab) · [awesome-dsh-skills](https://github.com/hackerFish/awesome-dsh-skills) · [awesome-dsh-presets](https://github.com/hackerFish/awesome-dsh-presets)

> The DSH ecosystem has 1000+ plugins — none does generative video. Whale brings the industry-validated motion-comic pipeline (waoowaoo / LumenX / MangaV / ArcReel pattern) into DSH as a hot-pluggable, per-step-controllable plugin.

**Live demo — real AI-generated video (wan2.2-t2v-plus, DashScope, 1080p 5s):**

![whale live demo](demos/whale-live-demo.gif)

[Quantified project assessment](docs/PROJECT-ASSESSMENT.md)

## Pipeline — the industry-standard seven-stage workflow

```
story (LLM, e.g. Doubao writes the novel) → script (LLM breaks it down)
→ storyboard (LLM shot list) → master asset (MJ-style hero image)
→ shot assets (image-model variations for consistency) → video (Seedance/Jimeng/Kling)
→ final cut (JianYing draft export / local ffmpeg render)
```

- **The first three stages are LLM stages**: inside DSH the session model itself does them — the model you're chatting with (Doubao, DeepSeek, whatever) is the brain that writes the story, script and shots; the plugin never calls an external LLM.

**Mapping to the industry-standard toolchain**:

| Industry flow | Whale |
|---|---|
| Doubao writes novel / script / shots | = DSH session model doing story/script/storyboard (pick a Doubao model and it IS "Doubao writes") |
| MJ hero asset | = master-asset stage (MJ official API adapter to come; wanx/Seedream can substitute) |
| image variations | = shot-assets stage (wanx ✅ verified · **Doubao Seedream ✅ wired**) |
| Seedance video | = video stage (**Doubao Seedance ✅** / jimeng / kling / ComfyUI) |
| JianYing final cut | = final-cut stage (JianYing draft export ✅ / ffmpeg ✅) |
- Every stage has a gate: auto / ask / manual.
- Parallel shots, quota scheduler, style genome, distribution pack as before.

- **Consistency asset library**: character/scene master assets + per-shot variations with automatic reference-image injection into prompts (the motion-comic standard technique)
- **Parallel shots**: batch submit → concurrent polling (configurable concurrency)
- **Account pool (quota scheduler)**: multi-account rotation per provider, per-day caps, exponential backoff on failure, automatic fallback re-submit to the next healthy account mid-pipeline, full audit trail
- **Credential vault**: `~/.whale/whale.json` (0600, atomic writes, follows `$DSH_HOME`), masked API responses, account management UI in the plugin settings (鲸影账号 tab)
- **Runtime account wiring**: accounts added in the UI feed straight into `whale_generate_video` — vault → pool pick (rotation/backoff) → per-account provider binding (`src/host/account-providers.ts`, single-field credentials as plain strings, multi-field as JSON) → usage/health persisted back to the vault
- **Style genome (memory)**: style DNA, shot-template scoring evolution, retry feedback — persists across sessions
- **Prompt engineering**: parameterized professional template library (character sheet / scene master / single shot) + composable quality boosters (8K / clean bg / neutral face / no text …) + optimizer; `whale_optimize_prompt` upgrades drafts to pro-grade prompts locally
- **Score-feedback loop (评分回写)**: every reviewed shot writes its score + booster combo back to the scorebook; the optimizer then picks boosters by real historical performance
- **Preset motion-comic pack**: 5 genres (city comeback / xianxia / suspense / sweet romance / sci-fi) with bilingual character cards, scene cards and shot scripts — `whale_story_presets` turns one preset id into a pipeline-ready script, `scripts/demo-presets.ts` runs it end-to-end with the mock provider (zero keys)
- **Self-analysis (自我审计)**: `whale_self_audit` tool + `npm run self-audit` scan the repo itself — source modules, test counts, provider matrix, capability inventory and a gap list — and write a generated report to `docs/AUDIT-REPORT.md`; the daily diff of that file IS the progress log. Session lessons are distilled into [docs/RETROSPECTIVE-2026-08-17.md](docs/RETROSPECTIVE-2026-08-17.md)

## Providers (verified matrix)

| Provider | Channel | Status |
|---|---|---|
| jimeng (即梦) | sessionid, free daily quota | ✅ protocol verified end-to-end; **text-to-video queue stays `SystemBusy` even off-peak (0 credits consumed)** — free route is now: wanx images → image-to-video |
| tongyi-wanx (通义万相) | cookie+xsrf, free credits | ✅ **live-verified: real whale image generated & downloaded** (free tier = text-to-image; video needs membership) |
| kling official (可灵) | accessKey:secretKey JWT, api-beijing.klingai.com | ✅ adapter written — not yet tested against a real key |
| kling via DashScope | `sk-` key | ✅ adapter written — not yet tested against a real key |
| **kling-lipsync (可灵对口型)** | official JWT, `/v1/videos/lip-sync` | ✅ adapter written against the official 3-13 contract (audio2video + text2video voice modes), 8 tests — not yet tested against a real key |
| **wan video via DashScope (通义万相视频)** | `sk-` key | ✅ **LIVE-VERIFIED: real whale video generated** (wan2.2-t2v-plus, 1920×1080 5s, ~90s turnaround) — ⚠️ metered at ¥0.70/s (1080p), NOT free quota; `demos/dashscope-wan-live.mp4` |
| doubao (火山方舟) | ARK API key | ✅ Seedance video + **Seedream image** (assets) — not yet tested against a real key |
| **doubao-web (豆包网页版)** | cookie, free web quota | ✅ **live-replayed & parsed**: SSE chat for the LLM stages (story/script/shots) + image bot for assets. Pro-tier free quota runs on a 7-day window (image bot pauses when spent; text keeps working) |
| ComfyUI local | workflow JSON builder + /prompt protocol | ✅ protocol-tested (mock server), real GPU pending |
| kling web (sessionid) | anti-bot one-time falcon token | 📄 anatomy documented; automation needs a capture bridge (deferred) |

## Editing & distribution

- **Final cut with optional lip-sync stage**: with a `capabilities.lipSync` provider configured, every voiced shot runs audio-driven lip sync (video reference + audio base64) and the synced clip replaces the original on the timeline; failure falls back to the original clip, never blocking the render. Shots can also carry a pre-recorded `voiceFile` (external TTS/voice actor) that skips local TTS
- **ffmpeg auto-render** — verified end-to-end (synthetic clips → timeline → burned subtitles → audio mix → final mp4, duration-checked)
- **JianYing (剪映) draft export** — editable tracks/keyframes/subtitles for manual polish; structure-validated
- **say TTS** — real Chinese voiceover with zero API keys: macOS `say`; Windows PowerShell SAPI (`System.Speech`, needs a Chinese voice pack). Any platform can also plug external audio per shot via `voiceFile`
- **Distribution pack** — platform specs + compliance precheck for 4 Chinese platforms

## DSH integration (deep invocation)

- **Model tools**: `whale_story_presets` (5-genre content pack → pipeline script), `whale_storyboard` (offline shot planning), `whale_generate_video` (provider routing), `whale_optimize_prompt` (pro-grade prompt upgrade), `whale_quality_review` (rule-level QC), `whale_comfyui_workflow` (ComfyUI workflow JSON), `whale_self_audit` (project self-analysis)
- **Host routes**: `/dsh-video-studio/health`, `/dsh-video-studio/runs`, `/dsh-video-studio/accounts` (GET list masked / POST add / DELETE remove)
- **Client UI**: 鲸影 (status) · 鲸影工作台 (live pipeline progress per run) · 鲸影账号 (account vault management) tabs in plugin settings; `whale_generate_video` gets a dedicated video card view
- Installs via `dsh plugin add`, boot-verified clean
- Planned: slash commands, subagent-parallel shots

## Verification discipline

123 unit tests green (account pool rotation/backoff/fallback, credential vault, runtime account→provider wiring, quota routing, prompt merging, score-feedback loop, lip-sync stage incl. fallback, Windows SAPI script + CJK font candidates, jianying draft structure, ffmpeg end-to-end render, provider protocols via mock servers incl. kling lip-sync, preset pack integrity, self-audit, live jimeng model probe, live wanx image generation). Test logs and proof artifacts live in `demos/`. The provider matrix lives in `src/selfaudit/matrix.ts` (single source of truth shared by the health route, the vault whitelist and the audit report).

## Install

```bash
dsh plugin --profile web add github:hackerFish/dsh-video-studio --ignore-workspace-root-check
# or npm once published
```

## Honesty notes

Model output quality is bounded by the vendor model; the pipeline maximizes it (consistency tokens, QC retry loop, score-feedback booster selection). sessionid/cookie usage is per-platform ToS — respect each platform's terms. Credentials live only in the local vault file and are never logged or returned unmasked. Not affiliated with DeepSeek.

## License

[MIT](LICENSE)