Comfyui Agent Skill Mie

comfyui-agent-skill-mie

Purpose

Run registered ComfyUI workflows through a stable Agent-facing CLI, with prompt enhancement, fail-fast errors, and structured JSON results.

Use this skill when the user asks to:

• Generate an image from text.
• Generate a new image inspired by a reference image.
• Edit an input image while preserving some structure or subject details.
• Generate text-to-video or image-to-video MP4 output.
• Generate music / instrumental / song-style MP3 output.
• Synthesize spoken voice audio with Qwen3-TTS.
• Cut out or mask an object in a picture by describing it in words.
• Transfer a face, expression, or pose from a driving video onto a reference face photo.
• Clone a specific voice from a reference audio sample and speak new text in that voice.
• Check whether a ComfyUI server is available.
• Import a workflow the user already validated in ComfyUI into their private registry, and activate it only after the generated config is reviewed.
• Preflight a registered workflow's nodes and models before a long run.
• Save a persistent ComfyUI server URL when the user asks.

Do not use this skill when the user only wants prompt writing, brainstorming, or discussion without actual generation.

A ComfyUI server being unavailable is not a reason to skip this skill. Return SERVER_UNAVAILABLE, ask whether ComfyUI is running on the local machine or on another machine, and point the user at check (single health check) and doctor (server health plus per-workflow preflight) to diagnose it.

Hard Rules

• Source mode: run CLI commands from the skill root (the directory containing SKILL.md and scripts/).
• Tool-install mode: comfyui-agent-skill-mie / comfyui-skill can be run from any directory.
• Source mode: use uv run --no-sync python -m comfyui (or uv run --no-sync comfyui-skill) for runtime calls.
• Tool-install mode: use comfyui-skill (or comfyui-agent-skill-mie) directly; do not wrap with uv run.
• Use registered workflows only. Do not run arbitrary unreviewed ComfyUI workflow JSON.
• If server health fails, stop generation and return/handle SERVER_UNAVAILABLE; do not search disk for ComfyUI installs or guess ports.
• Do not create or edit config.local.json unless the user explicitly wants a persistent server URL. For one-off runs, use --server or COMFYUI_URL.
• For reference_to_image, inspect the reference image with Agent vision and create a prompt. Do not upload that reference image to ComfyUI.
• Upload local media with the role's flag from the quick command table. --image is the image role. liveportrait also needs --video. qwen3_tts_clone needs -p, --text-input "ref_text=...", and --audio. Do not choose those flags from the capability name.
• Pass --width/--height only when the workflow config maps both dimensions and does not use size_strategy: "workflow_managed"; always pass both together. Otherwise the CLI returns INVALID_PARAM on both the sync and --submit paths instead of ignoring them.
• Analyzer-generated workflow configs require human review before activation.

Workflow Selection Policy

• Built-in defaults are fallback choices, not hard requirements.
• If another registered workflow is a stronger semantic match for the request, prefer the stronger match.
• Use workflow capability metadata and workflow selection guidance to choose among registered workflows.
• Any automatic workflow selection in the CLI is only a low-risk fallback when no explicit workflow has been chosen.
• If multiple workflows appear suitable and the user’s preference is ambiguous, ask a brief clarifying question.

User-Added Workflows

• Users may add custom workflows that were first validated in ComfyUI, then imported into the skill as reviewed registered capabilities.
• Do not execute arbitrary raw workflow JSON directly.
• Import the workflow, generate/review the config template, and activate it only after a reviewed workflow.config.json exists.
• User-added workflows must be stored in the per-user workflow registry so skill upgrades do not overwrite them.
• Once activated, user-added workflows may be selected like other registered workflows when they are a stronger semantic match and required inputs are available.
• import-workflow only accepts API format (ComfyUI's Save (API) button, requires Settings → Enable Dev mode options). A plain Save product (UI/Save format, {nodes, links}) is refused with WORKFLOW_NOT_API_FORMAT. To convert existing UI/Save workflows in place, run convert-ui (needs a running ComfyUI + Playwright).

Setup

Recommended install (tool-install mode):

bashCopy code
pipx install comfyui-agent-skill-mie

• Install package: comfyui-agent-skill-mie
• Main command: comfyui-agent-skill-mie
• Short alias: comfyui-skill

Prerequisites:

• ComfyUI server with GET /system_stats available.
• Python 3.10+.
• Source mode only: uv.
• Required ComfyUI models/custom nodes for the selected workflow.

Networking note:

• Default local examples use http://127.0.0.1:8188 for same-environment setups.
• If the agent runs inside WSL/container/sandbox while ComfyUI runs on the host OS, 127.0.0.1 may refer to the runtime itself. Try --server http://localhost:8188 or the host machine IP (and optionally persist it via save-server).

Initial setup from the skill root:

bashCopy code
uv sync
uv run --no-sync python -m comfyui --help

Tool-install mode:

bashCopy code
comfyui-agent-skill-mie --help
comfyui-agent-skill-mie check
comfyui-skill --help
comfyui-skill check

Quick Workflow Choice

Minimal decision tree:

• User gives text only → start from general T2I (z_image_turbo fallback), but prefer a stronger registered match when workflow guidance indicates one (e.g. posters / anime / artistic styles).
• Poster / text-in-image requests → prefer qwen_image_2512_4step
• Anime / manga / illustration style → prefer anima_turbo
• Artistic / painterly realism / concept art / product → prefer krea2_turbo
• User gives a reference image and wants a new similar image → vision → reference_to_image prompt → run z_image_turbo
• User gives an input image and wants edits → generate --workflow klein_edit --image input_image=... -p "..."
• User wants to mask / cut out an object by describing it → sam3_mat_image
• User wants face/expression transfer (face photo + driving video) → liveportrait
• User wants TTS / voice audio → generate --workflow qwen3_tts --speech-text "..." --instruct "..."
• User wants to clone a specific voice from a sample → qwen3_tts_clone
• User wants video → ltx_23_t2v_distill (text→video) or ltx_23_i2v_distilled (image→video)

For workflow-specific size rules, capability boundaries, and examples, read references/workflows.md.

Core Commands

Environment doctor (check server + preflight registered workflows):

Tool-install mode:

bashCopy code
comfyui-skill doctor

Source mode:

bashCopy code
uv run --no-sync python -m comfyui doctor

If it exits with code 0, the environment is ready for all checked workflows. Exit code 1 means missing nodes/models or server is unreachable (see JSON payload).

Health check:

bashCopy code
uv run --no-sync python -m comfyui check

Tool-install mode:

bashCopy code
comfyui-skill check

Generate an image:

bashCopy code
uv run --no-sync python -m comfyui generate -p "a cute cat sitting on a windowsill at golden hour"

Tool-install mode:

bashCopy code
comfyui-skill generate -p "a cute cat sitting on a windowsill at golden hour"

Generate with a specific workflow and server:

bashCopy code
uv run --no-sync python -m comfyui generate --workflow z_image_turbo --server http://192.168.1.100:8188 -p "a landscape"

Save a persistent server URL only when the user asks for it:

bashCopy code
uv run --no-sync python -m comfyui save-server http://192.168.1.100:8188

Preflight a workflow before a long run:

bashCopy code
uv run --no-sync python -m comfyui generate --workflow qwen_image_2512_4step --preflight

Show progress for long jobs:

bashCopy code
uv run --no-sync python -m comfyui generate --workflow ltx_23_t2v_distill -p "cinematic waves at sunset, slow pan" --progress

Convert UI/Save workflows to API format (opt-in; needs a running ComfyUI + Playwright, does not run inference):

bashCopy code
uv run --no-sync python -m comfyui convert-ui path/to/ui_workflow.json --server http://127.0.0.1:8188
uv run --no-sync python -m comfyui convert-ui path/to/workflows_dir -o api_workflows/

Full CLI options, output path behavior, async submit/poll, and error code details are in references/cli.md.

Prompt Enhancement

Before generation, convert the user's intent into the right workflow inputs.

Reference-to-image flow:

1. Ensure a usable reference image exists; otherwise return Agent error NO_REFERENCE_IMAGE.
2. Ensure this runtime can inspect images; otherwise return Agent error VISION_UNAVAILABLE.
3. Read the reference prompt enhancement file and create one English prompt.
4. Call T2I generation with that prompt. Do not pass the reference image to ComfyUI.

Image-to-image flow:

1. Ensure a local image path is available.
2. Read the image edit prompt enhancement file.
3. Call klein_edit with --image input_image=path.

Text-to-speech flow:

1. Split user intent into spoken content and voice/style instruction.
2. Read the TTS prompt enhancement file.
3. Call qwen3_tts with --speech-text and --instruct; do not use positional prompt.

Fail-Fast and Recovery

CLI failures are structured JSON on stdout. Agent-only pre-check failures for reference images should also be JSON.

Agent-only error shape:

jsonCopy code
{
  "source": "agent",
  "success": false,
  "error": {"code": "VISION_UNAVAILABLE", "message": "Cannot read reference image in this runtime."}
}

Required fail-fast behavior:

• Missing prompt: return/handle EMPTY_PROMPT.
• Unregistered workflow: return/handle WORKFLOW_NOT_REGISTERED.
• Server unavailable: return/handle SERVER_UNAVAILABLE and ask whether ComfyUI is running locally or on another machine.
• Missing reference image before reference_to_image: return Agent error NO_REFERENCE_IMAGE; do not call CLI.
• No vision for reference_to_image: return Agent error VISION_UNAVAILABLE; do not call CLI.
• Missing required input media: return/handle NO_INPUT_IMAGE when an image role is absent and NO_INPUT_MEDIA when a video or audio role is absent. If the local path does not exist, handle INPUT_IMAGE_NOT_FOUND or INPUT_MEDIA_NOT_FOUND respectively. Do not report a missing video or audio input as NO_INPUT_IMAGE.
• A required mapped role that is present but empty: return/handle MISSING_INPUT.
• Missing custom nodes/models during preflight: return/handle PREFLIGHT_MISSING_NODES or PREFLIGHT_MISSING_MODELS.

When the user provides a remote ComfyUI address, save it only if they want persistence:

bashCopy code
uv run --no-sync python -m comfyui save-server http://<address>:<port>

Otherwise retry the original command with --server http://<address>:<port>.

Output Handling

After successful generation, present the result to the user. Do not silently parse JSON and stop.

• For images, display the file when the runtime supports local image display; otherwise provide the absolute/local path from outputs[].path.
• For MP3/MP4, provide the path or use the runtime's media display/playback capability when available.
• Prefer omitting --output; the CLI writes to a per-job directory under results/ inside the per-user data root and returns exact paths in JSON. The per-user data root is %APPDATA%\comfyui-skill on Windows, ~/Library/Application Support/comfyui-skill on macOS, and $XDG_DATA_HOME/comfyui-skill (or ~/.local/share/comfyui-skill) on Linux. save-server writes config.local.json under that same root. Always use the returned outputs[].path.
• For --count > 1, parse the wrapper object and present each result.

See references/cli.md for JSON schemas and output directory rules.

References

• references/workflows.md — workflow selection, capabilities, size rules, examples.
• references/cli.md — CLI contract, async jobs, output paths, JSON schemas, error codes.
• references/prompt_enhancement/ — prompt enhancement instructions.