What does the Create Task AI skill do?

Create a new task in the Harbor task format for evaluating agents on Evolve. Use when the user wants to scaffold, build, or design a new task, benchmark problem, or eval. Guides through instruction writing, environment setup, verifier design (pytest vs Reward Kit vs custom), solution scripting, checking the task with evolve check, and publishing it.

Why use Create Task on TypingMind?

Because you install it once and use it with any model. Create Task is plain Markdown rather than provider-specific code, so the same skill runs on GPT-5, Claude, Gemini, Grok, or a local model — and you can switch model mid-chat without it breaking. TypingMind runs on your own API keys, so you pay providers directly instead of a per-seat subscription, and your skills and chats stay in your own storage.

How do I install Create Task in TypingMind?

Open Plugins → Skills → Install from GitHub in TypingMind and paste https://github.com/evolving-machines-lab/evolve/tree/main/skills/create-task. TypingMind reads its SKILL.md and installs it as a skill you can enable per chat.

Which AI models can use Create Task?

Any model you connect in TypingMind. AI skills are plain Markdown instructions rather than provider-specific code, so GPT, Claude, Gemini, Grok, and local models can all load this skill when a request matches it.

How many AI models can I use with Create Task?

As many as you like. As long as a model supports skills, you can use Create Task with it — GPT, Claude, Gemini, Grok, DeepSeek, Mistral, Llama and more — all on TypingMind with your own API keys.

Is the Create Task AI skill free?

Yes. It is published on GitHub by evolving-machines-lab under the Apache-2.0 license. You only pay your own AI provider for the tokens you use.

An AI skill is a reusable instruction bundle that teaches an AI model how to do one specific task. It follows the open Agent Skills format: a SKILL.md file with a name and description, plus any scripts, templates or reference files the model may need. The model reads the instructions only when your request matches the skill, so an installed skill costs nothing until it is used.

How are AI skills different from plugins or MCP servers?

A plugin or MCP server gives a model new tools to call — code that runs somewhere and returns a result. An AI skill gives the model knowledge and process instead: how to approach a task, which steps to follow, what good output looks like. Skills are plain Markdown, so they need no server, no API key and no runtime, and they work with any model.

Create Task by evolving-machines-lab — AI Skill for Any LLM

Guide the user through creating a new task end-to-end. Don't just dump commands — walk them through each decision, especially around the verifier (which is usually the hardest part).

The task format is Harbor's, and Evolve runs it unchanged. The full specification is at https://docs.harborframework.com/core-concepts/tasks/overview.

Step 1: Create the task directory

The evolve CLI has no scaffold command today. Create the layout by hand:

bash
mkdir -p "<task-name>/environment" "<task-name>/solution" "<task-name>/tests"

The files to write, one per step below:


<task-name>/
├── instruction.md         # Task prompt for the agent
├── task.toml              # Config and metadata
├── environment/Dockerfile # Container definition
├── solution/solve.sh      # Reference solution (optional)
└── tests/test.sh          # Verifier script

The directory name is the task's name on the platform: letters, digits, ., _ and -, at most 128 characters, starting with a letter or digit; use lowercase (Harbor's convention).

Where Harbor is installed, harbor task init "<org>/<task-name>" produces the same layout (optional).

If the user wants a multi-step task (ordered steps with per-step instructions, tests, and early stopping against a shared container), write the single-step layout first, then convert to the steps/ layout described in the Multi-step tasks section below.

Step 2: Write instruction.md

This is the prompt the agent receives. Help the user write it clearly:

State the goal concretely — what file to create, what behavior to produce
Specify expected outputs — paths, formats, content
Include constraints — language, tools, approach
Don't leak the tests — describe what "done" looks like, not how you'll check it

Example (from the ssh-key-pair tutorial):

markdown
# SSH Key Pair Generation

Generate an SSH key pair in the files `~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub`.

Don't make them password protected.

Step 3: Build the environment

Edit environment/Dockerfile to install dependencies the task needs. The agent works inside this container.

dockerfile
FROM ubuntu:24.04
WORKDIR /app

# Install what the task requires — NOT the solution
RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/*

For multi-container setups, use environment/docker-compose.yaml instead. After publishing, evolve dataset show "<dataset>@<version>" prints which sandbox providers can run each task.

Test the environment interactively before writing the solution or tests:

bash
docker build -t "<task-name>" "<task-path>/environment"
docker run --rm -it "<task-name>" bash

This is usually where task authors realize something is missing from the Dockerfile.

Step 4: Decide how to verify

This is the most important decision. Ask the user: "How do you want to grade this task?" Then help them pick:

Also ask: "Should the verifier run in the same environment as the agent, or in a separate verifier environment?"

Use the default shared environment when tests need to inspect the agent's full workspace, installed tools, or services.
Use a separate verifier environment when grading code, dependencies, API keys, or OS requirements should stay hidden from the agent, or when verification should run from a clean image.

For a separate verifier container with no pinned [verifier.environment] docker_image, tests/ is the verifier image's build context and its tests/Dockerfile must provide /tests/test.sh. A verifier that pins the task's own image gets tests/ uploaded to /tests; one that pins a distinct image boots as it is with nothing uploaded, so that image must carry /tests/test.sh itself. A separate verifier judges only what the task lists under a top-level artifacts = ["/app/out.json"] in task.toml, never the agent's whole workspace.

toml
[verifier]
environment_mode = "separate"

[verifier.environment]
docker_image = "ubuntu:24.04"

Option A: Reward Kit (recommended for most cases)

Use when the verifier has multiple criteria, needs partial credit, uses an LLM/agent judge, or would benefit from composable reusable checks. See evolve skills get rewardkit.

Good fit signals:

Multiple things to check (file exists + content correct + command works)
Subjective quality dimensions (readability, correctness of prose)
Want partial credit rather than pass/fail
Want to compose built-ins like file_contains, command_succeeds, json_key_equals

tests/test.sh:

bash
#!/bin/bash
uvx --from 'harbor-rewardkit==0.2.*' rewardkit /tests

Note: the package is named harbor-rewardkit but the executable is rewardkit, hence --from 'harbor-rewardkit==0.2.*' rewardkit. Running uvx harbor-rewardkit directly will fail.

Then add tests/checks.py and/or tests/judge.toml. Read evolve skills get rewardkit to design the criteria.

Option B: pytest (good for deterministic unit-style checks)

Use when the verification is straightforward assertion-style Python.

tests/test.sh:

bash
#!/bin/bash
apt-get update && apt-get install -y curl
curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh
source $HOME/.local/bin/env

uvx --with pytest==8.4.1 pytest /tests/test_outputs.py

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

Example tests/test_outputs.py:

python
from pathlib import Path

def test_file_exists():
    assert (Path.home() / ".ssh" / "id_rsa").exists()

Option C: Custom shell

For simple single-command checks (e.g. a binary pass/fail from one command):

bash
#!/bin/bash
if diff -q /app/output.txt /tests/expected.txt; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

Reward file format (all options)

/logs/verifier/reward.txt — single number (usually 0 or 1)
/logs/verifier/reward.json — {"accuracy": 0.95, "runtime_sec": 1.2} for multiple metrics

When both exist, reward.json wins. Everything the script prints is kept as the verifier log.

Always use absolute paths in test.sh.

Step 5: Write the solution

Write solution/solve.sh — a script that actually solves the task. evolve check runs it to confirm the task is solvable and the tests pass on a correct solution. The agent is never given it.

bash
#!/bin/bash
ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ""

Make it executable: chmod +x solution/solve.sh.

Step 6: Configure task.toml

Walk through the important fields:

toml
[task]
name = "<org>/<task-name>"
version = "1.0.0"
description = "One-line description"
keywords = ["jax", "mnist", "rewardkit"]  # 3–8 lowercase tokens: domain, verifier style, hardware

[metadata]
difficulty = "easy" | "medium" | "hard"
category = "programming" | "machine-learning" | "gpu" | ...
tags = ["..."]

[agent]
timeout_sec = 120.0       # How long the agent has

[verifier]
timeout_sec = 600.0       # How long tests have

[environment]
network_mode = "public"   # Baseline at env start (defaults to public)
cpus = 1                  # CPU cores
memory_mb = 2048          # RAM in MB
storage_mb = 10240        # Disk in MB

keywords: 3–8 lowercase tokens covering the domain (language/framework/benchmark family), the verifier style (rewardkit, judge-grading, pytest), and any notable hardware (gpu).

Network policy

Network access has two layers:

Baselines — set when an environment starts, restored between phases
Phase overrides — optional; only during agent.run() or verify()

Field	Layer	When applied
`[environment].network_mode`	Baseline	Agent env start; shared verifier uses this too
`[verifier.environment].network_mode`	Baseline	Separate verifier env start
`[agent].network_mode`, `[steps.agent].network_mode`	Override	During matching `agent.run()`
`[verifier].network_mode`, `[steps.verifier].network_mode`	Override	During matching `verify()`

Modes: public, no-network, or allowlist with allowed_hosts = ["pypi.org"] (exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostnames, when supported by the selected environment; not URLs, ports, or paths). Omitting [environment].network_mode defaults to public.

[agent] / [verifier] are optional phase overrides — only applied when set and different from the phase baseline. Matching the baseline is a no-op.

Shared verifier (default): verifier runs in the agent container; baseline is [environment]. Separate verifier: baseline is [verifier.environment] if set, else a copy of [environment].

toml
# Agent starts offline; agent phase opens network; verifier stays offline
[environment]
network_mode = "no-network"

[agent]
network_mode = "public"

[verifier]
network_mode = "no-network"

On Evolve a shared-mode verifier cannot switch egress: a [verifier] network_mode that differs from the [environment] baseline is refused at import. Prefer environment_mode = "separate" when agent and verifier need different baselines:

toml
[environment]
network_mode = "no-network"

[verifier]
environment_mode = "separate"

[verifier.environment]
network_mode = "public"   # Verifier baseline — not a phase override

Full reference: https://docs.harborframework.com/core-concepts/tasks/network-policies.

For Reward Kit judges needing API keys:

toml
[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

On Evolve you never put a real key in the task: write the template exactly as above, as the whole value, and the judge's credential is supplied at run time.

Step 7: Check the task

bash
evolve check "<task-path>" --watch

The check reads the task and, when it can, runs the environment, solution/solve.sh and the verifier, then rules on every criterion of a rubric (eleven by default); executed in the result says whether it ran the task. evolve check show <check-id> prints one entry per criterion, with an outcome, an explanation and evidence, and one label per task: has_a_problem, unclear or no_problem_found.

Where Harbor is installed, harbor run -p "<task-path>" -a oracle runs the solution and the verifier locally (optional). Its reward should be 1.0. If it's not, debug in this order:

Does solve.sh actually solve it? (run it by hand inside docker run --rm -it "<task-name>" bash)
Does the verifier correctly detect success? (check /logs/verifier/ output)
Are paths correct? (absolute vs relative)
Are dependencies installed in the Dockerfile?

Step 8: Test with a real agent (optional)

Publish a folder holding the task directory as a dataset, then run a job on it:

bash
evolve dataset check ./tasks
evolve dataset publish --dir ./tasks --name "<dataset>" --version 1.0 --watch
evolve run -d "<dataset>@1.0" -a codex -m gpt-5.5 --watch

If the task is too easy (every model 1.0) or impossible (every model 0.0), consider adjusting difficulty. evolve skills get publish covers every publish option.

Step 9: Write README.md (always the final step)

Add a README.md so future humans (and agents) can understand the task without reading every file. Include:

What the agent does — one paragraph, link to instruction.md.
Environment — base image, key installed packages, cached data, hardware (GPU/CPU/RAM), agent timeout.
Verifier — for Reward Kit tasks, a table of reward dimensions with type (programmatic / LLM judge / agent judge) and what each measures; how they're aggregated.
Layout — a tree of the task directory with one-line annotations.
Running — the concrete evolve check and evolve run commands, with the sandbox provider (-e) that evolve dataset show reports can run the task if it needs a GPU.

Treat this as docs, not marketing — the reader wants to know what they'd need to change to modify the task.

Multi-step tasks

Use when the work splits into ordered phases that should be scored separately, when you want early stopping between phases, or when you're testing an agent's ability to build on its own prior work. Steps share one container; files persist across steps.

Directory layout

Replace the task-root instruction.md, tests/, and solution/ with a steps/ directory containing one sub-directory per step:

Each [[steps]].name must match one directory name of at most 255 UTF-8 bytes, unique after case folding and Unicode normalization. Avoid path separators, control characters, Windows-reserved characters/device names, and trailing dots or spaces. Keep all task inputs and linked contents within the task directory; validation permits shared links inside the task. Use regular files and directories for shared inputs when publishing tasks.


<task-name>/
├── task.toml
├── environment/Dockerfile       # Built once, shared across all steps
├── steps/
│   ├── scaffold/
│   │   ├── instruction.md       # Prompt for this step
│   │   ├── workdir/             # Uploaded to WORKDIR before the agent runs
│   │   │   └── setup.sh         # Optional pre-agent hook (reserved filename)
│   │   ├── tests/test.sh        # Per-step verifier
│   │   └── solution/solve.sh    # Per-step reference solution (optional)
│   ├── implement/
│   │   └── ...
│   └── document/
│       └── ...
└── tests/                       # Optional shared helpers + fallback test.sh

Task-level tests/ is uploaded to /tests for each step's verification, then the step's own tests/ is layered on top (same-name files win). Use this for shared helpers.

steps/{name}/workdir/setup.sh is a reserved filename: if present, it runs after the workdir/ upload and before the agent, as the step's agent user, with cwd = WORKDIR. Non-zero exit aborts the step and the trial. Have it rm -- "$0" on its last line if the agent shouldn't see it.

task.toml

toml
schema_version = "1.4"

[task]
name = "<org>/<task-name>"
version = "1.0.0"

# How per-step rewards roll up into the trial-level verifier_result.
# "mean" (default): per-key mean across steps that produced a result.
# "final": the last step's verifier_result verbatim.
multi_step_reward_strategy = "mean"

[[steps]]
name = "scaffold"              # Must match the directory under steps/
min_reward = 1.0               # Abort trial if this step's reward < 1.0
[steps.agent]
timeout_sec = 60.0             # Overrides task-level [agent].timeout_sec
[steps.verifier]
timeout_sec = 30.0

[[steps]]
name = "implement"
# Dict form gates on specific keys from a multi-dim reward:
min_reward = { correctness = 0.8, style = 0.5 }
[steps.agent]
timeout_sec = 120.0
[steps.verifier]
timeout_sec = 30.0

[[steps]]
name = "document"
[steps.agent]
timeout_sec = 60.0
[steps.verifier]
timeout_sec = 30.0

Per-step overrides available: agent.timeout_sec, agent.user, agent.network_mode, verifier.timeout_sec, verifier.env, verifier.user, verifier.network_mode, verifier.environment_mode, verifier.environment, steps.verifier.environment.network_mode, healthcheck.*, artifacts. Unset fields fall back to the task-level values.

Choosing a reward strategy

"mean" — aggregate signal across all steps; good for continuous progress rewards.
"final" — last step's verifier_result is the trial reward. Right when the final step is an end-to-end check whose dict already represents the full task. Caveat: if min_reward triggers an early abort, "final" uses the aborted step's result, not the intended final step.

Artifacts

Step-level artifacts are collected into steps/{name}/artifacts/ after that step's verification. Task-level and trial-level artifacts are collected at every step in addition to the step-level ones.

Checking a multi-step task

Where Harbor is installed, harbor run -p "<task-path>" -a oracle runs each step's solution/solve.sh, then each step's verifier, in order (optional). Trial reward should be 1.0 across the aggregation strategy. Then publish and run it (Step 8).

Full reference + worked example

Docs: https://docs.harborframework.com/core-concepts/tasks/multi-step
Example task: https://github.com/laude-institute/harbor/tree/main/examples/tasks/hello-multi-step-advanced

Special features (mention if relevant)

Network policy: Baselines on [environment] / [verifier.environment]; phase overrides on [agent] / [verifier]; see Network policy under Step 6
MCP servers: Add [[environment.mcp_servers]] in task.toml for agent tooling
Healthcheck: Add [environment.healthcheck] for services that need to be ready
GPU: Set environment.gpus and optionally environment.gpu_types
Pre-built image: Set environment.docker_image instead of building from Dockerfile. You can omit environment/Dockerfile and place runtime files (configs, scripts, data) directly under environment/; the platform uploads them into the container workdir when the environment starts.
Non-root user: Set agent.user / verifier.user for isolation

Common pitfalls

Forgetting to write the reward file → task "passes" silently with reward 0
Using relative paths in test.sh → breaks when the verifier runs it from a different cwd
Installing the solution into the Dockerfile → agent already gets the answer
Test script leaks into instruction.md → agent sees the rubric and gaming becomes trivial
Forgetting chmod +x solution/solve.sh → the reference solution cannot run
Leaving README.md out → teammates have no way to understand the task at a glance
Putting network_mode on [agent] expecting it to apply at env start → use [environment].network_mode for the baseline; agent/verifier fields are phase overrides
A shared verifier's [verifier] network_mode differs from the [environment] baseline → refused at import; use a separate verifier env or match the baseline instead

Publisher	evolving-machines-lab
Repository	`evolve`
Skill name	`create-task`
Stars	76
Forks	5
Bundled files	Instructions only
License	Apache-2.0
Links	Skill source Repository

Create Task

Overview

Installation

Install in TypingMind

Install in another agent

Use it in TypingMind

Works with any AI model

Skill instructions

Step 1: Create the task directory

Step 2: Write instruction.md

Step 3: Build the environment

Step 4: Decide how to verify

Option A: Reward Kit (recommended for most cases)

Option B: pytest (good for deterministic unit-style checks)

Option C: Custom shell

Reward file format (all options)

Step 5: Write the solution

Step 6: Configure task.toml

Network policy

Step 7: Check the task

Step 8: Test with a real agent (optional)

Step 9: Write README.md (always the final step)

Multi-step tasks

Directory layout

task.toml

Choosing a reward strategy

Artifacts

Checking a multi-step task

Full reference + worked example

Special features (mention if relevant)

Common pitfalls

Frequently asked questions

Set up your own AI workspace now

Overview

Installation

Install in TypingMind

Install in another agent

Use it in TypingMind

Works with any AI model

Skill instructions

Step 1: Create the task directory

Step 2: Write instruction.md

Step 3: Build the environment

Step 4: Decide how to verify

Option A: Reward Kit (recommended for most cases)

Option B: pytest (good for deterministic unit-style checks)

Option C: Custom shell

Reward file format (all options)

Step 5: Write the solution

Step 6: Configure task.toml

Network policy

Step 7: Check the task

Step 8: Test with a real agent (optional)

Step 9: Write README.md (always the final step)

Multi-step tasks

Directory layout

task.toml

Choosing a reward strategy

Artifacts

Checking a multi-step task

Full reference + worked example

Special features (mention if relevant)

Common pitfalls

Frequently asked questions

Related AI Skills

Set up your own AI workspace now