Challenge Run Agent

challenge-run-agent — Launch the inference Agent

The Agent is the contestant's long-running process that:

1. Reverse-connects to the gateway over WebSocket using JOB_UUID and a per-process agent_id.
2. Receives observation frames, runs inference, returns action bytes.
3. Stays online until the platform sends a drain control frame (job complete).

Which client to run

There is no official published Simulation SDK package. Do not tell the user to pip install one, and do not point them at any client inside the platform's own repository — the platform repo ships only internal test clients, which are not supported for contest use and can change without notice.

Exactly two supported paths:

The rest of this skill assumes tunnel.sh. If the contestant is on their own client, the concurrency rules, launch plan, and lifecycle sections below still apply verbatim — only the command changes.

Preconditions

If any are missing, jump back to the producing skill instead of guessing.

JOB_TOKEN does not expire, so long-running agent processes won't hit a mid-run 401 from token expiry the way a CHALLENGE_TOKEN-based dial could.

Every Bash call in this skill should start with:

bashCopy code
[ -f ~/.simubotix-challenge.env ] && . ~/.simubotix-challenge.env

(Cross-shell state lives in ~/.simubotix-challenge.env; AI assistants spawn each command in a new subshell. See README "State file".)

Concurrency rules (read these before launching)

• The platform enforces at most $PARALLELISM concurrent agent processes per user. Excess connections are closed immediately after the WS upgrade.
• Each tunnel carries at most one active session at a time (1:1 contract). To run cases in parallel, run multiple agent processes — that's the whole point of parallelism.
• Each process must use a different GPU index if launched on the same host. If the user's box has only N GPUs, min(N, PARALLELISM) is the real cap.
• Each process must use a different agent_id. tunnel.sh auto-generates one — do not pass duplicates.

Step 1 — Confirm the launch plan with the user

Before running anything, summarise:

About to launch K agent processes against JOB_UUID=…, gateway TUNNEL_ENDPOINT=…. K must be ≤ PARALLELISM ($PARALLELISM) and ≤ available GPUs. Each process pins one GPU. Proceed?

Wait for explicit confirmation. Default K to $PARALLELISM only if the user said so — otherwise ask.

Step 2 — Launch a single agent (sanity check)

bashCopy code
cd <inference-repo>
CHALLENGE_TOKEN="${JOB_TOKEN:-$CHALLENGE_TOKEN}" ./scripts/tunnel.sh 0 "$JOB_UUID" "$TUNNEL_ENDPOINT"

The reference tunnel.sh takes no token argument — it reads the credential from $CHALLENGE_TOKEN in its environment. The inline CHALLENGE_TOKEN=... prefix therefore makes it dial with the non-expiring job_token for this tunnel process only; it does not overwrite the shell's own CHALLENGE_TOKEN, so other HTTP API calls (submit/poll/result) in the same session keep using the login token. Falls back to the login token if JOB_TOKEN is unset (older platforms). If the user's fork reads a differently-named variable, check the top of their tunnel.sh and adjust the prefix.

Argument order:

The baseline repo also needs a board selector env var so the loaded checkpoint matches the job's config.board — see challenge-baseline-model Step 4 for the exact variable it expects.

Verify the agent reaches the WARMUP → RUNNING lifecycle before scaling out. If it disconnects right after handshake, see challenge-troubleshoot.

Step 3 — Scale to PARALLELISM processes

First decide the real launch count K = min(PARALLELISM, NUM_GPUS). Detect available GPUs (one of):

bashCopy code
NUM_GPUS=$(nvidia-smi --query-gpu=index --format=csv,noheader 2>/dev/null | wc -l | tr -d ' ')
# fallback: if nvidia-smi is unavailable, ask the user how many GPUs to use.
K=$(( NUM_GPUS < PARALLELISM ? NUM_GPUS : PARALLELISM ))
echo "launching K=$K processes (parallelism=$PARALLELISM, gpus=$NUM_GPUS)"

If K < PARALLELISM, tell the user explicitly — they can either accept reduced concurrency or add more GPUs. Do not double-pin a GPU.

Map GPU i to agent process i:

bashCopy code
PIDS=()
for i in $(seq 0 $((K - 1))); do
  CHALLENGE_TOKEN="${JOB_TOKEN:-$CHALLENGE_TOKEN}" ./scripts/tunnel.sh "$i" "$JOB_UUID" "$TUNNEL_ENDPOINT" &
  PIDS+=($!)
done
trap 'kill "${PIDS[@]}" 2>/dev/null || true' EXIT
wait

Same scoped CHALLENGE_TOKEN="${JOB_TOKEN:-$CHALLENGE_TOKEN}" prefix as Step 2, applied per-process — each backgrounded tunnel.sh gets the job token in its own environment without touching the parent shell's CHALLENGE_TOKEN.

trap ensures Ctrl-C (exit 130) cleans up children. The wait at the end blocks the foreground shell until every agent exits — see "Driving from an AI assistant" below if you're orchestrating this from a non-interactive session.

Driving from an AI assistant

The launch loop is long-lived: it stays alive until every process gets drain (exit 0) or the user hits Ctrl-C. If you're an AI assistant orchestrating this from a single shell:

• Tell the user to run the loop in a dedicated terminal (or under tmux / nohup), then come back to your session for challenge-poll-result.
• Do NOT background the whole loop with & and continue polling from the same shell — when the assistant's session ends, the agents go with it.

If you have to launch and poll from the same automation, run the loop with nohup redirected to a log file and treat its PID as opaque until poll reaches a terminal status.

Lifecycle and exit codes

If a process dies abnormally, tunnel.sh will redial automatically with the same agent_id within the gateway-side 30-second reconnect window to resume the open session. Do not manually relaunch within that window — you'd race the automatic redial and the gateway will reject the second connection. Past 30 s, the open session is gone and the user must resubmit a new job (which consumes a daily submission slot — confirm before suggesting it, and re-probe /api/challenge/submission/quota first).

After launch

Cases queue and run as agents stay online. Hand off to challenge-poll-result to track status.

• Don't kill agents until the job's detailed_status reaches completed / failed — early termination loses in-flight cases.
• Once dispatch is done the platform sends drain and the agent exits 0 on its own. Wait for that natural exit; do not Ctrl-C preemptively.
• detailed_status: queued means an agent is attached but no case is running yet (usually no free GPU cluster-wide) — not that something is wrong. Keep the agents online; killing them discards the warmed-up slot.

Failure shortlist

For more, jump to challenge-troubleshoot.

Appendix — Tunnel protocol (for a self-built client)

Everything needed to write your own agent. The wire format is two WebSocket message types and six JSON control frames. No Protobuf, no gRPC. For what goes inside the binary payload (observation / action schema), see challenge-inference-protocol.

A.1 Dial

Copy code
GET <TUNNEL_ENDPOINT>?job=<JOB_UUID>&agent=<agent_id>
Authorization: Bearer <JOB_TOKEN or CHALLENGE_TOKEN>
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13

• agent_id — a UUIDv4 you generate (≤ 64 bytes UTF-8). Every process must use its own.
• Prefer JOB_TOKEN: scoped to that one job, no expiry, unaffected by password changes. CHALLENGE_TOKEN is accepted for back-compat but expires in 12h.
• 101 = handshake OK. 401 = bad/missing token, job not found or not owned by you, job in a terminal state, or the ?job= doesn't match the token's job. 500 = backend lookup failed, retry with backoff.
• Capacity exhaustion still returns 101 (HTTP can't 429 after Upgrade) and the server immediately sends a close frame. Treat an instant close after upgrade as "over the parallelism cap".
• Dialing with a (job_uuid, agent_id) pair that already has a live connection closes the new socket; the existing one survives.

After upgrade the connection sits in QUEUED, waiting for the server's first control frame.

A.2 Control frames — TextMessage, JSON

jsonCopy code
{ "type": "<control-type>", "session_id": "<optional>", "reason": "<optional>" }

• session_id is required on session_open / session_close; ignored on the rest.
• Unknown type values are logged and ignored by the gateway, not fatal — do the same so future server-side frames don't break your client.
• Wrong-direction frames are tolerated but have no effect: an agent sending warmup or session_open is silently ignored. Session lifecycle is server-driven.
• Do not invent a {"type":"ping"} — there is no application-level heartbeat (see A.6).

queued ({"type":"queued","reason":"no_gpu_available"}) is informational and one-shot: at most one per agent per job, only after ready and before the first session_open. Keep the socket open and the model resident; there is no "queue cancelled" frame and no server-side queue timeout. You leave the state on session_open or drain.

A.3 Data frames — BinaryMessage

Copy code
┌──────────────────┬───────────────────┬─────────────────────┐
│ sidLen (4 bytes) │ session_id (UTF-8)│ payload (arbitrary) │
│ big-endian uint32│ sidLen bytes      │ rest of the frame   │
└──────────────────┴───────────────────┴─────────────────────┘

• session_id must be ≤ 256 bytes. Reply with the same session_id.
• payload may be zero-length — the warmup round-trip uses an empty payload, and your handler must tolerate it (short-circuit and return a no-op action). This is the single most common contest bug.
• The gateway is payload-agnostic: bytes in, same bytes out on the simulator side.
• No framing checksum; WebSocket's per-message CRC is the integrity guarantee.
• A frame shorter than 4 bytes, or whose sidLen overruns the frame, drops the connection. A frame for an unregistered session_id is silently dropped.

A.4 State machine

Copy code
QUEUED ──warmup──▶ WARMUP ──ready──▶ RUNNING ──drain──▶ DRAINING

All other transitions are rejected. Sending ready before receiving warmup is ignored — don't try to short-circuit warmup. Data frames are accepted in any state as long as the session_id is known.

A.5 Sessions and reconnect

One active session per tunnel, sequentially reused across cases: session_open → data frames → session_close, then the scheduler may open the next one on the same tunnel. Echoing session_close back is optional. To run cases concurrently, run more processes (up to parallelism), not more sessions.

On an unclean close you get a 30-second reconnect window: redial with the same (job_uuid, agent_id) and Bearer token and the open session is restored. There's no explicit "I'm reconnecting" handshake — the server matches on the pair. Frames the server tried to send while you were down are dropped, not queued. Past 30 s the tunnel is gone and further dials on that pair are rejected; the user must submit a new job. Never reconnect after drain.

A.6 Heartbeat

Liveness is handled at the WebSocket library layer — no application-level heartbeat. The gateway pings every 20 s and enforces a 45 s read deadline; 45 s of silence opens the reconnect window. Most client libraries auto-reply to pings. Configure your own client-side ping to survive NATs that drop idle connections, e.g. Python websockets: websockets.connect(url, ping_interval=20, ping_timeout=10).

A.7 Minimal Python skeleton

Handshake, warmup, one session round-trip. Omits reconnect (A.5), per-session concurrency, and bounded buffering — all three are required in a real client.

pythonCopy code
import asyncio, json, struct, uuid, websockets

async def run_agent(tunnel_url: str, token: str, job_uuid: str, agent_id: str | None = None):
    agent_id = agent_id or str(uuid.uuid4())
    ws_url = f"{tunnel_url}?job={job_uuid}&agent={agent_id}"
    headers = {"Authorization": f"Bearer {token}"}

    async with websockets.connect(ws_url, additional_headers=headers,
                                  ping_interval=20, ping_timeout=10) as ws:
        async for msg in ws:
            if isinstance(msg, str):                       # control frame (A.2)
                ctl = json.loads(msg)
                t = ctl.get("type")
                if t == "warmup":
                    await load_model()                     # your code
                    await ws.send(json.dumps({"type": "ready"}))
                elif t == "session_open":
                    open_context(ctl["session_id"])        # your code
                elif t == "session_close":
                    close_context(ctl["session_id"])       # your code
                elif t == "drain":
                    return                                 # do NOT reconnect
                # any other type: log and ignore
            else:                                          # data frame (A.3)
                sid_len, = struct.unpack(">I", msg[:4])
                sid = msg[4:4 + sid_len].decode("utf-8")
                payload = msg[4 + sid_len:]
                action = infer(sid, payload)               # must handle payload == b""
                await ws.send(struct.pack(">I", sid_len) + sid.encode("utf-8") + action)

Run parallelism of these as separate OS processes (each with its own agent_id and GPU), not as asyncio tasks in one process — one process per GPU is the supported topology.