When to Use
- Sanity-check whether a running inference server actually accepts requests and returns valid actions, before launching a full task.
- User provides
ip:portand a payload, and asks to verify connectivity / output validity. - Quick smoke test in CI / pre-deploy.
- Diagnosing "the benchmark hangs / outputs garbage" — this probe surfaces protocol mismatches and NaN/Inf in actions before you sink time into a full simulator launch.
Do not use for:
- Running the benchmark itself → use the
run-benchmarkskill. - Submitting jobs to the Challenge platform →
challenge-submit-job.
What This Skill Does
Sends a saved corobot .pkl payload to ws://<HOST>:<PORT> and validates the reply:
- Loads the payload.
- Connects to the WebSocket server using msgpack-numpy.
- Sends one request, receives one action chunk.
- Validates: schema (key presence), per-dim min/max/mean/std, NaN/Inf flags, out-of-range checks (kind-aware: JOINT_ABS in radians, EEF_ABS in meters+quat, gripper in [0,1]), large jumps from the input state.
The payload is a corobot JSON-RPC envelope — {"method": "infer", "params": {...}} — and the server replies with {"result": {"left_arm": …, "right_arm": …, …}} (or {"error": …}).
Required Input
The user must provide:
HOST— server IP (e.g.127.0.0.1).PORT— server port (e.g.8999).
PAYLOAD is optional — it defaults to the bundled corobot_payload.pkl. Pass a
path only to override it (see Generating a payload below). If host/port are
missing, ask the user before running.
How to Run
bash# Bundled payload — just point it at the server geniesim benchmark check-inference --infer-host=<HOST>:<PORT> # Override with your own payload (positional) geniesim benchmark check-inference debug_preview/debug_0001.pkl \ --host <HOST> --port <PORT>
If geniesim isn't on $PATH (the launcher wasn't installed), substitute
python3 -m geniesim_cli benchmark check-inference … — same args, same behaviour.
Optional flags (forwarded to check_inference.py)
| Flag | Effect |
|---|---|
--iters N | Send N consecutive requests (default 1). Use 5–10 to catch flakiness. |
--max-dims N | Max idx rows printed per array (default 64). |
Generating a payload
A canonical corobot_payload.pkl ships next to the script and is used by
default, so you usually don't need to supply one. To probe with a fresh /
task-specific observation, run a benchmark task with the corobot policy's debug
dump enabled — it writes debug_preview/debug_NNNN.pkl (a {"payload": …, "obs": …} wrapper the probe unwraps automatically) — then pass that path.
Interpreting Output
The script prints structured sections:
| Section | What it tells you |
|---|---|
📦 Payload | Payload loaded and recognised as corobot. |
🔌 Connecting | Connected to ws server (or refused). |
📡 …response | Server returned a reply, schema check. |
📊 <key> | Per-dim min/max/mean/std + flags, per output (left_arm/right_arm/…). |
⏱ latency | Round-trip latency. |
| Outcome | Meaning |
|---|---|
✅ PASS — … | Server is up and returning sane actions. |
❌ Connection refused / timed out | Server isn't listening on that host:port. Verify it's running and the firewall is open. |
❌ … NaN / Inf … | Server responded but model output is broken. Check the policy checkpoint and normalization stats — not a network problem. |
❌ response missing 'result' dict | Server schema mismatch — it isn't speaking the corobot JSON-RPC protocol the probe expects. |
❌ server error: … | The server returned a JSON-RPC error; read the message. |
⚠️ OOB[…] flags | Action is finite but outside the kind's expected range. Could be a units bug (radians vs degrees) or an unnormalized output. |
Dependencies
The script needs (in the Python env that runs python3):
msgpacknumpywebsockets
It does not need Isaac Sim — pure-Python deps only. The CLI deliberately uses python3 instead of omni_python here so the probe is snappy.
Resources
- Script source:
source/geniesim_benchmark/src/geniesim_benchmark/scripts/check_inference.py(resolved via thegeniesim_benchmarkpackage) - CLI dispatcher:
source/geniesim_cli/src/geniesim_cli/commands/benchmark.py(_do_check_inference) - Payload dump hook:
source/geniesim_benchmark/src/geniesim_benchmark/benchmark/policy/corobotpolicy.py
