Skintokens Auto Rigging

Auto-Rigging Static Meshes with skin-tokens.cpp (SkinTokens/TokenRig) in Summer Engine

About the tool

skin-tokens.cpp is published by the localai-org GitHub organization. It is an Apache-2.0 C++23/GGML port of VAST-AI's MIT-licensed SkinTokens/TokenRig research model (arXiv:2602.04805). It ships a flat C11 API, an ASan/UBSan libFuzzer build, CPU/Vulkan numerical-parity tests against the upstream reference, and a Go/WebGL demo. The weights are a non-retrained GGUF conversion (~1.25 GB F16) on Hugging Face.

Limitations to plan around:

• The fully automatic rig command (unconstrained skeleton generation) is explicitly experimental — the policy can produce unsuitable bone topologies.
• The author's own demo shows open issues combining the output with kimodo.cpp skeletal animations.
• Generation recomputes the Qwen3 graph without a KV cache and may take several minutes per mesh; dense decoder queries are capped at 16K vertices per graph (chunked beyond that).
• Results are strongest on shapes resembling the training distribution (humanoids; trained on ArticulationXL 2.0, VRoid Hub, ModelsResource).

Recommendation: worth adopting as an offline asset-prep tool in the art pipeline, primarily the skin command (weights for an existing skeleton). Treat rig-generated skeletons as drafts that need artist review before animation use.

Outcome

Static .glb meshes (e.g. AI-generated or scanned characters/creatures) become rigged, skinned .glb assets that Summer Engine imports natively as Skeleton3D + skinned MeshInstance3D, ready for AnimationPlayer/retargeted animation — without manual weight painting.

When to Use

• You receive an unrigged character/creature mesh and need a fast first-pass rig or skin weights.
• You have an existing armature (e.g. a mocap/Kimodo-style skeleton GLB) and only need learned skin weights for a mesh (skin command — the reliable path).
• Batch-rigging many props/characters where hand-painting weights is not cost-effective.

When NOT to Use

• Hero characters needing production-quality, art-directed deformation — plan for manual weight cleanup after auto-rigging.
• Non-triangle meshes (tool accepts arbitrary triangle meshes only; triangulate first).
• Shapes far from the training distribution (expect weaker results; run --geometric diagnostic to compare).
• Runtime/in-engine rigging — this is an offline CLI/library step, not an editor plugin.

Core Principle

TokenRig models the whole rig — skeleton hierarchy followed by discrete skin-weight tokens (SkinTokens, an FSQ-CVAE vocabulary) — as one autoregressive sequence generated from a Michelangelo point-cloud encoding of the mesh (Qwen3-0.6B backbone, GRPO-refined). skin-tokens.cpp re-runs this stack through GGML on CPU or Vulkan and exports a standard skinned glTF/GLB with a one-frame rest pose, so it opens as a conventional rigged asset in any engine.

Pipeline shape (Summer Engine)

Copy code
static character.glb
      │
      ▼
[skin-tokens-cli]  rig | skin   (offline, CPU or Vulkan, minutes per mesh)
      │
      ▼
character-rigged.glb   (skeleton + skin weights + rest pose, skinned glTF)
      │
      ▼
Summer Engine import  →  character-rigged.glb (scene)
      └── Skeleton3D
            └── MeshInstance3D (skinned, Skin resource)
      └── AnimationPlayer / retargeted animations

Keep raw static meshes and rigged outputs as separate files under the asset folder convention, e.g. assets/chars/<name>/source.glb and assets/chars/<name>/rigged.glb, so re-rigging never destroys source art.

Setup (once)

1. Build the CLI (Linux; C++23 toolchain, CMake ≥ 3.25, Ninja, nlohmann-json, Vulkan unless disabled):

   bashCopy code
   git clone https://github.com/localai-org/skin-tokens.cpp
   cd skin-tokens.cpp
   git submodule update --init --recursive
   cmake -S . -B build/release -G Ninja -DCMAKE_BUILD_TYPE=Release   # add -DSKINTOKENS_ENABLE_VULKAN=OFF if no Vulkan
   cmake --build build/release -j
   cmake --install build/release --prefix ./dist
   ctest --test-dir build/release --output-on-failure
   ./build/release/bin/skintokens-cli inspect models/SkinTokens-GGUF/F16   # sanity check after weights download
   

2. Download the F16 GGUF bundle (keep the 3 files together; runtime checks their embedded identities):

   bashCopy code
   hf download LocalAI-io/SkinTokens-GGUF --include "F16/*" --local-dir models/SkinTokens-GGUF
   

   (~1.25 GB: mesh-encoder 57.8 MB + tokenrig 948.6 MB + skin-vae 244 MB. Use F32 only for numerical-parity work.)

Implementation Steps

1. Auto-rig a static mesh (draft skeleton + weights)

bashCopy code
./build/release/bin/skintokens-cli rig models/SkinTokens-GGUF/F16 \
  character.glb character-rigged.glb --device vulkan --postprocess

• --postprocess applies upstream's surface-locality heuristic (smoother, local influences); omit it only to preserve raw learned weights.
• Output stores a one-frame rest pose; inspect the skeleton topology before animating (experimental mode — see Failure Modes).
• Use --geometric as a non-learned baseline diagnostic when results look wrong.

2. Re-weight against an existing skeleton (preferred when an armature exists)

bashCopy code
# One rigged GLB supplies both geometry and skeleton (old weights ignored):
./build/release/bin/skintokens-cli skin models/SkinTokens-GGUF/F16 \
  character-rigged.glb character-rigged.glb character-reweighted.glb \
  --device vulkan --fit none

# Or separate mesh + animated skeleton GLBs (e.g. Kimodo animation output):
./build/release/bin/skintokens-cli skin models/SkinTokens-GGUF/F16 \
  character.glb motion.glb animated.glb --device vulkan --fit global

Fit modes:

3. Import into Summer Engine

Drop the rigged .glb into the project; the glTF importer creates the scene automatically:

• Skeleton3D with bones and a skinned MeshInstance3D (Skin resource), rest pose included.
• Check in the Import dock: mesh, skeleton, and (if supplied) animations present.
• If the source GLB contains animations and the hierarchy matches, the importer puts them into an AnimationPlayer; otherwise use retargeting (SkeletonProfileHumanoid, see the retarget skill) to map animations onto the generated skeleton.
• glTF round-trip note from upstream: on Blender export of results, a glTF_not_exported node may need removing (only relevant if the pipeline goes through Blender).

4. Optional editor tooling (typed GDScript)

An EditorScript wrapper to rig assets from inside the editor:

gdscriptCopy code
@tool
extends EditorScript

const CLI := "/opt/skin-tokens.cpp/build/release/bin/skintokens-cli"
const MODEL_DIR := "/opt/skin-tokens.cpp/models/SkinTokens-GGUF/F16"

func _run() -> void:
	var src := "res://assets/chars/goblin/source.glb"
	var dst := "res://assets/chars/goblin/rigged.glb"
	var args: PackedStringArray = [
		"rig", MODEL_DIR,
		ProjectSettings.globalize_path(src),
		ProjectSettings.globalize_path(dst),
		"--device", "vulkan", "--postprocess",
	]
	var output: Array = []
	var err := OS.execute(CLI, args, output, true, false)
	if err != 0:
		push_error("skin-tokens-cli failed (%d): %s" % [err, "\n".join(output)])
		return
	print("Rigged: ", dst)
	# Reimport so the new GLB appears in the FileSystem dock.
	EditorInterface.get_resource_filesystem().scan()

For embedding in a tool/asset server, prefer the flat C11 API (skintokens.h): st_model_load → st_rig_file / st_skin_files / st_inspect_glb_file, with caller-owned error buffers; C++ exceptions never cross the ABI.

Tunables

Failure Modes & Gotchas

• Unsuitable skeleton topology from rig (experimental policy) — always have an artist inspect bones before animating; re-run or fall back to a template skeleton + skin.
• Animation combination issues: the author reports unresolved problems combining skin-tokens.cpp rigs with kimodo.cpp skeletal animations (visible at the end of the demo video). Verify any animation transfer manually.
• Warped first frame when the supplied skeleton's pose doesn't match the mesh: use --fit global (or articulated for humanoid arms) so bind pose aligns with the mesh; --fit none only when coordinates already match.
• Out-of-distribution shapes degrade quality silently — no error is raised. Spot-check deformation visually.
• Runtime: several minutes per mesh (no KV cache); do not put this in a per-frame or hot-reload path — batch offline.
• Bundle integrity: the three GGUF files of a bundle must stay together; load fails identity checks otherwise.
• Vulkan precision: slightly looser numerics than CPU (7.1e-4 hidden state / 1.4e-3 logits relative L2 vs 8.1e-6 CPU F32); visually equivalent but use F32+CPU for parity debugging.
• Upstream GPU path (Python VAST-AI repo) needs an NVIDIA GPU ≥ 14 GB — use the C++ port instead for CPU/Vulkan machines.

Verification

Not yet verified in Summer Engine. To validate:

1. skintokens-cli inspect models/SkinTokens-GGUF/F16 succeeds after build + weights download.
2. Rig a known test mesh (rig … --postprocess) and open the output in the editor: Skeleton3D with sensible bone hierarchy + skinned MeshInstance3D, no import errors.
3. Pose one bone in the editor; confirm the surface deforms smoothly without bleeding across disconnected parts.
4. skin path: feed a mesh + animated skeleton GLB with --fit global; play the animation in the editor and confirm frame zero does not warp the mesh.
5. Re-run ctest / parity binaries if the CLI is rebuilt or GGML is updated.

Confidence

extracted — All tool behavior, commands, flags, sizes, numerics, and caveats come from the retrieved GitHub README, Hugging Face model card, upstream VAST-AI repo, and the author's X announcement (including the self-reported kimodo.cpp animation issue). The import mapping and GDScript wrapper are adaptations, not from the sources.