Lumera Single Image Scene Reconstruction

Single-Image → Editable Scene Reconstruction (Lumera-style) in Summer Engine

Outcome

Turn one reference image into an editable, engine-native scene in Summer Engine: individual MeshInstance3D objects with transforms, countable parametric light nodes, and an HDR environment probe — not a fused point cloud, Gaussian splat, or baked texture. This adapts the Lumera pipeline (arXiv:2607.20889, Tsinghua AIRSUN lab), which was built on another engine, to Summer Engine.

Caveat: Lumera has no public code, and the paper lists engine-portable extraction as future work. This skill documents the architecture and the parts an agent can build in Summer Engine today (assembly, scene representation, refinement loop, evaluation), and flags the parts that require external model components (VLM box/light parsers, per-object mesh generator, HDR estimator).

When to Use

• Bootstrapping a level/blockout from concept art or a screenshot, where artists need movable objects and re-tunable lights afterward.
• Producing 3D scene conditions for 3D-conditioned video generation (a use case the paper calls out).
• You specifically need editable entities (instances, lights, probes) rather than a render-able blob.

When NOT to Use

• You only need a visual backdrop: a Gaussian-splat or depth-mesh bake is cheaper and looks as good or better.
• Large unbounded outdoor scenes: the paper reports ~17 m Chamfer-L2 drift outdoors — geometry is unreliable at that scale.
• Exact light counts/positions are critical: individual-light localization F1 is 0.209 at 0.5 m; intensity error is ~2.7× (log-MAE 0.431). Expect hand-tuning.

Core Principle

Treat image-to-3D as game-engine structured parsing: engines already store exactly the entities downstream tools need — object components, transforms, cameras, materials, environment probes, and countable parametric lights. The pipeline therefore parses those entities directly instead of reconstructing geometry first and deriving entities later.

Lumera's stages (from paper abstract + appendices):

1. Lumera-Box — VLM parses object-level 3D boxes (position, size, yaw) from the image.
2. Lumera-Light — VLM parses parametric light tuples (x, y, z, r, g, b, I).
3. Per-object mesh reconstruction — a mesh per parsed box.
4. HDR environment estimation — environment probe (IntrinsicHDR partially compensates for SkyLight/unseen near-camera lights).
5. Bounded agentic refinement loop — dual-agent (geometry stage, then light stage) edits + verifies against the reference image.
6. Assembly — emit an engine-native scene (.tscn in Summer Engine).

Dataset backing it: Lumera-2K, 2,513 game-engine projects → 3.73M components, 63M object instances, 102.6K parametric lights, 95.1K camera views.

Scene / Node Shape (Summer Engine target)

The assembly stage should emit this node tree so output stays inspectable/editable:

Copy code
ReconstructedScene (Node3D)
├── Camera3D                    # recovered viewpoint (intrinsics from parsing stage)
├── WorldEnvironment
│   └── Environment
│       └── Sky (HDR panorama from stage 4) + ambient light settings
├── Objects (Node3D)
│   └── MeshInstance3D × N      # one per parsed box; transform = box center/yaw
│       └── unique material per object (so meshes are replaceable/refinable)
└── Lights (Node3D)
    ├── OmniLight3D / SpotLight3D / DirectionalLight3D × M
    │     # position/rotation from (x,y,z); color from (r,g,b);
    │     # light_energy from I (mind log-scale calibration, see Tunables)
    └── (the sky-light equivalent is carried by WorldEnvironment)

Keep objects and lights as separate siblings under named containers. That separation is what makes the scene editable rather than just looking 3D.

Implementation Steps

1. Define the intermediate scene schema

Before touching models, define the serializable tuple set the pipeline passes between stages (this is the engine-native contract):

gdscriptCopy code
# object record parsed by the box stage
class_name ParsedObject
var box_center: Vector3
var box_size: Vector3      # must be > 0; validate (paper strips non-positive sizes)
var yaw: float             # watch 0 vs ±π/2 symmetry for near-square objects
var label: String
var id: int                # unique; duplicates are a known model failure mode

# light record parsed by the light stage
class_name ParsedLight
var position: Vector3
var color: Color           # r,g,b
var intensity: float       # I, log-scale calibration needed (~2.7x error typical)
var kind: String           # point / spot / directional mapping

Add a repair layer on ingest: drop/fix invalid labels, non-positive sizes, duplicate IDs. The paper notes sanitized evaluation silently removed these, and recommends constrained decoding or a repair layer in the inference path.

2. Box + light parsing (external VLM components)

Lumera-Box/Lumera-Light are VLMs SFT'd on Lumera-2K; there is no public checkpoint. Options: use a hosted VLM prompted to emit the JSON tuples above, or fine-tune an open VLM on your own engine-exported scenes (.tscn scenes are text and easy to export; this is the "build Lumera-2K for your engine" step the paper identifies as missing for other engines).

3. Per-object mesh generation + HDR environment estimation

Also external model components in Lumera (per-object mesh reconstruction; IntrinsicHDR-style environment estimation). For a Summer Engine blockout you can substitute: box mesh (BoxMesh sized to the parsed box) per object and a generated/estimated panorama Sky. Swap in real reconstructed meshes later without changing the schema.

4. Assembly: write a .tscn

Deterministic GDScript/tool script: instantiate the node shape above from the repaired records. A .tscn is text, so generating it directly keeps the output diffable and versionable. Save with PackedScene.pack() / ResourceSaver.save() from an editor tool.

5. Bounded dual-agent refinement loop

The paper's Algorithm 1 (Appendix H.7), adapted:

• Two stages in order: geometry (objects/boxes), then lights. Switch the agent scope and prompt head per stage, but inherit scene state s and conversation history M across the switch — this avoids a cold start and lets the light-stage verifier refer to object identities already confirmed in the geometry stage.
• Each round: agent proposes an action a_t (move/resize/add/remove an entity) given the history, reference image, and current scene state; the executor applies it to the scene; a verifier renders the result and reports r_t against the reference image.
• Sliding-window history: keep only the system prompt + most recent L action/report pairs (M_{t+1} = Tail_L(M_t ∪ {a_t, r_t}), per VIGA). Skip records adjacent to rolled-back steps. This keeps 30+ rounds inside the VLM context window.
• Bounded: hard round limits T_g (geometry) and T_l (light), plus early termination when the verifier report triggers it. Never run unbounded — that is what keeps this agentic loop "bounded" and cheap.

6. Evaluation harness

Reproduce the paper's metrics on held-out scenes so improvements are measurable:

Tunables

Failure Modes & Gotchas

• Merged/baked outputs defeat the point. If assembly emits one combined mesh or a splat, you've built an image-to-3D renderer, not Lumera. Keep per-object nodes + parametric lights.
• Invalid model outputs are common: non-positive box sizes, duplicate IDs, ID fallbacks. Validate and repair before assembly; the paper's benchmark silently sanitized these.
• Yaw ambiguity on near-square objects (0 vs ±π/2 look identical) — single-token yaw serialization in prior work does not handle symmetry.
• Relation structure is unsolved (object-to-object spatial relations): anchor recall/SRF are Lumera's weakest scores. Don't promise correct relative layouts.
• Per-light localization is weak (F1 0.209 @ 0.5 m) while scene-level light count recall is strong (0.998). Frame the output as "right number of lights, roughly right places."
• SkyLight / near-camera lights whose source is off-screen are not supervised; the environment probe (IntrinsicHDR) only partially compensates.
• Large outdoor scenes drift geometrically (~17 m Chamfer-L2); consider joint camera+geometry calibration if you need metric outdoor consistency.
• Engine conventions don't transfer for free. The dataset's light conventions come from another engine; Summer Engine's light units and probes differ, so recalibrate intensity and validate the representation on Summer-native scenes.

Verification

Not yet verified in Summer Engine. To validate an implementation:

1. Assemble a scene from a test image; confirm the .tscn contains separate, selectable MeshInstance3Ds and light nodes (editability check).
2. Move/replace one mesh and re-tune one light in the editor without breaking the scene.
3. Run the evaluation harness (§6) against a held-out set of known scenes and compare to the reference numbers above.
4. Confirm the refinement loop terminates within T_g/T_l and the history window never exceeds the VLM context budget.

Confidence

extracted — Architecture, metrics, refinement-loop design, and limitations are drawn directly from the arXiv paper (abstract, intro, appendices H.7–I as retrieved; middle method sections were truncated and are not documented here) and the author's announcement post. The Summer Engine node mapping and GDScript schema are adaptations, not from the sources; no public Lumera code exists to reference.