Perceiving Next Item

perceiving-next-item

The repeating head of a pack-all loop. One robot.get_observation feeds two perceptions — the destination container and the next remaining item — and a decide router turns the item verdict into the loop's continue/terminate signal. It is perceiving-objects (the reliable pairwise-tournament perception) composed with (a) a second perception for the container and (b) a clean none exit, so a "pick up every object and place it in the basket" workflow has one subgraph that answers "is there still an item, and where is the basket?" every pass.

Detection uses the same pairwise VLM crop tournament as perceiving-objects (~30% → 97% object-ID over a one-shot Set-of-Marks pick on the LIBERO-PosVar study). The clean loop terminator comes from the item's container-excluding object_description: once only the basket remains, the tournament + verify gate answer "no grocery item", perceive_item returns found=False, and the decide router emits none → the loop exits to done.

When to use

• The repeating head of a pack-all / clean-all-items loop (transport routes its success edge back here), where each pass must reliably decide "grasp the next item" vs "everything is packed, stop".
• When the downstream place needs a fresh container OBB every pass (this skill emits container_obb/container_mask/container_cloud alongside the target).

When NOT to use

• Single pick-and-place (grab ONE named object). Use perceiving-objects (one target, no loop, no container co-perception).
• You want the container localized once, out of the loop. If the container never moves and you prefer to perceive it a single time before the loop, use a plain perceiving-objects subgraph for the basket + perceiving-objects-oneshot for the looping target. This skill deliberately re-localizes both each pass (robust to a nudged basket, one observation, one clean none).
• Cluttered scenes with look-alike distractors. Prefer perceiving-objects for the target identity — its pairwise-tournament plus object_description hints disambiguate look-alikes (it lacks the clean loop none, so you would add your own decide).

vs. perceiving-objects-oneshot

oneshot also has a clean not_found loop terminator, but it identifies with a single Set-of-Marks VLM pick (weaker on small/similar items) and perceives only the target (no container). This skill uses the pairwise tournament and co-localizes the container, so both the "is anything left?" decision and the downstream place are more reliable — at the cost of a second perception per pass.

Recommended subgraph state flow

textCopy code
observe → exterior_view → perceive_container → filter_obb_container
        → perceive_item → decide ──found──▶ filter_obb_item ──▶ found
                                └──none──▶ none

See examples/canonical_subgraph.json for the exact node/edge/output wiring — emit it verbatim, adapting only the object_name/object_description literals to the task's items and container.

State details:

1. observe — type: tool, robot.get_observation, inputs: {}.
2. exterior_view — type: script, scripts/<sg>/exterior_view.py, inputs: {cameras: Ref("observe.cameras")}. Drops the wrist cam so the container OBB is not bloated by the angled eye-in-hand view.
3. perceive_container — type: script, scripts/<sg>/perceive_dino_vlm.py, inputs: {cameras: Ref("exterior_view.cameras"), object_name: "basket"}.
4. filter_obb_container — type: tool, geometry.filter_and_compute_obb, inputs: {points: Ref("perceive_container.cloud")}.
5. perceive_item — type: script, scripts/<sg>/perceive_dino_vlm.py, inputs: {cameras: Ref("observe.cameras"), object_name: "grocery item", object_description: "…never the wicker basket…"} (the exclusion is a HARD rule above). For SUBSET tasks (specific items only), add reject_unverified: True and name exactly the allowed items in the literals — see the subset-scoping HARD rule.
6. decide — type: router, scripts/<sg>/decide_next_item.py, inputs: {found: Ref("perceive_item.found"), cloud: Ref("perceive_item.cloud"), container_obb: Ref("filter_obb_container.obb")}. Maps found → filter_obb_item, none → none.
7. filter_obb_item — type: tool, geometry.filter_and_compute_obb, inputs: {points: Ref("perceive_item.cloud")}.

Wiring the exit (HARD)

pythonCopy code
sg.add_exit("found")
sg.add_exit("none")
sg.set_exit(success_values=["found", "none"])   # BOTH are clean exits
sg.set_on_error("perception_failed")

sg.set_outputs(
    target_obb=Ref("filter_obb_item.obb"),
    target_mask=Ref("perceive_item.mask"),
    container_obb=Ref("filter_obb_container.obb"),
    container_mask=Ref("perceive_container.mask"),
    container_cloud=Ref("perceive_container.cloud"),
)

At the top level, route this subgraph's found → grasp, none → done, perception_failed → abort.

Required end states

See also

• scripts/perceive_dino_vlm.py — the pairwise-tournament perception (shared with perceiving-objects).
• scripts/decide_next_item.py — the unprivileged loop router: per-pass VLM all-packed check (primary stop) + same-target no-progress guard + perception verdict; sim.check_success is telemetry only (GAP_DECIDE_TRUST_ENV=1 opts into trusting it on benchmarks).
• scripts/exterior_view.py — wrist-cam drop for a clean container OBB.
• prompts/vlm_pairwise.md — the pairwise-tournament VLM prompt template.