Greedy Mask Overlap Resolution
Overview
Instance segmentation models often produce overlapping masks, but many evaluation metrics (and biological reality) require non-overlapping instances. This technique processes masks in descending confidence order and subtracts already-claimed pixels from each new mask. Simple, fast, and effective — commonly used in cell segmentation where dense packing makes overlap inevitable.
Quick Start
pythonimport numpy as np def resolve_overlaps(masks, scores, min_pixels=75): order = np.argsort(-scores) used = np.zeros(masks[0].shape, dtype=np.uint8) result = [] for idx in order: mask = (masks[idx] > 0).astype(np.uint8) mask = mask * (1 - used) # remove already-claimed pixels if mask.sum() >= min_pixels: used = np.clip(used + mask, 0, 1) result.append(mask) return result # Usage with Mask R-CNN output masks = output['masks'].cpu().numpy() # (N, H, W) scores = output['scores'].cpu().numpy() clean_masks = resolve_overlaps(masks, scores, min_pixels=75)
Workflow
- Sort all predicted masks by confidence score descending
- Initialize an empty occupancy map (zeros, same H×W as image)
- For each mask: subtract occupied pixels, check remaining area ≥ min_pixels
- If large enough, add to result and update occupancy map
- Encode surviving masks as RLE for submission
Key Decisions
- Sort by confidence: highest-confidence masks get priority for contested pixels
- min_pixels threshold: discard masks that become too small after overlap removal; tune per dataset (75-150 for cells)
- Per-class min_pixels: use different thresholds per class if instance sizes vary significantly
- Binary occupancy: simple and fast; for soft overlap, use IoU-based merging instead
