ICML 2026 submission

G2TAM

Geometry Grounded Track Anything Model

Promptable instance tracking in 3D space from unordered RGB images or videos, using geometry as an implicit memory for stable identity reasoning across views and time.

Chenming Zhu, Peizhou Cao, Jingli Lin, Wenbo Hu, Yunlong Ran, Jiangmiao Pang, Tai Wang, Xihui Liu

HKU · Shanghai AI Laboratory · BUAA · SJTU · UCLA · ZJU

Overview Results BibTeX
G2TAM teaser showing text, point, and box prompts producing consistent masks and 3D reconstruction across views.
Text, point, and box prompts are grounded in one geometry-semantic representation, enabling instance-consistent masks and reconstruction from RGB inputs.

Overview

Tracking with spatial memory, not only appearance memory.

TL;DR G2TAM takes unordered images or videos plus a text, point, or box prompt, then jointly predicts 3D geometry and spatially consistent instance masks.

Human spatial understanding arises from jointly perceiving geometry and semantics, enabling consistent object identification and localization across viewpoints and time. Current video segmentation models rely on explicit object appearance memory banks, but remain vulnerable to large viewpoint changes and long-term occlusions.

G2TAM uses spatially aligned geometric representations as implicit memory. A cross-modal spatial encoder integrates visual and textual prompts into a shared geometric space, supporting end-to-end reconstruction and instance-consistent mask prediction.

Method

Geometry as implicit memory.

G2TAM architecture with prompt encoders, cross-modal spatial encoder, geometry decoder, and mask decoder.
01

Prompt encoders

Point and box prompts are represented through coordinates and type embeddings. Text prompts are encoded with CLIP and projected into the same latent space.

02

Cross-modal spatial encoder

Prompt tokens are fused with per-frame DINOv2 vision tokens and register tokens, then processed through intra-view and global cross-view attention.

03

Geometry and mask decoders

The fused representation predicts camera geometry, point maps, confidence maps, and prompt-conditioned masks without a separate temporal memory bank.

Prompt handling modes in the cross-modal spatial encoder.
Visual prompts are localized to prompted frames, while text prompts act as global conditioning across frames.

InsTrack and PIST

A benchmark for promptable instance spatial tracking.

InsTrack builds on ScanNet++ and provides RGB images, multi-modal prompts, depth maps, camera poses, and 3D-consistent instance masks. The benchmark is split into InsTrack-Text and InsTrack-Visual to evaluate language and visual prompts separately.

856 training scenes
50 validation scenes
99,666 text prompts
77,007 visual prompts
Qualitative InsTrack validation examples with spatially consistent masks across frames.
Qualitative InsTrack examples show consistent target masks across large view changes and diverse prompt types.

Promptable Instance Spatial Tracking

Given a point, box, or referring expression on any view, PIST evaluates whether the same object is segmented consistently across all views.

S-mIoU S-SR

Results

Strong spatial consistency across static and dynamic settings.

Overall PIST 74.3 / 80.1

S-mIoU / S-SR on InsTrack validation, compared with SAM2 at 47.6 / 53.1 on visual prompts.

InsTrack Text 72.3 / 77.6

Substantially above ReferFormer at 37.6 / 43.7 and ReferDINO at 41.7 / 48.2.

InsTrack Visual 75.8 / 81.2

Large gains over Cutie-base and SAM2 under large viewpoint changes.

RGB-only geometry 2.51 / 86.91

Best Abs. Rel / delta on the InsTrack reconstruction metrics reported in the paper.

Qualitative comparison between G2TAM variants and SAM2 across a challenging sequence.
Joint reconstruction and segmentation stabilizes identity tracking where segmentation-only variants and SAM2 fail.
Task Setting Baseline G2TAM
3D visual grounding ScanRefer Acc@0.5 VLM-Grounder 32.8 45.7
3D visual grounding NR3D Acc@0.5 3D-VisTA 42.2 45.8
Semi-supervised VOS MOSE val J&F SAM2 75.2 77.8
Referring VOS Ref-YouTube-VOS J&F ReferDINO 69.3 72.2

Long-term Spatial Memory

Explore, leave, revisit, and keep the same identity.

Long-term explore-and-revisit trajectory showing G2TAM re-recognizing target objects after returning to the scene.
In explore-and-revisit trajectories, G2TAM re-recognizes targets after long temporal gaps by grounding them in spatial structure.

Citation

Reference the project.

@article{zhu2026g2tam,
  title={G2TAM: Geometry Grounded Track Anything Model},
  author={Zhu, Chenming and Cao, Peizhou and Lin, Jingli and Hu, Wenbo and Ran, Yunlong and Pang, Jiangmiao and Wang, Tai and Liu, Xihui},
  journal={arXiv preprint},
  year={2026}
}