Project page: inter-pose.github.io
Paper: arxiv.org/abs/2412.16155
Great thanks to the amazing team Jason Y. Zhang (@jasonyzhang.bsky.social), Philipp Henzler, Zhengqi Li (@zhengqili.bsky.social), Noah Snavely (@snavely.bsky.social), Ricardo Martin-Brualla.
This also applies to MASt3R. While MASt3R excels with overlapping pairs via feature matching, it struggles with non-overlapping ones due to unreliable correspondences. InterPose maintains robustness, outperforming MASt3R on outward-facing and matching it on center-facing datasets.
We show that InterPose generalizes across 3 SOTA video models (DynamiCrafter, Runway Gen-3, Luma Dream Machine) and consistently outperforms DUSt3R on 4 diverse datasets (indoor, outdoor, object) using our new benchmark, which selects challenging pairs with little to no overlap.
⚠️ Challenge: Generated videos may contain visual artifacts or implausible motion.
🔑 Solution: We generate multiple videos and use a self-consistency metric to select the most visually consistent sample.
💡 Motivation – Powerful Visual Priors: Video models are pre-trained on vast web-scale video data, enabling them to learn significantly more powerful priors of the visual world compared to 3D models like DUSt3R requiring 3D datasets.
🤔Can Generative Video Models Help Pose Estimation?
✅Yes!
We find that generative video models can hallucinate plausible intermediate frames that provide useful context for pose estimators (e.g. DUSt3R), especially for images with little to no overlap.
🔗 inter-pose.github.io
Introducing Doppelgangers++! 🚀 An enhanced pairwise image classifier that tackles visual aliasing (doppelgangers) to improve 3D reconstruction accuracy across diverse, real-world scenes. 🌍✨
🔗Project page: bit.ly/3VAPMJc. Code is also available.
Doppelgangers++: Improved Visual Disambiguation with Geometric 3D Features
Yuanbo Xiangli, Ruojin Cai, Hanyu Chen, Jeffrey Byrne,
@snavely.bsky.social
tl;dr: new dataset (55K pairs) + Mast3r == PROFIT
arxiv.org/abs/2412.05826
