PhysMem: Scaling Test-time Physical Memory for Robot Manipulation

View PDF HTML (experimental) Abstract:Reliable object manipulation requires understanding physical properties that vary across objects and environments. Vision-language model (VLM) planners can reason about friction and stability in general terms; however, they often cannot predict how a specific ball will roll on a particular surface or which stone will provide a stable foundation without direct experience. We present PhysMem, a memory framework that enables VLM robot planners to learn physical principles from interaction at test time, without updating model parameters. The system records experiences, generates candidate hypotheses, and verifies them through targeted interaction before promoting validated knowledge to guide future decisions. A central design choice is verification before application: the system tests hypotheses against new observations rather than applying retrieved experience directly, reducing rigid reliance on prior experience when physical conditions change. We evaluate PhysMem on three real-world manipulation tasks and simulation benchmarks across four VLM backbones. On a controlled brick insertion task, principled abstraction achieves 76% success compared to 23% for direct experience retrieval, and real-world experiments show consistent improvement over 30-minute deployment sessions. Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI) Cite as: arXiv:2602.20323 [cs.RO] (or arXiv:2602.20323v5 [cs.RO] for this version) https://doi.org/10.48550/arXiv.2602.20323 arXiv-issued DOI via DataCite Submission history From: Haoyang Li [view email] [v1] Mon, 23 Feb 2026 20:18:35 UTC (18,772 KB) [v2] Wed, 4 Mar 2026 04:33:20 UTC (18,767 KB) [v3] Mon, 23 Mar 2026 00:23:00 UTC (18,767 KB) [v4] Sat, 28 Mar 2026 02:43:53 UTC (18,768 KB) [v5] Tue, 21 Apr 2026 04:12:49 UTC (18,776 KB)