Pete Florence's Home Page

News

✨ We released an early preview of model capabilities from Generalist!
Recommended coverage of RT-2: NY Times Hard Fork Podcast, also Kevin Roose's article which was on the NYT front page.
Recommended coverage of Med-PaLM M: The Cognitive Revolution podcast, interview with Tao and Vivek.
New papers out in July 2023! RT-2, Med-PaLM M, "Scaling Up and Distilling Down", "LLMs as General Pattern Machines".
PaLM-E is out!
New pre-print! Interactive Language: Talking to Robots in Real Time.
ECCV 2022: come chat! Will be at the Frontiers of Monocular 3D Perception workshop.
NeurIPS 2022: NeRF-RL.
CoRL 2022: MIRA, InnerMonologue, VIRDO++.
Socratic Models pre-print out!
Giving a tutorial on "Vision-Based Robot Learning" at CVPR 2022 with Michael Ryoo, Shuran Song, and Andy Zeng on June 20, 2022.
4 papers at ICRA 2022! Thanks to all co-authors on these.
🎙️ Did a podcast! It's always great chatting with Eric Jang and Kevin Zakka -- thanks for setting this up, Kevin!
✍️ We wrote a blog post for the Google AI blog on Implicit BC!
XIRL was a best paper finalist at CoRL 2021.
IEEE Robotics and Automation Letters Best Paper Award (2020) for “Self-Supervised Correspondence in Visuomotor Policy Learning”.

Research

Med-PaLM M

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich.
[ website with videos ]

Med-PaLM M

Towards Generalist Biomedical AI.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, Christopher Semturs, S Sara Mahdavi, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, Vivek Natarajan.
[ pdf ]

pattern-machines

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition.
Huy Ha, Pete Florence, Shuran Song.
[ website with videos and code ]

pattern-machines

Large Language Models as General Pattern Machines.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, Andy Zeng.
[ website with videos and code ]

PaLM-E: An Embodied Multimodal Language Model.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence.
[ website with videos ]

RoboPianist: A Benchmark for High-Dimensional Robot Control.
Kevin Zakka, Laura Smith, Nimrod Gileadi, Taylor Howell, Xue Bin Peng, Sumeet Singh, Yuval Tassa, Pete Florence, Andy Zeng, Pieter Abbeel.
[ website with code and videos ]

grounded-decoding

Grounded Decoding: Guiding Text Generation with Grounded Models for Robot Control.
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, Brian Ichter.
[ website with videos ]

grounded-decoding

NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis.
Allan Zhou, Moo Jin Kim, Lirui Wang, Pete Florence, Chelsea Finn. CVPR 2023.
[ website with videos ]

Interactive Language: Talking to Robots in Real Time.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, Pete Florence. RA-L 2023.
[ website with code and videos ]

silico

Single-Level Differentiable Contact Simulation.
Simon Le Cleac'h, Mac Schwager, Zachary Manchester, Vikas Sindhwani, Pete Florence, Sumeet Singh. RA-L 2023.
[ pdf ]

CAP

Code as Policies: Language Model Programs for Embodied Control.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng. ICRA 2023.
[ website with code and videos ]

Heravi

Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations.
Negin Heravi, Ayzaan Wahid, Corey Lynch, Pete Florence, Travis Armstrong, Jonathan Tompson, Pierre Sermanet, Jeannette Bohg, Debidatta Dwibedi. ICRA 2023.
[ pdf ]

SMs

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence. ICLR 2023.
[ pdf ] [ website with code and videos ]

SMs

MIRA: Mental Imagery for Robotic Affordances.
Yen-Chen Lin, Pete Florence, Andy Zeng, Jonathan T. Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez Garcia, Phillip Isola. CoRL 2022.
[ pdf ] [ website ]

SMs

VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects.
Youngsun Wi, Andy Zeng, Pete Florence, Nima Fazeli. CoRL 2022.
[ pdf ] [ website with code and videos ]

Inner Monologue: Embodied Reasoning through Planning with Language Models.
Wenlong Huang*, Fei Xia*, Ted Xiao*, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter. CoRL 2022.
[ pdf ] [ website with code and videos ]

Reinforcement Learning with Neural Radiance Fields.
Danny Driess, Ingmar Schubert, Pete Florence, Yunzhu Li, Marc Toussaint. NeurIPS 2022.
[ pdf ] [ website with code and videos ]

NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields.
Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Tsung-Yi Lin, Alberto Rodriguez, Phillip Isola. ICRA 2022.
[ pdf ] [ website with code and videos ]

ikp

VIRDO: Visio-tactile Implicit Representations of Deformable Objects.
Youngsun Wi, Pete Florence, Andy Zeng, Nima Fazeli. ICRA 2022.
[ pdf ]

Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learning.
Aditya Ganapathi, Pete Florence, Jake Varley, Kaylee Burns, Ken Goldberg, Andy Zeng. ICRA 2022.
[ pdf ]

Trajectory Optimization with Optimization-Based Dynamics.
Taylor A. Howell, Simon Le Cleac'h, Sumeet Singh, Pete Florence, Zachary Manchester, Vikas Sindhwani. RA-L and ICRA 2022.
[ pdf ] [ code ]

Implicit Behavioral Cloning.
Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, Jonathan Tompson. CoRL 2021.
[ pdf ] [ code ] [ site with video ] [ blog post ]

XIRL: Cross-embodiment Inverse Reinforcement Learning.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, Debdidatta Dwibedi. CoRL 2021.
[ pdf ] [ code ] [ site with video ]
★ Best Paper Finalist, CoRL 2021
★ Oral Presentation, CoRL 2021

iNeRF

iNeRF: Inverting Neural Radiance Fields for Pose Estimation.
Yen-Chen Lin, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, Tsung-Yi Lin. IROS 2021.
[ pdf ] [ site with video ]

bags

Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks.
Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Goldberg, Andy Zeng. ICRA 2021.
[ pdf ] [ site with video ]

Transporter Network architecture

Transporter Networks: Rearranging the Visual World for Robotic Manipulation.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, Johnny Lee. CoRL 2020.
[ pdf ] [ site with videos ]
★ Best Presentation Finalist, CoRL 2020
★ Oral Presentation, CoRL 2020

Keypoints into the future

Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning.
Lucas Manuelli, Yunzhu Li, Pete Florence, Russ Tedrake. CoRL 2020.
[ pdf ] [ site with video ]

Pete Florence

Dense Visual Learning for Robot Manipulation.
Peter Florence. PhD thesis, Massachusetts Institute of Technology, September 2019.
[ pdf ] [ presentation video ]

Pete Florence

Self-Supervised Correspondence in Visuomotor Policy Learning.
Peter Florence, Lucas Manuelli, Russ Tedrake. RA-L and ICRA 2020.
[ pdf ] [ site ] [ video ]
★ Best Paper Award, RA-L 2020
Also at: NeurIPS 2019 Workshop on Robot Learning: Control and Interaction in the Real World

kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation.
Lucas Manuelli*, Wei Gao*, Peter Florence, Russ Tedrake. ISRR 2019.
[ pdf ] [ site with videos ]
Also at: CVPR 2019 Workshop on 3D Scene Understanding for Vision, Graphics, and Robotics

Pete Florence

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, Steven Lovegrove. CVPR 2019.
[ pdf ] [ video ] [ code ] [ talk ]
★ Best Paper Finalist, CVPR 2019
★ Oral Presentation, CVPR 2019

Pete Florence

Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation.
Peter R. Florence*, Lucas Manuelli*, and Russ Tedrake. CoRL 2018.
[ pdf ] [ video ] [ code ]
★ Best Paper Award, CoRL 2018
★ Best Technical Paper, 2018 Amazon Robotics Best Paper Awards in Manipulation
Press: [ WIRED ] [ CNN ] [ Newsweek ] [ Engadget ] [ VentureBeat ] [ more ]

Pete Florence

LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes.
Pat Marion*, Peter R. Florence*, Lucas Manuelli*, and Russ Tedrake. ICRA 2018. [ website with code, video, paper ]
★ Finalist, Best Vision Paper Award, ICRA 2018

Pete Florence

NanoMap: Fast, uncertainty-aware proximity queries with lazy search over local 3d data.
Peter R. Florence, John Carter, Jake Ware, and Russ Tedrake. ICRA 2018.
[ pdf ] [ video ] [ code ]
Press: [ IEEE Spectrum ] [ BBC ] [ The Verge ] [ Engadget ] [ Air & Space Magazine ] [ TechCrunch ] [ more ]

Pete Florence

High-speed autonomous obstacle avoidance with pushbroom stereo.
Andrew J. Barry, Peter R. Florence, and Russ Tedrake. Journal of Field Robotics (JFR) 2018.
[ pdf ] [ video ]

Pete Florence

Integrated perception and control at high speed: Evaluating collision avoidance maneuvers without maps.
Peter R. Florence, John Carter, and Russ Tedrake. WAFR 2016.
[ pdf ] [ presentation ]

Pete Florence

Aggressive quadrotor flight through cluttered environments using mixed integer programming.
Benoit Landry, Robin Deits, Peter R. Florence, and Russ Tedrake. ICRA 2016.
[ pdf ][ video ]

Patents

(Pending) Peter Raymond Florence, Christopher David Sachs, Kent W. Ryorchuk. Systems and methods for probabilistic semantic sensing in a sensory network. WO2015134879 A1 (2014)
Christopher David Sachs, Peter Raymond Florence. Dynamic spatially-resolved lighting using composited lighting models. U.S. Patent No. 9763306 (2014)

Technical Reports

Lucas Manuelli, Peter R. Florence. Reinforcement Learning for Autonomous Driving Obstacle Avoidance using LIDAR.
[ pdf ] [ video ] (2015)

Engineering Education for Kids

My friend Andy Barry and I started Stage One Education together. We develop and teach hands-on engineering workshops for kids across the country. Over 10,000 students to date have taken our workshops.