Hi there! I am a Research Scientist at Nvidia GEAR Lab. I graduated from Carnegie Mellon University’s Robotics Institute, School of Computer Science, advised by Prof. Kris Kitani. I earned my bachelor’s degree from University of Pennsylvania in 2019, where I worked with Prof. Kostas Daniilidis. My research has been supported by Qualcomm Innovation Fellowship and the Meta AI Mentorship Program.
My research interest lies at the intersection of robotics, vision, and learning. I am working on topcis including humanoid control, humnoid-object interaction, egocentric perception etc. Through my research, I want to create methods that effectively interpret spatial-temporal sensory input and build a representation of the 3D world to reason about the interactions between agents and the physical environment. On the application side, I am excited about humanoid robots and AR/VR.
Publications
- SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control
-
Zhengyi Luo†, Ye Yuan†, Tingwu Wang†, Chenran Li†, Sirui Chen*, Fernando Castañeda*, Zi-Ang Cao*, Jiefeng Li*, David Minor*, Qingwei Ben*, Xingye Da*, Runyu Ding, Cyrus Hogg, Lina Song, Edy Lim, Eugene Jeong, Tairan He, Haoru Xue, Wenli Xiao, Zi Wang, Simon Yuen, Jan Kautz, Yan Chang, Umar Iqbal, Linxi "Jim" Fan‡, Yuke Zhu‡
- In Submission
- † Equal contribution, * Core Contributor, ‡ Project Leads
-
[Paper]
[Project]
- BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning
-
Yitang Li*,
Zhengyi Luo*, Tonghe Zhang, Cunxi Dai, Anssi Kanervisto, Andrea Tirinzoni, Haoyang Weng, Kris Kitani, Mateusz Guzek, Ahmed Touati, Alessandro Lazaric, Matteo Pirotta, Guanya Shi
- In Submission
- * Equal contribution
-
[Paper]
[Project]
- Self-Improving Vision-Language-Action Models with Data Generation via Residual RL
-
Wenli Xiao*, Haotian Lin*, Andy Peng, Haoru Xue, Tairan He, Yuqi Xie, Fengyuan Hu, Jimmy Wu,
Zhengyi Luo, Linxi Fan, Guanya Shi, Yuke Zhu
- In Submission
- * Equal contribution
-
[Paper]
[Project]
- HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos
-
Haoyang Weng, Yitang Li, Nikhil Sobanbabu, Zihan Wang,
Zhengyi Luo, Tairan He, Deva Ramanan, Guanya Shi
- In Submission
-
[Paper]
[Project]
[Code]
- Learning Universal Humanoid Control
-
Zhengyi Luo
- PhD Thesis
-
[Paper]
- MaskedManipulator: Versatile Whole-Body Control for Loco-Manipulation
-
Chen Tessler, Yifeng Jiang, Erwin Coumans,
Zhengyi Luo, Gal Chechik, Xue Bin Peng
- SIGGRAPH Asia 2025
-
[Paper]
[Code]
- UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Control
-
Yan Wu, Korrawe Karunratanakul,
Zhengyi Luo, Siyu Tang
- ICCV 2025 (Highlight)
-
[Paper]
[Project]
[Code]
- Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning
-
Zhengyi Luo*, Chen Tessler*, Toru Lin, Ye Yuan, Tairan He, Wenli Xiao, Yunrong Guo, Gal Chechik, Kris Kitani, Linxi "Jim" Fan, Yuke Zhu
- In Submission
- * Equal contribution
-
[Paper]
[Project]
- ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills
-
Tairan He*, Jiawei Gao*, Wenli Xiao*, Yuanhang Zhang*, Zi Wang, Jiashun Wang,
Zhengyi Luo, Guanqi He, Nikhil Sobanbab, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, Linxi Fan, Yuke Zhu
- RSS 2025
-
[Paper]
[Project]
[Code]
- ISMimic: Learning Basketball Interaction Skills from Demonstrations
-
Yinhuai Wang*, Qihan Zhao*, Runyi Yu*, Ailing Zeng, Jing Lin,
Zhengyi Luo, Hok Wai Tsui, Jiwen Yu, Xiu Li, Qifeng Chen, Jian Zhang, Lei Zhang, Ping Tan
- CVPR 2025
- * Equal contribution
-
[Paper]
[Project]
[Code]
- HOVER: Versatile Neural Whole-Body Controller for Humanoid Robots
-
Tairan He*, Wenli Xiao*, Toru Lin,
Zhengyi Luo, Zhenjia Xu, Zhenyu Jiang, Jan Kautz, Changliu Liu, Guanya Shi, Xiaolong Wang, Linxi Fan, Yuke Zhu
- ICRA 2025
- * Equal contribution
-
[Paper]
[Project]
[Code]
- CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control
-
Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda,
Zhengyi Luo, Xue Bin Peng, Amit H. Bermano, Michiel van de Panne
- ICLR 2025
-
[Paper]
[Project]
[Code]
- Humanoidlympics: Sports Environments for Physically Simulated Humanoids
-
Zhengyi Luo*, Jiashun Wang*, Kangni Liu*, Haotian Zhang, Chen Tessler, Jingbo Wang, Ye Yuan, Jinkun Cao, Zihui Lin, Fengyi Wang, Jessica Hodgins, Kris Kitani
- In submission
- * Equal contribution
-
[Paper]
[Project]
[Code]
- Harmony4d: A video dataset for in-the-wild close human interactions
-
Rawal Khirodkar*, Jyun-Ting Song*, Jinkun Cao,
Zhengyi Luo, Kris Kitani
- NeurIPS Datasets and Benchmarks 2024
-
[Paper]
[Project]
[Code]
- Grasping Diverse Objects with Simulated Humanoids
-
Zhengyi Luo*, Jinkun Cao*, Sammy Christen, Alexander Winkler, Kris Kitani, Weipeng Xu
- NeurIPS 2024
- * Equal contribution
-
[Paper]
[Project]
- OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
-
Tairan He*,
Zhengyi Luo*, Xialin He*, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, Guanya Shi
- CORL 2024
- * Equal contribution
-
[Paper]
[Project]
- Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation
-
Tairan He*,
Zhengyi Luo*, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, Guanya Shi
- IROS 2024 (Oral Presentation)
- * Equal contribution
-
[Paper]
[Project]
- Real-Time Simulated Avatar from Head-Mounted Sensors
-
Zhengyi Luo, Jinkun Cao, Rawal Khirodkar, Alexander Winkler, Jing Huang, Kris Kitani, Weipeng Xu
- CVPR 2024 (Highlight — Top 11%)
-
[Paper]
[Project]
[Code]
- PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios
-
Jingbo Wang*,
Zhengyi Luo*, Ye Yuan, Yixuan Li, Bo Dai
- CVPR 2024
- * Equal contribution
-
[Paper]
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
-
K Grauman et al.
- CVPR 2024 (Oral — Top 3%)
-
[Paper]
[Project]
- PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction
-
Yinhuai Wang, Jing Lin, Ailing Zeng,
Zhengyi Luo, Jian Zhang, Lei Zhang
- In Submission
-
[Paper]
[Project]
- Universal Humanoid Motion Representations for Physics-Based Control
-
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, Weipeng Xu
- ICLR 2024 (Spotlight — Top 5%)
-
[Paper]
[Project]
[Code]
- Perpetual Humanoid Control for Real-time Simulated Avatars
-
Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, Weipeng Xu
- ICCV 2023
-
[Video]
[Paper]
[Project]
[Code]
- Learning Human Dynamics in Autonomous Driving Scenarios
-
Jingbo Wang, Ye Yuan,
Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, Sameh Khamis
- ICCV 2023
-
[Paper]
- Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion
-
Davis Rempe*,
Zhengyi Luo*, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, Or Litany
- CVPR 2023
- * Equal contribution
-
[Video]
[Paper]
[Project]
- Embodied Scene-aware Human Pose Estimation
-
Zhengyi Luo*, Shun Iwase*, Ye Yuan, Kris M. Kitani
- NeurIPS 2022
- * Equal contribution
-
[Video]
[Paper]
[Project]
[Code]
- From Universal Humanoid Control to Automatic Physically Valid Character Creation
-
Zhengyi Luo, Ye Yuan, Kris M. Kitani
- ArXiv
-
[Paper]
[Project]
[Code]
- Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design
-
Ye Yuan, Yuda Song,
Zhengyi Luo, Wen Sun, Kris M. Kitani
- ICLR 2022 (Oral Presentation — Top 1.6%)
-
[Paper]
[Project]
[Code]
- Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation
-
Zhengyi Luo, Ryo hachiuma, Ye Yuan, Kris M. Kitani
- ICCVW (CV4ARVR) 2021, NeurIPS 2021
-
[Video]
[Paper]
[Project]
[Code]
- 3D Human Motion Estimation via Motion Compression and Refinement
-
Zhengyi Luo, S. Alireza Golestaneh, Kris M. Kitani
- ACCV 2020
- Oral Presentation
-
[Video]
[Paper]
[Project]
[Code]
- Learning Shape Representations for Clothing Variations in Person Re-Identification
-
Yu-Jhe Li,
Zhengyi Luo, Xinshuo Weng, Kris M. Kitani
- arXiv, 2020
-
[Paper]
- Cross-Domain 3D Equivariant Image Embeddings
-
Carlos Esteves, Avneesh Sud,
Zhengyi Luo, Kostas Daniilidis, Ameesh Makadia
- ICML 2019
-
[Paper]
- Cloud Chaser: Real Time Deep Learning Computer Vision on Low Computing Power Devices
-
Zhengyi Luo, Austin Small, Liam Dugan, Stephen Lane
- ICMV 2018
-
[Paper]
[Project]
[Code]
- The rural–urban stress divide: Obtaining geographical insights through Twitter
-
Kokil Jaidka, Sharath Chandra Guntuku, Jane H Lee,
Zhengyi Luo, Anneke Buffone, Lyle H Ungar
- Computers in Human Behavior, 2020: 106544
-
[Paper]
[Project]
---
Professional Services
Conference Reviewer: ICML, ICLR, NeurIPS, CVPR, ICCV, ECCV, CORL, ICRA, RSS,
IROS, SIGGRAPH
Journal Reviewer: IJCV, TMM, RA-L
