Xinlei Yu (Leo)

My name is Xinlei (Leo) Yu, a second year CS PhD student at the University of Southern California, advised by Dr. Heather Culbertson. I’m also a Student Researcher at Google. I study how human experience can be represented and used to support robot reasoning and action.

My research asks how robots can use knowledge from people’s everyday experiences to interpret requests and act in the physical world. I develop systems and representations that connect first-person observations, spatial context, and human feedback. My recent work includes personalized robot manipulation from smart-glasses recordings and interactive 3D scene representations for haptic experiences.

Building on this work, I aim to develop structured, executable representations that people can inspect and refine, and that robots can use for reasoning and action. My broader goal is to enable robots to adapt to individual users and unfamiliar situations through experience and interaction.

Three-panel comic: a person asks a robot for coffee; the robot wonders which kind, then brings a cappuccino while the person thinks, 'Maybe next time I'll be more specific.'
How do we address this? (Image credit by GPT)

I want to extend my heartfelt thanks to the many brilliant colleagues and mentors I’ve had the pleasure of working with. Their support and guidance—from insightful conversations that sparked new ideas, to their steady direction whenever I felt lost—have been instrumental in my journey, and for that, I am sincerely grateful.

I am open and happy to discuss new research projects, innovative ideas, and potential collaborations. Please feel free to contact me at xinleiyu@usc.edu

Publications

Propeller-based Handheld Force-feedback Device for Navigation

Xinlei Yu, Yang Chen, Heather Culbertson

IEEE Transactions on Haptics 2026

Mobile robotic assistants need ways to guide people physically while preserving freedom of movement. We developed a propeller-driven robotic interface that translates navigation goals and the user’s pose into continuous handheld torque cues. In two studies with 12 participants each, it reduced completion time by 54.6% for visually occluded real-world wayfinding and 43.2% for VR target search relative to vibrotactile guidance.

CrazyJoystick: A Handheld Flyable Joystick for Providing On-Demand Haptic Feedback in Extended Reality

Yang Chen*, Xinlei Yu*, Heather Culbertson

IEEE World Haptics Conference 2025

Robotic interfaces can enrich XR experiences by bringing physical feedback to users on demand while preserving free-hand interaction. CrazyJoystick autonomously flies to the user and becomes a handheld force-feedback interface, reallocating propeller thrust to haptic rendering once grasped. In an 18-participant evaluation, users identified directional cues with 92.2% accuracy, and haptic feedback improved reported VR presence.

Under Review

RoboCodeVerse: Scaling 3D Coding to Robot Manipulation with Simulation-Verified Articulated Objects

Xinlei Yu · Co-first author

Under reviewCVPR 2027

Simulation can scale robot manipulation training, but operable articulated assets remain scarce. RoboCodeVerse generates articulated objects as executable 3D code and refines them with structured feedback from contact-level simulation tests. With a fixed training budget, scaling from 10 to 10,000 generated assets raises success on held-out RoboCasa fixtures from 14.8% to 45.2%; a policy trained on 3,000 generated assets opens a real microwave zero-shot in 9/10 trials.

RoboMemo: Personalized Robot Manipulation from Everyday Smart-Glasses Recordings

Xinlei Yu, Chen Yang, Andrew Nhu, Bridget Liu, Xiao Yuan, Zhiheng Jia, Heather Culbertson

Under reviewICRA 2027

Personalized robotic assistance requires knowledge of a user’s belongings, preferences, and interaction history that general-purpose models typically lack. RoboMemo turns everyday smart-glasses recordings into retrievable personal memory that grounds robot instructions and verifies object identity before invoking a frozen VLA. In ALOHA 2 look-alike trials, recovered context improves correct object selection from 5/16 to 14/16 and task completion from 3/16 to 9/16, matching manually authored context without policy retraining.

Cap2Hap: Contact-Conditioned Vibrotactile Retrieval for Captured 3D Gaussian Scenes

Xinlei Yu, Zeqing Wang, Haolin Xiong, Heather Culbertson

Under reviewIEEE VR 2027

Digital twins need to represent how environments feel during interaction, beyond their visual appearance. Cap2Hap turns RGB captures into haptically interactive 3D Gaussian scenes in Unity, using language-aligned 3D features to retrieve vibrotactile feedback at the point of contact. It reduces within-surface retrieval changes by 7–11-fold against screen-space baselines and improves perceived haptic realism over 2D multi-view retrieval.

3DCodeVerse: Exploring 3D Worlds as Code

Yipeng Gao, Ziyao Zeng, Jinfa Huang, Yijiang Li, Haolin Xiong, Wang Qin, Xinlei Yu, Youheng Yao, Jie Zhang, Kun-Yu Lin, Yinyuan Zhao, Jiebo Luo, Yajie Zhao, Lei Shu, Yunhao Ge, Meiqi Guo, Xinyu Hu, Laurent Itti

Under reviewICLR 2027

Generating editable 3D content as code is limited by scarce executable training data and inadequate verification. 3DCodeVerse combines a million-scale code dataset, a multi-agent generation and verification framework, and fine-tuned open models. Supervised fine-tuning raises 3DCodeBench execution rates from 0.0% to 90.1% for a 9B model and from 61.8% to 93.4% for a 27B model.

Gaussian PCA-Tree: Analytic Hierarchy for Multi-ratio 3DGS Compaction

Haolin Xiong, Gonglin Chen, Xinlei Yu, Butian Xiong, Yajie Zhao

Under reviewWACV 2027

Deploying 3D Gaussian scenes at different memory budgets often requires separate optimization for each compression ratio. Gaussian PCA-Tree builds an analytic hierarchy of merged Gaussians and jointly fine-tunes it across budgets. Across three benchmarks, one hierarchy supports 4–32-fold compaction without per-ratio fine-tuning, with higher PSNR and faster GPU extraction than the evaluated single-model multi-budget baselines.

TouchTwin: Human–AI Haptic Authoring for 3D-Scanned Tabletop Scenes through Language and Touch

Wanli Qian, Xinlei Yu, Heather Culbertson

Under reviewCHI 2027

Visual scans capture appearance but not material feel, while manual haptic authoring requires specialist knowledge. TouchTwin turns phone scans into editable haptic scenes, letting users feel and revise AI-proposed materials through language and spatial edits. With physical reference objects present, 12 participants improved realism ratings from 4.0/7 to 5.6/7 and material-family accuracy from 52% to 92% after a median 6.6 minutes of editing.

Shifting from Human Intent to AI Authorship in Mediated Social Touch

Premankur Banerjee, Xinlei Yu, Kyuhong Lee, Magen Mozeh, Heather Culbertson

Under reviewCHI 2027

AI-generated touch raises the question of whether a haptic message still conveys another person’s social intent. We built a video-call haptic system comparing direct human touch, human-prompted AI touch, and autonomously authored touch. Two 24-participant studies suggest that AI can render human-intended touch, while autonomous authorship reduces perceived social meaning and social presence.

Project Playground

Humanoid Image Semantic 3D Perception and VLM-based Planning for Humanoid Loco-Manipulation
[Demo Video Alternative]
System Image Real-Time UAV Teleoperation via an Immersive VR Interface
[Demo Video] [System Diagram]
VR Room Image VR Dressing Room
[Demo Video]


And More