Xinlei Yu (Leo)
My name is Xinlei (Leo) Yu, a second year CS PhD student at the University of Southern California, advised by Dr. Heather Culbertson. I’m also a Student Researcher at Google. I study how human experience can be represented and used to support robot reasoning and action.
My research asks how robots can use knowledge from people’s everyday experiences to interpret requests and act in the physical world. I develop systems and representations that connect first-person observations, spatial context, and human feedback. My recent work includes personalized robot manipulation from smart-glasses recordings and interactive 3D scene representations for haptic experiences.
Building on this work, I aim to develop structured, executable representations that people can inspect and refine, and that robots can use for reasoning and action. My broader goal is to enable robots to adapt to individual users and unfamiliar situations through experience and interaction.
I want to extend my heartfelt thanks to the many brilliant colleagues and mentors I’ve had the pleasure of working with. Their support and guidance—from insightful conversations that sparked new ideas, to their steady direction whenever I felt lost—have been instrumental in my journey, and for that, I am sincerely grateful.
I am open and happy to discuss new research projects, innovative ideas, and potential collaborations. Please feel free to contact me at xinleiyu@usc.edu
Publications
Propeller-based Handheld Force-feedback Device for Navigation
IEEE Transactions on Haptics 2026
Mobile robotic assistants need ways to guide people physically while preserving freedom of movement. We developed a propeller-driven robotic interface that translates navigation goals and the user’s pose into continuous handheld torque cues. In two studies with 12 participants each, it reduced completion time by 54.6% for visually occluded real-world wayfinding and 43.2% for VR target search relative to vibrotactile guidance.
CrazyJoystick: A Handheld Flyable Joystick for Providing On-Demand Haptic Feedback in Extended Reality
IEEE World Haptics Conference 2025
Robotic interfaces can enrich XR experiences by bringing physical feedback to users on demand while preserving free-hand interaction. CrazyJoystick autonomously flies to the user and becomes a handheld force-feedback interface, reallocating propeller thrust to haptic rendering once grasped. In an 18-participant evaluation, users identified directional cues with 92.2% accuracy, and haptic feedback improved reported VR presence.
Under Review
RoboCodeVerse: Scaling 3D Coding to Robot Manipulation with Simulation-Verified Articulated Objects
Under reviewCVPR 2027
Simulation can scale robot manipulation training, but operable articulated assets remain scarce. RoboCodeVerse generates articulated objects as executable 3D code and refines them with structured feedback from contact-level simulation tests. With a fixed training budget, scaling from 10 to 10,000 generated assets raises success on held-out RoboCasa fixtures from 14.8% to 45.2%; a policy trained on 3,000 generated assets opens a real microwave zero-shot in 9/10 trials.
RoboMemo: Personalized Robot Manipulation from Everyday Smart-Glasses Recordings
Under reviewICRA 2027
Personalized robotic assistance requires knowledge of a user’s belongings, preferences, and interaction history that general-purpose models typically lack. RoboMemo turns everyday smart-glasses recordings into retrievable personal memory that grounds robot instructions and verifies object identity before invoking a frozen VLA. In ALOHA 2 look-alike trials, recovered context improves correct object selection from 5/16 to 14/16 and task completion from 3/16 to 9/16, matching manually authored context without policy retraining.
Cap2Hap: Contact-Conditioned Vibrotactile Retrieval for Captured 3D Gaussian Scenes
Under reviewIEEE VR 2027
Digital twins need to represent how environments feel during interaction, beyond their visual appearance. Cap2Hap turns RGB captures into haptically interactive 3D Gaussian scenes in Unity, using language-aligned 3D features to retrieve vibrotactile feedback at the point of contact. It reduces within-surface retrieval changes by 7–11-fold against screen-space baselines and improves perceived haptic realism over 2D multi-view retrieval.
3DCodeVerse: Exploring 3D Worlds as Code
Under reviewICLR 2027
Generating editable 3D content as code is limited by scarce executable training data and inadequate verification. 3DCodeVerse combines a million-scale code dataset, a multi-agent generation and verification framework, and fine-tuned open models. Supervised fine-tuning raises 3DCodeBench execution rates from 0.0% to 90.1% for a 9B model and from 61.8% to 93.4% for a 27B model.
Gaussian PCA-Tree: Analytic Hierarchy for Multi-ratio 3DGS Compaction
Under reviewWACV 2027
Deploying 3D Gaussian scenes at different memory budgets often requires separate optimization for each compression ratio. Gaussian PCA-Tree builds an analytic hierarchy of merged Gaussians and jointly fine-tunes it across budgets. Across three benchmarks, one hierarchy supports 4–32-fold compaction without per-ratio fine-tuning, with higher PSNR and faster GPU extraction than the evaluated single-model multi-budget baselines.
TouchTwin: Human–AI Haptic Authoring for 3D-Scanned Tabletop Scenes through Language and Touch
Under reviewCHI 2027
Visual scans capture appearance but not material feel, while manual haptic authoring requires specialist knowledge. TouchTwin turns phone scans into editable haptic scenes, letting users feel and revise AI-proposed materials through language and spatial edits. With physical reference objects present, 12 participants improved realism ratings from 4.0/7 to 5.6/7 and material-family accuracy from 52% to 92% after a median 6.6 minutes of editing.
Shifting from Human Intent to AI Authorship in Mediated Social Touch
Under reviewCHI 2027
AI-generated touch raises the question of whether a haptic message still conveys another person’s social intent. We built a video-call haptic system comparing direct human touch, human-prompted AI touch, and autonomously authored touch. Two 24-participant studies suggest that AI can render human-intended touch, while autonomous authorship reduces perceived social meaning and social presence.
Project Playground
![]() | Semantic 3D Perception and VLM-based Planning for Humanoid Loco-Manipulation [Demo Video Alternative] |
| | Real-Time UAV Teleoperation via an Immersive VR Interface [Demo Video] [System Diagram] |
![]() | VR Dressing Room [Demo Video] |
And More


