Xinlei Yu (Leo)

My name is Xinlei (Leo) Yu, a second year CS PhD student at the University of Southern California, advised by Dr. Heather Culbertson. I’m also a Student Researcher at Google. I study how human experience can be represented and used to support robot reasoning and action.

My research asks how robots can use knowledge from people’s everyday experiences to interpret requests and act in the physical world. I develop systems and representations that connect first-person observations, spatial context, and human feedback. My recent work includes personalized robot manipulation from smart-glasses recordings and interactive 3D scene representations for haptic experiences.

Building on this work, I aim to develop structured, executable representations that people can inspect and refine, and that robots can use for reasoning and action. My broader goal is to enable robots to adapt to individual users and unfamiliar situations through experience and interaction.

Three-panel comic: a person asks a robot for coffee; the robot wonders which kind, then brings a cappuccino while the person thinks, 'Maybe next time I'll be more specific.' — **How do we address this?** (Image credit by GPT)

I want to extend my heartfelt thanks to the many brilliant colleagues and mentors I’ve had the pleasure of working with. Their support and guidance—from insightful conversations that sparked new ideas, to their steady direction whenever I felt lost—have been instrumental in my journey, and for that, I am sincerely grateful.

I am open and happy to discuss new research projects, innovative ideas, and potential collaborations. Please feel free to contact me at xinleiyu@usc.edu

Publications

A participant navigating between waypoints with the propeller-based handheld force feedback device.

Xinlei Yu, Yang Chen, Heather Culbertson

IEEE Transactions on Haptics 2026

A handheld, propeller-based device that delivers force feedback for navigation.

CrazyJoystick flying into a user's hand and providing haptic feedback in virtual reality.

CrazyJoystick: A Handheld Flyable Joystick for Providing On-Demand Haptic Feedback in Virtual Reality

Yang Chen*, Xinlei Yu*, Heather Culbertson

IEEE World Haptics Conference 2025

A flyable handheld joystick that brings on-demand haptic feedback into virtual reality.

Under Review

Smart-glasses recordings become a personal context store that helps a robot select the user's red sunglasses and abstain when the user's mug is absent.

RoboMemo: Personalized Robot Manipulation from Everyday Smart-Glasses Recordings

Xinlei Yu, Chen Yang, Andrew Nhu, Bridget Liu, Xiao Yuan, Zhiheng Jia, Heather Culbertson

Under reviewICRA 2027

Distills everyday smart-glasses recordings into object descriptions and visual references, helping robots identify a user's intended object—or decline when it is absent—without policy fine-tuning.

RGB video capture becomes a touch-queryable 3D Gaussian scene paired with a haptic library, enabling contact-based vibrotactile retrieval in VR.

Cap2Hap: Contact-Conditioned Vibrotactile Retrieval for Captured 3D Gaussian Scenes

Xinlei Yu, Zeqing Wang, Haolin Xiong, Heather Culbertson

Under reviewIEEE VR 2027

Retrieves vibrotactile feedback from contact regions in reconstructed 3D Gaussian scenes, without meshes or material labels. Reports 7–11× fewer spurious within-surface haptic changes than screen-space baselines and higher perceptual ratings across three user studies.

3DCodeVerse turns reference images and text prompts into executable 3D code, illustrated with generated scenes, graphics, articulated objects, and 3D assets.

3DCodeVerse: 3D Generation via Code

Yipeng Gao, Ziyao Zeng, Jinfa Huang, Yijiang Li, Haolin Xiong, Wang Qin, Xinlei Yu, Youheng Yao, Jie Zhang, Kun-Yu Lin, Yinyuan Zhao, Jiebo Luo, Yajie Zhao, Lei Shu, Yunhao Ge, Meiqi Guo, Xinyu Hu, Laurent Itti

Under reviewICLR 2027

Recasts 3D generation as executable code, combining a one-million-sample dataset, an automated multi-agent verification and curation pipeline, and fine-tuned open models for spatial reasoning and 3D creation.

A shared Gaussian PCA-Tree hierarchy exports a garden scene at 4, 8, 16, and 32 times compression, with rendering speed, quality, and memory comparisons.

Gaussian PCA-Tree: Analytic Hierarchy for Multi-ratio 3DGS Compaction

Haolin Xiong, Gonglin Chen, Xinlei Yu, Butian Xiong, Yajie Zhao

Under review

Builds an analytic hierarchy for compressing trained 3D Gaussian scenes to different primitive budgets. Enables linear-time extraction without learned importance scores while preserving rendering quality under aggressive compression.

TouchTwin captures and reconstructs a tabletop scene, proposes material regions, and supports language corrections and stylus-based haptic comparison with the real scene.

TouchTwin: Human–AI Haptic Authoring for 3D-Scanned Tabletop Scenes through Language and Touch

Wanli Qian, Xinlei Yu, Heather Culbertson

Under reviewCHI 2027

Turns phone scans into editable haptic scenes through a propose–feel–revise loop. In a 12-participant study, language and spatial edits improved realism from 4.0 to 5.6/7 and material-family accuracy from 52% to 92%.

Direct, AI-mediated, and autonomous touch workflows deliver social touch to a person through a wearable haptic interface.

Premankur Banerjee, Xinlei Yu, Kyuhong Lee, Magen Mozeh, Heather Culbertson

Under reviewCHI 2027

Examines directly enacted, AI-composed, and AI-authored touch in video calls. Two 24-participant studies distinguish AI's ability to preserve affective intent from the reduced social presence and naturalness associated with autonomous authoring.

Project Playground

	Semantic 3D Perception and VLM-based Planning for Humanoid Loco-Manipulation [Demo Video Alternative]
	Real-Time UAV Teleoperation via an Immersive VR Interface [Demo Video] [System Diagram]
	VR Dressing Room [Demo Video]

And More