Jin Ryong Kim

About me

I am an Associate Professor in the Department of Computer Science and a Director of the Multimodal Interaction Lab (MI Lab, https://mi-lab.io) at the University of Texas at Dallas. My research strives to create novel haptics interactions to tackle technical and human factors' challenges to amplify human satisfaction through enriched user experience. My research goal is to establish an interdisciplinary research collaboration program to solve real-world challenges in haptics interaction and address issues in human perception and multimodalities in immersive environments.

I received my Ph.D. in Electrical and Computer Engineering (Advisor: Dr. Hong Z. Tan) and an M.S. in Computer Science, both from Purdue University. I also received M.S. and B.S. in Electrical and Computer Engineering from Hanyang University, Seoul, Korea. Prior to the current position, I was a Staff Researcher at Alibaba Group in Sunnyvale, California, and a Senior Researcher at the Electronics and Telecommunications Research Institute (ETRI). I also worked at Microsoft Research Asia (Beijing, China), Samsung Advanced Institute of Technology (Yongin, Korea), and NHK Science and Technology Research Laboratories (Tokyo, Japan) as a research intern (CV).

Research Area

News

Publication

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2012

2011

2009

2007

2006

2005

2004

2003

Projects

Thermal Masking Across the Human Body: Patterns, Pathways, and Perceptual Boundaries (CHI 2026)

Thermal masking, a vibration-induced illusion in which concurrent tactile input induces a vivid thermal sensation at the tactile site, is a promising mechanism for wearable interfaces and extended reality because it can deliver rich thermal feedback with minimal hardware. While prior work has examined this phenomenon on limited body parts, its expression across the full body remains under exploration. We present four studies mapping thermal masking across eight regions: head, face, neck, arms, hands, torso, legs, and feet. Results show that masking strength is location-dependent, producing perceptual patterns that align primarily with somatosensory pathways rather than proximity. On smaller regions such as the fingers, masking was localized, while on larger areas such as the torso and neck, it extended more broadly. Dorsal–ventral and inter-body tests revealed viable pairings and perceptual boundaries. These findings provide the first comprehensive atlas of body-wide thermal masking, advancing understanding and guiding efficient thermal–tactile interface design. [Paper | Video]

Moisture Transfer: A Perceptual Wetness Illusion Through Thermal and Wet Integration (CHI 2026)

Simulating wetness in interactive systems is challenging due to the lack of dedicated hygroreceptors in human skin and the complexity of delivering physical moisture. We introduce Moisture Transfer, a perceptual wetness illusion in which users feel moisture at a dry site when cold and wet stimuli are applied nearby. This illusion arises from the brain’s spatial integration of thermal and tactile cues, offering a new pathway to render wetness without direct contact. We investigate this illusion by establishing it with a single finger and show that thermal congruence enhances perceived wetness. We then explored its spatial extent across five fingers, revealing lateral transfer of wetness. Finally, we applied these findings to create a proof-of-concept VR interface that evokes full-hand wetness using minimal actuation. We conclude with design implications for XR and wearable systems and outline future work exploring body-wide wetness illusions and multisensory integration. [Paper | Video]

Touch with Meaning: A Contextual Analysis of Social Touch (CHI 2026)

Social touch is a rich channel of human communication, conveying emotion, intent, and meaning embedded in context. Yet most HCI studies treat touch in isolation, overlooking the layered subtleties that shape interpretation. We present a contextual analysis of 5,016 social touch events, grounded in a large collection of annotated scenes from films, dramas, and documentaries. Using a computer vision pipeline, we segmented touch events from video and annotated them across dimensions, including who is involved, how the gesture is performed, where on the body it occurs, and the cultural backdrop. Our analysis shows that identical gestures can convey distinct meanings depending on body location, relationship type, and context. Similar intentions—like comfort, encouragement, or dominance—may be expressed through different gestures or locations, shaped by relational dynamics, cultural norms, and public or private settings. These insights inform the design of socially aware touch technologies, including avatars, social agents, and mediated communication systems. [Paper | Video]

SensoryBlox: Plug-and-Feel Modular Multi-Sensory User Interfaces for Immersive Cardboard VR (CHI 2026)

We present SensoryBlox, a modular, multi-sensory user interface designed for integration with cardboard-based virtual reality (VR) head-mounted displays (HMDs). SensoryBlox features interchangeable sensory modules—vibration, temperature, wind, and olfactory—that enable users to assemble customized multi-sensory configurations tailored to diverse VR contexts. The system includes in-VR interfaces for module scanning, spatial tracking, and real-time customization of feedback patterns. To inform SensoryBlox design, we conducted three user studies. The initial study explored application scenarios and associated sensory modalities to identify design requirements for a modular multi-sensory VR system. Based on these findings, we developed the hardware modules and in-VR software interfaces. In the second study, we evaluated the usability and interaction experience of SensoryBlox across all functionalities. Finally, a comparison study examined the impact of multi-sensory feedback on user experience. Our findings demonstrate the potential of a modular multi-sensory system in enriching immersion and engaging interactions within low-cost VR environments. [Paper | Video]

MetaTwin: A Collaborative XR Platform for Seamless Physical-Virtual Synchronization (IMWUT/Ubicomp 2025)

This paper presents MetaTwin, a collaborative platform that enables seamless synchronization between physical and virtual realms for co-existing Extended Reality (XR) experiences. MetaTwin employs a hybrid decentralized server architecture to synchronize user interactions and environments within a shared space, allowing users to collaborate and socialize across physical locations while experiencing the convergence of real and virtual spaces. Integrated IoT devices act as both physical and virtual entities, supporting shared control and enabling resource sharing, such as presentation slides and music. We detail the configuration and deployability of MetaTwin as a solution for XR collaboration. To evaluate performance and feasibility, we compared MetaTwin with an existing XR platform and conducted an ablation study to identify the benefits and limitations of our approach. Additionally, a user study investigates the impact of spatial and temporal synchronization offsets on collaboration quality. Our findings inform the development of operational guidelines for future collaborative XR platforms. [Paper | Video]

Understanding Latency Sensitivity in Thermal and Tactile Feedback for Multimodal Haptics in VR (VRST 2025)

Low-latency multimodal feedback is essential for maintaining a high-quality user experience in VR; however, unpredictable network conditions can introduce latency that negatively impacts user experience. This work investigates how users perceive multimodal haptic feedback—specifically thermal (hot/cold) and tactile stimuli—and how latency in such feedback affects user experience. We first measured users’ response times for thermal, tactile, and combined thermal-tactile stimuli. Subsequently, we conducted a psychophysical study to identify delay thresholds for each modality by examining temporal congruency between visual and haptic cues. We designed a haptic delay network simulator to emulate a realistic network environment. Results highlighted that combined thermal-tactile feedback has higher latency tolerance than thermal-only feedback, indicating that multimodal integration can buffer the negative effects of latency. Using these thresholds, we designed controlled latency conditions and assessed user experience. Based on our findings, we propose design recommendations for haptic data transmission in networked VR systems. [Paper | Video]

HeatFlow: A Thermal-Tactile Display for Dynamic 2D Thermal Movements (UIST 2025)

UIST 2025 Best Paper Honorable Mention Award 🏆

We introduce HeatFlow, a thermal display capable of generating dynamic thermal movements on a 2D surface by integrating thermal and tactile sensations. Leveraging a perception-driven approach, we combine vibration-induced thermal referral and apparent tactile motion to create the illusion of moving thermal flows. The system features a 2D array of nine tactile actuators paired with a single nearby thermal actuator. To validate our approach, we conducted three user studies. The first study determined the optimal parameters for perceivable 2D thermal flows by examining different array sizes and motion speeds under varying temperature conditions. The second study focused on optimizing our algorithm to produce smooth and continuous curved thermal flows. Comparative evaluations against two prior tactile motion techniques demonstrated that our algorithm outperforms existing methods in generating realistic thermal motion. Finally, we integrated HeatFlow into virtual reality (VR) environments, assessing its feasibility in six interactive scenarios across three body sites—the arm, palm, and face. Our findings highlight the potential of HeatFlow to enhance immersive VR experiences by providing realistic thermal feedback across multiple body locations. Additionally, we present HeatFlow as a design tool—an application that enables users to create custom thermal flows by adjusting key parameters such as motion speed, intensity, and temperature. This work lays the foundation for more advanced thermal interfaces in interactive systems. [Paper | Video]

UltraEdit: In-Situ Design Environment for Ultrasound Haptization (UIST 2025)

We present UltraEdit, an in-situ design environment that enables users to directly edit ultrasound haptic sensations in VR using barehand interactions. UltraEdit represents haptic sensations as tangible objects, called blobs, which users can modify or apply to 3D objects through intuitive hand gestures. Users can create, edit, copy, and apply blobs, allowing them to work with multiple haptic sensations to design virtual objects with diverse tactile feedback. Additionally, UltraEdit allows users to create spatial and temporal haptic patterns using time blobs and spatial drawing features. Users can draw custom-shaped haptic patterns directly on their hands, adapting to comprehensive design scenarios. To evaluate UltraEdit, we conducted an exploratory user study assessing its usability, effectiveness, and ease of learning. We also compared its performance to an existing desktop-based haptic editing tool. Participants found UltraEdit intuitive to learn, enjoyable to use, and effective for adding haptic feedback to virtual objects. [Paper | Video]

PropType: Everyday Props as Typing Surfaces in Augmented Reality (CHI 2025)

CHI 2025 Best Paper Honorable Mention Award 🏆

We introduce PropType, an interactive interface that transforms everyday objects into typing surfaces within an Augmented Reality (AR) environment. Users can interact with nearby props, such as cups, water bottles, boxes, and various other objects, utilizing them as on-the-go keyboards. To develop PropType, we conducted three studies. The first study involved observing users to understand how they naturally engage with prop surfaces for typing. The second study assessed the reachability and efficiency of touch input across four props with different sizes and shapes. Based on these insights, we designed customized keyboard layouts for each prop. In the third study, we evaluated typing performance using PropType, achieving an average typing speed of up to 26.1 words per minute (WPM) with 2.2% corrected error rate (CER) and 1.1% uncorrected error rate (UER). Finally, we present a PropType editing tool that allows users to customize keyboard layouts and visual effects for prop-based typing. [Paper | Video]

Thermal In Motion: Designing Thermal Flow Illusions with Tactile and Thermal Interaction (UIST 2024)

ISMAR 2024 Best Demo Award & UIST 2024 Best Demo Honorable Mention Award 🏆

This study presents a novel method for creating moving thermal sensations by integrating the thermal referral illusion with tactile motion. Conducted through three experiments on human forearms, the first experiment examined the impact of temperature and thermal actuator placement on perceived thermal motion, finding the clearest perception with a centrally positioned actuator under both hot and cold conditions. The second experiment identified the speed thresholds of perceived thermal motion, revealing a wider detectable range in hot conditions (1.8 cm/s to 9.5cm/s) compared to cold conditions (2.4cm/s to 5.0cm/s). Finally, we integrated our approach into virtual reality (VR) to assess its feasibility through two interaction scenarios. Our results shed light on the comprehension of thermal perception and its integration with tactile cues, promising significant advancements in incorporating thermal motion into diverse thermal interfaces for immersive VR experiences. [Paper | Video]

Fiery Hands: Designing Thermal Glove through Thermal and Tactile Integration for Virtual Object Manipulation (UIST 2024)

We present a novel approach to render thermal and tactile feedback to the palm and fingertips through thermal and tactile integration. Our approach minimizes the obstruction of the palm and inner side of the fingers and enables virtual object manipulation while providing localized and global thermal feedback. By leveraging thermal actuators positioned strategically on the outer palm and back of the fingers in interplay with tactile actuators, our approach exploits thermal referral and tactile masking phenomena. Through a series of user studies, we validate the perception of localized thermal sensations across the palm and fingers, showcasing the ability to generate diverse thermal patterns. Furthermore, we demonstrate the efficacy of our approach in VR applications, replicating diverse thermal interactions with virtual objects. This work represents significant progress in thermal interactions within VR, offering enhanced sensory immersion at an optimal energy cost. [Paper | Video]

Wetness Illusion in Mid-Air (ISMAR 2024)

This study explores the impact of combining multiple senses on how people perceive wetness in mid-air. We examine how factors like temperature, pressure, and visual stimuli influence the illusion of wetness on users’ palms. The first user study examines these effects and the complex relationships between these variables. The findings suggest that increased temperature and pressure increase the likelihood of perceiving wetness. Interestingly, we note the influence of visual scene content on wetness perception. This leads to our second user study, which investigates how visual context impacts wetness perception. The results demonstrate a strong link between visual content and wetness perception, offering valuable insights for designing immersive virtual reality experiences that enhance sensory perception. [Paper | Video]

Thermal Masking: When the Illusion Takes Over the Real (CHI 2024)

IEEE Haptics Symposium 2024 Best Demo Honorable Mention Award 🏆

This paper reports on a thermal illusion called thermal masking. Thermal masking is a phenomenon induced by thermal referral to completely mask the original thermal sensation, providing thermal sensation only at the tactile site. Three experiments are conducted using thermal and vibrotactile actuators to investigate the nature of thermal masking. The results show a higher percentage of thermal masking occurs in warm than hot or cold conditions. The second experiment examines how far the thermal masking can be perceived. The results show that masking can reach up to 24 cm from the thermal site. The third experiment explores the interaction space by placing the tactile actuators on the opposite side of the thermal actuator. The results confirm that thermal masking can reach the other side of the arm, and the performance was higher in warm conditions. [Paper | Video]

Let It Snow: Designing Snowfall Experience in VR (IMWUT/UbiComp 2024)

We present Snow, a cross-modal interface that integrates cold and tactile stimuli in mid-air to create snowflakes and raindrops for VR experiences. Snow uses six Peltier packs and an ultrasound haptic display to create unique cold-tactile sensations for users to experience catching snowflakes and getting rained on their bare hands. Our approach considers humans’ ability to identify tactile and cold stimuli without masking each other when projected onto the same location on their skin, making illusions of snowflakes and raindrops. We design both visual and haptic renderings to be tightly coupled to present snow melting and rain droplets for realistic visuo-tactile experiences. For multiple snowflakes and raindrops rendering, we propose an aggregated haptic scheme to simulate heavy snowfall and rainfall environments with many visual particles. The results show that the aggregated haptic rendering scheme demonstrates a more realistic experience than other schemes. We also confirm that our approach of providing cold-tactile cues enhances the user experiences in both conditions compared to other modality conditions. [Video]

Fabric Thermal Display using Ultrasonic Waves (ISMAR 2023)

This paper presents a fabric-based thermal display of a polyester fabric material combined with thermally-conductive materials using an ultrasound haptic display. We first empirically test the thermal generation process in five fabric materials by applying 40 kHz ultrasonic waves to the fabric materials. We also examine their thermal characteristics by applying different frequencies and amplitudes of ultrasonic cues. We show that polyester demonstrates the best thermal performance. We then combine it with thermally-conductive materials, including copper and aluminum, and compare them with the fabric-only condition. We integrate polyester with aluminum into a glove to explore the use cases in VR and share our findings, insights, limitations, and future works. [Paper | Video]

Upper Body Thermal Referral and Tactile Masking for Localized Feedback (TVCG/IEEE VR 2023)

This paper investigates the effects of thermal referral and tactile masking illusions to achieve localized thermal feedback on the upper body. We use a 2D array of sixteen vibrotactile actuators (4×4) with four thermal actuators to explore the thermal distribution on the user’s back. A combination of thermal and tactile sensations is delivered to establish the distributions of thermal referral illusions with different numbers of vibrotactile cues. The result confirms that localized thermal feedback can be achieved through cross-modal thermo-tactile interaction on the user’s back of the body. [Paper | Video]

MetaTwin: Synchronizing Physical and Virtual Spaces for Seamless World (VRST 2022)

MetaTwin is a collaborative Metaverse platform that supports one-to-one spatiotemporal synchrony between physical and virtual spaces. The users can interact with other users and surrounding IoT devices without being tied to physical spaces. Resource sharing is implemented to allow users to share media, including presentation slides and music. We deploy MetaTwin in two different network environments (i.e., within the US, Korea-US international) and summarize users’ feedback about the experience. [Paper | Video 1 | Video 2]

Mid-Air Thermo-Tactile Interaction in VR (VRST 2021)

Best Student Demo Award at SIGRAPH Asia 2023 Emerging Technologies 🏆

We design a proof-of-concept thermo-tactile feedback system with an open-top chamber, heat modules, and an ultrasound display. Our approach is to provide heated airflow along the path to the focused pressure point created from the ultrasound display to generate thermal and vibrotactile cues in mid-air simultaneously. [Paper | Video]

TangibleData (VRST 2021)

We build an interactive 3D data visualization tool that adapts hand gestures and mid-air haptics to provide tangible interaction in VR using ultrasound haptic feedback on 3Ddata visualization. We consider two types of 3D visualization datasets and provide different data encoding methods for haptic representations. [Paper 1 | Paper 2 | Video 1 | Video 2]

In-Air Text Input Technique (CHI 2022)

We empirically explore fundamental requirements for achieving VR in-air typing by observing the unconstrained eyes-free in-air typing of touch typists. We examine properties of finger kinematics, correlated movement of fingers, interrelation in consecutive key-strokes, and 3D distribution of key-stroke movements. We further test finger kinematic features, including 3D position, velocity, acceleration, and temporal features, including previous fingers and keys. Based on this analysis, we assess the performance of various classifiers, including Naive Bayes, Random Forest, Support Vector Machines, and Deep Neural Networks, in terms of the accuracy for correctly classifying the keystroke. [Paper 1 | Paper 2 | Video]

Refinity

Refinity is an interactive holographic signage for the new retail shopping experience. In this project, we show a concept of futuristic shopping experience with a tangible 3D mid-air interface that allows customers to directly select and explore realistic virtual products using autostereoscopic 3D display combined with mid-air haptics and finger tracker. We also present an example of in-store shopping scenario for natural interactions with 3D. This shopping experience will engage users in producing a memorable in-store experience with the merging of digital and physical interactions. [Paper | Video]

RealWalk (IEEE Transactions on Haptics | IEEE Haptics Symposium 2019)

RealWalk is a pair of haptic shoes for HMD-based VR, designed to create realistic sensations of ground surface deformation and texture through MR fluid actuators. RealWalk offers a novel interaction scheme through the physical interaction between the shoes and the ground surfaces while walking in VR. Each shoe consists of two MR fluid actuators, an insole pressure sensor, and a foot position tracker. When a user steps on the ground with the shoes, the two MR fluid actuators are depressed, creating a variety of ground material deformation such as snow, mud, and dry sand by changing its viscosity. We build an interactive VR application and compare RealWalk with vibrotactile-based haptic shoes to investigate its effectiveness. [Paper 1 | Paper 2 | Paper 3 | Video]

Touch3D (SIGGRAPH 2017 Poster)

Touch3D is an interactive mobile platform that provides realistic viewing and touching experiences through glasses-free 3D visualization with electrovibration. Touch3D is designed to take advantage of both visual and tactile illusions to maximize multimodal experience in touchscreen interaction. We seamlessly integrate two technologies: Automultiscopic 3D Display and Electrovibration Display; and weave both hardware and software into one fluid interface. Our museum application using Touch3D demonstrates important implications for the improvement of 3D perception in both visual and tactile modalities for enhanced touchscreen interaction. [Paper | Video]

AirPiano (IEEE World Haptics Conference 2017)

AirPiano is a music playing system to provide touchable experiences in HMD-based virtual reality with mid-air haptic feedback. AirPiano allows users to enjoy enriched virtual piano-playing experiences with touchable keys in the air. We implement two haptic rendering schemes to mimic the resisting force of piano keys using ultrasonic vibrations. Constant Feedback is designed to provide short and intense feedback whereas Adaptive Feedback is designed to follow the changes in feedback from a real keypress. [Paper | Video]

Interactive 3D Painting with Haptic Brush in Immersive Room (IEEE Access)

We propose an interactive artwork system with automultiscopic 3D and haptic paint brush in an immersive room. Our system consists of a 81-view automultiscopic display, a handheld haptic paint brush, and a large-scale color palette station in a CAVE-like cubic room filled with the visual scene of the artwork. The 81-view rendering and multiplexing technology is applied by setting up the virtual cameras in the off-axis layout. The haptic paint brush is designed and implemented using a 2D array of multiple piezoelectric actuators. It provides the tactile feedback of spatial distance information between a virtual brush and a distal 3D object displayed on the automultiscopic display for the precise control of the brush when it is interacting with automultiscopic 3D. [Paper]