Reson
Acoustic Signal Processing & Machine Learning Gesture System
“How do you detect physical gestures using sound waves?”
Case Study Index (5 Sections)↓
Research Problem & Motivation
Human-computer interfaces traditionally depend on physical touch, cameras, or specialized hardware. Reson investigates how standard audio hardware (speakers and microphones) can serve as an active sonar sensor for physical gesture recognition.
I was fascinated by the idea of touchless control using existing hardware without requiring cameras or dedicated depth sensors. Exploring audio signal processing offered a challenging intersection of physics, digital signal processing, and ML.
Signal & Data Pipeline
Audio Generation (18-22kHz continuous tone) -> Microphone Hardware Capture -> Bandpass Noise Filtering -> STFT Spectrogram Matrix -> Doppler Shift Extractor -> PyTorch Classifier -> Gesture Event Broadcast.
Experiments & Findings
Tested signal accuracy across varying hand distances (10cm - 50cm) and ambient noise levels, achieving reliable classification for swipe and push gestures in quiet room settings.
Engineering Decisions
Large FFT window sizes improve frequency resolution but introduce latency, breaking real-time gesture feedback.
Selected a 1024-sample FFT window with 75% overlap, balancing frequency resolution with sub-30ms temporal latency.
Slight loss in fine-grained frequency resolution.
Achieved responsive gesture classification speeds suitable for interactive software.
Challenges & Solutions
Ambient Acoustic Noise Cancellation
Issue: Background room audio and speaker hardware distortion polluted ultrasonic frequencies.
Decision/Solution: Applied bandpass filtering and dynamic ambient baseline subtraction to isolate intentional gesture Doppler shifts.
Future Research Directions
Key Takeaways
Digital signal processing requires careful balance between time resolution and frequency resolution.
Commodity hardware can act as novel sensor surfaces when coupled with proper signal filtering.
Related Engineering Projects
Reson Collector
Built a dedicated Python dataset recorder to capture, label, and sanitize audio samples for training the Reson gesture detection models.
MemoryOS
Built an AI semantic notes system during SRM Builds 7.0 that converts markdown notes into vector embeddings for instant contextual retrieval.