Selected projects in speech and audio, generative modeling, and embedded machine learning. Each project includes its approach, findings, and supporting materials.
Intelligent Information Processing Lab (IIP), Sogang University
Adapted Voxtral Mini 3B for Korean medical speech recognition and spoken question answering with 4-bit QLoRA and two-stage text/audio fine-tuning.
Results & limitations
Medical ASR CER 19.14% → 9.09%; spoken QA 2.07 → 4.01/5, judged by GPT-OSS. Both comparisons use the Voxtral 3B 4-bit baseline in the project evaluation.
Full size ↗Two-stage training: Stage 1 on medical text data, Stage 2 on general and clinical speech data, with QLoRA on Voxtral Mini 3B.
Methods & setup
Built medical text and spoken QA datasets through data cleaning, question generation, TTS synthesis, and CER-based filtering.
Compared RAG, LLM-only fine-tuning, and two-stage audio-adapter/LLM fine-tuning; analyzed ASR error propagation and transcription forgetting.
More figures (3)
Full size ↗Results: QLoRA fine-tuning reduces medical ASR CER from 19.14% to 9.09%, and raises the GPT-OSS-judged spoken QA score from 2.07 to 4.01 out of 5.Full size ↗Portfolio page: project overview, problem, role, and two-stage training pipeline.Full size ↗Portfolio page: results, acquired skills, and evaluation tables.
Audio-to-Audio Speech Unit Translation (Korean ↔ English)
Sep. 2024 – Present
ECE Capstone Design, Inha University
Exploring Korean–English speech-to-speech translation through discrete speech units, without an intermediate text representation at inference time.
Results & limitations
Generated English speech scored BLEU 42.8 / COMET 0.1009 in the project evaluation. Output was intelligible but had unstable pitch and background noise.
Full size ↗Textless speech-to-speech translation: HuBERT audio encoder, Transformer unit translator, and unit vocoder, extended from English and Chinese to Korean.
Methods & setup
Textless speech-to-speech pipeline: mHuBERT with a k-means quantizer (500 units) → Transformer encoder–decoder with language tags → HiFi-GAN unit vocoder conditioned on a speaker d-vector.
Explored Korean (SOV) and English (SVO) word-order differences by reordering training transcripts with Llama 3.1 8B Instruct before resynthesis.
Full size ↗Portfolio page: project overview, problem, role, model structure, and implementation steps.Full size ↗Portfolio page: results, acquired skills, model inputs and outputs, and evaluation.
Other projects
Scene Graph to Video Generation with Diffusion
Aug. 2024 – Dec. 2024
Machine Intelligence Lab, Inha University
Explored controllable video generation with scene graphs as structured conditions for a diffusion model.
Results & limitations
Implemented the full pipeline. Generation quality remained limited by scarce scene-graph–video paired data (Action Genome) and weak graph–video alignment.
Full size ↗From scene-graph-conditioned images to video generation with temporal graph conditions. Project portfolio, Figure 9.
Methods & setup
R-GCN scene-graph embedding aligned to a CLIP image encoder by contrastive learning (SGClip), conditioning a latent diffusion model through cross-attention with a time-extended U-Net.
Implemented an autoregressive long-video module that injects noise only into the first frame.
More figures (2)
Full size ↗Scene-graph and video-frame embedding alignment with contrastive learning. Implementation notes.Full size ↗Autoregressive video generation conditioned on the previous frame and scene graph. Implementation notes.
Water Quality Prediction: Embedded Device and Sensor Analysis
Mar. 2025 – Jun. 2025
R&D Department, STS Engineering
Developed soft-sensor models to estimate water quality indicators from low-cost IoT sensor time series, alongside embedded firmware for continuous monitoring.
Results & limitations
Delivered a monitoring prototype to the client. Reported project prediction error (NMAE) decreased from 15% to 8%.
Full size ↗Water quality monitoring architecture linking sensors, an MCU, LTE, the IoT server, and a dashboard. Project portfolio, Figure 12.
Methods & setup
Built FreeRTOS firmware for an IoT device with water quality sensors, RS485 serial, and LTE, focusing on fault handling and reboot recovery.
Used an ensemble to separate trend prediction from noise modeling in irregular sensor time series.
More figures (4)
Full size ↗Field installation of the water quality sensors at the treatment plant.Full size ↗Sensor enclosure with signal converters, power supply, and LTE wiring.Full size ↗Long-term trend forecast with Prophet.Full size ↗Estimated and measured values from the residual correction model.
Explored question answering over Korean economic articles through language-model fine-tuning.
Results & limitations
LoRA adaptation degraded the already well-aligned base model in project experiments; the approach shifted to minimal fine-tuning to preserve base behavior.
Global Wildfire Detection Challenge
Mar. 2024
6th AI SPARK Challenge
Built a satellite-image segmentation system for wildfire detection using an ensemble of TransUNet and Attention U-Net.