Selected work

Research & projects

Selected projects in speech and audio, generative modeling, and embedded machine learning. Each project includes its approach, findings, and supporting materials.

Speech & audio

Medical Speech Recognition and Spoken QA

Jul. 2025 – Feb. 2026

Intelligent Information Processing Lab (IIP), Sogang University

Adapted Voxtral Mini 3B for Korean medical speech recognition and spoken question answering with 4-bit QLoRA and two-stage text/audio fine-tuning.

Results & limitations

Medical ASR CER 19.14% → 9.09%; spoken QA 2.07 → 4.01/5, judged by GPT-OSS. Both comparisons use the Voxtral 3B 4-bit baseline in the project evaluation.

Two-stage training: Stage 1 on medical text data, Stage 2 on general and clinical speech data, with QLoRA on Voxtral Mini 3B.
Two-stage training: Stage 1 on medical text data, Stage 2 on general and clinical speech data, with QLoRA on Voxtral Mini 3B.
Methods & setup
  • Built medical text and spoken QA datasets through data cleaning, question generation, TTS synthesis, and CER-based filtering.
  • Compared RAG, LLM-only fine-tuning, and two-stage audio-adapter/LLM fine-tuning; analyzed ASR error propagation and transcription forgetting.

Audio-to-Audio Speech Unit Translation (Korean ↔ English)

Sep. 2024 – Present

ECE Capstone Design, Inha University

Exploring Korean–English speech-to-speech translation through discrete speech units, without an intermediate text representation at inference time.

Results & limitations

Generated English speech scored BLEU 42.8 / COMET 0.1009 in the project evaluation. Output was intelligible but had unstable pitch and background noise.

Textless speech-to-speech translation: HuBERT audio encoder, Transformer unit translator, and unit vocoder, extended from English and Chinese to Korean.
Textless speech-to-speech translation: HuBERT audio encoder, Transformer unit translator, and unit vocoder, extended from English and Chinese to Korean.
Methods & setup
  • Textless speech-to-speech pipeline: mHuBERT with a k-means quantizer (500 units) → Transformer encoder–decoder with language tags → HiFi-GAN unit vocoder conditioned on a speaker d-vector.
  • Explored Korean (SOV) and English (SVO) word-order differences by reordering training transcripts with Llama 3.1 8B Instruct before resynthesis.
  • Setup: Multilingual AIhub + LibriSpeech, Fairseq / PyTorch, NVIDIA A6000 48GB.

Other projects

Scene Graph to Video Generation with Diffusion

Aug. 2024 – Dec. 2024

Machine Intelligence Lab, Inha University

Explored controllable video generation with scene graphs as structured conditions for a diffusion model.

Results & limitations

Implemented the full pipeline. Generation quality remained limited by scarce scene-graph–video paired data (Action Genome) and weak graph–video alignment.

From scene-graph-conditioned images to video generation with temporal graph conditions. Project portfolio, Figure 9.
From scene-graph-conditioned images to video generation with temporal graph conditions. Project portfolio, Figure 9.
Methods & setup
  • R-GCN scene-graph embedding aligned to a CLIP image encoder by contrastive learning (SGClip), conditioning a latent diffusion model through cross-attention with a time-extended U-Net.
  • Implemented an autoregressive long-video module that injects noise only into the first frame.

Water Quality Prediction: Embedded Device and Sensor Analysis

Mar. 2025 – Jun. 2025

R&D Department, STS Engineering

Developed soft-sensor models to estimate water quality indicators from low-cost IoT sensor time series, alongside embedded firmware for continuous monitoring.

Results & limitations

Delivered a monitoring prototype to the client. Reported project prediction error (NMAE) decreased from 15% to 8%.

Water quality monitoring architecture linking sensors, an MCU, LTE, the IoT server, and a dashboard. Project portfolio, Figure 12.
Water quality monitoring architecture linking sensors, an MCU, LTE, the IoT server, and a dashboard. Project portfolio, Figure 12.
Methods & setup
  • Built FreeRTOS firmware for an IoT device with water quality sensors, RS485 serial, and LTE, focusing on fault handling and reboot recovery.
  • Used an ensemble to separate trend prediction from noise modeling in irregular sensor time series.

Competitions

Korean LLM Fine-tuning for Question Answering

Jul. 2024 – Aug. 2024

2024 Inha Artificial Intelligence Challenge (Dacon)

Explored question answering over Korean economic articles through language-model fine-tuning.

Results & limitations

LoRA adaptation degraded the already well-aligned base model in project experiments; the approach shifted to minimal fine-tuning to preserve base behavior.

Global Wildfire Detection Challenge

Mar. 2024

6th AI SPARK Challenge

Built a satellite-image segmentation system for wildfire detection using an ensemble of TransUNet and Attention U-Net.