Real-Time Voice-to-EHR for Surgical Documentation

The Problem

Surgeons are healers, not transcriptionists. Yet the average surgeon spends 2+ hours daily on documentation—dictating notes, reviewing transcriptions, correcting medical terminology errors, mapping to billing codes. It’s a leading cause of physician burnout and delays revenue cycles by days.

The Architecture

flowchart LR
  subgraph capture [Audio Capture]
      Mic[OR Microphone]
      NoiseFilter[Noise Filter]
      VAD[Voice Activity Detection]
  end
  
  subgraph transcription [Transcription Layer]
      Whisper[Whisper v3]
      Diarization[Speaker Diarization]
  end
  
  subgraph extraction [Medical NLP]
      EntityExtractor[Entity Extractor]
      MedPaLM[Med-PaLM 2]
      ICD10[ICD-10 Mapper]
  end
  
  subgraph output [EHR Output]
      StructuredNote[Structured Op Note]
      BillingCodes[Billing Codes]
      Review[Surgeon Review UI]
  end
  
  Mic --> NoiseFilter
  NoiseFilter --> VAD
  VAD --> Whisper
  Whisper --> Diarization
  Diarization --> EntityExtractor
  EntityExtractor --> MedPaLM
  MedPaLM --> ICD10
  MedPaLM --> StructuredNote
  ICD10 --> BillingCodes
  StructuredNote --> Review
  BillingCodes --> Review

Near-Real-Time Medical Entity Extraction

The operating room is a challenging audio environment—monitors beeping, equipment humming, multiple voices. The system handles this through:

1. Noise Filtering & VAD: Isolates the surgeon’s voice from OR ambient noise using specialized audio processing
2. Whisper v3 Transcription: Converts speech to text with medical vocabulary fine-tuning
3. Med-PaLM 2 Extraction: Identifies medical entities—procedures, anatomy, instruments, findings—and structures them into standard operative note format
4. ICD-10 Mapping: Automatically generates billing codes from the structured note

The surgeon reviews and approves via a simple mobile UI before the note hits the EHR.

Tech Stack

• Whisper v3 — Speech-to-text with medical vocabulary
• Med-PaLM 2 — Medical domain language model
• Azure OpenAI (HIPAA) — Compliant inference infrastructure
• React Native — Cross-platform surgeon review app

The Impact

Surgeons now complete documentation before leaving the OR. The system handles the heavy lifting; they just review and approve.