Vision-Guided Agents for Insurance Data Extraction

The Problem

Insurance underwriting requires verification—lots of it. An InsurTech startup’s underwriters were spending hours each day manually navigating government portals: DMV records, property databases, licensing boards. Each portal had different UIs, different login flows, different data formats. Traditional Selenium scripts broke constantly as these legacy sites updated.

The Architecture

flowchart TB
  subgraph request [Quote Request]
      Application[Policy Application]
      DataNeeds[Verification Checklist]
  end
  
  subgraph orchestration [Agent Orchestration]
      Router[Portal Router]
      Navigator[Vision Navigator]
      Extractor[Data Extractor]
  end
  
  subgraph browser [Browser Layer]
      Playwright[Playwright Instance]
      Screenshot[Screenshot Capture]
      DOM[DOM State]
  end
  
  subgraph inference [Local Inference]
      Llama[Llama 3.1 8B]
      Vision[Vision Analysis]
  end
  
  subgraph output [Verified Data]
      Structured[Structured Output]
      Confidence[Confidence Scores]
  end
  
  Application --> Router
  DataNeeds --> Router
  Router --> Navigator
  Navigator --> Playwright
  Playwright --> Screenshot
  Screenshot --> Vision
  Vision --> Llama
  Llama --> Navigator
  Playwright --> DOM
  DOM --> Extractor
  Extractor --> Structured
  Extractor --> Confidence

Vision-Guided Navigation

The breakthrough: agents that see the page like humans do.

1. Portal Router: Determines which government portals need to be accessed based on the policy application
2. Vision Navigator: Takes screenshots of each page state and uses LLM reasoning to determine next actions—no CSS selectors, no XPath, just “click the blue ‘Search’ button”
3. Data Extractor: Parses the final result pages using Pydantic models for structured output

When a portal redesigns their UI, the agent adapts. No code changes required.

Tech Stack

• Playwright — Headless browser automation
• PydanticAI — Structured data extraction with validation
• Llama 3.1 8B — Local inference for privacy (policyholder data never leaves the VPC)
• AWS Lambda — Serverless execution at scale

The Impact

The system handles portal updates gracefully—a redesigned DMV site that would have broken Selenium scripts for days is navigated correctly on the first attempt.