David Hawkins | Product Design

Human-in-the-loop AI review

Optimizing AI interactions with human judgement

Categories:

This is one of four Precedent case studies — the AI-review interaction model. For the system-level view, see Law Firm AI Ecosystem; for the onboarding flow, see AI Ecosystem Onboarding; for the editor’s process story, see AI Document Authoring.

Overview

Bodily injury demand letters are the backbone of personal injury litigation. They synthesize months or years of medical treatment into a persuasive narrative that communicates a client’s injuries, suffering, and financial losses to insurance companies. Before AI, producing these letters was a brutal process.

Law Firm Status Quo

The opportunity was clear: AI could dramatically accelerate this workflow. But the challenge wasn’t just automation,  it was building the right level of human oversight into an AI-powered pipeline so that attorneys could trust the output, catch edge cases, and maintain the professional judgment their clients depend on.

My Role

As the Principal Product Designer, I owned the design and front-end implementation of core workflows.

I worked inside a cross-functional pod: 1 ML engineer, 1 backend engineer, 1 FE engineer, and 2 attorney advisors from partner firms. I owned design of the review workflows; as well as requirements and prioritization; ML engineering owned extraction and confidence tuning. Every workflow below was shaped in a weekly working sessions with all three groups — the division of labor mattered, because the product’s core question (where should human judgment live?) could only be answered with attorneys and ML engineers in the same room.

My responsibilities spanned:

Design Approach: Trust Through Transparency

The central design challenge was this: how do you give legal professionals enough control to trust AI-generated content, without making the review process so burdensome that it erases the efficiency gains?

I framed the product around three principles. These principles weren’t drafted at a desk. Over the course of a month, I facilitated 3 working sessions with attorneys and paralegals: journey-mapping their existing demand process to find where trust broke down and co-reviewing early prototypes against redacted case files. The three principles below are the distilled output of those sessions — each one traces back to something a practitioner told us.

1. Surface the Right Decisions, Not All the Data

The ETL pipeline processes enormous volumes of medical records. Rather than exposing every extracted data point for review, I designed the system to surface only the items that require human judgment, flagged treatments that the AI identified as potentially problematic.

This meant designing a triage-first experience: the system does the heavy lifting of extraction and organization, and the attorney’s attention is directed to the decisions only they can make.

2. Make AI Output Reviewable, Not Just Readable

There’s a meaningful difference between presenting a wall of generated text and designing an interface that supports active review. I focused on making AI output decomposable, breaking narratives into discrete, reviewable sections tied to specific evidence, so attorneys could evaluate claims against source material rather than reading prose on faith.

3. Keep the Human in Command

Every AI-generated output is a draft, not a deliverable. The workflows I designed ensure that attorneys can accept, reject, edit, or regenerate any piece of content with custom instructions. The AI proposes; the human disposes.

The Workflow

Before the step-by-step breakdown, here is the flow in one view. Shaded stages run fully automated; highlighted stages are the deliberate human decision points. The entire product thesis is visible in where those decision points sit — early enough to catch problems, late enough that attorneys never do work the machine should have done.

Step 1: Document Upload & Processing

Attorneys or paralegals upload medical records, bills, and supporting documentation. The system ingests, parses, and extracts structured data — treatment dates, providers, diagnoses, procedures, imaging findings, and billing information

Step 2: Flagged Treatment Review

This is where the human-in-the-loop model earns its value. The system automatically flags treatments that need attorney review:

I designed the review interface around rapid decision-making. Attorneys see a filterable, sortable table of flagged items with contextual detail, enough information to make a judgment call without switching to the source document. This is recognition over recall applied to legal review: the interface carries the context, so the attorney’s memory doesn’t have to.

Design decisions that mattered here:

Step 3: Narrative Generation

Using RAG against the processed medical records and resolved treatment data, the system generates narrative sections for the demand letter:

Step 4: Narrative Review & Refinement

This is the most interaction-rich part of the workflow, and where the most design iteration happened. Attorneys need to review AI-generated prose with the same critical eye they’d apply to a junior associate’s draft.

I designed the review experience around three modes of interaction:

Read and accept. For narratives that are accurate and well-written, a simple approval flow. No friction added where none is needed.

Direct edit. For targeted changes — a word choice, a factual correction, a tone adjustment — attorneys can edit the narrative text directly in a rich text editor.

Instruct and regenerate. For narratives that need more substantial rework, attorneys can provide natural language instructions (e.g., “Emphasize the chronic nature of the lumbar injury” or “Remove references to the ER visit on 3/15”) and the system regenerates the section accordingly

Design decisions that mattered here:

Step 5: Narrative Context Management

For complex cases, attorneys may need to adjust the context the AI uses for generation — adding case-specific details, excluding certain records, or emphasizing particular aspects of the injury.

I designed a context management interface that gives attorneys control over the RAG pipeline’s inputs without requiring them to understand the underlying technology. They work with familiar legal concepts (providers, date ranges, document types) rather than technical abstractions.

Outcomes

Business Impact

User Impact

Design Impact