Andrew LinhAL
Back to work

2025 · Architecture lead

AgentOps Runner

Deterministic evaluation harness for AI agent workflows

Abstract neural network visualization on a dark display

Context

An AI product team needed reproducible evals for multi-step agents before promoting prompts and tools to production.

Problem

Ad-hoc notebook evals produced non-comparable scores. Regressions slipped into production twice a month.

Architecture

Built a runner with frozen fixtures, tool stubs, deterministic seeding, and structured trace artifacts with typed scorecards.

Result

Eval variance across identical runs dropped below 1%. Production agent regressions fell from ~8/month to 1/month.

Stack

PythonTypeScriptRedisOpenTelemetryDocker