01
Human-in-the-loop beats raw accuracy
Side-by-side verification reduces errors more than chasing a smarter model.
FIELD NOTES & PRINCIPLES
Architectural Insights
Pragmatic lessons and engineering principles from shipping production AI systems in regulated environments.
Related case study
02
Treat knowledge like code
Staging, promotion, one-click rollback, and audit trails for every document.
03
Measure before you believe
Run custom multi-run evaluation suites with at least three runs per question to counter LLM output variance, and guard the evaluation set against biased curation.
04
Refusing is a feature
In regulated industries, refusing is safer than hallucinating.
05
Defaults are silent bugs
Many retrieval problems come from default settings, not from the model.
06
Different documents, different pipelines
Documents full of screenshots and text-dominant documents need different parsing strategies.
07
Keep AI orchestration separate from the web app
The team that maintains the AI pipeline does not need to touch the web app.
Engineering Note
08
Size infrastructure with benchmarks, not vendor slides
Benchmark models on your own GPUs before sizing the cluster.
Engineering Note
09
AI agents need standards too
Written standards, memory, and permission limits let AI coding agents work safely in a shared codebase.
Engineering Note