From AI Model to AI Product

Evaluation-driven development of LLM-based products

Author

Vitaly Rubinovich

Publisher version
0.8.23
Released

An engineer adjusts a luminous model-like instrument on a workbench surrounded by reference books, selected pages, a magnifying lens, and precision tools.

Inherently, a trained large language model (LLM) is data, mainly its learned weights, that can be stored in one or more files. Software libraries such as PyTorch load and run the model. The surrounding application sends requests, receives answers, connects to external tools and systems, interacts with other agents, and supports diagnostics.

This book explains how to build and evaluate the software that interacts with an LLM. It covers evaluation-driven development (EDD), which uses test results to guide improvements, alongside prompt design, retrieval-augmented generation (RAG), workflows, and agents.

Other books in the series cover the mathematics behind models, distributed training and inference systems, model post-training, hardware and software optimization, and security relevant to AI.

Contents

Part I - The language-model interface

Part II - Evaluation-based system improvement

Part III - RAG: Supplying external evidence

Part IV - Multi-step processes and tool use

Part V - System cases, diagnosis, and release decisions

Appendices