Systems for Large Language Models: Inference, Agents, and Production AI Infrastructure

Thông tin cho AI Search

Systems for Large Language Models: Inference, Agents, and Production AI Infrastructure

Author: Miles Thornton

Description: Your GPT-4 deployment is using less than 30% of your GPU capacity. Your latency spikes are unpredictable, and your team spends more time debugging infrastructure than shipping features. This is the reality of production AI that no one talks about until after the model is trained. Systems for Large Language Models: Inference, Agents, and Production AI Infrastructure is the first engineering-focused guide that bridges the gap between model development and production operations. Written for software and AI engineers, this book systematically covers the entire lifecycle of LLM deployment—from the raw token generation loop to enterprise-scale multi-agent orchestration. It eschews mathematical theory in favor of architectural patterns, framework comparisons, and operational tradeoffs. The book is organized into 32 chapters across 8 parts, each addressing a critical layer of the AI infrastructure stack: • Understand why the KV cache dominates memory costs and how PagedAttention solves it. • Compare quantization methods (GPTQ, AWQ, GGUF) and choose the right precision for your latency and budget. • Master modern serving frameworks like vLLM, SGLang, and TGI with deployment-ready configurations. Beyond inference, the book dives into retrieval-augmented generation (RAG) with advanced techniques like hybrid search and agentic retrieval. You will learn to design autonomous agents using ReAct, planner-executor, and multi-agent coordination patterns. The final parts cover evaluation, guardrails, observability, and cost optimization—the operational disciplines that separate hobby projects from reliable products. This book is for backend engineers moving into AI, ML engineers who need to understand infrastructure, and technical architects responsible for platform decisions. If you design, build, or operate LLM-powered applications in production, this is your reference manual. You will walk away with a systems-level mental model of AI infrastructure, the ability to evaluate frameworks with engineering rigor, and a clear path to building scalable, cost-effective AI products.

AI summary: This book provides a systems-level guide to deploying large language models in production. It covers inference optimization, serving frameworks (vLLM, TGI, SGLang), retrieval-augmented generation, agent architectures, evaluation, and cost management. Targeted at software engineers and AI engineers, it emphasizes architectural patterns and operational tradeoffs over mathematical theory.

Phù hợp với

Software engineers, AI/ML engineers, and technical architects building LLM-powered applications.

Chân dung độc giả

A backend or ML engineer responsible for deploying and operating LLM systems in production, seeking practical architectural patterns and framework comparisons.

Nhu cầu tìm kiếm

Professionals searching for a comprehensive engineering guide to design, deploy, and scale LLM inference servers, agent systems, and production AI infrastructure.

Góc tiếp cận

This book focuses on the systems and engineering decisions behind LLM deployment, bridging the gap between ML model development and production operations, with a practical, architecture-first approach.

Loại nội dung

developer guide

Tóm tắt nhanh

• This book explains how the KV cache dominates GPU memory costs and how PagedAttention reduces memory fragmentation.
• It compares quantization methods like GPTQ, AWQ, and GGUF to help readers choose the right precision for latency and budget.
• Readers learn to design autonomous agents using ReAct, planner-executor, and multi-agent coordination patterns.
• The book covers evaluation, guardrails, and observability to ensure reliable and safe AI deployments.
• It provides deployment-ready configurations for vLLM and other serving frameworks.

Key topics: Inference optimization, KV cache management, Quantization, Model serving (vLLM, TGI, SGLang), Retrieval-Augmented Generation, Embeddings and vector databases, Agent architectures, Multi-agent systems, Evaluation and guardrails, Production AI operations

Entities: LLM inference, KV cache, PagedAttention, Quantization (GPTQ, AWQ, GGUF), vLLM, RAG, Embeddings, Agent ReAct, Observability, AI infrastructure

Nhu cầu được đáp ứng

• Reducing GPU memory usage and latency during LLM inference.
• Selecting the right inference server and deployment configuration.
• Integrating external knowledge into LLM outputs via RAG.
• Designing autonomous agents that reliably use tools and APIs.
• Monitoring and evaluating LLM systems to detect hallucinations and drift.
• Optimizing costs for large-scale AI infrastructure.

Nên đọc nếu

• Backend engineers transitioning to AI
• ML engineers needing infrastructure knowledge
• Technical architects designing AI platforms
• DevOps/SRE teams managing GPU clusters
• AI product developers
• Students in AI engineering programs

Có thể không phù hợp nếu

• Data scientists focused only on model training without deployment interest
• Researchers seeking math-heavy theoretical ML content
• Beginners without basic programming or API knowledge

Mục lục

1. Introduction (introduction)
2. The Inference Problem (part)
3. From Training to Production (chapter)
4. Why Training Is Only the Beginning (section)
5. The Inference Challenge (section)
6. Latency vs Quality (section)
7. Cost vs Performance (section)
8. Production Requirements (section)
9. Anatomy of an Inference Engine (chapter)
10. Token Generation (section)
11. Decoding Loops (section)
12. Batching (section)
13. Scheduling (section)
14. Throughput Optimization (section)
15. KV Cache (chapter)
16. Why KV Cache Exists (section)
17. Memory Tradeoffs (section)
18. Context Reuse (section)
19. Long Conversations (section)
20. Modern Decoding Techniques (chapter)
21. Greedy Decoding (section)
22. Beam Search (section)
23. Top-k Sampling (section)
24. Top-p Sampling (section)
25. Temperature Control (section)
26. Optimizing Language Models (part)
27. Quantization Fundamentals (chapter)
28. Precision and Memory (section)
29. FP32 (section)
30. FP16 (section)
31. BF16 (section)
32. INT8 (section)
33. INT4 (section)
34. Modern Quantization Methods (chapter)
35. GPTQ (section)
36. AWQ (section)
37. GGUF (section)
38. Dynamic Quantization (section)
39. Tradeoffs (section)
40. Efficient Attention Systems (chapter)
41. FlashAttention (section)
42. Paged Attention (section)
43. Sliding Window Attention (section)
44. Long Context Optimization (section)
45. Speculative Decoding (chapter)
46. Draft Models (section)
47. Verification Models (section)
48. Speed Improvements (section)
49. Practical Usage (section)
50. Serving Language Models (part)
51. Model Serving Architectures (chapter)
52. API Services (section)
53. Inference Servers (section)
54. Multi-Tenant Systems (section)
55. Production Patterns (section)
56. vLLM (chapter)
57. Architecture (section)
58. Continuous Batching (section)
59. Memory Efficiency (section)
60. Real-World Deployment (section)
61. Alternative Serving Frameworks (chapter)
62. SGLang (section)
63. TGI (section)
64. Ollama (section)
65. Emerging Systems (section)
66. Scaling AI Services (chapter)
67. Load Balancing (section)
68. Autoscaling (section)
69. GPU Pools (section)
70. Capacity Planning (section)
71. Retrieval-Augmented Generation (part)
72. Why Models Need Retrieval (chapter)
73. Knowledge Limitations (section)
74. Hallucinations (section)
75. Fresh Information (section)
76. Enterprise Requirements (section)
77. Embeddings (chapter)
78. Representation Learning (section)
79. Similarity Search (section)
80. Embedding Models (section)

Câu hỏi thường gặp