[Part 1] Basic LLMOps Flow

▣ Chapter 1: LLMOps Flow
1.1 Background: The iterative process of creating an LLM application
1.2 Why LLMOps is Needed
1.3 Differences between MLOps and LLMOps
__1.3.1 Model Complexity and Scale
__1.3.2 Data Management and Processing
__1.3.3 Customization and Optimization
__1.3.4 Monitoring
1.4 LLMOps Workflow

▣ Chapter 2: Background Knowledge for LLM Application Development
2.1 LLM model selection
__2.1.1 Commercial Closed Model
__2.1.2 Open Source Model
__2.1.3 Differences between closed and open source models
__2.1.4 Selecting a model based on business requirements
2.2 Adjusting LLM parameters according to application type
2.3 Elements of the prompt
__2.3.1 Components of the prompt
__2.3.2 Prompt Role
__2.3.3 Prompt templating

▣ Chapter 3: Developing a Practical Customer Inquiry Classification Application
3.1 Customer Inquiry Classification Application Overview
3.2 Language Model Selection
__3.2.1 Antropic's Messages API
__3.2.2 Open-source model access using Olama
3.3 Langchain Overview
__3.3.1 Why use Langchain?
__3.3.2 Installing Langchain and Building Example Applications
3.4 Langchain Basics
__3.4.1 Prompt Template
__3.4.2 Chat Model
__3.4.3 Output Parser
__3.4.4 LCEL
3.5 Developing a Practical Application Using LangChain
__3.5.1 Model Definition
__3.5.2 Defining the Output Parser
__3.5.3 Application Chain Development

▣ Chapter 4: LLMOps Tool Development
4.1 The Need for LLMOps Tools
__4.1.1 Prompt Versioning
__4.1.2 Managing Datasets for Evaluation
__4.1.3 Performance Evaluation Criteria and Methods
4.2 Essential features and components of the LLMOps tool
__4.2.1 Test function
__4.2.2 Prompt Management and Versioning Features
__4.2.3 Evaluation
__4.2.4 Dataset Management Function
4.3 Configuring the Development Environment
__4.3.1 Streamlet for creating easy Python-based web apps
__4.3.2 SQLite for storing data
4.4 LLM Test Function
__4.4.1 Why Testing UI is Needed
__4.4.2 Implementing a model management class supported by the test function
__4.4.3 Creating a test menu using Streamlet
4.5 Prompt Management and Versioning
__4.5.1 Why Prompt Management is Needed
__4.5.2 Table design for prompt management
__4.5.3 Implementing a class for prompt management
__4.5.4 Streamlet Test Menu Improvements: Support for Saving Prompts and Versioning
__4.5.5 Implementing a class to manage specific prompt templates
4.6 Evaluation Criteria
__4.6.1 Various evaluation indicators
__4.6.2 Implementing an evaluator class that generates evaluation metrics
__4.6.3 Creating and Using Evaluators
4.7 Dataset
__4.7.1 Table design for dataset management
__4.7.2 Implementing the dataset storage class
__4.7.3 Implementing a class for managing a single dataset
__4.7.4 Creating a dataset management menu using Streamlet
4.8 Dataset Evaluation
__4.8.1 Table design for dataset evaluation
__4.8.2 Implementing the dataset evaluation class
__4.8.3 Creating an evaluation execution menu using Streamlet
__4.8.4 Creating a menu to view evaluation results using Streamlet
4.9 LLMOps Menu Configuration

▣ Chapter 5: Managing LLM Applications Using LLMOps Tools
5.1 Design and create a draft prompt
5.2 Building the Dataset
5.3 Evaluation Progress
5.4 Added a new version of the prompt
5.5 Compare evaluation metrics by version to make a decision.
5.6 Answer the questions based on the evaluation results

[Part 2] LLMOps Flow for RAG

▣ Chapter 6: Practical RAG-Based Insurance Chatbot Application
6.1 Overview of the Insurance Inquiry Chatbot Application
6.2 General Augmented Search Generation (RAG) Workflow
__6.2.1 Document Indexing Process
__6.2.2 Answer Generation Process
__6.2.3 Further RAG Paradigm
6.3 Using the Vector Database Pinecone
__6.3.1 Practice: Finding Similar Documents Using Pinecone
6.4 Reading PDF files and indexing them into a vector database
__6.4.1 Document Chunking
__6.4.2 Document Chunk Vectorization
__6.4.3 Index
6.5 Find documents most similar to the entered question
__6.5.1 Searching for candidate documents using the embedding model (10 documents)
__6.5.2 Final filtering with re-ranking model (3)
6.6 Developing a Practice Application Chain
__6.6.1 Search: Find similar documents
__6.6.2 Generate: Generate answers based on documents

▣ Chapter 7: Development of LLMOps Tools for RAG
7.1 Tool functions for the RAG system
7.2 RAG Evaluator Implementation
__7.2.1 Understanding RAG Evaluation Metrics
__7.2.2 Lagas supporting RAG evaluation metrics
__7.2.3 Implementing an evaluator for RAG
__7.2.4 Tool Integration: Dynamic Evaluator Support
__7.2.5 Tool Integration: RAG Evaluation Support
__7.2.6 Added Streamlet Evaluation menu function
7.3 Synthetic dataset creation function
__7.3.1 RAG Question Types
__7.3.2 Knowledge Graph-Based Test Set Generation Pipeline
__7.3.3 Practice: Generating a Synthetic Test Set Based on PDF Documents
__7.3.4 Implementing a synthetic dataset class based on PDF documents

▣ Chapter 8: Managing LLM Applications Using LLMOps Tools
8.1 Creating a Prompt
8.2 Creating and Saving Synthetic Datasets
8.3 Evaluation Progress
8.4 Analysis of Evaluation Results
__8.4.1 Token Usage and Latency Analysis
__8.4.2 Analysis of context precision and reliability metrics
8.5 Answer questions based on evaluation results

[Part 3] Continuous Improvement

▣ Chapter 9: Ongoing Management of LLM Applications
9.1 Monitoring
__9.1.1 Tracking Model Usage
__9.1.2 Collecting Performance Metrics
__9.1.3 Detecting Prompt Injection Attacks
9.2 Resource Management: Cost Reduction and Model Lightness
__9.2.1 Caching
__9.2.2 Model Lightweighting: Quantization
__9.2.3 Model Lightening: Knowledge Distillation
9.3 Market Direction Driven by Deep Seek
__9.3.1 Development of low-cost, high-performance AI models
__9.3.2 Open Source Strategy
__9.3.3 Introduction of efficient learning techniques

▣ Chapter 10: Continuous Improvement of LLMOps Tools
10.1 Chaining, Agent Support, and Monitoring
__10.1.1 Chaining
__10.1.2 Agent
10.2 Model Deployment Process Support
__10.2.1 Model Training
__10.2.2 Model Serving

★ What this book covers ★

◎ Understanding the LLMOps flow for iterative development of LLM applications
◎ Selection criteria and prompt components for closed/open source LLM models
◎ Development of a customer inquiry classification application using Langchain
◎ Development of an LLMOps tool capable of prompt version management, testing, and evaluation.
◎ Streamlet-based prompt/dataset/metric management UI configuration
◎ Implementation of a RAG-based chatbot using Finecon and document chunking
◎ Creation of a synthetic dataset and a Lagas evaluator for measuring RAG system performance
◎ Integration of continuous improvement strategies and tools, such as caching, model lightweighting, and monitoring.