To begin with

Chapter 1: Basic Concepts of Reinforcement Learning
_1.1 What is reinforcement learning?
_1.2 Probability and Stochastic Processes
__1.2.1 Probability
__1.2.2 Conditional probability
__1.2.3 Stochastic Processes
_1.3.
Markov chain
__1.3.1 Markov property
__1.3.2 Markov property and Brownian motion
__1.3.3 Markov chain
_1.4 Markov Reward Process
__1.4.1 Composition of Markov Rewards
__1.4.2 What is the expected value of probability?
__1.4.3 Return value
__1.4.4 State Value Function

Chapter 2: Basic Reinforcement Learning Algorithms
_2.1 What is a Markov decision process?
_2.2 MDP Components
__2.2.1 State transition matrix and reward function in MDP
__2.2.2 Policy in MDP
__2.2.3 Comparison Case of MRP and MDP
__2.2.4 State Transition Matrix and Reward Function Considering Policy
_2.3 MDP State Value Function
__2.3.1 What is the MDP state value function?
__2.3.2 MDP State Value Function Example
_2.4 MDP Action Value Function
__2.4.1 What is an MDP action value function?
__2.4.2 Relationship between MDP action value function and state value function
__2.4.3 MDP Action Value Function Example
_2.5 MDP optimal value function
__2.5.1 What is the MDP optimal value function?
__2.5.2 MDP Optimal Value Function Example
_2.6 Various terms used in reinforcement learning
__2.6.1 Policy Evaluation and Policy Control
__2.6.2 Model-based and model-free

Chapter 3: Dynamic Programming and Monte Carlo Methods
_3.1.
dynamic programming
__3.1.1.
What is dynamic programming?
__3.1.2.
Grid World
__3.1.3 Dynamic Programming Example
_3.2 Monte Carlo method
__3.2.1 What is the Monte Carlo method?
__3.2.2 Monte Carlo method using incremental averaging
__3.2.3 Monte Carlo method example

Chapter 4: Time-Differential Learning, Salsa, and Q-Learning
_4.1 Time-lapse learning
__4.1.1 What is temporal difference learning?
__4.1.2 Action value function rather than state value function
__4.1.3 Time-Differential Learning Example
_4.2 Salsa
__4.2.1 SARSA Concept
__4.2.2 Salsa Example
_4.3 Q Running
__4.3.1 On-policy and off-policy
__4.3.2 Importance Sampling
__4.3.3 Q Running
__4.3.4 Q-Learning Example
__4.3.4 Comparison of Salsa and Q-Learning Examples

Chapter 5: Artificial Intelligence Concepts
_5.1 Machine Learning
_5.2 Linear Regression Analysis
_5.3 Classification Analysis
_5.4 Deep Learning
_5.5 Learning the Basics of Artificial Intelligence with Programs
__5.5.1 What is TensorFlow?
__5.5.2 Basic Artificial Neural Network Example

Chapter 6 Function Approximation
_6.1 Differentiation
_6.2 Partial differentiation
_6.3 Scalars and Vectors
_6.4 Gradient
_6.5 Gradient descent
_6.6 Stochastic Gradient Descent
_6.7 Notation for Partial Differentiation and Gradient Descent in Reinforcement Learning
_6.8 Function Approximation

Chapter 7: Value-Based Reinforcement Learning and the DQN Algorithm
_7.1 DQN Algorithm
_7.2 Cartpole
_7.3 The Problem of Exploration and Greed
_7.4 Basic Structure of the DQN Algorithm
_7.5 Full DQN Algorithm Code Review
_7.6 Detailed Structure of the DQN Algorithm
_7.7 Analysis of DQN Algorithm Learning Results

Chapter 8 Policy-Based Reinforcement Learning REINFORCE Algorithm
_8.1 Revisiting Artificial Neural Networks
_8.2 Policy Gradient
_8.3 How the REINFORCE Algorithm Works
_8.4 Basic Structure of the REINFORCE Algorithm
_8.5 REINFORCE Algorithm Full Code Review
_8.6 Detailed Structure of the REINFORCE Algorithm
_8.7 Analysis of REINFORCE Algorithm Learning Results

Chapter 9 Policy-Based A2C Algorithm
_9.1 Actor Critic Algorithm
__9.1.1 What is the Actor Critic Algorithm?
__9.1.2 Actor Critic Algorithm Structure and Operation
_9.2 Advantage Actor Critic
_9.3 Basic Structure of the A2C Algorithm
_9.4 A2C Algorithm Full Code Review
_9.5 A detailed look at the A2C algorithm structure
_9.6 Analysis of A2C Algorithm Learning Results
Chapter 10 Policy-Based PPO Algorithm

_10.1 Importance Sampling
_10.2 On-policy policy gradient
_10.3 Clipping Technique
_10.4 GAE
_10.5 Basic Structure of the PPO Algorithm
_10.6 Full Code Review of the PPO Algorithm
_10.7 Detailed Structure of the PPO Algorithm
_10.8 PPO Algorithm Algorithm Learning Results Analysis

Chapter 11: Tuning Artificial Neural Networks
_11.1 Overview of Artificial Neural Network Tuning
_11.2 Input data preprocessing
__11.2.1 Standardization
__11.2.2 Normalization
_11.3 Choosing a Cost Function
_11.4 Activation Algorithm
_11.5 Weight Initialization
_11.6 Optimization Algorithm
_11.7 Discussion on the number of nodes and hidden layers
_11.8 Other Model Training Stabilization and Performance Improvement Techniques
__11.8.1 Gradient Clipping
__11.8.2 Early Termination
_11.9 PPO Algorithm Artificial Neural Network Tuning
_11.10 Applying PPO algorithm tuning code
_11.11 Analysis of PPO Algorithm Tuning Results

Chapter 12 Bayesian Optimization Techniques
_12.1 Frequentist and Bayesian Probability
_12.2 Calculating Bayesian Probability
_12.3 Bayesian Optimization Package Optuna
__12.3.1 Optuna Example
_12.4 optuna optimization full code
_12.5 Analysis of Bayesian Optimization Results

Chapter 13 Stable Baselines 3
_13.1 What is Stable-Baselines3?
__13.1.1 Introduction to Open-Source Reinforcement Learning Frameworks
__13.1.2 Relationship with Baselines, Evolution from SB2 to SB3
__13.1.3 Why SB3? Direct Implementation vs.
Using the library
_13.2 Core Components of SB3
__13.2.1 Policy: The Agent's Brain
__13.2.2 Model: The Center of Learning
__13.2.3 Env: Link to the environment
__13.2.4 Callback: Control and Monitor During Learning
__13.2.5 VecEnv: Speeding Up with Parallel Environments
_13.3 Key Components of SB3 as Seen Through Examples
__13.3.1 Understanding SB3 Core Components
__13.3.2 SB3 PPO class hyperparameters
__13.3.3 SB3 PPO class tuning applied
__13.3.4 Running Example Code and Evaluation Metrics
__13.3.5 Key Indicators for Interpreting Learning Outcomes
__13.3.6 TensorBoard

Chapter 14: AI Asset Allocation Strategies
_14.1 What is an asset allocation strategy?
_14.2 Ray Dalio's All Weather Portfolio
__14.2.1 What is an All Weather Portfolio?
__14.2.2 Roles by Asset Class
__14.2.3 Ray Dalio's Asset Allocation
__14.2.4 Pros and Cons of the All-Weather Portfolio
__14.2.5 All-Weather Portfolio Investment Performance Evaluation
__14.2.6 All-Weather Portfolio and Reinforcement Learning
_14.3 All-Weather Portfolio Code Implementation Strategy
_14.4 All Weather Portfolio Full Code
_14.5 Detailed Structure of the All-Weather Portfolio

Chapter 15: Tuning AI Asset Allocation Strategies
_15.1 First Tuning: Applying Empirical Tuning Techniques
_15.2 Second Tuning: Utilizing the Optuna Package
_15.3 Comparison with the Buy and Hold Strategy
_15.4 Additional tuning elements

[Appendix A] Preparing the Practice Environment
_A.1 Preparing for Learning
__A.1.1 Installing Python
__A.1.2 Installing required packages
_A.2 Running Jupyter Notebook
_A.3 Check the program version

[Appendix B] ChatGPT Evolved into RLHF: The Intersection of Generative AI and Reinforcement Learning
_A.1 How were generative AI and Chat GPT created?
__A.1.1 The concept of generative AI
__A.1.2 How was ChatGPT trained?
__A.1.3 How does GPT understand and speak context?
__A.1.4 Summary
_A.2 What is RLHF?
__A.2.1 What is human feedback?
__A.2.2 Why does ChatGPT need a reward model to learn 'good answers'?
__A.2.3 Overall learning process of RLHF
_A.3 Connecting PPO and the Policy-Based Algorithms We Learned
__A.3.1 PPO we learned and PPO actually used
__A.3.2 How did ChatGPT become a representative example of policy-based reinforcement learning?
__A.3.3 The Position of PPO in the Flow of Practical Reinforcement Learning
_A.4 Understanding RLHF through examples
__A.4.1 Full Code Review
__A.4.2 Understanding the Code
_A.5 How will reinforcement learning evolve generative AI?
__A.5.1 The Sorting Problem and the Role of Reinforcement Learning
__A.5.2 Future Evolution Directions of Language Models
__A.5.3 A Philosophical Shift in Reinforcement Learning Toward 'Human-Centered AI'

Search

This book was written for countless programmers who hesitated before the mathematical theory and complex code of reinforcement learning.
Reinforcement learning, which has been known since the emergence of AlphaGo as an "artificial intelligence that is good at games," is actually an autonomous and flexible learning method that discovers optimal strategies on its own through continuous interaction with the environment.
These characteristics make them ideally suited to solving real-world problems characterized by unpredictability and complexity, and are particularly well-established as a viable weapon for profit generation in financial markets, including automated trading systems, high-frequency trading strategies, and risk management models.

This book is not simply a theoretical book; it is a practical guide that helps readers leverage reinforcement learning to achieve meaningful results in real-world markets.
The following are the main features and structure of this book.

Target audience for this book

Developers who want to study reinforcement learning but feel the barrier of mathematics or theoretical explanations
· A programmer who wants to go beyond simple task automation and create an intelligent system that can make decisions and adapt to the situation.
· Individual investors who are interested in asset markets such as stocks, cryptocurrencies, and raw materials and want to design their own profit-generating strategies.
· Startup founders or planners planning AI investment apps, robo-advisors, and financial SaaS products utilizing reinforcement learning
· Practitioners who want to automate repetitive transactions or workflows and empower them with learning and adaptability.

Structure of this book

This book is broadly divided into seven main parts and an appendix.
Each part follows the flow of concept → implementation → practice → application, and the difficulty level increases step by step.
· Basic concepts of reinforcement learning - Intuitive explanation of probability, Markov chain, MDP, value function, and Bellman equation.
· Basic Algorithms - Learn core techniques such as dynamic programming, Monte Carlo, TD learning, SARSA, and Q learning, along with code.
· Artificial Intelligence Concepts & Function Approximation - Covers the principles of neural networks, gradient descent, activation functions, and how to use deep learning frameworks.
Value-based Reinforcement Learning (DQN) - Implement DQN with the CartPole example and practice exploration-exploitation balance and replay memory.
· Tuning and Optimization - Practice activation functions, weight initialization, optimizers, data preprocessing, loss functions, gradient clipping, and Optuna-based Bayesian optimization.
· Financial Market Project - Create an asset allocation environment using yfinance data and perform learning, tuning, and performance analysis with PPO.
· Appendix - Guide to building a practice environment, in-depth explanation of RLHF