CHAPTER 01 Introduction 1
1.1 Reinforcement Learning 2
1.2 Example 5
1.3 Components of Reinforcement Learning 7
1.4 Limitations and Scope 9
1.5 Extended Example: Tic-Tac-Toe 10
1.6 Summary 16
1.7 Early History of Reinforcement Learning 17
Reference 27

PART I Solutions in Table Form
CHAPTER 02 Multiple Choice 31
2.1 Multiple Choice Problem 32
2.2 Behavioral Value Method 34
2.3 10-choice test 35
2.4 Incremental Implementation 38
2.5 Traces of Abnormal Problems 40
2.6 Positive initial value 42
2.7 Trust Limit Action Selection 44
2.8 Gradient Multiple Selection Algorithm 46
2.9 Related Search (Contextual Multiple Choice) 50
2.10 Summary 51
References and Historical Facts 54

CHAPTER 03 Finite Markov Decision Processes 57
3.1 Agent-Environment Interface 58
3.2 Goals and Rewards 64
3.3 Rewards and Episode 66
3.4 Unified Notation for Episodic and Continuous Work 69
3.5 Policy and Value Functions 70
3.6 Optimal Policy and Optimal Value Function 76
3.7 Optimality and Approximation 82
3.8 Summary 83
References and Historical Facts 84

CHAPTER 04 DYNAMIC PROGRAMMING 89
4.1 Policy Evaluation (Prediction) 90
4.2 Policy Improvement 94
4.3 Policy Repetition 97
4.4 Value Repeat 100
4.5 Asynchronous Dynamic Programming 103
4.6 Generalized Policy Iteration 104
4.7 Efficiency of Dynamic Programming 106
4.8 Summary 107
References and Historical Facts 109

CHAPTER 05 Monte Carlo Methods 111
5.1 Monte Carlo Forecasting 112
5.2 Monte Carlo Action Value Estimation 118
5.3 Monte Carlo Control 119
5.4 Monte Carlo Control Without Starting Exploration 123
5.5 Predicting Inactive Policies Using Importance Extraction 126
5.6 Incremental Implementation 133
5.7 Inactive Monte Carlo Control 135
Importance Extraction Method Considering Discounts 138
5.9 Decision-Level Importance Extraction Method 139
5.10 Summary 141
References and Historical Facts 143

CHAPTER 06 Time-Differential Learning 145
6.1 TD Prediction 146
6.2 Advantages of the TD Prediction Method 150
6.3 Optimality of TD(0) 153
6.4 Salsa: Active Policy TD Control 157
6.5 Q Learning: Inactive Policy TD Control 160
6.6 Expected value Salsa 162
6.7 Maximizing Variance and Dual Learning 163
6.8 Games, Post-Game Conditions, and Other Special Cases 166
6.9 Summary 168
References and Historical Facts 169

CHAPTER 07 n-step bootstrap 171
7.1 n-step TD prediction 172
7.2 n-level salsa 177
7.3 Learning n-step inactive policies 179
7.4 Step-by-step method for decision making with control variables 181
7.5 Passive Policy Learning Without Importance Extraction: An n-Stage Tree Augmentation Algorithm 184
7.6 Integration Algorithm: n-Step Q(σ) 187
7.7 Summary 189
References and Historical Facts 190

CHAPTER 08 Planning and Learning Using Table-Based Methods 191
8.1 Models and Plans 192
8.2 Dyna: Integrating Planning, Action, and Learning 194
8.3 When the model is wrong 199
8.4 Prioritized Batch Processing 202
8.5 Expected Updates vs. Sample Updates 206
8.6 Trajectory Sampling 210
8.7 Real-Time Dynamic Programming 213
8.8 Planning at the Decision Point 217
8.9 Empirical Exploration 219
8.10 Dice Rolling Algorithm 221
8.11 Monte Carlo Tree Search 223
8.12 Summary 227
8.13 Part 1 Summary: Dimension 228
References and Historical Facts 231

PART II Approximate Solutions
CHAPTER 09 Active Policy Prediction Using Approximation 237
9.1 Value Function Approximation 238
9.2 Predictive Objectives (VE) 239
9.3 Probabilistic Gradient and Semi-Gradient Methods 241
9.4 Linear Methods 246
9.5 Creating Features for Linear Methods 253
9.6 Manually selecting time interval parameters 268
9.7 Nonlinear Function Approximation: Artificial Neural Networks 269
9.8 Least Squares TD 275
9.9 Memory-Based Function Approximation 278
9.10 Kernel-Based Function Approximation 280
A Deeper Look at 9/11 Active Policy Learning: Focus and Emphasis 282
9.12 Summary 285
References and Historical Facts 286

CHAPTER 10 Active Policy Control with Approximation 293
10.1 Episodic Semi-Slope Control 294
10.2 Semi-slope n-level Salsa 297
10.3 Average Reward: Setting a New Problem for Continuous Tasks 300
10.4 Objection to Discounted Settings 304
10.5 Differential Semi-Slope n-Step Salsa 307
10.6 Summary 308
References and Historical Facts 308

CHAPTER 11: Inactive Policy Methods Using Approximation 311
11.1 Semi-slope Method 312
11.2 Example of Inactive Policy Emission 315
11.3 Deadly Trinity 320
11.4 Linear Value Function Geometry 322
11.5 Gradient Descent in Bellman Error 327
11.6 Bellman error cannot be learned 332
11.7 Gradient TD Method 337
11.8 Strong TD Method 341
11.9 Reducing Variance 343
11.10 Summary 345
References and Historical Facts 346

CHAPTER 12 Eligible Traces 349
12.1 λ gain 350
12.2 TD(λ) 355
12.3 Interrupted n-step λ gain method 359
12.4 Re-Updating: Online λ Gain Algorithm 361
12.5 True Online TD(λ) 363
12.6 Dutch Traces in Monte Carlo Learning 366
12.7 Salsa (λ) 368
12.8 Variable λ and γ 372
12.9 Inactive Policy Traces with Control Variables 374
12.10 From Watkins' Q(λ) to Tree Augmentation(λ) 378
12.11 Stable Inactivity Policy Method Using Traces 381
12.12 Implementation Issue 383
12.13 Conclusion 384
References and Historical Facts 386

CHAPTER 13 Policy Gradient Methods 389
13.1 Policy Approximation and Its Advantages 390
13.2 Policy Gradient Summary 393
13.3 REINFORCE: Monte Carlo Policy Gradient 395
13.4 REINFORCE 399 with reference values
13.5 The Doer-Critic Method 401
13.6 Policy Gradients for Continuous Problems 403
13.7 Policy Parameterization for Continuous Actions 406
13.8 Summary 408
References and Historical Facts 409

PART III A DEEPER DIG
CHAPTER 14 Psychology 413
14.1 Prediction and Control 414
14.2 Classical Conditioning 416
14.3 Instrumental Conditioning 433
14.4 Delayed Reinforcement 438
14.5 Cognitive Map 440
14.6 Habitual and Goal-Directed Behavior 442
14.7 Summary 447
References and Historical Facts 449

CHAPTER 15: NEUROSCIENCE 457
15.1 Neuroscience Fundamentals 458
15.2 Reward Signals, Reinforcement Signals, Value, and Prediction Error 460
15.3 Reward Prediction Error Hypothesis 463
15.4 Dopamine 465
15.5 Experimental Support for the Reward Prediction Error Hypothesis 469
15.6 TD error/dopamine similarity 473
15.7 Neurobehavior-Critic 479
15.8 Doer and Critic Learning Rules 482
15.9 Hedonistic Neurons 488
15.10 Collective Reinforcement Learning 490
15.11 Model-Based Methods in the Brain 494
15.12 Addiction 496
15.13 Summary 497
References and Historical Facts 501

CHAPTER 16 Applications and Case Studies 511
16.1 TD-Garmon 511
16.2 Samuel's Checkers Player 518
16.3 Watson's Double Wager 522
16.4 Memory Control Optimization 526
16.5 Human-level video game skills 531
16.6 Mastering the Game of Baduk 539
16.7 Personalized Web Services 550
16.8 Fever Rise 554

CHAPTER 17 Frontier 559
17.1 General Value Functions and Auxiliary Operations 559
17.2 Temporal Abstraction through Options 562
17.3 Observations and Conditions 565
17.4 Design of the Compensation Signal 572
17.5 Remaining Issues 576
17.6 The Future of Artificial Intelligence 580
References and Historical Facts 584

Reference 588
Search 626

The absolute bible of reinforcement learning, revised for the first time in 20 years with significantly enhanced content!
Understand the core concepts and latest algorithms of reinforcement learning in a simple and clear way!

Reinforcement learning, one of the most actively researched areas in artificial intelligence, is a numerical computational learning method that maximizes the reward given to a learner interacting with a complex and uncertain environment.
In their book, Robust Reinforcement Learning, Richard Sutton and Andrew Barto explain the core concepts and algorithms of reinforcement learning clearly and easily.
Since the publication of the first edition, new topics have been added, and topics already covered have been updated with the latest content.

Like the first edition, the second edition focuses on core online learning algorithms, but includes more mathematical content in separate text boxes.
This book is broadly divided into three parts:

■ In the first part, we covered as many reinforcement learning methods as possible, applying only table-based methods that can find accurate solutions.
Many of the algorithms presented in the first part are new to the second edition, including UCB, Expected Value Salsa, and Dual Learning.
■ In the second part, the methods presented in the first part are extended to function approximation-based methods, with new sections covering topics such as artificial neural networks and Fourier-based methods, and the content on inactive policy learning and policy gradient methods is enriched.
■ The third part includes new chapters on how reinforcement learning relates to psychology and neuroscience, and updated chapters on case studies such as AlphaGo and AlphaGo Zero, Atari games, and IBM Watson's betting strategies.
In the final chapter, we discussed the impact of reinforcement learning on future society.