Chapter 1: What is Deep Learning?

1.1 Artificial Intelligence, Machine Learning, and Deep Learning

__1.1.1 Artificial Intelligence

__1.1.2 Machine Learning

__1.1.3 Learning Representations from Data

__1.1.4 What does 'deep' mean in deep learning?

__1.1.5 Understanding the working principles of deep learning with three figures

__1.1.6 Deep learning achievements to date

__1.1.7 Don't believe short-term hype

__1.1.8 Outlook on AI

1.2 Before Deep Learning: A Brief History of Machine Learning

__1.2.1 Probabilistic Modeling

__1.2.2 Early Neural Networks

__1.2.3 Kernel method

__1.2.4 Decision Trees, Random Forests, and Gradient Boosting Machines

__1.2.5 Back to neural networks

__1.2.6 Characteristics of Deep Learning

__1.2.7 Recent Trends in Machine Learning

1.3 Why Deep Learning? Why Now?

__1.3.1 Hardware

__1.3.2 data

__1.3.3 Algorithm

__1.3.4 A new wind of investment

__1.3.5 Popularization of Deep Learning

__1.3.6 Will it last?


Chapter 2: Mathematical Components of Neural Networks

2.1 First Encounter with Neural Networks

2.2 Data Representation for Neural Networks

__2.2.1 Scalar (Rank-0 Tensor)

__2.2.2 Vectors (Rank-1 Tensors)

__2.2.3 Matrix (Rank-2 Tensor)

__2.2.4 Rank-3 tensors and higher-rank tensors

__2.2.5 Core Properties

__2.2.6 Manipulating Tensors with NumPy

__2.2.7 Batch Data

__2.2.8 Real-world examples of tensors

__2.2.9 Vector data

__2.2.10 Time series data or sequence data

__2.2.11 Image data

__2.2.12 Video Data

2.3 The Cogs of Neural Networks: Tensor Operations

__2.3.1 Element-wise operations

__2.3.2 Broadcasting

__2.3.3 Tensor multiplication

__2.3.4 Tensor size conversion

__2.3.5 Geometric Interpretation of Tensor Operations

__2.3.6 Geometric Interpretation of Deep Learning

2.4 The Engine of Neural Networks: Gradient-Based Optimization

__2.4.1 What is a derivative?

__2.4.2 Derivatives of Tensor Operations: Gradients

__2.4.3 Stochastic Gradient Descent

__2.4.4 Derivative Connection: Backpropagation Algorithm

2.5 Revisiting the First Example

__2.5.1 Reimplementing the first example from scratch using TensorFlow

__2.5.2 Running the training step

__2.5.3 Full training loop

__2.5.4 Evaluating the Model

2.6 Summary


Chapter 3: Introduction to Keras and TensorFlow

3.1 What is TensorFlow?

3.2 What is Keras?

3.3 A Brief History of Keras and TensorFlow

3.4 Setting up the deep learning working environment

__3.4.1 Jupyter Notebook: A Recommended Deep Learning Experimentation Tool

__3.4.2 Using Colab

3.5 Getting Started with TensorFlow

__3.5.1 Constant tensors and variables

__3.5.2 Tensor Operations: Computing Math in TensorFlow

__3.5.3 Revisiting the GradientTape API

__3.5.4 End-to-End Example: TensorFlow Linear Classifier

3.6 Neural Network Structure: Understanding the Core Keras API

__3.6.1 Layer: Components of Deep Learning

__3.6.2 model on the floor

__3.6.3 'Compile' Step: Setting Up the Learning Process

__3.6.4 Choosing a Loss Function

__3.6.5 Understanding the fit( ) method

__3.6.6 Monitoring Loss and Metrics in Validation Data

__3.6.7 Inference: Using the Trained Model

3.7 Summary


Chapter 4: Getting Started with Neural Networks: Classification and Regression

4.1 Movie Review Classification: A Binary Classification Problem

__4.1.1 IMDB dataset

__4.1.2 Data Preparation

__4.1.3 Creating a Neural Network Model

__4.1.4 Training Validation

__4.1.5 Predicting New Data with a Trained Model

__4.1.6 Additional Experiments

__4.1.7 Summary

4.2 News Article Classification: A Multi-Classification Problem

__4.2.1 Reuters Dataset

__4.2.2 Data Preparation

__4.2.3 Model Configuration

__4.2.4 Training Validation

__4.2.5 Predicting on New Data

__4.2.6 Other ways to deal with labels and losses

__4.2.7 Why you need a sufficiently large intermediate layer

__4.2.8 Additional Experiments

__4.2.9 Summary

4.3 Predicting House Prices: A Regression Problem

__4.3.1 Boston Housing Price Dataset

__4.3.2 Data Preparation

__4.3.3 Model Configuration

__4.3.4 Training validation using K-fold validation

__4.3.5 Predicting on New Data

__4.3.6 Summary

4.4 Summary


Chapter 5: Fundamentals of Machine Learning

5.1 Generalization: The Goal of Machine Learning

__5.1.1 Underfitting and Overfitting

__5.1.2 The essence of generalization in deep learning

5.2 Machine Learning Model Evaluation

__5.2.1 Training, validation, and test sets

__5.2.2 Exceeding the common sense threshold

__5.2.3 Things to keep in mind about model evaluation

5.3 Improving Training Performance

__5.3.1 Tuning the Key Parameters of Gradient Descent

__5.3.2 Making Better Assumptions About Structure

__5.3.3 Increasing model capacity

5.4 Improving generalization performance

__5.4.1 Dataset Curation

__5.4.2 Characteristic Engineering

__5.4.3 Using Early Termination

__5.4.4 Regulating the Model

5.5 Summary


Chapter 6: A General Machine Learning Workflow

6.1 Task Definition

__6.1.1 Problem Definition

__6.1.2 Data Collection

__6.1.3 Understanding Data

__6.1.4 Selecting success metrics

6.2 Model Development

__6.2.1 Data Preparation

__6.2.2 Choosing an Evaluation Method

__6.2.3 Going beyond the baseline model

__6.2.4 Increasing Model Capacity: Creating an Overfit Model

__6.2.5 Model Regularization and Hyperparameter Tuning

6.3 Model Deployment

__6.3.1 Explaining the Work to the Client and Setting Expectations

__6.3.2 Deploying the Inference Model

__6.3.3 Monitoring the model while it is running

__6.3.4 Model Maintenance

6.4 Summary


Chapter 7: Mastering Keras

7.1 Various Workflows

7.2 Multiple Ways to Create a Keras Model

__7.2.1 Sequential Model

__7.2.2 Functional API

__7.2.3 Model Subclassing

__7.2.4 Mixing multiple methods

__7.2.5 Using the Right Tool for the Job

7.3 Using the built-in training and evaluation loops

__7.3.1 Creating a Custom Indicator

__7.3.2 Using callbacks

__7.3.3 Creating a Custom Callback

__7.3.4 Monitoring and Visualization with TensorBoard

7.4 Creating Custom Training and Evaluation Loops

__7.4.1 Training vs. Inference

__7.4.2 Low-Level Usage of Measurement Indicators

__7.4.3 Complete training and evaluation loop

__7.4.4 Improving Performance with tf.function

__7.4.5 Using the fit( ) method as a custom loop

7.5 Summary


Chapter 8: Deep Learning for Computer Vision

8.1 Introduction to Convolutional Neural Networks

__8.1.1 Convolution Operation

__8.1.2 Max pooling operation

8.2 Training a Convnet from Scratch on a Small Dataset

__8.2.1 The Feasibility of Deep Learning in Small Dataset Problems

__8.2.2 Downloading data

__8.2.3 Creating a Model

__8.2.4 Data Preprocessing

__8.2.5 Using data augmentation

8.3 Using Pretrained Models

__8.3.1 Feature Extraction Using a Pretrained Model

__8.3.2 Fine-tuning a pre-trained model

8.4 Summary


Chapter 9: Advanced Deep Learning for Computer Vision

9.1 Three Major Computer Vision Tasks

9.2 Image Segmentation Example

9.3 Modern ConvNet Architecture Patterns

__9.3.1 Modularization, Layering, and Reuse

__9.3.2 Residual Linkage

__9.3.3 Batch Normalization

__9.3.4 Depth-wise Separate Convolution

__9.3.5 Apply to all Xception-like models

9.4 Interpreting What ConvNets Learn

__9.4.1 Intermediate Activation Visualization

__9.4.2 Visualizing ConvNet Filters

__9.4.3 Visualizing a heatmap of class activations

9.5 Summary


Chapter 10: Deep Learning for Time Series

10.1 Various types of time series operations

10.2 Temperature Prediction Problem

__10.2.1 Data Preparation

__10.2.2 Common sense level benchmark

__10.2.3 Trying a Basic Machine Learning Model

__10.2.4 Trying a 1D convolutional model

__10.2.5 The First Recurrent Neural Network

10.3 Understanding Recurrent Neural Networks

__10.3.1 Recurrent Layers in Keras

__10.4 Advanced Uses of Recurrent Neural Networks

__10.4.1 Using recurrent dropout to reduce overfitting

__10.4.2 Stacking Circular Layer

__10.4.3 Using Bidirectional RNNs

__10.4.4 Further

10.5 Summary


Chapter 11: Deep Learning for Text

11.1 Introduction to Natural Language Processing

11.2 Preparing Text Data

__11.2.1 Text Standardization

__11.2.2 Text Segmentation (Tokenization)

__11.2.3 Lexical Dictionary Indexing

__11.2.4 Using the TextVectorization Layer

11.3 Two ways to represent groups of words: sets and sequences

__11.3.1 Preparing IMDB Movie Review Data

__11.3.2 Treating words as sets: BoW method

__11.3.3 Processing Words as Sequences: The Sequence Model Approach

11.4 Transformer Architecture

__11.4.1 Understanding Self-Attention

__11.4.2 Multi-head attention

__11.4.3 Transformer Encoder

__11.4.4 When should I use a sequence model instead of a BoW model?

11.5 Beyond Text Classification: Sequence-to-Sequence Learning

__11.5.1 Machine Translation Example

__11.5.2 Sequence-to-sequence model using RNN

__11.5.3 Sequence-to-sequence model using transformers

11.6 Summary


Chapter 12: Deep Learning for Generative Models

12.1 Text Generation

__12.1.1 A Brief History of Deep Learning Models for Sequence Generation

__12.1.2 How to generate sequence data?

__12.1.3 The Importance of Sampling Strategy

__12.1.4 Implementing a Text Generation Model Using Keras

__12.1.5 Text Generation Callback Using Variable Temperature Sampling

__12.1.6 Summary

12.2 Deep Dream

__12.2.1 Keras DeepDream Implementation

__12.2.2 Summary

12.3 Neural Style Transfer

__12.3.1 Content Loss

__12.3.2 Style Loss

__12.3.3 Implementing Neural Style Transfer with Keras

__12.3.4 Summary

12.4 Image Generation Using Variational Autoencoders

__12.4.1 Sampling from the latent space of an image

__12.4.2 Concept vector for image transformation

__12.4.3 Variational Autoencoder

__12.4.4 Implementing VAE with Keras

__12.4.5 Summary

12.5 Introduction to Generative Adversarial Networks

__12.5.1 How to implement GAN

__12.5.2 Training Methods

__12.5.3 Preparing the CelebA Dataset

__12.5.4 Discriminator

__12.5.5 Constructor

__12.5.6 Hostile Networks

__12.5.7 Summary

12.6 Summary


Chapter 13: Best Practices for Solving Real-World Problems

13.1 Unleashing the full potential of your model

__13.1.1 Hyperparameter Optimization

__13.1.2 Model Ensemble

13.2 Training a Large-Scale Model

__13.2.1 Speeding up training on GPUs with mixed precision

__13.2.2 Multi-GPU Training

__13.2.3 TPU Training

13.3 Summary


Chapter 14 Conclusion

14.1 Review of Key Concepts

__14.1.1 Various Methods for AI

__14.1.2 Why Deep Learning is Special in the Field of Machine Learning

__14.1.3 About Deep Learning

__14.1.4 Core Technologies

__14.1.5 General Machine Learning Workflow

__14.1.6 Main network structure

__14.1.7 The Potential of Deep Learning

14.2 Limitations of Deep Learning

__14.2.1 The Risk of Anthropomorphism in Machine Learning Models

__14.2.2 Autonomous Machines vs. Intelligent Agents

__14.2.3 Local Generalization vs. Ultimate Generalization

__14.2.4 The Purpose of Intelligence

__14.2.5 Spectrum of Generalization

14.3 Methods for Improving Generalization in AI

__14.3.1 The Importance of Setting Proper Goals: The Shortcut Rule

__14.3.2 New Goals

14.4 Implementing Intelligence: The Missing Components

__14.4.1 Intelligence superior to abstract metaphors

__14.4.2 Two kinds of abstraction

__14.4.3 Missing Half of the Figure

14.5 The Future of Deep Learning

__14.5.1 Program-like model

__14.5.2 Mixing Deep Learning and Program Synthesis

__14.5.3 Persistent Learning and Modular Subroutine Reuse

__14.5.4 Long-term Vision

14.6 Keeping up with rapid changes

__14.6.1 Practice with Kaggle's real-world problems

__14.6.2 Read the latest papers through the archive (arXiv)

__14.6.3 Exploring the Keras Ecosystem

14.7 Conclusion

This book is written for anyone who wants to learn deep learning from scratch or broaden their understanding.
Whether you're a machine learning engineer, a software engineer, or a college student, you'll learn something from this book.

Let's explore deep learning by starting simple and then moving on to the latest technologies.
This book strikes a balance between intuition, theory, and practice.
We avoid mathematical notation and instead explain the core ideas of machine learning and deep learning with detailed code and intuitive analogies.
There are many code examples with detailed comments and practical guidelines.
It also includes a simple, high-level explanation of everything you need to know to get started with deep learning to solve specific problems.

The example code uses the Python deep learning framework Keras.
Keras uses TensorFlow 2 as its computational engine.
We will show you the latest best practices in Keras and TensorFlow 2.

After reading this book, you will have a clear understanding of what deep learning is, when to apply it, and what its limitations are.
Familiarize yourself with standard workflows for structuring and solving machine learning problems, and learn how to address common issues.
Therefore, you will be able to use Keras to tackle a wide range of real-world problems, from computer vision to natural language processing, image classification, image segmentation, time series forecasting, text classification, machine translation, and text generation.


What's changed in the 2nd edition:

- Reflects the latest TensorFlow and Keras APIs

- Implementing the transformer structure from scratch, including attention

- Added advanced models such as image segmentation, machine translation, text generation, and face image generation.

- Implement custom metrics, custom callbacks, and custom training loops

- Model implementation using gradient tape and subclassing

- Description of the entire machine learning workflow, including problem definition, deployment, and optimization.

- Use functional APIs in most of your code

- Introducing an image augmentation technique using a new image augmentation layer.

- Training methods using KerasTuner, mixed precision, multi-GPU, and TPU

- Reflecting on intelligence and generalization and envisioning the future of deep learning.


Pros of this book:

- Explains everything from basics to the latest advanced algorithms using Python code without using difficult formulas.

- It uses concise Keras code, so developers familiar with Python can easily understand it.

- Covers a wide range of applications, including convolutional neural networks, recurrent neural networks, transformers, and generative models.

- Learn the complete workflow and caveats for machine learning projects.

- For beginners: You can learn deep learning from the basics even if you don't know math.

- For developers: If you are familiar with Python, you can implement all the advanced models.

- For researchers: Learn how to create custom models with Keras.