Mastering Transformer
 |
Description
Book Introduction
Let's take a look at a transformer-based language model.
Let's solve problems effectively while exploring NLP and computer vision tasks!
Transformer-based language models such as BERT, T5, GPT, DALL-E, and ChatGPT are leading natural language processing (NLP) research and forming a new paradigm.
Thanks to their accurate and fast fine-tuning capabilities, these models outperform traditional machine learning-based approaches on challenging natural language understanding (NLU) problems.
Transformers are achieving remarkable results not only in NLP but also in the rapidly growing fields of multimodal learning and generative AI.
Through this book, readers will learn and implement a variety of text-based NLP tasks, as well as multimodal solutions including text-to-image transformation and transformer-based computer vision (CV) solutions.
This book first introduces various transformer models and explains how to train autoregressive language models such as GPT and XLNet.
You will also learn how to improve model performance and track model training progress using the TensorBoard toolchain.
The second half of the book focuses on solving computer vision problems using the Vision Transformer (ViT).
Finally, we will explore how to apply the capabilities of Transformers to modeling and forecasting time series data.
Through this book, readers will understand the Transformer model and learn how to use it to solve various tasks in the fields of NLP and computer vision.
Let's solve problems effectively while exploring NLP and computer vision tasks!
Transformer-based language models such as BERT, T5, GPT, DALL-E, and ChatGPT are leading natural language processing (NLP) research and forming a new paradigm.
Thanks to their accurate and fast fine-tuning capabilities, these models outperform traditional machine learning-based approaches on challenging natural language understanding (NLU) problems.
Transformers are achieving remarkable results not only in NLP but also in the rapidly growing fields of multimodal learning and generative AI.
Through this book, readers will learn and implement a variety of text-based NLP tasks, as well as multimodal solutions including text-to-image transformation and transformer-based computer vision (CV) solutions.
This book first introduces various transformer models and explains how to train autoregressive language models such as GPT and XLNet.
You will also learn how to improve model performance and track model training progress using the TensorBoard toolchain.
The second half of the book focuses on solving computer vision problems using the Vision Transformer (ViT).
Finally, we will explore how to apply the capabilities of Transformers to modeling and forecasting time series data.
Through this book, readers will understand the Transformer model and learn how to use it to solve various tasks in the fields of NLP and computer vision.
- You can preview some of the book's contents.
Preview
index
[Part 1] Recent developments in the field, preparation for the internship, and the first Hello World program
▣ Chapter 1: From BoW (Bag of Words) to Transformer
1.1 Evolution of NLP approaches
1.2 Summary of traditional NLP approaches
__1.2.1 Design and creation of language models
1.3 Deep Learning Utilization
__1.3.1 Considering word order using RNN models
__1.3.2 LSTM and Gated Recurrent Unit
__1.3.3 Contextual Word Embedding and Transfer Learning
1.4 Overview of Transformer Architecture
__1.4.1 Attention Mechanism
__1.4.2 Multi-head attention mechanism
1.5 Transfer Learning (TL) Using Transformers
1.6 Multimodal Learning
summation
References
▣ Chapter 2: Preparation of the Practice Environment and Basic Knowledge
2.1 Technical Requirements
2.2 Installing the Transformer library with Anaconda
__2.2.1 Installing Anaconda on Linux
__2.2.2 Installing Anaconda on Windows
__2.2.3 Installing Anaconda on macOS
__2.2.4 Installing TensorFlow, PyTorch, and Transformer
__2.2.5 Installing and Using Transformers in Google Colab
2.3 Handling Language Models and Tokenizers
2.4 Leveraging the Community-Provided Model
2.5 Using Multimodal Transformers
2.6 Benchmarks and Dataset Utilization
__2.6.1 Key Benchmarks
__2.6.2 GLUE Benchmark
__2.6.3 SuperGLUE Benchmark
__2.6.4 XTREME Benchmark
__2.6.5 XGLUE Benchmark
__2.6.6 SQuAD Benchmark
__2.6.7 Accessing Datasets Using the API
2.7 Handling Data Using the Datasets Library
__2.7.1 Sorting, subscript access, and shuffling
__2.7.2 Caching and Reusability
__2.7.3 Filtering using the filter function
__2.7.4 Data processing using the map function
__2.7.5 Handling local files
__2.7.6 Preparing the dataset for model training
2.8 Speed and Memory Usage Benchmarks
summation
[Part 2] Various Transformer Models: From Autoencoders to Autoregressive Models
▣ Chapter 3: Autoencoding Language Model
3.1 Technical Requirements
3.2 BERT: An Autoencoding Language Model
__3.2.1 Pre-training task for BERT language model
__3.2.2 A Closer Look at the BERT Language Model
3.3 Training an Autoencoding Language Model for an Arbitrary Language
3.4 Sharing Models with the Community
3.5 Other autoencoding models
__3.5.1 Introduction to ALBERT
__3.5.2 RoBERTa model
__3.5.3 ELECTRA
__3.5.4 DeBERTa model
3.6 Handling Tokenization Algorithms
__3.6.1 BPE
__3.6.2 WordPiece Tokenization
__3.6.3 SentencePiece Tokenization
__3.6.4 tokenizers library
summation
▣ Chapter 4: From Generative Models to LLM (Large-Scale Language Models)
4.1 Technical Requirements
4.2 Introduction to GLM
4.3 Working with GLMs
__4.3.1 GPT model series
__4.3.2 Transformer-XL
__4.3.3 XLNet
4.4 Handling Text-to-Text Models
__4.4.1 Multi-Task Learning with T5
__4.4.2 Zero-shot text generalization using the T0 model
__4.4.3 BART: Another noise removal-based seq2seq model
4.5 GLM training
4.6 Natural language generation using autoregressive models
summation
References
▣ Chapter 5: Fine-tuning a Language Model for Text Classification
5.1 Technical Requirements
5.2 Text Classification Overview
5.3 Fine-tuning the BERT model for single-sentence binary classification
5.4 Training a Classification Model Using Native PyTorch
5.5 Fine-tuning a BERT model for multi-class classification (using a custom dataset)
5.6 Fine-tuning the BERT model for sentence pair regression
5.7 Multi-label text classification
5.8 Fine-tuning the model using run_glue.py
summation
References
▣ Chapter 6: Fine-tuning a Language Model for Token Classification
6.1 Technical Requirements
6.2 Introduction to Token Classification
__6.2.1 Understanding NER
__6.2.2 Understanding Part of Speech Tagging
__6.2.3 Understanding Question Answering (QA)
6.3 Fine-tuning the language model for named entity recognition
6.4 Question-answering using token classification
6.5 Questions and Answers for Various Tasks
summation
▣ Chapter 7: Text Representation
7.1 Technical Requirements
7.2 Introduction to sentence embeddings
__7.2.1 Cross Encoder vs. Dual Encoder
7.3 Benchmarking sentence similarity models
7.4 Zero-Shot Learning Using the BART Model
7.5 Semantic Similarity Experiments Using FLAIR
__7.5.1 Average word embedding
__7.5.2 RNN-based document embedding
__7.5.3 Transformer-based BERT embedding
__7.5.4 SBERT Embedding
7.6 Text Clustering Using SBERT
__7.6.1 Topic Modeling Using BERTopic
7.7 Semantic Search Using SBERT
7.8 Fine-tuning the Embedding Model with Instructions
summation
Further Reading
▣ Chapter 8: Improving Model Performance
8.1 Technical Requirements
8.2 Performance Improvement through Data Augmentation
__8.2.1 Character-level augmentation
__8.2.2 Word-level augmentation
__8.2.3 Sentence-level augmentation
__8.2.4 Improving IMDb Text Classification Performance through Data Augmentation
8.3 Adapting the Model to the Domain
8.4 HPO technique for determining optimal hyperparameters
summation
▣ Chapter 9: Efficient Parameter Tuning
9.1 Technical Requirements
9.2 Introduction to PEFT
9.3 Different types of PEFT
__9.3.1 Additive method
__9.3.2 Optional Method
__9.3.3 Low coefficient fine tuning
9.4 PEFT Practice
__9.4.1 Fine-tuning BERT checkpoints through adapter adjustments
__9.4.2 Efficient Fine-tuning of FLAN-T5 for NLI Tasks (Using LoRA)
__9.4.3 Fine-tuning using QLoRA
summation
References
[Part 3] Advanced Topics
▣ Chapter 10: LLM (Large-Scale Language Model)
10.1 Technical Requirements
10.2 Why LLM is Important
__10.2.1 The Importance of Reward Functions
__10.2.2 Instruction-following ability of LLM
10.3 Fine-tuning large-scale language models
summation
▣ Chapter 11: NLP and Explainable AI (XAI)
11.1 Technical Requirements
11.2 Interpretation of the Head of Attention
__11.2.1 Visualizing Heads of Attention Using exBERT
__11.2.2 Multi-scale visualization of the head of attention using BertViz
__11.2.3 Understanding the Internal Structure of BERT Using a Probe Classifier
11.3 Explaining Model Decisions
__11.3.1 Transformer Decision Analysis Using LIME
__11.3.2 Transformer Decision Analysis Using SHAP (SHapley Additive exPlanations)
summation
▣ Chapter 12: efficient transformer
12.1 Technical Requirements
12.2 Introducing an efficient, lightweight, and fast transformer model
12.3 Model size reduction
__12.3.1 Knowledge Distillation with DistilBERT
__12.3.2 Pruning the Transformer Model
__12.3.3 Quantization
12.4 Utilizing efficient self-attention mechanisms
__12.4.1 Sparse self-attention based on fixed patterns
__12.4.2 Learnable Patterns
__12.4.3 Low-Count Decomposition, Nuclear Methods, and Other Approaches
Easier quantization using 12.5 bitsandbytes
summation
References
▣ Chapter 13: Cross-Language and Multilingual Language Modeling
13.1 Technical Requirements
13.2 Translation Language Modeling and Cross-Language Knowledge Sharing
13.3 XLM and mBERT
__13.3.1 mBERT model
__13.3.2 XLM
13.4 Cross-language similarity task
__13.4.1 Cross-Language Text Similarity
__13.4.2 Visualizing Cross-Language Text Similarity
13.5 Cross-Language Classification
13.6 Cross-Language Zero-Shot Learning
13.7 Large-scale multilingual translation
__13.7.1 Performance of the Fine-Tuned Multilingual Model
summation
References
▣ Chapter 14: Service Provision of Transformer Models
14.1 Technical Requirements
14.2 Providing Transformer Model Services Using FastAPI
14.3 Dockerization of the API
14.4 Providing faster transformer model service using TFX
14.5 Load Testing with Locust
14.6 Faster Inference with ONNX
14.7 SageMaker Inference
summation
Further Reading
▣ Chapter 15: Model Tracking and Monitoring
15.1 Technical Requirements
15.2 Tracking Model Indicators
__15.2.1 Tracking Model Training Using TensorBoard
__15.2.2 Real-time model training tracking using W&B
summation
Further Reading
[Part 4] Using Transformer Beyond NLP
▣ Chapter 16: Vision Transformer (ViT)
16.1 Technical Requirements
16.2 Vision Transformer (ViT)
16.3 Image Classification Using Transformers
16.4 Semantic Segmentation and Object Detection Using Transformers
16.5 Visual Prompt Model
summation
▣ Chapter 17: Generative Multimodal Transformers
17.1 Technical Requirements
17.2 Multimodal Learning
__17.2.1 Generative Multimodal AI
17.3 Stable Diffusion for Text-to-Image Generation
17.4 Image Generation Using Stable Diffusion
17.5 Creating Music with MusicGen
17.6 Text-to-Speech Generation (Speech Synthesis) Using Transformers
summation
▣ Chapter 18: Rethinking Transformer Architecture for Time Series Data
18.1 Technical Requirements
18.2 Basic Concepts of Time Series Data
18.3 Transformers and Time Series Modeling
summation
▣ Chapter 1: From BoW (Bag of Words) to Transformer
1.1 Evolution of NLP approaches
1.2 Summary of traditional NLP approaches
__1.2.1 Design and creation of language models
1.3 Deep Learning Utilization
__1.3.1 Considering word order using RNN models
__1.3.2 LSTM and Gated Recurrent Unit
__1.3.3 Contextual Word Embedding and Transfer Learning
1.4 Overview of Transformer Architecture
__1.4.1 Attention Mechanism
__1.4.2 Multi-head attention mechanism
1.5 Transfer Learning (TL) Using Transformers
1.6 Multimodal Learning
summation
References
▣ Chapter 2: Preparation of the Practice Environment and Basic Knowledge
2.1 Technical Requirements
2.2 Installing the Transformer library with Anaconda
__2.2.1 Installing Anaconda on Linux
__2.2.2 Installing Anaconda on Windows
__2.2.3 Installing Anaconda on macOS
__2.2.4 Installing TensorFlow, PyTorch, and Transformer
__2.2.5 Installing and Using Transformers in Google Colab
2.3 Handling Language Models and Tokenizers
2.4 Leveraging the Community-Provided Model
2.5 Using Multimodal Transformers
2.6 Benchmarks and Dataset Utilization
__2.6.1 Key Benchmarks
__2.6.2 GLUE Benchmark
__2.6.3 SuperGLUE Benchmark
__2.6.4 XTREME Benchmark
__2.6.5 XGLUE Benchmark
__2.6.6 SQuAD Benchmark
__2.6.7 Accessing Datasets Using the API
2.7 Handling Data Using the Datasets Library
__2.7.1 Sorting, subscript access, and shuffling
__2.7.2 Caching and Reusability
__2.7.3 Filtering using the filter function
__2.7.4 Data processing using the map function
__2.7.5 Handling local files
__2.7.6 Preparing the dataset for model training
2.8 Speed and Memory Usage Benchmarks
summation
[Part 2] Various Transformer Models: From Autoencoders to Autoregressive Models
▣ Chapter 3: Autoencoding Language Model
3.1 Technical Requirements
3.2 BERT: An Autoencoding Language Model
__3.2.1 Pre-training task for BERT language model
__3.2.2 A Closer Look at the BERT Language Model
3.3 Training an Autoencoding Language Model for an Arbitrary Language
3.4 Sharing Models with the Community
3.5 Other autoencoding models
__3.5.1 Introduction to ALBERT
__3.5.2 RoBERTa model
__3.5.3 ELECTRA
__3.5.4 DeBERTa model
3.6 Handling Tokenization Algorithms
__3.6.1 BPE
__3.6.2 WordPiece Tokenization
__3.6.3 SentencePiece Tokenization
__3.6.4 tokenizers library
summation
▣ Chapter 4: From Generative Models to LLM (Large-Scale Language Models)
4.1 Technical Requirements
4.2 Introduction to GLM
4.3 Working with GLMs
__4.3.1 GPT model series
__4.3.2 Transformer-XL
__4.3.3 XLNet
4.4 Handling Text-to-Text Models
__4.4.1 Multi-Task Learning with T5
__4.4.2 Zero-shot text generalization using the T0 model
__4.4.3 BART: Another noise removal-based seq2seq model
4.5 GLM training
4.6 Natural language generation using autoregressive models
summation
References
▣ Chapter 5: Fine-tuning a Language Model for Text Classification
5.1 Technical Requirements
5.2 Text Classification Overview
5.3 Fine-tuning the BERT model for single-sentence binary classification
5.4 Training a Classification Model Using Native PyTorch
5.5 Fine-tuning a BERT model for multi-class classification (using a custom dataset)
5.6 Fine-tuning the BERT model for sentence pair regression
5.7 Multi-label text classification
5.8 Fine-tuning the model using run_glue.py
summation
References
▣ Chapter 6: Fine-tuning a Language Model for Token Classification
6.1 Technical Requirements
6.2 Introduction to Token Classification
__6.2.1 Understanding NER
__6.2.2 Understanding Part of Speech Tagging
__6.2.3 Understanding Question Answering (QA)
6.3 Fine-tuning the language model for named entity recognition
6.4 Question-answering using token classification
6.5 Questions and Answers for Various Tasks
summation
▣ Chapter 7: Text Representation
7.1 Technical Requirements
7.2 Introduction to sentence embeddings
__7.2.1 Cross Encoder vs. Dual Encoder
7.3 Benchmarking sentence similarity models
7.4 Zero-Shot Learning Using the BART Model
7.5 Semantic Similarity Experiments Using FLAIR
__7.5.1 Average word embedding
__7.5.2 RNN-based document embedding
__7.5.3 Transformer-based BERT embedding
__7.5.4 SBERT Embedding
7.6 Text Clustering Using SBERT
__7.6.1 Topic Modeling Using BERTopic
7.7 Semantic Search Using SBERT
7.8 Fine-tuning the Embedding Model with Instructions
summation
Further Reading
▣ Chapter 8: Improving Model Performance
8.1 Technical Requirements
8.2 Performance Improvement through Data Augmentation
__8.2.1 Character-level augmentation
__8.2.2 Word-level augmentation
__8.2.3 Sentence-level augmentation
__8.2.4 Improving IMDb Text Classification Performance through Data Augmentation
8.3 Adapting the Model to the Domain
8.4 HPO technique for determining optimal hyperparameters
summation
▣ Chapter 9: Efficient Parameter Tuning
9.1 Technical Requirements
9.2 Introduction to PEFT
9.3 Different types of PEFT
__9.3.1 Additive method
__9.3.2 Optional Method
__9.3.3 Low coefficient fine tuning
9.4 PEFT Practice
__9.4.1 Fine-tuning BERT checkpoints through adapter adjustments
__9.4.2 Efficient Fine-tuning of FLAN-T5 for NLI Tasks (Using LoRA)
__9.4.3 Fine-tuning using QLoRA
summation
References
[Part 3] Advanced Topics
▣ Chapter 10: LLM (Large-Scale Language Model)
10.1 Technical Requirements
10.2 Why LLM is Important
__10.2.1 The Importance of Reward Functions
__10.2.2 Instruction-following ability of LLM
10.3 Fine-tuning large-scale language models
summation
▣ Chapter 11: NLP and Explainable AI (XAI)
11.1 Technical Requirements
11.2 Interpretation of the Head of Attention
__11.2.1 Visualizing Heads of Attention Using exBERT
__11.2.2 Multi-scale visualization of the head of attention using BertViz
__11.2.3 Understanding the Internal Structure of BERT Using a Probe Classifier
11.3 Explaining Model Decisions
__11.3.1 Transformer Decision Analysis Using LIME
__11.3.2 Transformer Decision Analysis Using SHAP (SHapley Additive exPlanations)
summation
▣ Chapter 12: efficient transformer
12.1 Technical Requirements
12.2 Introducing an efficient, lightweight, and fast transformer model
12.3 Model size reduction
__12.3.1 Knowledge Distillation with DistilBERT
__12.3.2 Pruning the Transformer Model
__12.3.3 Quantization
12.4 Utilizing efficient self-attention mechanisms
__12.4.1 Sparse self-attention based on fixed patterns
__12.4.2 Learnable Patterns
__12.4.3 Low-Count Decomposition, Nuclear Methods, and Other Approaches
Easier quantization using 12.5 bitsandbytes
summation
References
▣ Chapter 13: Cross-Language and Multilingual Language Modeling
13.1 Technical Requirements
13.2 Translation Language Modeling and Cross-Language Knowledge Sharing
13.3 XLM and mBERT
__13.3.1 mBERT model
__13.3.2 XLM
13.4 Cross-language similarity task
__13.4.1 Cross-Language Text Similarity
__13.4.2 Visualizing Cross-Language Text Similarity
13.5 Cross-Language Classification
13.6 Cross-Language Zero-Shot Learning
13.7 Large-scale multilingual translation
__13.7.1 Performance of the Fine-Tuned Multilingual Model
summation
References
▣ Chapter 14: Service Provision of Transformer Models
14.1 Technical Requirements
14.2 Providing Transformer Model Services Using FastAPI
14.3 Dockerization of the API
14.4 Providing faster transformer model service using TFX
14.5 Load Testing with Locust
14.6 Faster Inference with ONNX
14.7 SageMaker Inference
summation
Further Reading
▣ Chapter 15: Model Tracking and Monitoring
15.1 Technical Requirements
15.2 Tracking Model Indicators
__15.2.1 Tracking Model Training Using TensorBoard
__15.2.2 Real-time model training tracking using W&B
summation
Further Reading
[Part 4] Using Transformer Beyond NLP
▣ Chapter 16: Vision Transformer (ViT)
16.1 Technical Requirements
16.2 Vision Transformer (ViT)
16.3 Image Classification Using Transformers
16.4 Semantic Segmentation and Object Detection Using Transformers
16.5 Visual Prompt Model
summation
▣ Chapter 17: Generative Multimodal Transformers
17.1 Technical Requirements
17.2 Multimodal Learning
__17.2.1 Generative Multimodal AI
17.3 Stable Diffusion for Text-to-Image Generation
17.4 Image Generation Using Stable Diffusion
17.5 Creating Music with MusicGen
17.6 Text-to-Speech Generation (Speech Synthesis) Using Transformers
summation
▣ Chapter 18: Rethinking Transformer Architecture for Time Series Data
18.1 Technical Requirements
18.2 Basic Concepts of Time Series Data
18.3 Transformers and Time Series Modeling
summation
Detailed image
Publisher's Review
★ What this book covers ★
◎ Focus on solving various simple to complex NLP problems using Python.
◎ Learn how to solve classification/regression problems using traditional NLP approaches.
◎ Understand how to train a language model and fine-tune it for subtasks.
◎ Learn how to use Transformers for generative AI and computer vision tasks.
◎ Build a Transformer-based NLP app using Hugging Face's Transformers library.
◎ Performs multilingual language generation tasks such as machine translation and conversational AI.
◎ Describes an efficiency improvement technique to increase the inference speed of the transformer model and reduce the delay time.
◎ Focus on solving various simple to complex NLP problems using Python.
◎ Learn how to solve classification/regression problems using traditional NLP approaches.
◎ Understand how to train a language model and fine-tune it for subtasks.
◎ Learn how to use Transformers for generative AI and computer vision tasks.
◎ Build a Transformer-based NLP app using Hugging Face's Transformers library.
◎ Performs multilingual language generation tasks such as machine translation and conversational AI.
◎ Describes an efficiency improvement technique to increase the inference speed of the transformer model and reduce the delay time.
GOODS SPECIFICS
- Date of issue: January 10, 2025
- Page count, weight, size: 504 pages | 188*240*20mm
- ISBN13: 9791158395636
You may also like
카테고리
korean
korean
![GQ KOREA Mark (Monthly): December [2025]](http://librairie.coreenne.fr/cdn/shop/files/8ef265dbbfbf186523ed75ba7319009d.jpg?v=1765340328&width=3840)
