CHAPTER 1: INTRODUCING TRANSFORMERS

_1.1 Encoder-Decoder Framework
_1.2 Attention Mechanism
_1.3 Transfer Learning in NLP
_1.4 Hugging Face Transformers
_1.5 Tour of Transformer Applications
__1.5.1 Text Classification
__1.5.2 Named Entity Recognition
__1.5.3 Questions and Answers
__1.5.4 Summary
__1.5.5 Translation
__1.5.6 Text Generation
_1.6 Hugging Face Ecosystem
__1.6.1 Hugging Face Hub
__1.6.2 Hugging Face Tokenizer
__1.6.3 Hugging Face Dataset
__1.6.4 Hugging Face Accelerate
_1.7 Key Challenges of Transformers
_1.8 Conclusion

CHAPTER 2 Text Classification

_2.1 Dataset
__2.1.1 Using the Hugging Face Dataset for the First Time
__2.1.2 From dataset to dataframe
__2.1.3 Examining class distribution
__2.1.4 Check tweet length
_2.2 From text to tokens
__2.2.1 Character Tokenization
__2.2.2 Word tokenization
__2.2.3 Partial word tokenization
__2.2.4 Tokenizing the entire dataset
_2.3 Training a Text Classification Model
__2.3.1 Using Transformers as Feature Extractors
__2.3.2 Fine-tuning the transformer
_2.4 Conclusion

CHAPTER 3: Uncovering Transformers

_3.1 Transformer Architecture
_3.2 Encoder
__3.2.1 Self-Attention
__3.2.2 Feedforward layer
__3.2.3 Adding layer normalization
__3.2.4 Positional Embedding
__3.2.5 Adding a classification head
_3.3 decoder
_3.4 Transformers Universe
__3.4.1 Transformer Family Tree
__3.4.2 Encoder Type
__3.4.3 Decoder Type
__3.4.4 Encoder-Decoder Types
_3.5 Conclusion

CHAPTER 4 Multilingual Named Entity Recognition

_4.1 Dataset
_4.2 Multilingual Transformer
_4.3 XLM-R Tokenization
__4.3.1 Tokenization Pipeline
__4.3.2 SentencePiece tokenizer
_4.4 Transformer for Named Entity Recognition
_4.5 Transformer Model Class
__4.5.1 Body and Head
__4.5.2 Creating a Custom Model for Token Classification
__4.5.3 Loading a custom model
_4.6 Tokenizing Text for NER Tasks
_4.7 Performance Measurement
_4.8 Fine-tuning XLM-RoBERTa
_4.9 Error Analysis
_4.10 Cross-Language Transfer
__4.10.1 When Zero-Shot Transitions Are Useful
__4.10.2 Fine-tuning in multiple languages ​​simultaneously
_4.11 Using the Model Widget
_4.12 Conclusion

CHAPTER 5 Text Generation

_5.1 Difficulties in generating consistent text
_5.2 Greedy Search Decoding
_5.3 Beam search decoding
_5.4 Sampling Method
_5.5 Top-k and Nucleus Sampling
_5.6 Which decoding method is best?
_5.7 Conclusion

Chapter 6 Summary

_6.1 CNN/DailyMail dataset
_6.2 Text Summarization Pipeline
__6.2.1 Summary Criteria Model
__6.2.2 GPT-2
__6.2.3 T5
__6.2.4 BART
__6.2.5 PEGASUS
_6.3 Comparing summary results
_6.4 Evaluating the quality of the generated text
__6.4.1 BLEU
__6.4.2 ROUGE
_6.5 Evaluating PEGASUS on the CNN/DailyMail Dataset
_6.6 Training the Summary Model
__6.6.1 Evaluating PEGASUS in SAMSum
__6.6.2 Fine-tuning PEGASUS
__6.6.3 Generating a Conversation Summary
_6.7 Conclusion

CHAPTER 7 QUESTIONS AND ANSWERS

_7.1 Building a Review-Based QA System
__7.1.1 Dataset
__7.1.2 Extracting answers from text
__7.1.3 Building a QA Pipeline Using Haystack
_7.2 Improving the QA Pipeline
__7.2.1 Evaluating Retrievers
__7.2.2 Evaluating Leaders
__7.2.3 Domain Adaptation
__7.2.4 Evaluating the entire QA pipeline
_7.3 Beyond Extractive QA
_7.4 Conclusion

CHAPTER 8 Building an Efficient Transformer

_8.1 Intent Detection Example
_8.2 Creating a Benchmark Class
_8.3 Reducing model size with knowledge refinement
__8.3.1 Knowledge Refinement in Fine-Tuning
__8.3.2 Knowledge Refinement in Pre-Training
__8.3.3 Creating a Knowledge Refinement Trainer
__8.3.4 Choosing a Good Student
__8.3.5 Finding Good Hyperparameters with Optuna
__8.3.6 Performing Refined Model Benchmarks
_8.4 Speeding up models with quantization
_8.5 Benchmarking Quantized Models
_8.6 Optimizing Inference with ONNX and ONNX Runtime
_8.7 Creating a sparse model with weight pruning
__8.7.1 Sparsity in Deep Neural Networks
__8.7.2 Weight pruning method
_8.8 Conclusion

CHAPTER 9 Dealing with the Label Shortage Problem

_9.1 Creating a GitHub Issue Tag
__9.1.1 Downloading data
__9.1.2 Preparing Data
__9.1.3 Creating a training set
__9.1.4 Creating a Training Slice
_9.2 Building a Naive Bayes Model
_9.3 If there is no labeled data
_9.4 When there is little labeled data
__9.4.1 Data Augmentation
__9.4.2 Using embeddings as lookup tables
__9.4.3 Fine-tuning the basic transformer
__9.4.4 In-context and few-shot learning using prompts
_9.5 Using Unlabeled Data
__9.5.1 Fine-tuning the language model
__9.5.2 Fine-tuning the classifier
__9.5.3 Advanced Methods
_9.6 Conclusion

CHAPTER 10: COLLECTING LARGE-SCALE DATASETS

_10.1 Collecting Large Datasets
__10.1.1 Difficulties in Building Large Corpora
__10.1.2 Creating a Custom Code Dataset
__10.1.3 Handling Large Datasets
__10.1.4 Adding a Dataset to the Hugging Face Hub
_10.2 Building a Tokenizer
__10.2.1 Tokenizer Model
__10.2.2 Measuring Tokenizer Performance
__10.2.3 Tokenizer for Python Code
__10.2.4 Training the Tokenizer
__10.2.5 Saving a Custom Tokenizer to the Hub
_10.3 Training a model from scratch
__10.3.1 Pre-training objectives
__10.3.2 Model Initialization
__10.3.3 Building a Data Loader
__10.3.4 Defining the Training Loop
__10.3.5 Running Training
_10.4 Results and Analysis
_10.5 Conclusion

CHAPTER 11 Future Directions

_11.1 Transformer Expansion
__11.1.1 Law of Scale
__11.1.2 Difficulties in scaling
__11.1.3 Attention please!
__11.1.4 Sparse Attention
__11.1.5 Linear Attention
_11.2 Beyond Text
__11.2.1 Vision
__11.2.2 Table
_11.3 Multimodal Transformer
__11.3.1 Speech-to-Text
__11.3.2 Vision and Text
_11.4 What's next?

Cube and Transformer for creating natural language processing applications

This book is aimed at engineers and researchers with machine learning knowledge, providing practical methods for implementing models and applying Transformers to their work.
After introducing the basic theory and methods required to use Transformers, we will train various natural language processing models for purposes such as named entity recognition (NER) of multilingual text, text generation, text summarization, and question answering (QA).
Let's build the model I need, following the guidance of the team behind HuggingFace Transformers, a library that provides a standardized interface to various Transformer models.

Target audience
Data scientists and machine learning engineers who want to customize Transformer to their liking.
Developers who want to create their own natural language processing applications

Key Contents
ㆍHow to build, debug, and optimize Transformer models for NLP tasks such as text classification and named entity recognition.
How to use transformers for cross-language transfer learning
ㆍHow to improve model performance by applying transformers in situations where labeled data is insufficient.
ㆍMethods for improving the efficiency of transformer models using techniques such as knowledge refinement, quantization, and pruning.
How to train large-scale Transformer models from scratch and scale them to multiple GPUs and distributed environments.