Kafka Essentials Guide
 |
Description
Book Introduction
Written by engineers at Confluent and LinkedIn, who developed Kafka.
Core Practice Guide for Building and Operating a Kafka Environment
From application architects and developers to production engineers new to the Kafka Streaming Platform, this updated and expanded edition (with new chapters covering Kafka's AdminClient API, transactions, new security features, and tooling changes) will teach you how to process large-scale real-time data and streams.
Core Practice Guide for Building and Operating a Kafka Environment
From application architects and developers to production engineers new to the Kafka Streaming Platform, this updated and expanded edition (with new chapters covering Kafka's AdminClient API, transactions, new security features, and tooling changes) will teach you how to process large-scale real-time data and streams.
- You can preview some of the book's contents.
Preview
index
Translator's Preface xv
Beta Reader Review xvi
Recommendation for the Second Edition xviii
1st Edition Recommendation xx
Beginning xxiii
CHAPTER 1 Getting Started with Kafka 1
1.1 Publish/Subscribe Message Delivery 2
1.2 Introduction to Kafka 4
1.3 Why Kafka? 10
1.4 Data Ecosystem 12
1.5 Kafka's Origins 15
1.6 Getting Started with Kafka 18
CHAPTER 2 Installing Kafka 19
2.1 Preferences 19
2.2 Installing the Kafka Broker 23
2.3 Setting up a broker 25
2.4 Choosing Hardware 34
2.5 Using Kafka in the Cloud 36
2.6 Setting up a Kafka cluster 37
2.7 Considerations in Production Environments 44
2.8 Summary 48
CHAPTER 3 Kafka Producer: Writing Messages to Kafka 49
3.1 Producer Overview 50
3.2 Creating a Kafka Producer 52
3.3 Sending Messages with Kafka 54
3.4 Setting up the producer 57
3.5 Serializer 65
3.6 Partition 73
3.7 Header 76
3.8 Interceptor 77
3.9 quarters, throttled 79
3.10 Summary 81
CHAPTER 4 Kafka Consumer: Reading Data from Kafka 83
4.1 Kafka Consumer: Concept 83
4.2 Creating a Kafka Consumer 90
4.3 Subscribe to Topic 91
4.4 Polling Loop 92
4.5 Setting up the consumer 94
4.6 Commit and Offset 102
4.7 Rebalance Listener 109
4.8 Reading records at a specific offset 112
4.9 How to Break Out of a Polling Loop 113
4.10 Deserializer 115
4.11 Standalone Consumers: Why and How to Use Consumers Without Consumer Groups 121
4.12 Summary 123
CHAPTER 5 Managing Kafka with Code Within Your Program 125
5.1 AdminClient Overview 126
5.2 AdminClient Usage: Close Creation Settings 128
5.3 Essential Topic Management Features 130
5.4 Managing Settings 134
5.5 Consumer Group Management 136
5.6 Cluster Metadata 140
5.7 Advanced Admin Tasks 141
5.8 Testing 145
5.9 Summary 147
CHAPTER 6 Kafka's Internal Mechanisms 149
6.1 Cluster Membership 149
6.2 Controller 150
6.3 Replication 158
6.4 Request Processing 161
6.5 Physical Storage 169
6.6 Summary 181
CHAPTER 7 Reliable Data Delivery 183
7.1 Reliability Assurance 184
7.2 Replication 185
7.3 Broker Settings 186
7.4 Using Producers in Trusted Systems 192
7.5 Using Consumers in Trusted Systems 195
7.6 Verifying System Reliability 200
7.7 Summary 204
CHAPTER 8: The Semantic Structure of 'Exactly Once' 205
8.1 Idempotent Producer 206
8.2 Transaction 210
8.3 Transaction Performance 225
8.4 Summary 225
CHAPTER 9 Building a Data Pipeline 227
9.1 Considerations When Building a Data Pipeline 228
9.2 Kafka Connect vs. Producer/Consumer 235
9.3 Kafka Connect 235
9.4 Alternatives to Kafka Connect 257
9.5 Summary 258
CHAPTER 10 Mirroring Data Between Clusters 259
10.1 Inter-Cluster Mirroring Use Cases 260
10.2 Multi-Cluster Architecture 261
10.3 Apache Kafka's MirrorMaker 275
10.4 Other Intercluster Mirroring Solutions 288
10.5 Summary 293
CHAPTER 11 Security 295
11.1 Applying Security Settings 296
11.2 Security Protocols 298
11.3 Certification 301
11.4 Encryption 324
11.5 Approval 326
11.6 Thanks 333
11.7 Zookeeper Security 335
11.8 Platform Security 338
11.9 Summary 340
CHAPTER 12: RUNNING KAFKA 343
12.1 Topic Task 343
12.2 Consumer Group 350
12.3 Changing dynamic settings 353
12.4 Write and Read Operations 358
12.5 Partition Management 364
12.6 Other Tools 374
12.7 Unsafe Work 375
12.8 Summary 377
CHAPTER 13 MONITORING KAFKA 379
13.1 Indicator Basics 379
13.2 Service Level Objectives 383
13.3 Kafka Broker Metrics 386
13.4 Client Monitoring 410
13.5 Rack Monitoring 417
13.6 Endpoint Monitoring 418
13.7 Summary 419
CHAPTER 14 Stream Processing 421
14.1 What is Stream Processing? 423
14.2 Stream Processing Concepts 426
14.3 Stream Processing Design Patterns 434
14.4 Kafka Streams by Example 443
14.5 Kafka Streams: Architectural Overview 452
14.6 Stream Processing Use Cases 459
14.7 Choosing a Stream Processing Framework 460
14.8 Summary 462
APPENDEX A Installing Kafka on Other Operating Systems 465
A.1 Installing on Windows Systems 465
A.2 Installing on macOS 468
APPENDEX B Additional Tools 471
B.1 Integrated Platform 471
B.2 Cluster Installation and Management 473
B.3 Monitoring and Data Exploration 474
B.4 Client Library 476
B.5 Stream Processing 476
Search 477
Beta Reader Review xvi
Recommendation for the Second Edition xviii
1st Edition Recommendation xx
Beginning xxiii
CHAPTER 1 Getting Started with Kafka 1
1.1 Publish/Subscribe Message Delivery 2
1.2 Introduction to Kafka 4
1.3 Why Kafka? 10
1.4 Data Ecosystem 12
1.5 Kafka's Origins 15
1.6 Getting Started with Kafka 18
CHAPTER 2 Installing Kafka 19
2.1 Preferences 19
2.2 Installing the Kafka Broker 23
2.3 Setting up a broker 25
2.4 Choosing Hardware 34
2.5 Using Kafka in the Cloud 36
2.6 Setting up a Kafka cluster 37
2.7 Considerations in Production Environments 44
2.8 Summary 48
CHAPTER 3 Kafka Producer: Writing Messages to Kafka 49
3.1 Producer Overview 50
3.2 Creating a Kafka Producer 52
3.3 Sending Messages with Kafka 54
3.4 Setting up the producer 57
3.5 Serializer 65
3.6 Partition 73
3.7 Header 76
3.8 Interceptor 77
3.9 quarters, throttled 79
3.10 Summary 81
CHAPTER 4 Kafka Consumer: Reading Data from Kafka 83
4.1 Kafka Consumer: Concept 83
4.2 Creating a Kafka Consumer 90
4.3 Subscribe to Topic 91
4.4 Polling Loop 92
4.5 Setting up the consumer 94
4.6 Commit and Offset 102
4.7 Rebalance Listener 109
4.8 Reading records at a specific offset 112
4.9 How to Break Out of a Polling Loop 113
4.10 Deserializer 115
4.11 Standalone Consumers: Why and How to Use Consumers Without Consumer Groups 121
4.12 Summary 123
CHAPTER 5 Managing Kafka with Code Within Your Program 125
5.1 AdminClient Overview 126
5.2 AdminClient Usage: Close Creation Settings 128
5.3 Essential Topic Management Features 130
5.4 Managing Settings 134
5.5 Consumer Group Management 136
5.6 Cluster Metadata 140
5.7 Advanced Admin Tasks 141
5.8 Testing 145
5.9 Summary 147
CHAPTER 6 Kafka's Internal Mechanisms 149
6.1 Cluster Membership 149
6.2 Controller 150
6.3 Replication 158
6.4 Request Processing 161
6.5 Physical Storage 169
6.6 Summary 181
CHAPTER 7 Reliable Data Delivery 183
7.1 Reliability Assurance 184
7.2 Replication 185
7.3 Broker Settings 186
7.4 Using Producers in Trusted Systems 192
7.5 Using Consumers in Trusted Systems 195
7.6 Verifying System Reliability 200
7.7 Summary 204
CHAPTER 8: The Semantic Structure of 'Exactly Once' 205
8.1 Idempotent Producer 206
8.2 Transaction 210
8.3 Transaction Performance 225
8.4 Summary 225
CHAPTER 9 Building a Data Pipeline 227
9.1 Considerations When Building a Data Pipeline 228
9.2 Kafka Connect vs. Producer/Consumer 235
9.3 Kafka Connect 235
9.4 Alternatives to Kafka Connect 257
9.5 Summary 258
CHAPTER 10 Mirroring Data Between Clusters 259
10.1 Inter-Cluster Mirroring Use Cases 260
10.2 Multi-Cluster Architecture 261
10.3 Apache Kafka's MirrorMaker 275
10.4 Other Intercluster Mirroring Solutions 288
10.5 Summary 293
CHAPTER 11 Security 295
11.1 Applying Security Settings 296
11.2 Security Protocols 298
11.3 Certification 301
11.4 Encryption 324
11.5 Approval 326
11.6 Thanks 333
11.7 Zookeeper Security 335
11.8 Platform Security 338
11.9 Summary 340
CHAPTER 12: RUNNING KAFKA 343
12.1 Topic Task 343
12.2 Consumer Group 350
12.3 Changing dynamic settings 353
12.4 Write and Read Operations 358
12.5 Partition Management 364
12.6 Other Tools 374
12.7 Unsafe Work 375
12.8 Summary 377
CHAPTER 13 MONITORING KAFKA 379
13.1 Indicator Basics 379
13.2 Service Level Objectives 383
13.3 Kafka Broker Metrics 386
13.4 Client Monitoring 410
13.5 Rack Monitoring 417
13.6 Endpoint Monitoring 418
13.7 Summary 419
CHAPTER 14 Stream Processing 421
14.1 What is Stream Processing? 423
14.2 Stream Processing Concepts 426
14.3 Stream Processing Design Patterns 434
14.4 Kafka Streams by Example 443
14.5 Kafka Streams: Architectural Overview 452
14.6 Stream Processing Use Cases 459
14.7 Choosing a Stream Processing Framework 460
14.8 Summary 462
APPENDEX A Installing Kafka on Other Operating Systems 465
A.1 Installing on Windows Systems 465
A.2 Installing on macOS 468
APPENDEX B Additional Tools 471
B.1 Integrated Platform 471
B.2 Cluster Installation and Management 473
B.3 Monitoring and Data Exploration 474
B.4 Client Library 476
B.5 Stream Processing 476
Search 477
Detailed image
Into the book
Kafka resembles the concept of an "event stream" in stream processing (covered in Chapter 14).
Just as stream processing applications require a "state store" to view the "current state" of an "event stream," understanding a system like Kafka requires something that can clearly organize the "current state."
This book is a guide to Kafka written by people who have been developing Kafka for a long time.
You could say that it is a 'state storage' created by the people who created the 'event stream' that has developed Kafka so far.
Having contributed to the Kafka project for quite some time, I translated this book with the hope that it would serve as a good 'state repository' for Korean readers.
---From the "Translator's Preface"
Apache Kafka is a publish/subscribe message system designed to solve the problems described above.
It is also called a 'distributed commit log' or 'distributed streaming platform'.
Filesystem or database commit logs are designed to persistently preserve all transaction records, allowing the system's state to be consistently recovered.
Similarly, data stored in Kafka is persisted in an ordered, deterministic manner and can be read deterministically.
Additionally, data can be distributed and stored within the system to improve performance during expansion and ensure data availability even in the event of a failure.
--- p.4
Some people ask where the name Kafka comes from and whether it signifies some special function.
Jay Krebs once commented on this:
I thought it was right to use the author's name, since Kafka is a system optimized for writing.
I took a lot of literature classes in college and loved the works of Franz Kafka.
Plus, the name sounded cool as an open source project name.
Therefore, we can say that there is no relationship between the name and function of Kafka.
--- p.18
Since 2019, the Apache Kafka community has been working on an ambitious project.
This is a move away from a ZooKeeper-based controller to a raft-based controller quorum.
A preview version of a new controller called 'KRaft' was included in Apache Kafka 2.8, and starting with 3.3, it was officially designated as production-ready, dropping the 'experimental' designation.
--- p.152
Kafka's primary role in a data pipeline is to act as a very large and stable buffer between the various stages of the data pipeline.
This effectively decouples the data writer and reader ends of the data pipeline, allowing the same data from a single source to be sent to multiple target applications or systems with different timeliness and availability requirements.
This ability to decouple the two sides of the data pipeline is what makes Kafka a good fit for most data pipelines, along with its reliability, security, and efficiency.
--- p.227
As of version 3.4.0, Kafka only supports Java 8 and later.
However, if your environment is security-critical, we recommend using Java 11 if possible.
The text states, “Kafka supports only protocols after TLSv1.2 and TLSv1.3 by default,” but this is because TLSv1.3 is supported starting from Java 11.
Starting with Kafka 2.6.0, the default has been changed to automatically enable TLSv1.3 when using Java 11 or later.
Just as stream processing applications require a "state store" to view the "current state" of an "event stream," understanding a system like Kafka requires something that can clearly organize the "current state."
This book is a guide to Kafka written by people who have been developing Kafka for a long time.
You could say that it is a 'state storage' created by the people who created the 'event stream' that has developed Kafka so far.
Having contributed to the Kafka project for quite some time, I translated this book with the hope that it would serve as a good 'state repository' for Korean readers.
---From the "Translator's Preface"
Apache Kafka is a publish/subscribe message system designed to solve the problems described above.
It is also called a 'distributed commit log' or 'distributed streaming platform'.
Filesystem or database commit logs are designed to persistently preserve all transaction records, allowing the system's state to be consistently recovered.
Similarly, data stored in Kafka is persisted in an ordered, deterministic manner and can be read deterministically.
Additionally, data can be distributed and stored within the system to improve performance during expansion and ensure data availability even in the event of a failure.
--- p.4
Some people ask where the name Kafka comes from and whether it signifies some special function.
Jay Krebs once commented on this:
I thought it was right to use the author's name, since Kafka is a system optimized for writing.
I took a lot of literature classes in college and loved the works of Franz Kafka.
Plus, the name sounded cool as an open source project name.
Therefore, we can say that there is no relationship between the name and function of Kafka.
--- p.18
Since 2019, the Apache Kafka community has been working on an ambitious project.
This is a move away from a ZooKeeper-based controller to a raft-based controller quorum.
A preview version of a new controller called 'KRaft' was included in Apache Kafka 2.8, and starting with 3.3, it was officially designated as production-ready, dropping the 'experimental' designation.
--- p.152
Kafka's primary role in a data pipeline is to act as a very large and stable buffer between the various stages of the data pipeline.
This effectively decouples the data writer and reader ends of the data pipeline, allowing the same data from a single source to be sent to multiple target applications or systems with different timeliness and availability requirements.
This ability to decouple the two sides of the data pipeline is what makes Kafka a good fit for most data pipelines, along with its reliability, security, and efficiency.
--- p.227
As of version 3.4.0, Kafka only supports Java 8 and later.
However, if your environment is security-critical, we recommend using Java 11 if possible.
The text states, “Kafka supports only protocols after TLSv1.2 and TLSv1.3 by default,” but this is because TLSv1.3 is supported starting from Java 11.
Starting with Kafka 2.6.0, the default has been changed to automatically enable TLSv1.3 when using Java 11 or later.
--- p.307
Publisher's Review
A core practical book written by the creators of Kafka and translated by a Korean developer who participated in its development.
Every enterprise application generates a variety of data, from log messages and metrics to user behavior and outgoing messages, and how this data is moved is as important as the data itself.
From application architects and developers to production engineers new to the Kafka Streaming Platform, this updated and expanded edition (with new chapters covering Kafka's AdminClient API, transactions, new security features, and tooling changes) will teach you how to work with data in motion.
Developers from Confluent and LinkedIn, who worked on Kafka, explain how to deploy a Kafka cluster in production, develop reliable, event-driven microservices, and use Kafka as a platform for building scalable stream processing applications.
Through detailed examples, you can learn Kafka's design principles, reliability guarantees, core APIs, and architectural details ranging from replication protocols to controllers and storage layers.
Best practices for Kafka configuration and installation
How to use Kafka producers and consumers to write and read messages
ㆍ Patterns and use case requirements for reliable data transmission
Best practices for developing data pipelines and applications using Kafka
How to monitor, tune, and maintain Kafka when running it in a production environment
ㆍ The most important performance indicators for operating Kafka
ㆍ Delivery guarantees available in Kafka for stream processing systems
Every enterprise application generates a variety of data, from log messages and metrics to user behavior and outgoing messages, and how this data is moved is as important as the data itself.
From application architects and developers to production engineers new to the Kafka Streaming Platform, this updated and expanded edition (with new chapters covering Kafka's AdminClient API, transactions, new security features, and tooling changes) will teach you how to work with data in motion.
Developers from Confluent and LinkedIn, who worked on Kafka, explain how to deploy a Kafka cluster in production, develop reliable, event-driven microservices, and use Kafka as a platform for building scalable stream processing applications.
Through detailed examples, you can learn Kafka's design principles, reliability guarantees, core APIs, and architectural details ranging from replication protocols to controllers and storage layers.
Best practices for Kafka configuration and installation
How to use Kafka producers and consumers to write and read messages
ㆍ Patterns and use case requirements for reliable data transmission
Best practices for developing data pipelines and applications using Kafka
How to monitor, tune, and maintain Kafka when running it in a production environment
ㆍ The most important performance indicators for operating Kafka
ㆍ Delivery guarantees available in Kafka for stream processing systems
GOODS SPECIFICS
- Date of issue: April 14, 2023
- Page count, weight, size: 520 pages | 962g | 188*257*24mm
- ISBN13: 9791192469768
- ISBN10: 1192469763
You may also like
카테고리
korean
korean
![GQ KOREA Mark (Monthly): December [2025]](http://librairie.coreenne.fr/cdn/shop/files/8ef265dbbfbf186523ed75ba7319009d.jpg?v=1765340328&width=3840)
