Part 1 | What is Kafka Connect?

Chapter 1: Introduction to Kafka Connect

Kafka Connect features
__Pluggable architecture
__Scalability and reliability
__Declarative pipeline definition
__Kafka Connect is also part of Apache Kafka
Use Cases
__Capture database changes
__Kafka Cluster Mirroring
__Building a data lake
__Log aggregation
__Legacy System Modernization
Alternatives to Kafka Connect
organize

Chapter 2: Apache Kafka

Distributed Event Streaming Platform
__Open source
__distributed system
__Event Streaming
__platform
Kafka Key Concepts
__Publish-Subscribe
__Broker and Record
__Topics and Partitions
__Replication
__Storage cycle and compaction
__Craft and Zookeeper
Interacting with Kafka
__Producer
__Consumer
__Kafka Streams
Getting Started with Kafka
__Kafka execution
__Exchanging records
__Running a Kafka Streams application
organize

Part 2: Building Data Pipelines with Kafka Connect

Chapter 3 Components of the Kafka Connect Data Pipeline

Kafka Connect Runtime
__Running Kafka Connect
__Kafka Connect REST API
__Install the plugin
__Deployment mode
Source connector and sink connector
__Connectors and Tasks
__Configuring the connector
__Run the connector
converter
__Data format and schema
__Configuring the converter
Using the __converter
Transformation and Predicate
__Transformation Use Cases
__Predicate
__Configuring Transformations and Predicates
__Using Transformations and Predicates
organize

Chapter 4: Designing an Effective Data Pipeline

Selecting a Connector
__pipeline direction
__Licensing and Support
__Connector functions
Defining a data model
__Data conversion
__Mapping data between systems
Data formatting
__data format
__schema
A Look Inside Kafka Connect
__Internal Topic
__Group Membership
__Rebalance Protocol
Handling Outage Situations in Kafka Connect
__Walker's disability
__Connector/Task Failure
__Kafka/External System Failure
__Dead Letter Queue
Understanding processing semantics
__Sink connector
__Source Connector
organize

Chapter 5 Connectors in Action

Confluent S3 Sync Connector
__Configuring the connector
__Exactly once semantic
__Run the connector
Confluent JDBC Source Connector
__Configuring the connector
__Run the connector
Debisium MySQL Source Connector
__Configuring the connector
__Event Format
__Run the connector
organize

Chapter 6: Mirroring a Cluster with MirrorMaker

What is mirroring?
__Looking at Mirroring Use Cases
__Mirroring in Practice
Introducing Mirror Maker
__Common concept
__Deployment mode
Mirrormaker Connector
__MirrorSourceConnector
__MirrorCheckpointConnector
__MirrorHeartbeatConnector
Running MirrorMaker
__Disaster Recovery Example
__Cross-region replication example
organize

Part 3 | Running Kafka Connect in a Production Environment

Chapter 7: Deploying and Operating a Kafka Connect Cluster

Preparing the Kafka Connect environment
__Building a Kafka Connect Environment
__Install the plugin
__Networking and Permissions
Walker plugin
__Configuration Provider
__REST extension
__Connector Client Configuration Override Policy
Sizing and Capacity Planning
__Understanding Kafka Connect Resource Utilization
__Selecting the appropriate number of workers and tasks
Running a Kafka Connect Cluster
__Add a worker
__Remove Walker
__Upgrading and Maintaining Walkers
__Restarting tasks and connectors in a failed state
__Reset connector offset
Managing Kafka Connect Using the REST API
__Creating and Deleting Connectors
__Connector and Task Configuration
__Controlling the connector lifecycle
__List connector offsets
Debugging Issues
organize

Chapter 8: Configuring Kafka Connect

Configuring the runtime
__Configuration for commercial environments
__Fine Tuning Configuration
Configuring the Connector
__Topic composition
__Client Override
__Configuration for exactly one time
__Configuration for error handling
Configuring a Kafka Connect Cluster for Security
__Securing your connection to Kafka
__Configure permissions
__Configuring REST API Security
organize

Chapter 9: Monitoring Kafka Connect

Log monitoring
__Logging Configuration
__Understanding the Start Log
__Analyzing logs
Monitoring metrics
__Indicator Reporter
__Analyzing indicators
__Indicator Exploration
Key Indicators
__Kafka Connect Runtime Metrics
__Other system indicators
organize

Chapter 10: Managing Kafka Connect in Kubernetes

Introduction to Kubernetes
__Virtualization technology
__Kubernetes Basics
Running Kafka Connect on Kubernetes
__Container Image
__Walker deployment
__Networking and Monitoring
__composition
Deploying Kafka Connect Using a Kubernetes Operator
Introducing Kubernetes Operators
__Kubernetes Operator for Kafka Connect
Streamji
__Configuring the Kubernetes environment
__Starting the Operator
__Learn about Kafka Connect CRDs
__Deploying a Kafka Connect Cluster and Connectors
__Mirror Maker CRD
organize

Part 4: Building Custom Connectors and Plugins

Chapter 11: Building Source and Sink Connectors

Common concepts and APIs
__Building a Custom Connector
__Connector API
__composition
__Task API
__Kafka Connect Records
__ConnectorContext API
Implementing a Source Connector
__SourceTask API
__source record
__SourceConnectorContext and SourceTaskContext API
__Exactly once semantic support
Implementing a Sync Connector
__SinkTask API
__Sync Record
__SinkConnectorContext and SinkTaskContext API
organize

Chapter 12: Extending Kafka Connect with Connectors and Worker Plugins

Implementing a Connector Plugin
__Transformation API
__Predicate API
__Converter API and HeaderConverter API
Implementing a Walker Plugin
__ConfigProvider API
__ConnectorClientConfigOverridePolicy API
__ConnectRestExtension API
organize

│ What this book covers │

ㆍ Design a resilient and efficient data pipeline by combining core Kafka Connect components.
ㆍCapturing database changes, building a data lake, and mirroring Kafka clusters using existing connectors
ㆍ Deploying, configuring, and operating Kafka Connect clusters in a commercial environment
ㆍ Continuous monitoring of Kafka Connect clusters using logs and metrics
Running a Kafka Connect cluster on Kubernetes
ㆍ Writing your own connectors and plugins

│ Target audience for this book │

Data engineers, site reliability engineers, and developers who interact with the Kafka Connect environment.

│ Structure of this book │

Part 1 provides an introduction to Kafka and Kafka Connect in general, and is primarily intended for engineers who are new to or just getting started with Kafka Connect.

Part 2 explains how to build data pipelines using Kafka Connect and is particularly relevant to data engineers.
Chapters 3 and 4 discuss the core Kafka Connect components and explain how to combine them to design robust and efficient data pipelines.
The remaining chapters in Part 2 take a closer look at the most popular connectors.
Chapter 5 covers three community-provided connectors: Confluent S3 sink, Confluent JDBC source, and Debezium MySQL source.
Chapter 6 details how Kafka's mirroring tool, MirrorMaker, works.
This includes the functionality and configuration of the source, checkpoint, and heartbeat connectors.

Part 3 focuses on the operational aspects of Kafka Connect and is intended for site reliability engineers.
Chapter 7 shows how to deploy and operate a Kafka Connect cluster in a production environment.
Chapter 8 covers all the configuration settings Kafka Connect provides, providing background and context to help you decide how and when to adjust them.
Chapter 9 explains how to continuously monitor your Kafka Connect cluster using logs and metrics.
Finally, Chapter 10 discusses key considerations for running a Kafka Connect cluster on Kubernetes.
This article provides a general introduction to Kubernetes and explains the options available for deploying Kafka Connect on this type of infrastructure.

Part 4 describes how to implement custom connectors and plugins for Kafka Connect and covers the API in detail, making it suitable for developers who want to customize Kafka Connect for their own use cases.

[Author's Note]

Kafka Connect is a great tool for building reliable and scalable data pipelines.
Kafka Connect is part of the popular Apache Kafka streaming platform, and while it may not get as much attention as the broker, client, and Kafka Streams, it's a tool worth knowing about.
Kafka Connect allows you to exchange data with Kafka clusters and mirror data between clusters.
The pluggable design allows you to build powerful pipelines without writing a single line of code.

Whether it's speaking at conferences, writing blog posts, or simply helping fellow Kafka enthusiasts, we're both passionate about sharing our knowledge.
As a result, we've had many conversations with users and developers around the world about Kafka and Kafka Connect.
Because Kafka is such a popular technology, there are a lot of great resources available about it, including books, blog posts, and tutorials.
While many of these resources cover Kafka Connect, I felt there was a lack of in-depth coverage of various use cases, configurations, and operational procedures.
While you can get started with Kafka Connect with just a basic understanding, given its flexibility and versatility, a deeper understanding of how it works can make a huge difference.

We've given many conference presentations on Kafka Connect that go beyond the basics, but there's a limit to how much we can fit into a 40-minute session.
In writing this book, I've compiled everything I've learned about Kafka Connect over the past few years, plus anything I couldn't fit into conference sessions or blog posts.
It includes our own individual experiences running Kafka Connect, as well as insights gained from helping and advising customers.
We also took the time to go through all the settings, metrics, and APIs in detail to thoroughly explain how Kafka Connect works.
This process often involved writing custom plugins to test code paths, meticulously analyzing code, and discussing with other Kafka contributors.

This book will provide you with all the knowledge you need to build reliable data pipelines tailored to your use cases and run them in a production environment.


[Translator's Note]

Kafka Connect is not just a data application; it's a powerful distributed system designed to efficiently build and scale data pipelines in enterprise environments.
It is a key technology that addresses the complexity and reliability issues inherent in existing individual data integration methods and ensures stable data flow in real-time streaming environments.

While translating this book, I felt that it covered the inner workings of Kafka Connect and various operational techniques in depth.
In particular, I am confident that this book will be of practical help to readers as it explains in detail the problems frequently encountered in practice and solutions to them.
- Go Seung-beom

Operating a data lake often involves designing and developing various data pipelines.
Moreover, when the source systems are diverse and real-time data is required, it is especially recommended to consider using Kafka Connect.
This book provides detailed explanations of Kafka Connect components and tips for cluster management and operations.
Most developers who have used Kafka will be able to use Kafka Connect without much trouble, but readers who complete this book will clearly distinguish the roles of Kafka and Kafka Connect.
Therefore, I highly recommend reading this if you are looking to change/build a legacy ETL system into a scalable, real-time ETL system within the Kafka ecosystem.
- Lee Sang-heon

Working as a data engineer at a startup often means you're responsible for the entire data pipeline on your own.
Moreover, with so many tasks to perform, developing an application from scratch can be a huge burden.
In these situations, leveraging Kafka Connect can help you build data pipelines faster and more efficiently.
However, when you first encounter Kafka Connect, it may seem rather complicated.

This book systematically explains the core elements required for operation, helping even beginners reduce trial and error and handle Kafka Connect reliably.
While simple usage methods are available in various resources, a deep understanding of the Kafka Connect lifecycle can help you solve more complex problems.
There are many open-source connectors available, but not all of them are perfect for every situation, so sometimes you may feel the need to build your own custom Kafka connector.
This book takes the burden out of understanding the Kafka Connect lifecycle and provides practical guidance for developing and operating custom connectors.

I, too, had to build a custom Kafka connector to meet new requirements, and the knowledge I gained while translating this book allowed me to develop and operate it myself.
Kafka Connect may not be a silver bullet for every situation, but there's no doubt that it has a wide range of applications in data engineering.
I hope this book will serve as a solid guide for working users like me in finding clues to solve problems.
- Hwang Han-hee