[PART I: Building a Data Engineering Foundation]

CHAPTER 1 Data Engineering Details
_1.1 What is data engineering?
_1.2 Data Engineering Technologies and Activities
_1.3 Data Engineer within the organization
_1.4 Conclusion
_1.5 References

CHAPTER 2: THE DATA ENGINEERING LIFECYCLE
_2.1 What is the Data Engineering Lifecycle?
_2.2 Key, Unsung Elements of the Data Engineering Lifecycle
_2.3 Conclusion
_2.4 References

CHAPTER 3 Designing a Good Data Architecture
_3.1 What is data architecture?
_3.2 Principles of Good Data Architecture
_3.3 Key Architectural Concepts
_3.4 Cases and Types of Data Architecture
_3.5 Who is responsible for designing the data architecture?
_3.6 Conclusion
_3.7 References

CHAPTER 4 Technology Selection Across the Data Engineering Lifecycle
_4.1 Team size and capabilities
_4.2 Speed ​​to Market
_4.3 Interoperability
_4.4 Cost Optimization and Business Value
_4.5 Present vs. Future: Immutable vs. Temporary Technologies
_4.6 Location: On-premises, cloud, hybrid cloud, multi-cloud
_4.7 Comparison of Building and Purchasing
_4.8 Monolithic vs. Modular Comparison
_4.9 Comparing Serverless and Server
_4.10 Optimization, Performance, and Benchmark Wars
_4.11 The Hidden Elements of the Data Engineering Lifecycle
_4.12 Conclusion
_4.13 References

[PART II: A Deep Dive into the Data Engineering Lifecycle]

CHAPTER 5 STEP 1: GENERATE DATA FROM THE SOURCE SYSTEM
_5.1 Data Sources: How is data generated?
_5.2 Source System: Main Ideas
_5.3 Practical details of the source system
_5.4 Who to work with
_5.5 The impact of unseen factors on the source system
_5.6 Conclusion
_5.7 References

CHAPTER 6 STEP 2: SAVE DATA
_6.1 Basic Components of Data Storage
_6.2 Data Storage System
_6.3 Data Engineering Storage Overview
_6.4 Key Ideas and Trends in Storage
_6.5 Who to work with
_6.6 Unseen Elements
_6.7 Conclusion
_6.8 References

CHAPTER 7 STEP 3: DATA COLLECTION
_7.1 What is data collection?
_7.2 Key Engineering Considerations for the Collection Phase
_7.3 Batch Collection Considerations
_7.4 Considerations for Collecting Messages and Streams
_7.5 Data Collection Methods
_7.6 Person in charge of working together
_7.7 Unseen Elements
_7.8 Conclusion
_7.9 References

CHAPTER 8 STEP 4: Query Modeling and Data Transformation
_8.1 Query
_8.2 Data Modeling
_8.3 Conversion
_8.4 Person in charge of working together
_8.5 Unseen Elements
_8.6 Conclusion
_8.7 References

CHAPTER 9 Step 5: Serving Data for Analytics, Machine Learning, and Reverse ETL
_9.1 General Considerations for Data Serving
_9.2 Analysis
_9.3 Machine Learning
_9.4 What Data Engineers Need to Know About ML
_9.5 How to Serving Data for Analytics and ML
_9.6 Reverse ETL
_9.7 People I work with
_9.8 Unseen Elements
_9.9 Conclusion
_9.10 References

[PART III: The Future of Security, Privacy, and Data Engineering]

CHAPTER 10 Security and Privacy
_10.1 people
_10.2 Process
_10.3 Technology
_10.4 Conclusion
_10.5 References

CHAPTER 11 The Future of Data Engineering
_11.1 The Data Engineering Lifecycle That Never Disappears
_11.2 Reducing Complexity and the Rise of Easy-to-Use Data Tools
_11.3 Cloud-scale Data OS and Enhanced Interoperability
_11.4 'Enterprise' Data Engineering
_11.5 Changes in Positions and Responsibilities
_11.6 Beyond the Modern Data Stack to the Live Data Stack
_11.7 Conclusion

APPENDIX A Serialization and Compression Techniques in Detail
APPENDIX B Cloud Networking

Epilogue
Search

The hotly-talked-about book that data engineers in the field have already discovered!
The core principles of designing and building data pipelines in one book!

This book does not cover data engineering using specific tools, technologies, or platforms.
There are many books that approach data engineering-related technologies from this perspective, but they have a short shelf life.
Instead, this book focuses on the fundamental concepts behind data engineering.


The goal of this book is to fill the gap in current data engineering content and resources.
While there's no shortage of technical resources covering specific data engineering tools and techniques, people struggle to understand how to assemble these components into a coherent, holistic outcome that applies to the real world.
This book explores every step of the data lifecycle, from beginning to end.
In particular, it demonstrates how to combine various technologies to meet the needs of downstream data consumers such as analysts, data scientists, and machine learning engineers.
On the one hand, it serves as a complement to O'Reilly books that cover the details of specific technologies, platforms, and programming languages.

The main content of this book is the data engineering life cycle, which covers data creation, storage, collection, transformation, and serving.
Since the dawn of data, we've witnessed the rise and fall of numerous specific technologies and vendor offerings, but the data engineering lifecycle stages have remained essentially unchanged.
This framework provides readers with the proper understanding needed to apply technology to real-world business problems.


Our goal here is to establish principles that span two axes.
First, we aim to refine data engineering into a discipline that encompasses all relevant technologies.
Second, I want to present principles that remain unchanged over time.
We hope these ideas reflect lessons learned from the past two decades of data technology upheaval, and that our internal framework will remain relevant for the next decade or more.
_From the preface 'About this book'

Target audience

- Person in charge of data engineering practices
- Mid- to senior-level software engineers seeking data engineering work
- Data stakeholders or team leaders who work in conjunction with technical practitioners
- Anyone who wants to understand data engineering as a data analyst and data scientist
- Anyone who wants to draw the big picture in the field of data engineering

Key Contents

- A concise overview of the entire data engineering environment.
- Assess data engineering problems with an end-to-end framework of best practices.
- Avoid marketing hype when choosing data technologies, architectures, and processes.
- Design and build a robust architecture with a data engineering lifecycle.
- Integrate data governance and security across the data engineering lifecycle.