Technology that supports big data
 |
Description
Book Introduction
The success of modern business hinges on how data is collected, integrated, and processed!
Everything you need to know about big data and related technologies, from a data processing expert!
'How to systematize data processing?'
"Technology Supporting Big Data" organizes the elements and technologies required for a series of data processing, focusing on engineering issues, establishes a foundation for efficient data processing, and covers various technologies that support system automation on top of that.
As computer performance improves, expectations for the development of data-driven systems, led by machine learning, are growing.
Therefore, in the future, regardless of the system size, the demand for 'technology that makes data processing itself part of the system' will gradually increase.
The diverse visual materials and systematic introduction to related technologies presented in this book will be of great help to readers in their introduction to big data.
Everything you need to know about big data and related technologies, from a data processing expert!
'How to systematize data processing?'
"Technology Supporting Big Data" organizes the elements and technologies required for a series of data processing, focusing on engineering issues, establishes a foundation for efficient data processing, and covers various technologies that support system automation on top of that.
As computer performance improves, expectations for the development of data-driven systems, led by machine learning, are growing.
Therefore, in the future, regardless of the system size, the demand for 'technology that makes data processing itself part of the system' will gradually increase.
The diverse visual materials and systematic introduction to related technologies presented in this book will be of great help to readers in their introduction to big data.
- You can preview some of the book's contents.
Preview
index
CHAPTER 1 Basic Knowledge of Big Data _ 1
1-1 [Background] Big Data Establishment 3
Accelerated data processing through distributed systems: Two representative technologies that overcome the difficulty of handling big data.
Pioneering Business Use of Distributed Systems - Coexisting with Data Warehouses 7
Expanding the Breadth of Data Analysis You Can Do Yourself - Accelerate Big Data Utilization with Cloud Services and Data Discovery 8
1-2 Data Analysis Foundation in the Big Data Era 11
[Re-introduction] Big Data Technology - Data Processing Structure Using Distributed Systems 11
Data Warehouses and Data Marts - Data Pipeline Basics 16
Data Lakes - Accumulating Data as It Is 17
Developing a Data Analytics Foundation Step by Step - Teams and Roles, Starting Small and Expanding 19
Purpose of Data Collection - Three Examples: 'Retrieval,' 'Processing,' and 'Visualization' 22
Confirmatory and Exploratory Data Analysis 25
1-3 [Attribute Learning] Special Analysis and Data Frames Using Scripting Languages 26
Data Processing and Scripting Languages - Popular Python Languages and DataFrames 26
Data Frames, the Basics of the Basics - Building from 'Arrays within Arrays' 27
Example of a web server's access log - easily processed with pandas data frames 28
Interactively Aggregating Time Series Data - Aggregating Data Using DataFrames 30
Using SQL Results as a Data Frame 31
1-4 BI Tools and Monitoring 33
Monitoring with Spreadsheets - Understanding the Current Status of Your Project 33
Data-Driven Decision Making - KPI Monitoring 35
Identifying Changes and Understanding the Details - Using BI Tools 37
Determining the Line Between Manual and Automated Work 39
1-5 Summary 42
CHAPTER 2 Exploring Big Data _ 43
Basic 45 of 2-1 cross tally
Transaction Tables, Cross Tables, and Pivot Tables—The Concept of 'Cross Aggregation' 45
Lookup Tables - Adding Attributes by Combining Tables 47
Aggregating Tables with SQL - Preparing for Cross-Aggregation of Bulk Data 50
Data Aggregation? Data Mart? Visualization—System Configuration Determined by the Size of the Data Mart 55
2-2 High-speed through column-oriented storage 56
Reducing database latency 56
Column-Oriented Database Access - Compressing Columns to Reduce Disk I/O 58
An MPP Database Approach - Leveraging Multi-Core Power through Parallelism 61
2-3 Ad hoc analysis and visualization tools 64
Ad-hoc Analysis with Jupyter Notebooks - Recording Analysis Processes in a Notebook 64
Dashboard Tools - Visualize aggregated results regularly 68
BI Tools - Interactive Dashboards 75
2-4 Basic Structure of a Data Mart 77
Building Data Marts Perfect for Visualization - OLAP 77
Denormalizing a Table 79
Abstracting Tables in Preparation for Multidimensional Model Visualization 82
2-5 Summary 86
CHAPTER 3 Distributed Processing of Big Data _ 87
3-1 Framework for Large-Scale Distributed Processing 89
Structured and Unstructured Data 89
Hadoop - A Common Platform for Distributed Data Processing 92
Spark - High-Speed In-Memory Data Processing 99
3-2 Query Engine 101
Pipeline 101: Building a Data Mart
Creating Structured Data with Hive 102
The Structure of Presto, the Interactive Query Engine - Aggregating Structured Data with Presto 109
Choosing a Data Analysis Framework: MPP Databases, Hive, Presto, and Spark 115
3-3 Building a Data Mart 119
Fact Tables - Accumulating Time Series Data 119
Aggregate Tables - Reducing the Number of Records 122
Snapshot Table - Recording the Master's State 123
History Table - Recording Master Changes 127
[Final Step] Complete the Denormalized Table by Adding Dimensions 127
3-4 Summary 130
CHAPTER 4 Accumulation of Big Data _ 131
4-1 Bulk and Streaming Data Collection 133
Object Storage and Data Ingestion - Loading Data into Distributed Storage 133
Bulk Data Transfer - The Need for an ETL Server 135
Streaming data transfer - Data transfer for handling small data that is continuously transmitted 137
4-2 [Performance × Reliability] Tradeoffs in Message Delivery 143
Message Broker - Installing a Middle Layer to Solve Storage Performance Problems 143
Ensuring Message Delivery is Difficult - Reliability Issues and Three Design Approaches 146
Deduplication is a costly operation 149
Data Collection Pipelines - Storage Suitable for Long-Term Data Analysis 152
4-3 Optimization of Time Series Data 154
Process time and event time - The main target of data analysis is event time 154
Partitioning and Problems by Process Time - Full Scans You Want to Avoid as Much as Possible 154
Time Series Indexes - Efficient Aggregation by Event Time ① 156
Conditional Pushdown - Efficient Aggregation by Event Time ② 157
Partitioning by Event Time - Table Partitioning, Time Series Tables 158
4-4 Distributed Storage of Unstructured Data 161
[Basic Strategy] Data Utilization with NoSQL Databases 161
Distributed KVS - Improving Disk Write Performance 162
Wide Column Store - Analyzing and Storing Structured Data 166
Document Store - Managing Schemaless Data 169
Search Engines - Finding Data with Keyword Searches 171
4-5 Summary 175
CHAPTER 5 Big Data Pipeline _ 177
5-1 Workflow Management 179
[Basic Knowledge] Workflow Management - Managing the Flow of Data 179
Thinking First About How to Recover from Errors 183
Describing tasks as idempotent operations - executing the same task multiple times produces the same result 188
Making the entire workflow idempotent 194
Task Queues - Controlling Resource Consumption 195
5-2 Batch-type data flow 199
The Era of MapReduce Is Over - Data Flow and Workflow 199
A New Framework to Replace MapReduce - Internal Representation via DAGs 201
Combining Data Flows and Workflows 204
Separating Data Flow and SQL - Data Warehouse Pipelines and Data Mart Pipelines 207
5-3 Streaming Data Flow 209
Splitting the Path between Batch and Stream Processing 209
Integrating Batch and Stream Processing 211
Replacing the Results of Stream Processing with Batch Processing - Addressing Two Problems of Stream Processing 214
Out-of-order data processing 217
5-4 Summary 220
CHAPTER 6 Building a Big Data Analysis Foundation _ 223
6-1 Ad Hoc Analysis of Schemaless Data 225
Collecting Schemaless Data 225
Preparing the Interactive Execution Environment 228
Distributed Environments with Spark - Enabling Data Growth 232
Aggregating Data to Build a Data Mart 237
Visualizing Data with BI Tools 241
6-2 Data Pipelines with Hadoop 245
Tasking Daily Batch Processing 245
[Task 1] Data Extraction with Embulk 246
[Task 2] Structuring Data with Hive 248
[Task 3] Data Aggregation with Presto 250
6-3 Automation by Workflow Management Tools 253
Airflow - Script-based Workflow Management 253
Running a Workflow from the Terminal 257
Start the scheduler to run the DAG regularly 260
Controlling the Resources Consumed by Tasks 265
Running Data Pipelines in Hadoop 266
6-4 Data Pipeline by Cloud Services 268
The Relationship Between Data Analytics and Cloud Services 268
Amazon Web Services 270
Google Cloud Platform 272
Treasure Data 274
6-5 Summary 279
1-1 [Background] Big Data Establishment 3
Accelerated data processing through distributed systems: Two representative technologies that overcome the difficulty of handling big data.
Pioneering Business Use of Distributed Systems - Coexisting with Data Warehouses 7
Expanding the Breadth of Data Analysis You Can Do Yourself - Accelerate Big Data Utilization with Cloud Services and Data Discovery 8
1-2 Data Analysis Foundation in the Big Data Era 11
[Re-introduction] Big Data Technology - Data Processing Structure Using Distributed Systems 11
Data Warehouses and Data Marts - Data Pipeline Basics 16
Data Lakes - Accumulating Data as It Is 17
Developing a Data Analytics Foundation Step by Step - Teams and Roles, Starting Small and Expanding 19
Purpose of Data Collection - Three Examples: 'Retrieval,' 'Processing,' and 'Visualization' 22
Confirmatory and Exploratory Data Analysis 25
1-3 [Attribute Learning] Special Analysis and Data Frames Using Scripting Languages 26
Data Processing and Scripting Languages - Popular Python Languages and DataFrames 26
Data Frames, the Basics of the Basics - Building from 'Arrays within Arrays' 27
Example of a web server's access log - easily processed with pandas data frames 28
Interactively Aggregating Time Series Data - Aggregating Data Using DataFrames 30
Using SQL Results as a Data Frame 31
1-4 BI Tools and Monitoring 33
Monitoring with Spreadsheets - Understanding the Current Status of Your Project 33
Data-Driven Decision Making - KPI Monitoring 35
Identifying Changes and Understanding the Details - Using BI Tools 37
Determining the Line Between Manual and Automated Work 39
1-5 Summary 42
CHAPTER 2 Exploring Big Data _ 43
Basic 45 of 2-1 cross tally
Transaction Tables, Cross Tables, and Pivot Tables—The Concept of 'Cross Aggregation' 45
Lookup Tables - Adding Attributes by Combining Tables 47
Aggregating Tables with SQL - Preparing for Cross-Aggregation of Bulk Data 50
Data Aggregation? Data Mart? Visualization—System Configuration Determined by the Size of the Data Mart 55
2-2 High-speed through column-oriented storage 56
Reducing database latency 56
Column-Oriented Database Access - Compressing Columns to Reduce Disk I/O 58
An MPP Database Approach - Leveraging Multi-Core Power through Parallelism 61
2-3 Ad hoc analysis and visualization tools 64
Ad-hoc Analysis with Jupyter Notebooks - Recording Analysis Processes in a Notebook 64
Dashboard Tools - Visualize aggregated results regularly 68
BI Tools - Interactive Dashboards 75
2-4 Basic Structure of a Data Mart 77
Building Data Marts Perfect for Visualization - OLAP 77
Denormalizing a Table 79
Abstracting Tables in Preparation for Multidimensional Model Visualization 82
2-5 Summary 86
CHAPTER 3 Distributed Processing of Big Data _ 87
3-1 Framework for Large-Scale Distributed Processing 89
Structured and Unstructured Data 89
Hadoop - A Common Platform for Distributed Data Processing 92
Spark - High-Speed In-Memory Data Processing 99
3-2 Query Engine 101
Pipeline 101: Building a Data Mart
Creating Structured Data with Hive 102
The Structure of Presto, the Interactive Query Engine - Aggregating Structured Data with Presto 109
Choosing a Data Analysis Framework: MPP Databases, Hive, Presto, and Spark 115
3-3 Building a Data Mart 119
Fact Tables - Accumulating Time Series Data 119
Aggregate Tables - Reducing the Number of Records 122
Snapshot Table - Recording the Master's State 123
History Table - Recording Master Changes 127
[Final Step] Complete the Denormalized Table by Adding Dimensions 127
3-4 Summary 130
CHAPTER 4 Accumulation of Big Data _ 131
4-1 Bulk and Streaming Data Collection 133
Object Storage and Data Ingestion - Loading Data into Distributed Storage 133
Bulk Data Transfer - The Need for an ETL Server 135
Streaming data transfer - Data transfer for handling small data that is continuously transmitted 137
4-2 [Performance × Reliability] Tradeoffs in Message Delivery 143
Message Broker - Installing a Middle Layer to Solve Storage Performance Problems 143
Ensuring Message Delivery is Difficult - Reliability Issues and Three Design Approaches 146
Deduplication is a costly operation 149
Data Collection Pipelines - Storage Suitable for Long-Term Data Analysis 152
4-3 Optimization of Time Series Data 154
Process time and event time - The main target of data analysis is event time 154
Partitioning and Problems by Process Time - Full Scans You Want to Avoid as Much as Possible 154
Time Series Indexes - Efficient Aggregation by Event Time ① 156
Conditional Pushdown - Efficient Aggregation by Event Time ② 157
Partitioning by Event Time - Table Partitioning, Time Series Tables 158
4-4 Distributed Storage of Unstructured Data 161
[Basic Strategy] Data Utilization with NoSQL Databases 161
Distributed KVS - Improving Disk Write Performance 162
Wide Column Store - Analyzing and Storing Structured Data 166
Document Store - Managing Schemaless Data 169
Search Engines - Finding Data with Keyword Searches 171
4-5 Summary 175
CHAPTER 5 Big Data Pipeline _ 177
5-1 Workflow Management 179
[Basic Knowledge] Workflow Management - Managing the Flow of Data 179
Thinking First About How to Recover from Errors 183
Describing tasks as idempotent operations - executing the same task multiple times produces the same result 188
Making the entire workflow idempotent 194
Task Queues - Controlling Resource Consumption 195
5-2 Batch-type data flow 199
The Era of MapReduce Is Over - Data Flow and Workflow 199
A New Framework to Replace MapReduce - Internal Representation via DAGs 201
Combining Data Flows and Workflows 204
Separating Data Flow and SQL - Data Warehouse Pipelines and Data Mart Pipelines 207
5-3 Streaming Data Flow 209
Splitting the Path between Batch and Stream Processing 209
Integrating Batch and Stream Processing 211
Replacing the Results of Stream Processing with Batch Processing - Addressing Two Problems of Stream Processing 214
Out-of-order data processing 217
5-4 Summary 220
CHAPTER 6 Building a Big Data Analysis Foundation _ 223
6-1 Ad Hoc Analysis of Schemaless Data 225
Collecting Schemaless Data 225
Preparing the Interactive Execution Environment 228
Distributed Environments with Spark - Enabling Data Growth 232
Aggregating Data to Build a Data Mart 237
Visualizing Data with BI Tools 241
6-2 Data Pipelines with Hadoop 245
Tasking Daily Batch Processing 245
[Task 1] Data Extraction with Embulk 246
[Task 2] Structuring Data with Hive 248
[Task 3] Data Aggregation with Presto 250
6-3 Automation by Workflow Management Tools 253
Airflow - Script-based Workflow Management 253
Running a Workflow from the Terminal 257
Start the scheduler to run the DAG regularly 260
Controlling the Resources Consumed by Tasks 265
Running Data Pipelines in Hadoop 266
6-4 Data Pipeline by Cloud Services 268
The Relationship Between Data Analytics and Cloud Services 268
Amazon Web Services 270
Google Cloud Platform 272
Treasure Data 274
6-5 Summary 279
Detailed image
Into the book
The term 'big data' began to appear frequently in the media around late 2011 and 2012, around the time when many companies began adopting distributed systems for data processing.
Although data processing by computers had been done before, the term 'big data' began to be used here and there, and the movement to utilize data for business became active.
--- p.3
The difference between dashboard tools and BI tools is not that strict.
While the former emphasizes the ease of adding new graphs, the latter emphasizes more interactive data exploration.
For example, if you want to take your time and look at your data slowly, such as clicking on a graph to switch to a detailed view or displaying the raw data that forms the basis for an aggregation, a BI tool is a good choice.
--- p.68
Hadoop is currently known as a system representing big data, but historically, its development began around 2003 as a distributed file system for 'Nutch', an open source web crawler.
Afterwards, in 2006, it became an independent project and was distributed as Apache Hadoop.
--- p.92
First, files on object storage are difficult to replace.
Once you write a file, the only way is to replace it in its entirety.
This is fine for things like log files that will not be changed later, but it is not suitable for things that are changed frequently, like databases.
Data with a high write frequency should be stored in a separate RDB and snapshotted regularly, or stored in another 'distributed database'.
--- p.161
Meanwhile, the event data considered in this book, such as messages sent from millions of smartphones, cannot be used directly.
In the previous chapter, we covered the flow of data centered on message flow as a real-time message delivery method.
If batch processing starts with storing the data received in this way in distributed storage, then stream processing is continuing the processing without going through distributed storage.
Although data processing by computers had been done before, the term 'big data' began to be used here and there, and the movement to utilize data for business became active.
--- p.3
The difference between dashboard tools and BI tools is not that strict.
While the former emphasizes the ease of adding new graphs, the latter emphasizes more interactive data exploration.
For example, if you want to take your time and look at your data slowly, such as clicking on a graph to switch to a detailed view or displaying the raw data that forms the basis for an aggregation, a BI tool is a good choice.
--- p.68
Hadoop is currently known as a system representing big data, but historically, its development began around 2003 as a distributed file system for 'Nutch', an open source web crawler.
Afterwards, in 2006, it became an independent project and was distributed as Apache Hadoop.
--- p.92
First, files on object storage are difficult to replace.
Once you write a file, the only way is to replace it in its entirety.
This is fine for things like log files that will not be changed later, but it is not suitable for things that are changed frequently, like databases.
Data with a high write frequency should be stored in a separate RDB and snapshotted regularly, or stored in another 'distributed database'.
--- p.161
Meanwhile, the event data considered in this book, such as messages sent from millions of smartphones, cannot be used directly.
In the previous chapter, we covered the flow of data centered on message flow as a real-time message delivery method.
If batch processing starts with storing the data received in this way in distributed storage, then stream processing is continuing the processing without going through distributed storage.
--- p.210
GOODS SPECIFICS
- Date of issue: November 5, 2018
- Page count, weight, size: 312 pages | 170*225*21mm
- ISBN13: 9791188621439
- ISBN10: 1188621432
You may also like
카테고리
korean
korean
![GQ KOREA Mark (Monthly): December [2025]](http://librairie.coreenne.fr/cdn/shop/files/8ef265dbbfbf186523ed75ba7319009d.jpg?v=1765340328&width=3840)
