Running Spark
 |
Description
Book Introduction
The definitive Spark introductory book recommended by Spark founder Matei Zaharia!
This updated edition, which includes Spark 3.x, shows data engineers and data scientists why Spark's architecture and integration are important.
We perform data analysis from simple to complex and systematically explain how to use machine learning algorithms.
This updated edition, which includes Spark 3.x, shows data engineers and data scientists why Spark's architecture and integration are important.
We perform data analysis from simple to complex and systematically explain how to use machine learning algorithms.
- You can preview some of the book's contents.
Preview
index
Translator's Preface x
Beta Reader Review xii
Recommendation xiv
Starting with xv
About the cover xxi
CHAPTER 1: Introducing Apache Spark: A Unified Analytics Engine
The Beginning of Spark 1
What is Apache Spark? 4
Integrated Analysis 7
Developer Experience 15
CHAPTER 2 Downloading and Starting Apache Spark 19
Step 1: Download Apache Spark 19
Step 2: Using the Scala or PySpark Shell 22
Using a local machine 24
Step 3: Understanding Spark Application Concepts 26
Transformation, Action, and Delayed Evaluation 29
Spark UI 31
First standalone application 34
Summary 42
CHAPTER 3 Apache Spark's Formalized API 43
Spark: What's Underneath an RDD? 44
Establishing Spark's Structure 45
DataFrame API 48
Dataset API 71
DataFrames vs. DataSets 77
Spark SQL and the underlying engine 79
Summary 85
CHAPTER 4 Spark SQL and DataFrames: Introducing Built-in Data Sources 86
Using Spark SQL in Spark Applications 87
SQL Tables and Views 93
Data Sources for DataFrames and SQL Tables 98
Summary 119
CHAPTER 5 Spark SQL and DataFrames: Interacting with External Data Sources 120
Spark SQL and Apache Hive 120
Querying with Spark SQL Shell, Beeline, and Tableau 126
External Data Source 134
PostgreSQL 137
Higher-Order Functions in DataFrames and Spark SQL 144
150 Common DataFrame and Spark SQL Operations
Summary 163
CHAPTER 6 Spark SQL and Datasets 164
A Single API for Java and Scala 164
Working with Datasets 167
Memory Management for Datasets and DataFrames 175
Dataset Encoder 176
Dataset usage cost 178
Summary 180
CHAPTER 7 Optimizing and Tuning Spark Applications 181
Optimizing and Tuning Spark Efficiently 181
Data Caching and Persistence 191
Spark Join Types 196
A Look Inside Spark UI 206
Summary 213
CHAPTER 8: Streaming Standardization 214
Evolution of Apache Spark's Stream Processing Engine 214
Programming Model for Formatted Streaming 218
Fundamentals of Structured Streaming Queries 220
Internals of a Running Streaming Query 227
Streaming Data Sources and Sinks 233
Data Transformation 243
Streaming Aggregation with Stateful Information 246
Streaming Join 255
Arbitrary state maintenance operations 263
Performance Tuning 272
Summary 274
CHAPTER 9 Building a Reliable Data Lake with Apache Spark 275
The Importance of Optimal Storage Solutions 275
Database 277
Data Lake 279
Lakehouse: The Next Step in the Evolution of Storage Solutions 282
Building a Lakehouse with Apache Spark and Delta Lake 285
Summary 296
CHAPTER 10 Machine Learning with MLlib 298
What is Machine Learning? 299
Machine Learning Pipeline Design 302
Hyperparameter Tuning 322
Summary 338
CHAPTER 11: Managing, Deploying, and Scaling Machine Learning Pipelines with Apache Spark 339
Model Management 339
Model Deployment Options with MLlib 346
Leveraging Spark for Non-MLlib Models 352
Summary 358
CHAPTER 12 Epilogue: Apache Spark 3.0 359
Spark Core and Spark SQL 359
Jeonghyeonghwa Streaming 368
PySpark, Pandas UDF, Pandas Function API 370
Changed features 373
Summary 376
Search 379
Beta Reader Review xii
Recommendation xiv
Starting with xv
About the cover xxi
CHAPTER 1: Introducing Apache Spark: A Unified Analytics Engine
The Beginning of Spark 1
What is Apache Spark? 4
Integrated Analysis 7
Developer Experience 15
CHAPTER 2 Downloading and Starting Apache Spark 19
Step 1: Download Apache Spark 19
Step 2: Using the Scala or PySpark Shell 22
Using a local machine 24
Step 3: Understanding Spark Application Concepts 26
Transformation, Action, and Delayed Evaluation 29
Spark UI 31
First standalone application 34
Summary 42
CHAPTER 3 Apache Spark's Formalized API 43
Spark: What's Underneath an RDD? 44
Establishing Spark's Structure 45
DataFrame API 48
Dataset API 71
DataFrames vs. DataSets 77
Spark SQL and the underlying engine 79
Summary 85
CHAPTER 4 Spark SQL and DataFrames: Introducing Built-in Data Sources 86
Using Spark SQL in Spark Applications 87
SQL Tables and Views 93
Data Sources for DataFrames and SQL Tables 98
Summary 119
CHAPTER 5 Spark SQL and DataFrames: Interacting with External Data Sources 120
Spark SQL and Apache Hive 120
Querying with Spark SQL Shell, Beeline, and Tableau 126
External Data Source 134
PostgreSQL 137
Higher-Order Functions in DataFrames and Spark SQL 144
150 Common DataFrame and Spark SQL Operations
Summary 163
CHAPTER 6 Spark SQL and Datasets 164
A Single API for Java and Scala 164
Working with Datasets 167
Memory Management for Datasets and DataFrames 175
Dataset Encoder 176
Dataset usage cost 178
Summary 180
CHAPTER 7 Optimizing and Tuning Spark Applications 181
Optimizing and Tuning Spark Efficiently 181
Data Caching and Persistence 191
Spark Join Types 196
A Look Inside Spark UI 206
Summary 213
CHAPTER 8: Streaming Standardization 214
Evolution of Apache Spark's Stream Processing Engine 214
Programming Model for Formatted Streaming 218
Fundamentals of Structured Streaming Queries 220
Internals of a Running Streaming Query 227
Streaming Data Sources and Sinks 233
Data Transformation 243
Streaming Aggregation with Stateful Information 246
Streaming Join 255
Arbitrary state maintenance operations 263
Performance Tuning 272
Summary 274
CHAPTER 9 Building a Reliable Data Lake with Apache Spark 275
The Importance of Optimal Storage Solutions 275
Database 277
Data Lake 279
Lakehouse: The Next Step in the Evolution of Storage Solutions 282
Building a Lakehouse with Apache Spark and Delta Lake 285
Summary 296
CHAPTER 10 Machine Learning with MLlib 298
What is Machine Learning? 299
Machine Learning Pipeline Design 302
Hyperparameter Tuning 322
Summary 338
CHAPTER 11: Managing, Deploying, and Scaling Machine Learning Pipelines with Apache Spark 339
Model Management 339
Model Deployment Options with MLlib 346
Leveraging Spark for Non-MLlib Models 352
Summary 358
CHAPTER 12 Epilogue: Apache Spark 3.0 359
Spark Core and Spark SQL 359
Jeonghyeonghwa Streaming 368
PySpark, Pandas UDF, Pandas Function API 370
Changed features 373
Summary 376
Search 379
Detailed image
Into the book
By 2013, Spark had become so widely used that its original authors and researchers (Matej Zaharia, Ali Ghosh, Reynolds Shin, Patrick Wendell, Ion Stojka, and Andy Konvinski) transferred the Spark project to the Apache Software Foundation (ASF) and formed a company called Databricks.
Developers from Databricks and the open source community released Apache Spark 1.010 in May 2014 under the leadership of the ASF.
This first major release sets the stage for frequent releases and notable features from Databricks and over 100 commercial partners.
--- p.4
You can write a single Spark application and everything runs on it, without having to spin up separate engines for completely different tasks or learn separate APIs.
With Spark, you have a single, unified processing engine to handle your workload.
--- p.5
Of all the joys a developer can experience, one of the most appealing is a well-structured set of APIs that increase productivity, are easy to use, and are easy to understand.
One of the principles of Apache Spark is to appeal to developers with an easy-to-use API across multiple languages, including Scala, Java, Python, SQL, and R, regardless of the scale of the data.
--- p.15
One of the authors of this book is a data scientist who loves baking cookies using M&Ms, and she often gives them out as prizes to students from various states in her machine learning and data science courses.
But because she's a data-driven person, she wants to make sure that students in different states are given the right color of M&Ms.
Let's write a Spark program that reads a file containing over 100,000 data points (each line contains a state, a color, and a count of M&Ms) and aggregates them by color and state.
These aggregated results will tell you what color M&Ms students in each state like.
The complete Python program is in Example 2-1.
--- p.35
What's the difference between data caching and persistence? In Spark, the two terms are synonymous.
Two API calls, cache() and persist(), provide these features.
The latter can provide more granular settings about where and how data is stored - whether in memory or on disk, and whether it is serialized or not.
Developers from Databricks and the open source community released Apache Spark 1.010 in May 2014 under the leadership of the ASF.
This first major release sets the stage for frequent releases and notable features from Databricks and over 100 commercial partners.
--- p.4
You can write a single Spark application and everything runs on it, without having to spin up separate engines for completely different tasks or learn separate APIs.
With Spark, you have a single, unified processing engine to handle your workload.
--- p.5
Of all the joys a developer can experience, one of the most appealing is a well-structured set of APIs that increase productivity, are easy to use, and are easy to understand.
One of the principles of Apache Spark is to appeal to developers with an easy-to-use API across multiple languages, including Scala, Java, Python, SQL, and R, regardless of the scale of the data.
--- p.15
One of the authors of this book is a data scientist who loves baking cookies using M&Ms, and she often gives them out as prizes to students from various states in her machine learning and data science courses.
But because she's a data-driven person, she wants to make sure that students in different states are given the right color of M&Ms.
Let's write a Spark program that reads a file containing over 100,000 data points (each line contains a state, a color, and a count of M&Ms) and aggregates them by color and state.
These aggregated results will tell you what color M&Ms students in each state like.
The complete Python program is in Example 2-1.
--- p.35
What's the difference between data caching and persistence? In Spark, the two terms are synonymous.
Two API calls, cache() and persist(), provide these features.
The latter can provide more granular settings about where and how data is stored - whether in memory or on disk, and whether it is serialized or not.
--- p.191
Publisher's Review
The definitive Spark introductory book recommended by Spark founder Matei Zaharia!
The second edition of Running Spark: Lightning-Fast Data Analysis has been published.
As data grows larger, is generated faster, and is available in a variety of formats, large-scale processing for analytics and machine learning is also required.
Apache Spark is an alternative that can efficiently handle these large-scale workloads.
This updated edition, which includes Spark 3.x, shows data engineers and data scientists why Spark's architecture and integration are important.
We systematically explain how to perform data analysis from simple to complex and how to use machine learning algorithms.
Step-by-step exercises, code examples, notebooks, and more help you:
■ Learning high-level structured APIs using Python, SQL, Scala, and Java
■ Understanding Spark Jobs and the SQL Engine
■ Inspect, tune, and debug Spark jobs using Spark configuration and Spark UI
■ Connect to data sources such as JSON, Parque, CSV, Avro, ORC, Hive, S3, and Kafka
■ Perform analytics on batch and streaming data using structured streaming
■ Building a stable data pipeline with open source Delta Lake and Spark
■ Develop machine learning pipelines using MLlib and reproduce and deploy models using MLflow.
The second edition of Running Spark: Lightning-Fast Data Analysis has been published.
As data grows larger, is generated faster, and is available in a variety of formats, large-scale processing for analytics and machine learning is also required.
Apache Spark is an alternative that can efficiently handle these large-scale workloads.
This updated edition, which includes Spark 3.x, shows data engineers and data scientists why Spark's architecture and integration are important.
We systematically explain how to perform data analysis from simple to complex and how to use machine learning algorithms.
Step-by-step exercises, code examples, notebooks, and more help you:
■ Learning high-level structured APIs using Python, SQL, Scala, and Java
■ Understanding Spark Jobs and the SQL Engine
■ Inspect, tune, and debug Spark jobs using Spark configuration and Spark UI
■ Connect to data sources such as JSON, Parque, CSV, Avro, ORC, Hive, S3, and Kafka
■ Perform analytics on batch and streaming data using structured streaming
■ Building a stable data pipeline with open source Delta Lake and Spark
■ Develop machine learning pipelines using MLlib and reproduce and deploy models using MLflow.
GOODS SPECIFICS
- Publication date: June 24, 2022
- Page count, weight, size: 404 pages | 788g | 188*257*20mm
- ISBN13: 9791191600889
- ISBN10: 1191600882
You may also like
카테고리
korean
korean
![GQ KOREA Mark (Monthly): December [2025]](http://librairie.coreenne.fr/cdn/shop/files/8ef265dbbfbf186523ed75ba7319009d.jpg?v=1765340328&width=3840)
