arrow_back
Introduction
Introduction
What are the key features of Apache Spark that you like?
Which all kind of data processing supported by Spark?
What are benefits of Spark over MapReduce?
What does a Spark Engine do?
Spark Setup, Deployment, and Execution Modes
In which situation you will use Client mode and Cluster mode?
Do you need to install Spark on all nodes of Yarn cluster while running Spark?
How to stop a Running Spark Application
How to limit the number of retries on Spark job failure in YARN?
Is there any way to get Spark Application id, while running a job?
PySpark Code - Getting the Application ID
Where are logs in Spark on YARN How to view those logs?
How to prevent Spark Executors from getting Lost when using YARN client mode?
What is mount points? Why do you use it? in Databricks
PySpark Code - Using DBFS Mount Points
How can I run Spark on a cluster?
PySpark Code - spark-submit for Python
Spark Core Concepts – RDDs, Transformations, and Actions
How do you define RDD?
Explain about transformations and actions in the context of RDDs?
What is Lazy evaluated RDD mean?
What happens to RDD when one of the nodes on which it is distributed goes down?
What is the difference between map and flatMap and a good use case for each?
PySpark Code: map vs. flatMap
How to print the contents of RDD?
PySpark Code: Printing RDD Contents Safely
How to read multiple text files into a single RDD?
PySpark Code: Reading Multiple Files into an RDD
Explain sortByKey() operation
PySpark Code: sortByKey Usage
How would you control the number of partitions of a RDD?
PySpark Code: Partition Control (repartition vs coalesce)
DataFrames and Spark SQL
What is DataFrames?
PySpark Code: Creating a Basic DataFrame
What are the advantages of DataFrame?
What is Spark SQL and how does it differ from Hive?
What are the various data sources available in SparkSQL?
PySpark Code: Reading Various Data Sources
Does SparkSQL support subquery?
How to change column types in Spark SQL DataFrame?
PySpark Code: Casting Column Types
How to replace NULL value in Spark Dataframe?
PySpark Code: Handling NULL Values
How to add a constant column in a Spark DataFrame?
PySpark Code: Adding a Constant Column (lit)
How to add an index Column in Spark Dataframe?
PySpark Code: Adding an Index Column
How to concatenate columns in Spark Dataframe?
PySpark Code: Concatenating Columns
Is there any way for Spark to create primary keys?
PySpark Code: Generating Surrogate Primary Keys
File Formats, Storage, and Data Partitioning
List the advantages of Parquet file in Apache Spark
PySpark Code: Reading and Writing Parquet
How does Spark partition work on files in HDFS?
How to compress spark output written to HDFS in Standalone mode?
PySpark Code: Configuring Output Compression
Difference between partition and bucketing? in Apache Spark
PySpark Code: Partitioning vs. Bucketing
How to deal with a 100 GB table joined with a 1 GB table?
PySpark Code: Implementing a Broadcast Join
Spark Performance Optimization and Troubleshooting
How to get good performance with Spark?
What are the various levels of persistence in Apache Spark?
What is the difference between cache() and persist() method of RDD?
PySpark Code: cache() vs persist()
What is coalesce transformation?
What is Shuffling?
What is Speculative Execution of a tasks?
How to evaluate your Spark application?
Have you ever encountered Spark java.lang.OutOfMemoryError? How to fix it?
PySpark Code: Manual Salting for Skewed Joins
Do you know the top five secrets of performance tuning Apache Spark
Advanced Spark Concepts
What is Catalyst Optimizer? Explain with example.
PySpark Tip: Viewing the Catalyst Plan
What is Tungsten Project in Spark and how does it optimize execution?
What is WholeStageCodeGen in Spark SQL?
What is the difference between groupByKey and reduceByKey?
PySpark Code: groupByKey vs reduceByKey
How can you minimize data transfers when working with Spark?
What is the advantage of broadcasting values across Spark Cluster?
PySpark Code: Creating and Using Broadcast Variables
What is Broadcast Join and when should you use it?
What is the Default level of parallelism in Spark?
PySpark Code: Tuning Shuffle Partitions
Monitoring, Logs, and Spark UI
How to monitor and troubleshoot Spark jobs using Spark UI?
What does “Stage Skipped” mean in Spark web UI?
How do you disable Info Messages when running Spark Application?
PySpark Code: Changing Log Levels at Runtime
What are the different stages of query execution in Spark SQL?
Spark Streaming and Real-Time Processing
What is Apache Spark Streaming?
How Spark Streaming API works?
What do you understand by receivers in Spark Streaming?
What is DStream?
What is the significance of Sliding Window operation?
What is write-ahead log (journaling)?
What is Structured Streaming and how is it different from DStreams?
PySpark Code: Basic Structured Streaming (readStream & writeStream)
How do you implement watermarking in Structured Streaming?
How to handle late arriving data in Structured Streaming?
PySpark Code: Implementing Watermarks for Late Data
How does Spark integrate with Kafka for real-time streaming?
PySpark Code: Reading from Apache Kafka
Explain the role of checkpointing & stateful operations in Structured Streaming
PySpark Code: Enabling Checkpointing
Machine Learning and Graph Processing
What does MLlib do?
PySpark Code: Basic MLlib Pipeline
What is GraphX?
PySpark Note: GraphX vs. GraphFrames
What is PageRank?
Spark Architecture and Execution Flow
Define Spark architecture.
What is DAGScheduler and how it performs?
What are workers in Spark?
What happens when a job is submitted? (Job execution process)
What is Spark Driver?
Please define executors in detail.
What is stage, with regards to Spark Job execution?
What is checkpointing?
PySpark Code: Implementing Checkpointing
What is Data locality / placement?
Spark Integrations and Ecosystem Tools
Can you use Spark to access and analyse data stored in Cassandra?
PySpark Code: Reading from Apache Cassandra
Is it possible to run Apache Spark on Apache Mesos?
How to read a AWS S3 file in Spark?
PySpark Code: Reading from AWS S3
Do I need Hadoop to run Spark?
How does Spark relate to Apache Hadoop?
Who is using Spark in production?
Scenario-Based and Troubleshooting Questions
Scenario Based Question (Memory Management)
Scenario Based Question (Cache)
Scenario Based Question (Cluster)
Scenario Based Question (Recovery)
I’ve got big RDD (1 GB) in Yarn cluster. I can’t use collect(). How to handle?
PySpark Code: Safely Inspecting Large Datasets
Why does a job fail with “No space left on device,” but df says otherwise?
Certain data used again and again — how to improve performance?
While processing CSV, resultant output is multiple files — wanted a single file?
PySpark Code: Writing to a Single File
How to remove parentheses from output?
PySpark Code: Formatting RDD Output
What are possible reasons for TimeoutException?
PySpark Code: Handling Timeout Exceptions
Modern Spark Features – AQE, Delta Lake & More
What is Adaptive Query Execution (AQE) and how does it help?
How do you enable AQE and where is it useful?
PySpark Code: Configuring Adaptive Query Execution (AQE)
What is Dynamic Partition Pruning?
What is Delta Lake and why is it important?
What are the differences between Parquet and Delta Lake?
Explain ACID transactions in Delta Lake
How does Delta Lake handle schema evolution?
PySpark Code: Schema Enforcement and Evolution
How to implement Slowly Changing Dimensions (SCD) with Delta Lake
PySpark Code: Upserts (MERGE INTO) with Delta Lake
Miscellaneous and Community Insights
Does Spark require modified versions of Scala or Python?
Does my data need to fit in memory to use Spark?
How large a cluster can Spark scale to?
What is the role of Spark Accumulators?
PySpark Code: Using Accumulators
Is it possible to have multiple SparkContext in a single JVM?
Which all cluster managers can be used with Spark?
Name some sources from where Spark Streaming can process real-time
Name some companies that are already using Spark Streaming.
What is the difference between Apache Spark and Apache Storm?
What is the difference between Apache Spark and Apache Flink?
How to read a AWS S3 file in Spark?
Which all are the ways to configure Spark Properties and order them?
I want to find the moving average of the Time Series using Apache Spark
PySpark Code: Calculating a Moving Average
What does Stage Skipped mean in Apache Spark web UI?
What is Apache Spark Streaming?
Why Spark is good at low-latency iterative workloads?
We understand Spark Streaming uses micro-batching. Does this increase latency?