Master Jobs, Stages, and Tasks for Data Engineering Interviews

By -
Image
Mastering Spark execution internals is a "must-have" skill for Data Engineers. Whether you are prepping for an interview or debugging a slow production pipeline, understanding how Spark breaks down your code is the key to performance tuning. Spark applications follow a strict hierarchy: Jobs > Stages > Tasks . Let’s break down exactly how this works. 1. High-Level Architecture Before we dive into the code, let’s look at the components that manage the execution: Driver: The brain. It converts your code into a Directed Acyclic Graph (DAG) and schedules tasks. DAG Scheduler: Splits the graph into Stages based on "shuffles." Task Scheduler: Sends the individual Tasks to the executors. Executors: The workers that actually run the tasks in parallel. 2. Real-World Code Walkthrough: The "Wide" Transformation Let’s analyze a common scenario: reading data, filtering, grouping, and saving. # 1. Read Data (Narrow) df = sp...
Post a Comment

NVL vs COALESCE

By -

 NVL vs COALESCE to handle NULL values in SQL.



Both NVL and COALESCE are used in SQL to handle null values, but they have some differences:

Syntax: NVL takes two arguments, while COALESCE takes two or more arguments.

Return value: NVL returns the first argument if it is not null, otherwise it returns the second argument. COALESCE returns the first non-null value from its arguments.

Here are some examples to illustrate the differences:

NVL Example:


SELECT NVL(NULL, 'hello') FROM dual;
This will return 'hello', since the first argument is null.


SELECT NVL('world', 'hello') FROM dual;
This will return 'world', since the first argument is not null.

COALESCE Example:

SELECT COALESCE(NULL, NULL, 'hello', 'world') FROM dual;

This will return 'hello', since it is the first non-null value.


SELECT COALESCE(NULL, 'hello', 'world') FROM dual;
This will also return 'hello', since it is the first non-null value.

Hope it helps.

#sql #null #handling

Comments

Post a Comment

Popular posts from this blog

Master Jobs, Stages, and Tasks for Data Engineering Interviews

By -
Image
Mastering Spark execution internals is a "must-have" skill for Data Engineers. Whether you are prepping for an interview or debugging a slow production pipeline, understanding how Spark breaks down your code is the key to performance tuning. Spark applications follow a strict hierarchy: Jobs > Stages > Tasks . Let’s break down exactly how this works. 1. High-Level Architecture Before we dive into the code, let’s look at the components that manage the execution: Driver: The brain. It converts your code into a Directed Acyclic Graph (DAG) and schedules tasks. DAG Scheduler: Splits the graph into Stages based on "shuffles." Task Scheduler: Sends the individual Tasks to the executors. Executors: The workers that actually run the tasks in parallel. 2. Real-World Code Walkthrough: The "Wide" Transformation Let’s analyze a common scenario: reading data, filtering, grouping, and saving. # 1. Read Data (Narrow) df = sp...
Read more

RDD vs DATAFRAME vs DATASET

By -
Image
  Spark - RDD, Dataframe and Dataset!! Let's start with RDDs (Resilient Distributed Datasets).  Q  Explain what an RDD is and its role in distributed computing? RDD : An RDD is a fundamental data structure in Apache Spark, designed to handle large-scale data processing across clusters. It represents an immutable, partitioned collection of records that can be operated on in parallel. RDDs provide fault tolerance through lineage information, enabling recomputation of lost data partitions. Q  How does Spark's RDD differ from traditional data structures like arrays or lists?  Unlike arrays or lists, RDDs are distributed across multiple nodes in a cluster, allowing for parallel processing and fault tolerance. RDDs are immutable, meaning their contents cannot be changed once created. Operations on RDDs are lazily evaluated, allowing Spark to optimize execution plans and perform transformations efficiently. Q  Moving on to dataframes in Spark. What is a dataf...
Read more

How Delta Lake Prevents Conflicting Writes Using Optimistic Concurrency Control

By -
Delta Lake ensures reliable data operations by using optimistic concurrency control (OCC) . This mechanism prevents conflicting writes when multiple jobs or users attempt to update the same table simultaneously. Instead of locking resources, Delta Lake relies on its transaction log and version checks to guarantee consistency. Listen here about conflicting write in Delta Lake What is Optimistic Concurrency Control? Optimistic concurrency control assumes that most transactions will not conflict. Each writer reads the current table state, performs its changes, and then attempts to commit. Before committing, Delta Lake verifies against the transaction log that the underlying data has not changed since the read. If a conflict is detected, the write fails, and the user can retry safely. Why OCC is Better Than Locks Scalability: No need for heavy locking across distributed systems. Performance: Writers proceed in parallel without waiting for locks. Safety...
Read more