Posts

Showing posts with the label Optimistic Concurrency Control

Master Jobs, Stages, and Tasks for Data Engineering Interviews

By -
Image
Mastering Spark execution internals is a "must-have" skill for Data Engineers. Whether you are prepping for an interview or debugging a slow production pipeline, understanding how Spark breaks down your code is the key to performance tuning. Spark applications follow a strict hierarchy: Jobs > Stages > Tasks . Let’s break down exactly how this works. 1. High-Level Architecture Before we dive into the code, let’s look at the components that manage the execution: Driver: The brain. It converts your code into a Directed Acyclic Graph (DAG) and schedules tasks. DAG Scheduler: Splits the graph into Stages based on "shuffles." Task Scheduler: Sends the individual Tasks to the executors. Executors: The workers that actually run the tasks in parallel. 2. Real-World Code Walkthrough: The "Wide" Transformation Let’s analyze a common scenario: reading data, filtering, grouping, and saving. # 1. Read Data (Narrow) df = sp...
Post a Comment
Read more

How Delta Lake Prevents Conflicting Writes Using Optimistic Concurrency Control

By -
Delta Lake ensures reliable data operations by using optimistic concurrency control (OCC) . This mechanism prevents conflicting writes when multiple jobs or users attempt to update the same table simultaneously. Instead of locking resources, Delta Lake relies on its transaction log and version checks to guarantee consistency. Listen here about conflicting write in Delta Lake What is Optimistic Concurrency Control? Optimistic concurrency control assumes that most transactions will not conflict. Each writer reads the current table state, performs its changes, and then attempts to commit. Before committing, Delta Lake verifies against the transaction log that the underlying data has not changed since the read. If a conflict is detected, the write fails, and the user can retry safely. Why OCC is Better Than Locks Scalability: No need for heavy locking across distributed systems. Performance: Writers proceed in parallel without waiting for locks. Safety...
Post a Comment
Read more