Posts

Showing posts with the label DataCleaning

Master Jobs, Stages, and Tasks for Data Engineering Interviews

By -
Image
Mastering Spark execution internals is a "must-have" skill for Data Engineers. Whether you are prepping for an interview or debugging a slow production pipeline, understanding how Spark breaks down your code is the key to performance tuning. Spark applications follow a strict hierarchy: Jobs > Stages > Tasks . Let’s break down exactly how this works. 1. High-Level Architecture Before we dive into the code, let’s look at the components that manage the execution: Driver: The brain. It converts your code into a Directed Acyclic Graph (DAG) and schedules tasks. DAG Scheduler: Splits the graph into Stages based on "shuffles." Task Scheduler: Sends the individual Tasks to the executors. Executors: The workers that actually run the tasks in parallel. 2. Real-World Code Walkthrough: The "Wide" Transformation Let’s analyze a common scenario: reading data, filtering, grouping, and saving. # 1. Read Data (Narrow) df = sp...
Post a Comment
Read more

Techniques for handling skewed data in PySpark

By -
 Techniques for handling skewed data in PySpark : Handling skewed data in PySpark is crucial for preventing performance bottlenecks and ensuring efficient data processing. Here are some techniques to address data skew: Identify Skewed Keys : Use data profiling or analysis to identify keys or values that are skewed. Skewness can occur in join keys, group-by keys, or any other key-based operation. Preprocessing : Preprocess skewed data to reduce skewness. For example, you can use techniques like salting, where you add a random suffix to keys to distribute the data more evenly across partitions. Custom Partitioning : Use custom partitioning techniques to redistribute skewed data. For example, you can implement a custom partitioner that identifies skewed keys and redistributes them across partitions more evenly. Filtering : If possible, filter out skewed data early in your pipeline to reduce the impact of skewness on downstream operations. This can help reduce the amount of skewed data...
Post a Comment
Read more

NVL vs COALESCE

By -
  NVL vs COALESCE to handle NULL values in SQL. Both NVL and COALESCE are used in SQL to handle null values, but they have some differences: Syntax: NVL takes two arguments, while COALESCE takes two or more arguments. Return value: NVL returns the first argument if it is not null, otherwise it returns the second argument. COALESCE returns the first non-null value from its arguments. Here are some examples to illustrate the differences: NVL Example: SELECT NVL(NULL, 'hello') FROM dual; This will return 'hello', since the first argument is null. SELECT NVL('world', 'hello') FROM dual; This will return 'world', since the first argument is not null. COALESCE Example: SELECT COALESCE(NULL, NULL, 'hello', 'world') FROM dual; This will return 'hello', since it is the first non-null value. SELECT COALESCE(NULL, 'hello', 'world') FROM dual; This will also return 'hello', since it is the first non-null value...
Post a Comment
Read more