About Me

 

Welcome to my Big Data Engineering blog! Here, I share what I learn about Data Engineering. Through this blog, I aim to share my insights, tips, and learnings that help me navigate the complex world of big data. I hope you find it helpful. Let's explore the world of data together!

Comments

Post a Comment

Popular posts from this blog

Departmental Salary Insights: with SQL and PySpark

Image
Departmental Salary Insights: with SQL and PySpark   Description:  This article dives into the SQL and PySpark methodologies to identify employees who earn more than the average salary within their respective departments. INPUT: SQL QUERY: SELECT Name, Salary FROM ( SELECT Name, Salary, AVG (Salary) OVER ( PARTITION BY Department) AS avg_sal FROM Employee ) AS department_salaries WHERE Salary > avg_sal; PYSPARK QUERY: from pyspark.sql import SparkSession from pyspark.sql.window import Window as w from pyspark.sql.functions import avg, col from pyspark.sql.types import StructType, StructField, StringType, DoubleType # Define the schema for the DataFrame schema = StructType([ StructField( "EmpID" , StringType(), True ), StructField( "Name" , StringType(), True ), StructField( "Department" , StringType(), True ), StructField( "Salary" , DoubleType(), True ) ]) # Initialize a Spark session spark = SparkSessio...
Read more

Troubleshooting PySparkTypeError: [NOT_ITERABLE]

Image
Troubleshooting PySparkTypeError:  [NOT_ITERABLE]   Ever encountered a confusing error message like 'Column is not iterable' while working with PySpark? Here's a relatable scenario: You're trying to find the highest salary from a list of employees using PySpark . You've got your DataFrame set up, but when you run your code, The error pops up, leaving you scratching your head. The reason? You forgot a tiny but essential step: importing the ' max ' function! Here's the fix, in plain language: # Don't forget to import the ' max ' function! from pyspark.sql.functions import max # Now, let's find that maximum salary: data = [("John", 2500.0), ("Alice", 3000.0), ("Bob", 2200.0)] usr_data = spark.createDataFrame(data,["name","salary"]) +-----+----------+ | name|salary    | +-----+----------+ | John| 2500.0   | |Alice | 3000.0  | | Bob | 2200.0   | +-----+----------+ max_sal = usr_data.agg(max(us...
Read more

Understanding Union() and UnionByName() in PySpark

🚀 Union() vs UnionByName() in PySpark   In PySpark, when working with DataFrames, union() and unionByName() are two methods used for merging data from multiple DataFrames.   Purpose of the two functions: o    union() in PySpark is used to merge two tables with the same column count irrespective of the column names. It simply appends the rows of the second table below the rows of the first table. o    unionByName() on the other hand is used to merge data from two tables based on their column names. It ensures that the columns are matched and appended accordingly.   Differences : o    Example 1: §   union() simply merges two tables without considering column names or data types. §   unionByName() merges tables based on column names, without considering the order of columns.   Example Code: spark = SparkSession.builder.appName("Union_Example").getOrCreate()   # Creating DataFrames m...
Read more