Understanding Union() and UnionByName() in PySpark

🚀 Union() vs UnionByName() in PySpark

 

In PySpark, when working with DataFrames, union() and unionByName() are two methods used for merging data from multiple DataFrames.

 

Purpose of the two functions:

o   union() in PySpark is used to merge two tables with the same column count irrespective of the column names. It simply appends the rows of the second table below the rows of the first table.

o   unionByName() on the other hand is used to merge data from two tables based on their column names. It ensures that the columns are matched and appended accordingly.

 

Differences:

o   Example 1:

§  union() simply merges two tables without considering column names or data types.

§  unionByName() merges tables based on column names, without considering the order of columns.

 Example Code:


spark = SparkSession.builder.appName("Union_Example").getOrCreate()

 

# Creating DataFrames

mob = spark.createDataFrame([('Jio', 120, 'Ambani'), ('iphone', 2, 'Steve')], ["Company", "id", "Owner"])

mob1 = spark.createDataFrame([('Mark', 34, 'Fb', 2012), ('Billgates', 2, 'Windows', 2002)], ["Owner", "id", "Company", "year"])

 

# Performing union

mob_union = mob.union(mob1)

mob_union.show()

 

# Performing unionByName

mob_unionByName = mob.unionByName(mob1)

mob_unionByName.show()

 

+-------+---+------+

|Company| id| Owner|

+-------+---+------+

|    Jio|120|Ambani|

| iphone|  2| Steve|

+-------+---+------+

 

+---------+---+-------+----+

|    Owner| id|Company|year|

+---------+---+-------+----+

|     Mark| 34|     Fb|2012|

|Billgates|  2|Windows|2002|

+---------+---+-------+----+

 

# Output for unionByName():

+---------+---+-------+----+

|    Owner| id|Company|year|

+---------+---+-------+----+

|     Mark| 34|     Fb|2012|

|Billgates|  2|Windows|2002|

|     Jio|120|   null|null|

|   Steve|  2| iphone|null|

+---------+---+-------+----+

 

# output of union(): AnalysisException: [NUM_COLUMNS_MISMATCH]

This is because union() can only be performed with a matching column count

 

 

Conclusion:

o   Use union() when you want to merge two tables with the same column count, regardless of column names.

o   Use unionByName() when you want to merge tables based on column names, ensuring that columns are matched and appended accordingly.

 

 

Comments

Post a Comment

Popular posts from this blog

Departmental Salary Insights: with SQL and PySpark

Image
Departmental Salary Insights: with SQL and PySpark   Description:  This article dives into the SQL and PySpark methodologies to identify employees who earn more than the average salary within their respective departments. INPUT: SQL QUERY: SELECT Name, Salary FROM ( SELECT Name, Salary, AVG (Salary) OVER ( PARTITION BY Department) AS avg_sal FROM Employee ) AS department_salaries WHERE Salary > avg_sal; PYSPARK QUERY: from pyspark.sql import SparkSession from pyspark.sql.window import Window as w from pyspark.sql.functions import avg, col from pyspark.sql.types import StructType, StructField, StringType, DoubleType # Define the schema for the DataFrame schema = StructType([ StructField( "EmpID" , StringType(), True ), StructField( "Name" , StringType(), True ), StructField( "Department" , StringType(), True ), StructField( "Salary" , DoubleType(), True ) ]) # Initialize a Spark session spark = SparkSessio...
Read more

Handling a PySpark TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’

Image
Handling a PySpark TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’ You’ve been working with PySpark, trying to calculate the rolling sum of a column in your DataFrame, and you’ve set everything up perfectly. However, when you run your code, you face the dreaded TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’ error. The Reason: The error indicates that there’s a type mismatch in your DataFrame. Specifically, you’re trying to perform arithmetic operations on columns with different data types — an integer (int) and a string (str). In this scenario, you forgot a tiny but essential step: importing the ‘ sum ’ function! Here is the sample Hospital data, from pyspark.sql.window import Window as w from pyspark.sql.functions import col df_hospital = spark.read.options(header= 'True' ,inferSchema= 'True' ).csv( 'C:/Users/hp/Python_Sums_20thfeb/pyspark/hospital.csv' ) ovr_sp = w.partitionBy( 'department' ).orderBy( 'discharge_...
Read more

Troubleshooting PySparkTypeError: [NOT_ITERABLE]

Image
Troubleshooting PySparkTypeError:  [NOT_ITERABLE]   Ever encountered a confusing error message like 'Column is not iterable' while working with PySpark? Here's a relatable scenario: You're trying to find the highest salary from a list of employees using PySpark . You've got your DataFrame set up, but when you run your code, The error pops up, leaving you scratching your head. The reason? You forgot a tiny but essential step: importing the ' max ' function! Here's the fix, in plain language: # Don't forget to import the ' max ' function! from pyspark.sql.functions import max # Now, let's find that maximum salary: data = [("John", 2500.0), ("Alice", 3000.0), ("Bob", 2200.0)] usr_data = spark.createDataFrame(data,["name","salary"]) +-----+----------+ | name|salary    | +-----+----------+ | John| 2500.0   | |Alice | 3000.0  | | Bob | 2200.0   | +-----+----------+ max_sal = usr_data.agg(max(us...
Read more