Difference between distinct() and dropDuplicates()

 Difference between distinct() and  dropDuplicates()

distinct()

Checks the entire row, if all columns are same between two or more rows, then it considers the first row alone. In other words, it returns distinct rows based on the values of all columns in the DataFrame.

dropDuplicates(subset:optional)

It is more versatile than distinct() as it can be used to pick distinct values from specific columns as well. When no argument is passed to dropDuplicates(), it performs the same task as distinct(), checking the entire row. However, you can also specify a subset of columns to consider when looking for duplicates.

Example code:

distinct():

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('test').getOrCreate()
sc = spark.sparkContext
sales_data = [
    (1, "Product A", 10, 5.0, 50.0),
    (2, "Product B", 8, 7.5, 60.0),
    (1, "Product A", 10, 5.0, 50.0),
    (4, "Product D", 15, 8.0, 120.0),
    (5, "Product E", 20, 10.0, 200.0),
    (1, "Product E", 20, 10.0, 200.0)
]

df = spark.createDataFrame(sales_data, ["product_id", "product_name", "quantity_sold", "price_per_unit", "total_revenue"])


df.show()
df.distinct().show()

If you see, only record with product_id 1 is not present. It is because the last record with product_id 1 has different values in other columns.

dropDuplicates()

df.dropDuplicates().show()

This line performs the same task as the distinct, However we can pass subset here.

dropDuplicates(subset)

df.dropDuplicates(['product_id']).show()

If you run this query, only the first record with product_id 1 will be considered and other records will be neglected. This is because spark just checks for duplicates in the product_id column. We can even pass more than two columns inside the subset.

Conclusion

In summary, distinct() and dropDuplicates() are both useful methods for handling duplicate data in PySpark DataFrames. distinct() checks for duplicates in all columns, while dropDuplicates() allows you to specify a subset of columns to consider. Understanding the differences between these methods can help you choose the right one for your data processing needs.

Comments

Post a Comment

Popular posts from this blog

Departmental Salary Insights: with SQL and PySpark

Image
Departmental Salary Insights: with SQL and PySpark   Description:  This article dives into the SQL and PySpark methodologies to identify employees who earn more than the average salary within their respective departments. INPUT: SQL QUERY: SELECT Name, Salary FROM ( SELECT Name, Salary, AVG (Salary) OVER ( PARTITION BY Department) AS avg_sal FROM Employee ) AS department_salaries WHERE Salary > avg_sal; PYSPARK QUERY: from pyspark.sql import SparkSession from pyspark.sql.window import Window as w from pyspark.sql.functions import avg, col from pyspark.sql.types import StructType, StructField, StringType, DoubleType # Define the schema for the DataFrame schema = StructType([ StructField( "EmpID" , StringType(), True ), StructField( "Name" , StringType(), True ), StructField( "Department" , StringType(), True ), StructField( "Salary" , DoubleType(), True ) ]) # Initialize a Spark session spark = SparkSessio...
Read more

Handling a PySpark TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’

Image
Handling a PySpark TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’ You’ve been working with PySpark, trying to calculate the rolling sum of a column in your DataFrame, and you’ve set everything up perfectly. However, when you run your code, you face the dreaded TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’ error. The Reason: The error indicates that there’s a type mismatch in your DataFrame. Specifically, you’re trying to perform arithmetic operations on columns with different data types — an integer (int) and a string (str). In this scenario, you forgot a tiny but essential step: importing the ‘ sum ’ function! Here is the sample Hospital data, from pyspark.sql.window import Window as w from pyspark.sql.functions import col df_hospital = spark.read.options(header= 'True' ,inferSchema= 'True' ).csv( 'C:/Users/hp/Python_Sums_20thfeb/pyspark/hospital.csv' ) ovr_sp = w.partitionBy( 'department' ).orderBy( 'discharge_...
Read more

Troubleshooting PySparkTypeError: [NOT_ITERABLE]

Image
Troubleshooting PySparkTypeError:  [NOT_ITERABLE]   Ever encountered a confusing error message like 'Column is not iterable' while working with PySpark? Here's a relatable scenario: You're trying to find the highest salary from a list of employees using PySpark . You've got your DataFrame set up, but when you run your code, The error pops up, leaving you scratching your head. The reason? You forgot a tiny but essential step: importing the ' max ' function! Here's the fix, in plain language: # Don't forget to import the ' max ' function! from pyspark.sql.functions import max # Now, let's find that maximum salary: data = [("John", 2500.0), ("Alice", 3000.0), ("Bob", 2200.0)] usr_data = spark.createDataFrame(data,["name","salary"]) +-----+----------+ | name|salary    | +-----+----------+ | John| 2500.0   | |Alice | 3000.0  | | Bob | 2200.0   | +-----+----------+ max_sal = usr_data.agg(max(us...
Read more