Big Data Engineering

  Difference between distinct() and  dropDuplicates() distinct() Checks the entire row, if all columns are same between two or more rows, then it considers the first row alone. In other words, it returns distinct rows based on the values of all columns in the DataFrame. dropDuplicates( subset:optional ) It is more versatile than  distinct()  as it can be used to pick distinct values from specific columns as well. When no argument is passed to  dropDuplicates() , it performs the same task as  distinct() , checking the entire row. However, you can also specify a subset of columns to consider when looking for duplicates. Example code: distinct(): from pyspark.sql import SparkSession spark = SparkSession.builder.appName( 'test' ).getOrCreate() sc = spark.sparkContext sales_data = [ ( 1 , "Product A" , 10 , 5.0 , 50.0 ), ( 2 , "Product B" , 8 , 7.5 , 60.0 ), ( 1 , "Product A" , 10 , 5.0 , 50.0 ), ( 4 , "Product D" , 15 , 8.0 ...

🚀 Union() vs UnionByName() in PySpark   In PySpark, when working with DataFrames, union() and unionByName() are two methods used for merging data from multiple DataFrames.   Purpose of the two functions: o    union() in PySpark is used to merge two tables with the same column count irrespective of the column names. It simply appends the rows of the second table below the rows of the first table. o    unionByName() on the other hand is used to merge data from two tables based on their column names. It ensures that the columns are matched and appended accordingly.   Differences : o    Example 1: §   union() simply merges two tables without considering column names or data types. §   unionByName() merges tables based on column names, without considering the order of columns.   Example Code: spark = SparkSession.builder.appName("Union_Example").getOrCreate()   # Creating DataFrames m...

Repartition() vs Coalesce() in PySpark In PySpark, optimizing the number of partitions in a DataFrame is essential for performance. repartition() and coalesce() are two methods commonly used for this purpose.   1. Purpose:    - repartition() can both increase and decrease the number of partitions, whereas coalesce() is specifically used to decrease the number of partitions. These functions aim to enhance performance and reduce execution time.   2. Differences:    - Scenario: Suppose there are 3 partitions, and you want only 2 partitions.      - repartition() shuffles all partitions, distributing the data into 2 partitions to ensure balanced shuffling.      - coalesce() takes the data from the 3rd partition and places it into the 1st and 2nd partitions without balancing. It may allocate, for example, 70% of the 3rd partition to the 1st and 30% to the 2nd partition.   3. Example Code:    ```PySpark  ...