Troubleshooting PySparkTypeError: [NOT_ITERABLE]

Troubleshooting PySparkTypeError:

 [NOT_ITERABLE] 



Ever encountered a confusing error message like 'Column is not iterable' while working with PySpark? Here's a relatable scenario:

You're trying to find the highest salary from a list of employees using PySpark. You've got your DataFrame set up, but when you run your code, The error pops up, leaving you scratching your head.


The reason? You forgot a tiny but essential step: importing the 'max' function!


Here's the fix, in plain language:


# Don't forget to import the 'max' function!
from pyspark.sql.functions import max


# Now, let's find that maximum salary:
data = [("John", 2500.0), ("Alice", 3000.0), ("Bob", 2200.0)]
usr_data = spark.createDataFrame(data,["name","salary"])



+-----+----------+
| name|salary    |
+-----+----------+
| John| 2500.0   |
|Alice | 3000.0  |
| Bob | 2200.0   |
+-----+----------+

max_sal = usr_data.agg(max(usr_data['salary']).alias('max_sal'))
max_sal.show()









By adding that one line, you're back on track, finding the max salary without an obstacle.🚀

Comments

Post a Comment

Popular posts from this blog

Departmental Salary Insights: with SQL and PySpark

Image
Departmental Salary Insights: with SQL and PySpark   Description:  This article dives into the SQL and PySpark methodologies to identify employees who earn more than the average salary within their respective departments. INPUT: SQL QUERY: SELECT Name, Salary FROM ( SELECT Name, Salary, AVG (Salary) OVER ( PARTITION BY Department) AS avg_sal FROM Employee ) AS department_salaries WHERE Salary > avg_sal; PYSPARK QUERY: from pyspark.sql import SparkSession from pyspark.sql.window import Window as w from pyspark.sql.functions import avg, col from pyspark.sql.types import StructType, StructField, StringType, DoubleType # Define the schema for the DataFrame schema = StructType([ StructField( "EmpID" , StringType(), True ), StructField( "Name" , StringType(), True ), StructField( "Department" , StringType(), True ), StructField( "Salary" , DoubleType(), True ) ]) # Initialize a Spark session spark = SparkSessio...
Read more

Handling a PySpark TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’

Image
Handling a PySpark TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’ You’ve been working with PySpark, trying to calculate the rolling sum of a column in your DataFrame, and you’ve set everything up perfectly. However, when you run your code, you face the dreaded TypeError: ‘unsupported operand type(s) for +: ‘int’ and ‘str’ error. The Reason: The error indicates that there’s a type mismatch in your DataFrame. Specifically, you’re trying to perform arithmetic operations on columns with different data types — an integer (int) and a string (str). In this scenario, you forgot a tiny but essential step: importing the ‘ sum ’ function! Here is the sample Hospital data, from pyspark.sql.window import Window as w from pyspark.sql.functions import col df_hospital = spark.read.options(header= 'True' ,inferSchema= 'True' ).csv( 'C:/Users/hp/Python_Sums_20thfeb/pyspark/hospital.csv' ) ovr_sp = w.partitionBy( 'department' ).orderBy( 'discharge_...
Read more