Kunal Anand Kunal Anand - 1 month ago 13
Python Question

Getting int() argument must be a string or a number, not 'Column'- Apache Spark

I am getting this exception if I use the following code :


int() argument must be a string or a number, not 'Column'



df= df.withColumn('FY',
F.when((df['ID'].substr(5,2).isin({'11','12'})),int(df['ID'].substr(1,4))+1).
otherwise(int(df['ID'].substr(1,4))))


Basically I want to add 1 to the result if the result isin 11 or 12 otherwise just uder the ID. Please help, I am fairly new to Python.

Answer

Use:

df.withColumn('FY',F.when(df['ID'].substr(5,2).isin({'11','12'}),
  df['ID'].substr(1,4).cast("integer") + 1).
  otherwise(df['ID'].substr(1,4)).cast("integer"))