Pfthb

I'm trying to typecast the columns in the data frame df_trial which has all the columns as string, based on an XML file I'm trying to type cast each column.

val columnList = sXml \ "COLUMNS" "COLUMN"
val df_trial = sqlContext.createDataFrame(rowRDD, schema_allString)
columnList.foreach(i => 
 var columnName = (i \ "@ID").text.toLowerCase()
 var dataType = (i \ "@DATA_TYPE").text.toLowerCase()
 if (dataType == "number") 
 print("number")
 var DATA_PRECISION: Int = (i \ "@DATA_PRECISION").text.toLowerCase().toInt
 var DATA_SCALE: Int = (i \ "@DATA_SCALE").text.toLowerCase().toInt;
 var decimalvalue = "decimal(" + DATA_PRECISION + "," + DATA_SCALE + ")"
 val df_intermediate: DataFrame =
 df_trial.withColumn(s"$columnName",
 col(s"$columnName").cast(s"$decimalvalue"))
 val df_trial: DataFrame = df_intermediate
 else if (dataType == "varchar2") 
 print("varchar")
 var DATA_LENGTH = (i \ "@DATA_LENGTH").text.toLowerCase().toInt;
 var varcharvalue = "varchar(" + DATA_LENGTH + ")"
 val df_intermediate =
 df_trial.withColumn(s"$columnName",
 col(s"$columnName").cast(s"$varcharvalue"))
 val df_trial: DataFrame = df_intermediate
 else if (dataType == "timestamp") 
 print("time")
 val df_intermediate =
 df_trial.withColumn(s"$columnName", col(s"$columnName").cast("timestamp"))
 val df_trial: DataFrame = df_intermediate
 
);

In each branch of the if-else you're using the values called df_trial before you've defined them. You'll need to rearrange the code to define them first.

Note: the way you have it, the df_trial at the very top is not being used. Depending on what you are trying to do, you may want to change the first df_trial to a var and remove the val from the other usages. (This is probably still wrong since you will be overwriting the same variable multiple times as you loop over columnList).