get min and max from a specific column scala spark dataframe
How about getting the column name from the metadata:
val selectedColumnName = df.columns(q) //pull the (q + 1)th column from the columns array
df.agg(min(selectedColumnName), max(selectedColumnName))
You can use pattern matching while assigning variable:
import org.apache.spark.sql.functions.{min, max}
import org.apache.spark.sql.Row
val Row(minValue: Double, maxValue: Double) = df.agg(min(q), max(q)).head
Where q is either a Column
or a name of column (String). Assuming your data type is Double
.