Pass variables from Scala to Python in Databricks

The reverse will pretty much the same. In Scala:

scalaDF.registerTempTable("some_table")

In Python:

spark.table("some_table")

If you use recent Spark version you should use createOrReplaceTempView in place of registerTempTable.

You can make use of the .createOrReplaceTempView() method or sql().

Here is an example to pass a dataframe through from scala, python, onto sql with a modification along the way ...and back to scala.

%scala 
var df = spark.range(0,10).selectExpr("*","'scala' language_origin")
df.createOrReplaceTempView("tableName")
display(df)

%python
df = sql("select * from tableName")
df2 = df.selectExpr("*","'python' language_added")
df2.createOrReplaceTempView("tableName")
display(df2)

%sql
create or replace temp view tableName as
select *, 'sql' language_added from tableName;
select * from tableName

%scala
df = sql("select * from tableName")
display(df)