Setting YARN queue in PySpark

You can use below argument in you spark-submit command.

--queue queue_name

You can set this property in your code. spark.yarn.queue

Hope this will help.

Thanks

Try to use spark.yarn.queue rather than queue.

conf = pyspark.SparkConf().set("spark.yarn.queue", "your_queue_name")
sc

Related