Missing SPARK_HOME when using SparkLauncher on AWS EMR cluster

As of emr-4.0.0, all applications on EMR are in /usr/lib. Spark is in /usr/lib/spark.


I find out that Spark on AWS EMR (tested with version emr-5.23.0 & emr-5.22.0) doesn't install Spark on EMR CORE Nodes. Just check the EMR nodes installation on /usr/lib/spark, it's not really a SPARK_HOME like the one installed on the EMR MASTER node.

Installing Spark on EMR CORE Nodes solved my issue.

Tags:

Amazon Web Services

Amazon Emr

Emr

Apache Spark

Pyspark

Related

How to remove all element from array except the first one in javascript ESLint dollar($) is not defined. (no-undef) Override TestRun Parameters with value including semi colons in Visual Studio Team Services Huge files in Docker containers Is there a way to assign values to an Option Base 1 array in VBA using a single line? IllegalStateException in MediaPlayer set datasource method Calculate difference between two dates using Carbon and Blade docker: driver failed programming external connectivity on endpoint webserver How to execute background task when Android app is closed / set to background? Add margin row totals in dplyr chain How to add a custom domain for a serverless-1.0.0 framework defined/deployed API? Differences between Native Script and react native and ionic framework

Recent Posts