Running scheduled Spark job
You can use a cron tab, but really as you start having spark jobs that depend on other spark jobs i would recommend pinball for coordination. https://github.com/pinterest/pinball
To get a simple crontab working I would create wrapper script such as
#!/bin/bash
cd /locm/spark_jobs
export SPARK_HOME=/usr/hdp/2.2.0.0-2041/spark
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_USER_NAME=hdfs
export HADOOP_GROUP=hdfs
#export SPARK_CLASSPATH=$SPARK_CLASSPATH:/locm/spark_jobs/configs/*
CLASS=$1
MASTER=$2
ARGS=$3
CLASS_ARGS=$4
echo "Running $CLASS With Master: $MASTER With Args: $ARGS And Class Args: $CLASS_ARGS"
$SPARK_HOME/bin/spark-submit --class $CLASS --master $MASTER --num-executors 4 --executor-cores 4 $ARGS spark-jobs-assembly*.jar $CLASS_ARGS >> /locm/spark_jobs/logs/$CLASS.log 2>&1
Then create a crontab by
- crontab -e
- Insert 30 1 * * * /PATH/TO/SCRIPT.sh $CLASS "yarn-client"
Crontab is good enough only if you don't care about high availability, since it will run on a single machine that can fail.
The fact that you run in a stand alone mode indicate that you don't have hadoop and mesos installed, that have some tools to make this task more reliable.
An alternative to crontab (though it suffers from high availability issues as well at the moment) is airbnb's airflow. It was built for such use cases exactly (among others) see here: http://airflow.incubator.apache.org/scheduler.html.
Mesos users can try using chronos which is a cron job for clusters: https://github.com/mesos/chronos.
There is also oozie that comes from the hadoop world http://blog.cloudera.com/blog/2013/01/how-to-schedule-recurring-hadoop-jobs-with-apache-oozie/.
If this is a mission critical, you can even program it yourself if you use consul/zookeper or other tools that provide leader election - just have your processes run on multiple machines, have them compete on leadership and make sure the leader submits the job to the spark.
You can use spark job server to make the job submission more elegant: https://github.com/spark-jobserver/spark-jobserver
There is no built-in mechanism in Spark that will help. A cron job seems reasonable for your case. If you find yourself continuously adding dependencies to the scheduled job, try Azkaban.