Adds SPARK_DAEMON_MEMORY (driver memory) config var

Before we only set spark.driver.memory in spark-defaults.sh. This acutally only sets the driver memory for the executed job! SPAR_DAEMON_MEMORY sets the driver/master memory on startup. This should not have affected any previous benchmarks as 1. The default is 1gb 2. It only needs to be increased in case of very huge collects and broadcasts
peelframework · Aug 3, 2016 · 6102a8a · 6102a8a
1 parent 9b25872
commit 6102a8a
Show file tree

Hide file tree

Showing 2 changed files with 3 additions and 0 deletions.
diff --git a/peel-extensions/src/main/resources/reference.spark.conf b/peel-extensions/src/main/resources/reference.spark.conf
@@ -29,13 +29,15 @@ system {
                 SPARK_WORKER_CORES = ${system.default.config.parallelism.per-node}
                 SPARK_EXECUTOR_CORES = ${system.default.config.parallelism.per-node}
                 SPARK_EXECUTOR_MEMORY = "512m"
+                SPARK_DAEMON_MEMORY = "1024m"
                 # Enables periodic cleanup of worker / application dirs every 5 min. for data older than 1 hour.
                 SPARK_WORKER_OPTS = """"-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=300 -Dspark.worker.cleanup.appDataTtl=3600""""
             }
             # spark-defaults.conf entries
             defaults {
                 spark.master = "spark://"${runtime.hostname}":7077"
                 spark.executor.memory = "512m"
+                spark.driver.memory = "1024m"
                 spark.eventLog.enabled = "true"
                 spark.eventLog.dir = "file://"${system.spark.path.log}
                 spark.local.dir = "/tmp/spark"

diff --git a/peel-extensions/src/main/resources/templates/spark/conf/spark-env.sh.mustache b/peel-extensions/src/main/resources/templates/spark/conf/spark-env.sh.mustache
@@ -38,6 +38,7 @@
 {{#SPARK_MASTER_OPTS}}SPARK_MASTER_OPTS={{SPARK_MASTER_OPTS}}{{/SPARK_MASTER_OPTS}}{{^SPARK_MASTER_OPTS}}# - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y"){{/SPARK_MASTER_OPTS}}
 {{#SPARK_WORKER_CORES}}SPARK_WORKER_CORES={{SPARK_WORKER_CORES}}{{/SPARK_WORKER_CORES}}{{^SPARK_WORKER_CORES}}# - SPARK_WORKER_CORES, to set the number of cores to use on this machine{{/SPARK_WORKER_CORES}}
 {{#SPARK_WORKER_MEMORY}}SPARK_WORKER_MEMORY={{SPARK_WORKER_MEMORY}}{{/SPARK_WORKER_MEMORY}}{{^SPARK_WORKER_MEMORY}}# - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g){{/SPARK_WORKER_MEMORY}}
+{{#SPARK_DAEMON_MEMORY}}SPARK_DAEMON_MEMORY={{SPARK_DAEMON_MEMORY}}{{/SPARK_DAEMON_MEMORY}}{{^SPARK_DAEMON_MEMORY}}# - SPARK_DAEMON_MEMORY, to set how much total memory the master has (e.g. 1000m, 2g){{/SPARK_DAEMON_MEMORY}}
 {{#SPARK_WORKER_PORT}}SPARK_WORKER_PORT={{SPARK_WORKER_PORT}}{{/SPARK_WORKER_PORT}}{{^SPARK_WORKER_PORT}}# - SPARK_WORKER_PORT, to use non-default ports for the worker{{/SPARK_WORKER_PORT}}
 {{#SPARK_WORKER_WEBUI_PORT}}SPARK_WORKER_WEBUI_PORT={{SPARK_WORKER_WEBUI_PORT}}{{/SPARK_WORKER_WEBUI_PORT}}{{^SPARK_WORKER_WEBUI_PORT}}# - SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker{{/SPARK_WORKER_WEBUI_PORT}}
 {{#SPARK_WORKER_INSTANCES}}SPARK_WORKER_INSTANCES={{SPARK_WORKER_INSTANCES}}{{/SPARK_WORKER_INSTANCES}}{{^SPARK_WORKER_INSTANCES}}# - SPARK_WORKER_INSTANCES, to set the number of worker processes per node{{/SPARK_WORKER_INSTANCES}}