Read classpath entries from SPARK_EXTRA_CLASSPATH on executors. #383
Conversation
This makes executors consistent with the driver. Note that SPARK_EXTRA_CLASSPATH isn't set anywhere by Spark itself, but it's primarily meant to be set by images that inherit from the base driver/executor images.
There was a problem hiding this comment.
This reduces an inconsistency between the driver and executor docker images. We're in the process of baking an entire application into the docker images, and roughly the way we're doing it is creating a new Dockerfile for both driver and executor, inheriting from the published Spark Dockerfiles, adding jars into a path on the image, and setting ENV SPARK_EXTRA_CLASSPATH on the images.
However there is an inconsistency between the driver and executor Dockerfiles -- driver uses SPARK_EXTRA_CLASSPATH whereas the executor uses SPARK_EXECUTOR_EXTRA_CLASSPATH.
To eliminate this inconsistency, without breaking compatibility for consumers currently relying on SPARK_EXECUTOR_EXTRA_CLASSPATH, here we add support for SPARK_EXTRA_CLASSPATH on the executor as well.
|
+1. I ran into this issue the other day when I put /etc/hadoop/conf in a custom docker images and wanted put the dir in the classpath. Thanks for fixing this! |
|
|
This makes executors consistent with the driver. Note that SPARK_EXTRA_CLASSPATH isn't set anywhere by Spark itself, but it's primarily meant to be set by images that inherit from the base driver/executor images.
Apply patches for SPARK-24531 to fix tests
…he-spark-on-k8s#383) This makes executors consistent with the driver. Note that SPARK_EXTRA_CLASSPATH isn't set anywhere by Spark itself, but it's primarily meant to be set by images that inherit from the base driver/executor images.
This makes executors consistent with the driver. Note that SPARK_EXTRA_CLASSPATH isn't set anywhere by Spark itself, but it's primarily meant to be set by images that inherit from the base driver/executor images.