Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

[WIP] Use HDFS Delegation Token in driver/executor pods as part of Secure HDFS Support #379

Merged
merged 5 commits into from
Aug 1, 2017
Merged

[WIP] Use HDFS Delegation Token in driver/executor pods as part of Secure HDFS Support #379

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
48533ff
Mount a hadoop secret in the driver pod
kimoonkim Jul 18, 2017
3c3331a
Fix compile error
kimoonkim Jul 18, 2017
f2a4033
Mount a hadoop secret in the executor pod
kimoonkim Jul 18, 2017
aa04b04
Document the experimental config option
kimoonkim Jul 20, 2017
0141c0a
Address review comments and fix styles
kimoonkim Jul 20, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions ...e-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/kubernetes/config.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,18 @@ package object config extends Logging {
private[spark] val CLIENT_CERT_FILE_CONF_SUFFIX = "clientCertFile"
private[spark] val CA_CERT_FILE_CONF_SUFFIX = "caCertFile"

// TODO: This option is intended to be used for internal prototype only until the submission
// client automatically creates the secret file. Remove this option afterward
// unless other use is found.
private[spark] val MOUNTED_HADOOP_SECRET_CONF =
ConfigBuilder("spark.kubernetes.mounted.hadoopSecret")
.doc("Use a Kubernetes secret containing Hadoop tokens such as an HDFS delegation token." +
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make the key configurable as well so that it doesn't always have to be "hadoop-token-file".

" The secret should have an entry named 'hadoop-token-file' under the data section," +
" which contains binary dumps of Hadoop tokens.")
.internal()
.stringConf
.createOptional

private[spark] val RESOURCE_STAGING_SERVER_USE_SERVICE_ACCOUNT_CREDENTIALS =
ConfigBuilder(
s"$APISERVER_AUTH_RESOURCE_STAGING_SERVER_CONF_PREFIX.useServiceAccountCredentials")
Expand Down
8 changes: 8 additions & 0 deletions ...anagers/kubernetes/core/src/main/scala/org/apache/spark/deploy/kubernetes/constants.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,13 @@ package object constants {
s"$DRIVER_CREDENTIALS_SECRETS_BASE_DIR/$DRIVER_CREDENTIALS_OAUTH_TOKEN_SECRET_NAME"
private[spark] val DRIVER_CREDENTIALS_SECRET_VOLUME_NAME = "kubernetes-credentials"

// Hadoop credentials secrets for the Spark app.
private[spark] val SPARK_APP_HADOOP_CREDENTIALS_BASE_DIR = "/mnt/secrets/hadoop-credentials"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put these into /etc/hadoop/ so that only root access will be allowed to this folder. thoughts and reasoning for location? Also, should this be customizable in config.scala

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In YARN, the token file is placed into a work dir for the executors. So I don't think /etc/hadoop is a good location. And root access or not is irrelevant in pods since everything runs as root.

I chose this dir because we have the other code using it. I think it's better to follow the convention here. From line 30 above:

private[spark] val DRIVER_CREDENTIALS_SECRETS_BASE_DIR =
     "/mnt/secrets/spark-kubernetes-credentials"

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/mnt/secrets is good for the credentials. Use /etc/hadoop for configuration files.

private[spark] val SPARK_APP_HADOOP_TOKEN_FILE_SECRET_NAME = "hadoop-token-file"
private[spark] val SPARK_APP_HADOOP_TOKEN_FILE_PATH =
s"$SPARK_APP_HADOOP_CREDENTIALS_BASE_DIR/$SPARK_APP_HADOOP_TOKEN_FILE_SECRET_NAME"
private[spark] val SPARK_APP_HADOOP_SECRET_VOLUME_NAME = "hadoop-secret"

// Default and fixed ports
private[spark] val SUBMISSION_SERVER_PORT = 7077
private[spark] val DEFAULT_DRIVER_PORT = 7078
Expand All @@ -69,6 +76,7 @@ package object constants {
private[spark] val ENV_MOUNTED_FILES_DIR = "SPARK_MOUNTED_FILES_DIR"
private[spark] val ENV_PYSPARK_FILES = "PYSPARK_FILES"
private[spark] val ENV_PYSPARK_PRIMARY = "PYSPARK_PRIMARY"
private[spark] val ENV_HADOOP_TOKEN_FILE_LOCATION = "HADOOP_TOKEN_FILE_LOCATION"

// Bootstrapping dependencies with the init-container
private[spark] val INIT_CONTAINER_ANNOTATION = "pod.beta.kubernetes.io/init-containers"
Expand Down
4 changes: 3 additions & 1 deletion ...cala/org/apache/spark/deploy/kubernetes/submit/DriverConfigurationStepsOrchestrator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import org.apache.spark.SparkConf
import org.apache.spark.deploy.kubernetes.ConfigurationUtils
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.kubernetes.constants._
import org.apache.spark.deploy.kubernetes.submit.submitsteps.{BaseDriverConfigurationStep, DependencyResolutionStep, DriverConfigurationStep, DriverKubernetesCredentialsStep, InitContainerBootstrapStep, PythonStep}
import org.apache.spark.deploy.kubernetes.submit.submitsteps._
import org.apache.spark.deploy.kubernetes.submit.submitsteps.initcontainer.InitContainerConfigurationStepsOrchestrator
import org.apache.spark.launcher.SparkLauncher
import org.apache.spark.util.Utils
Expand Down Expand Up @@ -94,6 +94,7 @@ private[spark] class DriverConfigurationStepsOrchestrator(
submissionSparkConf)
val kubernetesCredentialsStep = new DriverKubernetesCredentialsStep(
submissionSparkConf, kubernetesResourceNamePrefix)
val hadoopCredentialsStep = new DriverHadoopCredentialsStep(submissionSparkConf)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need to be slightly refactored after merging this in after PR 373 because of the hadoopStepsOrchestrator that is leveraged.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know. Thanks.

val pythonStep = mainAppResource match {
case PythonMainAppResource(mainPyResource) =>
Option(new PythonStep(mainPyResource, additionalPythonFiles, filesDownloadPath))
Expand Down Expand Up @@ -131,6 +132,7 @@ private[spark] class DriverConfigurationStepsOrchestrator(
Seq(
initialSubmissionStep,
kubernetesCredentialsStep,
hadoopCredentialsStep,
dependencyResolutionStep) ++
initContainerBootstrapStep.toSeq ++
pythonStep.toSeq
Expand Down
55 changes: 55 additions & 0 deletions ...etes/core/src/main/scala/org/apache/spark/deploy/kubernetes/submit/HadoopSecretUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.kubernetes.submit

import io.fabric8.kubernetes.api.model.{Container, ContainerBuilder, Pod, PodBuilder}

import org.apache.spark.deploy.kubernetes.constants._

object HadoopSecretUtil {
Copy link
Member

@ifilonenko ifilonenko Jul 19, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason for why this needs to be a seperate util and cant just be as part of a step. where is this being re-used?

Copy link
Member

@ifilonenko ifilonenko Jul 19, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on another note the pod and container will be shared by the PodWithDetachedMainContainer() so it will need to be refactored when PRs are merged

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used by both the driver and executors.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #373 for how one can share a module between the driver and the executors. But we also can think about how to do this better since the abstraction is a little difficult to reason about right now. We do a similar operation with setting the init-container bootstrap on the executors.


def configurePod(secretNameOption: Option[String], pod: Pod) : Pod = {
secretNameOption.map { secret =>
new PodBuilder(pod)
.editOrNewSpec()
.addNewVolume()
.withName(SPARK_APP_HADOOP_SECRET_VOLUME_NAME)
.withNewSecret()
.withSecretName(secret)
.endSecret()
.endVolume()
.endSpec()
.build()
}.getOrElse(pod)
}

def configureContainer(secretNameOption: Option[String],
containerSpec: Container) : Container = {
secretNameOption.map { secret =>
new ContainerBuilder(containerSpec)
.addNewVolumeMount()
.withName(SPARK_APP_HADOOP_SECRET_VOLUME_NAME)
.withMountPath(SPARK_APP_HADOOP_CREDENTIALS_BASE_DIR)
.endVolumeMount()
.addNewEnv()
.withName(ENV_HADOOP_TOKEN_FILE_LOCATION)
.withValue(SPARK_APP_HADOOP_TOKEN_FILE_PATH)
.endEnv()
.build()
}.getOrElse(containerSpec)
}
}
38 changes: 38 additions & 0 deletions ...a/org/apache/spark/deploy/kubernetes/submit/submitsteps/DriverHadoopCredentialsStep.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.deploy.kubernetes.submit.submitsteps

import org.apache.spark.SparkConf
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.kubernetes.submit.HadoopSecretUtil

private[spark] class DriverHadoopCredentialsStep(submissionSparkConf: SparkConf)
extends DriverConfigurationStep {

private val maybeMountedHadoopSecret = submissionSparkConf.get(MOUNTED_HADOOP_SECRET_CONF)

override def configureDriver(driverSpec: KubernetesDriverSpec): KubernetesDriverSpec = {
val podWithMountedHadoopToken = HadoopSecretUtil.configurePod(maybeMountedHadoopSecret,
driverSpec.driverPod)
val containerWithMountedHadoopToken = HadoopSecretUtil.configureContainer(
maybeMountedHadoopSecret,
driverSpec.driverContainer)
driverSpec.copy(
driverPod = podWithMountedHadoopToken,
driverContainer = containerWithMountedHadoopToken)
}
}
13 changes: 10 additions & 3 deletions ...ala/org/apache/spark/scheduler/cluster/kubernetes/KubernetesClusterSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ import org.apache.spark.{SparkContext, SparkEnv, SparkException}
import org.apache.spark.deploy.kubernetes.{ConfigurationUtils, InitContainerResourceStagingServerSecretPlugin, PodWithDetachedInitContainer, SparkPodInitContainerBootstrap}
import org.apache.spark.deploy.kubernetes.config._
import org.apache.spark.deploy.kubernetes.constants._
import org.apache.spark.deploy.kubernetes.submit.InitContainerUtil
import org.apache.spark.deploy.kubernetes.submit.{HadoopSecretUtil, InitContainerUtil}
import org.apache.spark.network.netty.SparkTransportConf
import org.apache.spark.network.shuffle.kubernetes.KubernetesExternalShuffleClient
import org.apache.spark.rpc.{RpcAddress, RpcCallContext, RpcEndpointAddress, RpcEnv}
Expand Down Expand Up @@ -130,6 +130,8 @@ private[spark] class KubernetesClusterSchedulerBackend(
private implicit val requestExecutorContext = ExecutionContext.fromExecutorService(
ThreadUtils.newDaemonCachedThreadPool("kubernetes-executor-requests"))

private val maybeMountedHadoopSecret = conf.get(MOUNTED_HADOOP_SECRET_CONF)

private val driverPod = try {
kubernetesClient.pods().inNamespace(kubernetesNamespace).
withName(kubernetesDriverPodName).get()
Expand Down Expand Up @@ -582,9 +584,14 @@ private[spark] class KubernetesClusterSchedulerBackend(

val executorPodWithNodeAffinity = addNodeAffinityAnnotationIfUseful(
executorPodWithInitContainer, nodeToLocalTaskCount)
val resolvedExecutorPod = new PodBuilder(executorPodWithNodeAffinity)
val executorPodWithMountedHadoopToken = HadoopSecretUtil.configurePod(maybeMountedHadoopSecret,
executorPodWithNodeAffinity)
val containerWithMountedHadoopToken = HadoopSecretUtil.configureContainer(
maybeMountedHadoopSecret, initBootstrappedExecutorContainer)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above with detachedContainer logic

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend to leverage a Bootstrap method similair to how SparkPodInitBootStrap functions that could be passed in as an Option

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. This builder code would deserve some refactoring. But I'd like to put it off until the prototype works end-to-end and we see clearly what is needed.


val resolvedExecutorPod = new PodBuilder(executorPodWithMountedHadoopToken)
.editSpec()
.addToContainers(initBootstrappedExecutorContainer)
.addToContainers(containerWithMountedHadoopToken)
.endSpec()
.build()
try {
Expand Down