Spark 3.5.1 - Ozone client ignoring Kerberos cache #6912

maron546 · 2024-07-08T16:50:20Z

maron546
Jul 8, 2024

Hi.

In our environments, we are not able to upgrade Spark from version 3.1.3 to 3.5.1 due to an authentication error.
Steps to reproduce the issue:

Create the Spark image from the binary

bin/docker-image-tool.sh -t 3.5.1 build

We have the same behaviour with the official Spark image in Docker Hub, too.

In the image, add the ozone-filesystem-hadoop3-client-1.4.0.jar in /opt/spark/jars, as per HDDS-6926.
Edit the image entrypoint in order to require a Kerberos ticket with a dedicated principal and keytab (kinit).
Execute the submit from other tools, such as Airflow.
The driver fails with the following error:

24/07/08 15:58:16 WARN Client: Exception encountered while connecting to the server
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:392)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
        at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:843)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:839)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAs(Unknown Source)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:839)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1677)
        at org.apache.hadoop.ipc.Client.call(Client.java:1502)
        at org.apache.hadoop.ipc.Client.call(Client.java:1455)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:122)
        at jdk.proxy2/jdk.proxy2.$Proxy46.submitRequest(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at jdk.proxy2/jdk.proxy2.$Proxy46.submitRequest(Unknown Source)
        at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:33
0)
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1
784)
        at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:247)
        at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:248)
        at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:115)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.<init>(BasicRootedOzoneClientAdapterImpl.java:190)
        at org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:51)
        at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:97)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:189)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
        at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1831)
        at org.apache.spark.deploy.history.EventLogFileWriter.<init>(EventLogFileWriters.scala:60)
        at org.apache.spark.deploy.history.SingleEventLogFileWriter.<init>(EventLogFileWriters.scala:213)
        at org.apache.spark.deploy.history.EventLogFileWriter$.apply(EventLogFileWriters.scala:181)
        at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:64)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:636)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
        at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:43)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
        at org.apache.spark.deploy.SparkSubmit.$anonfun$submit$2(SparkSubmit.scala:169)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAs(Unknown Source)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:169)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
24/07/08 15:58:16 WARN Client: Exception encountered while connecting to the server
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:392)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
        at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:843)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:839)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAs(Unknown Source)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:839)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1677)
        at org.apache.hadoop.ipc.Client.call(Client.java:1502)
        at org.apache.hadoop.ipc.Client.call(Client.java:1455)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:122)
        at jdk.proxy2/jdk.proxy2.$Proxy46.submitRequest(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at jdk.proxy2/jdk.proxy2.$Proxy46.submitRequest(Unknown Source)
        at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:33
0)
        at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1
784)
        at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:247)
        at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:248)
        at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:115)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.<init>(BasicRootedOzoneClientAdapterImpl.java:190)
        at org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:51)
        at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:97)
        at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:189)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
        at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1831)
        at org.apache.spark.deploy.history.EventLogFileWriter.<init>(EventLogFileWriters.scala:60)
        at org.apache.spark.deploy.history.SingleEventLogFileWriter.<init>(EventLogFileWriters.scala:213)
        at org.apache.spark.deploy.history.EventLogFileWriter$.apply(EventLogFileWriters.scala:181)
        at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:64)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:636)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2888)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1099)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1093)
        at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:43)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
        at org.apache.spark.deploy.SparkSubmit.$anonfun$submit$2(SparkSubmit.scala:169)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
        at java.base/java.security.AccessController.doPrivileged(Unknown Source)
        at java.base/javax.security.auth.Subject.doAs(Unknown Source)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:169)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

The configurations are exactly the same as the ones we used with Spark 3.1.3 and the jar ozone-filesystem-hadoop3-1.4.0.jar.

Answered by maron546

Mar 5, 2025

Hi.
I solved the issue in Spark 3.5.4, without initializing the ticket cache but simply adding the following jars:

accessors-smart-2.4.7.jar
asm-9.1.jar
commons-configuration2-2.1.1.jar
hadoop-annotations-3.3.4.jar
hadoop-auth-3.3.4.jar
hadoop-common-3.3.4.jar
jcip-annotations-1.0-1.jar
json-smart-2.4.7.jar
nimbus-jose-jwt-9.8.1.jar
ozone-filesystem-hadoop3-client-1.4.1.jar
protobuf-java-2.5.0.jar
stax2-api-4.2.1.jar
woodstox-core-5.3.0.jar

View full answer

kerneltime · 2024-07-08T16:57:02Z

kerneltime
Jul 8, 2024
Collaborator

Did you move to hadoop-client jar as well? Ref: #3915

8 replies

jojochuang Jul 10, 2024
Collaborator

hi @maron546 can you provide DEBUG logs? Ideally also Kerberos debug log but it may contain sensitive information.
follow this instruction https://risdenk.github.io/2018/03/14/kerberos-debugging.html

If Spark 3.5 can kinit and access Hadoop services it should also be able to access Ozone too.

maron546 Jul 17, 2024
Author

Hi @jojochuang.
Unfortunately, even with the following properties for Spark:

"spark.driver.extraJavaOptions": "-Dsun.security.krb5.debug=true",
"spark.kubernetes.driverEnv.HADOOP_ROOT_LOGGER": "DEBUG,console",
"spark.kubernetes.driverEnv.HADOOP_JAAS_DEBUG": "true",
"spark.kubernetes.driverEnv.OZONE_ROOT_LOGGER": "DEBUG,console",
"spark.kubernetes.driverEnv.OZONE_SECURITY_LOGGER": "DEBUG,console",

we didn't get any additional informations from the Ozone client. The log is similar to the previous attached one. Is there other property we should use for debugging?
Furthermore, in the Kerberos logs there is only the initial AS_REQ:

Jul 17 08:30:08 krb5-0 krb5kdc[71](info): AS_REQ (8 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), DEPRECATED:des3-cbc-sha1(16), DEPRECATED:arcfour-hmac(23), camellia128-cts-cmac(25), camellia256-cts-cmac(26)}) 172.31.41.91: ISSUE: authtime 1721205008, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, spark@xxx for krbtgt/xxx@xxx

Does the client take the following properties (ozone-site.xml):

ozone.om.kerberos.principal
hdds.scm.kerberos.principal

into consideration?

maron546 Jul 18, 2024
Author

We also noticed that in the driver log there are no requests for delegation tokens for the services listed in the property spark.kerberos.access.hadoopFileSystems or for Hive.
For example, in version 3.1.3 we have:

24/07/18 10:13:19 INFO HadoopFSDelegationTokenProvider: getting token for: RootedOzoneFileSystem{URI=ofs://xxx, workingDir=ofs://xxx/user/xxx, userName=xxx, statistics=0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops} with renewer xxx
24/07/18 10:13:19 INFO KMSClientProvider: New token created: (Kind: kms-dt, Service: kms://https@xxx:9393/kms, Ident: (kms-dt owner=xxx, renewer=xxx, realUser=xxx, issueDate=1721297599498, maxDate=1721902399498, sequenceNumber=553, masterKeyId=128))
24/07/18 10:13:19 INFO HadoopFSDelegationTokenProvider: getting token for: S3AFileSystem{uri=s3a://xxx, workingDir=s3a://xxx/user/xxx, inputPolicy=normal, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432,...
...

This part is missing in version 3.5.1. Do you know if there are additional properties that must be configured?

yazynin Jul 18, 2024

Hi @maron546, Spark 3.5.1 worked with ozone 1.4 with kerberos auth in my environment.
do you have in spark-defults.conf
spark.kerberos.access.hadoopFileSystems ofs://xxxx/
in core-site.xml
<property><name>fs.ofs.impl</name><value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value></property> <property><name>fs.AbstractFileSystem.ofs.impl</name><value>org.apache.hadoop.fs.ozone.RootedOzFs</value></property> <property><name>hadoop.security.authentication</name><value>kerberos</value></property> <property><name>hadoop.security.authorization</name><value>true</value></property> <property><name>hadoop.security.auth_to_local</name><value>DEFAULT</value></property> <property><name>fs.defaultFS</name><value>ofs://xxx/</value></property> <property><name>hadoop.kerberos.keytab.login.autorenewal.enabled</name><value>true</value></property>

maron546 Jul 19, 2024
Author

Hi @yazynin.
Are you using a principal and a keytab for the Spark submit? Unfortunately, in our environments (Kubernetes) we can't do that, so we have to use the ticket cache, but Spark 3.5.1 seems to ignore it, even if in the driver log is correctly imported:

24/07/19 08:37:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
                [UnixLoginModule]: succeeded importing info:
                        uid = 0
                        gid = 0
                        supp gid = 0
Debug is  true storeKey false useTicketCache true useKeyTab false doNotPrompt true ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is true principal is null tryFirstPass is false useFirstPass is false storePass is false clearPass is false
Refreshing Kerberos configuration
Java config name: null
Native config name: /etc/krb5.conf
Loading config file from /etc/krb5.conf
Loading krb5 profile at /etc/krb5.conf
logging = {
default = STDERR
kdc = STDERR
admin_server = STDERR
}
libdefaults = {
dns_canonicalize_hostname = false
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = xxx
}
realms = {
xxx = {
kdc = krb5-0.xxx.svc.cluster.local
kdc = krb5-1.xxx.svc.cluster.local
admin_server = krb5-0.xxx.svc.cluster.local
primary_kdc = krb5-0.xxx.svc.cluster.local
max_renewable_life = 7d
}
yyy = {
kdc = krb5-0.xxx.svc.cluster.local
kdc = krb5-1.xxx.svc.cluster.local
admin_server = krb5-0.xxx.svc.cluster.local
primary_kdc = krb5-0.xxx.svc.cluster.local
max_renewable_life = 7d
}
zzz = {
kdc = krb5-0.xxx.svc.cluster.local
kdc = krb5-1.xxx.svc.cluster.local
admin_server = krb5-0.xxx.svc.cluster.local
primary_kdc = krb5-0.xxx.svc.cluster.local
max_renewable_life = 7d
}
}
domain_realm = {
...
}
capaths = {
...
}
Loaded from native config
>>> KdcAccessibility: reset
>>> KdcAccessibility: reset
Acquire TGT from Cache
>>>KinitOptions cache name is /tmp/krb5cc_0
>>>DEBUG <CCacheInputStream>  client principal is spark@yyy
>>>DEBUG <CCacheInputStream> server principal is krbtgt/yyy@yyy
>>>DEBUG <CCacheInputStream> key type: 18
>>>DEBUG <CCacheInputStream> auth time: Fri Jul 19 08:37:18 UTC 2024
>>>DEBUG <CCacheInputStream> start time: Fri Jul 19 08:37:18 UTC 2024
>>>DEBUG <CCacheInputStream> end time: Sat Jul 20 08:37:18 UTC 2024
>>>DEBUG <CCacheInputStream> renew_till time: Fri Jul 26 08:37:18 UTC 2024
>>> CCacheInputStream: readFlags()  FORWARDABLE; RENEWABLE; INITIAL;
>>>DEBUG <CCacheInputStream>  client principal is [email protected]
>>>DEBUG <CCacheInputStream> server principal is krb5_ccache_conf_data/fast_avail/krbtgt/yyy\@yyy@X-CACHECONF:
>>>DEBUG <CCacheInputStream> key type: 0
>>>DEBUG <CCacheInputStream> auth time: Thu Jan 01 00:00:00 UTC 1970
>>>DEBUG <CCacheInputStream> start time: null
>>>DEBUG <CCacheInputStream> end time: Thu Jan 01 00:00:00 UTC 1970
>>>DEBUG <CCacheInputStream> renew_till time: null
>>> CCacheInputStream: readFlags()
get normal credential
Principal is spark@yyy
                [UnixLoginModule]: added UnixPrincipal,
                                UnixNumericUserPrincipal,
                                UnixNumericGroupPrincipal(s),
                         to Subject
Commit Succeeded

The complete core-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>ofs://xxx</value>
  </property>
  <property>
    <name>fs.AbstractFileSystem.o3fs.impl</name>
    <value>org.apache.hadoop.fs.ozone.OzFs</value>
  </property>
  <property>
    <name>fs.ofs.impl</name>
    <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>
  </property>
  <property>
    <name>fs.s3a.impl</name>
    <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
  </property>
  <property>
    <name>fs.AbstractFileSystem.s3a.impl</name>
    <value>org.apache.hadoop.fs.s3a.S3A</value>
  </property>
  <property>
    <name>fs.s3a.endpoint</name>
    <value>ozone-s3g.xxx.svc.cluster.local:9878</value>
  </property>
  <property>
    <name>fs.s3a.connection.ssl.enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>fs.s3a.path.style.access</name>
    <value>true</value>
  </property>
  <property>
    <name>fs.s3a.bucket.probe</name>
    <value>0</value>
  </property>
  <property>
    <name>fs.s3a.change.detection.version.required</name>
    <value>false</value>
  </property>
  <property>
    <name>fs.s3a.change.detection.mode</name>
    <value>none</value>
  </property>
  <property>
    <name>dfs.data.transfer.protection</name>
    <value>authentication</value>
  </property>
  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hadoop.security.authorization</name>
    <value>true</value>
  </property>
  <property>
    <name>hadoop.rpc.protection</name>
    <value>authentication</value>
  </property>
  <property>
    <name>hadoop.security.auth_to_local</name>
    <value>
      RULE:[1:$1@$0](.*@.*.xxx)s/@.*///L
      RULE:[2:$1@$0](.*@.*.xxx)s/@.*///L
      DEFAULT
    </value>
  </property>
  <property>
    <name>hadoop.proxyuser.kyuubi.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.kyuubi.users</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hive.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hive.users</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.trino.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.trino.users</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hue.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hue.users</name>
    <value>*</value>
  </property>
  <property>
    <name>hue.kerberos.principal.shortname</name>
    <value>hue</value>
  </property>
  <property>
    <name>hadoop.proxyuser.superset.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.superset.users</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.airflow.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.airflow.users</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.livy.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.livy.users</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.spark.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.spark.users</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.security.key.provider.path</name>
    <value>kms://[email protected]:9393/kms</value>
  </property>
  <property>
    <name>hadoop.security.kms.client.authentication.retry-count</name>
    <value>10</value>
  </property>
  <property>
    <name>hadoop.security.kms.client.failover.max.retries</name>
    <value>10</value>
  </property>
  <property>
    <name>fs.AbstractFileSystem.ofs.impl</name>
    <value>org.apache.hadoop.fs.ozone.RootedOzFs</value>
  </property>
  <property>
    <name>hadoop.kerberos.keytab.login.autorenewal.enabled</name>
    <value>true</value>
  </property>
</configuration>

And the Spark properties:

"spark.kubernetes.file.upload.path": "ofs://xxx/xxx/spark"
"spark.hadoop.hive.exec.scratchdir": "ofs://xxx/xxx/hive-scratchdir"
"spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive": "true"
"spark.eventLog.enabled": "true"
"spark.eventLog.dir": "ofs://xxx/xxx/spark-history/logs"
"spark.hadoop.fs.s3a.access.key": "xxx"
"spark.hadoop.fs.s3a.secret.key": "xxx"
"spark.kerberos.access.hadoopFileSystems": "ofs://xxx,s3a://xxx"
"spark.hadoop.hive.metastore.kerberos.principal": "hive/xxx@xxx"
"spark.hadoop.hive.metastore.uris": "thrift://xxx:9083"
"spark.driver.memory": "1g"
"spark.executor.memory": "1g"
"spark.sql.adaptive.enabled": "true"
"spark.kubernetes.authenticate.driver.serviceAccountName": "xxx"
"spark.kubernetes.container.image": "xxx"
"spark.kubernetes.container.image.pullPolicy": "Always"
"spark.kubernetes.driver.request.cores": 0.5
"spark.kubernetes.driver.limit.cores": 2
"spark.kubernetes.executor.request.cores": 0.5
"spark.kubernetes.executor.limit.cores": 2
"spark.kubernetes.executor.secrets.xxx": "/opt/hadoop/conf"
"spark.executorEnv.HADOOP_CONF_DIR": "/opt/hadoop/conf"
"spark.driver.extraJavaOptions": "-Dsun.security.krb5.debug=true -Djavax.security.auth.useSubjectCreds=false"
"spark.kerberos.renewal.credentials": "ccache"
"spark.kubernetes.driverEnv.HADOOP_ROOT_LOGGER": "DEBUG,console"
"spark.kubernetes.driverEnv.HADOOP_JAAS_DEBUG": "true"
"spark.kubernetes.driverEnv.OZONE_ROOT_LOGGER": "DEBUG,console"
"spark.kubernetes.driverEnv.OZONE_SECURITY_LOGGER": "DEBUG,console"
"spark.kubernetes.namespace": "xxx"

@jojochuang In the ozone-site.xml file, the security is enabled.

jojochuang · 2024-07-18T16:13:50Z

jojochuang
Jul 18, 2024
Collaborator

Delegation token requires ozone.security.enabled = true for the client to request delegation tokens.
But for it to fetch DT successfully it needs to have a Kerberos tgt first.

0 replies

maron546 · 2025-03-05T10:13:10Z

maron546
Mar 5, 2025
Author

Hi.
I solved the issue in Spark 3.5.4, without initializing the ticket cache but simply adding the following jars:

accessors-smart-2.4.7.jar
asm-9.1.jar
commons-configuration2-2.1.1.jar
hadoop-annotations-3.3.4.jar
hadoop-auth-3.3.4.jar
hadoop-common-3.3.4.jar
jcip-annotations-1.0-1.jar
json-smart-2.4.7.jar
nimbus-jose-jwt-9.8.1.jar
ozone-filesystem-hadoop3-client-1.4.1.jar
protobuf-java-2.5.0.jar
stax2-api-4.2.1.jar
woodstox-core-5.3.0.jar

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spark 3.5.1 - Ozone client ignoring Kerberos cache #6912

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 8 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Spark 3.5.1 - Ozone client ignoring Kerberos cache #6912

maron546 Jul 8, 2024

Replies: 3 comments · 8 replies

kerneltime Jul 8, 2024 Collaborator

jojochuang Jul 10, 2024 Collaborator

maron546 Jul 17, 2024 Author

maron546 Jul 18, 2024 Author

yazynin Jul 18, 2024

maron546 Jul 19, 2024 Author

jojochuang Jul 18, 2024 Collaborator

maron546 Mar 5, 2025 Author

maron546
Jul 8, 2024

Replies: 3 comments 8 replies

kerneltime
Jul 8, 2024
Collaborator

jojochuang Jul 10, 2024
Collaborator

maron546 Jul 17, 2024
Author

maron546 Jul 18, 2024
Author

maron546 Jul 19, 2024
Author

jojochuang
Jul 18, 2024
Collaborator

maron546
Mar 5, 2025
Author