How to access secure kerberized hadoop using just java API

Question

I am pretty new to Kerberos. I am using a shared Hadoop cluster. My admin has provided me with a username and a password and the location of the KDC server.

Is it possible to use just the JAVA GSS API and the Hadoop USerGroupInformation class to access the Hadoop cluster.

For a non-kerberos hadoop cluster, this would be the code snippet I would use to, say read a file from HDFS :

String uname = <Some username>;
UserGroupInformation ugi = UserGroupInformation.createRemoteUser(uname);
ugi.doAs(new PrivilegedExceptionAction<Void>(){

   public Void run() throws Exception {

      HdfsConfiguration hdfsConf = new HdfsConfiguration();
      ... SETUP Configuration ...
      FileSystem fs = FileSystem.get(hdfsConf);
      ... Use 'fs' to read/write etc ...
   }

});

Now for a secure cluster, I am also provided the kerberos password for the user.. Could someone please provide me with what are the exact changes I need to make to the above code snippet to first access the KDC and do a kinit and then proceed with the HDFS operation

Please remember, in the environment I am planning to deploy the Java app, I might not have 'kinit' installed locally, so invoking the kinit process from within java is not an option.

Thanks in advance..

if you can't kinit before executing the java program from your client, then you cannot deploy your job against the secure cluster, unless you build a kerberos client into your program. — jtravaglini
– jtravaglini, Commented Jan 20, 2014 at 16:23

Amit Kumar · Accepted Answer · 2018-04-18 13:00:23Z

5

If you have a keytab file for the identity of the UserGroupInformation superuser, you can use the org.apache.hadoop.security.SecurityUtil#login(…) API offered in the hadoop-common library, to log in directly from your code. The full API can be found here.

This will need to be done before you create the remote user (or more generally, before any socket connection of any form can be made to the various service daemons), as the invocation would otherwise fail with no login credentials being found.

You do not need krb5-workstation packages and utilities installed or available on the execution machine for this to work.

edited Apr 18, 2018 at 13:00

Amit Kumar

2,7452 gold badges41 silver badges77 bronze badges

answered Feb 8, 2014 at 6:40

Harsh J

1,49413 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Gherbi Hicham Over a year ago

I'm curious about this, will it work if my server is outside the kerberos cluster? meaning that the server that launches the API is not kerberised, will it work by just specifying the provided keytab?

PS1 · Accepted Answer · 2015-01-28 22:54:28Z

0

I had similar issue. I'll just put what I could not find easily in docs.

Ensure your site XML files are in your classpath.

answered Jan 28, 2015 at 22:54

PS1

1351 gold badge1 silver badge6 bronze badges

Collectives™ on Stack Overflow

How to access secure kerberized hadoop using just java API

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related