NoClassDefFoundError Error in Hadoop

Question

So after going through many posts on SO and revising some java basics, I still get this error

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2451)
at java.lang.Class.getMethod0(Class.java:2694)
at java.lang.Class.getMethod(Class.java:1622)
at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)

I know the difference between java.lang.NoClassDefFoundError and ClassNotFoundException in Java, and have looked through what usually causes it. In a nutshell, it is because some class is unavailable to the program during Run Time but is available during Compile time. Hence I get no compile time errors.

I have added two classpaths, one to commons-logging-1.1.3.jar and the other to the hadoop-core.*jar. I am pretty confident that the classpaths are correct.

Here are the imports in my Program

import java.io.*;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.SequenceFile.*;
import org.apache.hadoop.io.SequenceFile.Writer;
import org.apache.hadoop.io.*;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.*;

This class is in hadoop-core jar, can you write exactly how you run your job (including the classpath)? — Charles Menguy
– Charles Menguy, Commented May 30, 2013 at 6:14
@CharlesMenguy I use ubuntu 12.04, this is the command line for compiling it javac -cp "/home/hduser/hadoop/hadoop-core-1.1.2.jar:/home/hduser/Documents/commons-logging-1.1.3/commons-logging-1.1.3.jar" TSVtoSeq.java To Run, java TSVtoSeq TrainingT1.tsv testOutput 2 arguments are passed to main, Input File and Output file — SLearner
– SLearner, Commented May 30, 2013 at 6:18
Are you not setting your classpath with java -cp $JARS when you run your program? — Charles Menguy
– Charles Menguy, Commented May 30, 2013 at 6:21
@CharlesMenguy So, I added the classpaths while running the program(Is it necessary though?), I still get the same exception thrown. However, the class now being pointed to is "org/apache/commons/configuration/Configuration" instead of "org/apache/hadoop/io/Writable" — SLearner
– SLearner, Commented May 30, 2013 at 6:33
@shashlearner thats becoz there are lots of jars hadoop is dependent on (can be found in HADOOP_HOME/lib) and at runtime several of those get referred to. Either keep doing manual trial-n-error untill u get it running or add all of those jars in the classpath while you run it. A standard way is to create a jar of your code and run using bin/hadoop script as described in my answer below. — Tejas Patil
– Tejas Patil, Commented May 30, 2013 at 6:48

Tejas Patil · Accepted Answer · 2013-05-30 12:42:07Z

2

You need to create a jar out of the java code like given here:

$ mkdir my_classes 
$ javac -classpath $HADOOP_HOME/hadoop-$HADOOP_VERSION-core.jar -d my_classes <name of the main class> 
$ jar -cvf <name of the jar> -C my_classes .

Run the jar this way:

$HADOOP_HOME/bin/hadoop jar <name of the jar> <name of the main class> <arguments to the program>

See the documentation page of hadoop for the description of jar command.

edited May 30, 2013 at 12:42

answered May 30, 2013 at 6:34

Tejas Patil

6,1892 gold badges26 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

SLearner Over a year ago

Thanks for replying. I have done exactly what they've asked in their documentation here On running, I get a new error, which is: Exception in thread "main" java.lang.ClassNotFoundException: WordCount

Tejas Patil Over a year ago

@shashlearner the one that i gave was for wordcount example. dont blindly use that command. u need to replace things with your class names.

Tejas Patil Over a year ago

@shashlearner ok. could you provide these details: the commands you used, your source code, the error u got etc.

SLearner Over a year ago

Thanks! Hadoop Root : /home/hduser/hadoop My source code is here: /home/hduser/hadoop/TST_CLASS/hello/WordCount.java Source Code is the exact core in the Documentation here /home/hduser/hadoop/TST_CLASS/hello/MANIFEST.MF MANIFEST.MF contains a single line : Main-Class: org.myorg.WordCount /home/hduser/hadoop/TST_CLASS/hello/commons-logging-1.1.3-tests.jar I first compile the WordCount.java using: sudo javac -classpath /home/hduser/hadoop/hadoop-core-1.1.2.jar WordCount.java

SLearner Over a year ago

let us continue this discussion in chat

Collectives™ on Stack Overflow

NoClassDefFoundError Error in Hadoop

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related