Hadoop class not found exception

Question

I'm working on simple program on hadoop, I followed this tutorial steps: http://www.bogotobogo.com/Hadoop/BigData_hadoop_Creating_Java_Wordcount_Project_with_Eclipse_MapReduce2.php

even though I tried it on two different machines, it keeps showing this exception:

Exception in thread "main" java.lang.ClassNotFoundException: test.java
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

package pa2;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;


public class test extends Configured implements Tool{


public int run(String[] args) throws Exception
{ if (args.length<2)
{
    System.out.println("plz give proper arguments");
    return -1;
}
      //creating a JobConf object and assigning a job name for identification purposes
      JobConf conf = new JobConf(test.class);

      FileInputFormat.setInputPaths(conf, new Path(args[0]));
      FileOutputFormat.setOutputPath(conf, new Path(args[1]));

      conf.setMapperClass(mapper.class);

      conf.setMapOutputKeyClass(Text.class);
      conf.setMapOutputValueClass(IntWritable.class);

      conf.setOutputKeyClass(Text.class);
      conf.setOutputValueClass(IntWritable.class);

      JobClient.runJob(conf);

      return 0;
}


public static void main(String[] args) throws Exception
{
      // this main function will call run method defined above.
  int exitcode = ToolRunner.run(new test(),args);
      System.exit(exitcode);
}
}

can you please tell me what is wrong here?

update:

mapper class:

package pa2;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;


public class mapper extends MapReduceBase 
        implements Mapper<LongWritable,Text, Text, IntWritable>
{
            public void map(LongWritable Key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter r)
            throws IOException {


            int i=0;
            String [] array = new String [50];


                        String name;
                        String year;
                        String s=value.toString();

                        for (String word:s.split(",")){

                   word = s.substring(0, s.indexOf(",")+1);
                   year= word.substring(0, s.indexOf(",")+1);
                   name=word.substring(s.indexOf(",")+1);
                   int theyear= Integer.parseInt(year);


                   if(theyear<2000){
                        array[i] =name;
                        output.collect(new Text(word),  new IntWritable(1));

                        i++;}

                    }       
    }
}

I haven't written the reducer class. I exported the project as jar file,and I made a text file called movies to be the input of the program. then wrote this in the terminal:

[cloudera@quickstart ~]$ cd workspace
[cloudera@quickstart workspace]$ ls
pa2  pa2.jar  training
[cloudera@quickstart workspace]$ hadoop jar pa2.jar test movies.txt output.txt
Exception in thread "main" java.lang.ClassNotFoundException: test
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

export >jar file > I named it pa2.jar (same as the project name> finish. — hailah
– hailah, Commented Dec 9, 2016 at 20:58
Try to follow that tutorial verbatim. test and mapper should 1) be capitalized as a Java classes, then the class is WordCount in the default package. — OneCricketeer
– OneCricketeer, Commented Dec 9, 2016 at 20:59

OneCricketeer · Accepted Answer · 2016-12-09 21:04:18Z

1

No guarantees this is the solution to the immediate problem, but

package pa2;

This is appended to the class name. In other words, the fully-qualified class name is pa2.test.

So, try

hadoop jar ~/workspace/pa2.jar pa2.test input output

If you used the default package like that tutorial showed, you wouldn't need to specify the package on the command line.

edited Dec 9, 2016 at 21:04

answered Dec 9, 2016 at 21:00

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

hailah Over a year ago

I tried this and it says: Not a valid JAR: /home/cloudera/pa2.jar

OneCricketeer Over a year ago

Well, it isn't in that folder / that JAR isn't valid, as it says.

hailah Over a year ago

Yes sorry I was wrong, it's finally working now! THANK you very much.

kashmoney · Accepted Answer · 2016-12-09 18:39:11Z

0

The actual name of your map class should be provided here

conf.setMapperClass(mapper.class);

If you are trying to use the default map class, then write "Mapper.class".

answered Dec 9, 2016 at 18:39

kashmoney

977 bronze badges

5 Comments

hailah Over a year ago

Yes the name of the map class I have is "mapper".

kashmoney Over a year ago

Did you add the hadoop jars to the buildpath in eclipse?

hailah Over a year ago

Yes. I added all of the hadoop jars to the project.

kashmoney Over a year ago

Great, is it possible for you to share the complete mapreduce code(including the map class) and the command you are using to run the job? It seems like a minor semantic/syntax issue is causing this error.

hailah Over a year ago

Thank you very much for your time! it's fixed now.

Collectives™ on Stack Overflow

Hadoop class not found exception

2 Answers 2

3 Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related