disjoint_molecule disjoint_molecule - 3 months ago 24
Java Question

RuntimeException, ClassNotFoundException: Class WordCount$Map not found . Mapper class issue

I am trying to run WordCount on sandBox HortonWork HDP. I have searched this error in many places and applied what they said and I could not make it run.

Step by Step:
1- First I compile the java program

javac -cp .:$(hadoop classpath) WordCount.java


2- Next bundle the classes as a jar file

jar cvf WordCount.jar *.class


3- Lets look at the content of the jar file:

jar tf WordCount.jar

META-INF/
META-INF/MANIFEST.MF
WordCount.class
WordCount$Map.class
WordCount$Reduce.class


4- Next set the input file into hdfs and run the job.

hdfs dfs -put file.txt

hadoop jar WordCount.jar WordCount /user/root/file.txt /user/root/output


5- And I got the following error :

WARNING: Use "yarn jar" to launch YARN applications.
16/08/12 04:02:49 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
16/08/12 04:02:49 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/10.0.2.15:8050
16/08/12 04:02:50 WARN mapreduce.JobResourceUploader: No job jar file set. User classes may not be found. See Job or Job#setJar(String).
16/08/12 04:02:50 INFO input.FileInputFormat: Total input paths to process : 1
16/08/12 04:02:51 INFO mapreduce.JobSubmitter: number of splits:1
16/08/12 04:02:51 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470929339157_0022
16/08/12 04:02:51 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
16/08/12 04:02:51 INFO impl.YarnClientImpl: Submitted application application_1470929339157_0022
16/08/12 04:02:52 INFO mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1470929339157_0022/
16/08/12 04:02:52 INFO mapreduce.Job: Running job: job_1470929339157_0022
16/08/12 04:03:03 INFO mapreduce.Job: Job job_1470929339157_0022 running in uber mode : false
16/08/12 04:03:03 INFO mapreduce.Job: map 0% reduce 0%
16/08/12 04:03:10 INFO mapreduce.Job: map 100% reduce 0%
16/08/12 04:03:10 INFO mapreduce.Job: Task Id : attempt_1470929339157_0022_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class WordCount$Map not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class WordCount$Map not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
... 8 more

16/08/12 04:03:11 INFO mapreduce.Job: map 0% reduce 0%
16/08/12 04:03:16 INFO mapreduce.Job: Task Id : attempt_1470929339157_0022_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class WordCount$Map not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class WordCount$Map not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
... 8 more

16/08/12 04:03:23 INFO mapreduce.Job: Task Id : attempt_1470929339157_0022_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class WordCount$Map not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class WordCount$Map not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
... 8 more

16/08/12 04:03:31 INFO mapreduce.Job: map 100% reduce 100%
16/08/12 04:03:31 INFO mapreduce.Job: Job job_1470929339157_0022 failed with state FAILED due to: Task failed task_1470929339157_0022_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

16/08/12 04:03:31 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=4
Killed reduce tasks=1
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=20865
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=20865
Total time spent by all reduce tasks (ms)=0
Total vcore-seconds taken by all map tasks=20865
Total vcore-seconds taken by all reduce tasks=0
Total megabyte-seconds taken by all map tasks=5216250
Total megabyte-seconds taken by all reduce tasks=0


`

And the following code is the one I try to run:

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.*;
import org.apache.hadoop.mapreduce.lib.output.*;
import org.apache.hadoop.util.*;

public class WordCount extends Configured implements Tool {

public static void main(String args[]) throws Exception {
int res = ToolRunner.run(new WordCount(), args);
System.exit(res);
}

public int run(String[] args) throws Exception {
Path inputPath = new Path(args[0]);
Path outputPath = new Path(args[1]);

Configuration conf = getConf();
Job job = new Job(conf, this.getClass().toString());
job.setJarByClass(WordCount.class);

FileInputFormat.setInputPaths(job, inputPath);
FileOutputFormat.setOutputPath(job, outputPath);

job.setJobName("WordCount");
//job.setJarByClass(WordCount.class);

job.setMapperClass(Map.class);
job.setCombinerClass(Reduce.class);
job.setReducerClass(Reduce.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
//FileInputFormat.setInputPaths(job, inputPath);
//FileOutputFormat.setOutputPath(job, outputPath);

return job.waitForCompletion(true) ? 0 : 1;
}

public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

@Override
public void map(LongWritable key, Text value,
Mapper.Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}

public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {

@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}

context.write(key, new IntWritable(sum));
}
}

}


Thanks in advance for any help/hint and advice.

Answer

Try adding following in your class

job.setJar("WordCount.jar");
Comments