Vishal Kharde Vishal Kharde - 4 months ago 27
Java Question

how to suppress the logging messages generated by hadoop on the console

I am using hadoop 2.5.2.I am new to hadoop and map reduce . I m trying a map reduce code to perform sentiment analysis.But when i execute the code, lot of logging messages are generated by hadoop on the console.Thats the reason it takes a lot of time in giving analysis result for each sentence.Following are the logging messages....

2016-05-12 23:03:05,396 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-05-12 23:03:05,397 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-05-12 23:03:05,400 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-05-12 23:03:05,403 WARN mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(259)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-05-12 23:03:05,412 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 3
2016-05-12 23:03:05,421 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:3
2016-05-12 23:03:05,430 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_local1427897879_0120
2016-05-12 23:03:05,444 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2016-05-12 23:03:05,445 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2016-05-12 23:03:05,445 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2016-05-12 23:03:05,482 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2016-05-12 23:03:05,482 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: hadoop.tmp.dir; Ignoring.
2016-05-12 23:03:05,483 WARN conf.Configuration (Configuration.java:loadProperty(2368)) - file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2016-05-12 23:03:05,483 INFO mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://localhost:8080/
2016-05-12 23:03:05,483 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_local1427897879_0120
2016-05-12 23:03:05,483 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2016-05-12 23:03:05,484 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2016-05-12 23:03:05,485 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2016-05-12 23:03:05,485 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1427897879_0120_m_000000_0
2016-05-12 23:03:05,486 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-12 23:03:05,486 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(462)) - Processing split: file:/home/hduser/workspace/Test1/training/pool.txt:0+17961
2016-05-12 23:03:05,487 INFO mapred.MapTask (MapTask.java:runOldMapper(416)) - numReduceTasks: 1
2016-05-12 23:03:05,487 INFO mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-05-12 23:03:05,513 INFO mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-12 23:03:05,513 INFO mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-05-12 23:03:05,514 INFO mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2016-05-12 23:03:05,516 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1437)) - Starting flush of map output
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1455)) - Spilling map output
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1456)) - bufstart = 0; bufend = 17961; bufvoid = 104857600
2016-05-12 23:03:05,516 INFO mapred.MapTask (MapTask.java:flush(1458)) - kvstart = 26214396(104857584); kvend = 26211024(104844096); length = 3373/6553600
2016-05-12 23:03:05,523 INFO mapred.MapTask (MapTask.java:sortAndSpill(1641)) - Finished spill 0
2016-05-12 23:03:05,524 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1427897879_0120_m_000000_0 is done. And is in the process of committing
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/home/hduser/workspace/Test1/training/pool.txt:0+17961
2016-05-12 23:03:05,525 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1427897879_0120_m_000000_0' done.
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1427897879_0120_m_000000_0
2016-05-12 23:03:05,525 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1427897879_0120_m_000001_0
2016-05-12 23:03:05,525 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-12 23:03:05,526 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(462)) - Processing split: file:/home/hduser/workspace/Test1/training/pool.txt~:0+17939
2016-05-12 23:03:05,526 INFO mapred.MapTask (MapTask.java:runOldMapper(416)) - numReduceTasks: 1
2016-05-12 23:03:05,527 INFO mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-05-12 23:03:05,550 INFO mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600
2016-05-12 23:03:05,552 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -


I have referred the similar question already asked on stackoverflow [how to suppress Hadoop logging message on the console but it hasnt helped me niether clear answer is discussed. I tried using following configuration suggested in some forum but its not working.I have tried setting following in hadoop-env.sh

export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"


I have also tried editing the log4j.properties file by setting following values..

hadoop.root.logger=WARN,DRFA
hadoop.log.dir=.
hadoop.log.file=hadoop.log


But still i am not able to get rid of this logging messages generated by hadoop mapreduce during runtime execution.Its delaying my output on the console as well.Is their any workaround or any java code which i can embed in my code and suppress the messages.any suggestions,help..Anyone knows??

Thanks a lot!!

Answer

I have found solution for this.All that it needs is changing the configuration file of mapreduce.

1.mapreduce.map.log.level can take values as OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting could be overridden if "mapreduce.job.log4j-properties-file" is set.

  1. mapreduce.reduce.log.level can also take values as OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting could be overridden if "mapreduce.job.log4j-properties-file" is set.So its better to make sure that "mapreduce.job.log4j-properties-file" is not set.

We have to set following properties in the mapred-site.xml.

<property>
<name>mapreduce.map.log.level</name>
<value>OFF</value>
</property>


<property>
<name>mapreduce.reduce.log.level</name>
<value>OFF</value>
</property>

​Now i can see no log messages on the console.But it also has disadvantage as we cant figure out any error if it occurs while executing the mapreduce code as no log messages are visible.