amoe amoe - 6 months ago 22
Java Question

How to change log level of userlogs in Hadoop 2?

I've been trying to change the log level on userlogs i.e the files that appear under

/var/log/hadoop-yarn/userlogs/application_<id>/container_<id>
on CDH 5.2.1. However, no matter what I try, only
INFO
level logs will appear. I want to enable
TRACE
level logs for debugging.

Things I have tried so far:


  • Setting all loggers to TRACE level in
    /etc/hadoop/conf/log4j.properties
    .

  • Setting
    mapreduce.map.log.level
    and
    mapreduce.reduce.log.level
    in
    mapred-site.xml
    .

  • Setting
    mapreduce.map.log.level
    and
    mapreduce.reduce.log.level
    in the job configuration before submitting it.

  • Including a
    log4j.properties
    in my job jar file that sets the root Log4j logger to TRACE.

  • Modifying
    yarn-env.sh
    to specify
    YARN_ROOT_LOGGER=TRACE,console



None of these worked -- they didn't break anything, but they didn't have any effect on the log outputs under the
userlogs
directory. Modifying yarn-env.sh did cause the ResourceManager and NodeManager logs to enter trace level. Unfortunately these are not useful for my purpose.

I get the following error appearing in
/var/log/hadoop-yarn/userlogs/application_<id>/container_<id>/stderr
that may be relevant.

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/hadoop-yarn/nm-local-dir/usercache/tomcat/appcache/application_1419961570089_0001/filecache/10/job.jar/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.


I don't understand why the log4j "no configuration" message would happen, given that there is a
log4j.properties
file at the root of the job jar file that specifies a root logger:

log4j.rootLogger=TRACE, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] %m%n


My code does not knowingly use SLF4J for logging, it purely uses Log4j.

Answer

The actual answer was to set yarn.app.mapreduce.am.log.level to the level you need, but, crucially, it needs to be set in the Hadoop job configuration at submission time. It cannot be set on the cluster globally. The cluster global will always default to INFO, as it is hardcoded.

Using container-log4j.properties alone will not work as YARN will override the log level value on the command line. See the method addLog4jSystemProperties of org.apache.hadoop.mapreduce.v2.util.MRApps and cross reference with org.apache.hadoop.mapreduce.MRJobConfig.

container-log4j.properties will indeed be honoured, but it can't override the level set by this property.