smuggledPancakes smuggledPancakes - 7 months ago 16
Linux Question

Why does zookeeper not use my file log directory

In my zookeeper/conf/ file I set the zookeeper.log.dir to $HOME/zklogs

When I use it does not use that directory. Instead it uses the ${ZOO_LOG_DIR} which when I echo it, comes out to "."

I don't understand how fix this issue, I don't see the ${ZOO_LOG_DIR} set anywhere. I am not sure how it gets set to "." at all. I also don't know how to launch zookeeper without I am noobish at linux too and a little lost on this issue...

Does anybody know how I can fix this issue so that it uses the directory set in my file in the conf directory?

***UPDATE, I found in in the bin directory of my zookeeper install. There is code:

if["x${ZOO_LOG_DIR}" = "x" ]

I am not sure what is going on in that first line, but it has to be here that something is going wrong. I expect it to look at zookeeper.log.dir from my file. Can anybody tell me if that should be true? I don't want to just hardwire the path here...


I wanted to add how I fixed this problem / customized my environment.

There are 2 logging mechanisms working here:

  • bin/ redirects the zookeeper server's stdout and stderr to zookeeper.out
  • log4j can append to logs to several places including:
    • CONSOLE - which ends up in zookeeper server's stdout and stderr
    • ROLLINGFILE - which is sent to zookeeper.log

bin/ uses :

  • ZOO_LOG_DIR to set the path for both zookeeper.out and log4j.
  • ZOO_LOG4J_PROP to set the log4j logging level and what log appenders are turned on

The "eventual" defaults in the setup conf/ are set by a combination of zookeeper bash scripts:

  • ZOO_LOG_DIR = . ( the working directory from which zookeeper is started )
    • set inside of conf/ as zookeeper.log.dir
    • set inside of conf/ as zookeeper.root.logger

The effect of turning on the log appender CONSOLE is that logs now go to stdout. Because bin/ redirects stdout and stderr to zookeeper.out, log4j logs end up in zookeeper.out. The effect of turning off ROLLINGFILE is that no zookeeper.log file is created.

The zookeeper.out log is not rotated. The zookeeper.log log is set to be rotated and can be set to expire old logs.

I wanted logs to roll and be expired. The first thing I had to do was change conf/ to cause the expiration/deletion of old logs. I did that by setting log4j.appender.ROLLINGFILE.MaxBackupIndex inside of conf/ The second thing I had to do was set the log directory, logging level and appenders.

I have a bash script that runs every minute. If it sees that zookeeper isn't running, it runs :

bin/ start

I changed it to specify environmental variables expected by bin/

sudo ZOO_LOG_DIR=/opt/zookeeper-3.4.6/logs ZOO_LOG4J_PROP='INFO,ROLLINGFILE' /opt/zookeeper-3.4.6/bin/ start

The effect of turning off the log appender CONSOLE is that log4j logs now no longer end up in zookeeper.out. The effect of turning on ROLLINGFILE is that zookeeper.log file is created, rotated, and expired.

BTW, conf/ was apparently already in my classpath. I had to make no changes in that regard.

This chain contributed significantly to my understanding:!msg/nosql-databases/aebIvNnT0xY/doky1X9-WfwJ