Simply_me Simply_me - 3 years ago 107
Python Question

Yet Another Python Logging Setup

I've been reading on logging module in python and several blogs on how to set it up, however, no blog gave a scheme for more complex setup.

I would like:


  1. Use external config file (see below).

  2. Have external module to handle log settings (create log file w/
    mylogfile+datetime
    (see below).

  3. Finally, instantiate a logger in a class to log from multiple methods to the output file.



Right now, admittedly, I've a mishmash of settings, and would like to have some pointers to clean this mess :-).

I believe the external config file loads up fine, but there is no log file created.

Example main:

#!/bin/env python

import os
import sys
import logging.config
from datetime import datetime

datetime.now().strftime('mylogfile_%H%M%d%m%Y.log')

LOG_CONFIG = '../config/logging.conf'
#logging.config.fileConfig(LOG_CONFIG)
#logger = logging.getLogger(datetime.now().strftime('mylogfile_%H%M%d%m%Y.log'))

def setup_logger():
logging.config.fileConfig(LOG_CONFIG)
datetime.now().strftime('mylogfile_%H%M%d%m%Y.log')
logger = logging.getLogger(datetime.now().strftime('mylogfile_%H%M%d%m%Y.log'))

class TestLog(object):
def __init__(self):
self.logger = logging.getLogger(__name__)
self.__sub_test = 0

def add_test(self):
self.logger.debug('addition')
a = 1 + 1
self.logger.debug('result {}'.format(a, 1))

def sub_test(self):
self.logger.debug('subtraction')
b = 5 -2
self.logger.debug('result {}'.format(b, 1))


def main():
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
#setup_logger()
test1 = TestLog()
print test1.add_test()
print test1.sub_test()

if __name__ == "__main__":
sys.exit(main())


Conf file:

[loggers]
keys=root

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler

[logger_sLogger]
level=DEBUG
handlers=consoleHandler
qualname=sLogger
propagate=0

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=fileFormatter
args=('%(logfilename)s',)

[formatter_fileFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=

Answer Source

I am doing a similar thing, here is how I have it set up

External config file in YAML format:

logging.yml

version: 1
formatters:
  default: # default debug logger
    format: '[%(asctime)s] (%(name)s:%(funcName)s:%(lineno)d:%(levelname)s) %(message)s' # %(module)s:
    datefmt: "%Y-%m-%d %H:%M:%S"
  info: # basic info logging, for easier reading
    format: '[%(levelname)-8s] %(message)s'
    datefmt: "%Y-%m-%d %H:%M:%S"
  console:
    format: '[%(asctime)s] (%(name)s:%(funcName)s:%(lineno)d:%(levelname)s) %(message)s'
    datefmt: "%Y-%m-%d %H:%M:%S"

handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: console
    stream: ext://sys.stdout
  main:
    () : __main__.logpath # output file path
    level: DEBUG
    formatter: default

loggers:
  app:
    level: DEBUG
    handlers: [console, main]
    propagate: true
  parse:
    level: DEBUG
    handlers: [console, main]
    propagate: true
  tools:
    level: DEBUG
    handlers: [console, main]
    propagate: true
  data:
    level: DEBUG
    handlers: [console, main]
    propagate: true

Note the line () : __main__.logpath here, that calls a function called logpath in the main script in order to get the filehandler. I do this for conditional setting of the output file name. Put whatever filenaming or other Filehandler logic you need in there. See this in the 'main' app Python program:

app.py

!/usr/bin/env python
# -*- coding: utf-8 -*-

'''
My app
'''
# ~~~~~ LOGGING SETUP ~~~~~ #
# set up the first logger for the app
import os
import log as vlog
# path to the current script's dir
scriptdir = os.path.dirname(os.path.realpath(__file__))

def logpath():
    '''
    Return the path to the main log file; needed by the logging.yml
    use this for dynamic output log file paths & names
    '''
    global scriptdir
    return(vlog.logpath(scriptdir = scriptdir, logfile = 'log.txt'))

config_yaml = os.path.join(scriptdir,'logging.yml')
logger = vlog.log_setup(config_yaml = config_yaml, logger_name = "app")
logger.debug("App is starting...")

This is accompanied by log.py (imported as vlog in my main app);

log.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''
Functions to set up the app logger
'''
import yaml
import logging
import logging.config
import os

def logpath(scriptdir, logfile = 'log.txt'):
    '''
    Return the path to the main log file; needed by the logging.yml
    use this for dynamic output log file paths & names
    '''
    log_file = os.path.join(scriptdir, logfile)
    return(logging.FileHandler(log_file))

def log_setup(config_yaml, logger_name):
    '''
    Set up the logger for the script
    config = path to YAML config file
    '''
    # Config file relative to this file
    loggingConf = open(config_yaml, 'r')
    logging.config.dictConfig(yaml.load(loggingConf))
    loggingConf.close()
    return(logging.getLogger(logger_name))

Additionally, any other modules of my own which I import in app.py (after setting up the logging there) includes this logging setup at the start of the module:

data.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''
Module to do data stuff for my app
'''
import logging
logger = logging.getLogger("data")

I think this covers all the points you were talking about. It took me a while to figure this out myself. Hope it helps.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download