Is it possible to write many different log files using a single logger?

Question

My application manages the state of multiple objects, called Requests, over a substantial period of time. Each Request has a unique identifier and goes through a distinct lifecycle. New Requests arise in the system over time.

I'd like to write a separate log file for each Request. The log would track every interesting change to the state of that Request. So if I wanted to know everything about the history of Request X, it would be simple to go and look at X.log.

Obviously, I could hand-roll a solution using plain files. But I'd like to do this with Python's logging framework. One way would be to create a new logger instance for every unique Request, configure it to point at the correct file, and then log away. But this feels like the wrong solution. It creates many loggers, which aren't garbage-collected, and is also unbounded, since new Requests will continue to enter the system.

I was hoping for some way to configure a single logger, perhaps using a custom handler, so that I could redirect output to different files depending on the ID of the incoming Request. I've looked at the docs but everything I see seems to work at the level of incoming records, not manipulating outgoing endpoints.

Is this possible?

How do your Request objects leave the system? It seems like you could make a Logger object for each Request, configured appropriately, and then when your Request is decommissioned, it could tell the Manager (a logging.Manager subclass?) to remove the Logger instance. — Matt Anderson
– Matt Anderson, Commented Apr 4, 2013 at 23:36
@Matt Anderson - That might be possible. It's quite hard to know when a Request is done, but perhaps I could come up with something. I'm not sure how one removes logger instances though. They are module-scoped to the logging module, and don't seem to come with any clean-up API. Is a del sufficient? — ire_and_curses
– ire_and_curses, Commented Apr 5, 2013 at 1:06

ire_and_curses · Accepted Answer · 2013-04-05 06:29:26Z

Looking at the code for the RotatingFileHandler in logging.handlers eventually gave me enough clues to solve this. The key realisation is that when logging a message an optional extra keyword may be passed, which is a dictionary of attributes to be stored in the Record. This can be accessed from the Handler. Within the Handler, we can trigger a change of output stream on the basis of the user-supplied value.

import logging

class MultiFileHandler(logging.FileHandler):

    def __init__(self, filename, mode, encoding=None, delay=0):
        logging.FileHandler.__init__(self, filename, mode, encoding, delay)

    def emit(self, record):
        if self.should_change_file(record):
            self.change_file(record.file_id)
        logging.FileHandler.emit(self, record)

    def should_change_file(self, record):
        if not hasattr(record, 'file_id') or record.file_id == self.baseFilename:
             return False
        return True

    def change_file(self, file_id):
        self.stream.close()

        self.baseFilename = file_id
        self.stream = self._open()

if __name__ == '__main__':

    logger = logging.getLogger('request_logger')
    logger.setLevel(logging.DEBUG)    
    handler = MultiFileHandler(filename='out.log', mode='a')
    handler.setLevel(logging.DEBUG)    
    logger.addHandler(handler)

    # Log some messages to the original file
    logger.debug('debug message')
    logger.info('info message')

    # Log some messages to a different file
    logger.debug('debug message',       extra={'file_id':'changed.log'})
    logger.info('info message',         extra={'file_id':'changed.log'})
    logger.warn('warn message',         extra={'file_id':'changed.log'})
    logger.error('error message',       extra={'file_id':'changed.log'})
    logger.critical('critical message', extra={'file_id':'changed.log'})

Sean Vieira · Accepted Answer · 2013-04-05 01:29:02Z

0

Sounds like you are looking for a different logging system entirely - one that does not maintain any global state. Have you looked at Logbook?

Alternately, if you must avoid third-party dependencies you could use logging.addLevelName to add a level for each request and add a handler with a filter that drops every non-matching log entry to your logger for each request. Once the request goes out of scope you could call the handler's close method to remove it from the tree.

This will very likely not scale well however, as every handler's filter will be called for every log message.

answered Apr 5, 2013 at 1:29

Sean Vieira

161k34 gold badges321 silver badges297 bronze badges

Collectives™ on Stack Overflow

Is it possible to write many different log files using a single logger?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related