Logging to multiple log files from different classes in Python

Question

I want to write a Python class which uses Python logging. This Python class will be responsible for the creating a file with a given name in init function.

I want to create a object of the above class in two or more classes and expect two or files getting generated.

I tried writing this class but I am not able to create multiple files.

Can anyone guide me how do I do that?

I have created the following class:

class Logger:
def __init__(self, log_filename = "test.log"):
    if not os.path.exists("LogFiles"):
        os.makedirs("LogFiles")
    self.Logger = logging.getLogger("main")
    logging.basicConfig(level=logging.DEBUG,
                        format='%(asctime)s : %(message)s',
                        filename= log_filename,
                        filemode='w')           # change filemode to 'w' to overwrite file on each run

    consoleHandler = logging.StreamHandler()
    consoleHandler.setLevel(logging.DEBUG)
    formatter = logging.Formatter('%(asctime)s - %(message)s')
    consoleHandler.setFormatter(formatter)
    logging.getLogger('').addHandler(consoleHandler)      # Add to the root logger
    self.Logger.info("Starting new logging sessions")


def writeToFile(self, line):
    if self.Logger.propagate == True:
        self.Logger.debug(line)

def closeFile(self):

    if self.Logger.propagate == True:
        self.Logger.propagate = False

@J.F.Sebastian I believe that doesn't quite do what he's after; it sends output from a single logger to two locations. It also makes use of the very magic parent logging. The OP wants to send output to two different locations with independent loggers. — jpmc26
– jpmc26, Commented Jun 12, 2013 at 6:08
Here is an example solution, the trick is naming the handler. stackoverflow.com/a/76941199/10389449 — Sevki Baba
– Sevki Baba, Commented Aug 20, 2023 at 20:27

Community · Accepted Answer · 2017-05-23 11:54:37Z

81

Sounds like the internals of your class should probably have a Logger and that you'll want to add a FileHandler to the Logger. You might want to consider just using a factory method that creates Loggers and adds the handler instead of implementing your own class. You may need to create the directories that hold the log files. See this answer for advice on creating directories.

Edit:

I don't think you need to write your own Logger class. Python's logging module has all the pieces you need. You probably just need a factory method. The key to realize is you need to create two separate, completely independent logging objects. You do this with logging.getLogger, and any time you pass it a different name, it gives you a different logger. You can use anything you want for the logger's name. For sure, you want to stay away from basicConfig for what you're doing. It's designed to be something simple for people who just want one Logger not doing anything too special.

I think this demonstrates the functionality you're after. The key is create two different loggers with different handlers. Then use them separately. Keep in mind that my second call to logging.getLogger doesn't create a new logger; it gets the one we set up initially in setup_logger.

log_test.py:

from __future__ import absolute_import

import logging

def setup_logger(logger_name, log_file, level=logging.INFO):
    l = logging.getLogger(logger_name)
    formatter = logging.Formatter('%(asctime)s : %(message)s')
    fileHandler = logging.FileHandler(log_file, mode='w')
    fileHandler.setFormatter(formatter)
    streamHandler = logging.StreamHandler()
    streamHandler.setFormatter(formatter)

    l.setLevel(level)
    l.addHandler(fileHandler)
    l.addHandler(streamHandler)    

def main():
    setup_logger('log1', r'C:\temp\log1.log')
    setup_logger('log2', r'C:\temp\log2.log')
    log1 = logging.getLogger('log1')
    log2 = logging.getLogger('log2')

    log1.info('Info for log 1!')
    log2.info('Info for log 2!')
    log1.error('Oh, no! Something went wrong!')

if '__main__' == __name__:
    main()

Sample run:

C:\temp>C:\Python\27\python.exe logtest.py
2013-06-12 02:00:13,832 : Info for log 1!
2013-06-12 02:00:13,832 : Info for log 2!
2013-06-12 02:00:13,832 : Oh, no! Something went wrong!

log1.log:

2013-06-12 02:00:13,832 : Info for log 1!
2013-06-12 02:00:13,832 : Oh, no! Something went wrong!

log2.log:

2013-06-12 02:00:13,832 : Info for log 2!

edited May 23, 2017 at 11:54

CommunityBot

11 silver badge

answered Jun 11, 2013 at 5:26

jpmc26

30.2k14 gold badges100 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

Shrikanth Kalluraya Over a year ago

I am new to logging module and I see even if I am giving different filename in basicConfig there is only one file created. Let me tell what my problem is Suppose there is a class A, class B and file C. C has a main function where I create object of class A and B. Class A logs should go to A.log file and Class B log should go to B.log file. I dont want any log of A to come B.log or viceversa. So can I create a common logger class and use this class in both A and B. Any example with psuedocode would help

Shrikanth Kalluraya Over a year ago

I have added the generic class which I wanna create. Can you tell me what am I doing wrong?

jfs Over a year ago

+1. setup_logger() might be a better name (getLogger() creates loggers if necessary). You might want l.propagate = False to avoid passing messages to ancestor loggers. Logger's name should probably be derived from a class name (a period in logger's name is special that is how logger hierarchy is created i.e., . in a name should be avoided unless explicitly desired (you could use logging_tree to visualize it)).

jpmc26 Over a year ago

@J.F.Sebastian I think whether to use class names for the logger names depends on the usage. If it's one per class, then yes, it makes sense. If you really just have two log files and you want some classes to use one and some classes to use another, then it probably makes more sense just to use an explicit name.

jpmc26 Over a year ago

@MakleBirt It's the file path, not the name. See here.

|

Sherd · Accepted Answer · 2019-03-05 15:56:56Z

This is better handled using dictConfig. You can then specify logging to two seperate files. I utilize the second_logger only when an API post is made so that I can log external data to the second log.

import os, logging
from logging.config import dictConfig

FORMAT = "%(asctime)s {app} [%(thread)d] %(levelname)-5s %(name)s - %(message)s. [file=%(filename)s:%(lineno)d]"
DATE_FORMAT = None


def setup_logging(name, level="INFO", fmt=FORMAT):
    formatted = fmt.format(app=name)
    log_dir = r'C:/log_directory'
    if not os.path.exists(log_dir):
        os.makedirs(log_dir)

    logging_config = {
        "version": 1,
        'disable_existing_loggers': False,
        "formatters": {
            'standard': {
                'format': formatted
            }
        },
        "handlers": {
            'default': {
                'class': 'logging.StreamHandler',
                'formatter': 'standard',
                'level': level,
                'stream': 'ext://sys.stdout'
            },
            'file': {
                'class': 'logging.handlers.TimedRotatingFileHandler',
                'when': 'midnight',
                'utc': True,
                'backupCount': 5,
                'level': level,
                'filename': '{}/app_manager.log'.format(log_dir),
                'formatter': 'standard',
            },
            'file2': {
                'class': 'logging.handlers.TimedRotatingFileHandler',
                'when': 'midnight',
                'utc': True,
                'backupCount': 5,
                'level': level,
                'filename': '{}/unified_log.log'.format(log_dir),
                'formatter': 'standard',
            }
        },
        "loggers": {
            "": {
                'handlers': ['default', 'file'],
                'level': level
            },
            "second_log": {
                'handlers': ['default', 'file2'],
                'level': level
            }
        }
    }

    dictConfig(logging_config)

log.setup_logging(name="log-name", level=LEVEL
logger = logging.getLogger(__name__)
second_logger = logging.getLogger('second_log')
second_logger.propagate = False

Collectives™ on Stack Overflow

Logging to multiple log files from different classes in Python

2 Answers 2

12 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

12 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related