Python logging module how to add class instance variable to message before sending it to parent logger

Question

I have some logger for multiple programs each with multiple handlers configured in a "__main__" logger and another package that creates a "__main__.package" logger so it uses that configuration for its messages. I dont have control of the loggers for the programs, only my package. My package should use that loggers settings. I can assume one is configured for every program using my package.

I want to inherit the formatter from main but modify the message/format before it gets sent to that formatter. Something very similar to this question except I am working with a child logger not the parent logger directly.

example:

main.py

import logging
import mypackage
logger = logging.getLogger("__main__")
logger.setLevel(logging.DEBUG)
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
stream_formatter = logging.Formatter('%(levelname)s:%(filename)s:%(lineno)d:%(message)s')
stream_handler.setFormatter(stream_formatter)
logger.addHandler(stream_handler)    

def main():
    a=1
    logger.info("message from main")
    b = mypackage.MyClass("EXTRA INSTANCE INFO")
    b.mymodule(a)

    c = mypackage.MyClass("EXTRA INSTANCE INFO2")
    c.mymodule(a+1)
    
main()

My attempt to add this info was to modify the message by adding a filter that modified the message before it was output:

mypackage.py

import logging
logger = logging.getLogger(f"__main__.{__name__}")

class _PackageContextFilter(logging.Filter):

    def __init__(self, needed_info_from_my_package):
        self.needed_info_from_my_package=needed_info_from_my_package

    def filter(self, record):
        record.msg = f"({self.needed_info_from_my_package}) " + record.msg
        return True
    
class MyClass():
    def __init__(self,init_cond) -> None:
        self.data=[]
        self.needed_info_from_my_package = [init_cond]
        # I think the issue arises here because a single logger object gets multiple filters
        logger.addFilter(_PackageContextFilter(self.needed_info_from_my_package))
        
        
    def mymodule(self,data):
        self.data.append(data)
        logger.info(f"message from mymodule about {self.data}")

output:

INFO:main.py:13:message from main
INFO:mypackage.py:24:(['EXTRA INSTANCE INFO']) message from mymodule about [1]
INFO:mypackage.py:24:(['EXTRA INSTANCE INFO2']) (['EXTRA INSTANCE INFO']) message from mymodule about [2]

my desired output is

INFO:main.py:13:message from main
INFO:mypackage.py:24:(['EXTRA INSTANCE INFO']) message from mymodule about [1]
INFO:mypackage.py:24:(['EXTRA INSTANCE INFO2']) message from mymodule about [2]

I think that this is being duplicated because I keep adding filters to the same logging object. I see in the logging cookbook there is an example that creates context vars to handle two instances of some class, but when I look up documentation on what a contextvar is it appears to be more related to threading. I'm not sure if it is a "good idea" to be creating them to solve this issue.

Another option I guess I can do is wrap every single input to the log message like logging.info(self.format_message_instance(msg)) but I figure there is a known best practice

Please provide a minimal reproducible example showing behavior that you believe should be modified. — chepner
– chepner, Commented Mar 21, 2024 at 20:40
Just to note, your logger probably should not be inheriting from the __main__ logger, but just documenting the logger(s) it does use for the importing application to configure (further) if it wants to. — chepner
– chepner, Commented Mar 22, 2024 at 17:56
@chepner can you explain why? I might be missing something but if there are a lot of packages each with their own logger then it seems the main programs are going to need to create handlers and formatters and filters for every one of them if the main one changes which is going to happen often. Is there an upside I'm not seeing? — guest
– guest, Commented Mar 22, 2024 at 18:24
also I dont see this in my questions list on my profile, the only way I can find it is by getting a notification comment. Am I still in some sort of moderator review sandbox? — guest
– guest, Commented Mar 22, 2024 at 18:26
The logger __main__ that your example creates is not the same as the root logger, which all loggers inherit from. Don't bake dependencies on any particular user into your package design. — chepner
– chepner, Commented Mar 22, 2024 at 20:16

Matt Prodani · Accepted Answer · 2024-03-22 18:31:28Z

1

Normally the filter wouldn't have reference to the MyClass instance that called it, therefore it can't know what info to add to the message. How about using an adapter that is unique to each instance similar to the cookbook example:

class CustomAdapter(logging.LoggerAdapter):
    def process(self, msg, kwargs):
        return f"({self.extra['info']}) {msg}", kwargs

    
class MyClass():
    def __init__(self,init_cond) -> None:
        self.data=[]
        self.needed_info_from_my_package = [init_cond]
        # here we initialize based on the main logger
        self.logger = CustomAdapter(logger, {'info': self.needed_info_from_my_package})
        
        
    def mymodule(self,data):
        self.data.append(data)
        self.logger.info(f"message from mymodule about {self.data}")

answered Mar 22, 2024 at 18:31

Matt Prodani

1099 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python logging module how to add class instance variable to message before sending it to parent logger

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related