Logging different python files in the same file

Question

I have a custom module called bar.py, which I import in some of my scripts. It is actually imported via pip install from the pypi index. That bar.py module looks like this (this is actually an oversimplified example of what I actually have so I can ringfence my doubt here.

import logging

def do_bar_logged():
    logging.INFO('START DOING SOMETHING')
    do_bar()
    logging.INFO('END DOING SOMETHING')

One of the scripts in which I import bar's do_bar_logged is called foo.py. It looks something like this.

import logging
from bar import do_bar_logged  # I need to install via pip bar before being able to import this, just like a regular module.

def main():
    logging.INFO('START CALLING DO_BAR')
    do_bar_logged()
    logging.INFO('END CALLING DO_BAR')

My question here is: would it be possible to have both loggings pointing to the same file without harcoding anything?

There's built-in logging which made specifically for this. Take a look on Logging HOWTO. — Olvin Roght
– Olvin Roght, Commented Jan 15, 2022 at 16:39
It is possible. But it’s not clear what you are asking. There are a lot of ways you can achieve what you are looking for — Adelin
– Adelin, Commented Jan 15, 2022 at 16:41
@Adelin I would need to log into the same file stuff placed in several scripts. Would that be possible without hardcoding the file name? — Javi Torre
– Javi Torre, Commented Jan 15, 2022 at 16:42
Sure. But you would also ideally not be logging using print calls. — khelwood
– khelwood, Commented Jan 15, 2022 at 16:48

Yevhen Bondar · Accepted Answer · 2022-01-15 17:00:24Z

2

You can use logging module like this

bar.py

import logging


def aux_print():
    logging.warning('This is my auxiliary script')

main.py

import logging
from bar import aux_print

logging.basicConfig(
    filename="log.txt",
    filemode='a',
    format='[%(asctime)s,%(msecs)d] [%(levelname)s]: %(message)s',
    datefmt='%H:%M:%S',
    level=logging.DEBUG
)


def main():
    logging.info('This is my main script')
    aux_print()


if __name__ == '__main__':
    main()

Output log.txt:

[18:58:23,708] [INFO]: This is my main script
[18:58:23,708] [WARNING]: This is my auxiliary script

answered Jan 15, 2022 at 17:00

Yevhen Bondar

4,7471 gold badge15 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Javi Torre Over a year ago

Thanks! Would this work also for the edit I made to my question?

Yevhen Bondar Over a year ago

@JaviTorre If bar library uses logging module than yes, it should write logs into file.

Javi Torre Over a year ago

Thanks! So, if I understood correctly, the logging library logs to the handler from which the parent script is calling it?

tdelaney · Accepted Answer · 2022-01-15 17:06:43Z

1

The logging module lets you replace calls to print with calls to special logging functions that can be configured to run in all sorts of ways. The top end logging calls stay the same while you substitute back end handlers to log where you want.

Logging multiple scripts to a single destination can be tricky. You can't just write a single file because the logs will tend to overwrite each other (a locking file handler can help). One of the simplest things to do is use the SysLogHandler (linux/mac) and NTEventLogHandler (windows) handlers to log to the regular system log.

Or you could use some of the low level handlers (HTTP, socket, dgram, queue), to write your own logging server that aggregates logs. You could even use a SQL or NOSQL handler to log to a database. Logging is a big deal and there are lots of 3rd party logging modules out there. Its worth a look to see if any are useful to you.

answered Jan 15, 2022 at 17:06

tdelaney

77.9k6 gold badges91 silver badges129 bronze badges

2 Comments

Javi Torre Over a year ago

Thanks! The logging to a DB part is actually a very good idea. Would this also work for the edit I just did to my answer?

tdelaney Over a year ago

Yes. Your scripts grab the logger, but you also need something to set them up. You can do that with a python script that knows how to create the logging handlers that you want. Put it in your module path and then have you top level scripts import it. One option is logging.config.fileconfig where you use a config file instead. You could program all of your scripts to grab, for instance, ~/.my_logging_config.ini and use whatever is in it.

Collectives™ on Stack Overflow

Logging different python files in the same file

2 Answers 2

3 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related