8

I like the python logging infrastructure and I want to use it for a number of different overnight jobs that I run. A lot of these jobs use module X let's say. I want the logging for module X to write to a log file not dependent on module X, but based on the job that ultimately led to calling functionality in module X.

So if overnight_script_1.py calls foo() in module X, I want the log of foo() to go to overnight_script_1.log. I also want overnight_script_2.py's call of foo() to log to overnight_script_2.log.

A potential solution to this problem would be to set up the log file based on looking at the 0th argument to sys.argv which can be mapped to my preferred file. Something seems hacky about doing it this way. Is there a preferred design pattern for doing this? I don't want to rummage through different log files based on the module where the function was called to find my diagnostic information for one of my scripts. Here is some code because I am not sure I am making myself clear.

Here is script1.py

import X
import logging_utils as lu
import sys

logname=sys.argv[0][:-3] # logname==script1 with the .py cut off
logger=lu.setup_logger(log_name) # assume this does the formatting and sets the filehandlers
# furthermore assume the file handler is set so that the output goes to script1.log.

# do a bunch of thing
logger.info('I am doing a bunch of things in script1 and I will now call X.foo()')
X.foo() # see module X below
logger.info('I finished X.foo()')

Similary, here is script2.py

import X
import logging_utils as lu
import sys

logname=sys.argv[0][:-3] # logname==script2 with the .py cut off
logger=lu.setup_logger(log_name) # assume this does the formatting and sets the filehandlers
# furthermore assume the file handler is set so that the output goes to script2.log.

# do a bunch of thing
logger.info('I am doing a bunch of things in script2 and I will now call X.foo()')
X.foo() # see module X below
logger.info('I finished X.foo()')

Here is X.py

import logging
import sys
logname=sys.argv[0][:-3] # could be script1 or script2
logger=logging.getLogger(logname)

def foo():
    try:
        i=1/0
    except:
        logger.error('oops - division by zero')

Then I want to run:

python script1.py

python script2.py

and get two log files script1.log and script2.log where the division by zero error that occurred in module X is logged in each.

Improve this question
10
  • Can you pass in the path of the log file to write to into foo() as a paramater? Commented Oct 12, 2016 at 14:17
  • @PrestonM - doesn't seem like the best solution to me. I really have foo1, foo2, foo3, ... foo945, ... etc. Commented Oct 12, 2016 at 14:19
  • 1
    Usually you set up logging in the script you execute, not in the module itself, e.g. by calling logging.basicConfig() with the appropriate arguments. Where and how are you setting up logging handlers, formatters etc... currently? Commented Oct 12, 2016 at 14:41
  • I don't understand what you are asking. The X modulo should not define any handler. The scripts are responsible for defining the handlers (either to a file, or to the console), set the level for logging and add the handlers to the logger... so the standard configuration already completely addresses the situation you are describing. Commented Oct 12, 2016 at 14:43
  • 1
    @FinanceGuyThatCantCode No. The handler decides where the output goes, and you should not define handlers in x at all. You define handlers only in your scripts. Commented Oct 12, 2016 at 15:34

2 Answers 2

Reset to default
11

I believe you should just follow the standard set up for library code:

Say you have a package mypkg and you want this package to log information while letting the user of the package decide which level of logging to use and where the output should go.

The mypkg should not set the level nor any handler for the logs. Module x of mypkg should be something like this:

import logging

logger = logging.getLogger(__name__)


def my_function():
    logger.info('something')

No configuration of the logging whatsoever just getLogger(__name__).

Then your script.py that uses mypkg and wants to log at level INFO on the console will do:

import logging
import mypkg

root_logger = logging.getLogger()
console_handler = logging.StreamHandler()

root_logger.setLevel(logging.INFO)
root_logger.addHandler(console_handler)

mypkg.my_function()  # produces output to stderr

While script2.py will log at level DEBUG to a file:

import logging
import mypkg


root_logger = logging.getLogger()
file_handler = logging.FileHandler('some_file.log')

root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(file_handler)

mypkg.my_function()

Note that by setting the level and handler on the root logger we are setting the level "globally". If the user has some own loggers that he wants to use with level DEBUG but he wants to use level INFO for mypkg he can do:

root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)

mypkg_logger = logging.getLogger('mypkg')
mypkg_logger.setLevel(logging.INFO)

handler = #... whatever you want
root_logger.addHandler(handler)

If your mypkg contains the modules x.py, y.py, a.py, b.py and c.py and you want to log the functions inside x at DEBUG and those inside y at WARNING and those for a/b/c.py at INFO you can do so by setting the levels on the corresponding loggers:

mypkg_logger = logging.getLogger('mypkg')
my_pkg_logger.setLevel(logging.INFO)

x_logger = logging.getLogger('mypkg.x')
x_logger.setLevel(logging.DEBUG)
y_logger = logging.getLogger('mypkg.y')
y_logger.setLevel(logging.WARNING)

Put the handlers only attached to the root_logger and everything should work fine.

Note: on some versions of python the logging module might warn you that you defined a logger with no handlers for the library modules. To fix ths you can use a NullHandler that just drops the logging messages:

# x.py

logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())

So the NullHandler literally does nothing, except from preventing the logging module from complaining. AFAIK newer versions do not need this at all.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Ahh so the key point that I seem to have been missing is that there is a root_logger which is the default return from getLogger() which we can modify from the calling script with a preferred FileHandler. It appears that you are saying that getLogger(name) will then use this FileHandler until told otherwise.
Also I edited my original question to have code snippets.
@FinanceGuyThatCantCode It's a bit more complicated. Loggers form a hierarchy based on their name so a logger with name mypkg.x is "a child" of the logger mypkg. Using getLogger(__name__) is just a simple way of calling your loggers mypkg, then mypkg.x, then mypkg.y then mypkg.subpkg.x etc. The root logger is a grandparent of all loggers. When you call .info or .debug on a logger the logger check if it matches its level, and if so it tries to log. If it doesn't find a handler it will send that item to the parent logger etc. up to the root logger.
@FinanceGuyThatCantCode If you want to set the level on a per-module basis you can directly get the logger for that module and set the level there, this prevents messages with lower level to go up to the root logger. Logging configuration can become really complicated since you have a level for each logger, logging propagation and you can also set the level of handlers...
@FinanceGuyThatCantCode You should probably read: Logging HOWTO, Logging Cookbook, Advanced logging tutorial.
|
-1

You could probably use as many RotatingFileHandler(s) as different modules you have and setup their filenames accordingly

https://docs.python.org/2/library/logging.handlers.html

-Edit-

Based on the comment then you may want to use a Formatter that takes into account where was the call made (to know if the call to the shared library was made from module_1 or module_2 and so on) and then and then simply log to the same file.

There are some caveats to think about:

  • I'm not sure if there is order preservation in the writing of the file because there may be two or more processes that write to that file
  • The log file may increase in size arbitrarily ending up with 0 disk space

https://docs.python.org/2/library/logging.html#formatter-objects https://docs.python.org/2/library/logging.html#logrecord-attributes https://docs.python.org/2.7/library/logging.config.html#configuration-file-format

Improve this answer

1 Comment

not sure I understand. The rotating handlers appear to rotate based on log file sizes - or at least that seems like the intent. Mainly I am trying to consolidate the logging for one of my scripts to a single log file - I don't want the logs for one app to be sprayed over many different log files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.