python: logging all errors into single log file

Check out the python module logging. This is a core module for unifying logging not only in your own project but potentially in third party modules too.

For a minimal logging file example, this is taken directly from the documentation:

import logging
logging.basicConfig(filename='example.log',level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')

Which results in the contents of example.log:

DEBUG:root:This message should go to the log file
INFO:root:So should this
WARNING:root:And this, too

However, I personally recommend using the yaml configuration method (requires pyyaml):

#logging_config.yml

version: 1
disable_existing_loggers: False
formatters:
  standard:
    format: '%(asctime)s [%(levelname)s] %(name)s - %(message)s'
handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: standard
    stream: ext://sys.stdout
  file:
    class: logging.FileHandler
    level: DEBUG
    formatter: standard
    filename: output.log
  email:
    class: logging.handlers.SMTPHandler
    level: WARNING
    mailhost: smtp.gmail.com
    fromaddr: [email protected]
    toaddrs: [email protected]
    subject: Oh no, something's gone wrong!
    credentials: [email, password]
    secure: []
root:
  level: DEBUG
  handlers: [console, file, email]
  propagate: True

Then to use, for example:

import logging.config
import yaml

with open('logging_config.yml', 'r') as config:
    logging.config.dictConfig(yaml.safe_load(config))

logger = logging.getLogger(__name__)

logger.info("This will go to the console and the file")
logger.debug("This will only go to the file")
logger.error("This will go everywhere")

try:
    list = [1, 2, 3]
    print(list[10])
except IndexError:
    logger.exception("This will also go everywhere")

This prints:

2018-07-18 13:29:21,434 [INFO] __main__ - This will go to the console and the file
2018-07-18 13:29:21,434 [ERROR] __main__ - This will go everywhere
2018-07-18 13:29:21,434 [ERROR] __main__ - This will also go everywhere
Traceback (most recent call last):
  File "C:/Users/Chris/Desktop/python_scratchpad/a.py", line 16, in <module>
    print(list[10])
IndexError: list index out of range

While the contents of the log file is:

2018-07-18 13:35:55,423 [INFO] __main__ - This will go to the console and the file
2018-07-18 13:35:55,424 [DEBUG] __main__ - This will only go to the file
2018-07-18 13:35:55,424 [ERROR] __main__ - This will go everywhere
2018-07-18 13:35:55,424 [ERROR] __main__ - This will also go everywhere
Traceback (most recent call last):
  File "C:/Users/Chris/Desktop/python_scratchpad/a.py", line 15, in <module>
    print(list[10])
IndexError: list index out of range

Of course, you can add or remove handlers, formatters, etc, or do all of this in code (see the Python documentation) but this is my starting point whenever I use logging in a project. I find it helpful to have the configuration in a dedicated config file rather than polluting my project with defining logging in code.

If I understand the question correctly, the request was to capture only the errors in a dedicated log file, and I would do that differently.

I would stick to the BKM that all modules in the package define their own logger objects (logger = logging.getLogger(__name__)).

I'd let them be without any handlers and whenever they will emit they will look up the hierarchy tree for handlers to actually take care of the emitted messages.

At the root logger, I would add a dedicated FileHandler(filename='errors.log') and I would set the log level of that handler to logging.ERROR.

That means, whenever a logger from the package will emit something, this dedicated file-handler will discard anything below ERROR and will log into the files only ERROR and CRITICAL messages.

You could still add global StreamHandler and regular FileHandler to your root logger. Since you'll not change their log levels, they will be set to logging.NOTSET and will log everything that is emitted from the loggers in the package.

And to answer the second part of the question, the logger handlers can define their own formatting. So for the handler that handles only the errors, you could set the formatter to something like this: %(name)s::%(funcName)s:%(lineno)d - %(message)s which basically means, it will print:

• the logger name (and if you used the convention to define loggers in every *.py file using __name__, then name will actually hold the hierarchical path to your module file (e.g. my_pkg.my_sub_pkg.module))

• the funcName will hold the function from where the log was emitted and

• lineno is the line number in the module file where the log was emitted.