Using program variables in python logging configuration file

Question

I am trying to enable my python logging using the following:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
import logging.config
import os
test_filename = 'my_log_file.txt'
try:
    logging.config.fileConfig('loggingpy.conf', disable_existing_loggers=False)
except Exception as e:
    # try to set up a default logger
    logging.error("No loggingpy.conf to parse", exc_info=e)
    logging.basicConfig(level=logging.WARNING, format="%(asctime)-15s %(message)s")
test1_log = logging.getLogger("test1")
test1_log.critical("test1_log crit")
test1_log.error("test1_log error")
test1_log.warning("test1_log warning")
test1_log.info("test1_log info")
test1_log.debug("test1_log debug")

I would like to use a loggingpy.conf file to control the logging like the following:

[loggers]
keys=root

[handlers]
keys=handRoot

[formatters]
keys=formRoot

[logger_root]
level=INFO
handlers=handRoot

[handler_handRoot]
class=FileHandler
level=INFO
formatter=formRoot
args=(test_filename,)

[formatter_formRoot]
format=%(asctime)s:%(name)s:%(process)d:%(lineno)d %(levelname)s %(message)s
datefmt=
class=logging.Formatter

Here I am trying to route the logging to the file named by the local "test_filename". When I run this, I get:

ERROR:root:No loggingpy.conf to parse
Traceback (most recent call last):
  File "logging_test.py", line 8, in <module>
    logging.config.fileConfig('loggingpy.conf', disable_existing_loggers=False)
  File "/usr/lib/python2.7/logging/config.py", line 85, in fileConfig
    handlers = _install_handlers(cp, formatters)
  File "/usr/lib/python2.7/logging/config.py", line 162, in _install_handlers
    args = eval(args, vars(logging))
  File "<string>", line 1, in <module>
NameError: name 'test_filename' is not defined
CRITICAL:test1:test1_log crit
ERROR:test1:test1_log error
WARNING:test1:test1_log warning

Reading the docs, it seems that the "args" value in the config is eval'd in the context of the logging package namespace rather than the context when fileConfig is called. Is there any decent way to try to get the logging to behave this way through a configuration file so I can configure a dynamic log filename (usually like "InputFile.log"), but still have the flexibility to use the logging config file to change it?

I misread the question and thus deleted my (wildly mistaken) comment. — msw
– msw, Commented Aug 23, 2015 at 19:09

Lukas Körfer · Accepted Answer · 2017-09-07 11:34:23Z

14

The args statement is parsed using eval at logging.config.py _install_handlers. So you can add code into the args.

[handler_handRoot]
class=FileHandler
level=INFO
formatter=formRoot
args=(os.getenv("LOG_FILE","default_value"),)

Now you only need to populate the environment variable.

edited Sep 7, 2017 at 11:34

Lukas Körfer

14.7k8 gold badges55 silver badges71 bronze badges

answered Sep 7, 2017 at 10:26

user7837926

Sign up to request clarification or add additional context in comments.

2 Comments

eyalstoler Over a year ago

Amazing! so simple.

Joe Over a year ago

Can this option work when using a dictConfig instead of fileConfig?

keocra · Accepted Answer · 2017-07-06 15:04:05Z

11

Even though it's an old question, I think this still has relevance. An alternative to the above mentioned solutions would be to use logging.config.dictConfig(...) and manipulating the dictionary.

MWE:

log_config.yml

version: 1
disable_existing_loggers: false
formatters:
  default:
    format: "%(asctime)s:%(name)s:%(process)d:%(lineno)d %(levelname)s %(message)s"
handlers:
  console:
    class: logging.StreamHandler
    formatter: default
    stream: ext://sys.stdout
    level: DEBUG
  file:
    class: logging.FileHandler
    formatter: default
    filename: "{path}/service.log"
    level: DEBUG
root:
  level: DEBUG
  handlers:
  - file
  - console

example.py

import logging.config
import sys
import yaml

log_output_path = sys.argv[1]

log_config = yaml.load(open("log_config.yml"))
log_config["handlers"]["file"]["filename"] = log_config["handlers"]["file"]["filename"].format(path = log_output_path)
logging.config.dictConfig(log_config)

logging.debug("test")

Executable as follows:

python example.py .

Result:

service.log file in current working directory contains one line of log message.
Console outputs one line of log message.

Both state something like this:
2016-06-06 20:56:56,450:root:12232:11 DEBUG test

answered Jul 6, 2017 at 15:04

keocra

6435 silver badges11 bronze badges

3 Comments

CrazyFrog Over a year ago

I like this. This is clean and not corrupt the namespace.

Grmpfhmbl Over a year ago

If I'm not mistaken, this requires that the key (handlers/file/filename) exists. If not this will fail with an exception. Every user of your script should be aware that in case they rename the file-handler or want to get rid of it completely etc this script would fail.

keocra Over a year ago

You are right, the script would fail if there are changes which are not reflected correctly at all places. That is why I labeled it as MWE. It is not intended to be copied and used in a product but should show a possible solution to the given problem.

Steve Hart · Accepted Answer · 2015-08-21 18:21:42Z

10

You could place the filename in the logging namespace with:

logging.test_filename = 'my_log_file.txt'

Then your existing loggingpy.conf file should work

You should be able to pollute the logging namespace with anything you like (within reason - i wouldn't try logging.config = 'something') in your module and that should make it referencable by the the config file.

answered Aug 21, 2015 at 18:21

Steve Hart

1164 bronze badges

1 Comment

rkh Over a year ago

This is simple and works great. With Sean's answer below, you can even do args=(os.path.join('.', test_filename),) to help put files where you want them. Thanks!

Sean Pradhan · Accepted Answer · 2015-08-21 18:29:38Z

2

This is very hacky so I wouldn't recommend it. But if you for some reason did not want to add to the logging namespace you could pass the log file name through a command line argument and then use sys.argv[1] to access it (sys.argv[0] is the script name).

[handler_handRoot]
class=FileHandler
level=INFO
formatter=formRoot
args=(sys.argv[1],)

edited Aug 21, 2015 at 18:29

answered Aug 21, 2015 at 18:16

Sean Pradhan

1015 bronze badges

1 Comment

rkh Over a year ago

This is good because it points out that certain things are available in the logging namespace (like sys.* and os.*), but not your local namespace.

RubenLaguna · Accepted Answer · 2023-12-18 08:42:06Z

The fileConfig configuration file format is sort of deprecated (see the note).

Using the dictConfig you can reference environment variable from a YAML configuration file using the ext:// prefix and referencing a variable that you populated with the envvar value like this

#logging.yaml
handlers:
  console:
    class: logging.FileHandler
    level: DEBUG
    formatter: formRoot
    filename: ext://__main__.log_file

where __main__.log_file is just a global variable log_file below:

import logging.config
import yaml

log_file = os.getenv("LOG_FILE","default_value")

with open('logging.yaml', 'r') as f:
    config = yaml.safe_load(f.read())
    logging.config.dictConfig(config)

log = logging.getLogger(__name__)
log.info("test")

Collectives™ on Stack Overflow

Using program variables in python logging configuration file

5 Answers 5

2 Comments

3 Comments

1 Comment

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

3 Comments

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related