Python module structure and logging

Question

I'm trying to log to the same file from the main flow and submodule:

/proxy
/proxy/proxy/network.py
/proxy/proxy.py

network.py:

import logging
print __name__
...

logging.getLogger(__name__).warning("test warn")

proxy.py:

import logging
print __name__
logging.basicConfig(filename="console", format="%(message)s", disable_existing_loggers=False)
logging.warning("test main")

Then I see logging in proxy.py works, while in network.py doesn't, obviously because their name's are:

proxy.network
__main__

respecitvely.

If I put them to the same folder their names would be like:

network
__main__

which also wouldn't work. Looks like I have to tell somehow that my project root is one level up relative to proxy.py (wherever it is), in this case they both would have name "proxy.*".

How to point the project root folder to interpreter? How to unify their module structure names, to enable logging(in either module structure)?

akash12300 · Accepted Answer · 2016-03-26 11:27:17Z

1

Simplest approach to save all logging info to same file is, do NOT provide a parameter in getLogger() [instead of what you are using now, getLogger(name)]

But keep in mind that if you are using some other module which use logging (outside your project, some third party maybe, like httplib) they might end up saving data in your log file with this approach

answered Mar 26, 2016 at 11:27

akash12300

4223 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

aluriak · Accepted Answer · 2016-03-26 13:14:21Z

1

EDIT: here is an access to the file name related to the whole package:

filepath = __file__ if __name__ == '__main__' else __name__

For the following structure, it gives:

mypack
├── __main__.py   # -> mypack.__main__
└── sub
    ├── __init__.py
    └── sub.py  # -> mypack.sub.sub

Here is a snippet:

def logger(name=None, root_name=PACKAGE_NAME):
    if name:
        return logging.getLogger(root_name + '.' + name)
    else:
        return logging.getLogger(root_name)

I usually define this function in a commons.py or utils.py, imported by all the modules of my package.

Taking advantage of the sublogger system, this allows a package to use the main logging of the package:

import utils
logger = utils.logger()
logger.info('module xxx started')

Or a sublogger dedicated to a particular concept:

import utils
logger = utils.logger('vm')
logger.info('module vm.xxx started')

The file path and the logging config are now decoupled and explicitely binded. Which can be automatize for each module:

import utils
logger = utils.logger(__file__)
logger.info('module ' + __file__ + ' started')

Because of the sublogger system, the logger 'pk.vm' will (by default) inherit for 'pk' configuration, that is to say using the same handlers, thus writing in the same logging file. However, it can be useful to define some particular behavior or handlers for any sublogger:

logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s',
        },
        'simple': {
            'format': '%(levelname)s %(message)s',
        },
    },
    'handlers': {
        'console':{
            'level':LOG_LEVEL,
            'class':'logging.StreamHandler',
            'formatter': 'simple',
        },
        'logfile': {
            'level': LOG_LEVEL,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': DIR_LOGS + LOGGER_NAME + '.log',
            'mode': 'w',
            'maxBytes': LOGFILE_MAX_SIZE,
            'formatter': 'verbose',
        },
        'logfile' + SUBLOGGER_SEPARATOR + SUBLOGGER_VM: {
            'level': LOG_LEVEL,
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': DIR_LOGS + LOGGER_NAME + '.' + SUBLOGGER_VM + '.log',
            'mode': 'w',
            'maxBytes': LOGFILE_MAX_SIZE,
            'formatter': 'verbose',
        },
    },
    'loggers': {
        PACKAGE_NAME: {
            'handlers':['console', 'logfile'],
            'propagate': True,
            'level':LOG_LEVEL,
        },
        PACKAGE_NAME + SUBLOGGER_SEPARATOR + SUBLOGGER_VM: {
            'handlers':['logfile' + SUBLOGGER_SEPARATOR + SUBLOGGER_VM],
            'level':LOG_LEVEL,
        },
    }
})

edited Mar 26, 2016 at 13:14

answered Mar 26, 2016 at 11:46

aluriak

5,9572 gold badges29 silver badges42 bronze badges

3 Comments

noname7619 Over a year ago

Should work, but what can you say about usage of name's reflecting the module structure? In different answers/manual they always say "file network.py is in submodule relative to proxy.py (main.py)" but I cannot realize how to make use of this if one of the files is on the top level.

aluriak Over a year ago

As far i can see, this requires the module to know where it is in the package architecture. This is probably not a good solution.

aluriak Over a year ago

i've edited my answer : _name_ and _file_ could be used for retrieving the file place in architecture.

Collectives™ on Stack Overflow

Python module structure and logging

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related