using python logging to the the existing script

Question

I have a chain of python scripts which are working perfectly fine. What I am trying to do is to using python logging module to log all the print statements I have in this script (forget about the purpose of this script. All I am trying to do is to add logging capability to save all those print statements intended to show up on console to be logged in to a log file):

from __future__ import print_function
#import MySQLdb
import pymysql
import shutil
import os
import sys
import io
import string
import traceback
import time  
from watchdog.observers import Observer  
from watchdog.events import PatternMatchingEventHandler
from colorama import *
init()
# ==== FOR THE TRANSFORMATION SCRIPT ====
from datetime import tzinfo, timedelta, datetime
import pytz
import codecs
from pytz import timezone
import csv
# =======================================

if not os.path.exists('Archive'):
    os.mkdir("Archive")
if not os.path.exists('Failed'):
    os.mkdir("Failed")

class MyHandler(PatternMatchingEventHandler):
    patterns = ["*.csv","*.processing", "*.transforming","*.loading"]

    def process(self, event):
        """
        event.event_type 
            'modified' | 'created' | 'moved' | 'deleted'
        event.is_directory
            True | False
        event.src_path
            path/to/observed/file
        """
        eventFileName = event.src_path
        eventType = event.event_type
        if eventType == 'moved':
            eventFileName = event.dest_path
        fileNameWithPath, fileExtension = os.path.splitext(eventFileName)
        if fileExtension == '.csv':
            print (Back.GREEN +'Entering copier.py...'+ Style.RESET_ALL)
            execfile('./copier_MySQL.py')
        #if fileExtension == '.processing':
        #    print (Back.GREEN +'Entering parser_MySQL.py...'+ Style.RESET_ALL)
        #    execfile('./parser_MySQL.py')
        if fileExtension == '.processing': #change this to '.transforming' if the above parser_MySQL.py is uncommented
            t = time.time()
            print (Back.GREEN +'Entering transform_MySQL.py...'+ Style.RESET_ALL)
            execfile('./transform_MySQL.py')
            elapsed = time.time() - t
            print (Back.MAGENTA +'Time took to transform the file = '+str(elapsed)+ Style.RESET_ALL)
            print (Back.BLUE + "Exiting transform_MySQL.py..." + Style.RESET_ALL)
            print (Fore.YELLOW + "-----------#*#*#*-----------#*#*#*-----------" + Style.RESET_ALL)
        if fileExtension == '.loading':
            if os.path.isfile(eventFileName):
                t = time.time()
                print (Back.GREEN +'Entering sweeper.py...'+ Style.RESET_ALL)
                execfile('./sweeper_MySQL.py')
                elapsed = time.time() - t
                print (Back.MAGENTA +'Time took to load the file into database = '+str(elapsed-1)+ Style.RESET_ALL)
                print (Back.BLUE +"Disconnected from the MySQL server. Exiting sweeper.py..."+ Style.RESET_ALL)
                print (Fore.YELLOW+'-----------#*#*#*-----------#*#*#*-----------'+ Style.RESET_ALL)
            else:
                print (Fore.RED+eventFileName+' is not created by transform_MySQL.py. Check the above messages. NOT executing sweeper_MySQL.py'+ Style.RESET_ALL) 
    def on_moved(self, event):
        self.process(event)

    def on_created(self, event):
        self.process(event)

if __name__ == '__main__':
    observer = Observer()
    observer.schedule(MyHandler(), path='.')
    observer.start()

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()

    observer.join()

I was looking at here and tried a few things but not able to quite fit that solution in my code. Any help is much appreciated.

Why not just redirect the output to a file? myscript.py > myscript.log — russdot
– russdot, Commented Mar 30, 2016 at 19:25
@russdot this script is intended to run on un-monitored computer where the progress (those print statements) are to be printed to screen and save the log messages to a file, so > myscript.log is what I really wish I can use but can not. — Nikhil Gupta
– Nikhil Gupta, Commented Mar 30, 2016 at 19:29
@russdot if you look at xi_'s comment to his answer below, he linked to python's logging module. All that functionality is automatically added to logging information buy a one line format statement. Doing this manually and writing to a log file is going to involve multiple lines of code which is kinda re-inventing the wheel, so is another reason for my reason to use python's logging module. — Nikhil Gupta
– Nikhil Gupta, Commented Mar 30, 2016 at 20:50
I'm glad you found a solution that works. The post you linked to describes using the python logging module to log to a file (though not as simply as xi_'s answer), so I presumed you needed an alternative. — russdot
– russdot, Commented Mar 30, 2016 at 21:01

xiº · Accepted Answer · 2016-03-30 19:28:19Z

3

You could use logging module instead of bunch of prints.

import logging
logging.basicConfig(level=logging.DEBUG)
logging.debug(Back.GREEN +'Entering sweeper.py...'+ Style.RESET_ALL)

To save output to file set:

logging.basicConfig(filename='myscript.log', level=logging.DEBUG)

edited Mar 30, 2016 at 19:28

answered Mar 30, 2016 at 19:25

xiº

4,7173 gold badges31 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Nikhil Gupta Over a year ago

that is what I thought but does it save those print statements to a file?

Nikhil Gupta Over a year ago

I will try it and get back. Also, can I use multiple levels in that statement something like level=logging.DEBUG,logging.ERROR,logging.INFO etc?

xiº Over a year ago

you could use messages with different level, for more info check this out docs.python.org/2/howto/logging.html

Nikhil Gupta Over a year ago

awesome, this worked and got to logging all including exceptions. :-)

Collectives™ on Stack Overflow

using python logging to the the existing script

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related