Python - run function when folder changes

Question

I want to run my function whenever a change occurs in my folder. The main thing to do here is to normalize new files, we do so by comparing my two folders, the folder that contains the normalized ones and the folder where we add new files and return a list of the unmatched files which our normalize function will modify and place in the folder of normalized ones.

Here is my code:

def normalize(c):
    PATH = path_to_file 
    fileNames = os.listdir(PATH)

    fileNames = c
    for file in fileNames:
        sfm = pd.read_csv(PATH + file, delimiter = ';', parse_dates=[[0, 1]], skiprows = 3)
  # other instructions then rewrite my new dataframe in my normalized folder
sfm.to_csv('.\dirnormalized'+str(file[:-4])+'.csv', sep=';', index=False, header=True)

def comparison():
    PATH = path_to_file 
    PATH_norm = path_to_norm
    fileNames = os.listdir(PATH)
    files = os.listdir(PATH_norm)
    fileNames = [file for file in fileNames if '.csv' in file]
    files = [fl for fl in files if '.csv' in fl]
    diff = [x for x in fileNames if x not in files]
    return diff

def change():

    logging.basicConfig(level=logging.INFO,
                        format='%(asctime)s - %(message)s',
                        datefmt='%Y-%m-%d %H:%M:%S')
# path_to_file and word_file are the same the only difference is that in path_to_file we have a back slash in the end in order to acces the files one by one 
    path = word_file
    event_handler = LoggingEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

def process(): 
    if(change()):
        c = comparison()
        normalize(c)
if __name__ == "__main__":
    import os 
    import time
    import logging
    from watchdog.observers import Observer
    from watchdog.events import LoggingEventHandler

    process()

The only problem here is that in my process() function the if() instruction doesn't seem pass to other ones, I can see when the new files are added which means when a modification occurs but the other instructions don't occur it stucks on if().

You have no return in your function change(). So the line if(change()): in your process() function is actually always if(None): so the code in that statement never runs. — Jeyekomon
– Jeyekomon, Commented May 15, 2018 at 7:32

vidstige · Accepted Answer · 2018-05-15 07:30:20Z

1

The last line of process() namely observer.join() will probably block the thread preventing the function from returning.

Moreover, control leave the function returning None causing the if statement to never be executed.

answered May 15, 2018 at 7:30

vidstige

13.1k9 gold badges70 silver badges115 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

RandomHash · Accepted Answer · 2018-05-15 07:59:50Z

Check out this for changes in a dir.

Use the MS ReadDirectoryChanges API, exposed via the pywin32 win32file module.

The way we employ it here is to use call ReadDirectoryChangesW in blocking mode.

The function returns a list of 2-tuples, each one representing an action and a filename. A rename always gives a pair of 2-tuples; other compound actions may also give a list.

import os
import win32file
import win32con

ACTIONS = {
  1 : "Created",
  2 : "Deleted",
  3 : "Updated",
  4 : "Renamed from something",
  5 : "Renamed to something"
}
# Thanks to Claudio Grondi for the correct set of numbers
FILE_LIST_DIRECTORY = 0x0001

path_to_watch = "."
hDir = win32file.CreateFile (
  path_to_watch,
  FILE_LIST_DIRECTORY,
  win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE | win32con.FILE_SHARE_DELETE,
  None,
  win32con.OPEN_EXISTING,
  win32con.FILE_FLAG_BACKUP_SEMANTICS,
  None
)
while 1:
  #
  # ReadDirectoryChangesW takes a previously-created
  # handle to a directory, a buffer size for results,
  # a flag to indicate whether to watch subtrees and
  # a filter of what changes to notify.
  #
  # NB Tim Juchcinski reports that he needed to up
  # the buffer size to be sure of picking up all
  # events when a large number of files were
  # deleted at once.
  #
  results = win32file.ReadDirectoryChangesW (
    hDir,
    1024,
    True,
    win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
     win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
     win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |
     win32con.FILE_NOTIFY_CHANGE_SIZE |
     win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |
     win32con.FILE_NOTIFY_CHANGE_SECURITY,
    None,
    None
  )
  for action, file in results:
    full_filename = os.path.join (path_to_watch, file)
    print full_filename, ACTIONS.get (action, "Unknown")

With some editing, this could easily achieve what you want, as the winApi is leveraged, its likely to be less error prone.

Thanks Tim Golden.

Collectives™ on Stack Overflow

Python - run function when folder changes

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related