30

So I'm developing a rather large python project with a number of modules. The "main" (runnable) module is a daemon (a Thrift daemon, actually) which calls off to other modules for its actual functionality. Starting up the daemon takes a long time because some of the modules have a rather lengthy and involved initialization processes.

So when I start the daemon, I wait... let's say... 2 minutes for everything to load, which isn't too bad in the grand scheme of things. However for development it becomes a major pain because I need to restart the daemon EVERY TIME which has been wasting a lot of my time.

Most modules only takes a few seconds to load. Ideally what I'd like to do is detect when any of the files in a particular module have changed, and reload that particular module. I've already figured out how to reload a module, but at this point I can't figure out how to watch a particular module for changes. Keep in mind that a module isn't a single .py file in this case, but rather a directory with __init__.py and 5-10 .py files, so I need to detect when any of them have changed.

Here is the project layout (if it makes any difference at all)

project
| -- daemonize.py
| -- main.py
| -- moduleA
|    | -- __init__.py
|    | -- happy_panda.py
|    ` -- sad_panda.py
| -- moduleB
|    | -- __init__.py
|    | -- takes_forever_to_load.py
|    ` -- seriously_get_some_coffee.py
| -- moduleC
|    | -- __init__.py
|    | -- frequently_changes.py
|    | -- reasons_i_hate_my_job.txt
|    ` -- home_address_of_moduleB_developer.txt
` -- service.py <-- uses modules A, B, and C

Any ideas or suggestions are appreciated.

EDIT

Thanks for the great feedback. Here is the code I created based on the suggestions. There's a small bug where pyinotify seems to be getting more than one notification, but it's a very small problem for me so I'm not going to fix it.

https://gist.github.com/1013122

Improve this question
2
  • Doesn't reload already check that the file has changed before doing anything? Commented Jun 7, 2011 at 19:19
  • @Bastien: After looking at the documentation for reload, no I don't believe that's how reload behaves. Correct me if I'm wrong. Commented Jun 7, 2011 at 19:23

3 Answers 3

Reset to default
9

Detect File Change Without Polling

Coupled with you already knowing how to reload your module this answer pretty much fills it out. It uses Inotify to "notify" (see what they did there) the program when the file is modified.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Very nice! I was thinking of trying to figure out if a module changed right before I call it, however doing it this way I can just reload the moment a file changes. Wouldn't have thought of that.
Glad to help, and glad I found your question actually, this'll come in handy down the line for something for I'm working on and I hadn't considered this technique as a time saver before.
2

It's an old question, but if you stumble upon it, here is a solution if you use IPython:

Enter the following lines in the IPython prompt at any time

%load_ext autoreload
%autoreload 2

For more options, check out this link: https://ipython.readthedocs.io/en/stable/

Improve this answer

1 Comment

Here is the source code if you want to browse how they implemented it, and see some caveats: github.com/ipython/ipython/blob/master/IPython/extensions/…
1

I would look at all of the files, and detect if a file was modified. If it was, I would reload it.

Improve this answer

5 Comments

How would you handle determining which files belong to a module? From the top level all I have is the module name.
I would look into the folder recursively. Are there files that change often that aren't code or configuration files?
Also, what are you doing on startup that takes 2 minutes?
Aside from reasons_i_hate_my_job.txt, no not really
Oh and you don't want to know. Part of it is loading a number of very large objects into memory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.