Python: maximum recursion depth

Question

A function for returning sum of the sizes of its arguments which could be single file/directory or a list of files/directories, is given below. The code gives an error message RuntimeError: maximum recursion depth exceeded while calling a Python object however I try to test this.

How to fix this?

Thanks

suresh

#!/usr/bin/python3.1
import os

def fileSizes(f):
    if hasattr(f,'__iter__'):
        return sum(filter(fileSizes,f))
    if os.path.isfile(f):
        return os.path.getsize(f)
    elif os.path.isdir(f):
        total_size = os.path.getsize(f)
        for item in os.listdir(f):
            total_size += fileSizes(os.path.join(f, item))
        return total_size

does listdir include current directories? what are the iterables that you are looping over? — andrew cooke
– andrew cooke, Commented Aug 29, 2011 at 1:02
os.listdir() does not include current or parent directory. — suresh
– suresh, Commented Aug 29, 2011 at 1:07

Can · Accepted Answer · 2011-08-29 01:00:58Z

9

Instead of writing your own adhoc directory-transversal method, use the built-in os.walk (Documentation) method.

In fact, the example in the documentation (link above) computes the total size of non-directory files.

answered Aug 29, 2011 at 1:00

Can

8,60151 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

suresh Over a year ago

But os.walk() calls listdir() in turn right? I am using listdir() in the above code..So can it still be called adhoc?

Can Over a year ago

os.walk calls os.listdir(), it behaves the same way as your code, but iteratively, and also, allowing to follow/ignore symlinks. You should really take a look at the documentation, everything is clear there.

Seth Carnegie · Accepted Answer · 2011-08-29 13:32:28Z

3

Last time I checked, the default maximum recursion depth was 1000. You have a few options:

Use Stackless Python which imposes no recursion limit (allowing infinite recursion).
Rewrite your function in an iterative style rather than recursive, so you don't overflow the stack in the first place.
Set the maximum recursion limit with sys.setrecursionlimit. Beware that this can cause a segmentation fault if abused.

edited Aug 29, 2011 at 13:32

answered Aug 29, 2011 at 0:54

Seth Carnegie

75.3k22 gold badges185 silver badges252 bronze badges

3 Comments

Devin Jeanpierre Over a year ago

-1: setting sys.setrecursionlimit in a program is not safe and can cause segfaults. The only time when that's correct is when you're writing a one-time use script. :/

Seth Carnegie Over a year ago

@Devin doesn't mean I can't tell him it's an option.

Seth Carnegie Over a year ago

@Devin should we never use C because every C program you write is "potentially segfaulting?" I'll switch the order and add a warning if it makes you feel better.

Roshan Mathews · Accepted Answer · 2011-08-29 01:08:30Z

1

The problem is in the line:

if hasattr(f,'__iter__'):
    return sum(filter(fileSizes,f))

Since f is a path, it is a string, and it has the attribute __iter__, so you loop there infinitely.

answered Aug 29, 2011 at 1:08

Roshan Mathews

5,9142 gold badges28 silver badges37 bronze badges

6 Comments

suresh Over a year ago

Oh, then how can I test for list which is not valid for a string? Okay I can use isinstance() I believe....

Roshan Mathews Over a year ago

did that fix your problem? about your second question, you could try type(f) != str and hasattr(f,'__iter__') ...

suresh Over a year ago

@rm I used isinstance. Recursion issue is gone but the sum(filter(...)) is giving me some other kind of errors...

Roshan Mathews Over a year ago

Great, I fixed your original code then. Try to fix the rest yourself or ask here in another question.

Roshan Mathews Over a year ago

Okay, this is a different question now, (not what the comments are meant for), but I think you mean to use a map in your code, not a filter. If you have any other question, ask other questions, don't keep adding them in the comments.

|

Michael Anderson · Accepted Answer · 2011-08-29 00:59:44Z

1

There are many tricks to avoid problems with deep recursion, but I suspect that is not your core issue. (Maybe it is if you have very deep directory structures.. so test on a shallow directory first).

I'm guessing you're finding a cyclical structure in your directory tree caused by links. I.e. a symlinked directory pointing back into a parent directory. (This kind of structure is not unusual).

You can avoid the infinite loop by checking whether a directory is a symlink before following it.

Another posibility is that you're getting "." or ".." somewhere in your lists. These will also generate infinite loops. I don't see that happening here, but its worth thinking about. os.listdir doesn't seem to return them.

answered Aug 29, 2011 at 0:59

Michael Anderson

74.3k9 gold badges147 silver badges196 bronze badges

2 Comments

suresh Over a year ago

@Anderson, I tried running on a shallow directory, but it looks like, irrespective of the argument, the code gives the same error message. So I cant test it as of now ...

Michael Anderson Over a year ago

Running your code "as is" in my directory produces results - in particular I tried fileSize(".") in a couple of directories. But the results look wrong.. (For example I get 68 when run in an empty directory).

Collectives™ on Stack Overflow

Python: maximum recursion depth

4 Answers 4

2 Comments

3 Comments

6 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

3 Comments

6 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related