Determining maximum valid setrecursionlimit value for Python

Question

In the Python 2 documentation, the sys library contains the following (bolded part is my edit):

sys.setrecursionlimit(limit)

Set the maximum depth of the Python interpreter stack to limit. This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python.

The highest possible limit is platform-dependent. A user may need to set the limit higher when she has a program that requires deep recursion and a platform that supports a higher limit. This should be done with care, because a too-high limit can lead to a crash.

What does this mean? Is this just a general "make sure you have enough memory to handle the extra stack space" statement, or is there a specific "per stack frame" size that can be used to calculate the memory value required? What happens to Python when it can't acquire the space?

What happens to Python when it can't acquire the space? Quoting from your quote: This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python. - you can safely assume CPython interpreter will "crash" - most likely it'll segfault. — Łukasz Rogalski
– Łukasz Rogalski, Commented Jan 19, 2016 at 18:06

Wayne Werner · Accepted Answer · 2016-01-19 18:15:30Z

3

Why let's find out:

me@host$ docker run -m 4MB --cpuset-cpus=0 -it --rm python:3.5.1
Python 3.5.1 (default, Dec  9 2015, 00:12:22)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, struct
>>> maxint = 2 ** (struct.Struct('i').size * 8 - 1) - 1
>>> sys.setrecursionlimit(maxint)
>>> def goodbye_world():
...  goodbye_world()
...
>>> goodbye_world()
me@host$

Welp. Looks like it crashes Python. Pretty quickly, too, when you only give it 4MB of RAM.

answered Jan 19, 2016 at 18:15

Wayne Werner

52.3k35 gold badges213 silver badges304 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mike Williamson Over a year ago

This is a great example, but I don't really understand everything that's happening. For instance, with a 64-bit machine + 64-bit Python, struct.Struct('i').size is 8, so that maxint is (2 ** 63) - 1. But it isn't obvious to me why this is a good choice for setting a recursion limit. Can I assume that each recursion will only consume 1 i-unit (32/64 bit)? Why? Is this because each recursion only consumes one address in the memory? Would you mind elaborating? Thanks so much!

Wayne Werner Over a year ago

@MikeWilliamson it's only been a few years since I wrote this answer, so I don't truly remember why I used that value, but I think it's because that's the maximum addressable size for that architecture (so, a 64-bit number for a 64-bit system). I definitely wasn't suggesting that it's a good recursion limit (as the crash evidences). A good recursion limit is one that 1) your platform supports, and 2) allows you to do what you need. Sometimes 1 prevents 2.

Collectives™ on Stack Overflow

Determining maximum valid setrecursionlimit value for Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related