Setting stacksize in a python script

Question

I am converting a csh script to a python script. The script calls a memory-intensive executable which requires a very large stack, so the csh script sets the stacksize to unlimited:

limit stacksize unlimited

When I try to reproduce this script in python, I execute them in a very naive manner, using os.system, e.g.:

os.system('some_executable')

But I do not know how to tell the OS to run these executables with unlimited stacksize. Is there a way to specify stacksize for calls within a python script? Is there some low-level system call that I should be using? And is there a module (similar to shutil) which controls this?

Attempting to set rlimit_stack after Stack Clash remediations may result in failure or related problems. Also see Red Hat Issue 1463241 — jww
– jww, Commented Jun 21, 2017 at 16:21

Thomas Ahle · Accepted Answer · 2022-04-03 00:59:20Z

56

I have good experience with the following code. It doesn't require any special user permissions:

import resource, sys
resource.setrlimit(resource.RLIMIT_STACK, (2**29,-1))
sys.setrecursionlimit(10**6)

It does however not seem to work with pypy.

If resource.setrlimit doesn't work, you can also try using threads:

sys.setrecursionlimit(10**6)
import threading
threading.stack_size(2**26)
threading.Thread(target=main).start()

edited Apr 3, 2022 at 0:59

answered Apr 27, 2013 at 3:45

Thomas Ahle

31.8k21 gold badges99 silver badges120 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Wilfred Hughes Over a year ago

sys.setrecursionlimit(10**6) will make the Python interpreter segfault if you overflow the stack.

ofer.sheffer Over a year ago

Note that, as of this moment, the resource module is not available for Windows OS.

Thomas Ahle Over a year ago

@ofer.sheffer Still better than the accepted answer which uses os.system.

garg10may Over a year ago

@ThomasAhle I am getting the error no module named resource

user3064538 Over a year ago

@ThomasAhle what is the -1? Did you mean resource.RLIM_INFINITY?

|

Nicholas Riley · Accepted Answer · 2011-02-21 02:07:24Z

20

You can just use the (u)limit command of your shell, if you want:

os.system('ulimit -s unlimited; some_executable')

Or (probably better) use resource.setrlimit:

resource.setrlimit(resource.RLIMIT_STACK, (resource.RLIM_INFINITY, resource.RLIM_INFINITY))

answered Feb 21, 2011 at 2:07

Nicholas Riley

44.5k6 gold badges107 silver badges125 bronze badges

2 Comments

marshall.ward Over a year ago

The first option doesn't work for me; I believe it is because os.system executes commands in the basic shell, sh, rather than bash or tcsh. But your second option is exactly what I needed, thank you. (I can't tell who suggested setrlimit first so I am awarding this one, since it includes the explicit code. I am grateful for everyone's answers, though).

Nicholas Riley Over a year ago

Yeah, I should have mentioned that - some shells use limit, some use ulimit, some allow both; the explicit invocation is a much better idea.

sarnold · Accepted Answer · 2011-02-21 02:04:13Z

15

You're looking for the Python setrlimit interface, resource.RLIMIT_STACK.

Note that standard users cannot raise their hard limits, only root (well, a process with the CAP_SYS_RESOURCE capability (see capabilities(7)) processes can raise their limits; so you may need to use the PAM pam_limits(8) limits.conf(5) file to raise the hard limits for the users in question.

answered Feb 21, 2011 at 2:04

sarnold

105k23 gold badges187 silver badges243 bronze badges

Comments

ncoghlan · Accepted Answer · 2020-05-05 15:05:29Z

5

You can alter the stack size of the current process via threading.stack_size(), but I don't know if that will be correctly inherited by subprocesses. That interface also requires a specific stack size - "unlimited" isn't an option.

edited May 5, 2020 at 15:05

user3064538

answered Feb 21, 2011 at 2:10

ncoghlan

41.8k11 gold badges77 silver badges83 bronze badges

Collectives™ on Stack Overflow

Setting stacksize in a python script

4 Answers 4

6 Comments

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

6 Comments

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related