How to keep track of virtual memory used when running Python code?

Question

I have a Python package I am benchmarking for virtual memory used.

At the moment, this isn't very precise. I submit the script e.g.

python script1.py

and look at the time and resources used in order to execute.

Is there some package/method to find out where my RAM bottlenecks are? I'm thinking of a tool like

ram_used python script1.py

that would keep track of how much RAM is used.

Does such a thing exist?

What do you mean with RAM bottlenecks? Do you want to know where the program is using a lot of memory? Or simply how much? — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Jul 7, 2017 at 17:05
The general type of tool is called a profiler, and it usually binds to your operating system, rather than a particular language. — Prune
– Prune, Commented Jul 7, 2017 at 17:07

Tom Wyllie · Accepted Answer · 2017-07-07 17:07:25Z

2

Yes, there is - It's called memory_profiler. Once you've gone through and added the @profile decorator to the functions you would like to profile, you can use;

python -m memory_profiler example.py

answered Jul 7, 2017 at 17:07

Tom Wyllie

2,08215 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Prune Over a year ago

What Python version? Both 2.7 and 3.5.4 deny any knowledge of memory_profiler for me.

Tom Wyllie Over a year ago

Have you installed it using pip? pip install -U memory_profiler works fine for me using python 2.7.1.

Prune Over a year ago

Thanks; my "install" command is aliased to yum; pip worked, and the profiling worked.

ShanZhengYang Over a year ago

@TomWyllie This is exceptionally useful for checking individual functions of the Python package. However, it's a bit cumbersome to add decorators to all functions in order to check the entire package. Is there a way to check the virtual memory used for the entire package?

Tom Wyllie Over a year ago

I've not tested this but my first idea would be to just decorate the entry point to the program, eg your if __name__ == '__main__'. Although this may not fit your use case.

|

ShanZhengYang · Accepted Answer · 2017-07-08 22:14:47Z

-1

There's also memory_info psutil. https://pythonhosted.org/psutil/

Place at the beginning of the script:

import psutil
p = psutil.Process()

and given that the code is running correctly, place this on the last line:

print(p.memory_info())

answered Jul 8, 2017 at 22:14

ShanZhengYang

18k52 gold badges143 silver badges249 bronze badges

Collectives™ on Stack Overflow

How to keep track of virtual memory used when running Python code?

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related