0

I have many hundreds of python functions I'm evaluating for memory usage, and many of them differ simply by the usage of intermediate variable assignments. I would like a way, if possible, to automatically refactor these functions to remove such intermediate variable assignments. This way, the memory differences between the functions would be due to algorithm logic instead of just additional variable assignments.

For example, consider the two functions

def se1(vals1, vals2):
    diff = [(v1 - v2) for v1, v2 in zip(vals1, vals2)]
    diff_sq = [d**2 for d in diff]
    return(sum(diff_sq))

def se2(vals1, vals2):
    return(sum([(x-y)**2 for x,y in zip(vals1, vals2)]))

When I measure memory usage for them using tracemalloc:

import numpy as np
import tracemalloc

listA = np.random.randint(0, 1000, 1000)
listB =  np.random.randint(0, 1000, 1000)

tracemalloc.start()
se1(listA, listB)
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
tracemalloc.reset_peak()
print("Peak memory usage of se1: ", peak)

tracemalloc.start()
se2(listA, listB)
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
tracemalloc.reset_peak()
print("Peak memory usage of se2: ", peak)

the usages are vastly different:

Peak memory usage of se1: 66722

Peak memory usage of se2: 34229

Even though they are producing the same answer using the same logic. I feel like there should be a way to remove intermediate variables using successive substitution within the function body, but I can't think of a general and streamlined way to do it.

4
  • I know in principle how to do this. But it requires parsing and rewriting code. Python ships with a disassembler named dis. I'd suggest seeing if you can find and fix the optimizations using that. Alternately, it is possible that pypy does this kind of optimization for you. Unfortunately its debugging tools aren't as rich. But if you simply want speed... Commented Oct 12, 2024 at 3:23
  • @btilly I’ve looked into dis and while it helps to do an initial partitioning of the functions, I’m not clear how to use it to automatically rewrite functions. Preferably I would want something like an operator that takes in the source code of a function and returns it with all intermediate values collapsed. Commented Oct 12, 2024 at 22:46
  • No, it doesn't do that for you. But it at least puts Python into a consistent and easier to parse format than ths original. It turns this problem from basically impossible to doable with standard compilation techniques. Which are themselves a whole body of theory that takss some time to learn. Sorry. Commented Oct 13, 2024 at 12:20
  • Reading some more, docs.python.org/3.9/library/ast.html#module-ast is a better option. It presents the parse tree in a form that can be edited, before compilation. At the bottom of the module they offer links to other documents explaining how to do interesting things with it. Commented Oct 13, 2024 at 12:36

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.