I am trying to use compile to runtime generate a Python function accepting arguments as follows.
import types
import ast
code = compile("def add(a, b): return a + b", '<string>', 'exec')
fn = types.FunctionType(code, {}, name="add")
print(fn(4, 2))
But it fails with
TypeError: <module>() takes 0 positional arguments but 2 were given
Is there anyway to compile a function accepting arguments using this way or is there any other way to do that?
Compile returns the code object to create a module. In Python 3.6, if you were to disassemble your code object:
>>> import dis
>>> dis.dis(fn)
0 LOAD_CONST 0 (<code object add at ...., file "<string>" ...>)
2 LOAD_CONST 1 ('add')
4 MAKE_FUNCTION 0
6 STORE_NAME 0 (add)
8 LOAD_CONST 2 (None)
10 RETURN_VALUE
That literally translates to make function; name it 'add'; return None.
This code means that your function runs the creation of the module, not returning a module or function itself. So essentially, what you're actually doing is equivalent to the following:
def f():
def add(a, b):
return a + b
print(f(4, 2))
For the question of how do you work around, the answer is it depends on what you want to do. For instance, if you want to compile a function using compile, the simple answer is you won't be able to without doing something similar to the following.
# 'code' is the result of the call to compile.
# In this case we know it is the first constant (from dis),
# so we will go and extract it's value
f_code = code.co_consts[0]
add = FunctionType(f_code, {}, "add")
>>> add(4, 2)
6
Since defining a function in Python requires running Python code (there is no static compilation by default other than compiling to bytecode), you can pass in custom globals and locals dictionaries, and then extract the values from those.
glob, loc = {}, {}
exec(code, glob, loc)
>>> loc['add'](4, 2)
6
But the real answer is if you want to do this, the simplest way is generally to generate Abstract Syntax Trees using the ast module, and compiling that into module code and evaluating or executing the module.
If you want to do bytecode transformation, I'd suggest looking at the codetransformer package on PyPi.
TL;DR using compile will only ever return code for a module, and most serious code generation is done either with ASTs or by manipulating byte codes.
is there any other way to do that?
For what's worth: I recently created a @compile_fun goodie that considerably eases the process of applying compile on a function. It relies on compile so nothing different than was explained by the above answers, but it provides an easier way to do it. Your example writes:
@compile_fun
def add(a, b):
return a + b
assert add(1, 2) == 3
You can see that you now can't debug into add with your IDE. Note that this does not improve runtime performance, nor protects your code from reverse-engineering, but it might be convenient if you do not want your users to see the internals of your function when they debug. Note that the obvious drawback is that they will not be able to help you debug your lib, so use with care!
See makefundocumentation for details.
compile does, and so does this decorator ;)compile is a "false friend" as we could think that it compiles (C-like, as in Cython compilation) python code and therefore can accelerate it. But it doesnt, it is just useful to create code dynamically from string.I fell into this trap myself some time ago,and lost time just experimenting in order to find the truth.Thanks to this decorator I am now able to prove myself easily that it does not bring any speed :) Still it removes the src from IDE debugging stack which may be convenient in rare edge cases. Thanks for clarifying your comment!I think this accomplishes what you want in a better way
import types
text = "lambda (a, b): return a + b"
code = compile(text, '<string>', 'eval')
body = types.FunctionType(code, {})
fn = body()
print(fn(4, 2))
The function being anonymous resolves the implicit namespace issues. And returning it as a value by using the mode 'eval' is cleaner that lifting it out of the code contents, since it does not rely upon the specific habits of the compiler.
More usefully, as you seem to have noticed but not gotten to using yet, since you import ast, the text passsed to compile can actually be an ast object, so you can use ast transformation on it.
import types
import ast
from somewhere import TransformTree
text = "lambda (a, b): return a + b"
tree = ast.parse(text)
tree = TransformTree().visit(tree)
code = compile(text, '<string>', 'eval')
body = types.FunctionType(code, {})
fn = body()
print(fn(4, 2))
exec(code)and thenadd(4, 2)?fnis not theadd()function. It is a function that will, upon execution, define anadd()function in its namespace (which you didn't save a reference to, so there's no way to actually invokeadd()).globals()['add'] = fnfollowed byadd(2, 4)should work? (it doesn't so I take it I understood it wrong).