subclassed python dictionary for custom namespace in exec() method

Question

I am trying to subclass a dictionary for use in an exec method. Ultimately, I would like the local function to have a custom name scoping behaviour.

In the code below, function b() does in fact have the correct globals() dictionary available to it, however it fails to search it when resolving z.

Does function b() first not search locals() then globals() ?

Very puzzling. Any help appreciated.

t = '''
def b():
#   return (globals()['z']) #works
    return z #fails

b()
'''

class MyDict(dict):
    def __init__(self, g):
        dict.__init__(self)
        self.my_g = g


    def __getitem__(self, key):
        print("GET ", key)
        try:
            val = dict.__getitem__(self, key)
        except:
            print("GET exception1")
            val = self.my_g[key]
        return val


g = {'z':123}

md = MyDict(g)

#fails to find z
exec(t, md, md)

#works
#exec(t, g, g)

output

GET  b
Traceback (most recent call last):
  File "/project1/text12", line 31, in <module>
  File "<string>", line 6, in <module>
  File "<string>", line 4, in b
NameError: global name 'z' is not defined

return z fails because there is no variable z. You say return (globals()['z']) works, what do you want to accomplish when returning z? — Niek de Klein
– Niek de Klein, Commented Aug 29, 2012 at 19:36
I want the z to be resolved from the global dictionary passed in as working in the answer below. Cheers — rbairos
– rbairos, Commented Aug 31, 2012 at 2:06

Martijn Pieters · Accepted Answer · 2019-03-04 19:09:46Z

6

Before python 3.3, you cannot use a custom dict subclass for the globals value of an exec statement. The underlying C code that executes the compiled code accesses the underlying C structures directly, ignoring any custom hooks you may have implemented.

In other words, when the code does a LOAD_GLOBAL operation (as is the case with z in your function b), the C bytecode evaluation loop accesses the globals structure in the current frame using the C API, bypassing any python overrides.

This is documented in the exec() function documentation as:

If only globals is provided, it must be a dictionary, which will be used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object.

This restriction has been loosened in Python 3.3, see issue 14385. The documentation hasn't been updated yet, but the bytecode evaluation loop has been updated to test for custom mappings before falling back to the C API access. If a custom mapping is used, the PyObject_GetItem function is used, which will call __getitem__ on custom classes.

edited Mar 4, 2019 at 19:09

answered Aug 29, 2012 at 19:39

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

16 Comments

rbairos Over a year ago

thank you very much. I havent had a chance to work this back into my original work but it looks great. Thanks again -Rob

rbairos Over a year ago

Sorry @martijn-pieters , its not quite there yet. Your addition works for finding variables in the local scope, but not in the function scope as the original example has. Im also using Python 3.2 if thats relevant. In other words: t = 'print(z)' now works but: t = ''' def b(): return z b() ''' still fails. Any insight as to how to get the function scope to search the globals dictionary in that case, as it does when I pass in g directly? Thanks.

Martijn Pieters Over a year ago

@rbairos: that's what I explained in the first half; unbound variables in the function scope are always looked up in the global() scope next. There is no module-local scope here that normally also puts things in the global() scope.

rbairos Over a year ago

But, @martijn-pieters isnt the global() scope the dictionary I pass in as an argument to exec() ? ie. exec(t, md, md) ? Confused as to why it doesnt attempt to search md, as it searches g when I call (t,g,g) ? (sorry this subtlety is new to me)

Martijn Pieters Over a year ago

@rbairos: investigating a little if python 3 makes a difference.

|

Collectives™ on Stack Overflow

subclassed python dictionary for custom namespace in exec() method

output

1 Answer 1

16 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

output

1 Answer 1

16 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related