saving variables within function - python

Question

I'd like to save to disk all the variables I create within a particular function in one go, so that I can load them later. Something of the type:

>>> def test():
      a=1
      b=2
      save.to.file(filename='file', all.variables)

>>> load.file('file')
>>> a
>>> 1

Is there a way to do this in python? I know cPickle can do this, but as far as I know, one has to type cPickle.dump() for every single variable, and my script has dozens. Also, it seems that cPickle stores only the values and not the names of the variables, so one has to remember the order the data was originally saved.

Why not pickle the whole function? You can use test.a or test.b to retrieve the variable. — zhangyangyu
– zhangyangyu, Commented Aug 5, 2013 at 21:51
@zhangyangyu: Nope. Function attributes aren't local variables. — user2357112
– user2357112, Commented Aug 5, 2013 at 21:57
Why do you have dozens of variables, each of which needs to be saved? It sounds to me like either you're using too many variables where something like a list would be more appropriate, or you're trying to save everything instead of considering which things need to be saved and which should be reset or reconstructed when the program restarts. — user2357112
– user2357112, Commented Aug 5, 2013 at 21:59

abarnert · Accepted Answer · 2013-08-07 18:01:39Z

2

Assuming all of the variables you want to save are local to the current function, you can get at them via the locals function. This is almost always a very bad idea, but it is doable.

For example:

def test():
    a=1
    b=2
    pickle.dump(file, locals())

If you print locals(), you'll see that it's just a dict, with a key for each local variable. So, when you later load the pickle, what you'll get back is that same dict. If you want to inject it into your local environment, you can… but you have to be very careful. For example, this function:

def test2():
    locals().update(pickle.load(file))
    print a

… will be compiled to expect a to be a global, rather than a local, so the fact that you've updated the local a will have no effect.

This is just one of the reasons it's a bad idea to do this.

So, what's the right thing to do?

Most simply, instead of having a whole slew of variables, just have a dict with a slew of keys. Then you can pickle and unpickle the dict, and everything is trivial.

Or, alternatively, explicitly pickle and unpickle the variables you want by using a tuple:

def test():
    a = 1
    b = 2
    pickle.dump(file, (a, b))

def test2():
    a, b = pickle.load(file)
    print a

In a comment, you say that you'd like to pickle a slew or variables, skipping any that can't be pickled.

To make things simpler, let's say you actually just want to pickle a dict, skipping any values that can't be pickled. (The above should show why this solution is still fully general.)

So, how do you know whether a value can be pickled? Trying to predict that is a tricky question. Even if you had a perfect list of all pickleable types, that still wouldn't help—a list full of integers can be pickled, but a list full of bound instance methods can't.

This kind of thing is exactly why EAFP ("Easier to Ask Forgiveness than Permission") is an important principle in duck-typed languages like Python.* The way to find out if something can be pickled is to pickle it, and see if you get an exception.

Here's a simple demonstration:

def is_picklable(value):
    try:
        pickle.dumps(value)
    except TypeError:
        return False
    else:
        return True

def filter_dict_for_pickling(d):
    return {key: value for key, value in d.items() if is_picklable((key, value))}

You can make this a bit less verbose, and more efficient, if you put the whole stashing procedure in a wrapper function:

def pickle_filtered_dict(d, file):
    for key, value in d.items():
        pickle.dump((key, value), file)
    except TypeError:
        pass

def pickles(file):
    try:
        while True:
            yield pickle.load(file)
    except EOFError:
        pass

def unpickle_filtered_dict(file):
    return {key: value for key, value in pickles(file)}

edited Aug 7, 2013 at 18:01

answered Aug 5, 2013 at 22:48

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

HappyPy Over a year ago

Thanks, that's exactly what I was looking for. Basically, I just want to save all variables within a given session in a file, for future reference (just in case I need to go back and see what values were assigned to what variables). But is there a way to tell pickle to save only those objects that are "pickleable", or perhaps telling locals() not to save the "non-pickleable" ones (I'm getting an error message - ValueError - due to these).

abarnert Over a year ago

@user2635863: Let me update the answer with something that might help for your new problem.

HappyPy Over a year ago

many thanks for both the clear explanation and demonstration.

GreyZmeem · Accepted Answer · 2013-08-05 22:00:05Z

1

You can use pickle to store objects in files:

>>> a = {'a': 1, 'b': 2, 'c': 3}
>>> a
{'a': 1, 'c': 3, 'b': 2}
>>>
>>> import pickle
>>> pickle.dump(a, open('a.dump', 'wb'))
>>>
>>> a = None
>>> a
>>>
>>> a = pickle.load(open('a.dump', 'rb'))
>>> a
{'a': 1, 'c': 3, 'b': 2}
>>>

More information at: http://docs.python.org/2/library/pickle.html

answered Aug 5, 2013 at 22:00

GreyZmeem

6601 gold badge7 silver badges11 bronze badges

Comments

roippi · Accepted Answer · 2013-08-05 22:08:34Z

1

If you are not satisfied with the API of pickle, consider shelve which does the pickling for you with a nicer dict-like front end.

ex.

>>> import shelve
>>> f = shelve.open('demo')
>>> f
<shelve.DbfilenameShelf object at 0x000000000299B9E8>
>>> list(f.keys())
['test', 'example']
>>> del f['test']
>>> del f['example']
>>> list(f.keys())
[]
>>> f['a'] = 1
>>> list(f.keys())
['a']
>>> list(f.items())
[('a', 1)]

edited Aug 5, 2013 at 22:08

answered Aug 5, 2013 at 21:57

roippi

26k4 gold badges52 silver badges75 bronze badges

Comments

gggg · Accepted Answer · 2013-08-05 21:52:19Z

0

You can pickle multiple variables by making them attributes of an object, or elements in a dictionary. In either case it saves the name as well. It's not quite as convenient as save in MATLAB, but it's better than nothing.

answered Aug 5, 2013 at 21:52

gggg

2,3961 gold badge17 silver badges19 bronze badges

Collectives™ on Stack Overflow

saving variables within function - python

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related