Recursively convert python object graph to dictionary

Question

I'm trying to convert the data from a simple object graph into a dictionary. I don't need type information or methods and I don't need to be able to convert it back to an object again.

I found this question about creating a dictionary from an object's fields, but it doesn't do it recursively.

Being relatively new to python, I'm concerned that my solution may be ugly, or unpythonic, or broken in some obscure way, or just plain old NIH.

My first attempt appeared to work until I tried it with lists and dictionaries, and it seemed easier just to check if the object passed had an internal dictionary, and if not, to just treat it as a value (rather than doing all that isinstance checking). My previous attempts also didn't recurse into lists of objects:

def todict(obj):
    if hasattr(obj, "__iter__"):
        return [todict(v) for v in obj]
    elif hasattr(obj, "__dict__"):
        return dict([(key, todict(value)) 
            for key, value in obj.__dict__.iteritems() 
            if not callable(value) and not key.startswith('_')])
    else:
        return obj

This seems to work better and doesn't require exceptions, but again I'm still not sure if there are cases here I'm not aware of where it falls down.

Any suggestions would be much appreciated.

in python it is not so bad to use exceptions and sometimes it can simplify the coding, a pythonic way- EAFP (Easier to Ask Forgiveness than Permission) — Anurag Uniyal
– Anurag Uniyal, Commented Jun 24, 2009 at 4:52
special case could be when object has slots, edited answer — Anurag Uniyal
– Anurag Uniyal, Commented Jun 24, 2009 at 4:56
point taken, but the exception thing is a bit of a holy war and i tend towards prefering them never to be thrown unless something is truly exceptional, rather than expected program flow. each to their own on that one :) — Shabbyrobe
– Shabbyrobe, Commented Jun 24, 2009 at 4:59
yes mostly it is style but sometime exception can really simply code e.g. when inner most functions communicate thru exceptions, and sometime it can speed up things e.g. if 99.99 % objects have dict in that case instead of checking 99.99% times for dict, check for exception .01% time — Anurag Uniyal
– Anurag Uniyal, Commented Jun 24, 2009 at 5:01

4 revs, 3 users 82% · Accepted Answer · 2018-07-16 12:52:10Z

66

An amalgamation of my own attempt and clues derived from Anurag Uniyal and Lennart Regebro's answers works best for me:

def todict(obj, classkey=None):
    if isinstance(obj, dict):
        data = {}
        for (k, v) in obj.items():
            data[k] = todict(v, classkey)
        return data
    elif hasattr(obj, "_ast"):
        return todict(obj._ast())
    elif hasattr(obj, "__iter__") and not isinstance(obj, str):
        return [todict(v, classkey) for v in obj]
    elif hasattr(obj, "__dict__"):
        data = dict([(key, todict(value, classkey)) 
            for key, value in obj.__dict__.items() 
            if not callable(value) and not key.startswith('_')])
        if classkey is not None and hasattr(obj, "__class__"):
            data[classkey] = obj.__class__.__name__
        return data
    else:
        return obj

edited Jul 16, 2018 at 12:52

community wiki

4 revs, 3 users 82%
Shabbyrobe

Sign up to request clarification or add additional context in comments.

5 Comments

Murat Ayfer Over a year ago

nicely done. only implementation that works just as i wanted it to, so far.

Sander Verhagen Over a year ago

Thank you, that mostly worked. Few caveats: in Python 3.5 iteritems() should be items(), and [todict(v, classkey) for v in obj] (line 10) tries to iterate characters in strings, fixed that with: elif hasattr(obj, "__iter__") and not isinstance(obj, str).

Vaibhav Tripathi Over a year ago

This converts my string values to None. A small but important miss. I solved this problem by catching objects of string type and returning str(obj) in those cases.

Cesar Flores Over a year ago

trying to convert a complex object with nested objects, I am getting this error. RecursionError: maximum recursion depth exceeded while calling a Python object and traceback points to this line data = dict([(key, todict(value, classkey)). Any idea of what this could be?

Grav Over a year ago

I needed to switch elif hasattr(obj, "__iter__") ... and elif hasattr(obj, "__dict__"): branches to avoid everything being turned into lists. The specific type was qdrant_client.http.models.models.CollectionInfo, but I suspect many classes implement iteration somehow

vvvvv · Accepted Answer · 2020-04-21 20:09:58Z

42

One line of code to convert an object to JSON recursively.

import json

def get_json(obj):
  return json.loads(
    json.dumps(obj, default=lambda o: getattr(o, '__dict__', str(o)))
  )

obj = SomeClass()
print("Json = ", get_json(obj))

edited Apr 21, 2020 at 20:09

vvvvv

32.9k19 gold badges70 silver badges103 bronze badges

answered Feb 27, 2018 at 8:08

Archit Dwivedi

4864 silver badges9 bronze badges

7 Comments

Tengerye Over a year ago

May I ask if there is a such concise way of loading an object from JSON?

Archit Dwivedi Over a year ago

You can do this obj.__dict__ = { 'key': 'value'}

Arigion Over a year ago

This solution did not work for me. Got an ValueError: Circular reference detected. Did not further investigate since accepted solution worked.

Cesar Flores Over a year ago

This doesn't work if you have complex objects. ValueError: Circular reference detected probably will pop up or you would get an error pointing that the object of type xxx is not JSON serializable.

Leon Over a year ago

Wow. This helper function worked very smooth for me when inserting a recursive object into mongodb. Thanks a lot!

|

Anurag Uniyal · Accepted Answer · 2009-06-24 04:55:40Z

10

I don't know what is the purpose of checking for basestring or object is? also dict will not contain any callables unless you have attributes pointing to such callables, but in that case isn't that part of object?

so instead of checking for various types and values, let todict convert the object and if it raises the exception, user the orginal value.

todict will only raise exception if obj doesn't have dict e.g.

class A(object):
    def __init__(self):
        self.a1 = 1

class B(object):
    def __init__(self):
        self.b1 = 1
        self.b2 = 2
        self.o1 = A()

    def func1(self):
        pass

def todict(obj):
    data = {}
    for key, value in obj.__dict__.iteritems():
        try:
            data[key] = todict(value)
        except AttributeError:
            data[key] = value
    return data

b = B()
print todict(b)

it prints {'b1': 1, 'b2': 2, 'o1': {'a1': 1}} there may be some other cases to consider, but it may be a good start

special cases if a object uses slots then you will not be able to get dict e.g.

class A(object):
    __slots__ = ["a1"]
    def __init__(self):
        self.a1 = 1

fix for the slots cases can be to use dir() instead of directly using the dict

edited Jun 24, 2009 at 4:55

answered Jun 24, 2009 at 4:41

Anurag Uniyal

89.2k41 gold badges181 silver badges223 bronze badges

3 Comments

Shabbyrobe Over a year ago

Thanks for the help and inspiration. I just realised that it doesn't handle lists of objects, so I've updated my version to test for iter. Not sure if that's a good idea though.

Anurag Uniyal Over a year ago

looks like it will get trickier because what happens for a object which provides a iter to iterate a list attribute which you have already put in dict, may be general solution is not possible.

Alexey Korolkov Over a year ago

Thanks @AnuragUniyal! You made my day! # noinspection PyProtectedMember def object_to_dict(obj): data = {} if getattr(obj, 'dict', None): for key, value in obj.__dict__.items(): try: data[key] = object_to_dict(value) except AttributeError: data[key] = value return data else: return obj

Tom · Accepted Answer · 2013-06-07 22:12:37Z

5

A slow but easy way to do this is to use jsonpickle to convert the object to a JSON string and then json.loads to convert it back to a python dictionary:

dict = json.loads(jsonpickle.encode( obj, unpicklable=False ))

answered Jun 7, 2013 at 22:12

Tom

19.4k16 gold badges78 silver badges83 bronze badges

1 Comment

Tom Over a year ago

Answer author's note: wouldn't recommend this now.

hbristow · Accepted Answer · 2014-03-27 06:24:19Z

I realize that this answer is a few years too late, but I thought it might be worth sharing since it's a Python 3.3+ compatible modification to the original solution by @Shabbyrobe that has generally worked well for me:

import collections
try:
  # Python 2.7+
  basestring
except NameError:
  # Python 3.3+
  basestring = str 

def todict(obj):
  """ 
  Recursively convert a Python object graph to sequences (lists)
  and mappings (dicts) of primitives (bool, int, float, string, ...)
  """
  if isinstance(obj, basestring):
    return obj 
  elif isinstance(obj, dict):
    return dict((key, todict(val)) for key, val in obj.items())
  elif isinstance(obj, collections.Iterable):
    return [todict(val) for val in obj]
  elif hasattr(obj, '__dict__'):
    return todict(vars(obj))
  elif hasattr(obj, '__slots__'):
    return todict(dict((name, getattr(obj, name)) for name in getattr(obj, '__slots__')))
  return obj

If you're not interested in callable attributes, for example, they can be stripped in the dictionary comprehension:

elif isinstance(obj, dict):
  return dict((key, todict(val)) for key, val in obj.items() if not callable(val))

Lennart Regebro · Accepted Answer · 2009-06-24 05:46:05Z

In Python there are many ways of making objects behave slightly differently, like metaclasses and whatnot, and it can override getattr and thereby have "magical" attributes you can't see through dict, etc. In short, it's unlikely that you are going to get a 100% complete picture in the generic case with whatever method you use.

Therefore, the answer is: If it works for you in the use case you have now, then the code is correct. ;-)

To make somewhat more generic code you could do something like this:

import types
def todict(obj):
    # Functions, methods and None have no further info of interest.
    if obj is None or isinstance(subobj, (types.FunctionType, types.MethodType))
        return obj

    try: # If it's an iterable, return all the contents
        return [todict(x) for x in iter(obj)]
    except TypeError:
        pass

    try: # If it's a dictionary, recurse over it:
        result = {}
        for key in obj:
            result[key] = todict(obj)
        return result
    except TypeError:
        pass

    # It's neither a list nor a dict, so it's a normal object.
    # Get everything from dir and __dict__. That should be most things we can get hold of.
    attrs = set(dir(obj))
    try:
        attrs.update(obj.__dict__.keys())
    except AttributeError:
        pass

    result = {}
    for attr in attrs:
        result[attr] = todict(getattr(obj, attr, None))
    return result

Something like that. That code is untested, though. This still doesn't cover the case when you override getattr, and I'm sure there are many more cases that it doens't cover and may not be coverable. :)

aminkr · Accepted Answer · 2021-03-18 11:29:04Z

2

No custom implementation is required. jsons library can be used.

import jsons

object_dict = jsons.dump(object_instance)

answered Mar 18, 2021 at 11:29

aminkr

943 bronze badges

Comments

Alexey Korolkov · Accepted Answer · 2022-03-06 00:03:08Z

1

Thanks @AnuragUniyal! You made my day! This is my variant of code that's working for me:

# noinspection PyProtectedMember
def object_to_dict(obj):
    data = {}
    if getattr(obj, '__dict__', None):
        for key, value in obj.__dict__.items():
            try:
                data[key] = object_to_dict(value)
            except AttributeError:
                data[key] = value
        return data
    else:
        return obj

answered Mar 6, 2022 at 0:03

Alexey Korolkov

2213 silver badges5 bronze badges

Comments

oroszgy · Accepted Answer · 2017-03-06 14:04:57Z

A little update to Shabbyrobe's answer to make it work for namedtuples:

def obj2dict(obj, classkey=None):
    if isinstance(obj, dict):
        data = {}
        for (k, v) in obj.items():
            data[k] = obj2dict(v, classkey)
        return data
    elif hasattr(obj, "_asdict"):
        return obj2dict(obj._asdict())
    elif hasattr(obj, "_ast"):
        return obj2dict(obj._ast())
    elif hasattr(obj, "__iter__"):
        return [obj2dict(v, classkey) for v in obj]
    elif hasattr(obj, "__dict__"):
        data = dict([(key, obj2dict(value, classkey))
                     for key, value in obj.__dict__.iteritems()
                     if not callable(value) and not key.startswith('_')])
        if classkey is not None and hasattr(obj, "__class__"):
            data[classkey] = obj.__class__.__name__
        return data
    else:
        return obj

Giorgio Di Guardia · Accepted Answer · 2018-09-14 17:57:17Z

0

def list_object_to_dict(lst):
    return_list = []
    for l in lst:
        return_list.append(object_to_dict(l))
    return return_list

def object_to_dict(object):
    dict = vars(object)
    for k,v in dict.items():
        if type(v).__name__ not in ['list', 'dict', 'str', 'int', 'float']:
                dict[k] = object_to_dict(v)
        if type(v) is list:
            dict[k] = list_object_to_dict(v)
    return dict

answered Sep 14, 2018 at 17:57

Giorgio Di Guardia

961 silver badge3 bronze badges

Comments

Will · Accepted Answer · 2020-01-10 09:00:21Z

Looked at all solutions, and @hbristow's answer was closest to what I was looking for. Added enum.Enum handling since this was causing a RecursionError: maximum recursion depth exceeded error and reordered objects with __slots__ to have precedence of objects defining __dict__.

def todict(obj):
  """
  Recursively convert a Python object graph to sequences (lists)
  and mappings (dicts) of primitives (bool, int, float, string, ...)
  """
  if isinstance(obj, str):
    return obj
  elif isinstance(obj, enum.Enum):
    return str(obj)
  elif isinstance(obj, dict):
    return dict((key, todict(val)) for key, val in obj.items())
  elif isinstance(obj, collections.Iterable):
    return [todict(val) for val in obj]
  elif hasattr(obj, '__slots__'):
    return todict(dict((name, getattr(obj, name)) for name in getattr(obj, '__slots__')))
  elif hasattr(obj, '__dict__'):
    return todict(vars(obj))
  return obj

Alex · Accepted Answer · 2021-03-31 05:35:48Z

0

I'd comment on the accepted answer but my rep is not high enough... The accepted answer is great but add another elif just after the if to support NamedTuples serialization to dict properly too:

    elif hasattr(obj, "_asdict"):
        return todict(obj._asdict())

answered Mar 31, 2021 at 5:35

Alex

411 silver badge3 bronze badges

Comments

Pavel Malyarevsky · Accepted Answer · 2021-12-19 09:10:29Z

0

Well. Added functionality of limiting the depth to @Shabbyrobe answer. Thought it might be worth for the objects which loop back.

def todict(obj, limit=sys.getrecursionlimit(), classkey=None):
        if isinstance(obj, dict):
            if limit>=1:
                data = {}
                for (k, v) in obj.items():
                    data[k] = todict(v, limit-1,classkey)
                return data
            else:
                return 'class:'+obj.__class__.__name__
        elif hasattr(obj, "_ast"):
            return todict(obj._ast(), limit-1) if limit>=1 else {'class:'+obj.__class__.__name__}
        elif hasattr(obj, "__iter__") and not isinstance(obj, str):
            return [todict(v, limit-1, classkey) for v in obj] if limit>=1 else {'class:'+obj.__class__.__name__}
        elif hasattr(obj, "__dict__"):
            if limit>=1:
                data = dict([(key, todict(value, limit-1, classkey)) 
                    for key, value in obj.__dict__.items() 
                    if not callable(value) and not key.startswith('_')])
                if classkey is not None and hasattr(obj, "__class__"):
                    data[classkey] = obj.__class__.__name__
                return data
            else:
                return 'class:'+obj.__class__.__name__
        else:
            return obj

edited Dec 19, 2021 at 9:10

answered Dec 19, 2021 at 8:31

Pavel Malyarevsky

11 bronze badge

1 Comment

Pavel Malyarevsky Over a year ago

if object is not doing dict method right way then you need to use dir()

sashabal · Accepted Answer · 2022-12-19 16:28:32Z

previous answers not work when class field is class instance. use this:

from dataclasses import dataclass, field

@dataclass
class BaseNumber:
    number:str = ''
    probability:float = 0.

@dataclass
class ContainerInfo:
    type:str = ''
    height:int = ''
    width:str = ''
    length:str = ''

@dataclass
class AdditionalNumber:
    number:str = ''
    prob:float = 0.
    info:ContainerInfo = ContainerInfo()

@dataclass  
class ContainerData:
    container_number = BaseNumber()
    container_type = AdditionalNumber()
    errors:list = field(default_factory=list)

    def todict(self, obj='sadasdas'):
        if obj == 'sadasdas':
            obj = self
            
        if isinstance(obj, dict):
            data = {}
            for (k, v) in obj.items():
                data[k] = self.todict(v)
            return data
        elif hasattr(obj, "_ast"):
            return self.todict(obj._ast())
        elif hasattr(obj, "__iter__") and not isinstance(obj, str):
            return [self.todict(v) for v in obj]
        elif hasattr(obj, "__dict__"):
            aaa = dir(obj)
            data = dict([(key, self.todict(value)) 
                for key, value in {field: getattr(obj, field) for field in dir(obj)}.items()
                if not callable(value) and not key.startswith('_')
            ])
            return data
        else:
            return obj

Collectives™ on Stack Overflow

Recursively convert python object graph to dictionary

14 Answers 14

5 Comments

7 Comments

3 Comments

1 Comment

Comments

1 Comment

Comments

Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

14 Answers 14

5 Comments

7 Comments

3 Comments

1 Comment

Comments

1 Comment

Comments

Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related