34

is there any elegant way to make Python JSON encoder support datetime? some 3rd party module or easy hack?

I am using tornado's database wrapper to fetch some rows from db to generate a json. The query result includes a regular MySQL timestamp column.

It's quite annoying that Python's default json encoder doesn't support its own datetime type, which is so common in all kinds of database queries.

I don't want to modify Python's own json encoder. any good practice? Thanks a lot!

ps: I found a dirty hack by modifying the Python JSON encoder default method:

Change:

def default(self, o):
    raise TypeError(repr(o) + " is not JSON serializable")

To:

def default(self, o):
    from datetime import date
    from datetime import datetime
    if isinstance(o, datetime):
        return o.isoformat()
    elif isinstance(o, date):
        return o.isoformat()
    else:
        raise TypeError(repr(o) + " is not JSON serializable")

well, it will be a temporary solution just for dev environment.

But for long term solution or production environment, this is quite ugly, and I have to do the modification every time I deploy to a new server.

Is there a better way? I do not want to modify Python code itself, neither Tornado source code. Is there something I can do with my own project code to make this happen? preferably in one pace.

Thanks a lot!

Improve this question
2
  • 1
    See: stackoverflow.com/questions/455580/… Commented Aug 25, 2012 at 12:49
  • the problem with the subclass method, is that it fails for all the other uses of json encoding, such as simple django, "dumpdata" Commented Nov 21, 2012 at 18:33

9 Answers 9

Reset to default
73

json.dumps(thing, default=str)

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

please upvote if you think this meets the definition of "easy hack" mentioned in the question.
I guess this should be the correct answer, Wonder what is wrong with this answer?
I find it amusing. 5 years later and this is still the most simple and Pythonic way to overcome the issue, and this answer is still underrated.
this actually preserves the float and int values when thing is a Dict.
you would rather agree on some common formatting and use strftime inside some function you're passing into default
37

The docs suggest subclassing JSONEncoder and implementing your own default method. Seems like you're basically there.

The reason dates aren't handled by the default encoder is there is no standard representation of a date in JSON. Some people are using the format /Date(1198908717056)/, but I prefer ISO format personally.

import json
import datetime


class DateTimeEncoder(json.JSONEncoder):

    def default(self, obj):
        if isinstance(obj, (datetime.datetime, datetime.date, datetime.time)):
            return obj.isoformat()
        elif isinstance(obj, datetime.timedelta):
            return (datetime.datetime.min + obj).time().isoformat()

        return super(DateTimeEncoder, self).default(obj)

now = datetime.datetime.now()
encoder = DateTimeEncoder()
encoder.encode({"datetime": now, "date": now.date(), "time": now.time()})
> {"datetime": "2019-07-02T16:17:09.990126", "date": "2019-07-02", "time": "16:17:09.990126"}
Improve this answer

4 Comments

You can combine the if and first elif by taking advantage of the fact that you do the same thing for both return obj.isoformat() and that isinstance supports tuples: isinstance(obj, (datetime.datetime, datetime.date))
this worked for me, I added the additional checks into my existing custom JSON encoder
This timedelta encoding fails for values over (or equal to) 24 hours. E.g. for a timedelta(hours=24), the encoded value will be 00:00:00 (0 hours), or for timedelta(hours=25), it will be 01:00:00 (1 hour).
@cod3monk3y if you have timedeltas over 24h, use the total_seconds method, or do manual string formatting. isoformat is an easy trick for stort ones.
24

I made my own classes for my project:

import datetime
import decimal
import json
import sys

class EnhancedJSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime.datetime):
            ARGS = ('year', 'month', 'day', 'hour', 'minute',
                     'second', 'microsecond')
            return {'__type__': 'datetime.datetime',
                    'args': [getattr(obj, a) for a in ARGS]}
        elif isinstance(obj, datetime.date):
            ARGS = ('year', 'month', 'day')
            return {'__type__': 'datetime.date',
                    'args': [getattr(obj, a) for a in ARGS]}
        elif isinstance(obj, datetime.time):
            ARGS = ('hour', 'minute', 'second', 'microsecond')
            return {'__type__': 'datetime.time',
                    'args': [getattr(obj, a) for a in ARGS]}
        elif isinstance(obj, datetime.timedelta):
            ARGS = ('days', 'seconds', 'microseconds')
            return {'__type__': 'datetime.timedelta',
                    'args': [getattr(obj, a) for a in ARGS]}
        elif isinstance(obj, decimal.Decimal):
            return {'__type__': 'decimal.Decimal',
                    'args': [str(obj),]}
        else:
            return super().default(obj)


class EnhancedJSONDecoder(json.JSONDecoder):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, object_hook=self.object_hook,
                         **kwargs)

    def object_hook(self, d): 
        if '__type__' not in d:
            return d
        o = sys.modules[__name__]
        for e in d['__type__'].split('.'):
            o = getattr(o, e)
        args, kwargs = d.get('args', ()), d.get('kwargs', {})
        return o(*args, **kwargs)

if __name__ == '__main__':
    j1 = json.dumps({'now': datetime.datetime.now(),
        'val': decimal.Decimal('9.3456789098765434987654567')},
        cls=EnhancedJSONEncoder)
    print(j1)
    o1 = json.loads(j1, cls=EnhancedJSONDecoder)
    print(o1)

Result:

{"val": {"args": ["9.3456789098765434987654567"], "__type__": "decimal.Decimal"}, "now": {"args": [2014, 4, 29, 11, 44, 57, 971600], "__type__": "datetime.datetime"}}
{'val': Decimal('9.3456789098765434987654567'), 'now': datetime.datetime(2014, 4, 29, 11, 44, 57, 971600)}

References:

Note: It can be made more flexible by passing a custom dictionary with types as keys and args, kwargs as values to the encoder's __init__() and use that (or a default dictionary) in the default() method.

Improve this answer

2 Comments

Best answer so far. It helps when I was trying to wrap my head around how to deserialize UUIDs. Great!
I am glad you are happy with it. I used it couple of times myself.
10 
json.dumps(r, default=lambda o: o.isoformat() if hasattr(o, 'isoformat') else o)
Improve this answer

1 Comment

While this code snippet may solve the question, including an explanation really helps to improve the quality of your post. Remember that you are answering the question for readers in the future, and those people might not know the reasons for your code suggestion.
2

Create a custom decoder/encoder:

class CustomJSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime.datetime):
            return http_date(obj)
        if isinstance(obj, uuid.UUID):
            return str(obj)
        return json.JSONEncoder.default(self, obj)

class CustomJSONDecoder(json.JSONDecoder):
    def __init__(self, *args, **kwargs):
        json.JSONDecoder.__init__(self, object_hook=self.object_hook, *args, **kwargs)

    def object_hook(self, source):
        for k, v in source.items():
            if isinstance(v, str):
                try:
                    source[k] = datetime.datetime.strptime(str(v), '%a, %d %b %Y %H:%M:%S %Z')
                except:
                    pass
        return source
Improve this answer

Comments

1

The Tryton project has a JSONEncoder implementation for datetime.datetime, datetime.date and datetime.time objects (with others). It is used for JSON RPC communication between the server and client.

See http://hg.tryton.org/2.4/trytond/file/ade5432ac476/trytond/protocols/jsonrpc.py#l53

Improve this answer

Comments

0

Convert the datetime type into a unix timestamp, then encode the contents into a json.

e.g. : http://codepad.org/k3qF09Kr

Improve this answer

2 Comments

you mean changing Python's default datetime type? how to do so? will it be too risky? will it break things?
@horacex no,just modify the resultset that came from tornado database wrapper.
-1

I recommend to use the ujson package or the orjson one.

They are much faster and still support several complex types.

Improve this answer

2 Comments

ujson didn't work for me (at least not out of the box). I got: TypeError: datetime.datetime(2020, 4, 18, 17, 50, 42, 418419, tzinfo=datetime.timezone(datetime.timedelta(seconds=10800))) is not JSON serializable
ujson doesn't natively serialize datetime.datetime. However, orjson did the trick. orjson also serializes other common types, such as uuid.UUID.
-2

Just create a custom encoder

(the small but important addition to Cole's answer is the handling of pd.NaT (or null/empty timestamp values), since without the addition you will get very weird timestamp conversions for NaT/missing timestamp data)

class CustomEncoder(json.JSONEncoder):
    def default(self, obj):
        if pd.isnull(obj):
            return None
        elif isinstance(obj, datetime):
            return obj.isoformat()
        elif isinstance(obj, date):
            return obj.isoformat()
        elif isinstance(obj, timedelta):
            return (datetime.min + obj).time().isoformat()
        else:
            return super(CustomEncoder, self).default(obj)

Then use it to encode a dataframe:

df_as_dict = df.to_dict(outtype = 'records')  # transform to dict

df_as_json = CustomEncoder().encode(df_as_dict) #transform to json

Since the encoder standardized the data, the regular decoder will act fine in transforming it back to a dataframe:

result_as_dict = json.JSONDecoder().decode(df_as_json) # decode back to dict

result_df = pd.DataFrame(result)  # transform dict back to dataframe

Of course this will also work if you put the dataframe into a larger dict before encoding, e.g

input_dict = {'key_1':val_1,'key_2':val_2,...,'df_as_dict':df_as_dict}
input_json = CustomEncoder().encode(input_dict)
input_json_back_as_dict = json.JSONDecoder().decode(input_json)
input_df_back_as_dict = input_json_back_as_dict['df_as_dict']
input_df_back_as_df = pd.DataFrame(input_df_back_as_dict)
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.