Protobuf to json in python

Question

I have an object that I de-serialize using protobuf in Python. When I print the object it looks like a python object, however when I try to convert it to json I have all sorts of problems.

For example, if I use json.dumps() I get that the object (the generated code from protoc) does not contain a _ dict _ error.

If I use jsonpickle I get UnicodeDecodeError: 'utf8' codec can't decode byte 0x9d in position 97: invalid start byte.

Test code below is using jsonpickle with the error shown above.

if len(sys.argv) < 2:
    print ("Error: missing ser file")
    sys.exit()
else :
    fileLocation = sys.argv[1]

org = BuildOrgObject(fileLocation) 

org = org.Deserialize()


#print (org)
jsonObj = jsonpickle.encode(org)
print (jsonObj)

This would be way easier to figure out if you showed us the relevant parts of your .proto file and the implementation of BuildOrgObject(). If we can reproduce the behavior you're seeing, it's much easier for us to figure out what's wrong. — Sam Mussmann
– Sam Mussmann, Commented Nov 1, 2013 at 20:59

denis-sumin · Accepted Answer · 2021-06-04 21:52:10Z

251

I'd recommend using protobuf↔json converters from google's protobuf library:

from google.protobuf.json_format import MessageToJson

json_obj = MessageToJson(org)

You can also serialise the protobuf to a dictionary:

from google.protobuf.json_format import MessageToDict
dict_obj = MessageToDict(org)

Refer to the protobuf package API documentation: https://developers.google.com/protocol-buffers/docs/reference/python/ (see module google.protobuf.json_format).

edited Jun 4, 2021 at 21:52

user3064538

answered Sep 25, 2016 at 8:50

denis-sumin

2,6311 gold badge14 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

18 Comments

Seanny123 Over a year ago

This doesn't seem to be available in version 3.2?

denis-sumin Over a year ago

I've done a quick test (clean venv, install protobuf, try to import MessageToJson) and it seems to be available. Python 3.6.

redacted Over a year ago

MessageToJson(org, preserving_proto_field_name=True) if you don't want your field_name converted into fieldName

Zioalex Over a year ago

It doesn't work for me. I get the error: AttributeError: 'Schema' object has no attribute 'DESCRIPTOR'

Assaf Mendelson Over a year ago

If you are running this on a repeated sub field (instead of on a regular message) then it will fail with missing DESCRIPTOR. Use the main message, or convert to dict each of the elements and combine

|

Seanny123 · Accepted Answer · 2017-02-09 08:39:07Z

17

If you need to go straight to json take a look at the protobuf-to-json library, but you'll have to install that manually.

But I would recommend that you use the protobuf-to-dict library instead for a few reasons:

It is accessible from pypi so you can simply pip install protobuf-to-dict or include it in a requirements.txt
dict can be converted to json and might be more useful than a json string

edited Feb 9, 2017 at 8:39

Seanny123

9,44617 gold badges74 silver badges136 bronze badges

answered Apr 13, 2016 at 3:28

Kevin Hill

5031 gold badge5 silver badges17 bronze badges

6 Comments

nmurthy Over a year ago

protobuf-to-dict feels more pythonic than google.protobuf.json_format

kgreenek Over a year ago

There is now a method include with protobuf for converting to a dict instead of json. It is called: google.protobuf.json_format.MessageToDict

Paul Kenjora Over a year ago

Requires implementing MyMessage to get things going. Kind of incomplete since thats a big part of parsing a proto?

Kevin Hill Over a year ago

That's a limitation of protobuf itself. Unlike something like json, a protobuf message cannot describe itself, and a schema must be created in order to either encode or decode a message. Without a schema, it is just a byte blob without enough information to parse.

lawson Over a year ago

@Stian - try the updated version: pip install protobuf3-to-dict

|

Nirav · Accepted Answer · 2022-02-13 17:53:25Z

3

Here's my function to convert a proto3 object to a JSON object (i.e. Python dictionary):

def protobuf_to_dict(proto_obj):
    key_list = proto_obj.DESCRIPTOR.fields_by_name.keys()
    d = {}
    for key in key_list:
        d[key] = getattr(proto_obj, key)
    return d

Since the converters from Google's protobuf library don't seem to work in some cases with the 3.19 version, this function leverages the Descriptor class present on each Protobuf object.

Here, getattr(obj, string_attribute) returns the value given by obj.attribute

edited Feb 13, 2022 at 17:53

answered Feb 6, 2022 at 9:23

Nirav

815 bronze badges

Comments

Simas Joneliunas · Accepted Answer · 2021-03-24 11:58:01Z

2

You can also user SerializeToString.

org.SerializeToString()

edited Mar 24, 2021 at 11:58

Simas Joneliunas

3,15620 gold badges32 silver badges39 bronze badges

answered Mar 24, 2021 at 11:33

Shubham Kanyal

371 bronze badge

1 Comment

whatyouhide Over a year ago

This does not return the JSON representation of the Protobuf message. It returns the Protobuf-encoded serialized messages as bytes (see the documentation).

jm-nab · Accepted Answer · 2024-07-19 21:22:13Z

1

In my case I was trying to convert google vision api response protobuf to dict.

You can try:

from google.protobuf.json_format import MessageToDict


(Pdb) type(response)
<class 'google.cloud.vision_v1.types.image_annotator.BatchAnnotateFilesResponse'>

print(MessageToDict(response._pb))
{'responses': [{'responses': [...]},...]}

If you try MessageToDict with just the response you'll get a missing DESCRIPTOR error

answered Jul 19, 2024 at 21:22

jm-nab

1246 bronze badges

Comments

vr286 · Accepted Answer · 2022-01-31 11:03:41Z

0

If you are using an older version that doesn't has the preserving_proto_field_name field:

from google.protobuf.json_format import MessageToJson
def proto_to_json(proto_obj):
    json_obj = MessageToJson(proto_obj):
    json_obj = MessageToJso, including_default_value_fields=True)
    # Change lowerCamelCase of google Json conversion to the snake_case as in original protobuf
    dict_obj = dict((re.sub(r'(?<!^)(?=[A-Z])', '_', k).lower(),v) for k, v in json.loads(json_obj).items())
    if hasattr(proto_obj, 'uuid'):
        dict_obj["uuid"] = proto_obj.uuid.encode("hex")
    return json.dumps(dict_obj, indent=4, sort_keys=True)

edited Jan 31, 2022 at 11:03

answered Jan 31, 2022 at 10:58

vr286

8467 silver badges6 bronze badges

Comments

Nikolay Spassov · Accepted Answer · 2025-11-10 20:50:39Z

0

#!/usr/bin/python

from google.transit import gtfs_realtime_pb2
from google.protobuf.json_format import MessageToDict, MessageToJson
import requests

feed = gtfs_realtime_pb2.FeedMessage()
response = requests.get('GTFS resource URL')
feed.ParseFromString(response.content)

print(MessageToJson(feed))

This returns proper JSON representation of a remote GTFS resource.

answered Nov 10 at 20:50

Nikolay Spassov

1,2961 gold badge12 silver badges22 bronze badges

Collectives™ on Stack Overflow

Protobuf to json in python

7 Answers 7

18 Comments

6 Comments

Comments

1 Comment

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

18 Comments

6 Comments

Comments

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related