How to know which protobuf message the byte array is?

Question

I want to use protobuf instead of Json to communicate between message queue.

I know how to deal with it when there is only one proto message.

Assume that the proto file is:

//person.proto
syntax = "proto3";

option java_outer_classname = "PersonProto";

message Person {
    int32 id = 2;
    string name = 1;
    string email = 3;
}

Now, i can deal with it with the approach below:

PersonProto.Person person = PersonProto.Person.newBuilder()
        .setEmail("[email protected]")
        .setId(1)
        .setName("name-test")
        .build();

byte[] bytes = person.toByteArray();

//Transfer from publisher to consumer between message queue.

//I can deserialise it, because i know the proto message is Person.
PersonProto.Person.parseFrom(bytes);

But what if there are multiple proto messages?

Assume that there is another proto message called Address.

syntax = "proto3";

option java_outer_classname = "PersonProto";

message Person {
    int32 id = 2;
    string name = 1;
    string email = 3;
}

message Address {
    string address = 1;
}

When consumer received byte array from message queue, how to know which proto message it is? and how to deserialise the byte array?

Nick Holt · Accepted Answer · 2018-05-11 11:45:50Z

26

Protobuf 3 introduced the concept of Any that can act in a similar way to the top-level-message pattern explained by @AdamCozzette.

On the write-side you pack your message in an Any:

Person person = ...
Any any = Any.pack(person);

out.write(any.toByteArray());

Then on the read-side you read into an Any and switch on the types you are interested in:

Any any = Any.parseFrom(in);

if (any.is(Person.class)
{
  Person person = any.unpack(Person.class);
  ...
}
else if (any.is(Address.class);
{
  Address address = any.unpack(Address.class);
  ...
}
else
{
  //Handle unknown message
}

Using Any removes the need for special a message type (the top-level-message), but also removes an element of type-safety in as much as you may receive messages that the consuming code does not know how to handle.

answered May 11, 2018 at 11:45

Nick Holt

34.5k4 gold badges55 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Grisgram Over a year ago

Any does not have a .parseFrom Method

Nick Holt Over a year ago

Any has a parseFrom method in my IDE and in the JavaDocs - developers.google.com/protocol-buffers/docs/reference/java/com/…

Grisgram Over a year ago

It seems to have changed in the latest versions. It's now any.Parser.ParseFrom

Nick Holt · Accepted Answer · 2018-05-11 11:28:02Z

Protocol buffers are not self-describing, so in general when you get a serialized protobuf there is no way to interpret its contents without knowing what schema to expect.

In your case I would recommend using a oneof field. You can have a single top-level message type for your queue messages and let that contain a oneof field that contains either a Person or an Address:

message TopLevelMessage {
  oneof inner_message {
    Person person = 1;
    Address address = 2;
  }
}

The consuming code would then need a switch statement like this in order to retrieve the inner-message:

TopLevelMessage topLevelMessage = TopLevelMessage.parseFrom(...);

switch (topLevelMessage.getInnerMessageCase()) 
{
  case PERSON:
    Person person = topLevelMessage.getPerson();
    ...
    break;

  case ADDRESS:
    Address address = topLevelMessage.getAddress();
    ...
    break;

  default:
     ... 
}

Collectives™ on Stack Overflow

How to know which protobuf message the byte array is?

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related