3

I'm working on a proprietary TCP protocol. This protocol sends and receive messages with a specific sequence of bytes.

I should be complaiant to this protocol, and i cant change it. So my input / output results are something like that :
\x01\x08\x00\x01\x00\x00\x01\xFF

\x01 - Message type
\x01 - Message type
\x00\x01 - Length
\x00\x00\x01 - Transaction
\xFF - Body

The sequence of field is important. And i want only the values of the fields in my serialization, and nothing about the structure of the class.

I'm working on a Java controller that use this protocol and I've thought to define the message structures in specific classes and serialize/deserialize them, but I was naive.

First of all I tried ObjectOutputStream, but it output the entire structure of the object, when I need only the values in a specific order.

Someone already faced this problem: Java - Object to Fixed Byte Array
and solved it with a dedicated Marshaller.

But I was searching for a more flexible solution.

For text serialization and deserialization I've found:
http://jeyben.github.io/fixedformat4j/
that with annotation defines the schema of the line. But it outputs a String, not a byte[]. So 1 is output like "1" that is represented differently based on encoding, and often with more bytes.

What I was searching for is something that given the order of my class properties will convert each property in a bunch of bytes (based on the internal representation) and append them to a byte[].

Do you know some library used for that purpose?
Or a simple way to do that, without coding a serialization algorithm for each of my entities?

Improve this question

1 Answer 1

Reset to default
3

Serialization just isn't easy; it sounds from your question like you feel you can just invoke something and out rolls compact, simple, versionable, universal data you can then put on the wire. What you need to fix is to scratch the word 'just' from that sentence. You're going to have to invest some time and care.

As you figured out already, java's baked in serialization has a ton of downsides. Don't use that.

There are various serializers. The popular ones are things like GSON or Jackson, which lets you serialize java objects into JSON. This isn't particularly efficient, and is string based. This sounds like crucial downsides but they really aren't, see below.

You can also spend a little more time specifying the exact format and use protobuf which lets you write a quite lean and simple data protocol (and protobuf is available for many languages, if eventually you want to write an participant in this protocol in non-java later).

So, those are the good options: Go to JSON via Jackson or GSON, or, use protobuf.

But JSON is a string.

You can turn a string to bytes trivially using str.getBytes(StandardCharsets.UTF_8). This cannot fail due to charset encoding differences (as long as you also 'decode' in the same fashion: Turn the bytes into a string with new String(theBytes, StandardCharsets.UTF_8). UTF-8 is guaranteed to be available on all JVMs; if it is not there, your JVM is as broken as a JVM that is missing the String class - not something to worry about.

But JSON is inefficient.

Zip it up, of course. You can trivially wrap an InputStream and an OutputStream so that gzip compression is applied which is simple, available on just about every platform, and fast (it's not the most efficient cutting edge compression algorithm, but usually squeezing the last few bytes out is not worth it) - and zipped-up JSON can often be more efficient that carefully handrolled protobuf, even.

The one downside is that it's 'slow', but on modern hardware, note that the overhead of encrypting and decrypting this data (which you should obviously be doing!!) is usually multiple orders of magnitude more involved. A modern CPU is simply very, very fast - creating JSON and zipping it up is going to take 1% of CPU or less even if you are shipping the collected works of shakespeare every second.

If an arduino running on batteries needs to process this data, go with uncompressed, unencrypted protobuf-based data. If you are facebook and writing the whatsapp protocol, the IAAS creds saved by not having to unzip and decode JSON is tiny and pales in comparison to the creds you spend just running the servers, but at that scale its worth the development effort.

In just about every other case, just toss gzipped JSON on the line.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thank you for your response. I've edited my post, with an example, beacuse maybe it was not so clear. The sequence of field is important. And i want only the values of the fields in my serialization, and nothing about the structure of the class. So i think i can exclude Json, even if is decoded in bytes.
Yes, now your question is something completely different. What you are talking about is writing a specific protocol. There is no serialization library that can do this for you, you have to handroll this code or find an existing library for this specific protocol.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.