Converting / Parsing C++ byte struct to Go

Question

I am reading some data packets in Go, where the fields are C++ data types. I tried parsing the data but I am reading garbage values.

Here is a small example - the data spec sheet for a particular datatype is as follows in C++,

struct CarTelemetryData
{
    uint16    m_speed;                      
    uint8     m_throttle;                   
    int8      m_steer;                      
    uint8     m_brake;                     
    uint8     m_clutch;                     
    int8      m_gear;                       
    uint16    m_engineRPM;                  
    uint8     m_drs;                        
    uint8     m_revLightsPercent;           
    uint16    m_brakesTemperature[4];       
    uint16    m_tyresSurfaceTemperature[4]; 
    uint16    m_tyresInnerTemperature[4];   
    uint16    m_engineTemperature;          
    float     m_tyresPressure[4];           
};

And below is what I have defined in Go

type CarTelemetryData struct {
    Speed                   uint16
    Throttle                uint8
    Steer                   int8
    Brake                   uint8
    Clutch                  uint8
    Gear                    int8
    EngineRPM               uint16
    DRS                     uint8
    RevLightsPercent        uint8
    BrakesTemperature       [4]uint16
    TyresSurfaceTemperature [4]uint16
    TyresInnerTemperature   [4]uint16
    EngineTemperature       uint16
    TyresPressure           [4]float32
}

For the actual un-marshalling, I am doing this -

func decodePayload(dataStruct interface{}, payload []byte) {
    dataReader := bytes.NewReader(payload[:])
    binary.Read(dataReader, binary.LittleEndian, dataStruct)
}

payload := make([]byte, 2048)
s.conn.ReadFromUDP(payload[:])
telemetryData := &data.CarTelemetryData{}
s.PacketsRcvd += 1
decodePayload(telemetryData, payload)

I suspect that this is because the datatypes are not equivalent and there is some conversion issue while reading the bytes into Go data-types, whereas they have been originally packages as C++. How can I deal with this?

Note: I don't have any control over the data that is sent, this is sent by a third party service.

Beware of stuff like alignment, which may cause differences. You basically have two choices: either you serialize your struct (slower, but more compatible) or you use a compiler-specific feature such as #pragma pack to tightly pack all struct members (faster, but highly error-prone). That, however, only handles the C++ side of things. PS: if this is the issue, and you're running on x86/x64, only members after m_engineRPM would be affected AFAIK. — LHLaurini
– LHLaurini, Commented Feb 19, 2022 at 23:04
Also, unless you're transmitting tremendous amounts of data, serialization is probably the way to go. — LHLaurini
– LHLaurini, Commented Feb 19, 2022 at 23:10
Thanks for your input. I'll look into this, however I don't have any control over the data that is transmitted to me. I simply have this Go server running which reads the data from the third party application, which is broadcasting this data to a port I'm listening on. — r4bb1t
– r4bb1t, Commented Feb 19, 2022 at 23:12
Then you'll probably have to add padding bytes in the Go struct to match (where exactly may be architecture-dependent). — LHLaurini
– LHLaurini, Commented Feb 19, 2022 at 23:14
Either by knowing the alignment rules for the architecture, or by using offsetof, like this (assuming your target is an x86 machine). By taking a quick look at binary.Read, it seems to expect packed data. So you need one byte after Gear and two bytes after EngineTemperature. Let me know if that works. — LHLaurini
– LHLaurini, Commented Feb 20, 2022 at 11:27

LHLaurini · Accepted Answer · 2022-02-20 12:23:24Z

The issue you're facing has to do with the alignment of struct members. You can read more about it here but, in short, the C++ compiler will sometimes add padding bytes in order to maintain the natural alignment expected by the architecture. If that alignment is not used, it may cause degraded performance or even an access violation.

For x86/x64, for example, the alignment of most types will usually (but not necessarily guaranteed to) be the same as the size. We can see that

#include <cstdint>
#include <type_traits>

std::size_t offsets[] = {
    std::alignment_of_v<std::uint8_t>,
    std::alignment_of_v<std::uint16_t>,
    std::alignment_of_v<std::uint32_t>,
    std::alignment_of_v<std::uint64_t>,
    std::alignment_of_v<__uint128_t>,
    std::alignment_of_v<std::int8_t>,
    std::alignment_of_v<std::int16_t>,
    std::alignment_of_v<std::int32_t>,
    std::alignment_of_v<std::int64_t>,
    std::alignment_of_v<__int128_t>,
    std::alignment_of_v<float>,
    std::alignment_of_v<double>,
    std::alignment_of_v<long double>,
    std::alignment_of_v<void*>,
};

compiles to

offsets:
        .quad   1
        .quad   2
        .quad   4
        .quad   8
        .quad   16
        .quad   1
        .quad   2
        .quad   4
        .quad   8
        .quad   16
        .quad   4
        .quad   8
        .quad   16
        .quad   8

Due to these (and other) implementation details, it may be advisable to not rely on the internal representation. In some cases, however, other methods may not be fast enough (such as serializing field by field), or you may not be able to change the C++ code, like OP.

binary.Read expects packed data, but C++ will use padding. We need to either use a compiler-dependent directive such as #pragma pack(1) or add padding the Go struct. The first is not an option for OP, so we'll use the second.

We can use the offsetof macro to determine the offset of a struct member relative to the struct itself. We can do something like

#include <array>
#include <cstddef>
#include <cstdint>

using int8 = std::int8_t;
using uint8 = std::uint8_t;
using uint16 = std::uint16_t;

struct CarTelemetryData {
    uint16 m_speed;
    uint8 m_throttle;
    int8 m_steer;
    uint8 m_brake;
    uint8 m_clutch;
    int8 m_gear;
    uint16 m_engineRPM;
    uint8 m_drs;
    uint8 m_revLightsPercent;
    uint16 m_brakesTemperature[4];
    uint16 m_tyresSurfaceTemperature[4];
    uint16 m_tyresInnerTemperature[4];
    uint16 m_engineTemperature;
    float m_tyresPressure[4];
};

// C++ has no reflection (yet) so we need to list every member
constexpr auto offsets = std::array{
    offsetof(CarTelemetryData, m_speed),
    offsetof(CarTelemetryData, m_throttle),
    offsetof(CarTelemetryData, m_steer),
    offsetof(CarTelemetryData, m_brake),
    offsetof(CarTelemetryData, m_clutch),
    offsetof(CarTelemetryData, m_gear),
    offsetof(CarTelemetryData, m_engineRPM),
    offsetof(CarTelemetryData, m_drs),
    offsetof(CarTelemetryData, m_revLightsPercent),
    offsetof(CarTelemetryData, m_brakesTemperature),
    offsetof(CarTelemetryData, m_tyresSurfaceTemperature),
    offsetof(CarTelemetryData, m_tyresInnerTemperature),
    offsetof(CarTelemetryData, m_engineTemperature),
    offsetof(CarTelemetryData, m_tyresPressure),
};

constexpr auto sizes = std::array{
    sizeof(CarTelemetryData::m_speed),
    sizeof(CarTelemetryData::m_throttle),
    sizeof(CarTelemetryData::m_steer),
    sizeof(CarTelemetryData::m_brake),
    sizeof(CarTelemetryData::m_clutch),
    sizeof(CarTelemetryData::m_gear),
    sizeof(CarTelemetryData::m_engineRPM),
    sizeof(CarTelemetryData::m_drs),
    sizeof(CarTelemetryData::m_revLightsPercent),
    sizeof(CarTelemetryData::m_brakesTemperature),
    sizeof(CarTelemetryData::m_tyresSurfaceTemperature),
    sizeof(CarTelemetryData::m_tyresInnerTemperature),
    sizeof(CarTelemetryData::m_engineTemperature),
    sizeof(CarTelemetryData::m_tyresPressure),
};

constexpr auto computePadding() {
    std::array<std::size_t, offsets.size()> result;

    std::size_t expectedOffset = 0;

    for (std::size_t i = 0; i < offsets.size(); i++) {
        result.at(i) = offsets.at(i) - expectedOffset;
        expectedOffset = offsets.at(i) + sizes.at(i);
    }

    return result;
}

auto padding = computePadding();

which compiles to (constexpr FTW)

padding:
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   1
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   0
        .quad   2

So, on x86, we need one byte before EngineRPM and two bytes before TyresPressure.

So, let's check if that works.

C++:

#include <cstddef>
#include <cstdint>
#include <iomanip>
#include <iostream>
#include <span>

using int8 = std::int8_t;
using uint8 = std::uint8_t;
using uint16 = std::uint16_t;

struct CarTelemetryData {
    uint16 m_speed;
    uint8 m_throttle;
    int8 m_steer;
    uint8 m_brake;
    uint8 m_clutch;
    int8 m_gear;
    uint16 m_engineRPM;
    uint8 m_drs;
    uint8 m_revLightsPercent;
    uint16 m_brakesTemperature[4];
    uint16 m_tyresSurfaceTemperature[4];
    uint16 m_tyresInnerTemperature[4];
    uint16 m_engineTemperature;
    float m_tyresPressure[4];
};

int main() {
    CarTelemetryData data = {
        .m_speed = 1,
        .m_throttle = 2,
        .m_steer = 3,
        .m_brake = 4,
        .m_clutch = 5,
        .m_gear = 6,
        .m_engineRPM = 7,
        .m_drs = 8,
        .m_revLightsPercent = 9,
        .m_brakesTemperature = {10, 11, 12, 13},
        .m_tyresSurfaceTemperature = {14, 15, 16, 17},
        .m_tyresInnerTemperature = {18, 19, 20, 21},
        .m_engineTemperature = 22,
        .m_tyresPressure = {23, 24, 25, 26},
    };

    std::cout << "b := []byte{" << std::hex << std::setfill('0');

    for (auto byte : std::as_bytes(std::span(&data, 1))) {
        std::cout << "0x" << std::setw(2) << static_cast<unsigned>(byte)
                  << ", ";
    }

    std::cout << "}";
}

results in

b := []byte{0x01, 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x00, 0x07, 0x00, 0x08, 0x09, 0x0a, 0x00, 0x0b, 0x00, 0x0c, 0x00, 0x0d, 0x00, 0x0e, 0x00, 0x0f, 0x00, 0x10, 0x00, 0x11, 0x00, 0x12, 0x00, 0x13, 0x00, 0x14, 0x00, 0x15, 0x00, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb8, 0x41, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xc8, 0x41, 0x00, 0x00, 0xd0, 0x41, }

Let's use that in Go:

// Type your code here, or load an example.
// Your function name should start with a capital letter.
package main

import (
    "bytes"
    "encoding/binary"
    "fmt"
)

type CarTelemetryData struct {
    Speed                   uint16
    Throttle                uint8
    Steer                   int8
    Brake                   uint8
    Clutch                  uint8
    Gear                    int8
    _                       uint8
    EngineRPM               uint16
    DRS                     uint8
    RevLightsPercent        uint8
    BrakesTemperature       [4]uint16
    TyresSurfaceTemperature [4]uint16
    TyresInnerTemperature   [4]uint16
    EngineTemperature       uint16
    _                       uint16
    TyresPressure           [4]float32
}

func main() {
    b := []byte{0x01, 0x00, 0x02, 0x03, 0x04, 0x05, 0x06, 0x00, 0x07, 0x00, 0x08, 0x09, 0x0a, 0x00, 0x0b, 0x00, 0x0c, 0x00, 0x0d, 0x00, 0x0e, 0x00, 0x0f, 0x00, 0x10, 0x00, 0x11, 0x00, 0x12, 0x00, 0x13, 0x00, 0x14, 0x00, 0x15, 0x00, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb8, 0x41, 0x00, 0x00, 0xc0, 0x41, 0x00, 0x00, 0xc8, 0x41, 0x00, 0x00, 0xd0, 0x41}

    var dataStruct CarTelemetryData

    dataReader := bytes.NewReader(b[:])
    binary.Read(dataReader, binary.LittleEndian, &dataStruct)

    fmt.Printf("%+v", dataStruct)
}

which prints

{Speed:1 Throttle:2 Steer:3 Brake:4 Clutch:5 Gear:6 _:0 EngineRPM:7 DRS:8 RevLightsPercent:9 BrakesTemperature:[10 11 12 13] TyresSurfaceTemperature:[14 15 16 17] TyresInnerTemperature:[18 19 20 21] EngineTemperature:22 _:0 TyresPressure:[23 24 25 26]}

Take the padding bytes out and it fails.

Thanks, I found that my issue was something else - and not related to this. It turns out the data I was reading was packed, but I had some hard to find bugs in my byte unmarshalling logic. However, this answer is great and really helpful - so I am marking this as accepted.

Collectives™ on Stack Overflow

Converting / Parsing C++ byte struct to Go

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related