95

I have an instance of a struct that I defined and I would like to convert it to an array of bytes. I tried []byte(my_struct), but that did not work. Also, I was pointed to the binary package, but I am not sure which function I should use and how I should use it. An example would be greatly appreciated.

11 Answers 11

51

One possible solution is the "encoding/gob" standard package. The gob package creates an encoder/decoder that can encode any struct into an array of bytes and then decode that array back into a struct. There's a great post, here.

As others have pointed out, it's necessary to use a package like this because structs, by their nature, have unknown sizes and cannot be converted into arrays of bytes.

I've included some code and a play.

package main

import (
    "bytes"
    "encoding/gob"
    "fmt"
    "log"
)

type P struct {
    X, Y, Z int
    Name    string
}

type Q struct {
    X, Y *int32
    Name string
}

func main() {
    // Initialize the encoder and decoder.  Normally enc and dec would be
    // bound to network connections and the encoder and decoder would
    // run in different processes.
    var network bytes.Buffer        // Stand-in for a network connection
    enc := gob.NewEncoder(&network) // Will write to network.
    dec := gob.NewDecoder(&network) // Will read from network.
    // Encode (send) the value.
    err := enc.Encode(P{3, 4, 5, "Pythagoras"})
    if err != nil {
        log.Fatal("encode error:", err)
    }

    // HERE ARE YOUR BYTES!!!!
    fmt.Println(network.Bytes())

    // Decode (receive) the value.
    var q Q
    err = dec.Decode(&q)
    if err != nil {
        log.Fatal("decode error:", err)
    }
    fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y)
}
Sign up to request clarification or add additional context in comments.

3 Comments

This might be a dumb question but does gob also serialize unexported fields?
@slim not at all! I'm actually not 100% sure but I would expect that it wouldn't. Same as encoding/json not serializing unexported fields.
WARNING: gob is very...particular about how you use it. It is NOTHING like JSON or any other format in that it prefixes your stream with type data, and then sends each instance of the type (rather than send the type data as well every time). This means you need a 1:1 correspondence between the same encoder decoder pair. So if you have multiple goroutines decoding from a stream, make sure they are all using the same decoder (same for encoding). They are thread safe for this reason.
42

I assume you want something like the way C handles this. There is no built in way to do that. You will have to define your own serialization and deserialization to and from bytes for your struct. The binary package will help you encode the fields in your struct to bytes that you can add to the byte array but you will be responsible for specifying the lengths and offsets in the byte array that will hold the fields from your struct.

Your other options are to use one of the encoding packages: http://golang.org/pkg/encoding/ such as gob or json.

EDIT:

Since you want this for making a hash as you say in your comment the easisest thing to do is use []byte(fmt.Sprintf("%v", struct)) like so: http://play.golang.org/p/yY8mSdZ_kf

2 Comments

Even better: You can just use fmt.Sprint(struct). Sprint formats its arguments according to the default format.
Once you []byte(fmt.Sprintf("%v", struct)), is there any way to convert the byte slice back to the original struct?
37

Just use json marshal, this is a very simple way.

newFsConfig := dao.ConfigEntity{EnterpriseId:"testing"}
newFsConfigBytes, _ := json.Marshal(newFsConfig)

3 Comments

Exactly, I only scrolled down looking if someone had given this answer. The rest of the answers are way too over convoluted for such a simple problem. Good job!
The other answers are complicated because we are looking for the literal binary representation of the struct. This and all the other json solutions add extra bytes and parsing overhead, which, aside from the worse memory and speed performance, also messes up protocols that expect binary encoding.
in case someone needs to change how keys are formatted in json - use tag on fields in struct: SomeKey string `json:"some_key"`
27

I know this thread is old, but none of the answers were accepted, and there's a pretty simple way to do this.

https://play.golang.org/p/TedsY455EBD

important code from playground

import (
  "bytes"
  "fmt"
  "encoding/json"
)

type MyStruct struct {
  Name string `json:"name"`
}

testStruct := MyStruct{"hello world"}
reqBodyBytes := new(bytes.Buffer)
json.NewEncoder(reqBodyBytes).Encode(testStruct)

reqBodyBytes.Bytes() // this is the []byte

4 Comments

This will convert the struct to a bytes array in "JSON" format. I'm assuming the question is about converting the struct to a bytes array in "Binary" format (For lower memory/disk consumption).
Indeed it will return an array of [123 34 110 97 109 101 34 58 34 104 101 108 108 111 32 119 111 114 108 100 34 125 10] or {"name":"hello world"}LF
how to read it back to struct?
22

Serialization is likely the proper answer.

But if you consent to unsafety and actually need to read struct as bytes, then relying on byte array memory representation might be a bit better than relying on byte slice internal structure.

type Struct struct {
    Src int32
    Dst int32
    SrcPort uint16
    DstPort uint16
}

const sz = int(unsafe.Sizeof(Struct{}))
var asByteSlice []byte = (*(*[sz]byte)(unsafe.Pointer(&struct_value)))[:]

Works and provides read-write view into struct, zero-copy. Two "unsafe" should hint enough that it may break badly.

1 Comment

it looks really C++ like, why is it unsafe...
11

You should use a bytes buffer instead of a string, the other suggested methods create a SHA1 of variable length, the SHA1 standard length must be 20 bytes (160 bits)

package main

import (
    "crypto/sha1"
    "fmt"
    "encoding/binary"
    "bytes"
)

type myStruct struct {
    ID   string
    Data string
}

func main() {
    var bin_buf bytes.Buffer
    x := myStruct{"1", "Hello"}
    binary.Write(&bin_buf, binary.BigEndian, x)
    fmt.Printf("% x", sha1.Sum(bin_buf.Bytes()))
}

Try it yourself: http://play.golang.org/p/8YuM6VIlLV

It's a really easy method and it works great.

6 Comments

Thank you Der, I'll edit this. Was my first comment.
Doesn't look like this works because myStruct isn't fixed size. play.golang.org/p/IGA_lgRVNX
Did it work? Check the err returned from binary.Write reveals "binary.Write: invalid type main.myStruct"....
Proof that this does not work. Would've been a great thing though.. play.golang.org/p/gpBUXMnaOUJ
This does not work, err - "binary.Write: invalid type main.myStruct"
|
10

json.Marshal is the best option to convert a struct to []byte, see example below:

package main

import (
    "encoding/json"
    "fmt"
)

type ExampleConvertToByteArray struct {
    Name    string
    SurName string
}

func main() {

    example := ExampleConvertToByteArray{
        Name:    "James",
        SurName: "Camara",
    }
    
    var exampleBytes []byte
    var err error

    exampleBytes, err := json.Marshal(example)
    if err != nil {
        print(err)
        return
    }

    fmt.Println(string(exampleBytes))
}

Go playground -> https://play.golang.org/p/mnB9Cxy-2H3

Comments

9
package main

import (
    "crypto/sha1"
    "fmt"
    "encoding/binary"
    "bytes"
)

type myStruct struct {
    ID   [10]byte
    Data [10]byte
}

func main() {
    var bin_buf bytes.Buffer
    x := myStruct{"1", "Hello"}
    binary.Write(&bin_buf, binary.BigEndian, x)
    fmt.Printf("% x", sha1.Sum(bin_buf.Bytes()))
}

binary.Write takes a struct which has fixed length memory allocated datatype.

1 Comment

This no longer compiles but it works for ints. Furthermore, the issue with this answer is that the size of the underlying byte slice is not always the size of the struct.
3

Take a look at https://blog.golang.org/go-slices-usage-and-internals Specifically slice internals. The idea is to mimic slice's internal structure and point to our struct instead of a byte sequence:

package main

import (
    "fmt"
    "unsafe"
)

// our structure
type A struct {
    Src int32
    Dst int32
    SrcPort uint16
    DstPort uint16
}

// that is how we mimic a slice
type ByteSliceA struct {
    Addr *A
    Len int
    Cap int
}

func main() {
    // structure with some data
    a := A{0x04030201,0x08070605,0x0A09, 0x0C0B}

    // create a slice structure
    sb := &ByteSliceA{&a, 12, 12} // struct is 12 bytes long, e.g. unsafe.Sizeof(a) is 12

    // take a pointer of our slice mimicking struct and cast *[]byte on it:     
    var byteSlice []byte = *(*[]byte)(unsafe.Pointer(sb))

    fmt.Printf("%v\n", byteSlice)
}

Output:

[1 2 3 4 5 6 7 8 9 10 11 12]

https://play.golang.org/p/Rh_yrscRDV6

1 Comment

@chewbapoclypse this only works with structs with no internal pointers. Tt won't work with strings, since string has an internal pointer to an underlying sequence of elements. You have to keep your string data in arrays for this to work.
2

Have you considered serializing it to bson? http://labix.org/gobson

Comments

0
var v any
b := (*[unsafe.Sizeof(v)]byte)(unsafe.Pointer(&v))
c := b[:]

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.