How to deal with []byte (byte array) with base64 encoding as input

Question

I want to send a json string with base64 encoded file in it from a client, basically it looks like this:

{  
   "data":"aGVscA==",
   "filename":"file.txt"
}

And I wrote this struct:

type StoredFile struct {
    Data     []byte `json:"data"`
    Filename string `json:"filename"`
}

Then I decode the json into the struct:

decoder := json.NewDecoder(request.Body)
storedFile := StoredFile{}
err := decoder.Decode(&storedFile)

And save it with gorm:

db.Create(&storedFile)

My question is:

How do json package decode the base64 string to byte array? Does it treat it like plain text, because I know that the data size will be increased by 33% when it is encoded to base64 and if it is treated like text the 33% increase will still be there right?
What kind of encoding does gorm use when it stores []byte to PostgreSQL database? Is it UTF-8? If not, how do I set the encoding to UTF-8?

Ullaakut · Accepted Answer · 2018-09-12 05:48:26Z

1

Yes, it considers your b64 string like plain text, because there's no way it can know that it's a base 64 string. So yes again, your size increase will still be there.
The encoding depends on the table. If gorm is simply writing into an already existing table, it will use the encoding that is specified in the structure. For example:

CREATE TABLE `users` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `email` varchar(255) NOT NULL,
  `password` varchar(255) NOT NULL,
  `token_user_id` varchar(255) NOT NULL,
  `is_admin` tinyint(4) NOT NULL DEFAULT '1',
  PRIMARY KEY (`id`),
  UNIQUE KEY (email)

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

For this table, the CHARSET is utf8mb4 so it would store using an UTF-8 encoding.

If gorm is also responsible for creating the table however, I believe it uses utf8 by default as the encoding, but I can't seem to find any source to back it up.

answered Sep 12, 2018 at 5:48

Ullaakut

3,7743 gold badges24 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

William Wino Over a year ago

If I store non-base64encoded bytes in the database, how does the json library convert it to json? Is it using a base64 encoding?

Ullaakut Over a year ago

It stores a StoredFile structure, right? So when you put it in your database, it's just inserting an entry to your StoredFiles table that contains two strings. Since your database doesn't know the difference between a slice of bytes and a string, it will just be inserted as two strings. There is no conversion or encoding made into base 64, no.

William Wino Over a year ago

Alright, thanks for the answer. What is your suggestion on this? Should I just override the MarshalJSON and UnmarshalJSON? Or there is a tag for this?

William Wino Over a year ago

there is someone who says that it is encoded with base64 encoding: stackoverflow.com/questions/36465065/…

Ezequiel Muns Over a year ago

If you wanted your data to be base64 encoded when in JSON format, and binary encoded when it's in the database, the best way is to create your own type (based on a []byte), which implements the driver.Valuer, sql.Scanner, json.Marshaler and json.Unmarshaler interfaces.

Collectives™ on Stack Overflow

How to deal with []byte (byte array) with base64 encoding as input

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related