Reading an io.Reader of unknown size into bytes array in golang

Question

I have a file (say the descriptor is named file) opened via the os.Open() method so I want to read its contents into a bytes array.

I assume the approach would be to create the later

data := make([]byte, 10000000)

and then read the contents into it

n, err := file.Read(data)

My question is whether there is a more elegant/go-idiomatic way of going about this since by not knowing in advance the file size, I just pass a number I estimate would do (10000000) upon initialisation of the bytes array.

golang.org/pkg/io/#ReadAll. Read doesn't work anyway because the underlaying reader isn't obligated to return all data in one go. — Peter
– Peter, Commented Mar 20, 2021 at 9:47
I think that's the correct answer; if you post it as an answer I will accept it — pkaramol
– pkaramol, Commented Mar 20, 2021 at 9:51

gonutz · Accepted Answer · 2022-03-25 07:49:35Z

10

You can use the io/ioutil (up to Go 1.15) or os (Go 1.16 and higher) package. There is a helper function that reads an entire file into a byte slice.

// For Go 1.15 and lower:
package main

import "io/ioutil"

func main() {
    data, err := ioutil.ReadFile("path/to/my/file")
}

// For Go 1.16 and higher:
package main

import "os"

func main() {
    data, err := os.ReadFile("path/to/my/file")
}

In Go version 1.16 the io/ioutil function is still there for compatibility reasons but it was replicated in os. I assume that it will stay in io/ioutil for as long as Go has a version 1.xx because of the compatibility promise so you might keep using that one.

In case you have a file descriptor or any io.Reader you can use the io/ioutil package as well:

package main

import (
    "io/ioutil"
    "os"
)

func main() {
    f, _ := os.Open("path/to/my/file")
    data, err := ioutil.ReadAll(f)
}

edited Mar 25, 2022 at 7:49

answered Mar 20, 2021 at 10:15

gonutz

5,7123 gold badges28 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

qxxt · Accepted Answer · 2022-01-23 02:13:07Z

1

The ioutil.ReadAll() is a wrapper for io.ReadAll() and it use predetermined size "512" bytes which enlarge as the read loops.

You can instead use the size from (*file.File).Stats().Size().

A more straightforward method would be using os.ReadFile() which automatically convert a file into []byte for you. It creates []byte with sizes from Stats().Size() method above before reading the file into byte.

answered Jan 23, 2022 at 2:13

qxxt

992 silver badges6 bronze badges

Collectives™ on Stack Overflow

Reading an io.Reader of unknown size into bytes array in golang

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related