Saving file to S3 using aws-sdk-go

Question

I'm having a bit of trouble saving a file in golang with the AWS S3 go sdk (https://github.com/awslabs/aws-sdk-go).

This is what I have:

import (
        "fmt"
        "bytes"

        "github.com/awslabs/aws-sdk-go/aws"
        "github.com/awslabs/aws-sdk-go/aws/awsutil"
        "github.com/awslabs/aws-sdk-go/service/s3"
)

func main() {    
    cred := aws.DefaultChainCredentials
    cred.Get() // i'm using environment variable credentials and yes, I checked if they were in here
    svc := s3.New(&aws.Config{Region: "us-west-2", Credentials:cred, LogLevel: 1})
    params := &s3.PutObjectInput{
                    Bucket:         aws.String("my-bucket-123"),
                    Key:            aws.String("test/123/"),
                    Body:         bytes.NewReader([]byte("testing!")),
            }
    resp, err := svc.PutObject(params)
    fmt.Printf("response %s", awsutil.StringValue(resp))
}

I keep receiving a 301 Moved Permanently response.

Edit: I created the bucket manually. Edit #2: Example response:

---[ RESPONSE ]--------------------------------------
HTTP/1.1 301 Moved Permanently
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 05 May 2015 18:42:03 GMT
Server: AmazonS3

POST sign is http as well.

can you post the logs of what is says ? usually it happens when you are accessing it to an http endpoint and is redirecting you to an httpS endpoint. Still the library should do it for you. — fabmilo
– fabmilo, Commented May 5, 2015 at 18:11
Also suspect that maybe it wants the Config to specify the Endpoint for your region (it's got a field for it), and also think there shoul dbe some good way to get it filled in automatically -- there's EC2.DescribeRegions, but that seems backwards. — twotwotwo
– twotwotwo, Commented May 6, 2015 at 5:20
@twotwotwo Specified s3-us-west-2.amazon.com as an endpoint but it didn't work. — John
– John, Commented May 6, 2015 at 13:53

Community · Accepted Answer · 2020-06-20 09:12:55Z

3

According to Amazon:

Amazon S3 supports virtual-hosted-style and path-style access in all regions. The path-style syntax, however, requires that you use the region-specific endpoint when attempting to access a bucket. For example, if you have a bucket called mybucket that resides in the EU, you want to use path-style syntax, and the object is named puppy.jpg, the correct URI is http://s3-eu-west-1.amazonaws.com/mybucket/puppy.jpg. You will receive a "PermanentRedirect" error, an HTTP response code 301, and a message indicating what the correct URI is for your resource if you try to access a bucket outside the US Standard region with path-style syntax that uses either of the following:

http://s3.amazonaws.com

An endpoint for a region different from the one where the bucket resides, for example, http://s3-eu-west-1.amazonaws.com for a bucket that was created in the US West (Northern California) region

I think the problem is that you are trying to access a bucket in the wrong region. Your request is going here:

https://my-bucket-123.s3-us-west-2.amazonaws.com/test/123

So make sure that my-bucket-123 is actually in us-west-2. (I tried this with my own bucket and it worked fine)

I also verified that it's using HTTPS by wrapping the calls: (their log message is just wrong)

type LogReadCloser struct {
    io.ReadCloser
}

func (lr *LogReadCloser) Read(p []byte) (int, error) {
    n, err := lr.ReadCloser.Read(p)
    log.Println(string(p))
    return n, err
}

type LogRoundTripper struct {
    http.RoundTripper
}

func (lrt *LogRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    log.Println("REQUEST", req)
    res, err := lrt.RoundTripper.RoundTrip(req)
    log.Println("RESPONSE", res, err)
    res.Body = &LogReadCloser{res.Body}
    return res, err
}

And then:

svc := s3.New(&aws.Config{
    Region:      "us-west-2",
    Credentials: cred,
    LogLevel:    0,
    HTTPClient:  &http.Client{Transport: &LogRoundTripper{http.DefaultTransport}},
})

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered May 6, 2015 at 21:58

Caleb

9,53843 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

John Over a year ago

How do you know that it's definitely in us-west-2? On the top right it just mentions global.

Caleb Over a year ago

In the AWS console, go to S3, then click on "Properties" on the right, and then select your bucket. It will show the region there.

Mat Jovanovic · Accepted Answer · 2020-05-27 07:05:06Z

I think you're better off using the S3 Uploader. Here's an example from my code, it's a web app, I use gin framework, and in my case I get a file from a web form, upload it to s3, and retrieve the URL to present a page in other HTMLs:

// Create an S3 Uploader
  uploader := s3manager.NewUploader(sess)

  // Upload 
  result, err := uploader.Upload(&s3manager.UploadInput{
   Bucket: aws.String(bucket),
   Key: aws.String(fileHeader.Filename),
   Body: f,
  })
  if err != nil {
   c.HTML(http.StatusBadRequest, "create-project.html", gin.H{
    "ErrorTitle":   "S3 Upload Failed",
    "ErrorMessage": err.Error()})
  } else {
   // Success, print URL to Console.
   ///"result.Location is the URL of an Image"
   fmt.Println("Successfully uploaded to", result.Location)
  }

You can find an entire example here, explained step by step: https://www.matscloud.com/docs/cloud-sdk/go-and-s3/

Collectives™ on Stack Overflow

Saving file to S3 using aws-sdk-go

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related