problem reading file from s3 using golang with lambda

Question

I have the following code in golang for read a file and extract some colums of the file, the fileName is received from my lambda, but in my first step that is get the file from s3 i am getting a problem.

My code is:

package main

import (
    "fmt"
    "packageXXX/cmd/utils"

    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
    "go.uber.org/zap"
)

var (
    log *zap.Logger
)

func init() {
    log = utils.InitLogger()
}

func DownloadFile(downloader *s3manager.Downloader, bucketName string, key string) error {

    buff := &aws.WriteAtBuffer{}

    numBytes, err := downloader.Download(
        buff,
        &s3.GetObjectInput{
            Bucket: aws.String(bucketName),
            Key:    aws.String(key),
        },
    )

    data := buff.Bytes() // now data is my []byte array

    log.Info("nnnnnnnnnnnnnnn", zap.Any("nnn", numBytes))
    log.Info("xxxxxxxxxxxxxxx", zap.Any("bytes", data))
    return err
}

func handler(fileName string) (string, error) {
    log.Info(fmt.Sprintf("received: %s", fileName))
    sess, err := session.NewSessionWithOptions(session.Options{
        Profile: "default",
        Config: aws.Config{
            Region: aws.String("us-west-2"),
        },
    })

    if err != nil {
        log.Info("errorrrrrrrrr", zap.Any("session", err))
        return "errorr,", nil
    } else {
        log.Info("sessionnnnnnnnnnnnnn", zap.Any("sess", sess))
    }

    bucketName := "s3-test-lambda-go-xxx"
    downloader := s3manager.NewDownloader(sess)
    log.Info("downloader", zap.Any("downloader", err))
    key := fileName
    err = DownloadFile(downloader, bucketName, key)

    if err != nil {
        fmt.Printf("Couldn't download file: %v", err)
        return "errror", nil
    }

    fmt.Println("Successfully downloaded file")

    return fmt.Sprintf("el file recibido es: %s", fileName), nil
}

func main() {
    lambda.Start(handler)
}

And the log that is printing is:

For verbose messaging see aws.Config.CredentialsChainVerboseErrorsSTART RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd Version: $LATEST
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"received: gggggg2.txt"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"sessionnnnnnnnnnnnnn","sessError":"json: unsupported type: endpoints.endpointDefaults"}
{"level":"info","ts":"2022-11-02T17:31:32Z","msg":"downloader","downloader":null}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"nnnnnnnnnnnnnnn","nnn":0}
{"level":"info","ts":"2022-11-02T17:31:38Z","msg":"xxxxxxxxxxxxxxx","bytes":""}
Couldn't download file: NoCredentialProviders: no valid providers in chain. Deprecated.
END RequestId: dba8ebd5-9102-4a2f-8e45-cdef355f1bbd

The problem is the session

Then Is possible create session with role, because like is production i don't want use access and secret key? Thanks a lot

I've found v2 of the aws SDK is easier to work with. aws.github.io/aws-sdk-go-v2/docs/configuring-sdk You can use config.LoadDefaultConfig() to assume the default config, including credentials. You can also specify a credentials file, if you're assuming another user or role. This may be similar with v1 of the sdk, but I'm not as familiar. — Brian Wagner
– Brian Wagner, Commented Nov 2, 2022 at 19:37

Timo Dempwolf · Accepted Answer · 2022-11-02 20:22:55Z

1

Session will load your default session if you but it like this

sess, err := session.NewSession(&aws.Config{
  Region: aws.String("us-east-1")},
)

instead of:

    sess, err := session.NewSessionWithOptions(session.Options{
        Profile: "default",
        Config: aws.Config{
            Region: aws.String("us-west-2"),
        },
    })

This means it will use whatever session you have configured not depending on whether you are inside the lambda or local. This also means you can assume any role you want to assume or even use SSO to login (so no access key and secret are required).

answered Nov 2, 2022 at 20:22

Timo Dempwolf

2622 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2291499 Over a year ago

nice! But now i am getting:

AccessDenied: Access Denied 	status code: 403, request id: 6GGDY59529A0J76Y, host id: 3Ta84qKYTRZNhc5XcwT1Mc/k0ffPlfGyDRR3FBXJpxDCF7PM5VWda8gNZpvOUkQfYkwdxom4sj4=: RequestFailure null

then is necessary create a specific role for my lambda?

Collectives™ on Stack Overflow

problem reading file from s3 using golang with lambda

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related