92

Is there a way to specify the CloudWatch log group that an AWS lambda logs to? It seems to be generated directly from the lambda name; however, it would be especially convenient to, for example, aggregate multiple lambdas to a single log group. We are especially interested in specifying the log group when the lambda is created by a CloudFormation template.

2
  • 4
    Request for a LogGroup property to be added to AWS::Lambda::Function: github.com/aws-cloudformation/… Commented Aug 26, 2019 at 17:45
  • 2
    It's now possible. See my answer below. Commented Jan 19, 2024 at 19:24

7 Answers 7

44

Actually, maybe you can, to an extent at least. I too was in search of the answer and gave this a try. Here's a snippet of two resources; the lambda function and the log group:

"MyLambdaFunction": {
    "Type": "AWS::Lambda::Function",
    "DependsOn": "ReadWriteRole",
    "Properties": {
        //snip
    }
},

"MyLambdaFunctionLogGroup": {
    "Type": "AWS::Logs::LogGroup",
    "DependsOn": "MyLambdaFunction",
    "Properties": {
        "LogGroupName": {"Fn::Join": ["", ["/aws/lambda/", {"Ref": "MyLambdaFunction"}]]},
        "RetentionInDays": 14
    }
},

I found that the log group was created with a retention of 14 days as indicated. When the lambda function runs, it does create log streams in this group. When I deleted the stack, however, it seems that the log groups is not deleted, and the retention is now set to never expire. Perhaps that's good enough so the streams don't get too out of hand...

Sign up to request clarification or add additional context in comments.

14 Comments

I was very optimistic about this answer but ran into the issue of the LogGroup failing because the LogGroupName already exists
you can remove DependsOn as we are using ARN of lambda in group name so its implicit.
@Purefan I was getting the same when updating a stack. I solved it by deleting the whole stack and creating it again.
This answer doesn't address the question asked
The Log Group must be created first. The Lambda function must depend on the log group.
|
32

I don't think that is possible.

Even if it were possible, each AWS Lambda instance would still write to its own log-stream. And though different invocations of the same lambda can write to the same log-stream (when the lambda instance is reused), this will definitely not be the case for different lambdas (since they must use different lambda instances).

As a result, you must have a tool that aggregates multiple log-stream. If so, what's the problem with making it a little more generic, so that it can aggregate log-streams from different log groups?

6 Comments

You are correct. I asked AWS support for the same thing and was told that it wasn't possible and probably would never be possible. However the reason multiple instances of a Lambda function do not log to the same log stream is because there is a limit of 5 requests/second/log-stream. So even if you tried to manually write to a single log stream from multiple Lambda instances you would soon run into that limit.
Sadly that there is no way to specify log group for individual Lambda function. Tools like this (github.com/TylerBrock/saw), where it can stream an aggregated multiple log-streams from within a particular log group will not be able to take benefit of it.
This answer doesn't address the question asked... sorry, couldn't resist. But in all seriousness if you look at the initial question, this more focuses on the use case at the end than the actual question, IMHO
@CraigBrett The answer to the question asked is in the first sentence. The essence of the answer is such that it doesn't allow for any elaboration (and neither it is possible to provide proof by linking to documentation, since unsupported features are usually simply missing from documentation and such facts are explicitly stated only in special cases). The rest of the answer are some additional thoughts to make it meet the criteria of an answer on SO (otherwise it would rather be written as a comment and would be impossible to accept).
@CraigBrett Anyone coming to this page with the purpose of specifying a log group for their AWS Lambda will find out that the most they can do is specify a log group that is no different from what would be generated by default. Therefore the other two answers, however useful from other perspectives unrelated to the OP's problem, do not address the question asked. Hence my comments. I think that it would be more adequate to post a new question focusing on the retention of logs (and maybe linking to this question) and then self-answering that very different question.
|
25

I find that @lingrlongr's answer is partially correct.

First, to answer the original question, you cannot specify a custom log group name for the lambda function to write to.

The lambda log group name always follows this pattern:

/aws/lambda/<function-name>

The lambda function will first check if a log group exists with this name.

  • If it exists, then it will use that.
  • If it does not exist, it will create a log group with that pattern.

Hence, if you want to add settings, such as RetentionInDays and SubscriptionFilter, make sure your CloudFormation or SAM template creates the LogGroup before the lambda function. If your lambda function is created first, an error will be thrown when creating the LogGroup saying that the LogGroup already exists. So, the lambda function should have DependsOn: LogGroup instead of the other way round.

Also, make sure you are not using Ref or GetAtt to reference the lambda function inside the LogGroup because that creates an implicit dependency on the lambda function causing the lambda function being created before the LogGroup.

Comments

9

Creating the log group as mentioned as one of the answers works. To keep the retention policy after the stack is deleted, just add a DeletionPolicy.

"MyLambdaFunctionLogGroup": {
  "Type": "AWS::Logs::LogGroup",
  "DependsOn": "MyLambdaFunction",
  "DeletionPolicy": "Retain",
  "Properties": {
    "LogGroupName": {"Fn::Join": ["", ["/aws/lambda/", {"Ref": "MyLambdaFunction"}]]},
    "RetentionInDays": 14
  }
}

2 Comments

This answer doesn't address the question asked
The Lambda should depend on the Log Group. The Log Group must be created first.
8

As of November 2023, you can now specify a custom CloudWatch Logs log group name when creating a Lambda function.

In CloudFormation for your Lambda function you can indicate the log group name, as follows:

Type: AWS::Lambda::Function
Properties:
  FunctionName: <my function name>
  LoggingConfig:
    ApplicationLogLevel: TRACE
    LogFormat: <optional custom format>
    LogGroup: <custom log group name>
    SystemLogLevel: WARN

Note that you can also indicate the application log level (DEBUG | ERROR | FATAL | INFO | TRACE | WARN) and system log level (DEBUG | INFO | WARN) which you previously could not do, as well as the log format.

1 Comment

This is the correct answer.
5

I attached LogGroup to Serverless function in SAM template as follows:

    MyFuncLogGroup:
        Type: AWS::Logs::LogGroup
        Properties:
            LogGroupName: '/aws/lambda/stackName-env-myFunc-v1'
            RetentionInDays: 30

    MyFunc:
        Type: AWS::Serverless::Function
        Properties:
            FunctionName: 'stackName-env-myFunc-v1'
            ...

Some users mentioned the requirement of DependsOn property but it is not required in my experience when using SAM. The only requirement is that the LogGroupName must be /aws/lambda/<FunctionName>. SAM will create the log group before the lambda function as long as your are not referencing the function's logical ID in the LogGroup

Also, if you are adding LogGroup to exisiting function, you can do so by simply updating your template.yaml with the LogGroup resource like above and add FunctionName property to the function resource. Obviously, the FunctionName should be different from your existing lambda function name that appears in your lambda's ARN.

WARNING: FunctionName property requires replacement. Your existing lambda will be deleted and new one will be created.

3 Comments

This seems to work initially but then when the lambda function is created, the retention is reset to 3 days.
davegallant, do you have some global settings for cloudwatch? I have been using this config for multiple projects and the retention period is persistent for all of them.. It is possible that the retention period was modified from console.
No. I got around it by using the awscli command aws logs put-retention-policy after the fact which isn't pretty but works!
5

Okay so I had to do this myself and @pat-myron commented a link to the way you do it below the answer.

He posted a feature request where @benbridts outlines the template here.

Here is the gist of what worked in my cloudformation template:

HelloWorldLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: "/aws/lambda/HelloWorld"
      RetentionInDays: 30

HelloWorldFunction:
  Type: AWS::Lambda::Function
  DependsOn: HelloWorldLogGroup
  Properties:
    FunctionName: HelloWorld
    Role:
      Fn::GetAtt:
        - LambdaExecRole
        - Arn

LambdaExecRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole
    Policies:
      - PolicyName: LambdaLogging
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action:
                - logs:CreateLogStream
                - logs:PutLogEvents
              Resource: !GetAtt HelloWorldLogGroup.Arn

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.