With due reference to the blog which helped me in the right direction, the tensult blogs article Exporting of AWS CloudWatch logs to S3 using Automation, though at some points I have deviated from the original author’s suggestion.
Some points are blindly my preference and some other due to the suggested best practices. I do agree that starters, would be better off with setting IAM policies with ‘*’ in resource field. But when you move things into production it is recommended to use least required permissions. Also some critical policies were missing from the assume role policy. Another unnecessary activity was the checking of existence of s3 bucket and attempt to create if not exists, at each repeated execution. Again for this purpose the lambda role needed create bucket permission. All these were over my head, and the out come is this article.
Well if you need cloudwatch logs to be exported to S3 for whatever reason, this could save your time a lot, though this needs to be run in every different region where you need to deploy the stack. Please excuse me as the whole article expects to have aws-cli and sam-cli pre installed.
First lets create the bucket and apply a policy that permits the cloud watch logs service to write into the bucket. Also the bucket owner has full permission on any created object.
aws s3 mb s3://<bucketname> --region <AWS_REGION>
aws s3api put-bucket-policy --bucket <bucketname> --policy file://policy.jsonThe first line creates the bucket and the second line adds the policy, for the sake of explanation the file is reproduced below.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": "logs.ap-south-1.amazonaws.com"
},
"Action": "s3:GetBucketAcl",
"Resource": "arn:aws:s3:::<bucketname>"
},
{
"Effect": "Allow",
"Principal": {
"Service": "logs.ap-south-1.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::<bucketname>/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
]
}Now we can deploy the template using
sam deploy -gThen respond with the responses as expected.
To cut the story short, using the template we create two roles, one for the state machine, the second for lambda, since the state machine requires lambda:InvokeFunction permission on the lambda and actual lambda does not need the same. Continuing the Lambda Function, which actually does the whole job, is mostly a branch from the same function provided in the original article but logically those parts which does the bucket existance check and the sort was removed.
The whole template and its accessories can be downloaded here. Download Export Cloudwatch Logs to AWS S3 SAM Template.
