How to Reduce Cost of SAM Artifact Storage using S3 & CloudFormation

One subtle thing that you do when working AWS SAM is that you upload the Lambda deployment packages (artifacts) to a staging bucket where Lambda service retrieve these packages. After the artifacts were retrieved by AWS Lambda from s3, those packages often stay in the staging bucket and may build up over time especially for large projects with tons of Lambda functions. This often lead to a slow and steady increase in S3 bills over time.

In this article, we will explored what are the common ways on to reduce the cost generated by SAM deployment artifacts over time.

Option 1: Add a 1-day Expiry Rule for Dev Environment Artifacts

Deployment artifacts for development environments are basically non-valuable to an organization, it doesn't allow developers to extract useful debugging information out of them except from cases where you really need to inspect the packages. To help your organization reduce cost, you can configure your development artifact storage for S3 bucket to implement an object lifecycle management rule that will expire any build that is older than 1 day. You can use the following CloudFormation template to achieve this:

You can release this template by either:

  1. Using guided deploying by executing the release script from this link. It will request for the Environment Name & AWS CLI Profile Name
  2. If you want to implement your own automation script, you can use the following command:

Option 2: Add a Sensible Expiration Date for UAT & PROD Environment Artifacts

UAT and PROD environment artifacts is a different discussion altogether from the ones from the DEV environments. They are often critical to understanding issues, security holes and defects that existed on your production software. The likeliness of them, providing valuable insights to debugging and investigative activities is higher compared to their dev counterparts. For this case, you might want to consider implementing a longer expiration date for these environments.

I won't be able to define what is a "sensible expiration period" for your organizations. This is usually determined by the following criteria:

  • International laws regarding data security
  • Your country's regulator laws regarding application data & artifacts
  • Your organizations internal policies

You can adjust the sample from dev environment by placing a value (in days format) that makes sense for your organization's use-case to this section of the cloud formation template: https://gist.github.com/allanchua101/4990ddf6c38dec76b5fae676a3b1177e

Option 3: Use a S3 Transition Rule to Update Storage Tier of Older records.

You can also implement a slowly shifting transition rule to your artifacts for production environment by ensuring cold artifacts (Less frequently retrieved) get transitioned to colder tiers for s3. Below is a sample of this implementation:

The sample above applies the following rules to save cost on S3 storage:

  • It shifts warm artifacts to S3 Standard Tier IA after a year
  • It shifts not-so-warm artifacts to S3 Glacier after three years
  • It deletes objects after seven years of existence in the storage

Wrapping Up

SAM artifacts stored in S3 buckets can pile up over a good period of time and it would be wise to implement a smart management of this artifacts. This article showcased how S3 object lifecycle management can help you achieve that.

  • Use one day expiration of dev artifacts
  • Use a sensible expiration date for your organization
  • Sensible artifact expiration dates vary between organizations and territories.
  • You can implement a set of S3 object lifecycle transition rules to slowly move less-frequent accessed artifacts to colder tiers.

Relevant Links

Get Some Cool Stuff From Amazon!


Comments

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. If you are looking for the ultimate learning recourse for .NET developers then Dotnetmastery.com is here to provide you best online E-courses to learn the .Net and more courses online at best prices.

    asp.net Core Web Tutorial

    ReplyDelete

Post a Comment

Popular posts from this blog

API Gateway: Response Aggregation with Ocelot and ASP.net Core

API Gateway in a Nutshell

Security: HTTP headers that expose web application / server vulnerabilities

Building Simple API Gateways with Ocelot and ASP.net Core