The Hidden Costs of AWS Storage Services – Spiceworks

Aviram Levy, head of the product enablement at Zesty, shares a detailed overview of the storage services available on the AWS Cloud platform. After the introduction, the pricing of each storage service will be provided to the reader as specified by Amazon. Aviram further lists the hidden culprits contributing to unexplained increments in the cost of running storage services on AWS. Where possible, these culprits will be exposed in detail to the reader, along with their added cost.

Cloud computing is an innovative technology, but the cost of working in the cloud can be more than we expect sometimes. Despite the growing number of cloud service providers, engineering teams experience unstable prices for the same monthly services, leading to 35% of wasted cloud spendOpens a new window .

Enterprise organizations, in particular, are feeling this pain. Pinterest experienced a 41% higher AWS bill in 2018 versus 2017Opens a new window due to having to purchase extra capacity at a higher cost despite the fact they had paid for capacity in advance. Furthermore, a 2021 report by VirtanaOpens a new window indicated that 82% of 350 companies based in the US and UK experience “unnecessary” cloud costs.” The same report noted a Gartner prediction that “60 percent of infrastructure and operations leaders will encounter cloud cost overruns by 2024.”

There are multiple cases, reported or not, where unnecessary cloud spend burns holes in a company’s technical purse. 

This article will help you identify the culprits responsible for cloud waste on AWS and teach you how to nip them in the bud. We’ll also discuss how S3 pricing is calculated and some general cost optimization tenets.

Brief Overview of AWS Storage Services

On the AWS cloud platform, customers can take advantage of a pay-as-you-go model, meaning customers only pay for the individual resources and services they consume for a period of time.

The price of using AWS’ state-of-the-art cloud tools depends on three factors: computing, storage, and outbound data transfers. In this article, we’ll discuss storage services with hidden costs, which we’ll classify as our primary  “culprits.” Several AWS storage services include

However, as S3 and EBS are the most commonly used, we will focus on them when discussing AWS culprits below. Also, this article will dive deeper into Amazon S3 since it has become the standard storage service for developers due to being more cost-efficient and durable.

Amazon S3 Pricing

It’s essential to know that for the S3 service, you only pay for the storage capacity you use, not the capacity provisioned. In other services, you pay for the resources provisioned, whether or not you used them.

Typically, there are six cost components for the S3 service:

How S3 Cost Is Calculated

AWS S3 service costs can be grouped into three major categories:

S3 Culprits and How to Optimize Them

There are several hidden culprits and techniques for optimizing AWS S3 cloud spend.

Compute and storage location

Having an Amazon EC2 instance and your S3 service set to different AWS regions reduces the performance overhead and increases the data transfer cost. You can transfer data for free when the EC2 instance and the S3 bucket are set to the same region.

If your data transfer usually includes another region, replicating the S3 bucket might be the best way to optimize spend. Remember that replicating the bucket may become expensive if the objects and total size are large or there are too many replications. Also, replication may lead to inconsistency in data. If there is high traffic in downloads from the S3 servers, migrate cacheable files to AWS CloudFront since it is more cost-effective and performant than S3 in most cases.

Not monitoring

Setting up CloudWatch alerts is important for monitoring usage metrics. Furthermore, you can also enable Cost Explorer or download reports about your S3 buckets to stay informed about your usage costs.

Keeping irrelevant files

Remember to delete files when they are no longer relevant. Also, if you have any relevant files that are not currently needed but can be recreated with relative ease, delete them too. Confirm that there are no incomplete uploads; if there are, delete them since if you leave them, you pay for them. The  S3 lifecycleOpens a new window helps you automate this process by enabling you to configure the lifecycle per various policies. In addition, this feature also lets you retain data at a reduced cost. 

S3 versioning

AWS charges for every version of a data object that is stored and transferred. Replications of an object are complete objects by themselves. If you have multiple versions of an object, you will be billed for them all. 

Large files

Compressing objects deliver better performance, reduce the storage capacity required, and, in turn, lowers costs. Depending on the use case, compressions such as LZ4, GZIP, or even ZSTD provide good performance and take up minimal space.

API calls remain the same.

Whether you’re uploading a large or a smaller object, the API cost is not dependent on the data size but on the type of call. That means you can only optimize it by uploading in bulk instead of in chunks, so you may have to batch or zip different files to create a single file for upload.

S3 intelligent tiering

Transitioning objects from the standard S3 service to the S3 Intelligent Tiering service attracts an additional fee. Depending on the access patterns of your objects, you may or may not be saving costs by using Intelligent Tiering. 

S3 Storage Tier

82.61%

95.70%

Data stored in Infrequent Access (IA) requires a retrieval fee of $0.01 per GB, but this is 4x cheaper than using S3 storage even though retrieval is free there. IA is the best option if you retrieve your data objects once a month. Also, you can use IA for disaster recovery. 

Data stored in the Frequent Access tier doesn’t save any cloud costs. It is noteworthy that S3 has a monitoring fee of $0.0025 per 1,000 data objects in a month. The formula for this is:

Monitoring costs per month = (number of objects with Intelligent Tiering/1000 objects) * $0.0025.

Note that transitioning objects to the lower tiers come with tradeoffs like reduced replications and SLA and a longer retrieval time for the lowest tiers. You should also note that there is a fee for transitioning data objects between two storage classes at a rate of $0.01 per 1,000 objects:

Transition costs per month = (number of data objects being transitioned / 1000 objects) * $0.01

AWS key management system alternative

Your cloud bill increases when you use Amazon S3 with the AWS KMS service because every data object requires a KMS encryption operation to write and a decryption operation to read. To reduce this cost, use the Amazon S3 Bucket Keys instead. This provides as much as a 99% reduction in unnecessary spending without changing dependent applications.

It is worth knowing that you are billed for disabled KMS keys on your account and active keys. So, consider deleting disabled keys if they don’t adversely affect services configured with those keys.

See More: How To Secure Enterprise Data Using the AWS Shared Responsibility Model

Amazon Elastic Block Store (EBS)

Amazon EBS comprises volume storage and snapshots. The storage volumes can be attached to EC2 instances to carry out block storage tasks. The snapshots provide an easier and more secure way to protect stored data such as data volumes, boot volumes, non-EBS volumes, and block data. For this reason, the pricing for EBS volumes is different from that of EBS snapshots.

The standard cost for snapshots is $0.05/GB/month. The cost for the archive is $0.0125/GB/month. 

Optimizing AWS EBS culprits

Optimizing your AWS EBS cost is a straightforward process that can be categorized into two parts.

Deleting independent volumes and old snapshots

Deleting a volume significantly reduces your cloud storage spend since EBS volumes are priced based on the amount of storage used. Contrarily, deleting a snapshot might not reduce cloud storage spending since only the data referenced by the snapshot is deleted.

Even after stopping an instance, the volumes configured to those instances retain information and add it to your bill. To stop EBS charges, consider deleting any volume or snapshot you’re not using.

Emptying the recycle bin

You pay for the EBS storage capacity provisioned to you. This can be either a volume, a snapshot, or in the recycle bin. The recycle bin secures snapshots that were deleted unintentionally. For this reason, snapshots in the recycle bin cost the same as a normal snapshot. So remember to delete the contents of the recycle bin to confirm deleted snapshots.

General Recommendations

Giving inexperienced users access to perform operations with the S3 service can also warrant unnecessary costs. Many developers have spawned demo instances but failed to turn them off after usage. It is advisable to set down and follow procedures while working with cloud infrastructure and promote a culture of ownership towards created resources.

When an AWS service is not being used for a long time, it’s always best to turn it off or delete it if possible, as it may be contributing to wastage without your knowledge.

Define and use monitoring metrics where possible. Also, endeavor to check these metrics regularly to avoid surprises. You can leverage alerts and notifications frequently to avoid waking up to outrageous bills. Building and training a team to architect and manage your cloud spend might be a good idea. Finally, a culture of retention-for-everything and backup in lower S3 tiers is beneficial to implement from the beginning, as it’s sometimes impossible to tell which data object is useful in the long term.

How do you think organizations can reduce the aforementioned costs while using the storage services? Share your thoughts with us on FacebookOpens a new window , TwitterOpens a new window , and LinkedInOpens a new window . We’d love to hear from you!

This content was originally published here.