AWS Tagging evolution: From simple labels to cost allocation and standardization

elcome to the world of AWS Tagging! If you are an AWS user, you must be familiar with the struggles of managing resources. AWS tagging is a lifesaver when organizing and managing your resources efficiently. AWS tagging allows you to label or assign metadata to your AWS resources.
AWS tagging has come a long way from mere labeling to cost allocation and organization-wide standardization. It has become an indispensable tool for managing AWS resources effectively. So, buckle up and join us on this exciting journey where we explore the AWS tagging evolution from labeling to cost allocation and organization-wide standardization.

AWS Tagging Evolution: Labelling to Cost Allocation to Standardization

AWS launched its maiden services - S3 and EC2, in 2006, ushering the world into the Cloud Computing era. Eventually, many companies started exploring AWS to bring in complex workloads. One of the biggest AWS customers - Netflix, started using AWS in 2008. By 2010, it was clear that cloud computing would grow by leaps and bounds in the coming years. Until then, with a handful of EC2 instances or EBS volumes, it was easy to correlate the cloud resources with their workloads/projects or teams. But with complex workloads, the number of cloud resources such as EC2 instances started rapidly growing, making it increasingly difficult to identify and correlate these resources.

‍

The AWS customers started asking for an easy mechanism to identify and correlate the cloud resources. In response to growing customer requests, AWS introduced resource tagging in 2010. This first piece was more for identifying or labeling the cloud resources. It came up with up to ten tags per resource to attach metadata for the cloud resource. It was limited only to EC2 and a few other shared resources. But still, this was a big step in the right direction. At this point, tags were supposed to be used by human users to identify cloud resources, and thus user-friendliness was a critical requirement.

The AWS tags are case-sensitive. This is official AWS stand but still some AWS services may not follow it (e.g. IAM Role). Perhaps user-friendliness is the reason the tags are case-sensitive. This case sensitiveness is one of the factors contributing to resource tags consistency problems that we often encounter. Each AWS tag has a key that may contain up to 128 characters and a value of up to 256 characters. The tags are stored as part of the user's AWS account and are private to the account. Users can manipulate the tags using API calls, including CreateTags, DescribeTags, and DeleteTags.

Initially, a single AWS account was the norm. With newer workloads, many departments and applications shared the AWS cloud. It caused the cost attribution problem - allocating costs to projects and budgets became increasingly difficult.

Using tags, you can easily filter through your resources and access only what you need. This idea came forward to use tagging not just for resource organization but also for cloud cost management. Using tags to track resource usage down to the department or project level, AWS users can monitor their spending and attribute the cost. With custom billing reports that use AWS tags, users can better break down their bills and allocate the costs appropriately.

In 2012, AWS tagging usage expanded to cloud cost tracking with the introduction of Cost Allocation Tags. The idea was straightforward - the users needed to tell AWS which tags were meaningful for cost allocation. Once AWS knows your cost allocation tags, each billing cycle will have the resource billing data with AWS tags. Of course, users need to ensure all their resources are tagged. With cost allocation tags, it became easy to slice and dice the billing data.

But remember, it was still a very primitive system than what we see today. It included only user-defined tags but was still a step forward. The resource tagging was now ready for many more applications. The majority of these applications were for automation. The tags user-friendliness eventually started becoming less important with the increasing use of tags in automation. In 2015, AWS introduced LookupEvent API, which provided users with detailed information about the activities performed on AWS resources in their accounts. It helped users monitor, audit, and troubleshoot their AWS infrastructure more effectively.

Wouldn't it be great if you could see your resources team-wise, application-wise, or any other custom criteria? And this is what many AWS customers wanted. Until then AWS console didn't have any feature to categorize the cloud resources per customers' needs. AWS addressed this shortcoming in 2014 with a great pair of powerful features: Resource Groups and a Tag Editor. Resource groups help users to easily create, maintain, and view a collection of resources that share common tags. As an example Users can tag 1000s of their EC2 instances with an AWS tag "Project=CloudYali-Inventory", and the Resource Group will group all these EC2 instances. But for it to work, the user had to tag each resource at the creation time. Any tag manipulation at a later stage would be a great pain - you possibly cannot add tags to all your resources one by one. The Tag Editor helped to address this pain point. The Tag editor allows you to manage tags across services and regions. Tag Editor allows global search and bulk update of the AWS tags with a few clicks. Isn't that great?? In the same year, AWS added tagging support to EC2 Auto Scaling. It marked the end of the first phase of AWS Tagging evolution.

In the next phase of tagging evolution, the focus would be on aws tagging strategy implementation with consistency and scale. It was a no-brainer as AWS had taken off in cloud computing.

Scaling the tagging for modern workloads

Now users were using tags for different automation purposes, not merely for resource organization or cost tracking. As the complexities of their applications increased, the limit of ten tags per resource started to constrain the users. During this period, to overcome these limitations, innovative approaches such as compound tag key emerged. In response to this growing need, AWS increased the limit to 50 tags per resource in 2016. And ever since, there has been no change in the maximum number of AWS tags per resource or in tag key/value length.

By this time, the workload started growing to thousands of resources, a lot of them with up to 50 tags. Managing the tags through the console, even with the Tag Editor was error-prone and a big operational headache. More and more teams needed APIs to tag resources programmatically instead of the AWS console. This demand was satisfied with the AWS Resource Tagging API release in 2017.

Over the years, tags have evolved to serve many important purposes. The AWS tags are used for identifying resources for cost allocation, resource access control (either directly or via tags on IAM users & roles), identifying associated security compliance standards or shutting down resources on weekends, and so on. Even a minor crack here could potentially exclude resources from their intended purposes.A formal aws tagging strategy may help ensure all AWS accounts get identically tagged.

The tagging standardization was problematic as there was no AWS tooling support. Eg. The tag key case sensitiveness means there could be many variations of the semantically same key but with different capitalizations. In many cases, multiple tags such as "costCenter", "CostCenter" or "costcenter" get added by different teams at different times in shared ac. Consequently, while standardizing tag key naming, tag values are simple - putting them into practice became the Achilles heel.

‍

AWS users needed a standard mechanism to enforce the tagging strategy in practice. Especially by this time, it was not just about a single account but hundreds of AWS accounts spanning across AWS Organizational Units within an AWS Organization. The idea was to have a Tag Policy which defines all allowed Tag keys (with capitalization) and their respective allowed values. The policy would also enforce if the resource creation should abort if mandatory tags were not present for the resource.

In 2019 AWS introduced the tag policies for AWS Organizations. The AWS Tag Policy, combined with Organization SCPs or IAM policies, assembles a powerful mechanism for tagging compliance and tagging enforcement. Note that AWS doesn't backfill the historical tagging data, AWS users still need to take care of this aspect of tagging.

Finally, in 2022, AWS introduced the Cost Allocation tag API, which provides users with a programmatic way to track costs associated with their resource tags. This feature helps users identify which resources are driving costs and where those costs should be allocated.

From mere labeling in 2010 to the sophisticated tagging services of today, AWS tagging has seen a rapid evolution. Each of these updates has provided users with additional functionality, greater control over their resources, and more effective monitoring, auditing, and management of their AWS infrastructure. But even after a decade, AWs tagging is still a work in progress for most AWS users.