Picking up on Diogo’s last post on how to obliterate all resources on your AWS Account, I thought it could also be useful to, instead, list all you have running.
Since I’m long overdue on a Go post, I’m going to share a one file app that uses the Go AWS SDK for to crawl each region for all taggable resources and pretty printing it on stdout, organised by Service type (e.g. EC2, ECS, ELB, etc.), Product Type (e.g. Instance, NAT, subnet, cluster, etc.).
The AWS SDK allows to retrieve all ARNs for taggable resources, so that’s all the info I’ll use for our little app.
Note: If you prefer jumping to full code code, please scroll until the end and read the running instructions before.
The objective
The main goal is to get structured information from the ARNs retrieved, so the first thing is to create a type that serves as a blue print for what I’m trying to achieve. Because I want to keep it simple, let’s call this type SingleResource.
Also, since we are taking care of the basics, we can also define the TraceableRegions that we want the app to crawl through.
Finally, to focus the objective, let’s also create a function that accepts a slice of []*SingleResource and will convert will print it out as a table to stdout:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // TraceableRegions is a list of AWS regions we want to crawl | |
| var TraceableRegions = […]string{"us-east-1", "us-east-2", "us-west-1", "us-west-2", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-north-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-southeast-1", "ap-southeast-2", "ap-south-1", "sa-east-1"} | |
| // SingleResource defines how we want to describe each AWS resource | |
| type SingleResource struct { | |
| Region *string | |
| Service *string | |
| Product *string | |
| Details *string | |
| ID *string | |
| ARN *string | |
| } | |
| // PrettyPrintResources makes use of a nice golang library to show | |
| // tables on stdout. Check it out: github.com/olekukonko/tablewriter | |
| func PrettyPrintResources(resources []*SingleResource) { | |
| var data [][]string | |
| for _, r := range resources { | |
| row := []string{ | |
| DerefNilPointerStrings(r.Region), | |
| DerefNilPointerStrings(r.Service), | |
| DerefNilPointerStrings(r.Product), | |
| DerefNilPointerStrings(r.ID), | |
| } | |
| data = append(data, row) | |
| } | |
| table := tablewriter.NewWriter(os.Stdout) | |
| table.SetHeader([]string{"Region", "Service", "Product", "ID"}) | |
| table.SetBorder(true) // Set Border to false | |
| table.AppendBulk(data) | |
| table.Render() | |
| } |
https://gist.github.com/joaoferrao/610482a2394b17546e12d1c172f5198e.js
Start parsing the ARN
All ARNs are composed of, at least, something like arn:partition:service:. Having this in mind, we can find a function to extract the Service Name portion of the SingleResource.
Because AWS SDK will allow to crawl only 1 region at a time, we will know which region each resource belongs to. We will also know which account, so… this means we can remove the four first sections of the ARN: arn:partition:service:region:account-id:resourcetype/resource:qualifier, so we can strike that information by creating our own version of a “shortened ARN”.
This will become useful down the line.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // GetServiceFromArn removes the arn:aws: component string of | |
| // the name and returns the first keyword that appears, svc | |
| func ServiceNameFromARN(arn *string) *string { | |
| shortArn := strings.Replace(*arn, "arn:aws:", "", –1) | |
| sliced := strings.Split(shortArn, ":") | |
| return &sliced[0] | |
| } | |
| // Short ARN removes the unnecessary info from the ARN we already | |
| // know at this point like region, account id and the service name. | |
| func ShortArn(arn *string) string { | |
| slicedArn := strings.Split(*arn, ":") | |
| shortArn := slicedArn[5:] // the first 5 we already have | |
| return strings.Join(shortArn, "/") | |
| } |
AWS Resource Types
Since not all ARNs are created equal, it’s useful to create a specific type for each of the services we are expecting to parse. While it’s easy to deduce the name of the service via any ARN name, independently of the resource, it’s not easy to treat all ARNs the same way to get the product of that service.
Let’s take these ARNs for example:
arn:aws:rds:eu-west-1:123456789012:db:mysql-db arn:aws:elasticbeanstalk:us-east-1:123456789012:environment/My App/MyEnvironment arn:aws:s3:::my_corporate_bucket/exampleobject
As you can see, there isn’t an obvious way of retrieving the service name (s3, Elasticbeanstalk, rds), but an obvious way of of stating one is a bucket or another is a db or an environment. This can be further illustrated by the possibilities for naming that AWS provides for ARNs:
arn:partition:service:region:account-id:resource arn:partition:service:region:account-id:resourcetype/resource arn:partition:service:region:account-id:resourcetype/resource/qualifier arn:partition:service:region:account-id:resourcetype/resource:qualifier arn:partition:service:region:account-id:resourcetype:resource arn:partition:service:region:account-id:resourcetype:resource:qualifier
To overcome this, if we really want a specific report at a Product level, we could do so by specifying a method for each type of service that interprets the “shortArn” resulting from the second function we already created. We we want to tell our application how to parse EC2 and ECS ARNs and a Generic method for everything else, we could create a type for them, giving them a method “ConvertToRow”.
Finally, at the bottom of the following snippet, I use a function that accepts the several different bits and pieces of information we were are able to generate so far (service name, ARN, region) and converts that into our intended SingleResource type.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // awsEC2 type is created for ARNs belonging to the EC2 service | |
| type awsEC2 string | |
| // awsECS type is created for ARNs belonging to the ECS service | |
| type awsECS string | |
| // awsGeneric is a is a generic AWS for services ARNs that don't have | |
| // a dedicated type within our application. | |
| type awsGeneric string | |
| // Generic Resource Handler | |
| func (aws *awsGeneric) ConverToResource(shortArn, svc, rgn *string) *SingleResource { | |
| return &SingleResource{ARN: shortArn, Region: rgn, Service: svc, ID: shortArn,} | |
| } | |
| // ConvertToRow converts EC2 shortened ARNs to to a SingleResource type | |
| func (aws *awsEC2) ConvertToResource(shortArn, svc, rgn *string) *SingleResource { | |
| // ec2 instance/i-23123jj1k1k23jh12 | |
| // ec2 security-group/sg-23bn1m231233123m1 | |
| // ec2 subnet/subnet-92i3i1i23i1ih1v23 | |
| // ec2 vpc/vpc-12i3o1ijkj12jh123 | |
| s := strings.Split(*shortArn, "/") | |
| return &SingleResource{ARN: shortArn, Region: rgn, Service: svc, Product: &s[0], ID: &s[1],} | |
| } | |
| // ConvertToRow converts ECS shortened ARNs to to a SingleResource type | |
| func (aws *awsECS) ConvertToResource(shortArn, svc, rgn *string) *SingleResource { | |
| // ecs cluster/some-ecs-cluster | |
| s := strings.Split(*shortArn, "/") | |
| return &SingleResource{ARN: shortArn, Region: rgn, Service: svc, Product: &s[0], ID: &s[1],} | |
| } | |
| // GetResourceRow shortens the ARN and assigns it to the right | |
| // service type calling its "ConvertToRow" method. Since we have | |
| // a default behaviour funneled towards our awsGeneric type, all | |
| // services will be handled. | |
| func ConvertArnToSingleResource(arn, svc, rgn *string) *SingleResource { | |
| shortArn := ShortArn(arn) | |
| switch *svc { | |
| case "ec2": | |
| res := awsEC2(*svc) | |
| return res.ConvertToResource(&shortArn, svc, rgn) | |
| case "ecs": | |
| res := awsECS(*svc) | |
| return res.ConvertToResource(&shortArn, svc, rgn) | |
| default: | |
| res := awsGeneric(*svc) | |
| return res.ConverToResource(&shortArn, svc, rgn) | |
| } | |
| } |
So far..
If you are still following along, we are now able to get the service with ServiceNameFromARN() and a shortened ARN with ShortArn() funcs, we are now able to pass them into ConvertArnToSingleResource() and get the correct SingleResource.
Finally, get the resources
All that’s left is to find a way to get all the ARNs in ones account so our code can feed on them. I left this part to the main() function and use AWS SDK for Resource Groups Tagging API to retrieve such info:
Because this code could be heavily improved, I walk you through the steps first:
- AWS SDK will crawl one region at a time, so I create an
aws.Configfor each one. - The calls to AWS’
GetResourcesmethod will be paginated with a max of 50 results per request, so I create an emptypaginationTokenoutside the most nested for loop to control whether it should keep going or break so the app can create a new config for the next region and a client to callGetResourceson.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| func main() { | |
| var resources []*SingleResource | |
| for _, region := range TraceableRegions { | |
| // We need to create a new CFG for each region. We | |
| // could actually update the region after the fact | |
| // but let's focus on the purpose, here 🙂 | |
| cfg := aws.Config{Region: aws.String(region)} | |
| s := session.Must(session.NewSessionWithOptions(session.Options{ | |
| SharedConfigState: session.SharedConfigEnable, | |
| Config: cfg, | |
| })) | |
| // Creating the actual AWS client from the SDK | |
| r := resourcegroupstaggingapi.New(s) | |
| // The results will come paginated, so we create an empty | |
| // one outside the next for loop so we can keep updating | |
| // it and check if there are still more results to come or | |
| // not. We could isolate this function and call it recursively | |
| // if we wanted to tidy up our code. | |
| var paginationToken string = "" | |
| var in *resourcegroupstaggingapi.GetResourcesInput | |
| var out *resourcegroupstaggingapi.GetResourcesOutput | |
| var err error | |
| // Let's start an infinite for loop until there are no | |
| for { | |
| if len(paginationToken) == 0 { | |
| in = &resourcegroupstaggingapi.GetResourcesInput{ | |
| ResourcesPerPage: aws.Int64(50), | |
| } | |
| out, err = r.GetResources(in) | |
| if err != nil { | |
| fmt.Println(err) | |
| } | |
| } else { | |
| in = &resourcegroupstaggingapi.GetResourcesInput{ | |
| ResourcesPerPage: aws.Int64(50), | |
| PaginationToken: &paginationToken, | |
| } | |
| } | |
| out, err = r.GetResources(in) | |
| if err != nil { | |
| fmt.Println(err) | |
| } | |
| for _, resource := range out.ResourceTagMappingList { | |
| svc := ServiceNameFromARN(resource.ResourceARN) | |
| rgn := region | |
| resources = append(resources, ConvertArnToSingleResource(resource.ResourceARN, svc, &rgn)) | |
| } | |
| paginationToken = *out.PaginationToken | |
| if *out.PaginationToken == "" { | |
| break | |
| } | |
| } | |
| } | |
| // Finally print the results | |
| PrettyPrintResources(resources) | |
| } |
Running Instructions
The full code is available in this Gist if you want to copy to a one .go file, but before running remember to activate your AWS_PROFILE environment variable so that the AWS SDK knows which account and credentials to use. export AWS_PROFILE=aws_named_profile.
In summary, copy everything into a main.go, open the terminal in the same dir and:
export AWS_PROFILE=aws_named_profile go run main.go
Voilá, you should see something similar to the following:
Data Center Notes 