AWS Distro for Open Telemetry (ADOT) is a secure, AWS-supported distribution of the Open Telemetry project. Users can instrument their ap...
AWS Distro for Open Telemetry (ADOT) is a secure, AWS-supported distribution of the Open Telemetry project. Users can instrument their applications just once and, using ADOT, send correlated metrics and traces to multiple monitoring solutions. Amazon EKS now allows users to enable ADOT as an add-on after the cluster is up and running. The ADOT add-on includes the latest security patches and bug fixes and is validated by AWS to work with Amazon EKS. In this blog post, we will take a look at how to install the ADOT add-on in an EKS cluster and then use it to collect metrics and traces from application workloads deployed to the cluster.
Open Telemetry is a set of APIs, SDKs, and tools that are designed for the creation and management of telemetry data such as traces, metrics, and logs. AWS Distro for Open Telemetry Collector (ADOT Collector) is an AWS-supported version of the upstream Open Telemetry Collector that is fully compatible with AWS computing platforms, including EKS. It enables users to send telemetry data to AWS-managed services such as Amazon CloudWatch, Amazon Managed Service for Prometheus, and AWS X-Ray.
The ADOT Collector has the concept of a pipeline that comprises three key types of components, namely 1. Receiver.
2. Processor.
3. Exporter.
A receiver is how data gets into the collector. It accepts data in a specific format, translates it into the internal format, and passes it to processors and exporters defined in the pipeline. It can be pull- or push-based.
A processor is an optional component that is used to perform tasks such as batching, filtering, and transformations on data between being received and being exported.
An exporter is used to determine which destination to send the metrics, logs, or traces to. The collector architecture allows multiple instances of such pipelines to be set up via a Kubernetes YAML manifest.
The traces pipeline comprises an instance of AWS X-Ray Receiver and AWS X-Ray Exporter and sends traces to AWS X-Ray. The metrics pipeline comprises a Prometheus Receiver, a processor, and an AWS Prometheus Remote Write Exporter that sends metrics data to a workspace in Amazon Managed Service for Prometheus.
The following is a list of prerequisites needed before we can install the ADOT add-on.
1. An EKS cluster supporting Kubernetes version 1.19 or higher.
2. Certificate Manager, if not already installed in the cluster. Make sure that you are using version < 1.6.0 (1.6.0 is not supported).
3. Kubernetes RBAC permissions specifically for EKS add-ons to install the ADOT add-on in your cluster.
If you are new to creating the cluster feel free to check out our blog Creation of AWS EKS cluster from Scratch.
The second prerequisite is to install the Cert Manager. check out the official documentation
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.1/cert-manager.yaml
Install the ADOT add-on with the following command or enable the Add-on using GUI.
Command: aws eks create-addon --addon-name adot --addon-version v0.45.0-eksbuild.1 --cluster-name $CLUSTER_NAME
Command: aws eks create-addon --addon-name adot --addon-version v0.102.1-eksbuild.1 --cluster-name Techiev-first-cluster
To add the Add-on using GUI click the Add-ons options in the EKS dashboard and click Get more Add-ons, choose AWS Distro for OpenTelemetry(ADOT) click next, and create the add-on.
The version string must match the value of the addon Version field in the previously shown output. The output from a successful execution of this command looks as follows:
Wait until the add-on is in ACTIVE status before proceeding to the next step. The status of the add-on can be checked using the following command.
command: aws eks describe-addon --addon-name adot --cluster-name $CLUSTER_NAME
command: aws eks describe-addon --addon-name adot --cluster-name Techiev-first-cluster
Before creating the Add-on in open-telemetry, we need to create the IAM role with the below listed required policy to be in the attached state.
1. AmazonPrometheusRemoteWriteAccess -- for Amazon-managed Prometheus
2. CloudWatchAgentServerPolicy -- for CloudWatch insights
3. AWSXRayDaemonWriteAccess and AWSXrayWriteOnlyAccess -- for x-ray
We need to create the IAM role with the below custom trust policy for the Service account.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "oidc_provider_arn"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc_provider_arn:sub": "system:serviceaccount:opentelemetry-operator-system:adot-collector"
}
}
}
]
}
Here opentelemetry-operator-system is a namespace and adot-collector is a service account name.
Once the add-on is installed, we can able to see the namespace opentelemetry-operator-system.
After installing the add-on Opentelemetry-operator installed automatically.
Once the Role is created, we can use the Role ARN to update the servicesaccount creation yaml file.
After updating all the required fields, we can create the cluster role, role binding, servicesaccount, and open-telemetry collector for X-ray.
kubectl apply -f "manifestfile.yaml" example:
kubectl apply -f adot-clusterrole.yaml
kubectl apply -f adot-clusterrolebinding.yaml
kubectl apply -f adot-serviceaccount.yaml
kubectl apply -f opentelemetrycollector.yaml
kubectl apply -f collector-config-xray.yaml
In the adot-serviceaccount.yaml we need to pass the Newly created Role Arn. 
In the collector-config-xray.yaml we need to pass the exporter aws x-ray region
opentelemetrycollector.yaml manifest we need to add our clustername, acccountid, region and aws managed Prometheus workspace write endpoint.
If you are deploying the x-ray, we need to add the Otel_exporter endpoints, and if you are using any attributes you need to add those attributes as well. For more details check opentelemetry.sdk.environment_variables
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://my-collector-xray-collector:4317
- name: OTEL_RESOURCE_ATTRIBUTES
value: service.namespace=GettingStarted,service.name=GettingStartedService kubectl apply -f sample-app.yaml kubectl apply -f traffic-generator.yaml
Once the application and Kubernetes resources have been deployed, we can check the status by running the command below.
kubectl get all -n opentelemetry-operator-system Now we can able to see the traces inside the Aws cloudwatch x-ray. Reference Blog:
Metrics and traces collection using Amazon EKS add-ons for AWS Distro for OpenTelemetry
