aws Distro for OpenTelemetry 可观测性workshop记录

参考资料

• https://aws-otel.github.io/docs/introduction
• https://aws-otel.github.io/docs/introduction

aws distro for opentelemetry 官方提供了不同语言不同使用场景下完善的使用实例和相关配置。

AWS Distro for OpenTelemetrics 由以下部分组成，用于向后端服务发送数据

• SDK
• Auto-instrumentation agent，自动注入代理
• OpenTelemetry Collector，收集器

使用opentelemetry的一般架构如以下ecs示例。将opentelemetry collector作为sidecar运行。

opentelemetry基础

所有的opentelemery关注点，比如跟踪和度量，都共享一个底层 Context 机制，用于在分布式事务的整个生命周期中存储状态和访问数据

• trace，通过span隐式定义trace，trace是span组成的有向无环图

• span，span代表事务中的一个操作，每个span封装以下状态：

  • 事件，每个事件都是元组（时间戳，名称，属性）

    • 操作名称
    • 开始和结束的时间戳
    • 属性（键值对）

  • span之间的因果关系链接

Http追踪

服务端追踪

使用gorilla/mux HTTP 多路复用器。通过opentelemetry库包装 HTTP 处理程序。对这些端点的所有调用都将根据现有的采样规则自动跟踪并发送到 AWS X-Ray

import(go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmuxgithub.com/gorilla/mux...
)
...
func MakeHttpHandler(s Service, Logger log.logger) http.Handler{r := mux.NewRouter()r.Use(otelmux.Middleware("petlistadoptions"))r.Methods("GET").Path("/api/adoptionlist/").Handler(handlerFunc)return r
}

客户端追踪

使用go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp封装客户端的http调用

client := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)

对于没有提供自动追踪检测支持的库调用，可以通过span和go context创建自定义追踪

tracer := otel.GetTracerProvider().Tracer("petlistadoptions")
_, span := tracer.Start(ctx, "PGSQL Query", trace.WithSpanKind(trace.SpanKindClient))sql := `SELECT pet_id, transaction_id, adoption_date FROM transactions ORDER BY id DESC LIMIT 25`// injecting custom attributes
span.SetAttributes(label.String("sql", sql),label.String("url", r.safeConnStr),
)rows, err := r.db.Query(sql)
if err != nil {handleErr(err)
}
span.End()

java追踪代理

https://github.com/aws-samples/one-observability-demo

使用java代理并修改JVM启动参数，使得自动台历能够收集追踪信息

ADD https://github.com/aws-observability/aws-otel-java-instrumentation/releases/download/version/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar
ENV JAVA_TOOL_OPTIONS "-javaagent:/app/aws-opentelemetry-agent.jar"

其他的opentelemetry配置

# OpenTelemetry agent configuration
ENV OTEL_TRACES_SAMPLER "always_on"
ENV OTEL_PROPAGATORS "tracecontext,baggage,xray"
ENV OTEL_RESOURCE_ATTRIBUTES "service.name=PetSearch" # 资源属性
ENV OTEL_IMR_EXPORT_INTERVAL "10000" # 将OTLP指标导出到collector的频率
ENV OTEL_EXPORTER_OTLP_ENDPOINT "http://localhost:4317"	 # collector监听断电

像aws sdk和流行框架都得到了opentelemetry的支持，对于java来说任何经由spring MVC的请求和对aws服务的调用都会被追踪和发布。

lambda追踪

lambda提供了专门的ADTO lambda layer，自动检测lambda函数，使用步骤如下

• 添加ADOT lambda层

• 在lambda函数中添加环境变量AWS_LAMBDA_EXEC_WRAPPER = /opt/otel-instrument

• 在lambda函数中启动Active Tracing

示例cdk代码如下

var adotLayerArn = "arn:aws:lambda:"+ process.env.CDK_DEFAULT_REGION + ":901920570463:layer:aws-otel-python-amd64-ver-1-11-1:2"
var adotlayer = lambda.LayerVersion.fromLayerVersionArn(this,'otelPythonLambdaLayer',adotLayerArn);
var layers: lambda.ILayerVersion[] = [adotlayer]// lambda自动追踪请求，业务代码中没有相关逻辑
new pythonlambda.PythonFunction(this, lambdaFileName, {entry: './resources/stepfn_lambdas/',index: lambdaFileName + '.py',handler: 'lambda_handler',memorySize: 128,runtime: lambda.Runtime.PYTHON_3_9,role: lambdaRole,layers: lambdalayers, //加入otel层tracing: Tracing.ACTIVE // 启用主动追踪});
pythonFn.addEnvironment("AWS_LAMBDA_EXEC_WRAPPER", "/opt/otel-instrument") //添加环境变量

python示例

https://github.com/aws-samples/one-observability-demo/tree/main/PetAdoptions/petadoptionshistory-py

https://catalog.workshops.aws/observability/en-US/aws-managed-oss/adot/python-instrumentation/tracing

手动追踪

配置环境

pip3 install opentelemetry-api
pip3 install opentelemetry-sdk
pip3 install opentelemetry-distro
pip3 install opentelemetry-exporter-otlp-proto-grpc
pip3 install opentelemetry-sdk-extension-aws
pip3 install opentelemetry-propagator-aws-xray

导入依赖

# OTLP Tracing
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource, get_aggregated_resources# Exporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter# Propagation
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.aws import AwsXRayPropagator# AWS X-Ray ID Generator
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator# Resource detector
from opentelemetry.sdk.extension.aws.resource.eks import AwsEksResourceDetector

手动追踪，最终去哦那个球

# Setup AWX X-Ray Propagator
# 跨 AWS 服务注入或提取跟踪上下文
set_global_textmap(AwsXRayPropagator())# Setup AWS EKS resource detector 
# 从 EKS 环境中检测到的额外资源字段(例如集群名称和容器 ID)添加到单个跟踪中
resource = get_aggregated_resources([AwsEksResourceDetector(), ]
)# Setup tracer provider with the X-Ray ID generator
tracer_provider = TracerProvider(resource=resource, id_generator=AwsXRayIdGenerator())
processor = BatchSpanProcessor(OTLPSpanExporter())
tracer_provider.add_span_processor(processor)# Sets the global default tracer provider
trace.set_tracer_provider(tracer_provider)# Creates a tracer from the global tracer provider
tracer = trace.get_tracer(__name__)

在flask中手动创建span追踪，注意start_as_current_span

@app.route('/api/home/transactions', methods=['GET'])
def transactions_get():with tracer.start_as_current_span("transactions_get") as transactions_span:transactions_get_counter.add(1)transactions = repository.list_transaction_history(db)return jsonify(transactions)

自动追踪

自动检测 boto3和 psycopg2之类的库调用来生成跟踪

pip3 install opentelemetry-instrumentation-botocore
pip3 install opentelemetry-instrumentation-flask

封装调用

# Instrumentation
from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor# Instrumentation
BotocoreInstrumentor().instrument()# Setup flask app
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
...