From 99592334a22ad0c245e021d9e181950bb92fdad2 Mon Sep 17 00:00:00 2001 From: Yilia Date: Wed, 8 Jan 2025 15:05:31 +0800 Subject: [PATCH] Revert "Update opentelemetry.md" This reverts commit fca1c33acafc89c5d747b5267bdb7e303d0ec835. --- docs/en/latest/plugins/opentelemetry.md | 263 +++++++++++------------- 1 file changed, 119 insertions(+), 144 deletions(-) diff --git a/docs/en/latest/plugins/opentelemetry.md b/docs/en/latest/plugins/opentelemetry.md index cd9c894390ed..09a17a82da07 100644 --- a/docs/en/latest/plugins/opentelemetry.md +++ b/docs/en/latest/plugins/opentelemetry.md @@ -5,8 +5,7 @@ keywords: - API Gateway - Plugin - OpenTelemetry -description: The `opentelemetry` Plugin instruments APISIX and sends traces to OpenTelemetry collector based on the OpenTelemetry specification, in binary-encoded OLTP over HTTP. - +description: This document contains information about the Apache opentelemetry Plugin. --- - - - - ## Description -The `opentelemetry` Plugin instruments APISIX and sends traces to OpenTelemetry collector based on the [OpenTelemetry specification](https://opentelemetry.io/docs/reference/specification/), in binary-encoded [OLTP over HTTP](https://opentelemetry.io/docs/reference/specification/protocol/otlp/#otlphttp). +The `opentelemetry` Plugin can be used to report tracing data according to the [OpenTelemetry specification](https://opentelemetry.io/docs/reference/specification/). + +The Plugin only supports binary-encoded [OLTP over HTTP](https://opentelemetry.io/docs/reference/specification/protocol/otlp/#otlphttp). + +## Attributes + +| Name | Type | Required | Default | Valid values | Description | +|---------------------------------------|---------------|----------|-------------------------------------------------|--------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| sampler | object | False | | | Sampling configuration. | +| sampler.name | string | False | always_off | ["always_on", "always_off", "trace_id_ratio", "parent_base"] | Sampling strategy. `always_on`: always samples, `always_off`: never samples, `trace_id_ratio`: random sampling result based on given sampling probability, `parent_base`: use parent decision if given, else determined by the root sampler. | +| sampler.options | object | False | | {fraction = 0, root = {name = "always_off"}} | Parameters for sampling strategy. | +| sampler.options.fraction | number | False | 0 | [0, 1] | Sampling probability for `trace_id_ratio`. | +| sampler.options.root | object | False | {name = "always_off", options = {fraction = 0}} | | Root sampler for `parent_base` strategy. | +| sampler.options.root.name | string | False | always_off | ["always_on", "always_off", "trace_id_ratio"] | Root sampling strategy. | +| sampler.options.root.options | object | False | {fraction = 0} | | Root sampling strategy parameters. | +| sampler.options.root.options.fraction | number | False | 0 | [0, 1] | Root sampling probability for `trace_id_ratio`. | +| additional_attributes | array[string] | False | | | Additional attributes appended to the trace span. Support built-in NGINX or APISIX variables in values, such as `http_header` or `route_id`. | +| additional_header_prefix_attributes | array[string] | False | | | Headers or header prefixes appended to the trace span's attributes. For example, use `x-my-header"` or `x-my-headers-*` to include all headers with the prefix `x-my-headers-`. | + +### Configuring the collector -## Static Configurations +You can set up the collector by configuring it in you configuration file (`conf/config.yaml`): -By default, configurations of the Service name, tenant ID, collector, and batch span processor are pre-configured in [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua). +| Name | Type | Default | Description | +|--------------------------------------------|---------|---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| trace_id_source | enum | x-request-id | Source of the trace ID. Valid values are `random` or `x-request-id`. When set to `x-request-id`, the value of the `x-request-id` header will be used as trace ID. Make sure that it matches the regex pattern `[0-9a-f]{32}`. | +| resource | object | | Additional [resource](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md) appended to the trace. | +| collector | object | {address = "127.0.0.1:4318", request_timeout = 3} | OpenTelemetry Collector configuration. | +| collector.address | string | 127.0.0.1:4318 | Collector address. If the collector serves on https, use https://127.0.0.1:4318 as the address. | +| collector.request_timeout | integer | 3 | Report request timeout in seconds. | +| collector.request_headers | object | | Report request HTTP headers. | +| batch_span_processor | object | | Trace span processor. | +| batch_span_processor.drop_on_queue_full | boolean | false | When set to `true`, drops the span when queue is full. Otherwise, force process batches. | +| batch_span_processor.max_queue_size | integer | 1024 | Maximum queue size for buffering spans for delayed processing. | +| batch_span_processor.batch_timeout | number | 2 | Maximum time in seconds for constructing a batch. | +| batch_span_processor.max_export_batch_size | integer | 16 | Maximum number of spans to process in a single batch. | +| batch_span_processor.inactive_timeout | number | 1 | Time interval in seconds between processing batches. | -To customize these values, add the corresponding configurations to `config.yaml`. For example: +:::note -```yaml +If you find a `bad argument #1 to '?' (invalid value)` error triggered by the `hex2bytes` function in error log, it's essential to verify if your traceId matches the specified regex pattern `[0-9a-f]{32}`, as required by opentelemetry's [traceId format](https://opentelemetry.io/docs/specs/otel/trace/api/#retrieving-the-traceid-and-spanid). + +For instance, a possible scenario occurs when the plugin attribute `trace_id_source` is configured as `x-request-id`, and requests include an x-request-id header generated by Envoy. Envoy typically uses a [UUID](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/tracing#trace-context-propagation) to create this header by default. When the opentelemetry plugin adopts this UUID as the traceId, the presence of hyphens in the UUID can cause issues. Since the UUID format with hyphens does not comply with the expected traceId format, it results in errors when attempting to push traces to the collector. + +::: + +You can configure these as shown below in your configuration file (`conf/config.yaml`): + +```yaml title="conf/config.yaml" plugin_attr: opentelemetry: - trace_id_source: x-request-id # Specify the source of the trace ID, `x-request-id` or `random`. When set to `x-request-id`, - # the value of the `x-request-id` header will be used as the trace ID. - resource: # Additional resource to append to the trace. - service.name: APISIX # Set the Service name for OpenTelemetry traces. + resource: + service.name: APISIX + tenant.id: business_id collector: - address: 127.0.0.1:4318 # Set the address of the OpenTelemetry collector to send traces to. - request_timeout: 3 # Set the timeout for requests to the OpenTelemetry collector in seconds. - request_headers: # Set the headers to include in requests to the OpenTelemetry collector. - Authorization: token # Set the authorization header to include an access token. - batch_span_processor: # Trace span processor. - drop_on_queue_full: false # Drop spans when the export queue is full. - max_queue_size: 1024 # Set the maximum size of the span export queue. - batch_timeout: 2 # Set the timeout for span batches to wait in the export queue before - # being sent. - inactive_timeout: 1 # Set the timeout for spans to wait in the export queue before being sent, - # if the queue is not full. - max_export_batch_size: 16 # Set the maximum number of spans to include in each batch sent to the OpenTelemetry collector. - set_ngx_var: false # Export opentelemetry variables to nginx variables. + address: 192.168.8.211:4318 + request_timeout: 3 + request_headers: + foo: bar + batch_span_processor: + drop_on_queue_full: false + max_queue_size: 6 + batch_timeout: 2 + inactive_timeout: 1 + max_export_batch_size: 2 ``` -Reload APISIX for changes to take effect. +## Variables -## Attributes +The following nginx variables are set by OpenTelemetry: -| Name | Type | Required | Default | Valid Values | Description | -|---------------------------------------|---------------|----------|--------------|--------------|-------------| -| sampler | object | False | - | - | Sampling configuration. | -| sampler.name | string | False | `always_off` | `always_on`, `always_off`, `trace_id_ratio`, or `parent_base` | Sampling strategy.
To always sample, use `always_on`.
To never sample, use `always_off`.
To randomly sample based on a given ratio, use `trace_id_ratio`.
To use the sampling decision of the span's parent, use `parent_base`. If there is no parent, use the root sampler. | -| sampler.options | object | False | - | - | Parameters for sampling strategy. | -| sampler.options.fraction | number | False | 0 | [0, 1] | Sampling ratio when the sampling strategy is `trace_id_ratio`. | -| sampler.options.root | object | False | - | - | Root sampler when the sampling strategy is `parent_base` strategy. | -| sampler.options.root.name | string | False | - | `always_on`, `always_off`, or `trace_id_ratio` | Root sampling strategy. | -| sampler.options.root.options | object | False | - | - | Root sampling strategy parameters. | -| sampler.options.root.options.fraction | number | False | 0 | [0, 1] | Root sampling ratio when the sampling strategy is `trace_id_ratio`. | -| additional_attributes | array[string] | False | - | - | Additional attributes appended to the trace span. Support [built-in variables](https://apisix.apache.org/docs/apisix/apisix-variable/) in values. | -| additional_header_prefix_attributes | array[string] | False | - | - | Headers or header prefixes appended to the trace span's attributes. For example, use `x-my-header"` or `x-my-headers-*` to include all headers with the prefix `x-my-headers-`. | +- `opentelemetry_context_traceparent` - [W3C trace context](https://www.w3.org/TR/trace-context/#trace-context-http-headers-format), e.g.: `00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01` +- `opentelemetry_trace_id` - Trace Id of the current span +- `opentelemetry_span_id` - Span Id of the current span -## Examples +How to use variables? you have to add it to your configuration file (`conf/config.yaml`): -The examples below demonstrate how you can work with the `opentelemetry` Plugin for different scenarios. +```yaml title="conf/config.yaml" +http: + enable_access_log: true + access_log: "/dev/stdout" + access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr","uri": "$uri"}' + access_log_format_escape: json +plugins: + - opentelemetry +plugin_attr: + opentelemetry: + set_ngx_var: true +``` -### Enable `opentelemetry` Plugin +## Enable Plugin -By default, the `opentelemetry` Plugin is disabled in APISIX. To enable, add the Plugin to your configuration file as such: +To enable the Plugin, you have to add it to your configuration file (`conf/config.yaml`): -```yaml title="config.yaml" +```yaml title="conf/config.yaml" plugins: - ... - opentelemetry ``` -Reload APISIX for changes to take effect. - -See [static configurations](#static-configurations) for other available options you can configure in `config.yaml`. +Now, you can enable the Plugin on a specific Route: -### Send Traces to OpenTelemetry +:::note +You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command: -The following example demonstrates how to trace requests to a Route and send traces to OpenTelemetry. - -Start an OpenTelemetry collector instance in Docker: - -```shell -docker run -d --name otel-collector -p 4318:4318 otel/opentelemetry-collector-contrib +```bash +admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g') ``` -Create a Route with `opentelemetry` Plugin: +::: ```shell -curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ - -H "X-API-KEY: ${admin_key}" \ - -d '{ - "id": "otel-tracing-route", - "uri": "/anything", +curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d ' +{ + "methods": ["GET"], + "uris": [ + "/uid/*" + ], "plugins": { - "opentelemetry": { - "sampler": { - "name": "always_on" + "opentelemetry": { + "sampler": { + "name": "always_on" + } } - } }, "upstream": { - "type": "roundrobin", - "nodes": { - "httpbin.org": 1 - } + "type": "roundrobin", + "nodes": { + "127.0.0.1:1980": 1 + } } - }' -``` - -Send a request to the Route: - -```shell -curl "http://127.0.0.1:9080/anything" -``` - -You should receive an `HTTP/1.1 200 OK` response. - -In OpenTelemetry collector's log, you should see information similar to the following: - -```text -2024-02-18T17:14:03.825Z info ResourceSpans #0 -Resource SchemaURL: -Resource attributes: - -> telemetry.sdk.language: Str(lua) - -> telemetry.sdk.name: Str(opentelemetry-lua) - -> telemetry.sdk.version: Str(0.1.1) - -> hostname: Str(e34673e24631) - -> service.name: Str(APISIX) -ScopeSpans #0 -ScopeSpans SchemaURL: -InstrumentationScope opentelemetry-lua -Span #0 - Trace ID : fbd0a38d4ea4a128ff1a688197bc58b0 - Parent ID : - ID : af3dc7642104748a - Name : GET /anything - Kind : Server - Start time : 2024-02-18 17:14:03.763244032 +0000 UTC - End time : 2024-02-18 17:14:03.920229888 +0000 UTC - Status code : Unset - Status message : -Attributes: - -> net.host.name: Str(127.0.0.1) - -> http.method: Str(GET) - -> http.scheme: Str(http) - -> http.target: Str(/anything) - -> http.user_agent: Str(curl/7.64.1) - -> apisix.route_id: Str(otel-tracing-route) - -> apisix.route_name: Empty() - -> http.route: Str(/anything) - -> http.status_code: Int(200) -{"kind": "exporter", "data_type": "traces", "name": "debug"} -``` - -To visualize these traces, you can export your telemetry to backend Services, such as Zipkin and Prometheus. See [exporters](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter) for more details. - -### Using Trace Variables in Logging - -The following example demonstrates how to configure the `opentelemetry` Plugin to set the following built-in variables, which can be used in logger Plugins or access logs: - -- `opentelemetry_context_traceparent`: [trace parent](https://www.w3.org/TR/trace-context/#trace-context-http-headers-format) ID -- `opentelemetry_trace_id`: trace ID of the current span -- `opentelemetry_span_id`: span ID of the current span - -Update the configuration file as below. You should customize the access log format to use the `opentelemetry` Plugin variables, and set `opentelemetry` variables in the `set_ngx_var` field. - -```yaml title="conf/config.yaml" -nginx_config: - http: - enable_access_log: true - access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}' - access_log_format_escape: json -plugin_attr: - opentelemetry: - set_ngx_var: true +}' ``` -Reload APISIX for configuration changes to take effect. +## Delete Plugin -You should see access log entries similar to the following when you generate requests: +To remove the `opentelemetry` Plugin, you can delete the corresponding JSON configuration from the Plugin configuration. APISIX will automatically reload and you do not have to restart for this to take effect. -```text -{"time": "18/Feb/2024:15:09:00 +0000","opentelemetry_context_traceparent": "00-fbd0a38d4ea4a128ff1a688197bc58b0-8f4b9d9970a02629-01","opentelemetry_trace_id": "fbd0a38d4ea4a128ff1a688197bc58b0","opentelemetry_span_id": "af3dc7642104748a","remote_addr": "172.10.0.1"} +```shell +curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d ' +{ + "methods": ["GET"], + "uris": [ + "/uid/*" + ], + "plugins": { + }, + "upstream": { + "type": "roundrobin", + "nodes": { + "127.0.0.1:1980": 1 + } + } +}' ```