[BUG] add_entries processor add_when is not working when checking for fields within log.attributes or resource.attributes

[alamzeeshan]

Describe the bug
The add_entries processor add_when is not working when checking for fields inside log.attributes or resource.attributes. It is working work when I check for a top level field such as severityText

I just setup data prepper docker container as mentioned in here

docker run --name data-prepper -p 2021:2021 -v /${PWD}/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml opensearchproject/data-prepper:latest

This is my data prepper config:

otel-opensearch-pipeline:
  workers: 1
  delay: "5000"
  source:
    otel_logs_source:
      ssl: false
      port: 2021
      #compression: gzip
  processor:
    - add_entries:
        entries:
        - key: "add_entry_test"
          value: "done"
          add_when: /severityText == "Info"
        - key: "add_entry_dot_test"
          value: done
          add_when: /log.attributes.foo == "bar"    

    - date:
        from_time_received: true
        destination: "@timestamp"

  sink:
    - stdout:

Also in the same machine I have setup opentelemetry-collector-contrib:0.116.1 to receive OTEL logs and forward to data prepper

docker run -p 4317:4317 -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml otel/opentelemetry-collector-contrib:0.116.1

  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  otlp:
    endpoint: "myhost:2021"
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

I am using Telemetry generator for OpenTelemetry to generate OTEL logs

./telemetrygen logs --body "2024-12-12:00:00:00 INFO This is a test message" --otlp-attributes host.name=\"mydevhost\" --telemetry-attributes foo=\"bar\" --trace-id ae87dadd90e9935a4bc9660628efd569 --span-id 5828fa4960140870 --duration 1s --otlp-insecure

OUTPUT

{
    "traceId": "ae87dadd90e9935a4bc9660628efd569",
    "spanId": "5828fa4960140870",
    "severityText": "Info",
    "flags": 0,
    "time": "2024-12-27T10:01:38.195840064Z",
    "severityNumber": 9,
    "droppedAttributesCount": 0,
    "serviceName": null,
    "body": "2024-12-12:00:00:00 INFO This is a test message",
    "observedTime": "1970-01-01T00:00:00Z",
    "schemaUrl": "https://opentelemetry.io/schemas/1.4.0",
    "add_entry_test": "done",
    "@timestamp": "2024-12-27T10:01:38.330Z",
    "log.attributes.app": "server",
    "log.attributes.foo": "bar",
    "resource.attributes.host@name": "mydevhost"
}

Expected behavior
I am expecting a new field add_entry_dot_test to be added based on the condition add_when: /log.attributes.foo == "bar". I am getting new field add_entry_test based on the condition add_when: /severityText == "Info". Looks like data prepper expression-syntax is unable to read fields inside log.attributes or resource.attributes.

Environment (please complete the following information):

• OS: [Ubuntu 22.04.5 LTS]
• Docker [27.3.1]
• Data Prepper [Latest]

[alamzeeshan]

Even when I use add_entries to create an entry if it doesn't exists. It is working as expected for top level field severityText and it is not getting overwritten. But the log.attributes.foo is getting overwritten even though my incoming event has the field log.attributes.foo=bar

./telemetrygen logs --body "2024-12-12:00:00:00 INFO This is a test message" --otlp-attributes host.name=\"mydevhost\" --telemetry-attributes foo=\"bar\" --trace-id ae87dadd90e9935a4bc9660628efd569 --span-id 5828fa4960140870 --duration 1s --otlp-insecure

otel-opensearch-pipeline:
  workers: 1
  delay: "5000"
  source:
    otel_logs_source:
      ssl: false
      port: 2021
      #compression: gzip

  processor:
    - add_entries:
        entries:
        - key: "severityText"
          value: "ERROR"
        - key: "log.attributes.foo"
          value: "baaz"    

    - date:
        from_time_received: true
        destination: "@timestamp"

  sink:
    - stdout:

OUTPUT

{
    "traceId": "ae87dadd90e9935a4bc9660628efd569",
    "spanId": "5828fa4960140870",
    "severityText": "Info",
    "flags": 0,
    "time": "2024-12-27T10:55:48.134602349Z",
    "severityNumber": 9,
    "droppedAttributesCount": 0,
    "serviceName": null,
    "body": "2024-12-12:00:00:00 INFO This is a test message",
    "observedTime": "1970-01-01T00:00:00Z",
    "schemaUrl": "https://opentelemetry.io/schemas/1.4.0",
    "log.attributes.foo": "baaz",
    "@timestamp": "2024-12-27T10:55:48.152Z",
    "log.attributes.app": "server",
    "resource.attributes.host@name": "mydevhost"
}

[timbastin]

I have the same issue when using the renaming keys processor

[LVMalekBe]

Also having the same issue with 'parse_json', 'key_value' and 'rename_keys' processors. it seems that they are ignoring fields with dots in them like "span.attributes.client_id".

[graytaylor0]

When I use the file source with this input and this configuration, I get the expected outcome

Config

version: "2"
test-pipeline:
  source:
    file:
      path: "/path/to/log.txt"
      format: "json"
      record_type: "event"
  processor:
    - add_entries:
        entries:
          - key: "my_key"
            value: 10
            add_when: '/log.attribute.foo == "bar"'
  sink:
     - stdout:   

Content of log.txt

{ "log.attribute.foo": "bar" }

Output to stdout

{"log.attribute.foo":"bar","my_key":10}

This makes it seem like the sources are manipulating the data differently and the issue is not necessarily with the . in the key name

[san81]

I can help

[alamzeeshan]

I was able to make this work. The otel_logs_source is flatting the attributes and nesting it under attributes key. So in order to make this work I had to use attributes/log.attributes.foo instead of just log.attributes.foo which we are getting as output.
If you want to make use of log, resource or trace attributes within processor you should be prepending the fields with attributes/. This behavior is seen only with the OTEL sources.

[LVMalekBe]

Thank you @alamzeeshan for the info.
I confirm that the below processor worked fine with your solution

  processor:
    - rename_keys:
        entries:
        - from_key: "attributes/span.attributes.app_id"
          to_key: "app_id"
          overwrite_if_to_key_exists: true

We definitely should add this to the docs