How to handle long log lines

[trahim]

Hi, I'm having issues using the file source and having my logs being truncated when they are longer log lines and subsequently having issues with parsing it. For context my message is essentially just a json string.

Below is my config for my source and my transform I am using.

    sources:
      tail_logs:
        data_dir: /vector-data-dir
        exclude:
        - /var/log/containers/vector-agent*.log
        - /var/log/containers/vector-aggregator*.log
        fingerprint:
          strategy: device_and_inode
        ignore_older_secs: 86400
        include:
        - /var/log/containers/*.log
        type: file
        max_line_bytes: 1024000
    transforms:
      content_logging_only:
        condition:
          source: |-
            remove_stdout, err = split(.message, r'stdout [a-zA-Z] ')[1]
            parsed_json, err = parse_json(remove_stdout)
            if err != null { log(remove_stdout, level: "error") }
            parsed_json.logType == "Special.Logger"
          type: vrl
        inputs:
        - tail_logs
        type: filter

In the log I've put in I can see the log get split into 2 in the agent logs.

Any ideas what I need to do? I couldn't find appropriate docs for this

[jszwedko]

Is it possible that the input log is split over multiple lines? Are you able to find it in the .log files?

[trahim]

So I've carried on playing with this sorted out issues with reduce for anyone who may look at this in the future looking for advice I've attached my updated config.

    sources:
      tail_logs:
        data_dir: /vector-data-dir
        exclude:
        - /var/log/containers/vector-agent*.log
        - /var/log/containers/vector-aggregator*.log
        fingerprint:
          strategy: device_and_inode
        ignore_older_secs: 86400
        include:
        - /var/log/containers/*.log
        type: file
        max_line_bytes: 1024000
    transforms:
      combine_long_lines:
        type: reduce
        group_by:
        - file
        inputs:
        - tail_logs
        merge_strategies:
          message: concat_raw
        starts_when:
            type: vrl
            source: match(string!(.message), r'^[1-2]\d{3}-(0[1-9]|1[0-2])-(3[0-1]|[1-2]\d|0[1-9])T(0\d|[0-2][0-9])(:[0-5]\d){2}.\d{9}Z (stdout|stderr) [a-zA-Z] \{"@timestamp":')
      content_logging_only:
        condition:
          source: |-
            remove_stdout, err = split(.message, r'stdout [a-zA-Z] ')[1]
            parsed_json, err = parse_json(remove_stdout)
            if err != null { log(remove_stdout, level: "error") }
            parsed_json.logType == "Special.Logger"
          type: vrl
        inputs:
        - tail_logs
        type: filter

But I now seem to have a different issue when I am trying to parse this and split it to work with the JSON string(strip out the timestamp and stdout/stderr from the message) it seems to truncate it again. Is there a size limit to strings? Or is there a better way to strip that out so I can parse the message json string?

This is my additional transformation

parse:                                                                                                                                                                                                                                         
  type: remap                                                                                                                                                                                                                                  
  inputs:                                                                                                                                                                                                                                      
  - content_logging_only                                                                                                                                                                                                                       
  source: |-                                                                                                                                                                                                                                     
    .message = split!(.message, r'(stdout|stderr) [a-zA-Z] ')[1]

[jszwedko]

I don't think there are any internal limits to string length. The only limit that should be applied here is the max_line_bytes limit in the source.

However, I think that:

.message = split!(.message, r'(stdout|stderr) [a-zA-Z] ')[1]

will result in you only getting the first segment of the joined lines (the [1] part). Instead I think you should consider using something like:

.message = replace!(.message, r'(stdout|stderr) [a-zA-Z] ', '')

to replace the line prefixes with an empty string to strip them. Let me know if that doesn't make sense.

[trahim]

Answering my own question but this is was the reduce was adding the stdout message again when concating, I'll look to see how to remove that

[jszwedko]

Answering my own question but this is was the reduce was adding the stdout message again when concating, I'll look to see how to remove that

Right, yeah, reduce will just concat the lines so any prefix will be repeated. You could also consider stripping the prefixes before reducing.

[trahim]

Hey yup I moved to replace which works well for removing the timestamp and stdout. Had some issues with starts_when where it was adding additional messages which matched the same pattern. I've moved to ends_when which seems to work better but am still seeing some occurrences of additional messages being appended. Not sure yet why but it does cause issues with parsing as it's not valid json. Are there any gotchas to be aware of?