#11185: Allow specifying a KMS key and tags for newly created AWS CloudWatch log groups.

[johannesfloriangeiger]

Summary

Implements the feature request #11185 by allowing users to specify a KMS key and tags for AWS CloudWatch log group sinks that are being used when creating new groups.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

• New integration test is provided in the PR.
• Manually
  • Get yourself an AWS environment.
  • Create a new KMS key and make sure that key can be used to encrypt CloudWatch log groups (for an example Key policy snippet see below).
  • Create a Vector config file as seen below, replace $KMS_KEY with the ARN of the key created in the previous step.
  • Run Vector: vector --config ./vector.yaml, see 3 new log groups being created: One without both custom KMS key and tags, one with only tags and one with both custom KMS key and tags.

Key policy that allows the usage in log groups in us-east-1:

{
  "Sid": "Allow use of the key for CloudWatch",
  "Effect": "Allow",
  "Principal": {
    "Service": "logs.us-east-1.amazonaws.com"
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

vector.yaml

sources:
  demo_logs:
    type: demo_logs
    format: json
sinks:
  cloudwatch_logs_without:
    type: aws_cloudwatch_logs
    inputs: [demo_logs]
    group_name: /without
    stream_name: demo-stream
    encoding:
      codec: json
  cloudwatch_logs_standard:
    type: aws_cloudwatch_logs
    inputs: [demo_logs]
    group_name: /standard
    stream_name: demo-stream
    encoding:
      codec: json
    tags:
      type: standard
  cloudwatch_logs_custom_kms_key:
    type: aws_cloudwatch_logs
    inputs: [demo_logs]
    group_name: /with-kms
    stream_name: demo-stream
    encoding:
      codec: json
    kms_key: $KMS_KEY
    tags:
      type: kms-key

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

• Please read our Vector contributor resources.
  • make check-all is a good command to run locally. This check is
    defined here. Some of these
    checks might not be relevant to your PR. For Rust changes, at the very least you should run:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

• Closes: Allow setting KMS key id and tags when creating a CloudWatch log group #11185

[bits-bot]

All committers have signed the CLA.

[pront]

Otherwise, this looks good and thank you for adding tests.

[pront]

Thanks @johannesfloriangeiger! Please add rustdocs and then make generate-components-docs.

[johannesfloriangeiger]

Done!

[neko-dd]

Is "ARN" a thing that will be obvious to all readers? If not, spell it out in parentheses after you mention it.

[johannesfloriangeiger]

Yes, the acronym is already used in the same file (L41) and is a widely used terminology in AWS context but I’ll add a non abbreviated version.

[pront]

Can you also do a git merge origin master to pick up the latest Cargo.lock? 🙏

[pront]

The docs team left a comment. Otherwise, LGTM