Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to customize fluent bit config via values.yaml #43

Merged
merged 1 commit into from
May 22, 2024
Merged

Add ability to customize fluent bit config via values.yaml #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 9 additions & 190 deletions charts/amazon-cloudwatch-observability/templates/linux/fluent-bit-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,195 +8,14 @@ metadata:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Grace 30
Log_Level error
Daemon off
Parsers_File parsers.conf
storage.path /var/fluent-bit/state/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M

@INCLUDE application-log.conf
@INCLUDE dataplane-log.conf
@INCLUDE host-log.conf

application-log.conf: |
[INPUT]
Name tail
Tag application.*
Exclude_Path /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
Path /var/log/containers/*.log
multiline.parser docker, cri
DB /var/fluent-bit/state/flb_container.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Read_from_Head ${READ_FROM_HEAD}

[INPUT]
Name tail
Tag application.*
Path /var/log/containers/fluent-bit*
multiline.parser docker, cri
DB /var/fluent-bit/state/flb_log.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[INPUT]
Name tail
Tag application.*
Path /var/log/containers/cloudwatch-agent*
multiline.parser docker, cri
DB /var/fluent-bit/state/flb_cwagent.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[FILTER]
Name kubernetes
Match application.*
Kube_URL https://kubernetes.default.svc:443
Kube_Tag_Prefix application.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Labels Off
Annotations Off
Use_Kubelet On
Kubelet_Port 10250
Buffer_Size 0

[OUTPUT]
Name cloudwatch_logs
Match application.*
region ${AWS_REGION}
log_group_name /aws/containerinsights/${CLUSTER_NAME}/application
log_stream_prefix ${HOST_NAME}-
auto_create_group true
extra_user_agent container-insights

dataplane-log.conf: |
[INPUT]
Name systemd
Tag dataplane.systemd.*
Systemd_Filter _SYSTEMD_UNIT=docker.service
Systemd_Filter _SYSTEMD_UNIT=containerd.service
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
DB /var/fluent-bit/state/systemd.db
Path /var/log/journal
Read_From_Tail ${READ_FROM_TAIL}

[INPUT]
Name tail
Tag dataplane.tail.*
Path /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
multiline.parser docker, cri
DB /var/fluent-bit/state/flb_dataplane_tail.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Read_from_Head ${READ_FROM_HEAD}

[FILTER]
Name modify
Match dataplane.systemd.*
Rename _HOSTNAME hostname
Rename _SYSTEMD_UNIT systemd_unit
Rename MESSAGE message
Remove_regex ^((?!hostname|systemd_unit|message).)*$

[FILTER]
Name aws
Match dataplane.*
imds_version v2

[OUTPUT]
Name cloudwatch_logs
Match dataplane.*
region ${AWS_REGION}
log_group_name /aws/containerinsights/${CLUSTER_NAME}/dataplane
log_stream_prefix ${HOST_NAME}-
auto_create_group true
extra_user_agent container-insights

host-log.conf: |
[INPUT]
Name tail
Tag host.dmesg
Path /var/log/dmesg
Key message
DB /var/fluent-bit/state/flb_dmesg.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[INPUT]
Name tail
Tag host.messages
Path /var/log/messages
Parser syslog
DB /var/fluent-bit/state/flb_messages.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[INPUT]
Name tail
Tag host.secure
Path /var/log/secure
Parser syslog
DB /var/fluent-bit/state/flb_secure.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[FILTER]
Name aws
Match host.*
imds_version v2

[OUTPUT]
Name cloudwatch_logs
Match host.*
region ${AWS_REGION}
log_group_name /aws/containerinsights/${CLUSTER_NAME}/host
log_stream_prefix ${HOST_NAME}.
auto_create_group true
extra_user_agent container-insights

{{- .Values.containerLogs.fluentBit.config.service | nindent 4 }}
{{- range $key, $val := .Values.containerLogs.fluentBit.config.extraFiles }}
@INCLUDE {{ $key }}
{{- end }}
parsers.conf: |
[PARSER]
Name syslog
Format regex
Regex ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S

[PARSER]
Name container_firstline
Format regex
Regex (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
Name cwagent_firstline
Format regex
Regex (?<log>(?<="log":")\d{4}[\/-]\d{1,2}[\/-]\d{1,2}[ T]\d{2}:\d{2}:\d{2}(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
{{- .Values.containerLogs.fluentBit.config.customParsers | nindent 4 }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason we split up the parsers and the fluentbit conf specifically? Seems like there's a lot of in-between configs too such as application-log, host-log, etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to keep it close to the actual fluent-bit helm chart so that it looks familiar for users already using that.
https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml#L367

Hence Ive exposed the service, customParsers and extraFiles as top level attributes. The application-log, host-log etc are specific to the way we have prescribed these files in our public docs in the past. We typically dont have root level inputs, filters and outputs in the main fluent-bit.conf and always take them in as additional/extra files.

{{- range $key, $val := .Values.containerLogs.fluentBit.config.extraFiles }}
{{ $key }}: |
{{- (tpl $val $) | nindent 4 }}
{{- end -}}
{{- end -}}
147 changes: 9 additions & 138 deletions charts/amazon-cloudwatch-observability/templates/windows/fluent-bit-windows-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,143 +8,14 @@ metadata:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Log_Level error
Daemon off
net.dns.resolver LEGACY
Parsers_File parsers.conf

@INCLUDE application-log.conf
@INCLUDE dataplane-log.conf
@INCLUDE host-log.conf

application-log.conf: |
[INPUT]
Name tail
Tag application.*
Exclude_Path C:\\var\\log\\containers\\fluent-bit*, C:\\var\\log\\containers\\cloudwatch-agent*
Path C:\\var\\log\\containers\\*.log
Parser docker
DB C:\\var\\fluent-bit\\state\\flb_container.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Rotate_Wait 30
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[INPUT]
Name tail
Tag application.*
Path C:\\var\\log\\containers\\fluent-bit*
Parser docker
DB C:\\var\\fluent-bit\\state\\flb_log.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Rotate_Wait 30
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[INPUT]
Name tail
Tag application.*
Path C:\\var\\log\\containers\\cloudwatch-agent*
Parser docker
DB C:\\var\\fluent-bit\\state\\flb_cwagent.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Rotate_Wait 30
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[OUTPUT]
Name cloudwatch_logs
Match application.*
region ${AWS_REGION}
log_group_name /aws/containerinsights/${CLUSTER_NAME}/application
log_stream_prefix ${HOST_NAME}-
auto_create_group true
extra_user_agent container-insights

dataplane-log.conf: |
[INPUT]
Name tail
Tag dataplane.tail.*
Path C:\\ProgramData\\containerd\\root\\*.log, C:\\ProgramData\\Amazon\\EKS\\logs\\*.log
Parser dataplane_firstline
DB C:\\var\\fluent-bit\\state\\flb_dataplane_tail.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Rotate_Wait 30
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[INPUT]
Name tail
Tag dataplane.tail.C.ProgramData.Amazon.EKS.logs.vpc-bridge
Path C:\\ProgramData\\Amazon\\EKS\\logs\\*.log.*
Path_Key file_name
Parser dataplane_firstline
DB C:\\var\\fluent-bit\\state\\flb_dataplane_cni_tail.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Rotate_Wait 30
Refresh_Interval 10
Read_from_Head ${READ_FROM_HEAD}

[FILTER]
Name aws
Match dataplane.*
imds_version v2

[OUTPUT]
Name cloudwatch_logs
Match dataplane.*
region ${AWS_REGION}
log_group_name /aws/containerinsights/${CLUSTER_NAME}/dataplane
log_stream_prefix ${HOST_NAME}-
auto_create_group true
extra_user_agent container-insights

host-log.conf: |
[INPUT]
Name winlog
Channels EKS, System
DB C:\\var\\fluent-bit\\state\\flb_system_winlog.db
Interval_Sec 60

[FILTER]
Name aws
Match winlog.*
imds_version v2

[OUTPUT]
Name cloudwatch_logs
Match winlog.*
region ${AWS_REGION}
log_group_name /aws/containerinsights/${CLUSTER_NAME}/host
log_stream_prefix ${HOST_NAME}.
auto_create_group true
extra_user_agent container-insights

{{- .Values.containerLogs.fluentBit.configWindows.service | nindent 4 }}
{{- range $key, $val := .Values.containerLogs.fluentBit.configWindows.extraFiles }}
@INCLUDE {{ $key }}
{{- end }}
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %b %d %H:%M:%S

[PARSER]
Name container_firstline
Format regex
Regex (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
Name dataplane_firstline
Format regex
Regex (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
{{- .Values.containerLogs.fluentBit.configWindows.customParsers | nindent 4 }}
{{- range $key, $val := .Values.containerLogs.fluentBit.configWindows.extraFiles }}
{{ $key }}: |
{{- (tpl $val $) | nindent 4 }}
{{- end -}}
{{- end -}}
Loading
Loading