Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/integration updates #1130

Merged
merged 12 commits into from
Jan 16, 2025
Merged

Feat/integration updates #1130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
6a08880
Aligned Loki Integration w/ Loki-Mixin
bentonam Jan 16, 2025
d620d80
Default Grafana Integration Label Selector
bentonam Jan 16, 2025
50f65cb
Updated Tests for Non-Capturing Groups
bentonam Jan 16, 2025
c89a4fc
Added MySQL Logs Test
bentonam Jan 16, 2025
17dcd5c
Rebuilt
bentonam Jan 16, 2025
52a79e8
Updated Meta-Monitoring Example
bentonam Jan 16, 2025
bee6038
Updated Integration Test
bentonam Jan 16, 2025
2da876c
Merge branch 'main' into feat/integration-updates
bentonam Jan 16, 2025
02fb43d
Rebuilt
bentonam Jan 16, 2025
410b181
Merge branch 'main' of github.com:grafana/k8s-monitoring-helm into fe…
bentonam Jan 16, 2025
165bf6a
Merge branch 'main' of github.com:grafana/k8s-monitoring-helm into fe…
bentonam Jan 16, 2025
ed5193a
Rebuilt
bentonam Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/k8s-monitoring/charts/feature-integrations/docs/integrations/grafana.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| fieldSelectors | list | `[]` | Discover Grafana instances based on field selectors. |
| labelSelectors | object | `{}` | Discover Grafana instances based on label selectors. |
| labelSelectors | object | `{"app.kubernetes.io/name":"grafana"}` | Discover Grafana instances based on label selectors. |
| metrics.portName | string | `"grafana"` | Name of the port to scrape metrics from. |
| namespaces | list | `[]` | The namespaces to look for Grafana instances in. Will automatically look for Grafana instances in all namespaces unless specified here |

Expand Down
15 changes: 7 additions & 8 deletions charts/k8s-monitoring/charts/feature-integrations/docs/integrations/loki.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,10 @@
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| fieldSelectors | list | `[]` | Discover Loki instances based on field selectors. |
| labelSelectors | object | `{}` | Discover Loki instances based on label selectors. |
| labelSelectors | object | `{"app.kubernetes.io/name":["loki","enterprise-logs"]}` | Discover Loki instances based on label selectors. |
| metrics.portName | string | `"http-metrics"` | Name of the port to scrape metrics from. |
| namespaces | list | `[]` | Namespaces to look for Loki instances in. Will automatically look for Loki instances in all namespaces unless specified here |

### General Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| jobLabel | string | `"integrations/loki"` | The value of the job label for scraped metrics and logs |
| name | string | `""` | Name for this Loki instance. |

### Logs Settings

| Key | Type | Default | Description |
Expand Down Expand Up @@ -49,3 +42,9 @@
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| metrics.scrapeInterval | string | `60s` | How frequently to scrape metrics from Loki. |

### General Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| name | string | `""` | Name for this Loki instance. |
3 changes: 2 additions & 1 deletion charts/k8s-monitoring/charts/feature-integrations/integrations/grafana-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ fieldSelectors: []

# -- Discover Grafana instances based on label selectors.
# @section -- Discovery Settings
labelSelectors: {}
labelSelectors:
app.kubernetes.io/name: grafana

# Settings for metrics collection
metrics:
Expand Down
7 changes: 2 additions & 5 deletions charts/k8s-monitoring/charts/feature-integrations/integrations/loki-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,6 @@
# @section -- General Settings
name: ""

# -- The value of the job label for scraped metrics and logs
# @section -- General Settings
jobLabel: integrations/loki

# -- Namespaces to look for Loki instances in.
# Will automatically look for Loki instances in all namespaces unless specified here
# @section -- Discovery Settings
Expand All @@ -18,7 +14,8 @@ fieldSelectors: []

# -- Discover Loki instances based on label selectors.
# @section -- Discovery Settings
labelSelectors: {}
labelSelectors:
app.kubernetes.io/name: [loki, enterprise-logs]

# Settings for metrics collection
metrics:
Expand Down
7 changes: 6 additions & 1 deletion ...oring/charts/feature-integrations/schema-mods/definitions/grafana-integration.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,12 @@
"type": "string"
},
"labelSelectors": {
"type": "object"
"type": "object",
"properties": {
"app.kubernetes.io/name": {
"type": "string"
}
}
},
"logs": {
"type": "object",
Expand Down
13 changes: 9 additions & 4 deletions ...nitoring/charts/feature-integrations/schema-mods/definitions/loki-integration.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,16 @@
"fieldSelectors": {
"type": "array"
},
"jobLabel": {
"type": "string"
},
"labelSelectors": {
"type": "object"
"type": "object",
"properties": {
"app.kubernetes.io/name": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
"logs": {
"type": "object",
Expand Down
14 changes: 7 additions & 7 deletions charts/k8s-monitoring/charts/feature-integrations/templates/_integration_grafana_logs.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,28 +19,28 @@
{{- $valueList := list }}
{{- if .namespaces }}
{{- $labelList = append $labelList "__meta_kubernetes_namespace" -}}
{{- $valueList = append $valueList (printf "(%s)" (join "|" .namespaces)) -}}
{{- $valueList = append $valueList (printf "(?:%s)" (join "|" .namespaces)) -}}
{{- end }}
{{- range $k, $v := .labelSelectors }}
{{- if kindIs "slice" $v }}
{{- $labelList = append $labelList (include "pod_label" $k) -}}
{{- $valueList = append $valueList (printf "(%s)" (join "|" $v)) -}}
{{- $valueList = append $valueList (printf "(?:%s)" (join "|" $v)) -}}
{{- else }}
{{- $labelList = append $labelList (include "pod_label" $k) -}}
{{- $valueList = append $valueList $v -}}
{{- $valueList = append $valueList (printf "(?:%s)" $v) -}}
{{- end }}
{{- end }}
rule {
source_labels = {{ $labelList | sortAlpha | toJson }}
source_labels = {{ $labelList | toJson }}
separator = ";"
regex = {{ $valueList | sortAlpha | join ";" | quote }}
regex = {{ $valueList | join ";" | quote }}
target_label = "job"
replacement = "integrations/grafana"
}
rule {
source_labels = {{ $labelList | sortAlpha | toJson }}
source_labels = {{ $labelList | toJson }}
separator = ";"
regex = {{ $valueList | sortAlpha | join ";" | quote }}
regex = {{ $valueList | join ";" | quote }}
target_label = "instance"
replacement = {{ $instance.name | quote }}
Comment on lines 20 to 45
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We gotta do something about predictable layout for this. We had the sortAlpha in there before because it wasn't deterministic and it failed autotests pretty regularly.

Maybe instead of range $k, $v := .labelSelectors we do something more like range $key := (keys .labelSelectors | sort)?

}
Expand Down
35 changes: 24 additions & 11 deletions charts/k8s-monitoring/charts/feature-integrations/templates/_integration_loki_logs.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,30 +19,43 @@
{{- $valueList := list }}
{{- if .namespaces }}
{{- $labelList = append $labelList "__meta_kubernetes_namespace" -}}
{{- $valueList = append $valueList (printf "(%s)" (join "|" .namespaces)) -}}
{{- $valueList = append $valueList (printf "(?:%s)" (join "|" .namespaces)) -}}
{{- end }}
{{- range $k, $v := .labelSelectors }}
{{- if kindIs "slice" $v }}
{{- $labelList = append $labelList (include "pod_label" $k) -}}
{{- $valueList = append $valueList (printf "(%s)" (join "|" $v)) -}}
{{- $valueList = append $valueList (printf "(?:%s)" (join "|" $v)) -}}
{{- else }}
{{- $labelList = append $labelList (include "pod_label" $k) -}}
{{- $valueList = append $valueList $v -}}
{{- $valueList = append $valueList (printf "(?:%s)" $v) -}}
{{- end }}
{{- end }}
// add static label of integration="loki" and instance="name" to pods that match the selector so they can be identified in the loki.process stages
rule {
source_labels = {{ $labelList | sortAlpha | toJson }}
source_labels = {{ $labelList | toJson }}
separator = ";"
regex = {{ $valueList | sortAlpha | join ";" | quote }}
target_label = "job"
replacement = "integrations/loki"
regex = {{ $valueList | join ";" | quote }}
target_label = "integration"
replacement = "loki"
}
rule {
source_labels = {{ $labelList | sortAlpha | toJson }}
source_labels = {{ $labelList | toJson }}
separator = ";"
regex = {{ $valueList | sortAlpha | join ";" | quote }}
regex = {{ $valueList | join ";" | quote }}
target_label = "instance"
replacement = {{ $instance.name | quote }}
}
{{- $labelList = append $labelList "__meta_kubernetes_namespace" -}}
{{- $valueList = append $valueList "([^;]+)" -}}
{{- $labelList = append $labelList (include "pod_label" "component") -}}
{{- $valueList = append $valueList "([^;]+)" }}
// override the job label to be namespace/component so it aligns to the loki-mixin
rule {
source_labels = {{ $labelList | toJson }}
separator = ";"
regex = {{ $valueList | join ";" | quote }}
target_label = "job"
replacement = "$1/$2"
}
{{- end }}
{{- end }}
Expand All @@ -58,9 +71,9 @@ rule {
{{- if .logs.enabled }}
stage.match {
{{- if $instance.namespaces }}
selector = "{job=\"integrations/loki\",instance=\"{{ $instance.name }}\",namespace=~\"{{ $instance.namespaces | join "|" }}\"}"
selector = "{integration=\"loki\",instance=\"{{ $instance.name }}\",namespace=~\"{{ $instance.namespaces | join "|" }}\"}"
{{- else }}
selector = "{job=\"integrations/loki\",instance=\"{{ $instance.name }}\"}"
selector = "{integration=\"loki\",instance=\"{{ $instance.name }}\"}"
{{- end }}

// extract some of the fields from the log line
Expand Down
38 changes: 36 additions & 2 deletions charts/k8s-monitoring/charts/feature-integrations/templates/_integration_loki_metrics.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,13 @@ declare "loki_integration" {
action = "keep"
}

// the loki-mixin expects the job label to be namespace/component
rule {
source_labels = ["__meta_kubernetes_namespace","__meta_kubernetes_pod_label_app_kubernetes_io_component"]
separator = "/"
target_label = "job"
}

{{ include "commonRelabelings" . | nindent 4 }}
}

Expand Down Expand Up @@ -140,9 +147,36 @@ declare "loki_integration" {
// drop metrics that match the drop_metrics regex
rule {
source_labels = ["__name__"]
regex = coalesce(argument.drop_metrics.value, "(^(go|process)_.+$)")
regex = coalesce(argument.drop_metrics.value, "")
action = "drop"
}

// keep only metrics that match the keep_metrics regex
rule {
source_labels = ["__name__"]
regex = coalesce(argument.keep_metrics.value, "(.+)")
action = "keep"
}

// the loki-mixin expects the instance label to be the node name
rule {
source_labels = ["node"]
target_label = "instance"
replacement = "$1"
}
rule {
action = "labeldrop"
regex = "node"
}

// set the memcached exporter container name from container="exporter" to container="memcached"
rule {
source_labels = ["component", "container"]
separator = ";"
regex = "memcached-[^;]+;exporter"
target_label = "container"
replacement = "memcached"
}
}
}
{{- range $instance := $.Values.loki.instances }}
Expand Down Expand Up @@ -177,7 +211,7 @@ loki_integration_discovery {{ include "helper.alloy_name" .name | quote }} {

loki_integration_scrape {{ include "helper.alloy_name" .name | quote }} {
targets = loki_integration_discovery.{{ include "helper.alloy_name" .name }}.output
job_label = {{ .jobLabel | quote }}
job_label = "integrations/loki"
clustering = true
{{- if $metricAllowList }}
keep_metrics = {{ $metricAllowList | join "|" | quote }}
Expand Down
14 changes: 7 additions & 7 deletions charts/k8s-monitoring/charts/feature-integrations/templates/_integration_mysql_logs.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,28 +17,28 @@
{{- $valueList := list }}
{{- if .logs.namespaces }}
{{- $labelList = append $labelList "__meta_kubernetes_namespace" -}}
{{- $valueList = append $valueList (printf "(%s)" (join "|" .logs.namespaces)) -}}
{{- $valueList = append $valueList (printf "(?:%s)" (join "|" .logs.namespaces)) -}}
{{- end }}
{{- range $k, $v := .logs.labelSelectors }}
{{- if kindIs "slice" $v }}
{{- $labelList = append $labelList (include "pod_label" $k) -}}
{{- $valueList = append $valueList (printf "(%s)" (join "|" $v)) -}}
{{- $valueList = append $valueList (printf "(?:%s)" (join "|" $v)) -}}
{{- else }}
{{- $labelList = append $labelList (include "pod_label" $k) -}}
{{- $valueList = append $valueList $v -}}
{{- $valueList = append $valueList (printf "(?:%s)" $v) -}}
{{- end }}
{{- end }}
rule {
source_labels = {{ $labelList | sortAlpha | toJson }}
source_labels = {{ $labelList | toJson }}
separator = ";"
regex = {{ $valueList | sortAlpha | join ";" | quote }}
regex = {{ $valueList | join ";" | quote }}
target_label = "integration"
replacement = "mysql"
}
rule {
source_labels = {{ $labelList | sortAlpha | toJson }}
source_labels = {{ $labelList | toJson }}
separator = ";"
regex = {{ $valueList | sortAlpha | join ";" | quote }}
regex = {{ $valueList | join ";" | quote }}
target_label = "instance"
replacement = {{ $instance.name | quote }}
}
Expand Down
21 changes: 18 additions & 3 deletions charts/k8s-monitoring/charts/feature-integrations/tests/grafana_logs_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@ tests:
rule {
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
separator = ";"
regex = "grafana"
regex = "(?:grafana)"
target_label = "job"
replacement = "integrations/grafana"
}
rule {
source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
separator = ";"
regex = "grafana"
regex = "(?:grafana)"
target_label = "instance"
replacement = "grafana"
}
Expand Down Expand Up @@ -84,7 +84,22 @@ tests:
of: ConfigMap
- matchRegex:
path: data["logs.alloy"]
pattern: namespace=~\\"k8smon\\"
# The pattern should look like this, but since the regex is escaped, it will be a bit different
# rule {
# source_labels = ["__meta_kubernetes_namespace","__meta_kubernetes_pod_label_app_kubernetes_io_name"]
# separator = ";"
# regex = "(?:k8smon);(?:grafana)"
# target_label = "job"
# replacement = "integrations/grafana"
# }
pattern: |-
\s*rule {
\s* source_labels = \["__meta_kubernetes_namespace","__meta_kubernetes_pod_label_app_kubernetes_io_name"\]
\s* separator = ";"
\s* regex = "\(\?:k8smon\)\;\(\?:grafana\)"
\s* target_label = "job"
\s* replacement = "integrations/grafana"
\s*}

- it: should allow you to set the timestamp format
set:
Expand Down
Loading
Loading