How to setup custom metrics

[yangvipguang]

What did you do?
now, does custom metrics files been removed? i do not know how to config this . Should I generate these files with jsonnet? because I find the custom.metrics with jsonnet format 。
Did you expect to see some different?

Environment

• Prometheus Operator version:

  quay.io/coreos/prometheus-operator:v0.39.0

• Kubernetes version information:
  1.16.x

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

Insert Prometheus Operator logs relevant to the issue here

• Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?:

[johanneswuerbach]

Hey @yangvipguang,

yes you need to setup jsonnet https://github.com/coreos/kube-prometheus/blob/master/README.md#compiling

Once that is done you can uncomment https://github.com/coreos/kube-prometheus/blob/master/example.jsonnet#L9 this line and you get the extra manifests needed for custom metrics.

Cheers,
Johannes

[hecheng64]

Using release-0.9, and uncomment https://github.com/coreos/kube-prometheus/blob/master/example.jsonnet#L9 this line and you get the extra manifests needed for custom metrics.

can,t use the metrics:http_requests:
unable to get metric http_requests: unable to fetch metrics from custom metrics API: the server could not find the metric http_requests for pods

[yangvipguang]

thanks a lot 。but I have switched to branch of release-0.4。 I found the cutom metrics in the experimental files 。 but I have some error with permissions。Could you give me a favor

I have to generate some other files like ca etcs

[johanneswuerbach]

This sounds like you didn't follow https://github.com/coreos/kube-prometheus#prerequisites, did you?

[yangvipguang]

i have figured 。may be i changed incrrectly when config custom metrics。 i have deploy the custom yaml files I have found the api versions ：custom api versions but i returned some errors 。

[metalmatze]

Maybe the video from last years FOSDEM might help too: https://www.youtube.com/watch?v=paO2kwHWOLw :)

[evgao]

Hi Guys,

I am trying to build release-0.5 with custom-metrics, I uncomment the line in example.jsonnet and it looks like the following:

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +

when I build with ./build.sh example.jsonnet command, it failed with the follow error message:

RUNTIME ERROR: couldn't open import "kube-prometheus/kube-prometheus-custom-metrics.libsonnet": no match locally or in the Jsonnet library paths

When I build without custom-metrics, it works fine. Kindly advise how to fix it. Thanks a lot.

[yangvipguang]

@evgao you should run ： jb install first and then ./build.sh

[evgao]

@yangvipguang Thanks Guang, It works now.

[rbtrun]

hi guys,

Using release-0.4

Also want open customer metrics with this:
(import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet')

But meet this error:
RUNTIME ERROR: couldn't open import "kube-prometheus/kube-prometheus-custom-metrics.libsonnet": no match locally or in the Jsonnet library paths

Indeed this file does not exist
kube-prometheus/kube-prometheus-custom-metrics.libsonnet

How can I get this file or is there a sample file for this?

[jnccneto]

Hello,

I have been trying this setup but I get an error when building with jsonet and enabled custom-metrics.libsonnet.
Any ideas on how to fix it?

error output:

RUNTIME ERROR: max stack frames exceeded.
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
...
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/addons/custom-metrics.libsonnet:7:18-54 object
vendor/kube-prometheus/components/prometheus-adapter.libsonnet:141:20-40 object
During manifestation

[Jaesang]

I have same problem, anyone could fix this? I tested this release-0.10 and main branch.

[rgershkovich]

This is due to:

{
  values+:: {
    prometheusAdapter+: {
      namespace: $.values.prometheusAdapter.namespace,

You can do something like:

local customMetrics = import 'kube-prometheus-release-0.10/addons/custom-metrics.libsonnet';
local prometheusAdapter = import 'kube-prometheus-release-0.10/components/prometheus-adapter.libsonnet';

{
  values:: {
    ...

  prometheusAdapter: prometheusAdapter($.values.prometheusAdapter) +
  ...
} + customMetrics + {
  values+:: {
    prometheusAdapter+: {
      namespace: 'monitoring',
    },
  },
}