Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regarding the bug report of volcano-sh kube-state-metrics monitoring project, is it still maintained? #3949

Open
gsmini opened this issue Jan 2, 2025 · 9 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@gsmini
Copy link

gsmini commented Jan 2, 2025

Description

  1. When I deployed the indicator collection according to the official documentation, I found that there was a bug in the collection code. I raised a PR in that repository, but it seems that no one maintains that project anymore, so I came here to ask if the indicator collection project is still maintained?(volcano-sh/kube-state-metrics):add filtering judgment for the kube_pod_volcano_container_status_runn… kube-state-metrics#3

Steps to reproduce the issue

Describe the results you received and expected

Theoretically, only the indicators of the volcano label pod should be collected. The current bug is that the indicators of the k8s pod are collected, so when configuring the Grafana dashboard based on the indicators, it does not match the volcano task.

What version of Volcano are you using?

volcanosh/vc-webhook-manager:v1.10.0

Any other relevant information

No response

@gsmini gsmini added the kind/bug Categorizes issue or PR as related to a bug. label Jan 2, 2025
@hwdef
Copy link
Member

hwdef commented Jan 2, 2025

Yes, it is also maintained.
PTAL @Monokaix @JesseStutler

@gsmini
Copy link
Author

gsmini commented Jan 2, 2025

Yes, it is also maintained.
Thank you for taking a look at my PR

@Monokaix
Copy link
Member

Monokaix commented Jan 2, 2025

Yes, it is also maintained. PTAL @Monokaix @JesseStutler

Maybe we can find another way to solve the metrics introduced in that repo, it's not a good way to do intrusive modification of kube-state-metrics.

@hwdef
Copy link
Member

hwdef commented Jan 2, 2025

Yes, it is also maintained. PTAL @Monokaix @JesseStutler

Maybe we can find another way to solve the metrics introduced in that repo, it's not a good way to do intrusive modification of kube-state-metrics.

I agree with this. I have long wanted to replace the current solution, but I have no good ideas. We may need a new exporter component

@Monokaix
Copy link
Member

Monokaix commented Jan 2, 2025

Hi, please use English first, which can allow other users around the world to understand.

@gsmini
Copy link
Author

gsmini commented Jan 2, 2025

Yes, it is also maintained. PTAL @Monokaix @JesseStutler

Maybe we can find another way to solve the metrics introduced in that repo, it's not a good way to do intrusive modification of kube-state-metrics.

Yes, but I saw that the volcano task indicator implementation of the submission record history is also implemented in an intrusive way based on the original k8s kube-state-metrics. So I also adopted this solution at present. In the long run, it may not be the best solution. I also hope that you can come up with a separate component to facilitate the subsequent grafana dashboard business implementation based on the volcano business implementation of custom indicators.
I hope that day will come as soon as possible, and I wish this project will get better and better.

@gsmini gsmini changed the title volcano-sh kube-state-metrics监控项目bug,还维护么 Regarding the bug report of volcano-sh kube-state-metrics monitoring project, is it still maintained? Jan 2, 2025
@gsmini
Copy link
Author

gsmini commented Jan 2, 2025

Hi, please use English first, which can allow other users around the world to understand.

Sorry, the current text has been changed to English.

@yccharles
Copy link

My temp solution is : only keep volcano's metrics when prometheus scrape kube-state-metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/instance: volcano-monitoring
  name: volcano-kube-state-metrics
  namespace: volcano-monitoring
spec:
  endpoints:
    - honorLabels: true
      interval: 30s
      port: http-metrics
      path: /metrics
      scheme: http
      scrapeTimeout: 30s
      metricRelabelings:
        - action: keep
          regex: .*_volcano_.*
          sourceLabels:
            - __name__
        - action: keep
          regex: '.+'
          sourceLabels:
            - queue
  jobLabel: app.kubernetes.io/name
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics

or config prometheus.yaml by hand

- job_name: volcano-kube-state-metrics
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: .*_volcano_.*
    replacement: $1
    action: keep
  - source_labels: [queue]
    separator: ;
    regex: .+
    replacement: $1
    action: keep

@gsmini
Copy link
Author

gsmini commented Jan 7, 2025

My temp solution is : only keep volcano's metrics when prometheus scrape kube-state-metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/instance: volcano-monitoring
  name: volcano-kube-state-metrics
  namespace: volcano-monitoring
spec:
  endpoints:
    - honorLabels: true
      interval: 30s
      port: http-metrics
      path: /metrics
      scheme: http
      scrapeTimeout: 30s
      metricRelabelings:
        - action: keep
          regex: .*_volcano_.*
          sourceLabels:
            - __name__
        - action: keep
          regex: '.+'
          sourceLabels:
            - queue
  jobLabel: app.kubernetes.io/name
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics

or config prometheus.yaml by hand

- job_name: volcano-kube-state-metrics
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: .*_volcano_.*
    replacement: $1
    action: keep
  - source_labels: [queue]
    separator: ;
    regex: .+
    replacement: $1
    action: keep

Thank you for your solution. The filtering solution after the indicator life cycle should meet the needs, but it does not solve the fundamental problem of https://github.com/volcano-sh
kube-state-metrics indicator monitoring.
It's ok. Let's put this issue aside for now. I guess the maintenance team is too busy to handle it. Let's solve it when they have time. 😁:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

4 participants