Problem with Incus / Prometheus and multiple projects

1

Hi, I’ve just managed to get Prometheus and Grafana set up with my cluster, each running in it’s own container. Basically seems to work in that I’ve configured up one node and it’s populating the Incus dashboard.

Problem: it only appears to be using the FIRST project listed in the config, i.e. it’s not iterating the project variable. This is my first time using Prometheus so I’ve no idea where to look, this is my config;

  - job_name: incus
    metrics_path: '/1.0/metrics'
    params:
      project: 
        - 'management'
        - 'default'
        - 'mms'
    scheme: 'https'
    static_configs:
      - targets: 
        - '10.2.0.9:8444'
    tls_config:
      ca_file: 'tls/server.crt'
      cert_file: 'tls/metrics.crt'
      key_file: 'tls/metrics.key'
      server_name: 'grok'
      insecure_skip_verify: true

Dashboard looks good, but the drop-down for projects only show “management”. If I create another job “incus2” and make project: [‘mms’,‘management’,‘default’], then it too works, but the drop-down on the dashboard only lists ‘mms’.

I “could” just replicate the stanza for each project and have “one” project in the project list, then use the “job” drop-down to effectively switch between projects, but it would be great to see the project drop-down working.

Am I doing something wrong, or is this a bug on the Prometheus fetch code? (or the Grafana display code?)

2

Any reason why you’re not pulling all projects?
(not mentioning the project parameter should do that)

3

When you do that, prometheus will make a single scrape to ..../1.0/metrics?project=management&project=default&project=mms

You can do the scrape yourself with curl and see what comes back. With incus 6.0.4, it seems to be looking at only the first value of the parameter provided:

# This shows only metrics from project "nsrc-builder"
curl -k 'https://localhost:8444/1.0/metrics?project=nsrc-builder&project=default' | perl -ne 'print "$1\n" if /project="(.*?)"/' | sort | uniq -c

I think it would be a feature request to honour multiple instances of the project parameter.

4

Any reason why you’re not pulling all projects?

Erm, I want all projects … maybe I read the documentation wrong, I thought it said that if you didn’t specify “project” it would use “default” … is the default to pull “all” ? (I’m trying it now …)

5

As Stéphane said, the default is all projects.

$ curl -fsSk 'https://localhost:8444/1.0/metrics' | perl -ne 'print "$1\n" if /project="(.*?)"/' | sort | uniq -c
    190 default
    193 nsrc-builder
6

Yep, cool, I just read the docs wrong … I’ve pulled the ‘param’ / project clause and now it’s filling all projects. Many thanks :slight_smile:

I’m not sure how the templates work, but as a really minor tweak, it sounds like changing the type of “project” from list to string would stop anyone else having this problem, if that’s possible?