New container not showing on prometheus

LXD
,
1

I have issue

In prometheus it doesn’t show new containers and only shows old containers . But when I check lxc query /1.0/metrics, the new container and copy containers is there.

Not only the new container, the results from the lxc copy container also don’t appear on Prometheus.

I checked Prometheus, there were no errors or bugs and I restarted Prometheus but it didn’t work.

Thanks

2

Sounds like you have Prometheus scrape rules setup incorrectly.

@sdeziel are you able to offer any advice/questions to ascertain what is wrong here?

Thanks

3

I cannot reproduce the issue with Prometheus 2.37.2. Please make sure you are checking Prometheus after it had the time to do at least one scrape of the LXD server. Also be sure that when you query Prometheus, the “evaluation time” is the current time rather than an older time that predates the existence of the container.

4

Thanks for the respon @tomp and @sdeziel

I think I got the problem, I’m using LXD cluster and made the prometheus on node 2. When I checked it turned out that prometheus on node 2 only reads containers and vm on node 1, so I created prometheus on node 1 so that it can read containers and vm on node 2. Maybe this is my solution for now.

I don’t know why, but before lxd version 5.7 prometheus was able to read containers and vm from all nodes.

5

In cluster environments, LXD will only return the values for instances running on the server being accessed. It’s expected that each cluster member will be scraped separately.

2 Likes