I’m reading prometheus metrics from lxc query /1.0/metrics. I’d like to add additional labels (like from lxc list, for example I’d want to add custom field {fqdn=user.fqdn}.
Is it possible to do without writing my on code getting metrics from lxc list?
What is the best way to do it?
Possibly. We could file it as a feature request at https://github.com/lxc/lxd/issues
There are some concerns about the performance of Prometheus queries as labels get added and depending on what’s done, it could also interfere with the default dashboard. But that’s something that can be documented and that can easily be spotted in config when someone files a support request (similar to a raw. config key, we wouldn’t offer support).
AFAIK, the user.* keys are free form. Maybe we could use something like metrics.label.* instead?
Well, I think the point here is to set a label to the value of an existing config key, not have to define a new one.
So say you could do privileged: security.privileged
But we could have a raw.metrics.labels config key that takes in key/value pairs of label name to instance config key name.
I think we’d want that set globally though (server config) as having the list of labels vary on a per-instance basis is going to cause a bunch of problems.
In my setup I have custom key user.fqdn with real hostname, and all my container’s name are like numbers. Example: container c123 with user.fqdn=mail.domain.tld
So I need to expose user.fqdn with metrics to prometheus/grafana for better/faster maintainability