Fetch Instance syslog

LXD
1

API “/1.0/instances/{name}/logs/{filename}” can get:
├── console.log
├── forkexec.log
├── forkstart.log
├── lxc.conf
├── lxc.log
├── proxy.portcache.log
└── proxy.ws.log
Which is useful for lxd cli admin, but from container user’s point of view, not really, as it shows a lot of:

lxc cont1 20220913090920.553 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc cont1 20220913090920.553 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing

… and similar.

It would be useful having access to Container syslog, as raw file or even better as stream.
Sure, file can be downloaded by GET “/1.0/instances/{name}/files/?path=/var/log/{syslog,error.log,messages}” but again the location and file name differ by distribution.

2

The logs presented are lxc/lxd logs though right?

Which is probably why LXD doesn’t export them for you? No?

3

I mean the system logs inside container (rsyslog local{0,1,2…}).

ubuntu :written to /var/log/syslog
alpine: /var/log/messages
…

When you create, start container or run applications, sometimes you need to check the sys logs, would be helpful to monitor it without explicitly accessing the instance by ssh/terminal …

4

Yea I get what your saying what i’m saying for the reasons your presenting its probably a no go.

Last time I checked the OS property of an image was optional, there is just no way of knowing where it is + what about windows in VM’s?

In LXDMosaic when we want to watch the cloud-init log I open a shell and execute tail -f /var/log/cloud-init thats probably the best way of doing it (the new SFTP might enable a better way but I haven’t experimented enough yet)

5

It also would be a nice feature for LXD Mosaic .

Many cloud/hosting providers offer a live console log to watch the advancement of vm boot process.
Sometimes it hangs due reasons and you even dont get network ready status or ssh … to investigate, something like:

image
image907×202 4.38 KB

6

LXDMosaic can do that already, you just need to use the “Terminal” (as LXDMosaic incorrectly calls it) :smile:

I dunno if you can scrape early boot logs before it boots, if you know a reliable way to get a VM into a bad boot state - id explore it for sure!

7

With “Terminal” you mean “console”?
Had tried that before, get “Not found”

curl -sk --cert $crt --key $key https://10.0.4.1:8443/instances/container/console | jq .

{
  "type": "error",
  "status": "",
  "status_code": 0,
  "operation": "",
  "error_code": 404,
  "error": "not found",
  "metadata": null
}
8

Well a 404 wont help but yes “consoles”, working out how they work with spice-html5 takes some work (took me ages to get my head around it but I “get it” now).

I cant speak for containers though, LXD hands over to the container at some point but I dont think the files API works at that point.

Here’s an screenshot of the boot sequence for a Ubuntu VM (ignore the stopped icon, I had to start node quickly);

Screenshot from 2022-09-13 17-20-07
Screenshot from 2022-09-13 17-20-071476×884 44.4 KB

9

Thanks for the hint.
So, going through all procedure, same as creating an interactive console, by creating operations, getting the ```
“fds”: {
“0”: “da3046cf02c0116febf4ef3fe4eaecdf308e720c05e5a9c730ce1a6f15417f66”,
“1”: “05896879d8692607bd6e4a09475667da3b5f6714418ab0ee0e5720b4c57f754b”

would keep a permanent flow of system log?

10

There’s a “gotta ya” in there but yes in principle its very similar!

11

Makes sense.
Let me try , than confirm your answer as solution. :pray:

1 Like
12

I feel you. Same here, though the approach in both, xterm and spice was bit different to yours (more nginx, haproxy websocket upgrades, rather pure php/node) , but yes it really is pain.
I think, it is intentional, a built in difficulty through lxd team for the aim of no pain no gain :slightly_smiling_face:

13

If you have spice im confused why your require syslogs? The users can just see whats going on (thats the screenshot above)

I’m not sure this is fair, the problems are “hard” - its not LXD’s fault (though the bandwidth cost of spice could be addressed, at the cost of even more complexity) :smile:

14

Because I use spice for graphical virtual machines only and mostly for windows VM graphical
Usually launched spice at a time when graphic had loaded already and missed the flow of boot messages.
Not really necessary to go spice in an alpine non-graphical mini image.

I really know LXDMosaic can do, yet
@stgraber
Mr. Graber, is there an easy way fetching the system logs, without bein member of elite club?

15

LXD isn’t generally in the business of integrating/parsing data from inside of its instances as would be required here to stream syslog (or journal).

For those kind of things, you’re likely better off either using an old school remote syslog server (rsyslog does that pretty easily) or using a modern log aggregate (greylog, loki, …).

The LXD file API can be used to fetch /var/log/syslog or similar paths from inside of the instance, but as that’s a one time transfer, it won’t work too well for streaming data like those log files. The more modern sshfs API would likely do a bit better though I don’t think that sshfs supports notifications/polling which would still limit what can be done as far as streaming.

16

Thanks for elaborating.
dmesg would not work within container and that is fine, as kernel.dmesg_restrict is set and container has nothing to do with kernel ring messages.
Would the result of exec tail -f /var/log/syslog passed to xterm work?