Is there already something like that for incus / incusOS ?
I have big interest into implementing such a service next if not.
What would such a service do:
Connect to the incus API as well listen to events.
Find instances that want to be logged (either by user.label.xyz or hardcoded ENV)
Listen on a incus websocket for instance logs (one per instance).
Use existing fluentd code (maybe even as a library) to forward them.
MAYBE offer a tee as in clone for console clients.
If anyone else is interested into implementing it ic-healthd from incus-compose might be a nice starting point - but don’t use tokens use certificates for long living services
A question to incus developers, is the events api made for such services and many of them?
We do support streaming of events to Loki, syslog or webhooks out of the box.
You can also get stuff through the events API as you suggested.
IncusOS supports streaming of the system journal over syslog only.
Note that this just gets you the log events, operation details, firewall hits (on OVN) and lifecycle events (audit trail).
In another issue you had interest in getting the container console output which none of this will help with. All our console outputs, whether container or VMs are stored in an in-memory ring buffer and that ring-buffer then gets flushed to disk when a client requests a dump of the console.
Incus is (by design) entirely unaware of whether new content showed up in the console log.
This is all for security reasons. There have been a bunch of attacks going on over the years where a malicious guest would try to inject escape sequences to their text console or would purposefully spam the console to either fill in the host’s disk or if actively monitored by the host, burn up all the CPU, in both cases causing a denial of service attack.
If you need the live text console, you can get it through the API, but there are a bunch of caveats to that:
You get attached to the bi-directional console which allows only a single client. Another incus console session will be prompted whether to disconnect the current session.
If a VM, only one client can be attached to any console, so connecting to the VGA output will require kicking out anyone connected to the text console. The access is exclusive so the ring-buffer gets disconnected, meaning that there is no more recorded log for anything that happens during a live connection.
If a VM on arm64, the live text console isn’t currently available (QEMU limitation).
That’s correct, the only way to get the live data is by connecting to the console but only a single client can be connected to it and any later connection will disconnect you.
Being attached to the console also prevents the console’s log from being retrieved later as the live console bypasses the ringbuffer when it is connected.
It’s also plausible that we may introduce overall restrictions on the number of console connections being established by a given client or have them auto-disconnect after a period of time as the mere relaying of the data could expose the Incus daemon to elevated load from an instance flooding its console.
You can’t have it be consistent on the filesystem. The backend is a ringbuffer and that ringbuffer gets dumped onto the filesystem every time you run an incus console --show-log. But now that we don’t reset it every time, it means you have no idea what has changed since the last time nor do you have any idea if you’re going to be missing some data.
The ringbuffer is 128kB, if you query it when more than 128kB of data was written, then you’ll be missing some content.
In general, I find the whole idea of using a bunch of CPU to read and forward the console output of every container to a central location to be just a bad idea as that means that all it takes is for one container to run cat /dev/urandom > /dev/console to take down:
Your local CPU (doing the read+forward)
Your network (sending potentially GB/s of crap over the network)
Your remote log analysis (trying to keep up with all the data being fed to it)
Your remote log storage (trying to store all that data)
I run a fair bit of OCI containers these days and I’ve never really had an issue with the ring buffer size. Either things are working correctly and I don’t care what’s in there or something is wrong and I pull the current log and/or attach to the live feed.
I generally don’t want to have a ton of logs all over the place, especially when that stuff can contain PII and needs a plan to handle GDPR right to be forgotten requests and similar.
Instead I use services with built-in metrics endpoints, get those scraped by Prometheus and use Grafana for dashboard and altering. Then when the metrics suggest something’s wrong, that’s when I go look at the live console data. That console data is never preserved so no issue around PII and similar data leaking in there.
The random thing or anything like that is bad, yes.
I’m sure forwarders have a mitigation for that, I can think of some secure envs where it is a requirement to log the applications output to a central stash.
Also not just for metrics but for searching in greylog, kibana and stuff. Something like that I did a long time with ago with Firewall logs, they where neat.
Let’s see what others tell, I can live without a thing like incus-fluentd
No, console.log is just a one time disk copy of what the last incus console --show-log showed you. It’s not otherwise kept updated at all though liblxc may be flushing the last chunk of the ringbuffer on container exit (which then allows incus console --show-log to retrieve it).
root@dakara:~# incus stop -f c1
root@dakara:~# incus console --show-log c1
Unable to open LoaderSystemToken EFI variable, ignoring: Permission denied
systemd 257.13-1~deb13u1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF -XKBCOMMON -UTMP +SYSVINIT +LIBARCHIVE)
Detected virtualization lxc.
Detected architecture x86-64.
Welcome to Debian GNU/Linux 13 (trixie)!
bpf-restrict-fs: BPF LSM hook not enabled in the kernel, BPF LSM not supported.
Queued start job for default target graphical.target.
[ OK ] Created slice system-getty.slice - Slice /system/getty.
[ OK ] Created slice system-modprobe.slice - Slice /system/modprobe.
[ OK ] Created slice user.slice - User and Session Slice.
[ OK ] Started systemd-ask-password-console.path - Dispatch Password Requests to Console Directory Watch.
[ OK ] Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[UNSUPP] Starting of proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point unsupported.
[ OK ] Reached target paths.target - Path Units.
[ OK ] Reached target remote-fs.target - Remote File Systems.
[ OK ] Reached target slices.target - Slice Units.
[ OK ] Reached target swap.target - Swaps.
[ OK ] Listening on systemd-creds.socket - Credential Encryption/Decryption.
[ OK ] Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[ OK ] Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[ OK ] Listening on systemd-journald.socket - Journal Sockets.
[ OK ] Listening on systemd-networkd.socket - Network Service Netlink Socket.
[ OK ] Listening on systemd-udevd-control.socket - udev Control Socket.
[ OK ] Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
Mounting run-lock.mount - Legacy Locks Directory /run/lock...
Mounting tmp.mount - Temporary Directory /tmp...
Starting modprobe@configfs.service - Load Kernel Module configfs...
Starting modprobe@drm.service - Load Kernel Module drm...
Starting modprobe@fuse.service - Load Kernel Module fuse...
Starting systemd-journald.service - Journal Service...
Starting systemd-network-generator.service - Generate Network Units from Kernel Command Line...
modprobe@drm.service: Unable to locate executable '/sbin/modprobe': No such file or directory
modprobe@configfs.service: Unable to locate executable '/sbin/modprobe': No such file or directory
Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
modprobe@fuse.service: Unable to locate executable '/sbin/modprobe': No such file or directory
Starting systemd-sysctl.service - Apply Kernel Variables...
Starting systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully...
Starting systemd-udev-load-credentials.service - Load udev Rules from Credentials...
Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[ OK ] Mounted run-lock.mount - Legacy Locks Directory /run/lock.
[ OK ] Mounted tmp.mount - Temporary Directory /tmp.
modprobe@configfs.service: Deactivated successfully.
[ OK ] Finished modprobe@configfs.service - Load Kernel Module configfs.
modprobe@drm.service: Deactivated successfully.
[ OK ] Finished modprobe@drm.service - Load Kernel Module drm.
modprobe@fuse.service: Deactivated successfully.
[ OK ] Finished modprobe@fuse.service - Load Kernel Module fuse.
[ OK ] Finished systemd-network-generator.service - Generate Network Units from Kernel Command Line.
[ OK ] Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.
[ OK ] Reached target network-pre.target - Preparation for Network.
[ OK ] Finished systemd-sysctl.service - Apply Kernel Variables.
[ OK ] Finished systemd-udev-load-credentials.service - Load udev Rules from Credentials.
[ OK ] Finished systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully.
Starting systemd-resolved.service - Network Name Resolution...
Starting systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev...
[ OK ] Started systemd-journald.service - Journal Service.
Starting systemd-journal-flush.service - Flush Journal to Persistent Storage...
[ OK ] Finished systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev.
[ OK ] Reached target local-fs-pre.target - Preparation for Local File Systems.
[ OK ] Reached target local-fs.target - Local File Systems.
[ OK ] Listening on systemd-sysext.socket - System Extension Image Management.
Starting systemd-udevd.service - Rule-based Manager for Device Events and Files...
[ OK ] Finished systemd-journal-flush.service - Flush Journal to Persistent Storage.
Starting systemd-tmpfiles-setup.service - Create System Files and Directories...
[ OK ] Finished systemd-tmpfiles-setup.service - Create System Files and Directories.
[ OK ] Started systemd-udevd.service - Rule-based Manager for Device Events and Files.
Starting systemd-networkd.service - Network Configuration...
[ OK ] Started systemd-resolved.service - Network Name Resolution.
[ OK ] Reached target nss-lookup.target - Host and Network Name Lookups.
[ OK ] Started systemd-networkd.service - Network Configuration.
[ OK ] Reached target network.target - Network.
Starting systemd-networkd-persistent-storage.service - Enable Persistent Storage in systemd-networkd...
Starting systemd-networkd-wait-online.service - Wait for Network to be Configured...
[ OK ] Finished systemd-networkd-persistent-storage.service - Enable Persistent Storage in systemd-networkd.
[ OK ] Finished systemd-udev-trigger.service - Coldplug All udev Devices.
[ OK ] Reached target sysinit.target - System Initialization.
[ OK ] Started apt-daily.timer - Daily apt download activities.
[ OK ] Started apt-daily-upgrade.timer - Daily apt upgrade and clean activities.
[ OK ] Started dpkg-db-backup.timer - Daily dpkg database backup timer.
[ OK ] Started systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories.
[ OK ] Reached target timers.target - Timer Units.
[ OK ] Listening on dbus.socket - D-Bus System Message Bus Socket.
[ OK ] Listening on systemd-hostnamed.socket - Hostname Service Socket.
[ OK ] Reached target sockets.target - Socket Units.
[ OK ] Reached target basic.target - Basic System.
Starting dbus.service - D-Bus System Message Bus...
Starting systemd-logind.service - User Login Management...
Starting systemd-user-sessions.service - Permit User Sessions...
[ OK ] Finished systemd-user-sessions.service - Permit User Sessions.
[ OK ] Started console-getty.service - Console Getty.
[ OK ] Reached target getty.target - Login Prompts.
[ OK ] Started dbus.service - D-Bus System Message Bus.
Starting systemd-hostnamed.service - Hostname Service...
[ OK ] Started systemd-logind.service - User Login Management.
[ OK ] Started systemd-hostnamed.service - Hostname Service.
Starting polkit.service - Authorization Manager...
[ OK ] Started polkit.service - Authorization Manager.
[ OK ] Finished systemd-networkd-wait-online.service - Wait for Network to be Configured.
[ OK ] Reached target network-online.target - Network is Online.
Starting nginx.service - A high performance web server and a reverse proxy server...
[ OK ] Started nginx.service - A high performance web server and a reverse proxy server.
[ OK ] Reached target multi-user.target - Multi-User System.
[ OK ] Reached target graphical.target - Graphical Interface.
Debian GNU/Linux 13 c1 console
c1 login:
root@dakara:~#
The only reason we have console.log at all is so you can retrieve it when the container is stopped. If it wasn’t for that, we wouldn’t even have the file to begin with.
As @jochumdev noted, the “big players” in OCI container runtimes direct console/stdout logs to somewhere.
In the absence of that support in Incus (for the reasons you outlined) I suppose the question is how do people with OCI containers in IncusOS handle web/application logs? Personally I’m already customising OCI images so disconnecting application outputs from stdout/stderr is an acceptable option in my environment, if there is a suitable replacement behaviour.
Lots of use cases require access to logs (everything from analytics to security investigations), and the ringbuffer data won’t be enough for them.
“Use applications which can stream their logs” is tempting, but unfortunately not the environment many people are working in.
