Unix-char device hotplugging does not work on Ubuntu 22.04 if source device is a symlink

chris42 · August 24, 2022, 10:48pm

I have a setup where I forward a joystick input device to a container as unix-char. The device is defined as follows in the container configuration:

js0:
  mode: "0666"
  path: /dev/input/js0
  required: "false"
  source: /dev/jsWirelessController
  type: unix-char

The source device file is a symlink created by a udev rule on the host:

ACTION=="add", ENV{ID_BUS}=="bluetooth", ENV{DEVNAME}=="/dev/input/js[0-9]*", SYMLINK="jsWirelessController"

The kernel version is 5.15.0-46-lowlatency and the container runs Ubuntu 20.04.
In case of an Ubuntu 20.04 host hotplugging works as expected, but on a 22.04 host the device only shows up in the container when it already has been present on container start.

However, if I replace the symlink in the source setting with the actual device file /dev/input/jsX on the host, hotplugging also works correctly with the Ubuntu 22.04 host. This is not a solution in my case since I need the udev rule to disambiguate between multiple different devices.

Both the 22.04 and 20.04 machine run exactly the same kernel version, lxd configurations and lxd versions. I have tried various lxd 5.x versions with the same result.

How can I debug this further?

tomp · August 25, 2022, 7:37am

This is where the event handler for the hotplug events is defined for the unix devices:

github.com

lxc/lxd/blob/master/lxd/device/unix_common.go#L98-L102

      
        
            	f := func(e UnixEvent) (*deviceConfig.RunConfig, error) {
            		// Check if the event is for a device file that this device wants.
            		if unixDeviceSourcePath(devConfig) != e.Path {
            			return nil, nil
            		}

I wonder if the e.Path variable from the event is the DEVNAME rather than the symlink target, which would explain why it doesn’t match, or perhaps there is no event being delivered for the symlink somehow.

The actual hotplug events are generated using fsnotify or fanotify watchers on /dev.

github.com

lxc/lxd/blob/4a9d85de837fed097736082f8a1806812ec8c941/lxd/fsmonitor/drivers/driver_fsnotify.go#L50

      
        
            		return fmt.Errorf("Failed to watch directory %q: %w", d.prefixPath, err)
            	}
            
            
	go d.getEvents(ctx)
            
            
	fsnotifyLoaded = true
            
            
	return nil
            }
            
            
func (d *fsnotify) getEvents(ctx context.Context) {
            	for {
            		select {
            		// Clean up if context is done
            		case <-ctx.Done():
            			_ = d.watcher.Close()
            			fsnotifyLoaded = false
            			return
            		case event := <-d.watcher.Events:
            			// Only consider create and remove events
            			if event.Op&fsn.Create == 0 && event.Op&fsn.Remove == 0 {

github.com

lxc/lxd/blob/4a9d85de837fed097736082f8a1806812ec8c941/lxd/fsmonitor/drivers/driver_fanotify.go#L74

      
        
            		fanotifyLoaded = false
            	}()
            
            
	go d.getEvents(fd)
            
            
	fanotifyLoaded = true
            
            
	return nil
            }
            
            
func (d *fanotify) getEvents(mountFd int) {
            	for {
            		buf := make([]byte, 256)
            
            
		// Although the event is less than 256 bytes, we read as much to ensure the entire event
            		// is captured and following events are readable. Using only binary.Read() would require
            		// more manual cleanup as otherwise bytes from a previous event would still be present and
            		// make everything unreadable.
            		_, err := unix.Read(d.fd, buf)
            		if err != nil {
            			// Stop listening for events as the fanotify fd has been closed due to cleanup.

@monstermunchkin are you aware of any limitations around watching for symlinks in this implementation?