As I can see fsnotify golang package provides us with no choice here.
IN_MODIFY is used to detect writes. But we are not interested in writes, we only want to subscribe to CREATE and DELETE events. While in fsnotify code we can see unconditional enabling of all flags (fsnotify/backend_inotify.go at main · fsnotify/fsnotify · GitHub)
var flags uint32 = unix.IN_MOVED_TO | unix.IN_MOVED_FROM |
unix.IN_CREATE | unix.IN_ATTRIB | unix.IN_MODIFY |
unix.IN_MOVE_SELF | unix.IN_DELETE | unix.IN_DELETE_SELF