Overall Architecture
1. Blender runs on macOS
2. User wears Apple Vision Pro, viewing Blender via Mac Virtual Display
3. Koen interacts with a custom visionOS app (e.g. a floating virtual keyboard panel) using eye tracking and tap gesture / dwell control
4. visionOS app sends interaction events to macOS companion app
5. macOS app translates these to system-level keyboard shortcuts sent to Blender
Connection Between visionOS and macOS App
We need real-time communication between the two apps. Here are the most suitable options:
1. Multipeer Connectivity
- Native Apple framework, originally for iOS/macOS.
- Supports peer-to-peer networking over Wi-Fi or Bluetooth.
- Works offline.
- Can be adapted for cross-device communication between visionOS and macOS.
Great for ease of use and fast discovery between nearby devices.
2. Bonjour + TCP/IP or WebSocket
- Advertise the macOS app on the local network using Bonjour (NSNetService).
- visionOS app discovers it and opens a socket connection.
- Send structured messages (e.g., JSON) for which key was “pressed.”
More control, scalable, and extendable.
Sending Keyboard Shortcuts to Blender from macOS App
On the macOS side, once the app receives a command (e.g., "shortcut": "G"), you can:
Use CGEvent or Quartz Event Services to send keypresses:
let src = CGEventSource(stateID: .hidSystemState)
let keyDown = CGEvent(keyboardEventSource: src, virtualKey: 5, keyDown: true) // e.g. 'G' key
let keyUp = CGEvent(keyboardEventSource: src, virtualKey: 5, keyDown: false)
keyDown?.post(tap: .cghidEventTap)
keyUp?.post(tap: .cghidEventTap)
Map keys using virtual key codes (e.g. kVK_ANSI_G from HIToolbox/Events.h)