Before his accident, Koen was already using Blender to create his art. He used it to sketch, sculpt, and build digital spaces as part of his broader visual practice. Returning to this tool is both a personal goal and a symbol of progress. It ties directly into the second objective of our project: getting Koen to the point where he can independently make art again.
Rethinking Interaction, A Complex Challenge
The question we’re now exploring is this: how can someone with limited physical mobility operate Blender in a meaningful way? The tool is powerful, but also dense. It’s filled with shortcuts, gestures, and UI conventions built for mouse and keyboard. Koen can’t use those the way most people can.
After several conversations with fellow designers and developers, we landed on a promising concept that combines existing technology in a new way. The next step is to build a working prototype so we can test and validate whether this approach is feasible for Koen.
Concept Setup
Blender will run on a Mac, and Koen will see and interact with it using Apple Vision Pro. macOS’s Mac Virtual Display makes this technically straightforward by mirroring the desktop environment into visionOS.
The innovation lies in the control layer:
- A custom visionOS app presents a floating virtual keyboard or control panel
- Koen navigates this interface using eye tracking. A tap gesture confirms selections
- The visionOS app sends the selected input to a macOS companion app
- The macOS app receives this input and translates it into keyboard shortcuts for Blender
This setup would create a simple feedback loop:
- Koen looks at a button in the virtual UI
- The visionOS app detects the gaze and tap
- A signal is sent to the Mac via local communication (for example, Multipeer or sockets)
- The macOS app converts the input into the correct Blender shortcut
- Blender responds as if the command was entered by keyboard
What’s Next
The concept is clear. The challenge now is to build a working prototype and test it with Koen. This will help us evaluate whether the setup is intuitive, responsive, and flexible enough for real creative work. We’ll also be able to explore how much control Koen can realistically manage, and where the UI may need to be simplified or adjusted.
The goal remains the same: enable Koen to express himself through digital art. The tools are just the means to get there.
As we start prototyping, we’ll keep sharing our progress and what we learn along the way. If you’re working on something similar or have ideas that might help, we’d love to hear from you.