Suresh has been hard at work expanding our prototype. With every step forward, we uncover new challenges. It’s exciting, but also a little daunting at times.
Right now, we’ve reached a point where we can switch between apps, read out their menus, and expose the functions within those menus.
In essence, we’re recreating macOS actions inside our visionOS dwell pad, so people can control Mac software using only their eyes. This takes some finesse. For example, in the Mail app, the flow might look like: “New Message” → “Type in title field” → “Type in body field” → “Send.” These actions are normally handled with a mouse or keyboard, but with the dwell pad, each step needs to trigger a logical suggestion for what comes next.
Beyond the technical side, there’s also a UX challenge. You have to look at the dwell pad while controlling something on your Mac. That creates a disconnect we want to minimize.
You can see this issue illustrated in the short video below.