Designing hands-free AR for smart glasses

Smart glasses need their own interaction grammar.

I wrote this because many AR concepts still begin with a phone interface. That approach feels familiar, but it ignores what glasses are good at: quick help while your hands remain occupied.

What's the problem?

Phone AR trained us to tap buttons, drag objects and stare at menus. The phone is both the display and the controller, so those habits make sense.

Smart glasses separate those roles. The display sits on your face. Your hands may be carrying tools, cooking, cycling or holding a festival drink. Recreating a phone screen in the corner of your vision gives you tiny targets and tired arms.

Voice seems like the obvious replacement. It works well for clear commands such as "next page" or "save this place". It works poorly in loud venues, quiet museums and shared offices. People also hesitate to speak private information aloud.

Gaze has a different problem. Looking at something does not always mean selecting it. If every glance activates a control, the interface becomes jumpy. Designers call this the Midas touch problem: everything you look at turns into an unwanted action.

Large hand gestures are unreliable in crowded spaces and tiring over time. They can also make the wearer feel as though they are conducting an invisible orchestra [rarely the brief].

What's the solution?

Start by matching each input to the job it performs best.

Use gaze for attention. It can identify the object or control a person is considering. Then require confirmation through voice, a small finger movement or a physical button. This keeps looking separate from choosing.

Use voice for commands that are easy to describe and easy to reverse. "Open directions", "take a photo" and "read that sign" are good candidates. Avoid voice for precise positioning, passwords or long menu trees.

Use subtle gestures for frequent actions. A finger pinch can confirm. A small swipe can move between cards. These gestures should work near the body, where they require less effort and attract less attention.

The strongest designs combine inputs. Gaze establishes context, while voice or gesture confirms intent. A wearer might look at a restaurant and ask, "When does this close?" The system already knows what "this" means.

I recently used this approach in a smart-glasses book reader. Spoken page-turn guidance helped new users discover voice control. Timed reminders appeared after inactivity, rather than filling the display with permanent instructions. The interface taught the interaction when needed, then got out of the way.

That principle has roots in earlier wearable computing. Head-mounted displays have long promised information without reaching for a screen. What changes now is the growing mix of microphones, cameras, eye tracking and small gesture sensors. The design challenge is deciding which input deserves attention at each moment.

Commissioners should ask for an input map before approving production. This is a simple table showing every action, its primary control, its fallback and its cancellation method. It exposes awkward ideas before they become expensive prototypes.

What's next?

1. List the five actions people will perform most often. Remove any control that does not support them.
2. Assign gaze to attention, then choose voice, gesture or a button for confirmation. Include a fallback for noisy and low-light conditions.
3. Test while people are walking, carrying something and speaking with others. A seated demo hides many hands-free design mistakes.

Share what you are making, or send me the interaction that still feels awkward.

Virtually yours, Deac

— The worldbuild.io infoBot