AI Trends

NodLink uses AirPods head sensors to control the Mac cursor, open-sourced under the MIT license

sspaiUpdated Nov 18 16:09

AI summary

The author Uinui developed NodLink, which reads head pose data from AirPods to move the Mac cursor, and completes actions such as clicking by tilting the head and voice commands, all without turning on the camera. It is based on the CMHeadphoneMotionManager interface opened up starting with macOS 14; the sensor reads head orientation 50 times per second, uses a One Euro filter to suppress jitter, and corrects gyroscope drift in the background. The tool is open-sourced under the MIT license, requires macOS 14 or later and AirPods that support head tracking, and currently supports Mac only.

Why read this

AI score 60

The author open-sourced a tool that uses AirPods head sensors to control the Mac cursor, and explained how calibration, jitter suppression, and drift correction are handled.

Topics

Tags

Body · AI translation

Click the button below to copy the link

Your AirPods have actually known which way your head is facing all along

If you use AirPods, you've probably tried Spatial Audio: when you turn your head, the sound stays in its original direction, as if a speaker were placed in front of you.

This works because the AirPods contain a gyroscope and an accelerometer that constantly read the orientation of your head. This data was originally used only for Spatial Audio, but starting with macOS 14, Apple opened up an interface called CMHeadphoneMotionManager, which third-party apps can also read.

What I built, NodLink, is simple: since the earbuds know which way your head is facing, let the cursor follow it too.

The motion sensor in the AirPods reads head orientation 50 times per second

I hope it can help people who have difficulty using their hands—for example, people with physical disabilities or those who are bedridden for long periods. Using a computer is so natural for most people, but for them, moving a mouse or clicking a button can be very hard.

Why not use a camera

Apple's system actually has a built-in "Head Tracking" accessibility feature that uses the front camera to watch your face. But the camera approach has several unavoidable problems: you need light, you need to face the screen, and your face has to be in the frame. Dedicated head-controlled mice often cost over a thousand dollars and require wearing an extra reflective sticker or sensor.

NodLink only reads the sensors in the AirPods and never turns on the camera. So in theory, lying down, lying on your side, or having the lights off doesn't affect it. For people who are bedridden, I feel this point is quite important.

Lying down, on your side, lights off—all fine

What it can do

Turn your head left or right and the cursor moves left or right; raise or lower your head and the cursor moves up or down. After calibration, your head angle is converted directly into a position on the screen—wherever you look, the cursor is there.

Turn your head to point: wherever the extension of your head's orientation lands, the cursor is there

When looking at things, people rarely tilt their heads, so head tilt is especially suitable as a "button" and doesn't conflict with moving the cursor:

And all of these can be customized. For example, you can change left head tilt to "hold Fn for voice input" so you can speak to type; when watching videos, left head tilt can also mean "next video."

One action, multiple mappings

If you find head tilt inconvenient, you can also just speak: "click," "double click," "right click," "scroll up," "scroll down," "copy," "paste," "undo," "close window"... both Chinese and English work, and things like "click" or "command tab" are recognized too. Speech recognition uses the system's built-in on-device recognition.

There's also a small detail here: as soon as you start speaking, the cursor freezes, and after recognition finishes it executes at the frozen position. Otherwise, during those few tenths of a second of recognition, a slight head movement would make the cursor drift. When buttons are too small and hard to aim at with your head, you can also say "numbers": buttons near where your head is pointing that can be clicked are labeled with numbers 1–9, and saying the corresponding number clicks it; when executing a click, the cursor also automatically snaps to a nearby clickable button and draws a box around it to show you which one is selected.

If you want to turn your head to look elsewhere without the cursor running off, say "pause," and say "continue" to resume. Slowly tilting your head, or tilting while turning, won't be treated as a click. Press Esc, or touch a real mouse or keyboard, and control immediately returns to you.

What it's like in actual use

The images below are all screenshots from my demo video, recorded while wearing real AirPods and without my hands touching the keyboard or mouse. The small window in the lower right corner shows my own movements.

On the left of the console is the head tilt angle, below is turning left and right, and on the right is nodding up and down. Once you actually put them on, you'll find that Apple's sensor is really sensitive and the values are very stable; the entire head-control system is based on its data.

Live shot: the console displays head angles in three directions in real time

The little character below is one I drew myself. The milk frog doesn't move on its own—each of its actions comes from your head: nod and it smiles, and the faster you nod, the faster it smiles; when you stop, it stops too; when you turn your head, it even attacks.

Possessing the milk frog: nod and it smiles

I also made four practice scenarios in the console, all of the things people most often do on a computer, so you can get familiar with the controls here first.

Close a pop-up. Point your head at the button and say "click." The moment you start speaking the cursor stops first, and after recognition it clicks at the stopped position.

Close a pop-up: point your head at the button and say "click"

Scroll short videos. Tilt left for the next one, say "double click" to like, tilt right to open the comments.

Live shot: scrolling short videos

Reply to messages. Tilt your head left and don't straighten it—that's equivalent to holding Fn and speaking with the input method; straighten it to release, and the text goes on screen. Finally say "click" to send.

Reply to messages: tilt to hold and speak, straighten to enter text

Order food delivery. String together the previous methods: use voice to select dishes and choose a delivery time, tilt your head to say a note, and finally place the order.

Live shot: ordering food delivery, with the note "leave at the door"

To be honest, voice recognition occasionally mishears things. While recording the video, it once heard my "click" as "weather"; in that case nothing happens, and you just say it again.

Beyond the practice scenarios, I also tried it on the real Douyin web version: tilt left to go to the next video, without using my hands at all. (Other people's videos in the image have been blurred.)

Real Douyin web version: tilt to go to the next video

For debugging convenience and also for looks, I built a 3D scene for the console. In third-person view, you can see a little figure wearing headphones, and which screen the extension of the direction their head is facing lands on; switch to first-person, and it's your own desktop view.

If you're interested in the implementation, here are a few things I find relatively interesting.

Calibration uses the tip of your nose, not your eyes. After putting on the headphones, first aim the tip of your nose at the point in the center of the screen and hold for 2 seconds to set the "straight ahead." For better accuracy, you can do another path calibration of about 30 seconds: first look at the center and the four corners, then use the tip of your nose to follow the cursor along the four edges twice. You must use the tip of your nose here, because AirPods measure the orientation of your head, not your line of sight; just moving your eyes won't work.

Anti-shake. A person's head can't stay completely still, and if the cursor follows it directly it will keep shaking. NodLink uses a filter called One Euro: when the head is still, it suppresses shaking hard, and when you turn your head quickly, it tries not to lag.

Drift correction. AirPods don't have a compass, so the left-right direction can only be accumulated little by little from the gyroscope; over time it will gradually drift, and eventually you may have to turn your head to one side to see the center of the screen. Fortunately, when using a computer, people are generally facing the screen, so NodLink quietly and slowly pulls it back in the background. If it really drifts too much, just say "recenter" or press C.

Technically, the Mac side is a small program written in Swift (I call it the "Headphone Bridge"), responsible for reading headphone data, listening for voice commands, and controlling the system mouse; calibration, filtering, gesture recognition, and the 3D scene are all in a web console, written with Node.js and three.js.

How to install

You need a Mac running macOS 14 or later, and a pair of AirPods that support head tracking (3rd generation and later, Pro, Max). The specific installation steps are all in the GitHub README; just follow along.

After installation, follow the guide at the top of the page: connect the headphones → set straight ahead → start head control. The permissions and data used are all processed locally on your machine. If you don't have AirPods, you can directly open the online version and use the mouse to simulate head movement to get a feel for it first.

NodLink currently only supports Mac, and installation has a bit of a barrier: you have to open a terminal and install Node.js, which isn't very friendly to friends who aren't familiar with computers. Later I'd also like to make it an app that can be installed by double-clicking. Also, voice occasionally mishears things; as mentioned above, if it mishears, just say it again.

Actually, what I most want to do is mobile. Phone screens are small and buttons are dense, and it's also the device many people use most while lying down. But iOS doesn't allow one app to control other apps; pure software can't do it. So I plan to add a very cheap ESP32 board: the app on the iPhone reads the AirPods' head data and sends it to the ESP32, and the ESP32 then connects back to the same iPhone as a Bluetooth mouse, using the system's AssistiveTouch to move the cursor and click. But that's all for later.

NodLink is open source under the MIT license; welcome to try it out, and welcome to submit issues. If you yourself use a head-controlled mouse or other assistive devices, or have family or friends who need them, I'd especially love to hear your opinions. Anything that's awkward to use or operations that still can't be done, you can tell me.

> Follow Shaoshupai on Xiaohongshu to experience a wonderful digital life 🍃

> Practical, easy-to-use genuine software, presented to you by Shaoshupai 🚀 Accessibility AirPods Mac macOS App Recommendations App+1 This article's editor:

English text is being prepared.

4 fellow readers have already charged

Source

sspai