The phone connection

Set up the iPhone over a sheet

The Mac runs the API and serves the app over HTTPS on the local network. The iPhone opens it in Safari, watches the printed sheet from a mount, and follows the fingertip. This guide follows docs/DEMO.md.

What you need

  • The Mac running make demo-live, or make demo-mock for no Meta calls (Get started has the setup, the phone certificate and the trusted mkcert root).
  • The iPhone on the same Wi-Fi as the Mac.
  • A published sheet, printed at 100 percent, its lines raised on swell paper or with puffy paint.
  • A mount that holds the phone still above the sheet.

The phone loads the app and the hand model from the Mac, so it needs the local network, not the internet. Questions need the internet.

1. The device check

On the iPhone, open the device check. Use the Mac’s .local name first; make demo-live prints the addresses with the real name.

https://<your-mac>.local:5173/explore?check=1

Press “Check this device”. Every row should say OK before filming or teaching:

  • Secure page: HTTPS, so the camera and the microphone are allowed to ask.
  • Camera: the rear camera, looking down at the sheet.
  • Hand tracker: 10 frames a second or more.
  • Local voice: a voice that works offline.
  • Screen Wake Lock: “Held and released”. If it fails, set Auto-Lock to Never in Settings, Display and Brightness.
  • Microphone: opened, so spoken questions can work.
The device check page: Secure page OK, Camera needs a look because the test camera does not say which way it faces, Hand tracker OK at 20.0 frames a second on the GPU, Local voice OK, Screen Wake Lock OK, Microphone OK.
The device check in a headless test run with a fake camera, which is why the camera row needs a look. On the iPhone, every row should say OK.
  1. Print the sheet’s PDF at 100 percent (“Actual size”, not “Fit”). Measure the bar at the bottom with a ruler: it must be 100 mm.
  2. Raise the lines, then tape the sheet flat at two corners.
  3. Fix the phone above the sheet with all four corner marks in view. It must not move once the corners are set.
  4. Turn the overhead light on and keep window glare off the sheet.
  5. Turn the silent switch off and the volume up: Safari mutes the tones in silent mode.

3. Set the corners

This is the calibration. It maps the camera picture to millimeters on the page, and a sighted helper does it once.

  1. Open the sheet in Explore (the addresses make demo-live prints, or Author’s “Open version 1 in Explore” link) and press “Start exploring”.
  2. Tap the four printed corner marks on the camera view, in order: A top left, B top right, C bottom right, D bottom left. The app asks for each one: “A helper sets the corners. Tap corner A, top left.”
  3. Check that the drawn outline sits on the printed marks. If not, tap “Set corners” and tap the four marks again. On a keyboard, R sets them again.
Explore on a phone-sized screen: the heart's title, the line Press Start, a large Start exploring button, and the settings for who speaks and the pointing hand.
Explore opens with one large button, “Start exploring”.
Explore after Start on a phone-sized screen: the prompt A helper sets the corners. Tap corner A, top left. Below it the question box, the Set corners button and the camera view, here a green test picture.
After Start, Explore asks for corner A. The camera view here is Chrome’s test picture from a headless mock-mode run, not a sheet.

The screenshots show text from the heart sheet: Heart diagram-en.svg by ZooFari, CC BY-SA 3.0, via Wikimedia Commons. Resized and placed on a white background.

The corners are saved for that camera size, so a reload with the same camera and sheet needs no new taps. Nothing detects the phone or the sheet moving afterward: if either moves, set the corners again.

4. Explore

  • Touch a part: a texture tone on entering, then its name when the finger rests.
  • Hold anywhere on the screen to ask out loud, and let go when you are done. “More”, “repeat” and “where is the aorta” are handled on the phone.
  • With a keyboard: H says the help, M more, period repeats, W what is here, Escape stops speech, R sets the corners.
  • With the network off, touch still speaks; a question says “Questions need the network. Labels still work.”

If Safari breaks on the security headers

Signs: the device check’s hand tracker row fails or never finishes, Start does nothing, or a page stays blank. Stop the servers with Ctrl-C and start them without the Content-Security-Policy, then reload the page on the phone:

TT_CSP=off make demo-live