Textbook figures a blind student can touch, and that talk back.

A teacher turns a printed figure into a raised sheet and sets a phone above it. The student explores with one finger. Rest on a part and the phone says what it is. Ask a question out loud and the answer comes from that same figure and the teacher’s notes.

Watch the demo Get started View project View on GitHub

Move your pointer over the dots. The camera follows a fingertip in much the same way.Drag a finger over the dots. The camera follows a fingertip in much the same way.

A college science textbook can have a picture on every other page. Very few of those pictures are ever made into something a blind student can touch.

Mostly it comes down to time, because each one is made by hand.

1 to 5 hoursto make a single tactile image by hand
200 to 1,000 hoursto convert the images in one textbook
Up to 500images in one book

Figures are estimates from Ladner et al., “Automating Tactile Graphics Translation,” University of Washington, ASSETS 2005. About the UW Tactile Graphics Project

How it works

The teacher prepares a sheet once. After that, the student can come back to it as often as they like.

Making the sheet

What the teacher does, on a computer

  1. Upload a figureA PNG, a JPEG or a PDF page. Muse Spark reads it, lists the parts in a few words each, and plans each one for touch: an area, a line or a group.
  2. Each part gets an outlineSAM 3.1 outlines each part from points Spark placed on it, with no training on that figure. Code, not a model, lays out the sheet in millimeters: gaps, textures, point symbols and braille.
  3. Check each partFix an outline with two clicks, merge, rename, describe. A part becomes touchable only once the teacher checks it.
  4. Print itPublish, print at 100 percent, and raise the lines on swell paper or with puffy paint. Publishing also writes a file of where every part sits on the page, which Explore reads.

Reading the sheet

What the student does, with a phone and no screen

  1. Set the sheet under a phoneA sighted helper taps the four printed corner marks once. The camera finds the fingertip, and the corners map it to the page.
  2. Touch a partA texture tone on entering, then its name when the finger rests. Touch needs no internet once the sheet has loaded from the Mac.
  3. Ask about itHold the screen and talk. Muse Voice Transcribe listens, with the figure’s part names as hints. Muse Spark answers from the figure or the teacher’s notes, or sends the student to the teacher.

Built on muse-spark-1.3, sam-3.1 and muse-voice-transcribe-1.0 through the Meta Model API. No model redraws the figure: a student learns from the textbook’s own picture.

The choices behind it

Four corner marks, no tags

The sheet carries the figure, its braille, a 100 mm scale bar and four printed corner marks. A sighted helper taps the marks once on the phone. No tags or stickers go on the paper.

The teacher has the last word

Models make mistakes with diagrams. A part is touchable only once a teacher has checked it, and publishing refuses a sheet that breaks a tactile rule.

Rules come from published guidance

I have no background in tactile graphics, so the spacing, texture and braille rules come from published guidance (BANA, APH, tactilegraphics.org, the NPS and the ADA), each with its source next to it in config/tactile_rules.yaml: at least 3.2 mm between raised parts, at most five textures, point symbols of at least 6.4 mm.

Answers stay on the figure

An answer comes from the figure’s parts and labels or the teacher’s notes, under a strict schema. A question the figure does not answer gets one fixed sentence that sends the student to the teacher.

Geometry comes from code

Directions, neighbors and distances are computed in code. The language model never guesses them; only picture questions (shape, biggest, top) are answered from the figure’s image.

A phone first

An ordinary phone camera is enough today. A path for the camera in Meta’s glasses is on the list for after the hackathon.

Where it stands

One person, building for the Meta Global AI Developer Hackathon, October 26 to November 4, 2026. The first code was written on October 24, before the build window; the Official Rules allow pre-event work. As of November 4, every number below names the committed file it comes from.

Results

Measured results, with the file each one comes from
WhatResultFile
Part plan, touchable areas (live recordings)Heart 14 of 15, Japan regions 9 of 9, brain 5 of 6, flower 7 of 13eval/plan/RESULTS.md
S2 against graded reference masksHeart: 0 of 4 regressed (3 held, 1 improved). Flower: none of 5 heldeval/plan/RESULTS.md
Questions, live, source rightHeart 36 of 38, brain 35 of 35eval/qa/RESULTS.md
Picture questions (shape, biggest, top)Heart 5 of 6, brain 6 of 6; 3 of 6 each without the imageeval/qa/RESULTS.md
eval/qa/RESULTS_image_no_image.md
Release to first spoken word, Mac, livep50 5.0 s over 12 questions; two took 13.8 and 14.7 seval/qa/VOICE.md
Release to first spoken word, iPhoneNot measured: the latency film was cuteval/gate-b/README.md
Touch to tone, iPhoneNot measured: the latency film was cuteval/gate-b/README.md
Fingertip error on the 9-point sheet (Gate B, goal 4 mm)mean 7.93 mm, median 4.0 mm, p95 25.77 mm over 45 touches (eval/gate-b/RESULTS.md)eval/gate-b/README.md
Meta credits spent$5.42 of $150 (Nov 4)eval/live/ledger.jsonl

Not done yet

  • Not yet tested with blind students or teachers. No tester session or expert review happened before the deadline.
  • Latency on the phone was not measured: the latency film was cut, so touch to tone and question latency on the iPhone have no numbers; the Mac’s release to first spoken word is p50 5.0 s (eval/qa/VOICE.md).
  • Gate B fails its 4 mm target: on the demo iPhone, mean fingertip error 7.93 mm, median 4.0 mm, p95 25.77 mm over 45 touches. Most of the error came from target 9 (five touches, mean 27.41 mm), including the first touch at 77.96 mm. The next fix snaps each tapped corner to its printed L mark (eval/gate-b/RESULTS.md).
  • Voice is tested on synthetic speech only, in English (eval/qa/VOICE.md).
  • Questions need the network and are slow: 5.0 s at p50 on a Mac, against a 2.0 s budget in RULES.md.
  • A sighted helper sets the corners, and nothing detects the phone or the sheet moving afterward.
  • Small parts are hard: 6 of the flower’s 13 areas need the teacher (eval/plan/RESULTS.md).

The entry

Track
Indie developer
Problem statement
Push the Frontier
Models
Muse Spark, SAM 3.1, Muse Voice Transcribe

Every limit is listed in the README.md.