Skip to content

Prospero

Prospero App — Claude Context

iOS app that turns hand tracking (Vision framework) into MIDI output and/or real-time audio via a built-in polyphonic synthesizer. Two input modes (camera hands, touch), two output modes (MIDI, audio). Retro rosewood/brass/cream UI with Art Deco typography (PoiretOne).

Concept

"Your hands play music. Move them up and down to change pitch, extend fingers to control expression. Play into any DAW via MIDI, or use the built-in synth with effects."

Data Flow

Camera frame → HandTracker (Vision) → [HandInputSource]
                                           ↓
                                  PositionStabilizer (EMA + velocity clamp)
                                           ↓
                                      MIDIRouter (scale quantization, CC mapping)
                                       ↙            ↘
              MIDIController (CoreMIDI)         AudioEngine (synthesis)
                     ↓                                ↓
           External DAW / synth              Speaker / headphones
Touch input follows the same path via TouchTracker → [TouchInputSource].

Dependencies

All four SPM packages: Horatio (types), Enobarbus (tracking), Touchstone (DSP/effects), Ariel (audio engine + MIDI).

Input Layer

  • HandTracker detects up to 4 hands via VNDetectHumanHandPoseRequest, publishes [HandInputSource] on main thread
  • TouchTracker / TouchInputView handles multi-touch (up to 4 slots: touch0–touch3)
  • PositionStabilizer smooths positions (EMA α=0.6 for hands, velocity clamp at 0.15 screen units, stale-frame recovery after 5 rejections)
  • Input mode toggled in HUD: camera or tap

MIDI Layer

  • MIDIRouter.process() maps input positions to MIDI messages (notes, CC, pitch bend, velocity, depth)
  • MIDIController sends raw MIDI packets via MIDIReceived on a virtual source named "Prospero"
  • MIDIClockReceiver listens for external MIDI clock (24 ticks/quarter), publishes BPM
  • MIDILearnSession captures incoming CC/note for parameter assignment
  • Routes are rebuilt from scratch on every settings change via updateRoutes()
  • All hands default to the same user-selected channel (1–16); per-hand channel selection available in settings

Audio Layer

  • AudioEngine — lock-free polyphonic synth (44.1 kHz, up to 16 voices)
  • 4 waveforms: sine, sawtooth (PolyBLEP), triangle, square (PolyBLEP, 70% attenuation)
  • Portamento (glide rate 0–1), ADSR envelope (configurable attack/release)
  • LFO modulation: vibrato (±0.4 semitones) or tremolo (0–80% depth), switchable via CC 1
  • Stereo/mono configurable
  • Inline effects in render callback: Tone → Fuzz → Delay → Reverb
  • Tone: ToneEffect (one-pole LPF)
  • Fuzz: FuzzEffect (tanh saturation, per-sample drive smoothing)
  • Delay: FolioDSP DelayLine (Hermite interpolation)
  • Reverb: FolioDSP FeedbackNetwork
  • All effect parameters routed through lock-free command buffer to audio thread
  • Output mode toggled in HUD: audio or MIDI

Tier Gating

  • #if DEBUG returns .pro, release checks StoreManager.shared.isPro
  • StoreManager handles StoreKit 2: monthly + lifetime products, transaction listener, restore
  • Free: Pentatonic Major only, keys C/F/Bb only, 1 hand, sine waveform only, limited effects
  • Pro: all 12 scales, all 12 keys, up to 4 hands, all waveforms, full effects + vibrato/tremolo

Critical Implementation Notes

  • MIDIRouter.activeNotes is keyed by sourceID, not channel. Multiple inputs can share a MIDI channel without killing each other's notes. On source loss → automatic noteOff.
  • Hand-tracking dots and fret overlay must share the same coordinate space. Both use GeometryReader with .ignoresSafeArea() for full-screen coordinates.
  • positionToNote uses index = min(Int(floatIndex), numNotes - 1) — same zone math as NoteFretOverlay.fretBoundaries.
  • AudioEngine uses a lock-free ring buffer for main thread → audio thread commands. Never block the render callback.
  • Finger counting compares fingertip vs PIP joint (not MCP) for more deliberate extension detection.

Design Language

  • Colors: rosewood (#2E1F14), brass (#B8945C), cream (#F2E1B8), nickel-silver frets (#BFB8A6), phosphor green (oscilloscope)
  • Font: PoiretOne-Regular (Art Deco display)
  • Visual motifs: CRT oscilloscope with bloom/vignette, analog gauge dials, procedural wood grain, metallic wire fret lines
  • Status indicators: connected (solid/glow green), disconnected (solid/glow red)

Files

File Purpose
ProsperoApp.swift @main entry point, --uitesting / --show-quotes launch args
ContentView.swift Master orchestrator: tracking, routing, audio, state, scene lifecycle
HUDView.swift Heads-up display: mode toggles, settings/help buttons, velocity/mod dials
SettingsView.swift Settings sheet: scale, key, note range, MIDI channels, visual toggles, pro upsell
AudioControlsPanel.swift Audio mode controls: oscilloscope, waveform picker, effect knobs
OnboardingView.swift 3-page full-screen onboarding (welcome → how it works → get started)
ProUpgradeView.swift Pro tier upgrade sheet with feature comparison + StoreKit purchase flow
NoteFretOverlay.swift Metallic fret lines + smart note label circles (root, 5th, 3rd, gap-fill)
OscilloscopeView.swift CRT phosphor-green waveform display with bloom + vignette
DialView.swift Vintage analog gauge (needle sweep 210°–510°), InteractiveDialView for knobs
CameraView.swift UIViewRepresentable wrapping AVCaptureVideoPreviewLayer
TouchInputView.swift UIViewRepresentable multi-touch layer (4 slots → TouchInputSource)
WoodGrainView.swift Procedural rosewood grain (seeded RNG, deterministic)
ArcShape.swift Custom Shape for dial arcs
BluetoothMIDIView.swift Wraps CABTMIDICentralViewController for BT MIDI pairing
TierConfig.swift Single source of truth for tier state and free-tier limits
StoreManager.swift StoreKit 2 integration: product loading, purchase, restore
AppTypes.swift InputMode, OutputMode enums, midiNoteName() helper
DesignTokens.swift ProsperoTokens design system (rosewood, brass, cream, phosphor green, PoiretOne font)
ProsperoQuotes.swift Array of Shakespeare quotes from The Tempest, shown at launch

Tests

UI Tests (ProsperoUITests)

~10 XCUITests covering HUD interactions, settings, onboarding, and mode switching. Use --uitesting launch arg to skip onboarding/quotes.

xcodebuild test -scheme Prospero -destination 'platform=iOS Simulator,name=iPhone 17 Pro' -only-testing:ProsperoUITests

All interactive elements have .accessibilityIdentifier() for testability (hud.settings, settings.done, etc.).

Rules

  • Never block the render callback — use the command buffer for main→audio thread communication.
  • Hand-tracking dots and fret overlay must use matching coordinate spaces (both .ignoresSafeArea()).
  • The MIDI virtual source name "Prospero" is user-visible — do not change.
  • @ObservationIgnored + _rt prefix for all audio-thread state.