Skip to content

Architecture

Prospero — Claude Context

Naming

  • Prospero — app name, used in app header UI, App Store, bundle, MIDI source name

Project Overview

iOS app that turns hand tracking (Vision framework) into MIDI output and/or real-time audio via a built-in polyphonic synthesizer. Two input modes (camera hands, touch), two output modes (MIDI, audio). Retro rosewood/brass/cream UI with Art Deco typography (PoiretOne).

App Trilogy (Shakespeare / Tempest universe)

  • Prospero — melodic instrument, hands → notes + CC, built-in audio + MIDI out, free/pro tiers
  • Caliban — expressive MIDI controller, hands/body/face/relationships → CC + notes, MIDI out only
  • Hecate — percussion instrument, body/face tracking → audio output
  • All three share the four SPM packages (Horatio, Enobarbus, Touchstone, Ariel)

Data Flow

Camera frame → HandTracker (Vision) → [HandInputSource]
                                           ↓
                                  PositionStabilizer (EMA + velocity clamp)
                                           ↓
                                      MIDIRouter (scale quantization, CC mapping)
                                       ↙            ↘
              MIDIController (CoreMIDI)         AudioEngine (synthesis)
                     ↓                                ↓
           External DAW / synth              Speaker / headphones
Touch input follows the same path via TouchTracker → [TouchInputSource].

Architecture

Input Layer

  • HandTracker detects up to 4 hands via VNDetectHumanHandPoseRequest, publishes [HandInputSource] on main thread
  • TouchTracker / TouchInputView handles multi-touch (up to 4 slots: touch0–touch3)
  • PositionStabilizer smooths positions (EMA α=0.6 for hands, velocity clamp at 0.15 screen units, stale-frame recovery after 5 rejections)
  • Input mode toggled in HUD: camera or tap

MIDI Layer

  • MIDIRouter.process() maps input positions to MIDI messages (notes, CC, pitch bend, velocity, depth)
  • MIDIController sends raw MIDI packets via MIDIReceived on a virtual source named "Prospero"
  • MIDIClockReceiver listens for external MIDI clock (24 ticks/quarter), publishes BPM
  • MIDILearnSession captures incoming CC/note for parameter assignment
  • Routes are rebuilt from scratch on every settings change via updateRoutes()

Audio Layer

  • AudioEngine — lock-free polyphonic synth (44.1 kHz, up to 16 voices)
  • 4 waveforms: sine, sawtooth (PolyBLEP), triangle, square (PolyBLEP, 70% attenuation)
  • Portamento (glide rate 0–1), ADSR envelope (configurable attack/release)
  • LFO modulation: vibrato (±0.4 semitones) or tremolo (0–80% depth), switchable via CC 1
  • Stereo/mono configurable
  • All effects are InlineEffects processed in the render callback: Tone → Fuzz → Delay → Reverb
  • Tone: ToneEffect (one-pole LPF, proven stable at all cutoffs; DSPComponent.makeTone/SVF abandoned — Chamberlin SVF unstable near Nyquist and per-sample bridge didn't propagate parameter changes)
  • Fuzz: FuzzEffect (tanh saturation, per-sample drive smoothing via targetDrive + slew rate 0.002)
  • Delay: FolioDSP DelayLine wrapped in MixedDSPComponent (DSPComponent.makeDelay, per-sample mix smoothing). DelayLine supports Linear and Hermite (cubic) interpolation modes (parameter index 2, default Hermite)
  • Reverb: FolioDSP FeedbackNetwork (DSPComponent.makeReverb)
  • Legacy ReverbEffect and DelayEffect remain in Touchstone but are unused
  • Effect parameter routing: All effect parameters (tone, fuzz, delay mix, reverb mix) are routed through the lock-free command buffer to the audio thread. No main-thread fxChain.setParameter() calls — eliminates data races and click artifacts. rtEffectsL/rtEffectsR arrays store audio-thread effect references, set in start(), cleared in stop().
  • Signal Graph + Recipe system (Touchstone library, not yet wired into Prospero's audio path):
  • SignalGraph = DAG of GraphNodes connected by GraphEdges, with topological sort (Kahn's algorithm)
  • Recipe = JSON-serializable definition: graph + metadata + ExposedParameter mappings
  • RecipeEngine = mono RT-safe runtime: instantiates DSPComponents, serial chain fast path, general graph fan-in, pre-allocated buffers, built-in Mixer/Crossfade node (2-input weighted blend by Balance parameter, handled at engine level without DSPComponent)
  • RecipeLoader = JSON load/save, loadAll(from: directoryURL) for batch loading
  • Modulation routing: ModulationRoute connects modulator node output → target node parameter; block-rate (once per process call, not per sample)
    • ModulationCurve: linear, exponential (x²), logarithmic (√x), sCurve (smoothstep)
    • ModulationPolarity: bipolar (-1..1) or unipolar (0..1)
    • Multi-output modulators (ChaosAttractor, KnotModulator) read x/y/z via FDSPSnapshot field
    • Modulator-only nodes (no audio edges) are ticked for state but excluded from audio path
    • setExposedParameter updates modulation route base values to prevent drift
  • Starter recipes bundled as SPM resources in Packages/Touchstone/Sources/Touchstone/Recipes/: miranda (shimmer reverb), falstaff (warm saturation), puck (chaos glitch), iago (creeping corruption, modulated), miranda-modulated (breathing shimmer, modulated), ariel (airy shimmer), caliban (raw distortion), hamlet (dual-path crossfade), lady_macbeth (dark ambience), lear (stormy chaos), mercutio (sharp flanger), oberon (spectral freeze), ophelia (gentle chorus), prospero (warm reverb), the_witches (eerie modulation), titiana (phaser shimmer), warm_bus (analog console channel strip)
  • RecipeLoader.bundledRecipesURL — runtime accessor for the bundled recipes directory via Bundle.module
  • Recipe JSON parameter names must use exact FolioDSP bridge names (capitalized, with spaces, e.g. "Mod Rate", "Grain Size")
  • RecipeEngine.resetAllNodes() — resets all DSP components to initial state (used on voice steal/allocation)
  • SynthRecipe system (Touchstone library):
  • SynthRecipe = voice graph + optional effects graph + note bindings + polyphony config
  • VoiceSynthEngine = polyphonic runtime: N voice RecipeEngines + optional post-mix effects pair
  • PolyphonyConfig includes ADSR: attackTime, decayTime, sustainLevel, releaseTime (seconds/level)
  • SynthRecipe.effectsGraph: SignalGraph? — optional post-mix effects, runs once on summed stereo output
  • SynthRecipe.effectsExposedParameters / effectsModulations — controls for the effects graph
  • Signal chain: voices → sum + polyGain → effectsGraph L/R → inputLevel → autoGain → limiter → output
  • Voice reset: resetAllNodes() called on fresh allocation and voice steal (not retrigger)
  • Bundled synth recipes in Packages/Touchstone/Sources/Touchstone/SynthRecipes/: warm_pad, simple_lead, sub_bass
  • SynthRecipeLoader — parallel to RecipeLoader for synth recipes
  • Output mode toggled in HUD: audio or MIDI

Tier Gating (TierConfig.swift)

  • #if DEBUG returns .pro, release checks StoreManager.shared.isPro
  • StoreManager handles StoreKit 2: monthly + lifetime products, transaction listener, restore
  • Free: Pentatonic Major only, keys C/F/Bb only, 1 hand, sine waveform only, limited effects
  • Pro: all 12 scales, all 12 keys, up to 4 hands, all waveforms, full effects + vibrato/tremolo

Critical Implementation Notes

  • MIDIRouter.activeNotes is keyed by sourceID, not channel. Multiple inputs can share a MIDI channel without killing each other's notes. On source loss → automatic noteOff.
  • The hand-tracking dots and fret overlay must share the same coordinate space. Both use a GeometryReader with .ignoresSafeArea() to position in full-screen coordinates. Without this, the safe area inset creates a ~1-zone offset between the dots and the fret lines.
  • positionToNote uses index = min(Int(floatIndex), numNotes - 1) — the same zone math as NoteFretOverlay.fretBoundaries. No position offset is needed when coordinate spaces match.
  • AudioEngine uses a lock-free ring buffer for main thread → audio thread commands. Never block the render callback.
  • Finger counting compares fingertip vs PIP joint (not MCP) for more deliberate extension detection.
  • See docs/note-mapping-reference.md for all mapping formulas.

MIDI Channels

  • All hands default to the same user-selected channel (1–16)
  • Per-hand channel selection available in settings (4 channel pickers for 4 hands)

File Guide — App Target (Prospero/)

File Purpose
ProsperoApp.swift @main entry point, --uitesting / --show-quotes launch args
ContentView.swift Master orchestrator (636 lines): tracking, routing, audio, state, scene lifecycle
HUDView.swift Heads-up display: mode toggles, settings/help buttons, velocity/mod dials
SettingsView.swift Settings sheet: scale, key, note range, MIDI channels, visual toggles, pro upsell
AudioControlsPanel.swift Audio mode controls: oscilloscope, waveform picker, effect knobs
OnboardingView.swift 3-page full-screen onboarding (welcome → how it works → get started)
ProUpgradeView.swift Pro tier upgrade sheet with feature comparison + StoreKit purchase flow
NoteFretOverlay.swift Metallic fret lines + smart note label circles (root, 5th, 3rd, gap-fill)
OscilloscopeView.swift CRT phosphor-green waveform display with bloom + vignette
DialView.swift Vintage analog gauge (needle sweep 210°–510°), InteractiveDialView for knobs
CameraView.swift UIViewRepresentable wrapping AVCaptureVideoPreviewLayer
TouchInputView.swift UIViewRepresentable multi-touch layer (4 slots → TouchInputSource)
WoodGrainView.swift Procedural rosewood grain (seeded RNG, deterministic)
ArcShape.swift Custom Shape for dial arcs
BluetoothMIDIView.swift Wraps CABTMIDICentralViewController for BT MIDI pairing
TierConfig.swift Single source of truth for tier state and free-tier limits
StoreManager.swift StoreKit 2 integration: product loading, purchase, restore
AppTypes.swift InputMode, OutputMode enums, midiNoteName() helper
DesignTokens.swift ProsperoTokens design system (rosewood, brass, cream, phosphor green, PoiretOne font)
ProsperoQuotes.swift Array of Shakespeare quotes from The Tempest, shown at launch

File Guide — Workbench Target (Workbench/)

Internal developer tool for loading, auditioning, and tweaking Touchstone recipes and synth voices. Depends on Touchstone + Horatio (no Ariel, Enobarbus, or camera/Bluetooth entitlements). Stock SwiftUI, multi-platform (iOS + macOS) NavigationSplitView. Two modes: Effects (signal gen → recipe → limiter) and Synth (MIDI → VoiceSynthEngine → limiter).

File Purpose
WorkbenchApp.swift @main entry point, macOS defaultSize(1000×700)
ContentView.swift NavigationSplitView: sidebar recipe picker + detail area; switches between Inspector View and Graph View via state.viewMode; synth mode detail content
Audio/WorkbenchAudioEngine.swift @Observable AVAudioEngine wrapper: dual render path (effects mode: signal gen → RecipeEngine L/R; synth mode: MIDI → VoiceSynthEngine → stereo output) → auto-gain → brickwall limiter → waveform ring buffer; CPU metering, live audio input, per-node profiling, recipe bypass, runtime sample rate negotiation, handleMIDIEvent() → SPSCRingBuffer<SynthCommand>
Audio/SignalGenerator.swift Phase-accumulator signal generator (15 types: basic, tonal, transient, modulated, sample playback, live)
Audio/MIDIInputListener.swift Persistent CoreMIDI listener: connects all sources, handles hot-plug, optional channel filter (omni default), forwards ParsedMIDIEvents to callback
Models/NodeLayout.swift Node position for graph canvas layout
Models/RecipeDocument.swift FileDocument wrapper for recipe JSON export/import
Models/SignalType.swift Signal type enum with 15 cases (incl. live), grouped sections, usesFrequency/usesSamplePicker/requiresLiveInput properties
Models/SynthesizedSample.swift SynthesizedSample enum: chord (C major), drums (kick-snare), melody (C scale) — generates loopable buffers
Models/WorkbenchState.swift @Observable app state: WorkbenchMode (.effects/.synth), recipe loading, selection, synth recipe management, MIDI listener lifecycle, graph editing, A/B comparison, ViewMode enum (.graph/.inspector)
Views/TransportView.swift Play/stop, mode picker (Effects/Synth), signal menu picker (effects) or MIDI status + voice count (synth), view mode picker, input/limiter controls, bypass, auto-gain, CPU meter, live input, profiling, file importer
Views/RecipePickerView.swift Sidebar recipe list (effects mode) with selection highlight, New Recipe / Clear Selection buttons, Edit context menu
Views/SynthRecipePickerView.swift Sidebar synth recipe list (synth mode) with node/binding counts, New/Clear/Edit actions, context menu
Views/RecipeInfoView.swift Recipe metadata: name, category badge, description, node/edge/mod/control counts
Views/ParameterPanelView.swift Exposed parameter sliders
Views/WaveformView.swift Canvas waveform display (green on black, 30fps polling)
Views/GraphEditorView.swift Main graph editor: HSplitView with ZStack canvas (left) + SelectedNodePanel (right, 300px, shows on node selection); toolbar (add/modulate/info/A/B/save/export)
Views/EdgeCanvasView.swift Canvas drawing audio edges (solid Bezier) and modulation routes (dashed orange)
Views/NodeBoxView.swift Individual graph node: role-colored rounded rect, input/output ports, context menu delete
Views/ComponentBrowserView.swift Searchable component picker grouped by ComponentRole (69 components)
Views/AddModulationSheet.swift Modulation route creation: source/target/depth/curve/polarity pickers
Views/RecipeMetadataEditor.swift Recipe metadata editor: id, name, category, description fields
Views/SnapshotView.swift Live FDSPSnapshot display polling at 15 Hz (float/bool/vec3 field rendering)
Views/OnScreenKeyboardView.swift 2-octave on-screen piano keyboard (synth mode only): octave shift, velocity control, drag-to-play glissando, sends noteOn/noteOff via sendSynthCommand()
Views/PerformanceMeterView.swift CPU load bar with peak marker and glitch count indicator
Views/NodeInspectorComponents.swift Shared components used by both Inspector View and Graph View: NodeVisualization (snapshot polling at 15 Hz + visualization dispatch), NodeParameterSlider, ConnectionsSection
Views/NodeInspectorView.swift Per-node DisclosureGroup: uses shared components from NodeInspectorComponents, bypass toggle, per-node timing display
Views/Visualizations/SnapshotReader.swift SnapshotField model + SnapshotReader byte-level field extraction from FDSPSnapshot, extractFloatArray helper
Views/Visualizations/TransferCurveView.swift 16-point transfer curve + signal dot for Overdrive, Tape Saturation, Tube Warmth, Transformer Color
Views/Visualizations/GainReductionView.swift Vertical GR meter + I/O levels for Compressor, BrickwallLimiter, Sag
Views/Visualizations/FrequencyResponseView.swift Biquad/SVF frequency response curve (Audio EQ Cookbook math, 128 log-spaced points)
Views/Visualizations/StaircaseView.swift Smooth sine + purple quantized staircase overlay for BitCrush
Views/Visualizations/WavetableMorphView.swift 32-point waveform preview for Oscillator
Views/Visualizations/AnalysisReadoutView.swift PitchReadoutView (note name, cents, confidence, history trace) + OnsetReadoutView (flash, flux meter, band indicators)
Views/Visualizations/ComponentVisualizationView.swift Dispatcher: maps component name → specialized visualization, falls back to raw field dump

Workbench Signal Types

Section Types Description
Basic sine, sawtooth, square, noise, silence Standard test tones (use frequency slider)
Tonal gatedBurst, pwm, fm Gated 200ms/500ms burst, PWM sweep 10–90% duty, 2-op FM with mod index sweep
Transient impulse, pluck, arpeggio Single-sample click, Karplus-Strong string, C major arpeggio at 3 notes/sec
Modulated sweep, sampleAndHold Log chirp 80–8000 Hz / 3s, random pitch jumps at 4 Hz
Playback sample Synthesized samples (chord/drums/melody) or user audio file import
Live live Microphone input via AVAudioSinkNode, with feedback protection

Workbench Signal Chain (Effects Mode)

SignalGenerator / Live Input (AVAudioSinkNode flat buffer) → Input Level (0 dBFS default)
                       ↓
              RecipeEngine L/R (Touchstone graph processing)
                       ↓
              Auto-Gain Normalization (off by default; RMS-based, -18 dBFS target, ±12 dB range)
                       ↓
              BrickwallLimiter L/R (always on, -6 dBFS default ceiling)
                       ↓
              Feedback Mute (live input + no headphones → silence)
                       ↓
              Waveform ring buffer → AVAudioSourceNode output

Workbench Signal Chain (Synth Mode)

MIDI Keyboard → MIDIInputListener (CoreMIDI, persistent)
    → SPSCRingBuffer<SynthCommand> (lock-free, main→audio)
        → VoiceSynthEngine (audio thread)
            ├─ Voice 0: RecipeEngine L/R (clone of voice graph)
            ├─ Voice 1: RecipeEngine L/R
            └─ Voice N: RecipeEngine L/R
                Each voice: frequency + amplitude envelope + stereo pan
            → Sum voices → Input Level
                → Auto-Gain → BrickwallLimiter → Waveform → Output
- Input level: -40..0 dB slider, default 0 dBFS (full scale — lets effects work with a strong signal). Applied before recipe processing. - Auto-gain: Off by default (hear raw effect output). RMS measurement over 500ms window, per-sample slew (rate 0.001), max ±12 dB correction. Resets accumulator on recipe change but keeps current gain to avoid spikes. - Brickwall limiter: Separate L/R DSPComponent("BrickwallLimiter") instances (FolioDSP kernels have internal state). Never bypassable. GR metering via FDSPSnapshot with cached field offset, throttled reads (~23ms). - CPU metering: mach_absolute_time() timing around render callback. Smoothed load (0–1+), peak hold (~3 sec decay), glitch count (callbacks > 100% deadline). Transferred to UI via updateWaveformSamples(). - Per-node profiling: RecipeEngine.profilingEnabled flag adds per-node mach_absolute_time() timing. Separate profiled processing paths to avoid overhead when disabled. Toggle in TransportView, per-node timing in NodeInspectorView. - Live audio input: AVAudioSinkNode connected to inputNode writes to a flat buffer read by AVAudioSourceNode in the same IO render cycle — no ring buffer needed. Uses setPreferredSampleRate(44100) and setPreferredIOBufferDuration(0.005) for ~5ms IO buffers. Runtime sample rate (_runtimeSampleRate) negotiated from AVAudioSession (iOS) or engine output format (macOS), used everywhere instead of hardcoded 44100. AVAudioSession switches to .playAndRecord on iOS. Feedback protection: iOS mutes output without headphones; macOS shows one-time warning. Route change notifications update protection state. Note: installTap was abandoned because iOS batches tap delivery into OS-determined chunks (~100ms) regardless of the bufferSize hint, making low-latency impossible. - @ObservationIgnored: All audio-thread state uses this instead of nonisolated(unsafe) to avoid Swift 6.2 warnings while opting out of observation tracking. Naming convention: prefix with _rt for audio-thread variables to avoid collisions with @Observable backing stores.

SPM Packages

Package Module Depends on Purpose
Packages/Horatio Horatio — InputSource protocol, all concrete source types, MIDI types, Scale enum (12 scales), MIDIPacketParser, ParsedMIDIEvent, midiToHz()
Packages/Enobarbus Enobarbus Horatio HandTracker, BodyTracker, FaceTracker, CameraManager, PositionStabilizer, RelationshipProcessor, DepthTracker, BodyCalibration
Packages/Touchstone Touchstone Horatio, FolioDSPBridge AudioEffect protocol, FXChain, DSPComponent, MixedDSPComponent, BridgeAdapter, ComponentRegistry, ComponentDescriptor, ParameterDescriptor, ComponentTypes, SignalGraph, GraphNode, GraphEdge, Recipe, ExposedParameter, RecipeLoader, RecipeEngine, RecipeError, ModulationRoute, ModulationCurve, ModulationPolarity, RTHelpers (rtOnePole, rtAsymmetricSlew, rtSpikeGain), SPSCRingBuffer, SynthRecipe, NoteBinding, PolyphonyConfig, SynthCommand, VoiceSynthEngine, SynthRecipeLoader; legacy: ReverbEffect, DelayEffect, ToneEffect, FuzzEffect
Packages/Ariel Ariel Horatio, Touchstone AudioEngine, MIDIRouter, MIDIController, MIDIClockReceiver, MIDILearnSession (re-exports MIDIPacketParser from Horatio)

All packages target iOS 16+ / macOS 13+ (Touchstone and Ariel require macOS 13+ due to FolioDSPBridge dependency). All have DocC catalogs (Sources/<Module>/<Module>.docc/) with landing pages and /// doc comments on all public APIs. Build docs with ⌃⇧⌘D in Xcode or xcodebuild docbuild -scheme <Module> -destination generic/platform=iOS.

Tests

Unit Tests (per-package)

Package Test files
Horatio MIDIRouteTests, InputSourceTests, ScaleTests
Enobarbus HandTrackerTests, BodyTrackerTests, FaceTrackerTests, DepthTrackerTests, RelationshipProcessorTests, PositionStabilizerTests
Touchstone FXChainTests, ToneEffectTests, FuzzEffectTests, DSPComponentTests, SignalGraphTests, RecipeTests, RecipeEngineTests, RecipeLoaderTests, ModulationRouteTests
Ariel MIDIRouterTests, MIDIClockReceiverTests, AudioEngineTests, MIDILearnSessionTests, MIDIPacketParserTests

UI Tests (ProsperoUITests)

~10 XCUITests covering HUD interactions, settings, onboarding, and mode switching. Tests use --uitesting launch arg to skip onboarding/quotes.

When to run: Before merging UI-facing branches — especially changes to HUDView, SettingsView, OnboardingView, or ContentView layout/presentation. Don't run on every change; a full suite takes ~4 minutes in the simulator.

# Full UI test suite (~4 min)
xcodebuild test -scheme Prospero -destination 'platform=iOS Simulator,name=iPhone 17 Pro' -only-testing:ProsperoUITests

# Single test (fastest feedback)
xcodebuild test -scheme Prospero -destination 'platform=iOS Simulator,name=iPhone 17 Pro' -only-testing:ProsperoUITests/ProsperoUITests/testHUDSettingsButtonOpensSheet

Accessibility identifiers: All interactive elements tested have identifiers (hud.settings, settings.done, etc.). When adding new interactive UI, add an .accessibilityIdentifier() for testability.

Design Language

  • Colors: rosewood (#2E1F14), brass (#B8945C), cream (#F2E1B8), nickel-silver frets (#BFB8A6), phosphor green (oscilloscope), wood grain variants
  • Font: PoiretOne-Regular (Art Deco display)
  • Visual motifs: CRT oscilloscope with bloom/vignette, analog gauge dials, procedural wood grain, metallic wire fret lines
  • Status indicators: connected (solid/glow green), disconnected (solid/glow red)

Docs

File Content
docs/note-mapping-reference.md Note mapping formulas
docs/plan-docc-documentation.md DocC documentation plan
docs/plan-midi-learn.md MIDI learn feature design
docs/plan-stereo-mono-output.md Stereo/mono audio output design
docs/plan-vision-body-face-lidar.md Body/face/depth tracking architecture (now largely in Caliban)
docs/analog-character.md DelayLine Hermite interpolation + ConsoleBus component design