# Listen relay + stamp spec (v2, 2026-11-18)

Rough but binding. The live stream (`sender.py`, `server.py`, the web listener), the
receiver AU (`au/ListenReceiver`) and kelushael's take-stamper (`/workspace/takestamp`)
all speak this. Little-endian unless noted.

## 1. Relay endpoint + auth
Base: `https://marcusx.fun/listen` (Caddy `handle_path /listen/*` -> 127.0.0.1:19350, `flush_interval -1`).
One fixed session ("main"). One fixed link. Password `[redacted]` is the only gate. No other tokens.

| what | how |
|---|---|
| login | `POST /api/login` `{"password":"[redacted]"}` -> `Set-Cookie: listen_s=...` (HMAC of expiry, in memory, not written down) |
| or | header `X-Listen-Password: [redacted]` on any request below (used by the AU and scripts) |
| send | WebSocket `/ws?role=send` (binary frames = section 2, text = JSON control) |
| listen (browser) | WebSocket `/ws?role=listen&mode=<mode>&latency=<sec>` |
| listen (AU / curl) | `GET /api/stream?mode=<mode>` -> endless `application/octet-stream` of section-2 frames, back to back. Through Caddy this body is HTTP chunked; clients (AU via libcurl/JUCE, curl, scripts) must dechunk. |
| state | `GET /api/me` (401 until signed in), `GET /api/health` (open, no audio) |
| AU download | `GET /au/` (signed in) |

Modes (chosen per listener; relay converts per listener in memory):
- `pcm` = relay passes the sender's frames untouched (type 3 f32 or 2 s24). Bit-perfect. No chirp.
- `pcm24` = 48 kHz s24, TPDF-dithered from f32 only if the source is float; chirp on.
- `mid` = 24 kHz s24 mono, windowed-sinc polyphase (Kaiser, scipy `resample_poly`) + chirp.
- `low` = 16 kHz s24 mono, same resampler + chirp.
- `flac` = FLAC (type 4): 24-bit, source rate and channel count, TPDF-dithered from f32 like pcm24.
  Chirp is added before encoding. Each payload is a complete standalone FLAC stream
  (`fLaC` + STREAMINFO + one 20 ms frame) so a listener can join mid-stream. Compression level 0.
- `opus64|opus96|opus128|opus256` = Opus 20 ms packets (type 5, 48 kHz, libopus, chirp added before encoding).
  Decoded audio is late by the encoder lookahead (312 samples); receivers subtract it from chirp/stamp math.
- `opus` = adaptive per listener: starts at 128 kbps; the relay steps that listener down one tier when its
  own relay queue exceeds 0.25 s and back up after 20 s under 0.05 s. All four encoders stay warm, so switches are seamless.
- Latency: listener picks 0.05-2.0 s (default 0.2). Relay never holds more than 2.0 s per listener;
  older frames are dropped and the next frame carries DISCONT.

Store-nothing: audio exists only in RAM for the hop. No files, recordings, audio logs or history on the relay.
The producer's side writes the aligned take while (and only while) the producer is recording (section 8).

## 2. Frame format ("LSTN" v2), 56-byte header
| off | size | field |
|---|---|---|
| 0 | 4 | magic `0x4C53544E` **big-endian** ("LSTN") |
| 4 | 1 | type: 1=legacy s16 (24-byte header, v1), 2=PCM s24le, 3=PCM f32le, 4=FLAC frame, 5=Opus packet, 6=CTRL (side-channel lock/nudge, no audio), 7=TRANSPORT (record link, JSON payload, no audio; section 8) |
| 5 | 1 | channels |
| 6 | 2 | seq (u16, wraps) |
| 8 | 4 | rate (Hz of the decoded audio) |
| 12 | 4 | nframes (decoded sample frames per channel) |
| 16 | 4 | stamp_us high 32 bits (sender wall clock, UNIX us, at first sample) |
| 20 | 4 | stamp_us low 32 bits |
| 24 | 2 | hdr_len (56) - readers MUST skip hdr_len bytes to reach the payload |
| 26 | 2 | flags: b0 MARKER, b1 PLAYING (song_pos valid), b2 BITPERFECT, b3 DISCONT |
| 28 | 8 | sample_index u64: index of first frame on the stream timeline, at `rate` |
| 36 | 8 | song_pos_ns i64: sender song position of first sample in ns; -1 = unknown |
| 44 | 4 | crc32 (zlib/IEEE) of the payload bytes |
| 48 | 2 | marker_off: frame offset of chirp start inside this frame, 0xFFFF = none |
| 50 | 2 | tier id (0 pcm,1 pcm24,2 mid,3 low,4 flac,5 opus-64,6 -96,7 -128,8 -256) |
| 52 | 4 | payload_len (bytes). Frame size = hdr_len + payload_len (lets byte streams split frames) |
| 56 | .. | payload: PCM interleaved, or one FLAC frame, or one Opus packet (48 kHz decode, 20 ms) |

Take Stamp (`/takestamp`) splits PCM type 2/3 as `hdr_len + nframes * channels * (3 or 4)`,
which equals `hdr_len + payload_len` for those types. Listen byte streams should split on
`hdr_len + payload_len` so type 4/5 (compressed) frames parse correctly.

Sender frames are 20 ms (960 frames @ 48 kHz). Sender default payload: type 3 f32, 48 kHz, 2 ch,
exactly CoreAudio's float capture (BITPERFECT set). `--pcm24` sends type 2 with TPDF dither.
Legacy type 1 still decodes in the web listener.

Null test / CRC: in `pcm` mode the receiver recomputes crc32 over the received payload and compares
with the header (badge "bit-perfect" when every frame matches). The test harness also nulls
the received samples against what was sent (difference must be exactly 0).

## 3. Chirp lock signal
- Log chirp, 16 ms, amp 0.03 (-30 dBFS), 1 ms raised-cosine fades.
- Band scales with rate: `f0 = rate/6`, `f1 = rate/3` (48k: 8->16 kHz, 24k: 4->8 kHz, 16k: 2.67->5.33 kHz),
  always below the resampler cutoff, so it survives every tier without aliasing.
- Inserted by the relay into derived tiers (pcm24/mid/low/flac/opus) at every sample_index that is a
  multiple of `2*rate` (every 2.0 s on the stream timeline). Never inserted in `pcm` (bit-perfect).
- Frame with the chirp has MARKER set and marker_off = its offset.
- Receivers high-pass (2nd order @ 0.75*f0), normalized cross-correlate with the same kernel,
  parabolic sub-sample peak; `lock_err = detected - (sample_index+marker_off)` in samples. A
  receiver may subtract the known kernel at the detected spot (the AU does) so it is not recorded.
- Kernel formula (Python and JS and C++ must match): phase accumulates `2*pi*f0*exp(k*i/rate)/rate`,
  `k = ln(f1/f0)/((n-1)/rate)`, `n = round(rate*0.016)`.

## 4. Stamps (song position)
- `song_pos_ns` = position in Marcus's song (Logic timeline) of the frame's first sample, in ns
  (rate-independent; convert with `round(song_pos_ns*rate/1e9)`).
- Sender gets it from a stamp source. Any process (takestamp, a MIDI/MTC bridge) sends UDP JSON to
  `127.0.0.1:19351` on the sender Mac: `{"song_pos_ns":int,"host_us":int,"playing":bool}` where
  host_us = UNIX us when the song was at song_pos. The sender extrapolates linearly to each frame's
  stamp_us; stamps older than 1 s or `playing:false` -> song_pos_ns=-1, PLAYING clear.
- Same thing over the WebSocket from any sender: text `{"type":"stamp","song_pos_ns":..,"host_us":..,"playing":..}`.
- Take-stamper compatibility: a stamped take uploaded/streamed by takestamp uses the same header
  (type 2 or 3, song_pos_ns per frame, sample_index monotonic, crc32 of payload). A file of
  back-to-back frames is a valid stream.

## 5. Clock + drift semantics
- Master timeline = sample_index. stamp_us is only for measuring one-way latency.
- Receiver keeps a FIFO and a fixed target fill = selected latency.
  Tight deadband (5 ms). Inside it, play at ratio 1 (no slow PLL glide). Outside it, snap the
  read pointer directly to the target in one step. The join is hidden by a ≤3 ms crossfade,
  starting at a nearby zero crossing. This is the silent always-on lock (section 6).
  Browser: AudioContext created at the stream rate, AudioWorklet does the resampling.
- Underrun: output silence and keep the timeline only until a snap is possible. If the gap
  exceeds the target latency, or the listener stays dry for longer than the target (a delay jump
  between 1x and 2x the target used to leave it silent indefinitely), snap immediately. Also snap
  down when the fill is more than max(150 ms, target) above target (a delay drop used to leave it
  permanently late). [2026-11-18 live-test fix, now a direct snap rather than a 1 s wait + PLL]
- AU placement:
  - If PLAYING and Lou's host transport is playing: the AU outputs the audio whose song position
    equals Lou's playhead (`ppqPosition/timeInSamples`), i.e. the take lands where Marcus's stamps say.
    Audio is held in an in-RAM song-keyed ring (60 s) so a cycle/loop region plays aligned. Nothing on disk.
  - Otherwise: live monitor at the selected latency (0.05-2.0 s), drift-corrected as above.
- Latency compensation: the AU reports 0 latency to the host (live source); song mode is exact
  because placement is by stamp, not by arrival.

## 6. Silent always-on lock (one-way offsets + Logic nudge)

Replaces Antonio's manual round-trip clap (route the feed back, measure a transient, type the
number into a monitor delay and Logic's nudge, redo before each set of takes). Both sides stay
engaged: each holds its own target offset for as long as it can, and when a side drifts past a
tight deadband the correction is one direct snap into place (a Newton's cradle, not a slow push).

### Clock + one-way measurement
- NTP-style four-timestamp exchange over the WebSocket (text `{"type":"sync","phase":"req","id","t1"}`,
  reply `phase:"rep"` with `t2`,`t3`; initiator records `t4`). The sender answers when one is
  connected, so the offset is against `stamp_us`'s clock; otherwise the relay answers.
  AU / scripts can also `POST /api/sync` `{"t1"}` → `{t2,t3,lock}`.
- Clock offset from the **lowest-RTT** samples (NTP best practice):
  `theta_us = ((t2-t1)+(t3-t4))/2` so sender ≈ listener + theta.
- **Forward one-way (Marcus → listener)** is measured on the **audio** path, not as RTT/2:
  `fwd_us = arrival_listener_us - stamp_us + theta_us`.
- **Reverse one-way (listener → Marcus)** is measured the same way when that side sends audio
  (`rev_us = arrival_sender_us - stamp_listener_us - theta_us`). If only one side is sending,
  the return leg of the sync probe (clock-corrected) is used until return audio exists.
- Path-asymmetry bias in theta cancels in the **sum**: `nudge = fwd + rev` is the exact Logic
  nudge (Antonio's 1.859 s number), in milliseconds and in samples at the stream rate.
- Nothing here alters audio samples. Bit-perfect `pcm` is untouched (no chirp, no pilot in the
  payload). Sync is a WebSocket / HTTP side channel; the published lock also travels as an
  LSTN **type 6 CTRL** frame (see below) so `/api/stream` clients (the AU, Take Stamp-adjacent
  tools) can read it. Type 6 is skipped by audio decoders and by Take Stamp's type 2/3 splitter.

### Sticky hold + snap
- Deadband 5 ms. Each direction holds its number while new measurements stay inside the band.
- Outside the band for 80 ms, or immediately on a jump ≥ 80 ms: the held number snaps to the
  new measurement. Play-out snaps the read pointer the same way (≤3 ms crossfade + zero-crossing).
- GET `/api/lock` (auth) and WS text `{"type":"lock",...}` publish the current holds.

### Type 6 CTRL payload (24 bytes, little-endian), after the usual 56-byte header
| off | size | field |
|---|---|---|
| 0 | 4 | fwd_us i32; -1 = unknown |
| 4 | 4 | rev_us i32; -1 = unknown |
| 8 | 8 | nudge_ns i64 (`(fwd+rev)*1000`; 0 if both unknown) |
| 16 | 4 | rate i32 (Hz), for converting nudge to samples: `round(nudge_ns*rate/1e9)` |
| 20 | 4 | flags: b0 LOCKED (both directions have a held number) |

`nframes=0`, `channels=0`, `payload_len=24`. Frame size = `hdr_len + payload_len`. Injected
by the relay every ~200 ms and whenever a hold snaps. Never written to disk.

The AU has no controls. It replaces the track input with Marcus's voice, reports the
round-trip nudge as plugin latency when Logic is recording and stamps are not yet placing
him, and otherwise reports 0 (talk is just the live path). Song-mode placement is by
`song_pos_ns` (already exact). The password is baked in and is not stored.

## 7. Always-on sender + cue return (no DAW on the singer side)

Marcus runs a tiny background sender (login agent on macOS, Startup folder on Windows).
No UI, no dialing, mic only, password [redacted] baked in, nothing stored. It auto-reconnects.
The same WebSocket (`role=send`) also receives **cue** audio.

Lu's AU sends his cue (Logic sidechain: the beat and talkback) on a second WebSocket
(`role=cue`) as ordinary LSTN v2 PCM. Each cue frame's `song_pos_ns` is Lu's playhead at
the first sample (`timeInSamples * 1e9 / hostRate`, or -1 when not playing). The flag
`PLAYING` is set when the playhead is running.

Marcus's sender plays that cue to his headphones and **echoes** the cue's `song_pos_ns`
onto the mic frames he is sending at that moment. Lu's AU places those echoed stamps
against the playhead, and the silent-lock nudge (`fwd + rev`) is the round trip: Marcus
sings to a cue that is late by one way, and his voice returns late by the other. The
recorded take is shifted back by that amount so it lands on Lu's grid.

Cue audio is RAM-only, like the main stream. It never touches bit-perfect mic payloads.

## 8. Record link: the producer's record button runs both sides (2026-11-18)

Marcus's design: the producer controls recording. One press of RECORD on the producer's side starts
both sides at the same song position; the performer never presses play. While recording, the
producer's monitor of the performer's track is the performer's live audio (the track input is muted
and replaced). Lag doesn't matter because every sample is placed by the song's time relative to
itself; on STOP the take is lined up by song position ("lag adjusted on the way over").
Pull, not push: the performer's sender sits **armed** and idle and only sends while the producer records.

### Messages (relay `/ws`, WebSocket text JSON)
| from | message |
|---|---|
| producer (`role=listen` or `role=cue`) | `{"type":"transport","event":"rec_start","take_id":str,"song_pos_ns":int,"host_us":int,"bpm":float?,"rate":48000}` |
| producer | `{"type":"transport","event":"stop","take_id":str,"song_pos_ns":int,"host_us":int}` |
| relay -> sender | the same object plus `relay_us` (relay clock at receipt), `seq`, `by` (`listen`/`cue`/`http`/`relay`); `late_join:true` when a sender (re)connects mid-take (song_pos_ns then = the song position *now*, extrapolated on the relay clock) |
| relay -> producer | `{"type":"transport_ack","from":"relay","ok":true,"delivered":bool,...}` (`delivered` = a sender was connected) |
| sender -> relay -> listeners | `{"type":"transport_ack","from":"sender","event","take_id","rx_us","capturing":bool}` |
| relay -> listeners that opt in with `?xport=1` (on `/api/stream` or `/ws`) | LSTN type 7 frame: `nframes=0`, `channels=0`, payload = compact UTF-8 JSON of the transport message; `song_pos_ns` header = the event's song position; `PLAYING` set for rec_start. While the sender is armed/idle the relay also sends a type-7 `{"event":"state",...}` once a second (keepalive; listeners that don't opt in never receive type 7, so older readers are unaffected). |

HTTP for clients without a WebSocket: `POST /api/transport` (same JSON; auth `X-Listen-Password: [redacted]`),
`GET /api/transport` (current state). `/api/health` adds `tx_frames` (audio frames from the current
sender), `tx_idle_s`, `armed` (sender connected and not streaming) and `transport` (`state` idle|rec, `take_id`, ...).
Field rules: `take_id` `[A-Za-z0-9_.-]{1,64}` (relay makes one if missing), `song_pos_ns` -1..1e15, `bpm` 20..400 or null.
If the producer connection that sent `rec_start` drops, the relay sends `stop` itself (never leaves the performer streaming).
Backward compatible: a pre-link sender ignores transport text and keeps streaming as before.

### Performer (sender.py `--armed`, the setup default)
- Connected and idle: nothing streamed, a text ping every 5 s. The mic stays open and its audio is read and
  thrown away ("warm"), because opening the device on `rec_start` measured 1.7-2 s of lost audio at the start of
  every take; `--cold-mic` closes it while idle instead.
- `rec_start`: starts sending mic frames, starts the cue player (producer's cue frames if they come; otherwise a click on
  the song grid from `bpm`, accent on every 4th beat), streams frames until `stop`; then closes the mic and player.
  Frames captured after `stop` are never sent.
- Stamps (`song_pos_ns`, PLAYING set): the song clock starts at `song_pos_ns` when `rec_start` arrives and is
  advanced by the capture **sample count** (anchor = first block's capture time, pulled earlier once if a
  device hands over a pre-buffered burst). Cue echo (section 7) wins when cue frames are playing.
  `--io-latency-ms` (or `LISTEN_IO_LATENCY_MS`) subtracts the performer gear's measured click-out + mic-in delay.
- Hidden on Windows: every child process gets CREATE_NO_WINDOW (sender.py itself and launch.py). The cue/click
  player is `ffplay -nodisp` (mainline ffmpeg has no wasapi/coreaudio output, and `-fflags nobuffer` on a raw
  pipe made ffmpeg 7 output nothing, so the old player was silent).

### Producer: box REAPER (`/home/box/listen-feed/reaper-link.py`; setup Producer-Linux installs it)
- Polls REAPER's web remote every 10 ms (`/_/TRANSPORT`, playstate 5/6 = recording; tempo from `/_/BEATPOS`).
- Record edge -> `rec_start` (song_pos_ns = REAPER position, bpm, take_id `take-YYYYmmdd-HHMMSS`); sets the
  Marcus track's input monitoring on (OSC `/track/2/monitor`), so the producer hears the live performer on it.
- Stop edge -> `stop`; restores monitoring. Collects the performer's PLAYING frames (RAM) for the take.
- After stop: reads REAPER's own (lag-shifted) raw recording on the track to get the exact record start
  `P0` (and length); waits until frames stop arriving (0.8 s quiet, 6 s max); writes
  `~/ListenSession/takes/<take_id>.wav` (32-bit float, 48 kHz, stereo) with every frame at
  `round((song_pos_ns - P0) * 48000 / 1e9)`, so sample 0 = P0; mutes REAPER's raw item, inserts the aligned
  take on the Marcus track at P0, saves the project; writes `<take_id>.json` (frames, coverage, arrival lag, placement).
- REAPER's "Select files to save or delete" prompt (on stop) blocks its web remote; reaper-link presses Save All.

### Producer: Listen AU (Logic), compile-ready, NOT yet run in Logic
- Reads the host playhead every block; on `isRecording` edges posts `rec_start`/`stop` (song_pos = timeInSamples,
  bpm from the host) through its cue WebSocket (`role=cue`), from the CueOut thread, never the audio thread.
- While playing it already outputs Marcus by song position (section 5) and replaces the track input.
- While recording it writes the take aligned by song position to `~/Music/Listen/takes/lu-<time>_at-<s>s.wav`
  (+ `.txt` with the position); finished 1.5 s after stop. Lu drags it onto the track at that position
  (Logic records a track's input, not the plug-in output, so the plug-in can't record onto the track itself).
- `LISTEN_URL` env overrides the relay for the box test (Logic never sets it). Wss also speaks plain `ws://`.

### Measured (box, local dev relay, 2026-11-18)
See `test_reclink_e2e.py` (REAPER) and `test_au_reclink.py` (AU harness on Linux).
