Milestone 3: compatibility, resource and performance proof

A serving defect found and fixed

A non-draining stdout collector blocked the original synchronous tracing formatter. At 160 listener churn connections the healthy player stopped receiving and SIGTERM could not complete. The retained baseline (evidence--milestone-3--log-sink-baseline.json) documents the failure. With the replacement logger, 1,000 churn connections (evidence--milestone-3--log-sink-fixed.json) completed while audio and graceful shutdown continued.

src/logging.rs writes each tracing record through one nonblocking syscall, caps it at 4,096 bytes, and counts truncated, partial or failed writes in logging_dropped_total. It has no queue, worker, task, retries or shared logging lock. Loss is intentional when the collector cannot keep up; a partial socket write may leave a partial record. There is no guarantee of lossless operational logs. Metrics remain the reliable aggregate signal. Concurrent records have bounded memory per executing formatter; the runtime's task/worker limits bound concurrent execution.

The Unix CLI duplicates stdout and sets its open file description nonblocking. Direct regular-file stdout is rejected at startup because O_NONBLOCK cannot bound storage latency. Use a draining pipe, service collector or Docker logging driver; a shell pipeline through tee is acceptable. The CLI currently requires Unix for this sink. The server library's I/O ownership remains independent of this CLI choice. rustix is the concrete safe syscall wrapper; Quikcast still forbids its own unsafe code. This is the only production behavior change introduced during this proof phase.

Protocol evidence and deliberate differences

The complete capture (evidence--milestone-3--compatibility-complete--summary.json) and its provenance (evidence--milestone-3--compatibility-complete--provenance.json) retain raw request/response bytes, event order, commands, versions, proxy directions, actual received audio and external decoder results. The original Icecast reference matrix remains separate from these Quikcast observations.

• Raw PUT and legacy SOURCE (ICE/1.0 and HTTP/1.0) are admitted with early HTTP/1.0 200 OK, and their live audio is byte-exact. Raw PUT with Expect returns 100 then 200 before source audio. These are Quikcast's chosen contract, not a claim that Icecast 2.5.0 emits the same early final response.

• Wrong or missing Basic credentials receive 401 without an interim 100. Production comparisons use bounded high-entropy credentials and constant-time comparison, without SHA-256 password storage.

• Actual curl finite Content-Length upload with Expect, FFmpeg's Icecast output and Liquidsoap 2.1.3 output all admit. The listener bytes match a contiguous region of the captured producer bytes, and all three captured audio regions decode successfully in external FFmpeg. FFmpeg adds muxer headers and Liquidsoap encodes its own output; comparison uses those actual producer bytes.

• Actual curl players with and without Icy-MetaData: 1 receive byte-exact plain audio. Station ICY headers are emitted where supplied; icy-metaint is absent. Curl's bounded open-stream capture exits 28 intentionally. Interleaving remains milestone 4.

• Liquidsoap's separately captured metadata request receives 404, and its source continues serving. /admin/metadata implementation and track changing/clearing remain milestone 4. This is an intentional current gap, not broad Icecast admin compatibility.

• Content-Length and chunked source parsing, malformed chunk isolation, source idle deadlines, duplicate framing/auth headers, invalid paths, over-read initial bytes, normal disconnect/reconnect and generation protection are checked by loopback integration/unit tests. Quikcast decodes HTTP chunk markers rather than copying Icecast 2.4.4's observed raw markers.

Source EOF closes the generation and cancels its listeners immediately. It does not promise draining the retained tail after an offline transition. In the finite curl capture, 16,384 bytes reached the listener from a 16,718-byte upload. Therefore real-client success means the captured live region was intact; it does not imply whole-file delivery after source EOF. The fanout workloads keep sources open until every healthy reader has consumed the complete sent byte count and verify that separate guarantee.

Temporary joins can begin at arbitrary bytes within the 64 KiB retained window. Decoder success on these particular captures does not establish all-offset MP3 decodability. Frame-boundary joining remains a later narrowly scoped framing change.

Memory ownership and limits

The fixed ring caps both payload bytes and entry count. Published frames share immutable owners; payload accounting remains charged through the last reference, including a listener slice retaining the full allocation. A listener can have one frame awaiting underlying transport flush, and its cursor monitor detects eviction even when Hyper stops polling. Tests cover tiny chunks, eviction, old-generation cleanup and final-owner release. All completed load runs check source/listener gauges and retained backing bytes return to zero after sockets close.

retained_audio_allocation_bytes counts backing payload, not process memory. Separately budget ring entry allocation/capacity, Bytes control blocks, generation metadata, task/future state, parser/Hyper buffers, tracing records, allocator arenas and kernel sockets. The configured admission reservation is described in milestone 2. A native debug-layout test measured the connection future as 17,792 bytes before polling; this is compiler/build-dependent type size, not a promise of release heap allocation size. At high listener counts that fixed state warrants measurement before optimization. No per-chunk task or unbounded per-listener channel was introduced.

The proof server also has an explicit 512 MiB container memory ceiling, 20,000 descriptors and 64 PID slots; these are lab process limits, not application admission-budget defaults. Cgroup memory includes the sampler and kernel charges. RSS can retain allocator pages after owners drop. Zero payload/gauges demonstrates owner release but does not by itself prove an RSS plateau. The sample files retain both quantities; newer runs additionally record anonymous/file RSS, swap, huge pages and page faults.

Workloads, measurements and provenance

All workloads use the pinned release binary identified by the embedded source manifest in each provenance file. Docker Desktop provides four vCPUs and about 4 GiB of VM memory; server and generator have disjoint CPU sets and two CPU quotas each. They still share the VM/kernel, host and loopback network. This is a reproducible development lab, not the dedicated Linux deployment/generator setup required for production capacity figures.

Every healthy reader checks every byte. The Python generator retains counts, not audio bodies. It pauses reads on stopped sockets with a 1 KiB receive buffer, alternates FIN/reset in listener churn, and reconnects a separate source mount. Container/process/task lifetimes and waits are bounded and joined. Reproduction instructions (tools--proof--README.md) document exact safety ceilings and commands. Per-run harness snapshots and both current-source hashes and the image's embedded build-source manifest prevent confusing changed scripts with the tested binary.

The long default-ring run used the initial generator; later runs precomputed the repeating comparison buffer to reduce generator allocation overhead. They are not interchangeable before/after server benchmarks. A fuzz run occurred during the long run's warmup; a short compatibility capture overlapped early in its measured interval. These limitations are retained rather than silently discarded. The initial repeated-baseline series was interrupted: a 231-second sampling gap invalidates the first repeat, and the next has only 24 completed samples and no final provenance. Docker failed normal and forced removal of its owned container with “did not receive an exit event.” These captures remain retained failures. After explicit user authorization, Docker Desktop was restarted; its ordinary restart timed out, requiring its own processes to be force-closed before start. All four previously running RadioPlatform containers recovered healthy, and both verified stopped proof containers were removed. The interruption/recovery record (evidence--milestone-3--interruption.json) preserves the sequence. The formerly live sampler file is now closed and immutable; its detached snapshot remains separate.

Recovered runs use caffeinate -i -s, persist container ownership immediately, handle SIGTERM through bounded cleanup, fail on cleanup errors, and reject measurement intervals with sampler failures or gaps over five seconds. Three new baseline repetitions, the corrected 10,000-listener run, a 15-minute quiet default-ring soak and the per-process control all passed; completion/status is recorded individually in the results index. The generator regression test holds back its last chunk to check that the producer join precedes reader equality. No failed capture is silently promoted to a passing benchmark.

Measured results and RSS plots are generated from the raw samples in the evidence package; CPU is process user+system time divided by wall time, expressed as consumed cores. Egress is bytes accepted by sockets; complete generator byte checks establish healthy-client receipt separately. Metrics endpoint latency is sampled once per second and is not a saturated API benchmark. Admission quantiles cover a stated bounded sample, not source-to-player end-to-end latency.

RSS finding and control experiment

The measured finding (evidence--milestone-3--memory-finding.json) closes the local RSS-step investigation. In the quiet run, increases of roughly 3.95 MiB and 3.34 MiB occurred around streaming seconds 48 and 58, alongside increases in AnonHugePages of 16 MiB and 8 MiB, respectively. Minor page faults increased by only 13 and 9. Thirty-second mapping snapshots show the residency changes within existing writable anonymous regions. The kernel reported THP always, max_ptes_none=511, and a ten-second scanning interval; these were read, never changed.

The 900-second run delivered every byte to 1,000 listeners. During its last 600 seconds, RSS stayed within 44.63–45.07 MiB, payload backing stayed capped at 4 MiB, and all tracked owners released after disconnect. A later 2 MiB promotion explains another approximately 0.41 MiB RSS increment. The counter and mapping evidence support kernel promotion filling previously sparse mapped pages, rather than a corresponding jump in retained audio.

The control uses the same executable, 128 kbit/s source, ring limits, 1,000 listeners and harness logic. An external Linux-only launcher disables THP for its own process, checks the result, then replaces itself with Quikcast using exec. It changes no production Rust code or shared kernel setting. Throughout the 300-second control, THP_enabled and AnonHugePages remained zero; exact delivery, owner release and graceful shutdown passed. At the matching 300-second point, RSS was 37.58 MiB in the control versus 44.63 MiB in the promotion-enabled run. Compare their common prefix, not their differently selected steady intervals.

Linux documents that promoting sparse mappings can increase memory consumption, and that per-process THP opt-out survives exec. Linux THP documentation, Linux v6.12 prctl definitions. This experiment validates the cause of the reproduced steps. The earliest captures lack huge-page counters; attributing their similar steps to the same mechanism remains a retrospective inference. There is no automatic deployment tuning change or claim of exhaustive leak freedom. Production budgeting must include page-size policy, allocator residency and kernel socket charges, beyond logical payload ownership.

Validation depth

The native final suite passed 17 unit tests and 8 loopback integration tests with property seed 20261007. Four parser/authentication properties each ran 2,048 generated cases, including arbitrary bounded prefaces, mutated real-shaped request heads, path canonicalization and credential input. The Linux image build passed 16 unit tests and the same 8 integration tests; the subsequently added connection-layout measurement is test-only and explains that count difference. Formatting, Clippy with warnings denied, all-target compilation, locked build and CLI curl/SIGTERM smoke passed. Four proof-tool tests, including the final-chunk race regression, and five reference-tool tests passed. Cargo audit was unavailable on the host; no dependency-audit pass is claimed.

AddressSanitizer fuzzing (evidence--milestone-3--fuzz--provenance.json) exercised the actual parser/auth/config modules for 1,000,000 inputs in 48 seconds, exit zero. The source target directly includes production modules; it does not duplicate their parser. Input length is capped at 16,385 bytes; time, RSS, container memory and per-input timeout are bounded. The 26 real-reference request-head seeds, 1,495 retained resulting corpus entries, fuzzer log, image identity, toolchain and separate dependency lock are preserved. This is a finite test budget, not exhaustive proof. Proptest and libFuzzer are test dependencies only.

The reference verifier rechecked all nine Icecast datasets, 178 raw dialogues and the 20-case reference compatibility matrix. Failed early proof captures remain visible: the first result export attempted Docker archive-copy from tmpfs, and the first Liquidsoap proxy allowed only one connection, preventing its parallel metadata request. The corrected capture uses durable bounded result mounts and a two-connection proxy. The first 10,000-listener run additionally exposed a generator shutdown-accounting race while its single Python event loop fell over ten seconds behind schedule: one final chunk was emitted after the equality check. The harness now joins all finite producers before checking reader totals. That failed run is retained; it is not a measured server eviction or proof of 10,000-listener capacity. Those harness failures are not classified as server protocol rejection.

The verifier (tools--proof--verify.py) checks the recursive evidence SHA-256 manifest, exact reader/source byte totals, unexpected EOF/mismatch lists, sampler errors, final-generation release, stalled-client eviction, successful server exit, real-client comparisons, and the logged fuzz budget. SHA-256 here checks fixture integrity; it is never credential storage.

Review gate and remaining work

Application ownership bounds, source/listener isolation, generation churn, stopped-reader behavior, measured real-client interoperability and nonblocking log loss policy have concrete evidence. The package does not certify deployment capacity, audio latency, arbitrary-offset MP3 decodability or complete tail draining after source EOF.

Before marking the entire milestone complete, run the recorded scenarios on the dedicated Linux host required by the approved benchmark strategy, with independently provisioned generators and deployment-representative CPU/network/allocator settings. The local RSS finding is explained by the promotion-enabled/control experiment above. Repeat the ownership/RSS measurements under the deployment host’s allocator and huge-page policy, separating logical backing, mapped residency and kernel charges. Establish a production admission ceiling from those measurements rather than treating the configured 10,000-listener limit as proven capacity.

On 2026-10-07 the user instructed development to continue after confirming no dedicated Linux host is available. The development gate is accepted for proceeding to milestone 4; dedicated-host production capacity validation remains deferred and unproven. The subsequent sequence stays ICY metadata → AAC ADTS/Ogg compatibility → independent authenticated MPEG-TS HLS. No later feature has been introduced to make the tests pass.