# Milestone 3 proof tooling

These are isolated external tests, not production dependencies. Run from the repository root with Docker and Python 3.9+. Real-client captures also require host curl/FFmpeg and the already provisioned Liquidsoap reference image. Never use production credentials; the fixtures deliberately retain disposable authorization headers.

## Build and checks

```sh
docker build -f tools/proof/Dockerfile -t quikcast-proof:validated .
PROPTEST_RNG_SEED=20261007 cargo test --all
cargo fmt --all --check
cargo clippy --all-targets --all-features -- -D warnings
cargo check --all-targets
cargo build --locked
python3 -B tools/cli_smoke.py
python3 -B tools/proof/log_sink.py --output /tmp/quikcast-log-proof.json
python3 -B tools/proof/compat.py --image quikcast-proof:validated --output /tmp/quikcast-compat-new
```

Output directories must not exist. Server images use a pinned Rust 1.99.0 base and Debian base, a locked release build, the Linux Rust suite, and an embedded source manifest. Package versions are recorded; future apt repository changes do not imply bit-identical rebuilds. Image IDs identify tested builds. The runtime image contains external test encoders only to provision the lab; Quikcast neither links nor executes them.

`compat.py` owns its server and source client processes/containers. Host bindings are loopback only. Raw source request/response bytes, event ordering, external-client versions/commands, curl traces, produced listener audio, decoder results, package/source identity, and manifests are retained. The bounded proxy forwards at most two concurrent connections and 32 connections in total, including Liquidsoap's separate metadata request. That request currently receives 404; implementing track updates remains milestone 4. Each direction is capped at 1 MiB per connection and lifetime at 30 seconds. Body comparison removes HTTP upload framing, then matches the captured listener bytes within the actual captured source payload. FFmpeg's muxer ID3 header and Liquidsoap's newly encoded bytes are therefore accounted for; they are not compared falsely to the original source file. Curl player timeouts return 28 intentionally for open-ended streams.

## Resource and performance runs

```sh
python3 -B tools/proof/run.py --image quikcast-proof:validated --output /tmp/quikcast-steady-new --listeners 1000 --seconds 600 --warmup 300 --ring-bytes 4194304 --ring-entries 4096
python3 -B tools/proof/run.py --image quikcast-proof:validated --output /tmp/quikcast-stalled-new --listeners 1000 --seconds 180 --warmup 30 --stopped 16 --kbps 1024
python3 -B tools/proof/run.py --image quikcast-proof:validated --output /tmp/quikcast-churn-new --listeners 1000 --seconds 180 --warmup 30 --churn 8 --reconnect
python3 -B tools/proof/run.py --image quikcast-proof:validated --output /tmp/quikcast-mounts-new --listeners 1000 --mounts 8 --seconds 180 --warmup 30
python3 -B tools/proof/run.py --image quikcast-proof:validated --output /tmp/quikcast-10000-new --listeners 10000 --seconds 180 --warmup 30
```

The server and generator use separate containers and disjoint CPU sets (0–1, 2–3), with two CPU quotas each. Their memory ceilings are 512 MiB and 1 GiB. The server has 64 PID slots and a 20,000-descriptor ceiling. Root filesystems are read-only, capabilities dropped, privilege escalation disabled, and writable temporary filesystems bounded. They share the server's isolated loopback namespace with no external network. Result mounts are owned by the calling local UID, and harness snapshots are mounted read-only. Every owned container has a bounded stop/remove attempt; sampler subprocesses are joined. Engine cleanup failures must be surfaced as blockers rather than claiming successful removal. Ownership IDs are persisted immediately so interrupted runs can be recovered without touching unrelated containers. SIGTERM goes through the script cleanup path. On macOS, run the host controller under `caffeinate -i -s` to prevent idle sleep; this cannot protect against an engine failure or explicit suspension. Cleanup never prunes other containers.

The asyncio generator is separately measured and owns a fixed bounded set of reader/source/churn tasks. It compares every healthy audio byte against a repeating captured MP3 fixture, retains counts rather than stream bodies, limits durations/listeners/mounts/churn/stopped clients, and gives I/O and shutdown explicit deadlines. The first 100,000 admission latencies are retained; this is a stated sample bound. Stopped sockets have a 1 KiB receive buffer and transport reading paused so asyncio does not mask the stall. Churn alternates FIN and reset while a separate mount repeatedly reconnects; an unaffected healthy source keeps streaming. A failure is retained and reported, not dropped from results.

The sampler records `/proc` RSS, user/system CPU, threads, descriptors, cgroup memory/throttling, application metrics and metrics-request latency once per second (newer runs also retain anonymous/file RSS, swap, huge pages and page faults). Server CPU excludes the separate sampler process; cgroup memory includes that process and kernel charges. Raw JSONL is primary evidence. Summaries select the post-warmup interval and compute rate deltas, CPU cores consumed, RSS range/median growth, allocation maxima and latency quantiles. Metric-request latency at one request per second is not a general API saturation benchmark. Socket-accepted byte counters do not assert player consumption; the generator checks consumption separately.

Do not compare scenarios as if their source rates, ring sizes, mount counts, or generators were identical. Repeat capacity/latency tests on a quiet dedicated Linux machine with independently provisioned generators before publishing production sizing. A shared Docker Desktop VM can demonstrate measured correctness/resource behavior and expose failure modes; it is not dedicated-host capacity proof.

## Sanitizer fuzzing

```sh
docker build -f tools/proof/Dockerfile.fuzz -t quikcast-proof:fuzz .
```

The fuzz target compiles the actual private parser/config/authentication modules directly, without copying protocol code or adding production fuzz exports. The checked-in corpus consists of 26 deduplicated real Icecast request heads. The lab pins cargo-fuzz 0.13.2 in its recorded toolchain; libfuzzer-sys and the complete fuzz dependency graph are retained separately from the production lock. Rust 1.99.0 with `RUSTC_BOOTSTRAP=1` is used only in the disposable image for sanitizer instrumentation; production is an ordinary stable locked build.

The recorded run executes the already built AddressSanitizer binary with no external network, one CPU, 768 MiB container memory, 512 MiB fuzzer RSS limit, 16,385-byte maximum input, a five-second per-input timeout, deterministic seed 20261007, and a ceiling of 120 seconds or one million inputs. The exact command, corpus, toolchain, lock and exit code are in the evidence. A clean short run is evidence of that tested corpus/budget, not proof of absence of all parser bugs.

## Review artifacts

Run `python3 -B tools/proof/verify.py` to check the retained recursive evidence manifest and semantic workload checks. The interruption record retains the former live sampler output and its detached snapshot. After the authorized Docker recovery the descriptor was closed and both files are immutable and hashed; original incomplete runs remain invalid benchmark evidence. `plot.py` renders the raw memory series with external matplotlib; it is a report dependency only. The measured conclusion and remaining dedicated-host/allocator gates are in `docs/Writerside/topics/milestone-3.md`. Failed diagnostic captures are retained alongside successful cases.

Recovered measurements reject sampler errors or a post-warmup sampling gap above five seconds, and record every owned-container stop/removal result. `--memory-maps` additionally captures at most 2 MiB / 4,096 mappings from `/proc/1/smaps` every 30 seconds, without process injection. This locates RSS changes by mapping, not by Rust allocation call stack.


For a per-process THP control, add `--disable-thp --memory-maps` to a run. The external `no_thp.py` launcher checks Linux `PR_SET_THP_DISABLE`/`PR_GET_THP_DISABLE` and replaces itself with the same Rust executable via `exec`. Python does not remain in the server process. The experiment changes no shared sysfs setting, production source, or deployment default. Compare the first 300 seconds of the quiet default-ring run with the 300-second control, respecting their different complete durations and warmup windows.

## Current ICY capture tooling

`compat.py` now validates milestone 4 ICY-enabled curl audio after deinterleaving and requires successful title setting/clearing. Use `--image quikcast-proof:milestone-4` built from current sources. The milestone 3 evidence retains its own original harness snapshots and image identities; those historical captures remain unchanged.

Verify the milestone 4 capture with `python3 -B tools/proof/verify_icy.py evidence/milestone-4/compatibility-final`. This checks hashes, image/production source identity, actual deinterleaved audio, set/clear blocks, successful Liquidsoap metadata replies and the focused reference matrix.

## Milestone-5 media delivery

Build the native binary with `cargo build --locked`, then run `python3 -B tools/proof/media_compat.py --output NEW_DIRECTORY`. The script owns a separate server on temporary loopback ports, uses disposable credentials, captures six late-join media cases, checks unchanged audio/page bytes, and invokes external FFmpeg to decode saved outputs. It preserves binary/source identities and request/response files and joins the server after SIGTERM. The user dev server is not involved.

The fuzz corpus additionally seeds actual AAC and Ogg initialization/page prefixes; the same production MPEG/ADTS boundary detector and Ogg parser run on every bounded fuzz input.

## Milestone 6 HLS

`cargo build --locked` then `python3 tools/proof/hls_compat.py --output NEW_DIRECTORY` captures typed MPEG-TS upload/publication/serving and rejection dialogues against an owned loopback server. It uses the fixture described in `fixtures/hls/provenance.json`, curl HEAD, and paced external FFmpeg HLS playback with seeking disabled while a continuous listener checks exact bytes. The 500-request sequential playlist workload is a local observation, not a capacity benchmark. Raw captures, process exits, source/binary/fixture hashes, commands and client versions are retained.

`python3 tools/proof/verify_hls.py` verifies final-source captures, typed compatibility matrix, fixture/compiled fuzz-module identities, tests and the sealed checksum inventory. `--seal` writes the inventory only after verification. The final package is under `evidence/milestone-6`; historical failures/superseded captures are retained and identified in its validation summary. Quikcast never invokes FFmpeg or accepts fixture `.m3u8` uploads; those remain external test-tool operations.