Milestone 6: independent MPEG-TS HLS

Scope and module ownership

src/hls/config.rs owns typed capacities and credential configuration; types.rs owns bounded management descriptions; ts.rs validates narrow transport initialization; budget.rs owns allocation reservations through final drop; store.rs owns stream/rendition state, publication and retirement; http.rs owns finite ingest and public responses. The existing server routes both listeners and runs maintenance. There is no storage trait, generic upload layer, codec factory, media subprocess, database or on-disk segment store.

The concrete media object contains immutable Bytes, a format enum, sequence-associated duration/discontinuity and retention bookkeeping. The only current format is MPEG-TS. A future format can extend object initialization and format-specific validation without replacing registry, publication, allocation ownership or HTTP serving. fMP4/CMAF, init objects, byte ranges, encryption, LL-HLS and raw AAC HLS are absent.

All previous hard invariants remain: listeners cannot backpressure ingest; every resident queue, buffer, request and task population has a hard bound; no lock spans I/O or await; older generations cannot mutate newer state; malformed clients remain isolated; normal disconnects cannot panic; and every long-lived task has cancellation and joining ownership. Published segments are immutable. A playlist commit requires every referenced segment to be completely received, validated and accepted.

Configuration and credentials

Enable optional [hls] alongside existing mounts, or with no continuous mounts. The private API uses the existing loopback-only admin_listen socket. HLS credentials are separate machine credentials; startup rejects reuse of any continuous-source secret, even under another username. The default username is hls. Specify exactly one bounded secret_file or explicit secret; relative secret paths resolve beside the configuration. Startup discards the raw credential configuration after loading its fixed-size representation. Comparison uses the existing padded constant-time credential implementation. No password hashing was introduced.

These are the defaults. The configurable 2 MiB segment maximum is a safety ceiling, not an expected segment size. The explicit upper configuration bound is 16 MiB. CORS is disabled by default; when enabled, it is either a validated exact HTTP(S) origin or *. Public responses use that configured literal, without reflecting client credentials or arbitrary origins. Process HTTP/connection limits still apply alongside HLS limits.

Typed producer API

Every private operation requires HLS Basic authentication. Stream and rendition names are 1–64 ASCII alphanumeric, underscore or hyphen characters. They never become filesystem paths. Numeric headers and sequence path components are canonical unsigned decimal. Generation tokens are returned as JSON strings, avoiding JavaScript integer precision loss; send them verbatim as headers. Publication revisions start at zero.

Let S=/internal/v1/hls/streams/{stream} and R=S/renditions/{rendition}:

• PUT S, empty body: register a stream; 201 returns generation and revision zero.

• GET S, empty body: recover its generation, master revision and ended/deleted state.

• PUT R, JSON definition, X-HLS-Generation: register a rendition; 201 returns its generation and revision zero.

• GET R, empty body: recover rendition generation, media revision, state and accepted-segment count.

• PUT R/segments/{sequence}, finite MPEG-TS body: upload one segment, with both generation headers, Content-Type: video/mp2t, X-HLS-Duration-Ms, and optional X-HLS-Discontinuity: 0|1; success is 201.

• PUT R/publication, JSON publication, both generation headers: atomically publish a media playlist; success returns its next revision.

• PUT S/master, JSON master description, stream generation: atomically publish a master; success returns its next revision. An empty rendition list withdraws the master.

• POST R/end, JSON expected revision, both generation headers: publish the current media window with ENDLIST.

• POST S/end, empty body, stream generation: end the stream after all its renditions have ENDLIST.

• DELETE R, empty body, both generations: hide its playlist and retire data; first withdraw any master reference.

• DELETE S, empty body, stream generation: hide its playlists and retire every rendition. Object retention promises remain in force.

JSON operations require Content-Type: application/json. Unknown fields are rejected. No endpoint accepts a .m3u8 document or arbitrary filename; Quikcast writes the playlists itself. Private query strings are rejected. Public query strings are ignored within the existing target-length ceiling.

Rendition definition:

Target duration is fixed for this generation, from 1–60 seconds. Window capacity is 3–256 segments, subject to reservation checks. Bandwidth must be positive; bounded CODECS text is producer-declared. Quikcast does not infer bitrate or codec details from media. Producers must supply correct durations, peak bandwidth, timestamps, codec declarations and discontinuity markers.

Media publication:

Master publication:

End description: {"expected_revision":1}. Each media reference may additionally specify "discontinuity":true, matching its accepted upload metadata. Collections stop at 256 entries during deserialization; codec and identity visitors bound strings before retaining them. The body ceiling is 256 KiB.

The server rejects stale generations/revisions, duplicate or retired sequence identities, missing accepted objects, changed duration/discontinuity metadata and illegal window progression with 409. Bad syntax/description is 400, failed authentication 401, unknown identity/object 404, unsupported MIME 415, excessive body 413, and exhausted allocation/upload/registry capacity 503. Existing typed timeout mappings apply. Authentication and declared-body limits are checked before body polling, so rejected Expect: 100-continue uploads receive no interim admission. Successful uploads use ordinary HTTP 100/finite-response semantics, separately from continuous source early admission.

Publication, validation and retention

Upload admission reserves a global upload slot, global allocation and rendition allocation before collecting bytes. Content-Length and actual chunked bytes are capped; interrupted/truncated bodies, trailers and invalid media never become objects. Body inactivity is limited to ten seconds and total transfer/validation admission to thirty seconds. The final commit rechecks rendition state and sequence ownership; concurrent duplicate uploads have one winner.

Transport validation requires nonempty whole 188-byte packets, sync bytes, valid adaptation envelope lengths, no transport-error indicator and no transport scrambling. Each segment must contain an initial CRC-valid, current, single-section PAT describing one program and a matching CRC-valid PMT with bounded descriptor structure and at least one elementary stream. This parser does not decode AAC, MPEG audio, PES, timestamps or samples. Later PSI changes and semantic media correctness are producer responsibilities. Validation of this narrow contract is distinct from proving every possible player can decode an upload.

Publication takes only a short rendition mutex. It verifies the expected revision, accepted objects, metadata equality, consecutive sequence numbers, fixed target-duration compatibility using nearest-second rounding, monotonic first/last sequence and no skipped window boundary. When pruning a live playlist, its duration must remain at least three target durations. It builds a charged immutable playlist before committing state; allocation failure leaves the prior revision intact. Discontinuity sequence increases when marked segments leave the window. Master publication serializes on its stream, references only ready renditions and uses declared bandwidth/CODECS.

Advertised segments cannot expire. Removed segments remain available for their own duration plus the longest published playlist duration that contained them, as required by RFC 8216. Accepted unpublished staging expires after thirty seconds. End leaves the final advertised window available until explicit deletion. Delete hides playlists immediately and converts advertised objects to grace retention. Active object responses retain their own backing allocation after registry expiry.

Accepted object URIs are never reused within a rendition generation. A publication prefix fence rejects older sequences; bounded retired-sequence bookkeeping rejects expired sparse staging without wrongly forbidding lower out-of-order uploads. Stored objects plus retired identifiers share the configured max_segments slot ceiling. Pathological sparse uploads can exhaust those slots and receive a controlled capacity error; a fresh rendition generation resets that bounded state.

Concurrency, generations and shutdown

The registry has a serialized writer and immutable ArcSwap index snapshots. Public lookup takes a reference without a global read mutex. Stream and rendition mutexes cover synchronous local state only. Lock order is registry writer → stream → rendition where multiple locks are required; publication uses rendition alone. Network reads/writes and body awaits occur after releasing locks. No per-upload, per-chunk, listener or sweeper task is spawned.

Generation allocation starts from a fallible OS-random 63-bit namespace and increments with checked arithmetic. Segment paths and strong ETags include generation tokens. This avoids ordinary restart namespace reuse probabilistically; it is not durable globally unique identity. HLS media are memory-resident and not recovered after restart. Producers must register/recover the current identities and publish anew.

In-flight operations hold the concrete old Arc and generation tokens. Deleted identities remain reserved until retention expires, response-held budgets release and outstanding state references drop. A later registration gets fresh state; old cleanup never finds a replacement by name to mutate it. A shared one-second supervisor timer performs bounded sweeping. On service shutdown, admission closes, the existing JoinSet cancels/drains owned HTTP tasks and aborts/joins remaining tasks at the shared deadline; only afterward is HLS registry state closed and released. Tests explicitly cover cancellation during incomplete uploads and zero allocation/upload gauges after joining.

Allocation and copy accounting

HLS global and per-rendition budgets charge full backing capacity before allocation, not merely visible object length. Charge ownership follows immutable Bytes until its last response or registry owner drops. Immutable response clones copy reference metadata, not media. Allocator reserve is fallible; unexpected capacity rounding beyond the allowed buffer cap is rejected.

The aggregate allocation budget includes all of the following simultaneously:

• Advertised, grace-retained and accepted unpublished segments: each reserves max_segment_bytes + 128, even for a small segment.

• Uploads in progress: the same full-cap charge transfers into accepted object ownership without a release/reacquire gap. Four global upload permits by default.

• Response-held expired objects: final-drop ownership keeps the original full-cap reservation alive.

• Current, newly built and old response-held playlists: each owns 256 KiB + 128 until final drop; this includes master snapshots, with a separate 4 MiB stream-local master budget.

• Finite management body plus JSON scratch/description: 2 * 256 KiB + 64 KiB per active mutation; eight permits. Bounded status responses reserve 16 KiB + 4 KiB, retained through response ownership.

• Rendition bookkeeping: 2 * max_segments * (size_of(Segment) + 40) + 32 KiB, covering rounded HashMap backing, retired-sequence storage, window references and bounded structural metadata.

• Stream bookkeeping: 16 KiB per stream. Registry base reserves 3 * max_streams * 512 + 4096.

• Each nonempty registry snapshot reserves max_streams * 512 + 1024 separately, including snapshots retained by a reader across later updates. Charges release only when the final index reader drops.

Registration requires both (2 * window_segments + 1 + max_uploads) <= max_segments and this conservative local byte condition:

This is minimum admission headroom, not permission to exceed budgets if rapid publication builds a larger grace population or slow readers hold old snapshots. Actual reservations enforce the ceiling continuously, rejecting new work rather than deleting promised objects. Global reservation may still fail even when local headroom exists.

Network data are copied once into the bounded upload Vec; freezing transfers ownership without another payload copy. Playlist text is formatted directly into its bounded reserved Vec. JSON retains a bounded body and deserialization scratch while parsing. Source and listener memory use the earlier independent admission accounting.

hls_allocation_reserved_bytes reports conservative charged capacities, including response-held backing; it is not process RSS. Allocator size-class/arena retention after free, runtime tasks, bounded HTTP read/write buffers, kernel socket buffers and the existing connection-future size are separate process-resource considerations from the milestone-3 package. There is no claim that the allocation gauge bounds every byte of process RSS. Finite response headers/small acknowledgements and bounded path vectors fit the existing connection/header allocation accounting.

Observability and dependencies

Fixed-label metrics are hls_requests_total, hls_rejections_total, hls_allocation_reserved_bytes, hls_active_uploads, hls_segments_accepted_total, hls_publications_total and hls_bytes_served_total. Served bytes count public HLS HTTP body bytes written to the socket, including finite error bodies; headers/private management bodies are excluded. Socket acceptance is not proof of remote playback. Source/audio/ICY counters retain their separate meanings.

Three concrete dependencies were added and locked: arc-swap 1.9.2 for immutable registry snapshots, serde_json 1.0.151 for typed bounded descriptions, and getrandom 0.4.3 for startup generation entropy. Their documented interfaces are ArcSwap, serde_json and getrandom. No new task framework, unsafe Quikcast code or media library was introduced.

Evidence and acceptance limits

The compatibility matrix, capture provenance, final-source hashes and validation summary are linked in milestone-6-compatibility.json (docs--milestone-6-compatibility.json) and evidence/milestone-6. Native/Linux tests cover revision races, immutable backing held after expiry, stale generations, out-of-order staging, discontinuities, malformed TS/JSON, partial upload invisibility, admission limits, chunked/oversized bodies, credential separation, status recovery, source/HLS independence, and joined incomplete-upload shutdown. Existing continuous-stream regression tests run unchanged alongside these cases.

The external fixture encoder generates fourteen seconds of AAC-in-TS solely as reference-test tooling. Quikcast never invokes it. tools/proof/hls_compat.py owns a loopback server, captures raw typed requests/responses, checks GET/HEAD/ETag/Range/CORS/query handling and invokes curl and FFmpeg as external clients. Its paced decode runs concurrently with byte-exact continuous streaming, followed by 500 sequential new-connection playlist reads. These latencies are local workload observations, not a production capacity rating.

Range requests deliberately return 416. The first FFmpeg capture exposed its default root HTTP range probe; that failed capture remains preserved. Successful full-object playback uses both -seekable 0 and -http_seekable 0, with persistent HTTP disabled for the fixture test. Generic player defaults, browser HLS support and broad producer interoperability are not inferred from that success. Additional player/producer certification and dedicated-host mixed-load capacity remain review gaps; no Icecast HLS compatibility claim is made.

Final validation and local demo

The final reviewed source passes 34 unit/property tests and 15 TCP integration tests on both macOS and pinned Linux, with formatting, Clippy warnings denied, all-target checking and the fuzz crate check passing. The AddressSanitizer campaign processed 949,932 inputs in 121 seconds under its 120-second clock limit and one-million-input ceiling, with no reported failure; it did not reach one million inputs. Reported sanitizer RSS ended at 225 MiB under its 512 MiB ceiling. The final connection future measures 18,000 bytes, below its 64 KiB test bound. cargo-audit is not installed; no advisory-audit pass is claimed. Direct dependency licenses and repository provenance are recorded in validation/dependencies.json.

The final HLS capture decoded 14,037,333 microseconds of fixture audio while checking 213,928 continuous-source bytes exactly. Its 500 sequential new-TCP playlist reads measured p50/p95/p99 of 1.60/2.32/2.67 ms and approximately 591 requests/s on this local machine. These figures describe this short recorded workload only. The final Linux image additionally passes curl, FFmpeg and Liquidsoap source/ICY regressions; raw captures and current-source verification remain preserved.

Reproduce the final HLS capture with cargo build --locked, then python3 tools/proof/hls_compat.py --output PATH_TO_NEW_CAPTURE. The harness chooses isolated loopback ports, uses disposable test secrets, owns and joins its server/client thread, and records exact binary/source/fixture hashes, raw requests/responses, versions, commands and results. Reproduce parser fuzzing using tools/proof/Dockerfile.fuzz and its recorded CPU/memory/network/pid limits. Use python3 tools/proof/verify_hls.py to verify the sealed review package; earlier failed and superseded captures are explicitly historical, rather than final-source claims.

The running developer configuration enables HLS with its own secret file. A fourteen-second AAC-in-MPEG-TS tone is available at http://127.0.0.1:8000/hls/demo/master.m3u8 for an HLS-capable player. This final ENDLIST fixture remains available until deletion or server restart; it is independent of BUTT on /radio.mp3. Its non-secret generation/verification details are in target/dev-butt/hls-demo.json.