TSDuck-based MPEG Transport Stream multiplexer with full PSI/SI support


Stream Manager mux assembles multiple SPTS streams into one CBR MPTS output (DVB/IPTV). Adaptive Hybrid Statmux combines a CBR core fill with VBR balanced encoders for constant mux output.
When the parent mux has statmux enabled, each FFmpeg encoder runs capped-VBR: -b:v <avg> -minrate <min> -maxrate <max> -bufsize <max×2>. VBV buffer = 2× max gives ~2 s peak headroom. Statmux streams also get a deterministic GOP offset (-force_key_frames) per stream_id so I-frame peaks don't coincide across channels.
tsp -I null (∞ NULL packets @ line rate) → -P merge tsp -I ip <stream> (overlays real packets) → -P continuity (drop alarm) → -P pcradjust --bitrate X (rewrites PCRs) → -P pcrverify --jitter-max 500µs → -P bitrate_monitor (rate alarm ±5%) → -P regulate --bitrate X --packet-burst 32 → -O ip <out> --packet-burst 7 --enforce-burst.
regulate plugin emits exactly mux.bitrate bps on the wire. Inputs lagging → NULL packets fill the gap. Inputs peaking → internal FIFO buffer (4096 packets ≈ 0.3 s @ 19 Mbps) absorbs. --enforce-burst forces 7 TS packets per UDP datagram (1316 B, IPTV standard).
Mux top bar shows 3 sums with color status. Rules:
Σ avg ≤ 85 % mux.bitrate — average target sum across all channels plus audio. That is the mux 'budget'. The 15 % statistical reserve protects against peak coincidence. Green ≤ 85 %, amber ≤ 100 %, red > 100 %.
Σ max ≤ 160 % mux.bitrate — peak ceiling sum. Allowed to exceed capacity (that's the point of statmux — peaks rarely coincide). Green ≤ 160 %, amber ≤ 200 %, red > 200 % (drops on sustained peak).
Σ min ≤ 50 % mux.bitrate — encoder floor sum. A safe floor ensures that on still scenes the encoder doesn't drop to needlessly low quality. Green ≤ 50 %, amber ≤ 90 %, red > 90 %.
Available budget after PSI/SI/EIT: ~18.5 Mbps. Σ avg = 4 × 4.0 = 16 Mbps (86 % — amber borderline, lower further). Σ max = 4 × 6.0 = 24 Mbps (130 % — green, OK). Σ min = 4 × 3.0 = 12 Mbps (65 % — amber, OK). Target: avg ~85 % of budget, max ≤ 160 %.
On ≥ 10 alarms in 60s a flash notification appears in the Mux UI recommending a bitrate reduction. Auto-back-off is NOT active — the operator decides.
-P bitrate_monitor --periodic 60 --min/max ±5% — alarm if actual rate strays beyond ±5 % of target. Means regulate is starved (input died).
-P continuity — tracks TS continuity counters. Logs a warning on every drop (over-subscription, FFmpeg crash, NIC jitter).
-P pcrverify --jitter-max 500000 — DVB allows PCR jitter ≤ 500 µs. Alarm = pcradjust can't keep up or source has broken PCRs.
GOP coincidence — all encoders emit I-frames simultaneously, peak coincidence. Fix: GOP offset (already on for statmux, hash(stream_id) mod gop_size).
Input loss-of-signal — single channel goes black, mux continues (thanks to -I null base). Duration ~3-8 s for FFmpeg restart.
Over-subscription (sustained peaks > capacity) — blocks/audio stutter ~50-200 ms. Fix: lower stream avg by 5-10 % or remove a channel.
PCR drift — STB stutter every few seconds. Fix: pcradjust in the pipeline (already there), watch pcrverify alarms.
STB VBV underrun — STB loses sync and re-acquires (~2-3 s). Fix: encoder bufsize = 2 × maxrate (already), lower maxrate.
--max-input-packets 4096 / --max-flushed-packets 4096 (internal FIFO ≈ 0.3 s @ 19 Mbps). --receive-buffer-size 16777216 (16 MB UDP rcv buffer per input). --packet-burst 32 (regulate) / 7 (output, IPTV standard). --enforce-burst (every UDP datagram is exactly 7 packets).
Top-bar badge `📐 rec X.XM` shows the recommended mux capacity derived from the central limit theorem. The sum of independent VBR streams approaches a normal distribution, so Mux = μ_total + z·σ_total + margin. σ_i is estimated from each input's VBR range as (max−min)/4 (assuming the range covers ±2σ). The margin = 10% of μ_total compensates for PSI/SI overhead, encoder rate-control jitter, and stream correlation that pure CLT under-estimates.
Badge color: 🟢 green = cap ≥ 99.7% rec (conservative) · 🟡 amber = between 99% and 99.7% · 🔴 red = cap < 99% (under-provisioned, risk of drops) · 🔵 blue = cap unset.
z-score per coverage target: 90% z=1.28 · 95% z=1.65 · 99% z=2.33 · 99.7% z=3.00 (default) · 99.99% z=3.72 (broadcast-grade). The badge tooltip lists all levels; clicking applies the 99.7% recommendation to mux.bitrate (mux restart required).
σ_eff = σ_total × statmux_factor (statmux_factor = 0.7 when Statmux is ON, else 1.0)
1080p25 HD channel: avg = 4000-5000 kbps, min = avg × 0.75 (~3000-3750), max = avg × 1.5 (~6000-7500). UI auto-fills min/max from avg. Statmux UI auto-fills avg from the stream's transcoder profile. SD channel: avg = 2000-2500 kbps. UHD: avg = 12000-18000 kbps.
Adaptive Hybrid Statmux is a runtime mechanism that shifts unused capacity from core streams (protected, full VBR) to balanced streams (sacrificial). "Hybrid" because two distinct rate-control regimes run side by side (CBR core fill + VBR balanced); "adaptive" because the router re-evaluates once per second from an HLS-lookahead prediction. The operator marks streams as balanced via the right-side checkbox; the system pre-encodes them into three tiers (HIGH/MED/MIN) and the router atomically swaps the symlink based on remaining headroom after predicting core peaks.
Mux capacity 38 Mbps
┌──────────────────────────────────────────────────────────────────┐
│█████████████████████████████████████████████████████░░░░░░░░░░░░│
└──────────────────────────────────────────────────────────────────┘
◀────── core_max (predicted from 6 HLS segments) ─▶◀── balanced ──▶
per_balanced
= (cap − core_max
− SI overhead
− audio) / N
Example: cap=38000 core_max=29000 SI=200 audio_total=1280 N_balanced=2
remaining = 38000 − 29000 − 200 − 1280 = 7520 kbps
per_balanced_max = 7520 / 2 = 3760 kbps
clamp(3760, 500, 3000) = 3000 kbps ← ceiling for balanced VBRWithout Adaptive Hybrid Statmux you must size the mux for the worst-case Σ max. With Adaptive Hybrid Statmux you size around Σ avg + a statistical buffer (broadcast correlation ρ=0.15). Real gain ~5 Mbps on a 38 Mbps mux, or +1 stream at the same cap.
Pure CBR (encoder fixed)
VBR without Statmux
Adaptive Hybrid Statmux ON
Custom URL input override and Statmux are mutually exclusive. With Statmux on, the balanced transcoder relies on the local encoder HLS to control bitrate. An external URL would bypass that control loop. The UI enforces this both ways — the checkbox locks when there is a conflicting combination, and the tooltip shows which stream is blocking.
┌─────────────────────────┐
│ Encoder (NVDEC→NVENC) │ ← UDP satellite
STREAM │ /dev/shm/stream_X/h264/│
│ → muxed.m3u8 + segments│
└────────────┬────────────┘
│
┌────────┴────────┐
│ │
role="core" role="balanced"
(protected) (sacrificial)
│ │
│ ▼
│ ┌───────────────────────────┐
│ │ BalancedTranscoder │
│ │ ffmpeg p1 NVENC VBR │
│ │ -maxrate dynamic │ ← Adaptive Hybrid router
│ │ /dev/shm/stream_X_ │ 1 s tick (predicts from
│ │ balanced/h264/...m3u8 │ 6 HLS segments, atomic
│ └───────────┬───────────────┘ symlink swap HIGH/MED/MIN)
│ │
▼ ▼
┌─────────────────────────┐
│ ffmpeg-as-mux │ ← /dev/shm/mux_X/pipe.sh
│ -f mpegts pipe:1 │
│ -muxrate <cap> │
└────────────┬────────────┘
│ shell pipe
▼
┌─────────────────────────┐
│ tsp -I file - │
│ continuity → regulate │
│ → pcradjust → bitrate_ │
│ monitor │
│ -P nit -P sdt -P inject│
│ (DVB SI tables) │
└────────────┬────────────┘
│
▼ UDP multicast
239.x.x.x:1234Statistical gain only happens when encoders have a wide VBR range — when max ≈ avg the encoder is effectively CBR and the router has nothing to redistribute. Configure `min/avg/max` per stream so Σmax is comfortably above the mux cap (over-subscription = headroom for statistical gain) while Σavg sits comfortably below it.
Worked example — 38 Mbps mux, 10 HD core + 2 balanced:
Old (CBR-locked): min=2000 avg=3000 max=4000
Σmax = 10×4000 + 2×7600 = 55200 kbps (1.45× cap — small gain)
σ_per_stream = (4000−2000)/4 = 500 kbps (trivial — flat)
Σ statistical headroom ≈ 1.5 Mbps
Recommended (wide VBR): min=1500 avg=3000 max=6000
Σmax = 10×6000 + 2×7600 = 75200 kbps (1.98× cap — healthy gain)
Σavg = 10×3000 + 2×3000 = 36000 kbps (95% cap — comfortable)
Σmin = 10×1500 + 2×500 = 16000 kbps (42% cap — comfortable floor)
σ_per_stream = (6000−1500)/4 = 1125 kbps (significant)
Σ statistical headroom ≈ 4 Mbps
Real benefit: on action scenes core peaks at 5−6 Mbps (HD detail), the
router sheds balanced to MIN tier, total wire stays at 38 Mbps.Recommended range per core stream: • min ≈ avg × 0.5 (encoder may drop on still scenes) • avg = nominal target by resolution (HD ≈ 3 Mbps, FullHD ≈ 5 Mbps, UHD ≈ 12 Mbps) • max ≈ avg × 2.0 (encoder may peak on action — sport, action film) Mux capacity guards (must hold): • Σ avg core + Σ avg balanced ≤ mux.bitrate × 0.85 (comfortable target) • Σ max core + Σ max balanced ≤ mux.bitrate × 2.0 (over-subscription gain) • Σ min core + Σ min balanced ≤ mux.bitrate × 0.6 (worst-case floor) σ_per_stream = (max − min) / 4 (assumes ±2σ range) σ_total = √((1−ρ) · Σσ²ᵢ + ρ · (Σσᵢ)²) ρ = 0.15 broadcast typical σ_eff = σ_total × 0.7 (statmux factor when ON) Mux_99.7% = μ_total + 3 · σ_eff (shown in top-bar 📐 rec)
At least 1 balanced stream — without one the Adaptive Hybrid router has nothing to squash during peaks, so gain = 0
Music and info channels are the best candidates — 90 %+ tolerance to rate shedding
Track via `[balance-router]` log entries — shows `core_max`, `remaining`, tier transitions for each 1 s tick
If `core_max > capacity` (CORE OVERPEAK warning in log), reduce core stream count or lower encoder maxrate
Add streams to the mux (channel assignment stays manual)
Toggle Statmux on the mux header (yellow chip). When enabled, all encoders switch to VBR and restart
Mark sacrificial streams via the right-side orange checkbox — typically music/info channels where rate-shedding is invisible
Done. The Adaptive Hybrid router reads a 6-segment HLS prediction from core streams every 1 s and atomically swaps the balanced symlink between HIGH/MED/MIN tiers
Multiple variable-bitrate channels share one fixed transport stream (MPTS). Bits are dynamically reallocated by content complexity — a demanding scene gets more, a static one less — so quality stays equalized and the sum never exceeds the wire capacity.
Class: closed-loop feedback statmux with C1 look-ahead (feedforward) over VBR encoders — broadcast statmux principle, driven from source complexity.
UDP multicast SSM, per channel
Encoders
source complexity
Loopback
own line / stream
allocator (Σ ≤ pool)
buffer
null padding → CBR
CBR wire at cap
anticipate complex scene before the dip
measured wire — calibration
target bitrate
Each channel runs on its own line up to the mux (separate encoder + loopback connection), where all merge into one MPTS multicast line at an exactly fixed bitrate. The allocator is the central brain (1 GenServer per mux), tick ~0.85 s.
The allocator combines two signals — a prediction from source complexity (forward) and the measured wire (backward). This approaches broadcast statmuxes that know a frame's cost before encoding.
C1 measures content complexity at the decoder — which leads the encoder output by the pipeline latency. The relative complexity change (correlation r≈0.92 with bits) is applied to the target bitrate in BOTH directions BEFORE the encoder produces it: a rising scene is pre-allocated (no dip), a falling one frees bits early for others.
Exact per-stream bits emitted by the mux (sender-side counters) calibrate the complexity→bitrate mapping and hold the sum exactly at the pool. Ground-truth anchor — the C1 prediction isn't perfect (r 0.92), the trailing wire corrects it.
A stream's claim is backed by what the encoder actually produces (× peak headroom). An easy channel can't hoard bandwidth it can't spend — it releases it to the hungry.
The goal isn't equal bitrate but equal quality — continuous pressure moves bits from high-quality channels (low QP) toward low-quality ones (high QP).
When C1 signals a channel dropping, its bandwidth is handed to the others in the same tick — they ramp up before the drop actually reaches the wire.
Pool = cap N − overhead (PSI/EIT) − audio − reserve (slider). Measured from the wire, windowed-max against bursts.
Per-stream demand = deliverable wire × C1 look-ahead (relative complexity change) + quality-equalizing pressure.
Proportional pool allocation by demand, clamped to [min, max] from the video profile; free bandwidth goes to the hungry.
Encoder target = alloc / calibration (per-stream EMA measured/target); dispatched with a threshold (Δ≥3%/1s) and slew limit against churn.
Buffer-hold servo keeps the queue level at ~1 s (band), emergency backstop on overload; null padding tops up the rest to CBR.
The output is an exact CBR stream at the configured cap. Three layers guarantee it:
The send queue holds a standing level of ~1 s of wire that absorbs sub-second GOP dips (between I-frames). Verified from the recording: 0.5–1 s absorbs 88–91% of dips.
When content is below cap, the difference is filled with MPEG-TS null packets (PID 0x1FFF). The wire thus ALWAYS hits the cap — DVB-compliant.
Egress is paced to the exact cap (busy-spin ABSTIME); a 12 ms debt clamp prevents catch-up bursts after a brief stall → no PCR jitter, no over-cap spikes.
A live stacked graph of the real wire — each band is one channel, flowing smoothly from the right. Data is measured on the wire (tsp probe + NIF counters), not estimated.
Each channel has a stable color; thickness = its instantaneous bitrate. Overhead (PSI/EIT) is grey at the bottom, null padding is textured white on top.
The dashed line at the top = configured cap (wire ceiling). Content + null padding together always sit under it.
The dotted white band on top = null padding (free capacity). The thinner it is, the fuller the wire; it vanishes on demanding content.
Red marks on the timeline = send-queue overflow (drop). Zero in steady state; bursts only at boot/flood.
Buttons switch the time window instantly (client-side), the active one has a grey background. The graph accumulates 30 min in the browser.
Hover shows per-channel bitrate, QP and quality index at that moment (historical value, not live).
Current pool regulation level (pool/(pool−trim)). >100% = overpressure (filling the buffer), <100% = trimming.
Tolerance band above cap — how much to keep as null reserve. 0 = fill exactly N; positive = larger reserve (safer); negative = slight overfill.
The header shows send-queue util % and cumulative drops. The "clear drops" button resets the counter.
Common GOP length for all channels in the mux (locked into video profiles on start). Wall-stagger spreads I-frames in time so they don't collide.
Encoder VBV buffer size (ms). Smaller flattens I-frames (good for motion channels), larger gives a more detailed I-frame. Default 850; per-stream override possible.
Each channel in the mux has a card with name, PNR, current and target bitrate, and state:
Running — channel is encoding and flowing to the wire.
Starting — encoder is spinning up / settling.
Off — channel is in the mux but not running (0 kbps).
Fault — input lost / encoder crashed.
StatMux generates the full set of mandatory MPEG-TS / DVB tables directly in the NIF mux — no external SI inserter. Tables are evenly timed into the stream so an STB always acquires PSI/SI quickly after retuning.
Table
PID
Interval
Purpose / source
EIT (Event Information Table, PID 0x12) is not static — it is fully wired to the application's EPG system. The operator assigns each channel in the mux an EPG service (eit_epg_service_id), and the mux generates live EIT from its scheduled events.
min = lower floor (static scene). Recommended avg × 0.5–0.75. Set low enough to save on calm content for others.
avg = target level. Σ avg of all channels ≈ video pool (cap − overhead − audio).
max = ceiling (VBR headroom for peaks/cuts). Recommended avg × 1.5. Don't hard-cap to avg — VBR needs room for peaks.
Examples (avg): SD 1.5–2.5M · HD 2.5–4M · FHD 4–6M · UHD 12–18M. For statmux keep a reasonable spread so the pool can be redistributed.
We do real statistical multiplexing — dynamic bit redistribution across VBR channels sharing a fixed wire, with quality equalization. Mechanically we're close to professional headend statmuxes (cap alignment, 1 s buffer, null padding, GOP/B-frames, floor override).
CBR wire at cap · 1 s buffer · per-stream proportional allocation · C1 look-ahead feedforward · quality equalization · predictive redistribution · null padding · exact pacing.
A broadcast statmux uses the encoder's rate-distortion estimate (exact frame cost, the encoder is a slave to the allocator). We use C1 source complexity (r≈0.92) + a VBR encoder that picks its own QP — a hardware limit, not a design one. Class: feedback + look-ahead instead of pure feedforward.
Symptom
Cause / fix
Gaps / null spikes (5–7M) in the chart
Content dips — either GOP between I-frames (absorbed by the 1 s buffer) or a collective content stall (handled by redistribution). C1 look-ahead reduced these substantially.
Drops > 0 climbing
Send queue overflowing — content above cap. Check the reserve (slider), util and that the allocator isn't stuck (tick running). Boot bursts are normal.
One channel stays low bitrate / high QP
Either static content (correct — it releases bandwidth) or too low a max in the profile. Check min/avg/max and the real source complexity.
Chart stutters / left edge erases
Client-side — hard refresh (Ctrl+Shift+R) for the new JS bundle. Data flows via incremental push, axis = selected range.
cap % / buffer doesn't change after edit
LiveView re-render — hard refresh. Values update once per second via a tracked assign.
SAT UDP SSM ─┐
├─ gst_encoder.py(K1) CBR ─ TCP127 ─┐ ZMQ complexity
(per ├─ gst_encoder.py(K2) CBR ─ TCP127 ─┤ ▲ │
chan) └─ gst_encoder.py(Kn) CBR ─ TCP127 ─┤ │ ▼
▼ BalanceRouter
┌─────── NIF StatMux ───────┐ │ set_bitrate
│ per-input ring (drop-old) │◄─┘
│ 2-pass drain (A prio, V) │
│ allocator: w=cmplx×max │
│ ×headroom% (slider 78-90) │
│ stale-exclusion (>2s) │
│ NULL pad -> wire cap CBR │
│ PCR pass-through │
└────────────┬──────────────┘
▼
MPTS UDP multicast (CBR cap)gst_encoder.py per stream: SAT UDP SSM input → nvh264enc rc-mode=cbr. Bitrate is driven by BalanceRouter via set_bitrate. Scene complexity (SAD/px) reported over ZMQ to the allocator.
Per-input ring (TCP loopback from encoder), 2-pass drain (audio priority, video full), proportional allocator, NULL pad exactly to wire cap (CBR), PCR pass-through from encoder.
One MPTS (PAT/PMT/SDT generated by NIF) → UDP multicast on the output VLAN. Wire is strictly CBR at mux.bitrate (NULL stuffing).
Row 1: ⚠ (validation issues) · PNR · name in a frame (colour = state).
Row 2+: min·avg·max kbps · predicted complexity Δ low/med/high/peak · INPUT LOST · ⚙URL · encoder Mbps · SID · service_type.
Name colour = stream state:
Mbps
37 ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ <= cap 37.0 Mbit (red, wire CBR ceiling)
· · · · · · · · · · · · · · · · · <= free space = NULL padding
·· grey · black dots · reserve ·· (unused capacity)
31 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ <= headroom 85% · 31.5 Mbit (orange,
▓▓▓▓ JOJ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ encoder targets aim here)
▒▒▒▒ Markíza ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒
░░░░ TA3 ░░░░░░░░░░░░░░░░░░░░░░░░░░ <= click band -> expand card
0 └──────────────────────────────────
time -> 10m │ 1h │ 24h CC0 PCR0 BR0 OVF0Red 'cap' line: wire CBR ceiling (mux.bitrate) — NULL fills up to here.
Orange 'headroom N%' line: = cap × headroom slider / 100. Encoder set_bitrate targets aim here. Moves with the slider.
Grey area with black dots between content and cap = NULL padding (unused reserve). More dots = more reserve.
Click a channel's colour band → automatically expands that channel's card below.
MuxEitAdapter reads events from the EPG system per service, builds DVB EIT sections (name, time, duration, description, genre) and packetizes them onto PID 0x12. Section versions increment on EPG change — the STB picks up the update.
EIT is injected directly in the NIF mux, evenly INTO the null space (1 packet / N output + p/f periodically) — replacing the bursty external tsp cascade. Peak/avg ≈ 1.1, no content eviction, cap holds.
Table 0x4E — the currently running + next event. Injected every ≤2 s for instant "now/next" currency on the STB.
The mux periodically regenerates the EIT .bin from EPG (~15 s) — an EPG schedule change reaches the wire without restart. The mux header shows "EIT ✓" when active.
Tables 0x50/0x51 — the full schedule hours/days ahead (segments). Spread evenly into the stream.
Indicator of real flow vs cap. Center = cap (0). Marker left (green) = reserve (Σ content under cap); right (red) = overflow. Live, moves every tick.
-5 Mbps CAP 37000 kbps +5 Mbps
[#### green = reserve ####│::: red = over cap :::]
^ -3200k
(Σ content ~3.2 Mbps under cap = healthy reserve)
Lower (→78): more NULL reserve, safer, less tight cap utilization.
Higher (→90): tighter cap utilization, less NULL.
⚠ ≥88 %: slider turns yellow — close to the ring-overflow freeze ceiling. Hard-clamp 78–90 (cannot reach 100 — guard against frozen video).
Asymmetry: lowering reliably adds reserve; raising is limited by per-stream max_bps clamp (allocator won't push a channel past its max → safe).
Controls how much of the cap the encoders target: encoder target = budget × headroom% / 100 (orange line in the chart). Changes live (effect from the next 10 ms tick), persisted in SystemConfig.
One channel fails (encoder crash / signal loss) ⇒ the mux keeps running, other channels unaffected, wire holds the cap (NULL absorbs the outage), CC=0 on survivors. Per-input ring threads are independent. The failed channel shows a red blinking name + INPUT LOST badge; the encoder is supervised-restarted and the channel self-recovers.
PCR and PES PTS/DTS are pass-through from the encoder (single clock domain ⇒ coherent, no drift). Residual PCR jitter ~tens of ms is inherent (10 ms output tick + UDP) — STB/VLC PLL tolerates it, CC=0. Hard DVB-S2 modulator compliance is out of scope for the current IPTV use case.
CAT — conditional access (only when scrambling / ECM).
EIT — event information (EPG) — from the EPG system, see below.
NIT — network info (network ID, name, transport stream).
PAT — program list (maps PNR → PMT PID).
PMT — per program: video/audio/PCR PIDs, codecs, descriptors.
SDT — service names and providers (what the STB shows in the list).
TDT/TOT — time and date + local offset (STB clock).