Monitoring and metrics

Per-channel metrics

Every channel exposes the following in the interface and over the HTTP API.

Wire bitrate

The rate of transport-stream data actually leaving the interface for that channel’s destination, measured by packet capture rather than reported by the engine. This is the one figure to trust. It is engine-agnostic, and it is the only measurement taken outside the process producing the stream, so it cannot be fooled by an engine that believes it is working.

Frames per second

Decoded frame rate. Remux engine only.

Playback speed

Progress through the source relative to real time. A sustained value below 1.0 means the channel cannot keep up. Remux engine only.

Encoder bitrate

The rate the engine believes it is producing, as distinct from the rate observed on the wire. Remux engine only.

Uptime

Time since the current engine process started. It resets on every restart, which makes a short uptime on a channel you have not touched a signal in itself.

Viewers

Number of televisions currently joined to the destination group, and their addresses. See On-demand streaming for how this is measured and when it is marked degraded.

Round-trip time to source

Latency to the origin host. Useful for separating “the origin is slow” from “the path to the origin is slow”.

Channel log

A per-channel buffer of the last 1000 entries or 24 hours, whichever is reached first, readable in the interface without shell access to the appliance.

Do not build alerts on frames-per-second or playback speed

These come from the remux engine’s own reporting. The pass-through engine never decodes video — that is the entire point of it — so it produces no frame rate and no playback speed. Those fields read n/a for such channels, and that is the correct value, not a fault.

Any watchdog keyed on them will read n/a as zero and restart a perfectly healthy channel, indefinitely. This has happened twice in production, from two different checks. Gate such checks on the engine in use, or key them on wire bitrate, which exists for both.

Transport-stream quality

Beyond “is data leaving the box”, Castmux measures whether the transport stream itself is healthy. This is what turns “channel 4 breaks up sometimes” into a number that says whether the damage arrived with the source or was introduced on the way out. The Stats button on a channel’s card graphs the last 30 minutes. All of them are served by GET /api/channels/{id}/ts-quality, and the metrics listed under OpenTelemetry are exported.

Bitrate and CBR spread

The transport-stream rate, and how much it varies over the last minute. A large spread on a channel expected to be constant-rate is worth knowing about before it is carried on DVB-C, where every multiplex has a fixed capacity.

Continuity errors

Packets missing or out of order, detected from each packet’s continuity counter and summed across the channel’s PIDs. This is the most direct measure of loss. On the pass-through engine it reflects the source faithfully. The remux engine renumbers continuity counters as it writes, so loss upstream of it does not show here.

Transport errors and sync losses

Packets the sender itself flagged as damaged, and places where the stream lost its packet alignment. Either one is a fault upstream of the appliance.

PCR leaps and PCR accuracy

PCR leaps are jumps in the programme clock. PCR accuracy checks whether each clock reference arrived when its timestamp says it should. A reference further out than ts_pcr_jitter_max_ms (200 ms by default) counts as a jitter event. That threshold is set to flag delivery bursts a receiver’s buffer would have to ride out, not ordinary interleaving. PCR accuracy is measured for SRT and MPEG-TS sources, and on the wire for remux-engine channels. It is not measured for HLS on the pass-through engine: segments arrive in bursts by design, and a measurement there would be meaningless.

Scrambled PIDs

The number of the channel’s PIDs that carry scrambled content. Castmux does not descramble, so any non-zero value on a channel meant to be in the clear is a source problem.

How it is measured depends on the engine. A pass-through channel is measured inside its own pipeline, at the point the stream leaves for the wire. A remux channel gets a second, read-only process that joins the channel’s multicast group and measures what a receiver actually gets. Neither alters the stream. The measurement is on by default and can be turned off with ts_monitoring: false to save the small per-channel CPU cost — see Configuration.

Counts are summed across the channel’s PIDs today, so they show that a channel is damaged but not whether it is the video or an audio track. A per-PID breakdown is on the roadmap.

System metrics

Measured for the appliance as a whole and shown alongside the channel list, because most capacity problems are visible here before they are visible per channel:

• CPU usage. Aggregate across cores.
• Memory usage. The binding constraint on these appliances. Watch it before adding channels rather than after — see Requirements.
• CPU temperature. Fanless boxes in cupboards behave differently in August.
• Per-interface bitrate, inbound and outbound. Inbound tells you what the uplink is pulling from your origins; outbound tells you what the televisions are being sent, and on which interface, which is frequently the fastest way to notice that it is the wrong one.

DVB-C metrics

Where a DVB-C modulator is fitted, each multiplex reports its running state, its uptime and a restart count. See DVB-C multiplexing.

Health checks Castmux performs

Castmux supervises its own channels. Each condition below is detected continuously and has a fixed, deliberate response.

Condition	What it means	What Castmux does
No data at startup	The engine started but nothing has been produced yet.	Allows a grace period — cold starts legitimately take seconds — then restarts.
Mid-stream silence	A channel that was producing data stopped. Usually the origin dropped it.	Restarts once the stall timeout expires.
Output silent, process alive	The engine is running and believes it is fine, but nothing is on the wire. The worst failure mode, because every internal counter looks healthy.	Restarts after several minutes of confirmed silence.
Output dip	Wire bitrate fell but did not stop. Brief dips are normal at segment boundaries and during origin hiccups.	Tolerated inside a grace window measured from the last known-healthy point. Acting on every dip would restart channels that were about to recover on their own.
Repeated early exits	The engine keeps dying shortly after start. Normally a source that is unreachable or unusable.	Backs off progressively rather than hammering the origin. The backoff resets after a sustained healthy run, so an intermittent source does not accumulate a punitive delay forever.
Destination unroutable	The multicast route does not lead anywhere usable. Checked against the routing table from outside the process, because a socket cannot detect it.	Alerts only. Never restarts. See Troubleshooting.
Duplicate senders	Something else is transmitting to the same multicast group — commonly an older deployment that was disabled but not stopped. Televisions receive both streams interleaved and show pixelation.	Alerts. Restarting would not help, and stopping the wrong sender is not Castmux’s decision to make.
Orphan process	An engine process left behind by a previous supervision cycle.	Reaped.

Restart what a restart can fix, alert on what it cannot

This is the rule the table above encodes, and it is worth stating plainly because it explains why some serious conditions produce no restart at all.

A restart is the correct response to a stuck process, a dead connection to an origin, or an engine that has wedged. It is the wrong response to a network fault. If the multicast route points at an interface with no carrier, restarting the channel produces a fresh process that sends into the same void — and it does so every few minutes, filling the log with restart events that bury the one line naming the real cause. A restart loop against a network fault is worse than the fault, because it destroys the evidence.

So Castmux restarts what a restart can plausibly fix, and for everything else it stops, says what is wrong, and leaves the channel alone.

Exports

OpenTelemetry metrics

Castmux exports its metrics and events over OpenTelemetry (OTLP over gRPC) to the Castmux telemetry collector at otel.alteox.app, port 4317. The export is always on, and the collector is fixed: it cannot currently be pointed at a collector of your own. Each series is labelled with the site’s location_name and re_seller, the appliance’s hostname and software version, and each channel’s name, protocol, source host and port, and engine. If the collector cannot be reached, the export fails quietly and streaming is unaffected.

The transport-stream quality figures are exported per channel as go_udp_channel_ts_bitrate_kbps, go_udp_channel_cbr_spread_percent, go_udp_channel_ts_continuity_errors_total, go_udp_channel_ts_transport_errors_total, go_udp_channel_ts_pcr_leaps_total, go_udp_channel_pcr_jitter_max_microseconds and go_udp_channel_pcr_violation_percent.

Series are labelled by channel name

Renaming a channel starts a new series; the old one stops receiving points. If you rename channels routinely, expect a discontinuity in dashboards at that moment. Deletes are handled properly — the series is retired rather than left to go stale — so removing a channel does not leave a permanent flat line in your graphs.

Structured events

Alongside the numeric metrics, Castmux emits structured events for the things that change state. These are what you build a timeline from after an incident:

• engine started, engine exited
• restart initiated
• output silent, output recovered, output degraded
• destination unroutable
• duplicate senders detected
• transport-stream errors (ts_quality_errors), with the counts that triggered it, at most once a minute per channel
• viewers arrived, viewers departed
• settings changed

A restart on its own says little. A restart preceded by “output silent” and followed by “output recovered” is a source hiccup handled correctly; the same restart preceded by “destination unroutable” is somebody having restarted a channel by hand against a network fault.

Logs

Castmux writes structured JSON to stdout, which a container log collector can ingest without a parsing layer. The same material is available per channel in the interface, buffered to 1000 entries or 24 hours, which is normally enough to diagnose a fault reported that morning without shell access to the appliance.

When opening a support case, the per-channel log for the affected channel is the single most useful attachment. Troubleshooting lists the rest.

Roadmap

The following are committed but not yet released. They are listed here so you can plan around their absence rather than discover it during commissioning.

Prometheus endpoint roadmap

A scrape endpoint alongside the OpenTelemetry export, with a supplied dashboard so a site does not have to build panels from scratch.

Email and webhook alerting roadmap

Delivery of the alert conditions above without an external monitoring stack in between.

Transport-stream analyser roadmap

Built-in analysis reporting continuity-counter errors, PES errors and scrambled packet counts per PID — the measurements that distinguish source-side loss from loss introduced locally.

Restart counter and availability report roadmap

Per-channel restart counts and availability over a period. Today restart frequency can only be recovered by reading the logs, which makes “is this channel getting worse?” harder to answer than it should be.