Day/Night — Design Notes¶
Superseded twice: the 2026-08-17 redesign, and in v1.10.1 the simple engine of
dev_notes/dn-redesign.md.Status: historical, and deliberately kept. This page analysed the machine as it stood on 2026-08-14 and argued for a targeted restructure once an executable oracle existed. That oracle was then built (trace recorder, replay harness, scenario corpus), and on 2026-08-17 the file was redesigned rather than restructured — see
dev_notes/DAYNIGHT_REDESIGN_2026-08-17.mdfor the new design and Day/Night for what ships. Read this page for the evidence: the twelve incidents, the invariant they all violate (§3), the failure generators (§4) and the harness design (§6) are all still the reason the current code looks the way it does, and any future change should be measured against them. Section 5's "no rewrite" verdict was explicitly conditional on the harness being missing, and that condition no longer holds; section 7's "what not to do" list still does.
Day/Night documents what src/daynight.c does. This page
is about why it kept needing fixing, and what to change about that. It is a
working document for whoever touches the file next, not user documentation.
Why this page exists¶
Of the 17 fix(daynight): commits to date, 12 are the same behavioural
class: the camera latches into the wrong mode and does not get out, or it
flips between modes. 53c21b4, c78dbcb, b3eec71, f8a7b21, 0f5fc80, bd21ce6,
fad4f40, a5dae07, b4a54f0, 43c2b16, 8c8dc1f, 14a1d61, plus the 2026-08-14
cam-L dawn incident that produced this page.
Twelve fixes for one symptom class is a signal about the design, not about the fixes. Each of those commits is individually well-reasoned and each closed a real, reproduced failure. So the interesting question is not "was fix N correct" (they were) but "why does fix N+1 keep arriving".
1. What the machine actually is¶
One thread, one 500 ms poll loop, one metric (total_gain, IMP [24.8]
linear, 256 = 1×, inverted: high gain = dark). Everything else is state
carried between ticks. Current state, grouped by what it is for:
| Group | State |
|---|---|
| Decision | cur, pending_target/pending_since_ms (hysteresis candidate), last_switch_ms (dwell) |
| Measurement conditioning | hist[] (brightness ring), settle_hist[]/settle_n (AE-stability ring), settle_floor_ms/settle_hard_ms, smooth_tg (night-only EMA), scraped_b/next_scrape_ms |
| Night model | night_baseline, baseline_logged, night_entered_ms |
| Probe scheduling | verify{mode,at_ms}, brighten_since_ms, brighten_armed, probe_day_ms, probe_verdict_at_ms, probe_backoff, probe_fail_smooth, last_phys_probe_ms, day_verify_ref/day_verify_ext |
| Backstops | osc_hist[]/osc_n/osc_freeze_until_ms, reassert_at_ms/reassert_left |
Counting the gates — the independent mechanisms that can block, delay, trigger or override a mode resolution — gives fifteen, not the eight one might guess from the section headings:
- boot/re-enable settle (floor + AE-stability extension)
- dwell (
transition_s) - pre-switch hysteresis (5 s)
- adaptive baseline + symmetric EMA drift
- sustained-brightening hold, with its arming edge
- …and its arming margin (
DN_BRIGHTEN_MARGIN), against a frozen rest reference (brighten_ref/DN_HOLD_REF_LEAD) - failure ratchet (
probe_fail_smooth×DN_RATCHET_MARGIN) - exponential backoff on failed probes
- passive-evidence skip gate
probe_max_skip_souter bound on the skip gate- ratchet-anchor override of the skip gate (
14a1d61) - dead-zone adoption + its verify deadline, both directions
- probe three-outcome classification → ambiguous arms a verify (
8c8dc1f) - post-probe AE-stability revert gate
- oscillation-breaker freeze
(plus the post-switch re-assert series, which is actuation rather than decision). Nine of the fifteen exist purely to decide when to spend an audible IR-cut click. That ratio is the story of this file.
2. What the incident record actually says¶
The twelve incidents are not twelve unrelated bugs, and they are not one undifferentiated mass either. They sort into seven generators (five, until two more named themselves on 2026-08-16):
| Generator | Incidents | Status |
|---|---|---|
| A. Actuation / measurement plumbing — the Set landed mid-ramp, the AE hadn't converged | 53c21b4, c78dbcb, b3eec71 (part) |
Closed. No recurrence since 2026-08-02. |
| B. A reachable state with no pending decision | bd21ce6 (UNKNOWN), 43c2b16 (adopted DAY), 8c8dc1f (ambiguous probe landing) |
Closed, and closed checkably: after 43c2b16+8c8dc1f every reachable (cur, verify) pair has an armed deadline. |
| C. A derived bar that goes unsatisfiable | f8a7b21, a5dae07(1), 2026-08-14, cam-H 2026-08-16 |
Open. Self-reporting since 5423b79, but cam-H shows the guard only covers the sensor's range: a bar can clear the 256 floor and still sit under everything the scene ever reads. |
| D. Two guards validating each other | 0f5fc80, 14a1d61 |
Open. |
| E. The evidence model is missing a dimension | b4a54f0, 2026-08-14, cam-H 2026-08-16 |
Half closed. The unverified-day revert (19dcd74) and probe_backoff are trend-aware; the latter only genuinely so since 2026-08-16, when the suspension stopped being a snapshot predicate. The brightening hold's ratchet is still level-only. |
| F. A property stated over one snapshot | cam-H 2026-08-16 | Open, and newly named. The defect is in the oracle, not the machine: purity bought counterfactual reach, not temporal claims. See §4F. |
| G. The corpus cannot falsify what its noise model cannot express | min_smooth_since_probe 2026-08-16, cam-D 2026-08-16 | Open, named the day the fix for F shipped with a bug its own green corpus could not see. See §4G. |
That table is the substance of my disagreement with the "these mechanisms now interact combinatorially, producing edge cases faster than patches retire them" reading. Two of the seven generators genuinely did close, and B closed because someone stated a closure criterion ("every guess must have a deadline") rather than patching one more path. The remaining five are not combinatorial noise — they are five nameable, recurring mistakes, each with a statable rule that would prevent it. That is a much more tractable situation than "combinatorial explosion", and a much worse one than "naturally converging".
One honest correction to the optimism above, from 2026-08-16: F was found not by another incident of an old kind but by the fix for E producing a new instance of E, and it took a live camera and a person flipping a light switch to notice. The generators are tractable; the oracle for them is still behind the code.
3. The invariant nobody wrote down¶
Every stuck-mode incident in the record shares one signature that nobody has remarked on:
A service restart fixes it.
cam-E 2026-08-03. A dim outbuilding 2026-08-13 — "only a manual service restart
(which replants the baseline from scratch) recovered it". cam-L
2026-08-14 — restarted at 08:07 and switched to day within ten seconds, off
gain 257, the very same reading it had been calling night for four hours.
That is not a coincidence and it is not a workaround. It is the diagnosis. A restart discards all accumulated history and re-decides from present evidence alone, and it lands on the right answer because the evidence was never ambiguous — only the history was. So the property this subsystem is supposed to have, and has never stated, is:
Restart-equivalence. If a cold start on the current evidence would decide mode M unambiguously, the running machine must converge to M within a bounded time T.
Two refinements make it honest rather than glib:
- "Unambiguously" matters: inside the dead-zone a cold start cannot decide
either — it adopts and schedules a verification. So the invariant only binds
where the evidence is decisive. That qualifier is already encoded in exactly
one place in the code (the adopted-day confirm test: "the reading would
have decided day on its own from
DN_UNKNOWN"`). Generalising it is the cheap part. - T is not zero. History legitimately matters here: it is what keeps the
IR-cut from clunking every hour in a dark closet. So the invariant must be
bounded-time, not instantaneous.
T = night_reconfirm_sis the natural bound — one reconfirm interval, which is precisely the guaranteenight_reconfirm_swas introduced (b3eec71) to provide, and whichprobe_backoff× the skip gate has been quietly eroding ever since.
Stated that way, every one of the twelve incidents is a violation of the same invariant, and the reason patches haven't converged is that each one closed a path to the violation rather than the violation. There are many paths.
This is also the most valuable thing on this page, because it is executable: given a gain trace, you can evaluate the cold-start verdict at every sample, compare it to what the running machine believed, and flag any decisive divergence lasting longer than T. That is a test oracle, and it is the thing this subsystem has never had.
4. The three open generators, and the rule that closes each¶
C. Bars derived from measurements, with no awareness of the measurement's range¶
total_gain has a hard floor at 256 (1.0× — units 0) and a practical
ceiling at the sensor's max analog+digital gain. Several bars are computed as
a fraction of a previously measured gain, with nothing checking the result
lands inside that range:
f8a7b21:day_gain_pct% of a baseline sampled mid-transition → a trigger no real light source could reach.a5dae07(1): flooring the trigger attotal_gain_day_thresholdwhen the resting baseline is below that threshold → a permanent false "day".- 2026-08-14: the failure ratchet demanded
60%of315= 189, below the floor of 256. Unsatisfiable, so the entire sustained-brightening path was silently dead — and nothing said so.
Note the shape: the ratchet is a geometric sequence. Each dawn probe latches
probe_fail_smooth at roughly the previous one's level, and each failure
demands another day_gain_pct% below that. On any camera whose dawn ramp
carries the night-pipeline gain toward the floor, the ratchet will
eventually demand a gain below the floor. That is not an edge case; it is the
terminal state of the recursion, reached on cam-L after four probes.
Rule: any bar derived from a measurement must be checked against the measurement's physical range, and an unsatisfiable bar must be logged.
Cheapest useful action in the whole file: a dn_bar_reachable() helper plus a
LOGW at the point a bar is latched. On 2026-08-14 that would have printed
"brighten ratchet 189 is below the gain floor 256 — brightening path
disabled" at 06:20:29, four hours before anyone noticed by eye. Done in
5423b79.
Amendment, cam-H 2026-08-16 — the rule as written is too weak. A
bar can clear the sensor's floor by a wide margin and still be unreachable,
because the range that matters is not the sensor's, it is the scene's. On
that camera the ratchet latched at the room's ordinary resting night level
(1653), demanded 60 % of it (992 — analog gain 62), and the room lives between
analog 85 and 100 all night. 992 is nowhere near the 256 floor, so
dn_bar_reachable() had nothing to say; the sustained-brightening path was
nevertheless dead for the whole night, and turning the room light on moved
nothing at all. The diagnosis is not that the guard failed — it is that the
bar was derived with the wrong constant, and that a guard keyed to the
hardware could never have caught it:
Rule (extended): a bar derived from a measurement must be checked against the range the SCENE can reach, not only the range the sensor can represent. If the fraction that defines the bar was borrowed from a test answering a different question, that is the bug — fix the derivation, not the guard.
Second amendment, cam-D 2026-08-16 (~10:03–11:28) — the guard was
checking a different number from the gate it guards. dn_bar_check() was
handed the nominal bar, baseline * (100+pct)/200, while the comparison that
actually gates the hold is that bar times DN_BRIGHTEN_MARGIN. At
baseline = 326 the nominal bar is 260.8 — above the 256 floor, so the guard
stayed silent — and the operative gate is 252.98, below it, so the brightening
path was dead. An hour in night mode in daylight with zero diagnostic
output, ended by a human raising total_gain_day_threshold to 700 for that
camera by hand. The silent dead band is exactly baseline ∈ [320.0, 329.9),
and the guard fired correctly either side of it (at baseline = 315 after a
restart the nominal bar is 252 and it warned), which is what made the failure
read as a camera quirk rather than an off-by-one multiplication.
The arithmetic is trivial; the structural cause is not, and it is the same one
as generator F. The guard and the gate each derived the bar themselves from
baseline and day_gain_pct, so nothing forced them to agree, and they
quietly stopped agreeing when DN_BRIGHTEN_MARGIN was introduced into one of
them. The fix is dn_hold_gate(): one definition, called by both.
Rule: a guard must be handed the value the guarded comparison uses — not a value derived alongside it. If the guard recomputes the bar, the guard is a second implementation and will drift.
Third amendment, same day, three cameras: the rule applies to CONFIG bars
too. total_gain_day_threshold is not derived from a measurement, so
dn_bar_check() has never looked at it — but it fails identically. Set below
what a room's day pipeline actually produces, the comparison it gates
(tg < total_gain_day_threshold → confirm day) can never come true: every
probe lands ambiguous, the verify expires, the camera reverts, and it spends
one audible IR-cut pair per reconfirm interval sitting in night mode in
daylight. Measured on 2026-08-16, all within a few hours:
| camera | threshold | best day-pipeline gain | ratio |
|---|---|---|---|
| cam-D | 300 (default) | — (fixed by hand to 700) | — |
cam-E 192.168.1.100 |
300 (default) | 531 (from 700 → 591 → 531) | 1.77× |
cam-F 192.168.1.100 |
450 | 2250 (from 3299 → 2629 → 2250) | 5.0× |
The machine's behaviour is correct in all three: the day pipeline is the
trustworthy judge, it judged "not day", and night is the recoverable side. What
was wrong is that it never said why — the revert logs unverified day, gain N
and never names the threshold that made it unverifiable, so each diagnosis cost
an SSH session and a hand-read of day-pipeline gains out of syslog. There is one
important difference from a derived bar, and it makes the diagnostic more
valuable rather than less: a derived bar gets re-derived and may fix itself,
whereas this one sits there until a human edits the config.
Rule: the unsatisfiable-bar rule is about the comparison, not about where the number came from. A CONFIG threshold that no reading in this scene can satisfy is as dead as a derived one, and the machine is the only party in a position to notice — it has both numbers.
The warn is opt-in (daynight.diagnose_thresholds, default 0), for the
same reason trace_path is: its condition persists until a human edits the
config, so unconditionally it is one WARN per reconfirm interval forever — pure
log growth on a fleet that does not need it, and far more than is needed to
make the point on one that does. Corpus 13 carries the key and asserts the
message; corpus 14 is the default-off twin and asserts the silence, so removing
the gate fails exactly one scenario instead of none.
DN_DAY_THR_UNREACHABLE (1.5×) gates the warn so a dawn ramp stays quiet: a
ramp's best reading sits just above the threshold and crosses it a probe or two
later, while a mis-set threshold is missed by a wide factor (1.77× and 5.0×
above). Corpus 09, 08 and 06 are the negative controls and emit it zero times;
corpus 13 is the positive one.
Considered and rejected: refusing to plant a baseline that makes the hold
unsatisfiable (baseline >= floor / (0.8 · 0.97) ≈ 1.29× floor). It is the
more ambitious fix and it does subsume the arithmetic, but it is wrong here for
a measured reason. night_baseline <= 0 is not a neutral state: the skip
gate's can_judge requires a baseline, so refusing to plant makes
solidly_night permanently false and every periodic reconfirm fire a
physical probe. On the exact camera class this would target — cam-J and
the cam-K closet, resting at 256–268 — that converts roughly zero clicks a
night into one per night_reconfirm_s, i.e. it re-creates the complaint the
skip gate was written for (2026-08-04, "das klacken der IR blende nervt").
Flooring the planted baseline instead of refusing it fares no better: it
fabricates a measurement, and at the floor the bar is satisfiable only by a
reading strictly below 256, which no sensor produces — still dead, now with a
lie in the log. The honest position is that near the floor all three
baseline-derived bars degenerate together, the hold is legitimately dead, and
what the machine owes the operator is to say so — which is what the guard now
does correctly.
Which is exactly what DN_RATCHET_MARGIN does: the ratchet stopped borrowing
day_gain_pct (a pipeline discriminator, 0.74 stops, correct for the IR-cut
transitions it was written for) and got its own constant sized for the
question it actually asks — "is this new evidence, distinguishable from the
evidence that already failed?" — at one quarter stop. Incidentally this also
pushes the geometric recursion above further from the floor: the same
2026-08-14 latch now derives 265 rather than 189.
D. Two guards validating each other¶
Both instances have an identical structure: mechanism A derives its reference from a value that mechanism B is concurrently moving, so A and B confirm each other instead of either being the other's escape hatch.
0f5fc80: the baseline drifted toward raw gain, and the brightening bar was derived from the baseline. Noise ratcheted both together → overnight flap.14a1d61: the baseline drifts towardsmooth_tg, and the skip gate's "solidly night" bar is derived from the baseline — so over hours the bar chased the (day-level) gain down and "still deep in night" stayed true.- cam-H 2026-08-16: the same chase in the third consumer, the
sustained-brightening hold, and the fastest-acting instance of it by a wide
margin. The hold's bar is
(100+pct)/200of the baseline, so it sits 22.4 % away;DN_BASELINE_ALPHAcloses 22.4 % in about 25 s, against aDN_BRIGHTEN_CONFIRM_MSof 30 s. The debounce loses that race by construction. Traced tick by tick on the corpus 10 replay: a 23 % light-on left the margin test 2.2 % short at the instant the step completed and falling from there, so it never opened on any tick — and 25 s later the bar had dropped throughsmooth_tg, taking the>= barbranch, which re-arms and clears the hold. A permanently brighter room therefore reads as "nothing has changed" from then on. Not merely slow: erased.
The fix in all three cases was the same idea, found three times independently:
anchor the test on a value frozen at the moment of an actual measurement
(probe_fail_smooth, or for the hold a snapshot of the baseline taken while
the scene was still at rest) rather than one that drifts.
Rule: an "is there evidence of change?" test must be anchored on a frozen measurement, never on a value that any other mechanism updates.
Corollary (2026-08-16): where a bar is derived from a drifting value, the drift rate and the confirm window are not independent knobs. If the drift can close the bar's own gap faster than the confirm can elapse, the test is not "strict", it is unreachable — the same failure mode as an unsatisfiable bar (generator C), arriving through time rather than through range. State the ratio when either constant is touched.
The audit that item asked for is now two-thirds done: night_baseline fed the
brightening bar, the skip gate and dn_day_trigger() and drifted under all
three. The skip gate got its frozen anchor in 14a1d61, the hold got
brighten_ref on 2026-08-16, and dn_day_trigger() is informational in the
adaptive regime (night→day is probe-mediated since b4a54f0), so it is the
one consumer where the drift is harmless. Note also where the hold's
snapshot has to be taken: freezing it when the scene crosses the bar is far
too late, because the bar is 22.4 % away and the baseline has been chasing for
tens of seconds by then — measured, that late freeze captured 2353 where the
true pre-event rest level was 2397, and 1.8 % of contamination was itself
enough to keep the margin shut. It freezes at the 3 % DN_BRIGHTEN_MARGIN
band instead: the reference stops tracking the moment the scene is measurably
brighter, not once it is decisively brighter.
E. The evidence model is missing a dimension¶
Every test in the file is metric <op> threshold — a level, sampled at an
instant. But the signal that distinguishes the two scenes the dead-zone
conflates is the derivative:
- a static dim scene (the one the unverified-day revert exists for) sits still;
- a scene mid-dawn is in free fall through the dead-zone.
b4a54f0 is the same gap seen from the other side: night-pipeline gain has no
usable level semantics at all, so the fix was to stop deciding on it. The
2026-08-14 fix (19dcd74) adds the file's first derivative-aware rule — revert
an unverified day only once the improvement has stopped.
Two more places are still level-only and demonstrably shouldn't be:
probe_backoff. ~~Doubling the reconfirm interval in the middle of a monotone descent is exactly backwards.~~ Closed by the trend suspension indn_next_probe()(dn_trend_falling()): while the gain sits belowDN_BRIGHTEN_MARGINof the frozenprobe_fail_smooth, the multiplier is suspended and the deadline falls back to one plainnight_reconfirm_safter arming. Note the reference point it needed was already in the file and already frozen — generator D's rule supplied it. On cam-L the backoff hit its ×4 cap at 05:58 while gain was falling through three orders of magnitude, and that cap is what turned a bad revert into a four-hour outage; corpus scenario 08 now measures the difference (recovery at 1276 s instead of 3976 s) and fails without it.
Closed properly on 2026-08-16, not before. As first written the
suspension read the instantaneous smooth_tg, which means it was still a
level test sampled at an instant — generator E's own defect, reintroduced
inside generator E's own fix. It therefore un-fired as readily as it fired.
On cam-H it suspended an ×4 backoff at 23:21:08 (gain 1599 against a
1653 anchor), pulling the reconfirm from 03:12 in to 00:12; the room then
dimmed on its own, the predicate went false, and the four hours came
straight back — long before the deadline it had pulled in. The evidence had
been measured and then discarded. It now reads min_smooth_since_probe,
the lowest smoothed gain since the anchor was frozen, so "the scene has been
measurably brighter than confirmed night" is a fact about the interval
rather than about this tick. The skip gate's ratchet-anchor override reads
the same value for the same reason: a deadline is a single instant, so
testing the instantaneous gain there throws away every brightening that
happened between deadlines.
- The brightening hold, whose ratchet asks "how far below the last failure"
when at dawn the answer that matters is "still falling, and for how long".
DN_RATCHET_MARGIN (2026-08-16) fixed the scale of that question, not its
level-only-ness; the hold is still the one place in the file that asks it
purely as a threshold.
Rule: any test that must distinguish "dim" from "getting brighter" needs a reference point in time, not just a threshold.
F. A property stated over one snapshot cannot see a defect that lives between snapshots¶
Worth naming separately, because it is a defect in the oracle, not in the
machine, and it is the reason the 2026-08-16 bug survived a week of work aimed
squarely at it. dn_next_probe() is pure over one dn_evidence, and both
properties tests/dn-probe-props.c asserted were statements about a single
evaluation: monotonicity compares two evidence values, the reconfirm bound
reads one. The trend suspension satisfied both, on every one of ~10⁶ points,
while still handing four hours back the moment a room dimmed again — because
across a sequence of evaluations, brighter evidence had no durable effect.
Purity bought counterfactual reachability, which is real and was the point;
it did not by itself buy temporal claims.
The corpus caught nothing either, and for a related reason: the nine scenarios all replay monotone transitions (dawn ramps, boots, a single dip). None of them contained a brightening that came and then went, which is precisely the trajectory that distinguishes a latched predicate from a sampled one. Scenario 10 exists to be that trajectory.
Rule: for every piece of state the schedule carries between ticks, state a property over the SEQUENCE, not only over the snapshot — at minimum "evidence once measured is never worth less later". And when adding a scenario, ask what its gain curve does non-monotonically; a corpus of monotone traces cannot falsify a memory bug.
G. The corpus can only falsify what its noise model can express¶
Named the same day as F, for the same reason: the fix for F shipped with a bug
its own tests could not see. min_smooth_since_probe was a plain running
minimum of the smoothed gain — an order statistic, and the only quantity in
this file with no noise rejection whatsoever. The expected depth of a running
minimum grows without bound in the number of samples, so on a perfectly static
scene it descends until it crosses the 0.97 anchor bar and suspends the
backoff. Worse, the sample count is set by the reconfirm interval, which is the
very thing the backoff lengthens: longer backoff → more samples → deeper
spurious minimum → suspension fires → backoff defeated. Self-reinforcing, in
the wrong direction. Measured on a static 3500 scene with realistic stochastic
noise and a real 3600 s reconfirm: an audible probe every hour all night, 7
board switches against a budget of 4, emitting log lines structurally
identical to the ones corpus 10 asserts as proof the fix works.
The header comment shipped with the mechanism correctly described and then
dismissed it, on the strength of a green corpus. The corpus was green because
noise_factor() was a bounded sum of two sinusoids. A deterministic periodic
signal has a fixed, finite set of extremes, so any running minimum over it
converges after one period and stops moving — the failure mode was
structurally inexpressible, at any noise_pct, for any duration. Eleven
scenarios reported zero click regressions while the backoff was inoperative.
Two things follow, and the second is the one worth keeping:
Rule: if a mechanism's behaviour depends on a statistic of a window of samples (a minimum, a maximum, a count of crossings, a time-to-first-event), deterministic test noise cannot falsify it. Use stochastic noise, and say in the scenario what statistic it is there to stress.
Rule: naming a risk in a comment is not bounding it. The comment that predicted this failure and waved it off was written in the same commit as the bug. If a risk is real enough to write down, it is real enough to need either a test that would fail on it or a stated reason no test can — and the second answer obliges you to fix the harness.
The corpus also had a hole that no amount of noise realism would have closed: every scenario asserted that changed evidence is eventually acted on, and not one asserted the converse. Scenario 12 is the null hypothesis — a static scene, stochastic noise, a real 3600 s reconfirm, and a click floor. It is the only scenario in the corpus that fails when the machine becomes too eager, which is the direction every fix in this record pushes.
assert_dip_monotone() is that property for min_smooth_since_probe: for two
evidence values with identical current readings, the one that records a
brighter past minimum must never be granted a later probe.
5. Verdict on a redesign¶
No rewrite. A targeted restructure, and a test harness first.
The switching machinery — settle, dwell, hysteresis, re-assert, the board hook chain — is correct and expensively earned; nothing in the incident record since 2026-08-02 touches it. The defects are all in probe scheduling and evidence modelling. A rewrite would relitigate twelve incidents' worth of hard-won detail with no test net underneath it, which is a reliable recipe for incidents 13 through 18.
What is actually missing is not a better structure. It is an executable oracle. You cannot safely refactor this file without one; with one, you can refactor it incrementally and cheaply.
The destination: dn_next_probe(evidence)¶
Collapsing verify, brighten_*, probe_backoff, probe_fail_smooth,
last_phys_probe_ms, the skip gate and the anchor override into one pure
function over a small typed evidence struct is the right end state. The win is
not fewer lines — it is that a pure function is directly unit-testable and,
more importantly, property-testable:
Monotonicity. For evidence
e1,e2identical except thate2is strictly brighter:dn_next_probe(e2) <= dn_next_probe(e1). Brighter evidence must never buy a later correction.
This single property is violated by 0f5fc80, by 14a1d61, and by the
2026-08-14 incident — where an ambiguous (dead-zone, i.e. brighter) revert
produced the same 4 h schedule as a confirmed-night revert. Three of the
open incidents, one assertion. That is what makes the collapse worth doing.
But it is a bad first step, because there is currently no way to show the collapse is behaviour-preserving.
6. The replay harness¶
Step 0 — there is no trace to replay¶
Checked on cam-L 2026-08-14: no gain recorder exists anywhere on the camera, nothing but syslog lines. Every past "verified against timpsd-sim" was therefore a hand-reconstructed scenario, not a replay of recorded data — including the verification of the 2026-08-14 fix itself. That is the binding constraint, and it is why the harness idea has to start with a recorder rather than a replayer.
Sketch: a daynight.trace_path key appending one line per N samples —
t_mono, cur, tg, luma, baseline, smooth_tg, day_trigger, probe_fail_smooth,
verify_mode, verify_in_s, backoff. Roughly 30 lines of code. Write to
tmpfs with a size cap, not to flash (these are camera-grade eMMC/NAND and
a 5 s cadence is ~900 KB/day); persist or upload only on demand. The /events
SSE stream already carries most of this and a LAN-side collector is the other
viable shape.
Step 1 — the two-pipeline problem, stated honestly¶
A faithful replay needs the gain for both pipelines at each timestamp, because which one gets read depends on the decision the machine under test makes — and a patch that changes the decision immediately needs the counterfactual reading. In the field only one pipeline is observable at a time. True replay from field data alone is therefore impossible, and the harness should say so rather than pretend:
- (a) Regression replay on synthetic two-pipeline traces — fully
faithful, the workhorse. This is what the 2026-08-14 fix was verified
against: a fake
isp-m0scrape file plus aswitch_cmdstub that records the active mode, with the driver serving the matching pipeline's gain. - (b) Incident replay on a field trace — faithful only up to the first point where the machine's decision diverges from the recorded one, and it must then stop and report "diverged at T" rather than continue on invented data. Limited, but it answers the exact question every one of these twelve commits tried to answer by hand: would this patch have diverged from the recorded bad behaviour, and when?
Step 2 — time compression¶
A dawn incident spans hours. ms_now_us() is a single static inline in
src/util.h — one chokepoint — so an opt-in MS_CLOCK_SCALE for sim builds
is straightforward. The DN_* compile-time constants must scale with it,
which needs a one-line SIM_CFLAGS hook on the sim Makefile target (the
2026-08-14 verification worked around this by compressing the config
timings 15× and leaving the constants alone, which is adequate for a targeted
check and not for a corpus).
Step 3 — assertions, not eyeballs¶
Each scenario declares expectations: mode at time T, maximum wrong-mode duration, maximum board switches (the click budget is a first-class requirement here, not a nicety), plus the two global properties from above — restart-equivalence and probe-time monotonicity.
Step 4 — the corpus¶
One scenario per historical incident. The numbers are all in the commit
messages already, which is the one genuinely good side-effect of this repo's
verbose-commit convention: 53c21b4 (boot transient 15000–20000),
f8a7b21 (65 % of baseline), 0f5fc80 (post-revert elevated resample),
bd21ce6 (boot at 731 in the dead-zone), a5dae07 (resting 256–268 vs
threshold 300; baseline planted mid-descent at 10856/5148), b4a54f0 (10856
resting, dawn dip to 820, day pipeline 8192), 43c2b16 (booted after dark on
a stale day), 14a1d61 (fail at 284, baseline chases to 257–266), and
2026-08-14 (9024 → 4813 → 2425 → 708, reverts at 1436 and 452, ratchet 189 vs
floor 256).
7. What not to do¶
- Do not add more escalation timers. A "dead-zone revert retry ladder" —
shortening the reconfirm interval after an ambiguous revert and doubling it
per consecutive one — was written and considered for the 2026-08-14 incident
and rejected: it recovers from the wrong revert instead of preventing
it, and it does so by re-introducing a bounded version of the pre-dawn probe
volley that
fad4f40was written to remove — several audible IR-cut pairs through every dawn that lands in the dead-zone, on every camera. The fix that shipped prevents the revert and costs fewer clicks than the status quo. Prefer that direction: this subsystem already has nine gates whose only job is rationing clicks, and adding a tenth that spends them is moving backwards. - Do not rewrite before the harness exists.
- Do not treat
probe_max_skip_sas the answer to anything. It is a 12 h backstop. Twice now (14a1d61, 2026-08-14) an incident has been characterised as "well withinprobe_max_skip_s", which is true and irrelevant — a camera in the wrong mode for four hours is already a failure.
8. Suggested order¶
- ~~Derivative-aware unverified-day revert~~ — done,
19dcd74. - ~~
dn_bar_reachable()+ a warn on an unsatisfiable bar.~~ — done,5423b79. Generator C is self-reporting instead of silent. - ~~Trace recorder (tmpfs, capped).~~ — done,
c84fcef(daynight.trace_path, opt-in). - ~~Replay harness + scenario corpus, with the restart-equivalence and
monotonicity assertions.~~ — done,
e06bf41(virtual clock) +e0c629b(scripts/dn-replay.py, nine scenarios). - ~~
dn_next_probe()collapse, validated against the corpus.~~ — done,src/daynight_probe.h+tests/dn-probe-props.c. - ~~Revisit
probe_backoff's trend-blindness (generator E).~~ — done,dn_trend_falling(); see the generator-E section above. - ~~Latch the trend suspension on a measurement, not a predicate
(generators E + F).~~ — done, 2026-08-16:
min_smooth_since_probeindn_evidence, read by bothdn_trend_falling()and the skip gate's ratchet-anchor override, plusassert_dip_monotone()as the sequence property that would have caught it. Corpus scenario 10. - ~~Give the failure ratchet its own margin (generator C, extended).~~ —
done, 2026-08-16:
DN_RATCHET_MARGIN(one quarter stop) instead of borrowingday_gain_pct. Pinned by thecam-H B1cases in the property test — both that a 21 % brightening can now start a hold, and that an identical re-dip still cannot. - ~~Stop the brightening hold's bar being derived from the drifting
baseline (generator D, third consumer).~~ — done, 2026-08-16:
brighten_ref, released again byDN_HOLD_REF_LEAD. This is the one that makes the fast path actually fire for the everyday event — corpus 10 goes from 489 s to 35 s — and the one that most needed its second-order effect caught, since the first version of it walkedfad4f40's probe volley down corpus 09's dawn ramp. - ~~Hand the generator-C guard the value the gate uses (
dn_hold_gate()).~~ — done, 2026-08-16 after the cam-D incident. One definition, called by the schedule and both guard call sites, because the arithmetic error was only possible while the guard re-derived the bar itself. - ~~Make
min_smooth_since_probenoise-safe (generator G).~~ — done, 2026-08-16: a reading must hold forDN_BRIGHTEN_CONFIRM_MSbefore it enters the minimum. A running minimum is an order statistic and had no business being the one unsmoothed quantity in the file. - ~~Make the harness noise stochastic, and add a null hypothesis
(generator G).~~ — done, 2026-08-16:
noise_factor()is seeded gaussian + AR(1), and corpus 12 asserts a click floor on a static scene. These are the two changes that would have caught items 10 and 11 before they shipped, and they are the general fix rather than the two specific ones.
What the corpus is worth, and how to keep it worth that¶
The corpus is only evidence if a scenario fails against a build without the
fix it is named for. Four of the twelve assert a decision rule rather than
a new log line, and all four were checked that way (08 against the anchor
override and against the trend suspension, 09 against 19dcd74's extension
and — newly — as the negative control for DN_HOLD_REF_LEAD, 10 against
both the latched suspension and the frozen hold reference, 11 as the
sensitivity floor via forbid_log) — each fails on behavioural assertions,
not merely on expect_log.
Scenario 09 earned a second job on 2026-08-16 and it is worth naming, because
it is the only reason a fix that looked finished was not shipped broken: the
first version of the frozen hold reference passed 10 and 11 and the whole
property test, and 09 failed it instantly on the click budget — 14 board
switches against 9, six sustained-brightening pairs walking down one dawn
ramp. The mode assertions all still passed. A corpus that only checked "did it
end up in the right mode" would have shipped it. max_switches is not a
nicety. Scenario 10 is the sharpest of the three: pre-fix it
never leaves night at all inside the run (wrong-mode 3100 s, mode@3500 and
mode@5900 both wrong), and probe_max_skip_s is deliberately left at its
43200 default so the backstop cannot quietly rescue it.
Every scenario now pins total_gain_day_threshold and
total_gain_night_threshold explicitly (at the historical 300/3000), rather
than inheriting the compiled defaults. This is not tidiness: the corpus encodes
incidents, and an incident happened under a specific configuration. While the
scenarios inherited the defaults, raising a default silently re-ran all thirteen
against a configuration none of them was calibrated for — so the defaults could
not be changed without invalidating the evidence, and the evidence could not
show whether the change was safe. Pinning decoupled the two, which is what made
the 300 → 768 default bump measurable at all.
Rule: a scenario must pin every config value its incident depended on. A corpus that inherits defaults cannot be used to evaluate a change to those defaults — precisely when you most need it.
Scenario 10 also carries the generator-F lesson in its shape. Its gain curve is non-monotone on purpose — a brightening that comes and then goes, before the deadline it pulled in. Every one of the first nine replays a monotone transition, which is why none of them could falsify a schedule that forgets. When adding a scenario, ask what its curve does non-monotonically before asking what it asserts.
Build a negative control by reverting the fix's hunk in a copy of the
current tree, never by building the historical commit. Trees older than
e06bf41 have no MS_CLOCK_SCALE hook: they silently ignore SIM_CFLAGS,
run in real time while the harness drives them 60× faster, reach none of their
own deadlines, and come back green on everything — a false negative that reads
exactly like "this scenario does not discriminate". dn-replay.py now refuses
such a binary outright (SimRun.check_clock), but the cheaper habit is not to
build one.
Next candidates, in the same spirit: the brightening hold's ratchet is the
last level-only test generator E names — DN_RATCHET_MARGIN corrected its
scale but it is still a threshold, not a trend — and night_baseline still
feeds three consumers while drifting under all three (the generator-D audit).
cam-H put a sharper edge on that last one: on the night in question the
hold was blocked not by the ratchet but by the baseline-relative arming
margin, because the baseline had already drifted down to chase the very
brightening the hold was meant to notice. That is 14a1d61 again, in the one
consumer the anchor override does not cover.
Also worth a pass now that generator F is named: audit the remaining
between-tick state in dn_thread the same way. brighten_since_ms,
day_verify_ref, brighten_ref and probe_backoff itself are each carried
across ticks and each has only snapshot properties asserted about it.
And the generator-G pass that has not been done: min_smooth_since_probe was
the only order statistic in the file and it is now debounced, but the audit
question generalises — for each derived quantity, is it a smoothed value, a
frozen measurement, or a statistic of a window? The first two are safe; the
third needs stochastic noise pointed at it and a stated reason it is bounded.
Right now the answer is "none remaining", and the value of writing that down is
that the next one added has to justify itself.
One more sensitivity fact worth recording rather than discovering again: the
debounce on the minimum means a dip must be deeper than the scene's own noise
band to latch. The real cam-H 23:21 dip (1599 against a 1653 anchor,
3.3%) would not latch under realistic AGC noise. That is the correct trade —
the alternative is no working backoff on any camera — and the periodic
reconfirm still bounds recovery at one night_reconfirm_s. But it means corpus
10's dip is deliberately deeper than the incident's, and anyone reading that
scenario as "this is what the camera saw" will be misled about the sensitivity.
And one specific piece of unfinished business from the third fix: the hold now
distinguishes a step from a ramp by how far its frozen reference has led the
baseline, which is a proxy for the derivative. The direct test — "is the
gain still falling?" — is the open generator-E item two bullets up, and it
would replace DN_HOLD_REF_LEAD with the thing that constant is standing in
for. Do that when the hold is next opened, not before: the proxy is measured,
bounded on both sides and has a negative control, which is more than the
subsystem's last four constants started with.
v1.12.0 dawn¶
The 1.10.1 engine wasted the dawn: the fleet logs of 125 mornings show 1276
morning looks and 238 failed morning checks (click pairs), about 10 looks
and 1.9 failed checks per camera and morning. The first failed check per dawn measures k (Kf); the
rest were spent not using it. Changes in src/daynight.c (a7721cb):
- Skip a look that cannot find day: without the illuminator the scene reads no brighter, so a jump, colour or near look with lit × 0.9 ≥ bar is not run; the lit level becomes the reference. Timer and requested looks, and a look that only learns the bar, always run. 640 of the 1276 morning looks sat above the bar.
- Creep or lamp by a 10 s step (more than 15 % between two 10 s samples
is a lamp), replacing the 15 %-over-a-minute ramp test that dawn jumps
failed. A creeping look, also a colour look, gets the jump bar and
day_gain / Kf. - Near-bar look: last dark reading × lit / reference under 0.95 × bar
starts a look without a
jump_pctstep (cam-wohn 2026-11-07: a 0.5 % miss at 08:21, then night until 09:21 although the scene would have read day from 08:25). - No lamp as the reference after a check: a settled level
jump_pctbrighter than the lit level before the check keeps the earlier reference. - Two one-line safety fixes: a railed check does not set
Kf, and a check under 1.5 ×day_gainwaits the 10 s hold (the AE still settling).
Measured: replay corpus 77 pairs, 0 failed (six new families and profiles from the measured 2026-11-07 dawns of cam-wohn and cam-schuppen); a fleet what-if over 121 mornings: failed morning checks 178 → 94, day on average 5 min later, about half of the morning looks skipped.
Deliberately not done:
- The K cap stays 2. It protects lamps after nights under foreign IR; a
second "creep K" would help only the first check after a restart, and
kdrifts 20-40 % within one morning anyway. Kfstays 1 h. Six expiries in 125 mornings against outlierKfvalues that would block for the rest of the morning; 3 h or the whole morning bought 3 failed checks.- No persisted or per-camera
k. It varies 1.5-2× per camera across mornings, and a stale value is a new stuck mode that only a restart clears. - No prior
kbefore the first check. −84 % failed checks on paper, +1 h of night in the low-k rooms, where one lamp decides day.
Report, with the field numbers, traces and the rejected variants:
dev_notes/DAWN_FABLE_REPORT_2026-11-07.md.