Skip to content

Day/Night — Design Notes

Superseded twice: the 2026-08-17 redesign, and in v1.10.1 the simple engine of dev_notes/dn-redesign.md.

Status: historical, and deliberately kept. This page analysed the machine as it stood on 2026-08-14 and argued for a targeted restructure once an executable oracle existed. That oracle was then built (trace recorder, replay harness, scenario corpus), and on 2026-08-17 the file was redesigned rather than restructured — see dev_notes/DAYNIGHT_REDESIGN_2026-08-17.md for the new design and Day/Night for what ships. Read this page for the evidence: the twelve incidents, the invariant they all violate (§3), the failure generators (§4) and the harness design (§6) are all still the reason the current code looks the way it does, and any future change should be measured against them. Section 5's "no rewrite" verdict was explicitly conditional on the harness being missing, and that condition no longer holds; section 7's "what not to do" list still does.

Day/Night documents what src/daynight.c does. This page is about why it kept needing fixing, and what to change about that. It is a working document for whoever touches the file next, not user documentation.

Why this page exists

Of the 17 fix(daynight): commits to date, 12 are the same behavioural class: the camera latches into the wrong mode and does not get out, or it flips between modes. 53c21b4, c78dbcb, b3eec71, f8a7b21, 0f5fc80, bd21ce6, fad4f40, a5dae07, b4a54f0, 43c2b16, 8c8dc1f, 14a1d61, plus the 2026-08-14 cam-L dawn incident that produced this page.

Twelve fixes for one symptom class is a signal about the design, not about the fixes. Each of those commits is individually well-reasoned and each closed a real, reproduced failure. So the interesting question is not "was fix N correct" (they were) but "why does fix N+1 keep arriving".

1. What the machine actually is

One thread, one 500 ms poll loop, one metric (total_gain, IMP [24.8] linear, 256 = 1×, inverted: high gain = dark). Everything else is state carried between ticks. Current state, grouped by what it is for:

Group State
Decision cur, pending_target/pending_since_ms (hysteresis candidate), last_switch_ms (dwell)
Measurement conditioning hist[] (brightness ring), settle_hist[]/settle_n (AE-stability ring), settle_floor_ms/settle_hard_ms, smooth_tg (night-only EMA), scraped_b/next_scrape_ms
Night model night_baseline, baseline_logged, night_entered_ms
Probe scheduling verify{mode,at_ms}, brighten_since_ms, brighten_armed, probe_day_ms, probe_verdict_at_ms, probe_backoff, probe_fail_smooth, last_phys_probe_ms, day_verify_ref/day_verify_ext
Backstops osc_hist[]/osc_n/osc_freeze_until_ms, reassert_at_ms/reassert_left

Counting the gates — the independent mechanisms that can block, delay, trigger or override a mode resolution — gives fifteen, not the eight one might guess from the section headings:

  1. boot/re-enable settle (floor + AE-stability extension)
  2. dwell (transition_s)
  3. pre-switch hysteresis (5 s)
  4. adaptive baseline + symmetric EMA drift
  5. sustained-brightening hold, with its arming edge
  6. …and its arming margin (DN_BRIGHTEN_MARGIN), against a frozen rest reference (brighten_ref / DN_HOLD_REF_LEAD)
  7. failure ratchet (probe_fail_smooth × DN_RATCHET_MARGIN)
  8. exponential backoff on failed probes
  9. passive-evidence skip gate
  10. probe_max_skip_s outer bound on the skip gate
  11. ratchet-anchor override of the skip gate (14a1d61)
  12. dead-zone adoption + its verify deadline, both directions
  13. probe three-outcome classification → ambiguous arms a verify (8c8dc1f)
  14. post-probe AE-stability revert gate
  15. oscillation-breaker freeze

(plus the post-switch re-assert series, which is actuation rather than decision). Nine of the fifteen exist purely to decide when to spend an audible IR-cut click. That ratio is the story of this file.

2. What the incident record actually says

The twelve incidents are not twelve unrelated bugs, and they are not one undifferentiated mass either. They sort into seven generators (five, until two more named themselves on 2026-08-16):

Generator Incidents Status
A. Actuation / measurement plumbing — the Set landed mid-ramp, the AE hadn't converged 53c21b4, c78dbcb, b3eec71 (part) Closed. No recurrence since 2026-08-02.
B. A reachable state with no pending decision bd21ce6 (UNKNOWN), 43c2b16 (adopted DAY), 8c8dc1f (ambiguous probe landing) Closed, and closed checkably: after 43c2b16+8c8dc1f every reachable (cur, verify) pair has an armed deadline.
C. A derived bar that goes unsatisfiable f8a7b21, a5dae07(1), 2026-08-14, cam-H 2026-08-16 Open. Self-reporting since 5423b79, but cam-H shows the guard only covers the sensor's range: a bar can clear the 256 floor and still sit under everything the scene ever reads.
D. Two guards validating each other 0f5fc80, 14a1d61 Open.
E. The evidence model is missing a dimension b4a54f0, 2026-08-14, cam-H 2026-08-16 Half closed. The unverified-day revert (19dcd74) and probe_backoff are trend-aware; the latter only genuinely so since 2026-08-16, when the suspension stopped being a snapshot predicate. The brightening hold's ratchet is still level-only.
F. A property stated over one snapshot cam-H 2026-08-16 Open, and newly named. The defect is in the oracle, not the machine: purity bought counterfactual reach, not temporal claims. See §4F.
G. The corpus cannot falsify what its noise model cannot express min_smooth_since_probe 2026-08-16, cam-D 2026-08-16 Open, named the day the fix for F shipped with a bug its own green corpus could not see. See §4G.

That table is the substance of my disagreement with the "these mechanisms now interact combinatorially, producing edge cases faster than patches retire them" reading. Two of the seven generators genuinely did close, and B closed because someone stated a closure criterion ("every guess must have a deadline") rather than patching one more path. The remaining five are not combinatorial noise — they are five nameable, recurring mistakes, each with a statable rule that would prevent it. That is a much more tractable situation than "combinatorial explosion", and a much worse one than "naturally converging".

One honest correction to the optimism above, from 2026-08-16: F was found not by another incident of an old kind but by the fix for E producing a new instance of E, and it took a live camera and a person flipping a light switch to notice. The generators are tractable; the oracle for them is still behind the code.

3. The invariant nobody wrote down

Every stuck-mode incident in the record shares one signature that nobody has remarked on:

A service restart fixes it.

cam-E 2026-08-03. A dim outbuilding 2026-08-13 — "only a manual service restart (which replants the baseline from scratch) recovered it". cam-L 2026-08-14 — restarted at 08:07 and switched to day within ten seconds, off gain 257, the very same reading it had been calling night for four hours.

That is not a coincidence and it is not a workaround. It is the diagnosis. A restart discards all accumulated history and re-decides from present evidence alone, and it lands on the right answer because the evidence was never ambiguous — only the history was. So the property this subsystem is supposed to have, and has never stated, is:

Restart-equivalence. If a cold start on the current evidence would decide mode M unambiguously, the running machine must converge to M within a bounded time T.

Two refinements make it honest rather than glib:

  • "Unambiguously" matters: inside the dead-zone a cold start cannot decide either — it adopts and schedules a verification. So the invariant only binds where the evidence is decisive. That qualifier is already encoded in exactly one place in the code (the adopted-day confirm test: "the reading would have decided day on its own from DN_UNKNOWN"`). Generalising it is the cheap part.
  • T is not zero. History legitimately matters here: it is what keeps the IR-cut from clunking every hour in a dark closet. So the invariant must be bounded-time, not instantaneous. T = night_reconfirm_s is the natural bound — one reconfirm interval, which is precisely the guarantee night_reconfirm_s was introduced (b3eec71) to provide, and which probe_backoff × the skip gate has been quietly eroding ever since.

Stated that way, every one of the twelve incidents is a violation of the same invariant, and the reason patches haven't converged is that each one closed a path to the violation rather than the violation. There are many paths.

This is also the most valuable thing on this page, because it is executable: given a gain trace, you can evaluate the cold-start verdict at every sample, compare it to what the running machine believed, and flag any decisive divergence lasting longer than T. That is a test oracle, and it is the thing this subsystem has never had.

4. The three open generators, and the rule that closes each

C. Bars derived from measurements, with no awareness of the measurement's range

total_gain has a hard floor at 256 (1.0× — units 0) and a practical ceiling at the sensor's max analog+digital gain. Several bars are computed as a fraction of a previously measured gain, with nothing checking the result lands inside that range:

  • f8a7b21: day_gain_pct% of a baseline sampled mid-transition → a trigger no real light source could reach.
  • a5dae07(1): flooring the trigger at total_gain_day_threshold when the resting baseline is below that threshold → a permanent false "day".
  • 2026-08-14: the failure ratchet demanded 60% of 315 = 189, below the floor of 256. Unsatisfiable, so the entire sustained-brightening path was silently dead — and nothing said so.

Note the shape: the ratchet is a geometric sequence. Each dawn probe latches probe_fail_smooth at roughly the previous one's level, and each failure demands another day_gain_pct% below that. On any camera whose dawn ramp carries the night-pipeline gain toward the floor, the ratchet will eventually demand a gain below the floor. That is not an edge case; it is the terminal state of the recursion, reached on cam-L after four probes.

Rule: any bar derived from a measurement must be checked against the measurement's physical range, and an unsatisfiable bar must be logged.

Cheapest useful action in the whole file: a dn_bar_reachable() helper plus a LOGW at the point a bar is latched. On 2026-08-14 that would have printed "brighten ratchet 189 is below the gain floor 256 — brightening path disabled" at 06:20:29, four hours before anyone noticed by eye. Done in 5423b79.

Amendment, cam-H 2026-08-16 — the rule as written is too weak. A bar can clear the sensor's floor by a wide margin and still be unreachable, because the range that matters is not the sensor's, it is the scene's. On that camera the ratchet latched at the room's ordinary resting night level (1653), demanded 60 % of it (992 — analog gain 62), and the room lives between analog 85 and 100 all night. 992 is nowhere near the 256 floor, so dn_bar_reachable() had nothing to say; the sustained-brightening path was nevertheless dead for the whole night, and turning the room light on moved nothing at all. The diagnosis is not that the guard failed — it is that the bar was derived with the wrong constant, and that a guard keyed to the hardware could never have caught it:

Rule (extended): a bar derived from a measurement must be checked against the range the SCENE can reach, not only the range the sensor can represent. If the fraction that defines the bar was borrowed from a test answering a different question, that is the bug — fix the derivation, not the guard.

Second amendment, cam-D 2026-08-16 (~10:03–11:28) — the guard was checking a different number from the gate it guards. dn_bar_check() was handed the nominal bar, baseline * (100+pct)/200, while the comparison that actually gates the hold is that bar times DN_BRIGHTEN_MARGIN. At baseline = 326 the nominal bar is 260.8 — above the 256 floor, so the guard stayed silent — and the operative gate is 252.98, below it, so the brightening path was dead. An hour in night mode in daylight with zero diagnostic output, ended by a human raising total_gain_day_threshold to 700 for that camera by hand. The silent dead band is exactly baseline ∈ [320.0, 329.9), and the guard fired correctly either side of it (at baseline = 315 after a restart the nominal bar is 252 and it warned), which is what made the failure read as a camera quirk rather than an off-by-one multiplication.

The arithmetic is trivial; the structural cause is not, and it is the same one as generator F. The guard and the gate each derived the bar themselves from baseline and day_gain_pct, so nothing forced them to agree, and they quietly stopped agreeing when DN_BRIGHTEN_MARGIN was introduced into one of them. The fix is dn_hold_gate(): one definition, called by both.

Rule: a guard must be handed the value the guarded comparison uses — not a value derived alongside it. If the guard recomputes the bar, the guard is a second implementation and will drift.

Third amendment, same day, three cameras: the rule applies to CONFIG bars too. total_gain_day_threshold is not derived from a measurement, so dn_bar_check() has never looked at it — but it fails identically. Set below what a room's day pipeline actually produces, the comparison it gates (tg < total_gain_day_threshold → confirm day) can never come true: every probe lands ambiguous, the verify expires, the camera reverts, and it spends one audible IR-cut pair per reconfirm interval sitting in night mode in daylight. Measured on 2026-08-16, all within a few hours:

camera threshold best day-pipeline gain ratio
cam-D 300 (default) — (fixed by hand to 700) —
cam-E 192.168.1.100 300 (default) 531 (from 700 → 591 → 531) 1.77×
cam-F 192.168.1.100 450 2250 (from 3299 → 2629 → 2250) 5.0×

The machine's behaviour is correct in all three: the day pipeline is the trustworthy judge, it judged "not day", and night is the recoverable side. What was wrong is that it never said why — the revert logs unverified day, gain N and never names the threshold that made it unverifiable, so each diagnosis cost an SSH session and a hand-read of day-pipeline gains out of syslog. There is one important difference from a derived bar, and it makes the diagnostic more valuable rather than less: a derived bar gets re-derived and may fix itself, whereas this one sits there until a human edits the config.

Rule: the unsatisfiable-bar rule is about the comparison, not about where the number came from. A CONFIG threshold that no reading in this scene can satisfy is as dead as a derived one, and the machine is the only party in a position to notice — it has both numbers.

The warn is opt-in (daynight.diagnose_thresholds, default 0), for the same reason trace_path is: its condition persists until a human edits the config, so unconditionally it is one WARN per reconfirm interval forever — pure log growth on a fleet that does not need it, and far more than is needed to make the point on one that does. Corpus 13 carries the key and asserts the message; corpus 14 is the default-off twin and asserts the silence, so removing the gate fails exactly one scenario instead of none.

DN_DAY_THR_UNREACHABLE (1.5×) gates the warn so a dawn ramp stays quiet: a ramp's best reading sits just above the threshold and crosses it a probe or two later, while a mis-set threshold is missed by a wide factor (1.77× and 5.0× above). Corpus 09, 08 and 06 are the negative controls and emit it zero times; corpus 13 is the positive one.

Considered and rejected: refusing to plant a baseline that makes the hold unsatisfiable (baseline >= floor / (0.8 · 0.97) ≈ 1.29× floor). It is the more ambitious fix and it does subsume the arithmetic, but it is wrong here for a measured reason. night_baseline <= 0 is not a neutral state: the skip gate's can_judge requires a baseline, so refusing to plant makes solidly_night permanently false and every periodic reconfirm fire a physical probe. On the exact camera class this would target — cam-J and the cam-K closet, resting at 256–268 — that converts roughly zero clicks a night into one per night_reconfirm_s, i.e. it re-creates the complaint the skip gate was written for (2026-08-04, "das klacken der IR blende nervt"). Flooring the planted baseline instead of refusing it fares no better: it fabricates a measurement, and at the floor the bar is satisfiable only by a reading strictly below 256, which no sensor produces — still dead, now with a lie in the log. The honest position is that near the floor all three baseline-derived bars degenerate together, the hold is legitimately dead, and what the machine owes the operator is to say so — which is what the guard now does correctly.

Which is exactly what DN_RATCHET_MARGIN does: the ratchet stopped borrowing day_gain_pct (a pipeline discriminator, 0.74 stops, correct for the IR-cut transitions it was written for) and got its own constant sized for the question it actually asks — "is this new evidence, distinguishable from the evidence that already failed?" — at one quarter stop. Incidentally this also pushes the geometric recursion above further from the floor: the same 2026-08-14 latch now derives 265 rather than 189.

D. Two guards validating each other

Both instances have an identical structure: mechanism A derives its reference from a value that mechanism B is concurrently moving, so A and B confirm each other instead of either being the other's escape hatch.

  • 0f5fc80: the baseline drifted toward raw gain, and the brightening bar was derived from the baseline. Noise ratcheted both together → overnight flap.
  • 14a1d61: the baseline drifts toward smooth_tg, and the skip gate's "solidly night" bar is derived from the baseline — so over hours the bar chased the (day-level) gain down and "still deep in night" stayed true.
  • cam-H 2026-08-16: the same chase in the third consumer, the sustained-brightening hold, and the fastest-acting instance of it by a wide margin. The hold's bar is (100+pct)/200 of the baseline, so it sits 22.4 % away; DN_BASELINE_ALPHA closes 22.4 % in about 25 s, against a DN_BRIGHTEN_CONFIRM_MS of 30 s. The debounce loses that race by construction. Traced tick by tick on the corpus 10 replay: a 23 % light-on left the margin test 2.2 % short at the instant the step completed and falling from there, so it never opened on any tick — and 25 s later the bar had dropped through smooth_tg, taking the >= bar branch, which re-arms and clears the hold. A permanently brighter room therefore reads as "nothing has changed" from then on. Not merely slow: erased.

The fix in all three cases was the same idea, found three times independently: anchor the test on a value frozen at the moment of an actual measurement (probe_fail_smooth, or for the hold a snapshot of the baseline taken while the scene was still at rest) rather than one that drifts.

Rule: an "is there evidence of change?" test must be anchored on a frozen measurement, never on a value that any other mechanism updates.

Corollary (2026-08-16): where a bar is derived from a drifting value, the drift rate and the confirm window are not independent knobs. If the drift can close the bar's own gap faster than the confirm can elapse, the test is not "strict", it is unreachable — the same failure mode as an unsatisfiable bar (generator C), arriving through time rather than through range. State the ratio when either constant is touched.

The audit that item asked for is now two-thirds done: night_baseline fed the brightening bar, the skip gate and dn_day_trigger() and drifted under all three. The skip gate got its frozen anchor in 14a1d61, the hold got brighten_ref on 2026-08-16, and dn_day_trigger() is informational in the adaptive regime (night→day is probe-mediated since b4a54f0), so it is the one consumer where the drift is harmless. Note also where the hold's snapshot has to be taken: freezing it when the scene crosses the bar is far too late, because the bar is 22.4 % away and the baseline has been chasing for tens of seconds by then — measured, that late freeze captured 2353 where the true pre-event rest level was 2397, and 1.8 % of contamination was itself enough to keep the margin shut. It freezes at the 3 % DN_BRIGHTEN_MARGIN band instead: the reference stops tracking the moment the scene is measurably brighter, not once it is decisively brighter.

E. The evidence model is missing a dimension

Every test in the file is metric <op> threshold — a level, sampled at an instant. But the signal that distinguishes the two scenes the dead-zone conflates is the derivative:

  • a static dim scene (the one the unverified-day revert exists for) sits still;
  • a scene mid-dawn is in free fall through the dead-zone.

b4a54f0 is the same gap seen from the other side: night-pipeline gain has no usable level semantics at all, so the fix was to stop deciding on it. The 2026-08-14 fix (19dcd74) adds the file's first derivative-aware rule — revert an unverified day only once the improvement has stopped.

Two more places are still level-only and demonstrably shouldn't be:

  • probe_backoff. ~~Doubling the reconfirm interval in the middle of a monotone descent is exactly backwards.~~ Closed by the trend suspension in dn_next_probe() (dn_trend_falling()): while the gain sits below DN_BRIGHTEN_MARGIN of the frozen probe_fail_smooth, the multiplier is suspended and the deadline falls back to one plain night_reconfirm_s after arming. Note the reference point it needed was already in the file and already frozen — generator D's rule supplied it. On cam-L the backoff hit its ×4 cap at 05:58 while gain was falling through three orders of magnitude, and that cap is what turned a bad revert into a four-hour outage; corpus scenario 08 now measures the difference (recovery at 1276 s instead of 3976 s) and fails without it.

Closed properly on 2026-08-16, not before. As first written the suspension read the instantaneous smooth_tg, which means it was still a level test sampled at an instant — generator E's own defect, reintroduced inside generator E's own fix. It therefore un-fired as readily as it fired. On cam-H it suspended an ×4 backoff at 23:21:08 (gain 1599 against a 1653 anchor), pulling the reconfirm from 03:12 in to 00:12; the room then dimmed on its own, the predicate went false, and the four hours came straight back — long before the deadline it had pulled in. The evidence had been measured and then discarded. It now reads min_smooth_since_probe, the lowest smoothed gain since the anchor was frozen, so "the scene has been measurably brighter than confirmed night" is a fact about the interval rather than about this tick. The skip gate's ratchet-anchor override reads the same value for the same reason: a deadline is a single instant, so testing the instantaneous gain there throws away every brightening that happened between deadlines. - The brightening hold, whose ratchet asks "how far below the last failure" when at dawn the answer that matters is "still falling, and for how long". DN_RATCHET_MARGIN (2026-08-16) fixed the scale of that question, not its level-only-ness; the hold is still the one place in the file that asks it purely as a threshold.

Rule: any test that must distinguish "dim" from "getting brighter" needs a reference point in time, not just a threshold.

F. A property stated over one snapshot cannot see a defect that lives between snapshots

Worth naming separately, because it is a defect in the oracle, not in the machine, and it is the reason the 2026-08-16 bug survived a week of work aimed squarely at it. dn_next_probe() is pure over one dn_evidence, and both properties tests/dn-probe-props.c asserted were statements about a single evaluation: monotonicity compares two evidence values, the reconfirm bound reads one. The trend suspension satisfied both, on every one of ~10⁶ points, while still handing four hours back the moment a room dimmed again — because across a sequence of evaluations, brighter evidence had no durable effect. Purity bought counterfactual reachability, which is real and was the point; it did not by itself buy temporal claims.

The corpus caught nothing either, and for a related reason: the nine scenarios all replay monotone transitions (dawn ramps, boots, a single dip). None of them contained a brightening that came and then went, which is precisely the trajectory that distinguishes a latched predicate from a sampled one. Scenario 10 exists to be that trajectory.

Rule: for every piece of state the schedule carries between ticks, state a property over the SEQUENCE, not only over the snapshot — at minimum "evidence once measured is never worth less later". And when adding a scenario, ask what its gain curve does non-monotonically; a corpus of monotone traces cannot falsify a memory bug.

G. The corpus can only falsify what its noise model can express

Named the same day as F, for the same reason: the fix for F shipped with a bug its own tests could not see. min_smooth_since_probe was a plain running minimum of the smoothed gain — an order statistic, and the only quantity in this file with no noise rejection whatsoever. The expected depth of a running minimum grows without bound in the number of samples, so on a perfectly static scene it descends until it crosses the 0.97 anchor bar and suspends the backoff. Worse, the sample count is set by the reconfirm interval, which is the very thing the backoff lengthens: longer backoff → more samples → deeper spurious minimum → suspension fires → backoff defeated. Self-reinforcing, in the wrong direction. Measured on a static 3500 scene with realistic stochastic noise and a real 3600 s reconfirm: an audible probe every hour all night, 7 board switches against a budget of 4, emitting log lines structurally identical to the ones corpus 10 asserts as proof the fix works.

The header comment shipped with the mechanism correctly described and then dismissed it, on the strength of a green corpus. The corpus was green because noise_factor() was a bounded sum of two sinusoids. A deterministic periodic signal has a fixed, finite set of extremes, so any running minimum over it converges after one period and stops moving — the failure mode was structurally inexpressible, at any noise_pct, for any duration. Eleven scenarios reported zero click regressions while the backoff was inoperative.

Two things follow, and the second is the one worth keeping:

Rule: if a mechanism's behaviour depends on a statistic of a window of samples (a minimum, a maximum, a count of crossings, a time-to-first-event), deterministic test noise cannot falsify it. Use stochastic noise, and say in the scenario what statistic it is there to stress.

Rule: naming a risk in a comment is not bounding it. The comment that predicted this failure and waved it off was written in the same commit as the bug. If a risk is real enough to write down, it is real enough to need either a test that would fail on it or a stated reason no test can — and the second answer obliges you to fix the harness.

The corpus also had a hole that no amount of noise realism would have closed: every scenario asserted that changed evidence is eventually acted on, and not one asserted the converse. Scenario 12 is the null hypothesis — a static scene, stochastic noise, a real 3600 s reconfirm, and a click floor. It is the only scenario in the corpus that fails when the machine becomes too eager, which is the direction every fix in this record pushes.

assert_dip_monotone() is that property for min_smooth_since_probe: for two evidence values with identical current readings, the one that records a brighter past minimum must never be granted a later probe.

5. Verdict on a redesign

No rewrite. A targeted restructure, and a test harness first.

The switching machinery — settle, dwell, hysteresis, re-assert, the board hook chain — is correct and expensively earned; nothing in the incident record since 2026-08-02 touches it. The defects are all in probe scheduling and evidence modelling. A rewrite would relitigate twelve incidents' worth of hard-won detail with no test net underneath it, which is a reliable recipe for incidents 13 through 18.

What is actually missing is not a better structure. It is an executable oracle. You cannot safely refactor this file without one; with one, you can refactor it incrementally and cheaply.

The destination: dn_next_probe(evidence)

Collapsing verify, brighten_*, probe_backoff, probe_fail_smooth, last_phys_probe_ms, the skip gate and the anchor override into one pure function over a small typed evidence struct is the right end state. The win is not fewer lines — it is that a pure function is directly unit-testable and, more importantly, property-testable:

Monotonicity. For evidence e1, e2 identical except that e2 is strictly brighter: dn_next_probe(e2) <= dn_next_probe(e1). Brighter evidence must never buy a later correction.

This single property is violated by 0f5fc80, by 14a1d61, and by the 2026-08-14 incident — where an ambiguous (dead-zone, i.e. brighter) revert produced the same 4 h schedule as a confirmed-night revert. Three of the open incidents, one assertion. That is what makes the collapse worth doing.

But it is a bad first step, because there is currently no way to show the collapse is behaviour-preserving.

6. The replay harness

Step 0 — there is no trace to replay

Checked on cam-L 2026-08-14: no gain recorder exists anywhere on the camera, nothing but syslog lines. Every past "verified against timpsd-sim" was therefore a hand-reconstructed scenario, not a replay of recorded data — including the verification of the 2026-08-14 fix itself. That is the binding constraint, and it is why the harness idea has to start with a recorder rather than a replayer.

Sketch: a daynight.trace_path key appending one line per N samples — t_mono, cur, tg, luma, baseline, smooth_tg, day_trigger, probe_fail_smooth, verify_mode, verify_in_s, backoff. Roughly 30 lines of code. Write to tmpfs with a size cap, not to flash (these are camera-grade eMMC/NAND and a 5 s cadence is ~900 KB/day); persist or upload only on demand. The /events SSE stream already carries most of this and a LAN-side collector is the other viable shape.

Step 1 — the two-pipeline problem, stated honestly

A faithful replay needs the gain for both pipelines at each timestamp, because which one gets read depends on the decision the machine under test makes — and a patch that changes the decision immediately needs the counterfactual reading. In the field only one pipeline is observable at a time. True replay from field data alone is therefore impossible, and the harness should say so rather than pretend:

  • (a) Regression replay on synthetic two-pipeline traces — fully faithful, the workhorse. This is what the 2026-08-14 fix was verified against: a fake isp-m0 scrape file plus a switch_cmd stub that records the active mode, with the driver serving the matching pipeline's gain.
  • (b) Incident replay on a field trace — faithful only up to the first point where the machine's decision diverges from the recorded one, and it must then stop and report "diverged at T" rather than continue on invented data. Limited, but it answers the exact question every one of these twelve commits tried to answer by hand: would this patch have diverged from the recorded bad behaviour, and when?

Step 2 — time compression

A dawn incident spans hours. ms_now_us() is a single static inline in src/util.h — one chokepoint — so an opt-in MS_CLOCK_SCALE for sim builds is straightforward. The DN_* compile-time constants must scale with it, which needs a one-line SIM_CFLAGS hook on the sim Makefile target (the 2026-08-14 verification worked around this by compressing the config timings 15× and leaving the constants alone, which is adequate for a targeted check and not for a corpus).

Step 3 — assertions, not eyeballs

Each scenario declares expectations: mode at time T, maximum wrong-mode duration, maximum board switches (the click budget is a first-class requirement here, not a nicety), plus the two global properties from above — restart-equivalence and probe-time monotonicity.

Step 4 — the corpus

One scenario per historical incident. The numbers are all in the commit messages already, which is the one genuinely good side-effect of this repo's verbose-commit convention: 53c21b4 (boot transient 15000–20000), f8a7b21 (65 % of baseline), 0f5fc80 (post-revert elevated resample), bd21ce6 (boot at 731 in the dead-zone), a5dae07 (resting 256–268 vs threshold 300; baseline planted mid-descent at 10856/5148), b4a54f0 (10856 resting, dawn dip to 820, day pipeline 8192), 43c2b16 (booted after dark on a stale day), 14a1d61 (fail at 284, baseline chases to 257–266), and 2026-08-14 (9024 → 4813 → 2425 → 708, reverts at 1436 and 452, ratchet 189 vs floor 256).

7. What not to do

  • Do not add more escalation timers. A "dead-zone revert retry ladder" — shortening the reconfirm interval after an ambiguous revert and doubling it per consecutive one — was written and considered for the 2026-08-14 incident and rejected: it recovers from the wrong revert instead of preventing it, and it does so by re-introducing a bounded version of the pre-dawn probe volley that fad4f40 was written to remove — several audible IR-cut pairs through every dawn that lands in the dead-zone, on every camera. The fix that shipped prevents the revert and costs fewer clicks than the status quo. Prefer that direction: this subsystem already has nine gates whose only job is rationing clicks, and adding a tenth that spends them is moving backwards.
  • Do not rewrite before the harness exists.
  • Do not treat probe_max_skip_s as the answer to anything. It is a 12 h backstop. Twice now (14a1d61, 2026-08-14) an incident has been characterised as "well within probe_max_skip_s", which is true and irrelevant — a camera in the wrong mode for four hours is already a failure.

8. Suggested order

  1. ~~Derivative-aware unverified-day revert~~ — done, 19dcd74.
  2. ~~dn_bar_reachable() + a warn on an unsatisfiable bar.~~ — done, 5423b79. Generator C is self-reporting instead of silent.
  3. ~~Trace recorder (tmpfs, capped).~~ — done, c84fcef (daynight.trace_path, opt-in).
  4. ~~Replay harness + scenario corpus, with the restart-equivalence and monotonicity assertions.~~ — done, e06bf41 (virtual clock) + e0c629b (scripts/dn-replay.py, nine scenarios).
  5. ~~dn_next_probe() collapse, validated against the corpus.~~ — done, src/daynight_probe.h + tests/dn-probe-props.c.
  6. ~~Revisit probe_backoff's trend-blindness (generator E).~~ — done, dn_trend_falling(); see the generator-E section above.
  7. ~~Latch the trend suspension on a measurement, not a predicate (generators E + F).~~ — done, 2026-08-16: min_smooth_since_probe in dn_evidence, read by both dn_trend_falling() and the skip gate's ratchet-anchor override, plus assert_dip_monotone() as the sequence property that would have caught it. Corpus scenario 10.
  8. ~~Give the failure ratchet its own margin (generator C, extended).~~ — done, 2026-08-16: DN_RATCHET_MARGIN (one quarter stop) instead of borrowing day_gain_pct. Pinned by the cam-H B1 cases in the property test — both that a 21 % brightening can now start a hold, and that an identical re-dip still cannot.
  9. ~~Stop the brightening hold's bar being derived from the drifting baseline (generator D, third consumer).~~ — done, 2026-08-16: brighten_ref, released again by DN_HOLD_REF_LEAD. This is the one that makes the fast path actually fire for the everyday event — corpus 10 goes from 489 s to 35 s — and the one that most needed its second-order effect caught, since the first version of it walked fad4f40's probe volley down corpus 09's dawn ramp.
  10. ~~Hand the generator-C guard the value the gate uses (dn_hold_gate()).~~ — done, 2026-08-16 after the cam-D incident. One definition, called by the schedule and both guard call sites, because the arithmetic error was only possible while the guard re-derived the bar itself.
  11. ~~Make min_smooth_since_probe noise-safe (generator G).~~ — done, 2026-08-16: a reading must hold for DN_BRIGHTEN_CONFIRM_MS before it enters the minimum. A running minimum is an order statistic and had no business being the one unsmoothed quantity in the file.
  12. ~~Make the harness noise stochastic, and add a null hypothesis (generator G).~~ — done, 2026-08-16: noise_factor() is seeded gaussian + AR(1), and corpus 12 asserts a click floor on a static scene. These are the two changes that would have caught items 10 and 11 before they shipped, and they are the general fix rather than the two specific ones.

What the corpus is worth, and how to keep it worth that

The corpus is only evidence if a scenario fails against a build without the fix it is named for. Four of the twelve assert a decision rule rather than a new log line, and all four were checked that way (08 against the anchor override and against the trend suspension, 09 against 19dcd74's extension and — newly — as the negative control for DN_HOLD_REF_LEAD, 10 against both the latched suspension and the frozen hold reference, 11 as the sensitivity floor via forbid_log) — each fails on behavioural assertions, not merely on expect_log.

Scenario 09 earned a second job on 2026-08-16 and it is worth naming, because it is the only reason a fix that looked finished was not shipped broken: the first version of the frozen hold reference passed 10 and 11 and the whole property test, and 09 failed it instantly on the click budget — 14 board switches against 9, six sustained-brightening pairs walking down one dawn ramp. The mode assertions all still passed. A corpus that only checked "did it end up in the right mode" would have shipped it. max_switches is not a nicety. Scenario 10 is the sharpest of the three: pre-fix it never leaves night at all inside the run (wrong-mode 3100 s, mode@3500 and mode@5900 both wrong), and probe_max_skip_s is deliberately left at its 43200 default so the backstop cannot quietly rescue it.

Every scenario now pins total_gain_day_threshold and total_gain_night_threshold explicitly (at the historical 300/3000), rather than inheriting the compiled defaults. This is not tidiness: the corpus encodes incidents, and an incident happened under a specific configuration. While the scenarios inherited the defaults, raising a default silently re-ran all thirteen against a configuration none of them was calibrated for — so the defaults could not be changed without invalidating the evidence, and the evidence could not show whether the change was safe. Pinning decoupled the two, which is what made the 300 → 768 default bump measurable at all.

Rule: a scenario must pin every config value its incident depended on. A corpus that inherits defaults cannot be used to evaluate a change to those defaults — precisely when you most need it.

Scenario 10 also carries the generator-F lesson in its shape. Its gain curve is non-monotone on purpose — a brightening that comes and then goes, before the deadline it pulled in. Every one of the first nine replays a monotone transition, which is why none of them could falsify a schedule that forgets. When adding a scenario, ask what its curve does non-monotonically before asking what it asserts.

Build a negative control by reverting the fix's hunk in a copy of the current tree, never by building the historical commit. Trees older than e06bf41 have no MS_CLOCK_SCALE hook: they silently ignore SIM_CFLAGS, run in real time while the harness drives them 60× faster, reach none of their own deadlines, and come back green on everything — a false negative that reads exactly like "this scenario does not discriminate". dn-replay.py now refuses such a binary outright (SimRun.check_clock), but the cheaper habit is not to build one.

Next candidates, in the same spirit: the brightening hold's ratchet is the last level-only test generator E names — DN_RATCHET_MARGIN corrected its scale but it is still a threshold, not a trend — and night_baseline still feeds three consumers while drifting under all three (the generator-D audit). cam-H put a sharper edge on that last one: on the night in question the hold was blocked not by the ratchet but by the baseline-relative arming margin, because the baseline had already drifted down to chase the very brightening the hold was meant to notice. That is 14a1d61 again, in the one consumer the anchor override does not cover.

Also worth a pass now that generator F is named: audit the remaining between-tick state in dn_thread the same way. brighten_since_ms, day_verify_ref, brighten_ref and probe_backoff itself are each carried across ticks and each has only snapshot properties asserted about it.

And the generator-G pass that has not been done: min_smooth_since_probe was the only order statistic in the file and it is now debounced, but the audit question generalises — for each derived quantity, is it a smoothed value, a frozen measurement, or a statistic of a window? The first two are safe; the third needs stochastic noise pointed at it and a stated reason it is bounded. Right now the answer is "none remaining", and the value of writing that down is that the next one added has to justify itself.

One more sensitivity fact worth recording rather than discovering again: the debounce on the minimum means a dip must be deeper than the scene's own noise band to latch. The real cam-H 23:21 dip (1599 against a 1653 anchor, 3.3%) would not latch under realistic AGC noise. That is the correct trade — the alternative is no working backoff on any camera — and the periodic reconfirm still bounds recovery at one night_reconfirm_s. But it means corpus 10's dip is deliberately deeper than the incident's, and anyone reading that scenario as "this is what the camera saw" will be misled about the sensitivity.

And one specific piece of unfinished business from the third fix: the hold now distinguishes a step from a ramp by how far its frozen reference has led the baseline, which is a proxy for the derivative. The direct test — "is the gain still falling?" — is the open generator-E item two bullets up, and it would replace DN_HOLD_REF_LEAD with the thing that constant is standing in for. Do that when the hold is next opened, not before: the proxy is measured, bounded on both sides and has a negative control, which is more than the subsystem's last four constants started with.

v1.12.0 dawn

The 1.10.1 engine wasted the dawn: the fleet logs of 125 mornings show 1276 morning looks and 238 failed morning checks (click pairs), about 10 looks and 1.9 failed checks per camera and morning. The first failed check per dawn measures k (Kf); the rest were spent not using it. Changes in src/daynight.c (a7721cb):

  • Skip a look that cannot find day: without the illuminator the scene reads no brighter, so a jump, colour or near look with lit × 0.9 ≥ bar is not run; the lit level becomes the reference. Timer and requested looks, and a look that only learns the bar, always run. 640 of the 1276 morning looks sat above the bar.
  • Creep or lamp by a 10 s step (more than 15 % between two 10 s samples is a lamp), replacing the 15 %-over-a-minute ramp test that dawn jumps failed. A creeping look, also a colour look, gets the jump bar and day_gain / Kf.
  • Near-bar look: last dark reading × lit / reference under 0.95 × bar starts a look without a jump_pct step (cam-wohn 2026-11-07: a 0.5 % miss at 08:21, then night until 09:21 although the scene would have read day from 08:25).
  • No lamp as the reference after a check: a settled level jump_pct brighter than the lit level before the check keeps the earlier reference.
  • Two one-line safety fixes: a railed check does not set Kf, and a check under 1.5 × day_gain waits the 10 s hold (the AE still settling).

Measured: replay corpus 77 pairs, 0 failed (six new families and profiles from the measured 2026-11-07 dawns of cam-wohn and cam-schuppen); a fleet what-if over 121 mornings: failed morning checks 178 → 94, day on average 5 min later, about half of the morning looks skipped.

Deliberately not done:

  • The K cap stays 2. It protects lamps after nights under foreign IR; a second "creep K" would help only the first check after a restart, and k drifts 20-40 % within one morning anyway.
  • Kf stays 1 h. Six expiries in 125 mornings against outlier Kf values that would block for the rest of the morning; 3 h or the whole morning bought 3 failed checks.
  • No persisted or per-camera k. It varies 1.5-2× per camera across mornings, and a stale value is a new stuck mode that only a restart clears.
  • No prior k before the first check. −84 % failed checks on paper, +1 h of night in the low-k rooms, where one lamp decides day.

Report, with the field numbers, traces and the rejected variants: dev_notes/DAWN_FABLE_REPORT_2026-11-07.md.