Elder Motor — Safety Interlocks and Error Recovery

Document 4 of 5 in the Elder Motor Build Specification
Date: 2026-04-19
Author: Elder, Lobe 6 Structure
Prerequisite: Documents 1 (Coil), 2 (Bridge), 3 (Black-Start)

1. Purpose

Define the safety interlock tree, fault detection, and recovery protocols. This is the "what goes wrong and what saves us" document. Ultracapacitors at 225-1000 V with 5-100 kJ stored energy are dangerous — a fault that dumps the bank through an unintended path can be explosive. Hardware-level protection is primary; software is secondary.

2. Layered defense principle

  Layer 1: HARDWARE (physical, does not require code to work)
     - Fuses, crowbars, mechanical E-stop
     - Must fail safe even if computer is completely dead

  Layer 2: GATE-DRIVER FIRMWARE (FPGA or gate-driver IC)
     - Desaturation detection, DC-bus OVP, immediate switch-off
     - Operates in <1 µs, no round-trip to host computer

  Layer 3: HOST SOFTWARE (Rust φΩ v3)
     - Higher-level state machine faults, load mis-match, thermal
     - Operates at 100 µs cadence
     - Commands Layer 2 to safe state on fault detection

  Layer 4: OPERATOR / SUPERVISORY (human + ForgePipe telemetry)
     - Reviews telemetry, spots slow-drift anomalies
     - Can command manual abort

A fault in any layer MUST trigger safe state in all lower layers within their response time.

3. Hardware-level interlocks (Layer 1)

3.1 DC bus fuse

• Type: fast-acting semiconductor fuse (e.g., Bussmann 170M-series)
• Rating: 125% of peak continuous current (75 A for 5 kW, 150 A for 40 kW)
• Location: between ultracap bank positive terminal and DC bus
• Purpose: last-resort protection against short-circuit; blows in <10 ms at 2× rated
• Replaceable: yes, after fault diagnosis

3.2 Crowbar circuit

• Topology: SCR + series fuse across DC bus
• Trigger: bus voltage > 110% nominal for > 100 µs (hardware comparator, no software)
• Action: SCR conducts, shorts the bus through the fuse, fuse blows, ultracaps dump through the short path
• Result: bus goes to zero instantly, all downstream protected from overvoltage
• Recovery: replace SCR + fuse after fault diagnosis

3.3 Mechanical E-stop

• Button: 22 mm mushroom-head red pushbutton, twist-to-release latching
• Location: on device front panel, visible and reachable without leaning in
• Circuit: normally-closed contact, breaks the FPGA safe-enable line
• Effect: FPGA interprets broken enable as "all switches off immediately" (gate drives pulled low)
• Hardware-only: does NOT depend on computer, does NOT depend on firmware beyond the basic enable line
• Reset: requires manual twist-release + physical power-cycle-and-armed sequence

3.4 Thermal fuse (irreversible)

• Type: metal alloy thermal link, opens permanently at 120°C
• Location: embedded in the coil winding potting compound, near hottest expected spot
• Purpose: last-resort thermal runaway protection; if this blows, coil is compromised and unit must be rebuilt
• Rating: 120°C (well above design max 85°C)

3.5 Insulation monitor

• Device: IMD (insulation monitoring device, e.g., Bender IRDH265)
• Function: continuously measures isolation resistance between DC bus and chassis ground
• Fault: if R_iso drops below 100 kΩ (IT-system threshold), alarm and trip contactor
• Purpose: detects degrading insulation before catastrophic arc-over

4. FPGA / gate-driver interlocks (Layer 2)

4.1 Desaturation detection (DESAT)

• Per-MOSFET: gate driver monitors V_DS during on-state
• If V_DS exceeds a threshold (typically 9 V for SiC vs. normal ~0.5 V) for > 2 µs, DESAT fault declared
• Action: immediately turn off the affected switch (soft turn-off to avoid avalanche), signal host
• Cause: short circuit, over-current, or component failure
• Recovery: latch fault, require manual clear

4.2 Bus OVP at gate-driver level

• Hardware comparator on the DC bus vs. reference (typically 105% of nominal)
• If exceeded, assert "switches off" signal directly to gate drivers
• Bypass computer entirely — 500 ns response time
• Relaxed (but not removed) if bus returns to nominal

4.3 Gate driver under-voltage lockout (UVLO)

• If the +15 V / -5 V gate supply rails drop below threshold (typically +12 V / +4 V)
• Gate driver itself refuses to turn switches on until rails recover
• Prevents half-turned-on MOSFET linear-region operation (which is destructive)

4.4 Watchdog timer

• FPGA maintains a counter; host must reset it every 500 µs
• If counter times out, FPGA enters safe state (all switches off)
• Purpose: detects host computer hang, crash, or communication failure
• No reset from host possible during watchdog timeout — manual clear only

5. Host software interlocks (Layer 3)

5.1 Over-current per segment

• Sample each segment current sensor at 100 µs cadence
• If any segment > 150% of commanded amplitude for > 10 µs: FAULT
• Action: command FPGA to safe state, log event, enter FAULT state

5.2 Under-voltage on bus during operation

• Bus should hold within ±1% during sustained operation
• If bus drops > 5% unexpectedly during SUSTAIN: FAULT ("self-sustain broken")
• Cause: coil coupling failure, load exceeds capacity, ultracap bank degradation
• Action: wind down gracefully if possible, else immediate safe state

5.3 Phase imbalance

• 6 doubling-circuit segments should carry equal-amplitude currents within ±10%
• If imbalance > 20% detected, FAULT ("coil asymmetry")
• Cause: open winding, shorted turn, gate driver fault, cap bank degradation on one phase

5.4 Throat-3 coupling anomaly

• Throat current should be 25-35% of segment amplitude
• If < 15% or > 50%, FAULT ("hemisphere coupling broken")
• Cause: throat winding damage, hemisphere imbalance, asymmetric heating

5.5 Thermal drift

• Per-segment temperature sensor sampled at 10 Hz
• Limits:
• Normal: < 60°C
• Warning: 60-75°C (host logs, host reduces firing cadence 20%)
• Fault: > 85°C (host commands safe state immediately)
• Prevents winding insulation breakdown (usually rated 180°C but with margin)

5.6 Resonance loss

• Phi-lock must maintain hemisphere-return ≈ hemisphere-out within ±5% amplitude
• If deviation > 10% for > 100 ms, FAULT ("phi-lock lost")
• Recovery: return to TUNING state, attempt phi-lock re-acquisition

5.7 Chain-code integrity check

• Host computes hash of loaded Rust φΩ v3 binary at boot
• Compares to known-good hash baseline (stamped to chain at deployment time)
• Mismatch: REFUSE TO RUN. Device locked until manual intervention.
• Prevents: firmware tampering, bit-rot in flash storage, unauthorized firmware updates

5.8 Operator input fault

• Remote commands (via secure channel to computer) require per-observer signature (φΩ v3 SIGN opcode)
• Unsigned or invalid-signature commands: REJECT, log attempt
• Prevents: unauthorized remote operation

6. FAULT state behavior

6.1 Immediate actions upon FAULT (any source)

1. FPGA commands all 14 gate drivers to low state (switches off)
2. Host sets state = FAULT, latches
3. Ultracaps retain their charge (they are the static energy reservoir — they don't self-discharge on fault)
4. Computer continues to run, serving telemetry + diagnostic endpoints
5. Load output relay opens (disconnects load from bus)
6. Front panel indicator: red FAULT LED + alarm tone

6.2 Telemetry on fault

Record to local ring buffer + ForgePipe message:

{
  "ts": 1776625000.000,
  "event": "FAULT",
  "prior_state": "SUSTAIN",
  "cause": "segment_4_overcurrent",
  "sensor_snapshot": {
    "bus_v": 221.3,
    "segment_currents_a": [58.2, 57.9, 58.1, 148.3, 57.7, 57.8],
    "throat_a": 17.1,
    "temperatures_c": [28.3, 28.5, 28.4, 42.1, 29.1, 28.9]
  },
  "last_5_firing_cycles": [...],
  "hardware_faults_active": ["none"]
}

6.3 Recovery procedure

1. Human reviews FAULT telemetry
2. If obvious physical damage (burned smell, visible char): DO NOT RESET. Unit back to shop.
3. If transient (brief over-current, thermal blip, operator error): clear fault via front-panel button + ARM sequence
4. If persistent (fault re-fires on re-ARM): back to shop for diagnosis

No auto-reset. Every fault requires human review. Once.

7. Failure mode analysis

7.1 MOSFET short-circuit (source-to-drain)

• Detection: DESAT on that switch, over-current on that segment
• Immediate result: FPGA turns off that switch (via gate pull-low)
• Consequence: that bridge leg is dead, but other segments still operational
• Recovery: replace MOSFET; coil and other electronics usually survive if detected fast

7.2 MOSFET open-circuit

• Detection: segment current fails to rise when commanded
• Immediate result: host sees phase imbalance, declares FAULT
• Consequence: that bridge leg non-functional
• Recovery: replace MOSFET

7.3 Coil winding short (shorted turn)

• Detection: reduced inductance on that segment, elevated current draw, rapid heating
• Immediate result: thermal fault or over-current, FAULT triggered
• Consequence: coil compromised
• Recovery: rewind affected segment (or whole coil for safety)

7.4 Coil winding open (broken wire)

• Detection: infinite impedance on that segment, zero current despite command
• Immediate result: phase imbalance detected, FAULT
• Recovery: rewind

7.5 Ultracap cell failure

• Detection: bank voltage drops unevenly, individual cell voltage monitoring (optional but recommended)
• Immediate result: bus under-voltage during load
• Recovery: replace affected cell(s); balance remaining bank

7.6 Gate driver failure

• Detection: UVLO from driver IC, or switch fails to respond to command
• Immediate result: FPGA sees unresponsive switch, FAULT
• Recovery: replace driver IC (typically DIP-socketed for easy replacement in prototypes)

7.7 Computer hang / crash

• Detection: FPGA watchdog timeout in <500 µs
• Immediate result: FPGA enters safe state, all switches off
• Recovery: reboot computer; if persistent, diagnose OS/software/hardware

7.8 Resonance lost mid-operation

• Detection: phase asymmetry, throat coupling drift, sustained bus sag
• Immediate result: host reverts SUSTAIN → TUNING, attempts re-lock
• Recovery: if re-lock fails within 10 seconds, FAULT and wind down

7.9 Load short-circuit

• Detection: output current spike well beyond commanded load
• Immediate result: output relay opens, FAULT
• Recovery: diagnose external load, clear fault, restart

7.10 Lightning / grid surge (while charger connected)

• Detection: crowbar triggers on DC bus overvoltage
• Immediate result: bus fuse blows, bank dumps through fuse + SCR
• Consequence: unit needs fuse + SCR replaced, possibly charger input damage
• Recovery: inspect, replace fuses, test

8. Cap bank handling (the most dangerous part)

Ultracaps at 225-1000 V with 5-100 kJ stored: lethal energy levels.

8.1 Discharge before service

• Hardware discharge resistor (ceramic wire-wound, 50 Ω, 200 W) connected via contactor
• Discharge from nominal to < 5 V takes 5-30 seconds
• Monitor bus voltage before opening any enclosure; require bus < 5 V AND mechanical indicator latched (capacitor-discharged LED)
• Multiple grounding staff available for manual shorting confirmation

8.2 Enclosure interlock

• Service enclosure door has interlock switch
• Opening door triggers: crowbar (immediate discharge), contactor open (disconnect from charger), output relay open
• Cannot operate device with service door open

8.3 HV warning labeling

• Appropriate IEC 60417 warning labels on all enclosures
• "DANGER: LETHAL VOLTAGE INSIDE — ENERGIZED EVEN WHEN POWERED DOWN"
• Symbol-based (language-agnostic)

8.4 Training requirement

• Anyone servicing the device must have reviewed this document
• Understand: ultracaps don't "turn off" — they hold charge until discharged
• A powered-off device with charged caps is MORE DANGEROUS than a running device (no protection circuits active)

9. Redundancy and fail-safe architecture

9.1 Critical signal paths are redundant

• DC bus voltage: measured by 2 independent ADCs through separate isolation amps (disagreement > 5% = FAULT)
• E-stop: wired in parallel with crowbar trigger signal (one button, two effects)
• Watchdog: FPGA watchdog + host-side dead-man's switch + hardware 555-based timer

9.2 Fail-safe defaults

• Default gate state: OFF (low)
• Default contactor state: OPEN
• Default output relay: OPEN
• Any uncommanded state → OFF/OPEN

9.3 Power loss behavior

• If AC/external power lost: charger disconnects (contactor opens), device continues running on cap bank (in SUSTAIN) or winds down (if in TUNING)
• If internal DC rail lost (computer dead): FPGA watchdog triggers → safe state within 500 µs
• If FPGA lost (rare): hardware crowbar + fuses handle catastrophic scenarios

10. FAULT state diagnostic tree

When FAULT fires, human operator walks this tree:

Is the enclosure hot to the touch?
  YES → evacuate, cool, do not open for 30 min minimum
  NO → continue

Is there a burned smell?
  YES → do not re-ARM. Unit to shop for full diagnosis.
  NO → continue

What was the FAULT cause reported?
  - Over-current → check coil resistance, look for shorted turn
  - Over-voltage → check crowbar, load disconnect, charger regulation
  - Under-voltage → check cap bank cell voltages, phase imbalance
  - Thermal → check ventilation, airflow, sustained load pattern
  - Phase imbalance → check individual segment resistance/inductance
  - Throat coupling → check throat-3 winding integrity
  - Resonance lost → check for physical vibration source, mechanical damage
  - Watchdog → check computer logs for crash cause
  - DESAT → specific switch failed, replace

Is the issue resolvable without opening the enclosure?
  YES (software/config) → clear FAULT, re-ARM, monitor
  NO → follow cap bank handling (Section 8) before opening

11. Periodic maintenance

11.1 Weekly (for production-deployed units)

• Review telemetry logs for slow-drift anomalies (temperature trend, cap ESR growth)
• Check E-stop button function
• Verify charger contactor operation

11.2 Monthly

• Insulation resistance test (> 100 MΩ)
• Cap bank balance check (individual cell voltages)
• Cooling system check (if present)

11.3 Annual

• Full DC resistance + inductance survey of coil segments
• Thermal imaging under load (identify hot spots)
• Firmware integrity check (hash verify)
• Fuse integrity (visual + continuity)

11.4 Event-triggered

• After any FAULT: full diagnosis before return to service
• After any grid event (lightning, brownout): insulation + component check

12. Prototype vs. production differences

ONE-ZERO prototype at The Outpost: all safety features enabled, Layer 1-4 active. No compromises; the first unit is also the test unit and must fail gracefully to inform future design.

Consumer production unit: same Layer 1 hardware, same Layer 2 FPGA, Layer 3 hardened (no developer shell access), Layer 4 simplified (preset alerts, no raw telemetry access by operator — prevents tampering with safety margins).

13. Shutdown-on-command behavior

Normal shutdown (operator command):

1. Host reduces firing cadence linearly to zero over 10 seconds
2. Ring-down energy dumped to resistive load bank (not the primary output load)
3. At zero firing: state = STANDBY
4. Caps remain charged (unless explicit discharge commanded)
5. Computer may power down or remain running for monitoring

Emergency shutdown (FAULT):

1. All switches off instantly
2. Output relay opens
3. Charger contactor opens
4. State = FAULT, caps stay charged
5. Alarm + logged

Full power-down (maintenance):

1. Normal shutdown first
2. Operator commands discharge
3. Discharge resistor contactor closes
4. Caps bleed down to < 5V over 30 seconds
5. Ground stick applied to confirm (manual)
6. Enclosure can be opened

14. Certification path (production)

For consumer sale:

• UL 1741 SA (smart inverter) — applies to grid-tie output
• UL 9540 (energy storage systems) — applies to ultracap bank
• IEC 62477-1 (safety of power electronic converter systems)
• FCC Part 15 Subpart B (EMI)
• CE marking (LVD + EMC + machinery if motion included)
• CSA for Canadian market
• Individual state certifications for grid interconnect

Not required for ONE-ZERO prototype or Outpost-internal use.

15. The prime directive

No matter what else goes wrong, no matter what firmware state or software state, the user must never be hurt by this device. Hardware Layer 1 is designed to be sufficient for safety even if all software is absent.

If Layer 1 fails to protect, that's a design defect requiring redesign, not a software patch.

Signed: Elder, Lobe 6 Structure
Authority: Node Zero — building at The Outpost
NODEZEROINSIDE. ∞.