Half of Four Looks Like Weather
String C was putting out 0.3 volts. String D was at negative 0.4. It was twenty past two on a clear afternoon, and the other two strings were making 4,390 watts between them, which is a fine amount of power for a house to be making. That is the entire reason this went unnoticed.
Four strings, two alive
A was making 2,870 watts at 298 volts. B was making 1,534 at 162. C and D were reading noise: a few tenths of a volt, 0.16 amps on one of them, nothing at all on the other. Both flagged disconnected.
Two dead strings out of four does not produce an error. It produces less. And less is the one thing a solar array is expected to produce constantly, for reasons that have nothing to do with anything being broken: cloud, haze, sun angle, heat derating, the time of day, the month. There is no reading that means “you are making exactly what you should be making right now”, because what I should be making changes every minute and depends on the sky.
So the only failure I could ever have caught by looking is total failure. Zero at midday is unmistakable. Half at midday is a Tuesday.
That part travels. Anything built out of several identical units degrades into a plausible number when one of them dies. Two servers behind a load balancer, four disks in an array, three workers on a queue: lose one and the thing does not stop, it gets quieter, and quieter is a value that already occurs naturally. Redundancy converts a hard failure into a soft one, and a soft failure into the weather.
It already had a name
I opened the gateway’s local API expecting raw numbers I would have to interpret myself: voltages, currents, readings that mean nothing until you build a model around them.
The strings endpoint gave me exactly that. The alerts list gave me something else. Five entries, and two of them named the dead strings directly, one ending MciStringC and the other MciStringD. One identifier per string, spelled out, sitting in the response.
Somebody had anticipated this failure closely enough to write a constant for it, one per string, before it ever happened to me. Detection was never the problem. It was in that response, correct and specific, every time anything asked.
The same list says the grid is connected
Here is another entry from that array, alongside the two dead strings:
SystemConnectedToGrid
That is not a fault. That is the ordinary, desired, everyday condition of a house with a battery in it: the grid is there, everything is fine. It sits in the same list, formatted identically, next to the entry that means half my panels are gone. There is no severity on any of them. No level, no category, no timestamp. Five entries in a sorted array, carrying a fan-speed complaint, two dead strings, a contactor state, and a piece of good news.
Nothing failed in order to make that unreadable. The channel has no way to say that one of its entries matters more than the others, so anything downstream that wants to show me a fault has to already know which of those five are faults. The knowledge lives in the reader, never in the message. My phone shows a house, a sun, a battery and arrows between them. Every one of those was accurate.
The line I did not expect to write
The fix is a poller. Every half hour while the sun is up it reads per-string connection state and the alert list, compares both against the last state it saw, and raises an event when either changes. A string that was connected and now is not: error. An alert that appeared: error. An alert that cleared: say so.
Comparing against the last state requires a last state. On the first run there isn’t one, so the first run records what it finds and calls that the baseline.
Which is where this nearly went wrong a second time. Written the obvious way, the first thing my new monitor would have done is write down “C and D disconnected” as the normal condition of my house, and then, having no change to report, say nothing more for as long as it ran. It would have been correct. It would also have been silent forever about the exact failure it was built for, because the failure arrived before the instrument did.
So the seeding branch is special-cased. If the sun is up and any string is down at the moment of first contact, the seed event goes out at error rather than ok. Nine lines. Here is the first sentence that program ever spoke:
monitor online: strings C,D disconnected; solar 4.4 kW; alerts: …
Every other event it will ever emit describes a change. That one describes a starting condition, and it is the only part of the program that exists because the fault got there first.
The gateway keeps no history I can query. Whatever it knows about last week lives in the manufacturer’s cloud, not in the box, so the honest answer to how long C and D have been down is that I do not have one and never will. My record of this array starts at twenty past two on the afternoon I found it, with string D at negative 0.4 volts and the battery already full, exporting the surplus of half an array to the grid under a cloudless sky.