sluice

Your retry loop's blind spot is its own reconnect — and the errors there arrive with their causes stripped

Three multi-day soaks against real managed infrastructure each died on an error the retry machinery was built for but couldn't recognize. The machinery existed and worked; the gap was classification coverage, twice over: a gRPC transport drop reported under the one status code a careful policy refuses to blanket-retry, and a reconnect failure inside the retry loop itself, where the driver flattens 'the peer dropped your pooled connection' into bare text with the structured cause gone.

Landed in sluice · 2026-07-23 · Cross-cutting

Observed — sluice's multi-day soak fleet (PlanetScale MySQL, PlanetScale Postgres, Cloudflare D1), 2026-07. Every death was LOUD — clean exit, named error, durable resume position, zero data loss; each restart warm-resumed and drained its backlog in minutes. Fixed across sluice v0.99.286 and v0.99.288.

Three deaths, one class #

The soaks died hours apart on routine transients: a VStream read returning rpc error: code = Internal desc = server closed the stream without sending trailers after ~17 healthy hours; a D1 poll hitting net/http: TLS handshake timeout and, separately, a plain HTTP 500; and — the sharpest one — a ~30-second network blip where the stream error was correctly classified, the retry engaged… and the reopen died terminal at open target change applier: mysql: ping: invalid connection. In every case the bounded-retry machinery already existed. It just couldn't see these errors: the classifier is interface-driven, and these shapes arrived without the wrapper.

The two coverage gaps #

From the stream: grpc-go reports a routine long-lived stream drop under codes.Internal — exactly the status a careful retry policy refuses to blanket-retry, because a genuine server-authored Internal is a fault that must stay loud. The fix has to discriminate the transport-authored wordings (server-closed-without-trailers, unexpected EOF, RST_STREAM) from a server-authored Internal by message text, because the code alone cannot tell you.

From the reconnect: every retry attempt first re-establishes its connections — and a failure there is a site most classifiers never covered, in a wire format that has lost its cause. go-sql-driver reduces “the peer dropped your pooled connection” to the bare text invalid connection; pgx v5 flattens multi-host connect errors so even errors.Is(err, syscall.ECONNREFUSED) misses a refused dial. At the driver boundary you often have nothing but text. And the class keeps paying out: the very next post-release regression cycle caught two more sibling wordings (the Windows connectex: … actively refused dial text, and pgx's conn closed from a severed pool connection — the latter now honoured via pgconn's own SafeToRetry contract, whose definition is exactly the property a retry classifier wants: the error is guaranteed to predate any byte reaching the server), shipped a release later in v0.99.289 along with the SQLSTATE-level transients (57P01–57P03, class 08) the trigger-CDC poll's transport classifier had deferred.

Update — the ladder kept extending, twice. A third instance arrived within a day of the second (Bug 200, fixed v0.99.290): the retry stack's own APPLY path dials too. When a target restart severs the pool, the next apply's pool acquire (Postgres) or begin tx (MySQL) dials fresh into the refused window — and neither applier classifier had a dial-shape leg, so a mid-outage CDC apply exited terminally with zero retries, making the whole v0.99.288/289 connect-retry stack unreachable whenever writes were pending during the outage (the realistic case; the focus checks had passed only because they stopped the target with no writes in flight). Three instances on a ladder — the stream reopen, the connect phase, the apply path's own dial — give the thesis its short form: every seam that can dial is a classification site. Then a fourth instance (v0.99.291) proved the corollary. The mechanism this time was not a missing site but maintenance drift: the transient-shape vocabulary lived in four hand-mirrored lists (trigger-CDC poll, pipeline connect phase, both engine appliers' text legs), and within ONE release of Bug 199 they had already diverged — the Windows dial wordings never reached the trigger-CDC list, so a postgres-trigger-source sync on Windows still exited terminally on a routine managed-PG restart, the exact class fixed twice one file over. The fix collapsed all four sites onto one single-homed matcher (internal/nettransient) with per-site corpus-parity change-detectors, so a one-sided addition or a site that stops delegating fails CI. Corollary: if the sites can't share one classifier they WILL drift — single-home the list, and parity-gate every consumer against it.

Update — a fifth instance (Bug 203, fixed v0.99.292) compounded that corollary. Single-homing the text corpus is not enough when a whole evidence class is missing from one site: the connect-phase classifier delegated to the shared corpus — network shapes only, which deliberately excludes PG server-lifecycle wordings — and had no structured SQLSTATE leg at all, so a re-establish attempt that landed in a restarting Postgres's FATAL: the database system is starting up (SQLSTATE 57P03) window exited terminal, while the applier and trigger-CDC poll classifiers both classified the identical SQLSTATE retriable. (The window is a second or two on a local container but 10–60 s on managed-PG restarts — exactly the real-world restart shape.) The fix single-homes the SQLSTATE set the same way as the text corpus: nettransient.IsConnectionAvailabilitySQLState (57P01–57P03 + class 08, matched structurally via the driver error's SQLState(), never message text), with the postgres-side predicate delegating to it and a delegation-parity pin that fails CI if either home drifts one-sided. And the MySQL sibling was ground-truthed rather than assumed — a restarting MySQL presents no structured handshake errno; the whole startup window surfaces as transport shapes the corpus already classifies — so no MySQL leg exists. Evidence, not symmetry, decides what each classification site needs.

What sluice does about it #

Classify positively and narrowly, at the site that still holds the signal. Structured checks first (sentinels, net.Error timeouts, syscall errnos, pgconn's SafeToRetry contract), then a pinned text-fallback list for the shapes that reach you as prose — and the list is a change-detector test, so widening the retry surface fails a pin instead of slipping in. Everything unmatched stays terminal: wrong DSN, bad credentials, unknown host, and every unknown shape. The bounded budget is the loud-failure floor either way — a target that never comes back exhausts it and fails with the cause named.

The transferable lesson #

A retry loop is only as good as its classifier's coverage, and the coverage has more sites than the happy read path: the reconnect inside the retry loop is itself a failure site. Audit where errors can arrive, not just where they usually do — and expect the driver boundary to hand you text with the structure stripped, so classify by positive match with a pinned shape set and let the budget, not the classifier, be your guarantee against looping forever.

Primary sources #