17 May 2026 · DevTools Connect

Sampling gaps that hide CI template abandonment | Field Notes

CI usage analytics often sample successful jobs more generously than failed or abandoned ones. That skew changes what platform teams decide to retire.

Laptop open on a wooden desk with coffee nearby

A platform team in our instrumentation reviews discovered that failed pipeline starts were throttled more aggressively than green builds. Leadership saw a healthy adoption curve for a shared template that engineers privately described as “the one we fork and gut.”

The fix was unglamorous: equalize sampling for start, cancel, and success events; tag template lineage; and stop counting wrapper jobs as first-class template use. After two sprints, the abandonment story matched the interviews.

If you govern application analytics for developer tool usage inside engineering platforms, ask your maintainers which failure paths are under-sampled. Quiet drops in cancel events are often where consolidation opportunities hide.