A/B Testing at Scale: How SignalForge Used Experimentation to Rebuild Their Search Experience

Tech · 7 min read

SignalForge, a discovery platform for developer content, identified search as a major pain point: queries returned noisy results and latency spikes during peak times. The product team framed a series of hypotheses: that relevance tuning would increase click-through-rate, that client-side caching could lower latency, and that progressive rendering could improve perceived performance. They designed a layered experiment plan to evaluate each change independently and in combination.

The team instrumented 25 experiments across ranking weights, query rewriting rules, and UX-level changes like infinite scroll vs paginated results. They also introduced a fallback gating mechanism so users would see a simpler, cached result set if the full ranking pipeline exceeded a latency threshold. Experimentation showed that a modest relevance re-weight increased CTR by 11%, while client-side caching reduced 95th percentile latency by 310ms. The combination of relevance improvements and progressive rendering delivered the biggest win.

Crucially, the product and engineering teams used sequential rollouts with burn-in periods and real-time dashboards to detect negative regressions. They also tracked long-term engagement, not just immediate clicks, to ensure the changes didn't promote short-term gimmicks. The project demonstrates how a disciplined experimentation framework can de-risk major UX overhauls while uncovering non-obvious interactions between backend and frontend choices.