On-device LLM or Cloud: How SignalWrite Chose Offline AI for Its Writing Assistant

AI · 6 min read

When SignalWrite began prototyping an AI writing assistant, the product team faced a classic trade-off: cloud-hosted LLMs with larger models and frequent updates versus a smaller on-device model offering lower latency and stronger privacy guarantees. Initial user interviews consistently surfaced two non-negotiables for the core user persona: instant suggestions during composition and no server-side retention of drafts. Those constraints pushed designers and engineers to test an on-device first approach.

The team evaluated three compression strategies—distillation, quantization, and retrieval-augmented light models—while keeping an eye on 200MB download caps for first-run installs. Designers adjusted UI expectations: suggestions needed to be more conservative, inline rather than generative, and offer a clear “explain suggestion” affordance to address occasional hallucinations from the compressed model. Product prioritized transparency microcopy and single-tap rollback controls.

In a 6-week beta with 12,000 DAUs, latency dropped by ~60% for suggestion generation and privacy-related NPS improved by 8 points. Cost per MAU fell by roughly 35% since fewer requests hit the cloud, allowing a hybrid rollout where heavy generative tasks gracefully escalate to servers when users opt-in. The decision required upfront engineering and UX trade-offs but aligned with the startup’s core promises—fast, private writing assistance that felt native on-device.