# Nightjar hosted backend - infrastructure plan Plan for the infrastructure that runs the **hosted** Nightjar service (the commercial plans). The open-source project itself does not require any of this; self-hosters run their own. This is for 12 Hobbies Studio to operate the managed service. Status: **plan only, not provisioned** (decided 2026-07). Stand it up at first beta / paying users. See `docs/hosted-service.md` for the plan tiers this maps to. ## Two backends, matching the two plan families | Plan family | AI runs on | Infra | |-------------|-----------|-------| | **Cloud Starter / Pro** | commercial API (Groq / OpenAI / Anthropic) | small orchestration node; heavy compute offloaded | | **Private** | self-hosted open models (faster-whisper + local LLM) | a GPU node; audio never leaves our hardware | Cloud tiers are cheap to run (compute is per-minute on a commercial provider, keeping the GBP 6 / 18 pricing profitable). The Private tier needs real hardware but is the privacy / sovereignty story. ## Primary option: self-host on the MS-02 cluster (best for small scale) Hardware on hand: **3x Minisforum MS-02-class mini-workstations**, one fitted with an **NVIDIA RTX 3050 6GB**. High-core CPUs, dual 10GbE, NVMe, PCIe GPU slot. This is a strong small-scale backend and a better fit for the Private tier than rented cloud. ### Node roles - **Node A (RTX 3050) - AI worker.** faster-whisper on the GPU + a local LLM (Ollama) for summaries. This is the Private-tier engine; models run on our own hardware. - **Node B - app tier.** FastAPI ingest + open API, Postgres (accounts, metadata, billing), Redis queue, reverse proxy. - **Node C - storage + overflow.** MinIO (S3-compatible object storage for audio), a CPU transcription worker for extra throughput, backups, and HA spare. Dual 10GbE makes shared storage and job distribution between the three effortless. Run Docker Compose per node, or k3s / Docker Swarm across all three for failover. ### The RTX 3050 6GB in practice - **Transcription: excellent.** faster-whisper (CTranslate2, int8) runs whisper medium/large-v3 within 6GB and several times faster than real-time - hundreds of hours of audio per day on one GPU. Far beyond small-scale need. - **Summaries: fine with 7-8B models.** A quantised 8B model (Llama 3.1 8B, Qwen2.5 7B, Mistral 7B at Q4) fits in 6GB and writes good summaries in seconds. 13B+ will not fit in VRAM; for bigger, use CPU (slower) or add a larger GPU later. Not needed for summaries. So the single 3050 covers both AI jobs the Private tier needs. Cloud tiers barely touch this hardware (orchestration + storage only). ### Getting to it from the internet: Cloudflare Tunnel Do not expose the home/office IP or open ports. We are already on Cloudflare - run a **Cloudflare Tunnel**: the nodes make an outbound connection and `api.nightjar...` is served through Cloudflare with TLS and DDoS protection, no inbound ports, no static IP. Free. ### Caveats of self-hosting a paid service - **Uptime / power / bandwidth.** Fine for beta / early customers. Add a UPS, decent upload bandwidth, tunnel resilience. For SLA-bound customers later, consider colo or a hybrid. - **Backups.** NVMe fails. Keep offsite backups of Postgres + audio (cheap cloud object storage target). - **Data residency.** On our own UK premises - a strong Private-tier pitch ("processed on our own hardware in the UK, never sent to a third-party AI"). Mind UK GDPR obligations. - **When we outgrow it.** Many concurrent Private customers, or wanting large LLMs - add VRAM / a second GPU, or a cloud GPU node. ## Fallback / alternative option: cloud VMs (Hetzner) If self-hosting uptime is a concern, or to launch before the cluster is ready: - **Phase 1 (Cloud tiers):** one small Hetzner VM (CPX31, 4 vCPU / 8 GB, ~EUR 15/mo, EU region for GDPR). Docker Compose: FastAPI + workers + Redis + Postgres + Caddy/LE. Object storage via Hetzner Object Storage (~EUR 5/mo). AI offloaded to a commercial API. All-in ~EUR 20-30/mo. - **Phase 2 (Private tier):** a CPU-optimised (CCX23/33, ~EUR 30-60/mo) or GPU VM for self-hosted whisper + LLM. Priced into the Private tier. A hybrid also works: MS-02 cluster does the compute; a tiny cloud VM (or just the Cloudflare Tunnel) is the public front door. ## Architecture (both options) ``` device --sync--> ingest API --> queue --> transcription worker --> summary (LLM) | (GPU or commercial) (local or commercial) v Postgres (metadata/accounts) object storage (audio + artefacts) | open REST API + exports <-- app / customers ``` ## Provisioning checklist (when ready) - [ ] Decide compute home: MS-02 cluster (primary) vs cloud (fallback) vs hybrid. - [ ] Cloudflare Tunnel for `api.nightjar...` (if self-hosting). - [ ] Docker Compose (or k3s): FastAPI, workers, Redis, Postgres, MinIO/Object Storage. - [ ] GPU node: faster-whisper (CUDA) + Ollama (7-8B) for the Private tier. - [ ] Backups (Postgres + audio) offsite; UPS on the nodes. - [ ] Metering + billing hookup for the plans (see `docs/hosted-service.md`). - [ ] Data-processing agreement + retention policy (Private tier).