openscribe/docs/infrastructure.md

Nightjar hosted backend - infrastructure plan

Plan for the infrastructure that runs the hosted Nightjar service (the commercial plans). The open-source project itself does not require any of this; self-hosters run their own. This is for 12 Hobbies Studio to operate the managed service.

Status: plan only, not provisioned (decided 2026-07). Stand it up at first beta / paying users. See docs/hosted-service.md for the plan tiers this maps to.

Two backends, matching the two plan families

Plan family	AI runs on	Infra
Cloud Starter / Pro	commercial API (Groq / OpenAI / Anthropic)	small orchestration node; heavy compute offloaded
Private	self-hosted open models (faster-whisper + local LLM)	a GPU node; audio never leaves our hardware

Cloud tiers are cheap to run (compute is per-minute on a commercial provider, keeping the GBP 6 / 18 pricing profitable). The Private tier needs real hardware but is the privacy / sovereignty story.

Primary option: self-host on the MS-02 cluster (best for small scale)

Hardware on hand: 3x Minisforum MS-02-class mini-workstations, one fitted with an NVIDIA RTX 3050 6GB. High-core CPUs, dual 10GbE, NVMe, PCIe GPU slot. This is a strong small-scale backend and a better fit for the Private tier than rented cloud.

Node roles

• Node A (RTX 3050) - AI worker. faster-whisper on the GPU + a local LLM (Ollama) for summaries. This is the Private-tier engine; models run on our own hardware.
• Node B - app tier. FastAPI ingest + open API, Postgres (accounts, metadata, billing), Redis queue, reverse proxy.
• Node C - storage + overflow. MinIO (S3-compatible object storage for audio), a CPU transcription worker for extra throughput, backups, and HA spare.

Dual 10GbE makes shared storage and job distribution between the three effortless. Run Docker Compose per node, or k3s / Docker Swarm across all three for failover.

The RTX 3050 6GB in practice

• Transcription: excellent. faster-whisper (CTranslate2, int8) runs whisper medium/large-v3 within 6GB and several times faster than real-time - hundreds of hours of audio per day on one GPU. Far beyond small-scale need.
• Summaries: fine with 7-8B models. A quantised 8B model (Llama 3.1 8B, Qwen2.5 7B, Mistral 7B at Q4) fits in 6GB and writes good summaries in seconds. 13B+ will not fit in VRAM; for bigger, use CPU (slower) or add a larger GPU later. Not needed for summaries.

So the single 3050 covers both AI jobs the Private tier needs. Cloud tiers barely touch this hardware (orchestration + storage only).

Getting to it from the internet: Cloudflare Tunnel

Do not expose the home/office IP or open ports. We are already on Cloudflare - run a Cloudflare Tunnel: the nodes make an outbound connection and api.nightjar... is served through Cloudflare with TLS and DDoS protection, no inbound ports, no static IP. Free.

Caveats of self-hosting a paid service

• Uptime / power / bandwidth. Fine for beta / early customers. Add a UPS, decent upload bandwidth, tunnel resilience. For SLA-bound customers later, consider colo or a hybrid.
• Backups. NVMe fails. Keep offsite backups of Postgres + audio (cheap cloud object storage target).
• Data residency. On our own UK premises - a strong Private-tier pitch ("processed on our own hardware in the UK, never sent to a third-party AI"). Mind UK GDPR obligations.
• When we outgrow it. Many concurrent Private customers, or wanting large LLMs - add VRAM / a second GPU, or a cloud GPU node.

Fallback / alternative option: cloud VMs (Hetzner)

If self-hosting uptime is a concern, or to launch before the cluster is ready:

• Phase 1 (Cloud tiers): one small Hetzner VM (CPX31, 4 vCPU / 8 GB, ~EUR 15/mo, EU region for GDPR). Docker Compose: FastAPI + workers + Redis + Postgres + Caddy/LE. Object storage via Hetzner Object Storage (~EUR 5/mo). AI offloaded to a commercial API. All-in ~EUR 20-30/mo.
• Phase 2 (Private tier): a CPU-optimised (CCX23/33, ~EUR 30-60/mo) or GPU VM for self-hosted whisper + LLM. Priced into the Private tier.

A hybrid also works: MS-02 cluster does the compute; a tiny cloud VM (or just the Cloudflare Tunnel) is the public front door.

Architecture (both options)

device --sync--> ingest API --> queue --> transcription worker --> summary (LLM)
                     |                          (GPU or commercial)   (local or commercial)
                     v
                 Postgres (metadata/accounts)      object storage (audio + artefacts)
                     |
                 open REST API + exports  <-- app / customers

Provisioning checklist (when ready)

• Decide compute home: MS-02 cluster (primary) vs cloud (fallback) vs hybrid.
• Cloudflare Tunnel for api.nightjar... (if self-hosting).
• Docker Compose (or k3s): FastAPI, workers, Redis, Postgres, MinIO/Object Storage.
• GPU node: faster-whisper (CUDA) + Ollama (7-8B) for the Private tier.
• Backups (Postgres + audio) offsite; UPS on the nodes.
• Metering + billing hookup for the plans (see docs/hosted-service.md).
• Data-processing agreement + retention policy (Private tier).