openscribe/state/ARCHITECTURE.md

# Architecture

> How the system is built and why. Update this when the structure changes; a change is
> not finished until this reflects it.

## Overview

Four parts, connected by an open REST API and a shared recording data model:

```
  [ Device: ESP32-S3 ]                          [ Self-hosted server ]
   mic -> I2S -> ring buffer (PSRAM)             FastAPI
        -> encoder (WAV) -> microSD              +-- ingest (from cloud store / upload)
   button/LED/haptic UX                          +-- faster-whisper (transcribe)
   power + charge detect                         +-- Ollama LLM (summarise)
   |  BLE (control/provision)                    +-- object store (MinIO/local) + DB
   |  WiFi REST API (LAN)                         `-- open REST API + exports
   |  WiFi uploader (on charge) --> cloud store -----------^
                     |                                     |
                     v                                     v
            [ Flutter app: Android + iOS ] <---- open REST API (device + server)
```

Three sync paths, exactly as specified:
1. BLE: control, status, and WiFi provisioning (small data). Portable/battery mode.
2. WiFi to app: bulk recording transfer via the device REST API (fast).
3. Independent WiFi upload: when on charge / hard-powered the device auto-joins WiFi and
   pushes recordings to generic cloud storage with no phone present.

## Components

### firmware (device)
- Responsibility: capture audio, store it, manage power/controls, expose control + data
  over BLE and WiFi, and upload autonomously when powered.
- Location: `firmware/` (PlatformIO, Arduino-ESP32, target ESP32-S3).
- Modules (M1 landed: audio, storage, recorder, ux; rest planned per milestone):
  - `audio`  - I2S/PDM mic capture presenting 16-bit PCM mono (ESP_I2S). [M1]
  - `storage`- microSD (FAT) streaming WAV writer + sidecar JSON metadata. [M1]
  - `recorder` - session state machine (idle/recording), file naming, metadata. [M1]
  - `ux`     - button (short-press start/stop) + status LED (haptic later). [M1]
  - `power`  - battery read (ADC), charge/VBUS detect -> mode switch.
  - `config` - NVS-stored settings (device id, WiFi creds, api token, upload). [M2]
  - `net_wifi` - WiFi station join/reconnect + SoftAP provisioning fallback + mDNS. [M2]
  - `api_http` - on-device REST server implementing the device paths. [M2]
  - `uploader` - S3-compatible / WebDAV client; pushes audio + metadata when powered.
  - `ble`    - GATT: device info, battery, record control, WiFi provisioning, status.
  - `ota`    - firmware update over HTTP.
- Depends on: microSD, I2S mic, LiPo + charge IC (see `hardware/BOM.md`).
- Why this way: ESP32-S3 has WiFi + BLE 5 + PSRAM + USB in one cheap chip, so all three
  sync paths and audio buffering fit on one board with off-the-shelf modules.

### server (self-hosted AI)
- Responsibility: ingest recordings, transcribe, summarise, store, and serve the open
  API with exports.
- Location: `server/` (FastAPI).
- Pipeline: ingest (from cloud store or direct upload) -> store raw audio (object store)
  -> transcribe (faster-whisper) -> summarise (Ollama LLM) -> index metadata (DB) ->
  expose REST API + exports (audio, TXT, SRT, VTT, Markdown, JSON).
- AI is provider-pluggable (`server/app/providers/`, see docs/ai-providers.md): transcription
  and the LLM each target a configured provider - open-standard (OpenAI-compatible / local
  faster-whisper / Ollama) or commercial (OpenAI, Anthropic, Gemini). No lock-in.
- Depends on: object storage (MinIO or local FS), a DB (SQLite to start, Postgres later),
  and whichever AI providers are configured (default: self-hosted faster-whisper + Ollama).
- Why this way: keeps the device cheap and low-power (no on-device AI); all heavy compute
  runs on hardware the user owns; every step swappable and open.

### app (Flutter)
- Responsibility: provision the device, browse the library, play audio, show transcripts
  and summaries, export/share, manage settings.
- Location: `app/`.
- Depends on: device BLE + REST API (provisioning/transfer) and server REST API (library,
  transcripts, summaries).
- Why this way: one codebase for Android + iOS. iOS restricts background BLE, so BLE is
  used for control/provisioning and WiFi for bulk transfer, which matches the design.

### case (3D print)
- Responsibility: enclosure for the chosen board + battery + mic + button + USB + LED.
- Location: `case/` (OpenSCAD, parametric).
- Why this way: code-defined parametric model re-tunes to exact module dimensions and
  stays fully open and diffable.

### hardware
- Responsibility: BOM, wiring/pinout, build notes. No custom PCB in v1.
- Location: `hardware/`.

### api
- Responsibility: the single source of truth for the open API (device + server).
- Location: `api/openapi.yaml`.

## Data and state

Recording metadata (sidecar JSON on device; row in server DB), canonical shape:

```json
{
  "id": "rec_20260703T101500Z_ab12",
  "device_id": "openscribe-abc123",
  "started_at": "2026-07-03T10:15:00Z",
  "duration_s": 372.5,
  "sample_rate": 16000,
  "channels": 1,
  "codec": "wav_pcm_s16le",
  "size_bytes": 11920000,
  "sha256": "…",
  "source": "device",
  "sync_state": "local | uploaded | ingested | transcribed | summarised",
  "transcript_ref": null,
  "summary_ref": null
}
```

- On device: files on microSD (`/recordings/<id>.wav` + `<id>.json`); config + secrets in
  NVS (never on the SD card in clear).
- In transit: audio + metadata JSON uploaded to the configured object store; server
  ingests from there (or accepts direct upload).
- On server: audio + artefacts (transcript, summary, subtitle files) in the object store;
  metadata + refs in the DB.

## External dependencies

- ESP32-S3 (Espressif), Arduino-ESP32, PlatformIO - mature, free, WiFi + BLE + PSRAM.
- faster-whisper (CTranslate2) - fast self-hosted STT, CPU or GPU.
- Ollama - self-hosted local LLM runtime for summaries.
- MinIO (or any S3-compatible / WebDAV target) - self-hosted object storage.
- FastAPI, Flutter - open, well supported.
All chosen to be self-hostable and open; no required proprietary SaaS.

## Constraints and trade-offs

- Audio default is WAV PCM 16 kHz mono for simplicity and quality; larger files, so WiFi
  is the real transfer channel and Opus/ADPCM is a later size optimisation.
- No on-device transcription: keeps the device cheap/low-power; needs the server for AI.
- BLE bulk transfer is slow and iOS-restricted, so BLE only does control/provisioning and
  hands transfers to WiFi.
- v1 uses off-the-shelf modules (no PCB): easier to build, bigger case than a Plaud.
- Security: device REST API and config writes must be authenticated (token in NVS);
  independent uploads use scoped object-store credentials. Hardening tracked in TODO.