openscribe/state/ARCHITECTURE.md

Architecture

How the system is built and why. Update this when the structure changes; a change is not finished until this reflects it.

Overview

Four parts, connected by an open REST API and a shared recording data model:

  [ Device: ESP32-S3 ]                          [ Self-hosted server ]
   mic -> I2S -> ring buffer (PSRAM)             FastAPI
        -> encoder (WAV) -> microSD              +-- ingest (from cloud store / upload)
   button/LED/haptic UX                          +-- faster-whisper (transcribe)
   power + charge detect                         +-- Ollama LLM (summarise)
   |  BLE (control/provision)                    +-- object store (MinIO/local) + DB
   |  WiFi REST API (LAN)                         `-- open REST API + exports
   |  WiFi uploader (on charge) --> cloud store -----------^
                     |                                     |
                     v                                     v
            [ Flutter app: Android + iOS ] <---- open REST API (device + server)

Three sync paths, exactly as specified:

1. BLE: control, status, and WiFi provisioning (small data). Portable/battery mode.
2. WiFi to app: bulk recording transfer via the device REST API (fast).
3. Independent WiFi upload: when on charge / hard-powered the device auto-joins WiFi and pushes recordings to generic cloud storage with no phone present.

Components

firmware (device)

• Responsibility: capture audio, store it, manage power/controls, expose control + data over BLE and WiFi, and upload autonomously when powered.
• Location: firmware/ (PlatformIO, Arduino-ESP32, target ESP32-S3).
• Modules (M1 landed: audio, storage, recorder, ux; rest planned per milestone):
  • audio - I2S/PDM mic capture presenting 16-bit PCM mono (ESP_I2S). [M1]
  • storage- microSD (FAT) streaming WAV writer + sidecar JSON metadata. [M1]
  • recorder - session state machine (idle/recording), file naming, metadata. [M1]
  • ux - button (short-press start/stop) + status LED (haptic later). [M1]
  • power - battery read (ADC), charge/VBUS detect -> mode switch.
  • config - NVS-stored settings (device id, WiFi creds, api token, upload). [M2]
  • net_wifi - WiFi station join/reconnect + SoftAP provisioning fallback + mDNS. [M2]
  • api_http - on-device REST server implementing the device paths. [M2]
  • uploader - S3-compatible / WebDAV client; pushes audio + metadata when powered.
  • ble - GATT: device info, battery, record control, WiFi provisioning, status.
  • ota - firmware update over HTTP.
• Depends on: microSD, I2S mic, LiPo + charge IC (see hardware/BOM.md).
• Why this way: ESP32-S3 has WiFi + BLE 5 + PSRAM + USB in one cheap chip, so all three sync paths and audio buffering fit on one board with off-the-shelf modules.

server (self-hosted AI)

• Responsibility: ingest recordings, transcribe, summarise, store, and serve the open API with exports.
• Location: server/ (FastAPI).
• Pipeline: ingest (from cloud store or direct upload) -> store raw audio (object store) -> transcribe (faster-whisper) -> summarise (Ollama LLM) -> index metadata (DB) -> expose REST API + exports (audio, TXT, SRT, VTT, Markdown, JSON).
• AI is provider-pluggable (server/app/providers/, see docs/ai-providers.md): transcription and the LLM each target a configured provider - open-standard (OpenAI-compatible / local faster-whisper / Ollama) or commercial (OpenAI, Anthropic, Gemini). No lock-in.
• Depends on: object storage (MinIO or local FS), a DB (SQLite to start, Postgres later), and whichever AI providers are configured (default: self-hosted faster-whisper + Ollama).
• Why this way: keeps the device cheap and low-power (no on-device AI); all heavy compute runs on hardware the user owns; every step swappable and open.

app (Flutter)

• Responsibility: provision the device, browse the library, play audio, show transcripts and summaries, export/share, manage settings.
• Location: app/.
• Depends on: device BLE + REST API (provisioning/transfer) and server REST API (library, transcripts, summaries).
• Why this way: one codebase for Android + iOS. iOS restricts background BLE, so BLE is used for control/provisioning and WiFi for bulk transfer, which matches the design.

case (3D print)

• Responsibility: enclosure for the chosen board + battery + mic + button + USB + LED.
• Location: case/ (OpenSCAD, parametric).
• Why this way: code-defined parametric model re-tunes to exact module dimensions and stays fully open and diffable.

hardware

• Responsibility: BOM, wiring/pinout, build notes. No custom PCB in v1.
• Location: hardware/.

api

• Responsibility: the single source of truth for the open API (device + server).
• Location: api/openapi.yaml.

Data and state

Recording metadata (sidecar JSON on device; row in server DB), canonical shape:

{
  "id": "rec_20260703T101500Z_ab12",
  "device_id": "openscribe-abc123",
  "started_at": "2026-07-03T10:15:00Z",
  "duration_s": 372.5,
  "sample_rate": 16000,
  "channels": 1,
  "codec": "wav_pcm_s16le",
  "size_bytes": 11920000,
  "sha256": "…",
  "source": "device",
  "sync_state": "local | uploaded | ingested | transcribed | summarised",
  "transcript_ref": null,
  "summary_ref": null
}

• On device: files on microSD (/recordings/<id>.wav + <id>.json); config + secrets in NVS (never on the SD card in clear).
• In transit: audio + metadata JSON uploaded to the configured object store; server ingests from there (or accepts direct upload).
• On server: audio + artefacts (transcript, summary, subtitle files) in the object store; metadata + refs in the DB.

External dependencies

• ESP32-S3 (Espressif), Arduino-ESP32, PlatformIO - mature, free, WiFi + BLE + PSRAM.
• faster-whisper (CTranslate2) - fast self-hosted STT, CPU or GPU.
• Ollama - self-hosted local LLM runtime for summaries.
• MinIO (or any S3-compatible / WebDAV target) - self-hosted object storage.
• FastAPI, Flutter - open, well supported. All chosen to be self-hostable and open; no required proprietary SaaS.

Constraints and trade-offs

• Audio default is WAV PCM 16 kHz mono for simplicity and quality; larger files, so WiFi is the real transfer channel and Opus/ADPCM is a later size optimisation.
• No on-device transcription: keeps the device cheap/low-power; needs the server for AI.
• BLE bulk transfer is slow and iOS-restricted, so BLE only does control/provisioning and hands transfers to WiFi.
• v1 uses off-the-shelf modules (no PCB): easier to build, bigger case than a Plaud.
• Security: device REST API and config writes must be authenticated (token in NVS); independent uploads use scoped object-store credentials. Hardening tracked in TODO.