Laurence 87a00265a3

ci / server (pull_request) Failing after 27s

Details

ci / firmware (pull_request) Failing after 36s

Details

feat(firmware): M2 WiFi + on-device REST API

Adds the WiFi-to-app sync path and the on-device open REST API implementing the
device paths of api/openapi.yaml, plus persistent config in NVS.

What changed:
- firmware/src/config.{h,cpp}: NVS (Preferences) store - stable device id from MAC,
  WiFi SSID/PSK, API bearer token, upload settings. Secrets stay off the SD card.
- firmware/src/net_wifi.{h,cpp}: station join with stored creds; SoftAP provisioning
  fallback when no creds / join fails; mDNS openscribe.local; reconnect in loop().
- firmware/src/api_http.{h,cpp}: synchronous WebServer on :80. Routes: GET /device,
  GET/PUT /device/config, GET /recordings, GET /recordings/{id}, GET
  /recordings/{id}/audio (HTTP Range -> 206 + Content-Range), DELETE /recordings/{id},
  POST /record/start|stop. Bearer-token auth on mutating calls (open when no token).
- firmware/src/storage.{h,cpp}: list/read/delete/count helpers + path builders.
- firmware/src/main.cpp: wire config + WiFi + API into setup/loop; bump fw to 0.2.0.
- firmware/platformio.ini: add bblanchon/ArduinoJson@^7 (only new dep).
- state/: ARCHITECTURE, NOTES, TODO updated; security + NTP follow-ups recorded.

Why:
- This is the WiFi transfer channel and the device half of the "completely open API":
  clients can list, download (resumable), delete and control without any cloud.

Notes:
- Not compiled locally (no PlatformIO on the dev host) or hardware-verified; CI builds
  both board profiles. Security follow-ups logged: API is open when no token is set,
  and the SoftAP uses a fixed default passphrase (make user-set at M4/BLE).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-03 12:09:41 +01:00

6.3 KiB

Raw Blame History

Architecture

How the system is built and why. Update this when the structure changes; a change is not finished until this reflects it.

Overview

Four parts, connected by an open REST API and a shared recording data model:

  [ Device: ESP32-S3 ]                          [ Self-hosted server ]
   mic -> I2S -> ring buffer (PSRAM)             FastAPI
        -> encoder (WAV) -> microSD              +-- ingest (from cloud store / upload)
   button/LED/haptic UX                          +-- faster-whisper (transcribe)
   power + charge detect                         +-- Ollama LLM (summarise)
   |  BLE (control/provision)                    +-- object store (MinIO/local) + DB
   |  WiFi REST API (LAN)                         `-- open REST API + exports
   |  WiFi uploader (on charge) --> cloud store -----------^
                     |                                     |
                     v                                     v
            [ Flutter app: Android + iOS ] <---- open REST API (device + server)

Three sync paths, exactly as specified:

BLE: control, status, and WiFi provisioning (small data). Portable/battery mode.
WiFi to app: bulk recording transfer via the device REST API (fast).
Independent WiFi upload: when on charge / hard-powered the device auto-joins WiFi and pushes recordings to generic cloud storage with no phone present.

Components

firmware (device)

Responsibility: capture audio, store it, manage power/controls, expose control + data over BLE and WiFi, and upload autonomously when powered.
Location: firmware/ (PlatformIO, Arduino-ESP32, target ESP32-S3).
Modules (M1 landed: audio, storage, recorder, ux; rest planned per milestone):
- audio - I2S/PDM mic capture presenting 16-bit PCM mono (ESP_I2S). [M1]
- storage- microSD (FAT) streaming WAV writer + sidecar JSON metadata. [M1]
- recorder - session state machine (idle/recording), file naming, metadata. [M1]
- ux - button (short-press start/stop) + status LED (haptic later). [M1]
- power - battery read (ADC), charge/VBUS detect -> mode switch.
- config - NVS-stored settings (device id, WiFi creds, api token, upload). [M2]
- net_wifi - WiFi station join/reconnect + SoftAP provisioning fallback + mDNS. [M2]
- api_http - on-device REST server implementing the device paths. [M2]
- uploader - S3-compatible / WebDAV client; pushes audio + metadata when powered.
- ble - GATT: device info, battery, record control, WiFi provisioning, status.
- ota - firmware update over HTTP.
Depends on: microSD, I2S mic, LiPo + charge IC (see hardware/BOM.md).
Why this way: ESP32-S3 has WiFi + BLE 5 + PSRAM + USB in one cheap chip, so all three sync paths and audio buffering fit on one board with off-the-shelf modules.

server (self-hosted AI)

Responsibility: ingest recordings, transcribe, summarise, store, and serve the open API with exports.
Location: server/ (FastAPI).
Pipeline: ingest (from cloud store or direct upload) -> store raw audio (object store) -> transcribe (faster-whisper) -> summarise (Ollama LLM) -> index metadata (DB) -> expose REST API + exports (audio, TXT, SRT, VTT, Markdown, JSON).
Depends on: object storage (MinIO or local FS), a DB (SQLite to start, Postgres later), faster-whisper, Ollama. All self-hostable.
Why this way: keeps the device cheap and low-power (no on-device AI); all heavy compute runs on hardware the user owns; every step swappable and open.

app (Flutter)

Responsibility: provision the device, browse the library, play audio, show transcripts and summaries, export/share, manage settings.
Location: app/.
Depends on: device BLE + REST API (provisioning/transfer) and server REST API (library, transcripts, summaries).
Why this way: one codebase for Android + iOS. iOS restricts background BLE, so BLE is used for control/provisioning and WiFi for bulk transfer, which matches the design.

case (3D print)

Responsibility: enclosure for the chosen board + battery + mic + button + USB + LED.
Location: case/ (OpenSCAD, parametric).
Why this way: code-defined parametric model re-tunes to exact module dimensions and stays fully open and diffable.

hardware

Responsibility: BOM, wiring/pinout, build notes. No custom PCB in v1.
Location: hardware/.

api

Responsibility: the single source of truth for the open API (device + server).
Location: api/openapi.yaml.

Data and state

Recording metadata (sidecar JSON on device; row in server DB), canonical shape:

{
  "id": "rec_20260703T101500Z_ab12",
  "device_id": "openscribe-abc123",
  "started_at": "2026-07-03T10:15:00Z",
  "duration_s": 372.5,
  "sample_rate": 16000,
  "channels": 1,
  "codec": "wav_pcm_s16le",
  "size_bytes": 11920000,
  "sha256": "…",
  "source": "device",
  "sync_state": "local | uploaded | ingested | transcribed | summarised",
  "transcript_ref": null,
  "summary_ref": null
}

On device: files on microSD (/recordings/<id>.wav + <id>.json); config + secrets in NVS (never on the SD card in clear).
In transit: audio + metadata JSON uploaded to the configured object store; server ingests from there (or accepts direct upload).
On server: audio + artefacts (transcript, summary, subtitle files) in the object store; metadata + refs in the DB.

External dependencies

ESP32-S3 (Espressif), Arduino-ESP32, PlatformIO - mature, free, WiFi + BLE + PSRAM.
faster-whisper (CTranslate2) - fast self-hosted STT, CPU or GPU.
Ollama - self-hosted local LLM runtime for summaries.
MinIO (or any S3-compatible / WebDAV target) - self-hosted object storage.
FastAPI, Flutter - open, well supported. All chosen to be self-hostable and open; no required proprietary SaaS.

Constraints and trade-offs

Audio default is WAV PCM 16 kHz mono for simplicity and quality; larger files, so WiFi is the real transfer channel and Opus/ADPCM is a later size optimisation.
No on-device transcription: keeps the device cheap/low-power; needs the server for AI.
BLE bulk transfer is slow and iOS-restricted, so BLE only does control/provisioning and hands transfers to WiFi.
v1 uses off-the-shelf modules (no PCB): easier to build, bigger case than a Plaud.
Security: device REST API and config writes must be authenticated (token in NVS); independent uploads use scoped object-store credentials. Hardening tracked in TODO.

6.3 KiB Raw Blame History