Laurence
054b2f6981
Implements the first working feature: the device records audio to the microSD
card, toggled by the button, with LED status and per-recording JSON metadata.
What changed:
- firmware/src/audio.{h,cpp}: mic capture via the core-bundled ESP_I2S. Dev board
uses I2S standard mode (INMP441/ICS-43434); XIAO uses PDM mode (onboard mic).
Presents 16-bit PCM mono to callers regardless of board.
- firmware/src/storage.{h,cpp}: microSD on a dedicated HSPI bus + a streaming
WavWriter that writes a 44-byte PCM header and patches RIFF/data sizes on close;
plus sidecar JSON metadata writer.
- firmware/src/recorder.{h,cpp}: idle/recording state machine; creates
/recordings/<id>.wav, pumps mic chunks in on update(), finalises + writes
<id>.json (Recording schema from api/openapi.yaml) on stop.
- firmware/src/ux.{h,cpp}: debounced button (short press toggles) + status LED
patterns (idle/recording/error), active-low aware.
- firmware/src/main.cpp: wires ux + recorder; loop toggles on button and drains
the mic while recording.
- firmware/include/audio_config.h: 16 kHz mono 16-bit, chunk size, rec dir.
- firmware/include/pins.h: added XIAO PDM mic + onboard SD pins, LED active-low flag.
- state/: TODO, ARCHITECTURE, NOTES updated for M1 and the deferred follow-ups.
Why:
- Recording is the foundation every later milestone (transfer, upload, transcription)
builds on. Kept dependency-free (only core-bundled ESP_I2S + SD) for simple CI builds.
Notes:
- Not compiled locally (no PlatformIO on the dev host) or hardware-verified; Forgejo
Actions CI builds both board profiles. Follow-ups tracked in state/TODO.md:
INMP441 16-bit level calibration, and real NTP timestamps (ids are uptime-based
until M2 brings WiFi/NTP).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
134 lines
6.3 KiB
Markdown
134 lines
6.3 KiB
Markdown
# Architecture
|
|
|
|
> How the system is built and why. Update this when the structure changes; a change is
|
|
> not finished until this reflects it.
|
|
|
|
## Overview
|
|
|
|
Four parts, connected by an open REST API and a shared recording data model:
|
|
|
|
```
|
|
[ Device: ESP32-S3 ] [ Self-hosted server ]
|
|
mic -> I2S -> ring buffer (PSRAM) FastAPI
|
|
-> encoder (WAV) -> microSD +-- ingest (from cloud store / upload)
|
|
button/LED/haptic UX +-- faster-whisper (transcribe)
|
|
power + charge detect +-- Ollama LLM (summarise)
|
|
| BLE (control/provision) +-- object store (MinIO/local) + DB
|
|
| WiFi REST API (LAN) `-- open REST API + exports
|
|
| WiFi uploader (on charge) --> cloud store -----------^
|
|
| |
|
|
v v
|
|
[ Flutter app: Android + iOS ] <---- open REST API (device + server)
|
|
```
|
|
|
|
Three sync paths, exactly as specified:
|
|
1. BLE: control, status, and WiFi provisioning (small data). Portable/battery mode.
|
|
2. WiFi to app: bulk recording transfer via the device REST API (fast).
|
|
3. Independent WiFi upload: when on charge / hard-powered the device auto-joins WiFi and
|
|
pushes recordings to generic cloud storage with no phone present.
|
|
|
|
## Components
|
|
|
|
### firmware (device)
|
|
- Responsibility: capture audio, store it, manage power/controls, expose control + data
|
|
over BLE and WiFi, and upload autonomously when powered.
|
|
- Location: `firmware/` (PlatformIO, Arduino-ESP32, target ESP32-S3).
|
|
- Modules (M1 landed: audio, storage, recorder, ux; rest planned per milestone):
|
|
- `audio` - I2S/PDM mic capture presenting 16-bit PCM mono (ESP_I2S). [M1]
|
|
- `storage`- microSD (FAT) streaming WAV writer + sidecar JSON metadata. [M1]
|
|
- `recorder` - session state machine (idle/recording), file naming, metadata. [M1]
|
|
- `ux` - button (short-press start/stop) + status LED (haptic later). [M1]
|
|
- `power` - battery read (ADC), charge/VBUS detect -> mode switch.
|
|
- `config` - NVS-stored settings (WiFi creds, upload target + keys, codec).
|
|
- `net_wifi` - WiFi manager (join, reconnect), mDNS.
|
|
- `api_http` - on-device REST server (see `api/openapi.yaml`).
|
|
- `uploader` - S3-compatible / WebDAV client; pushes audio + metadata when powered.
|
|
- `ble` - GATT: device info, battery, record control, WiFi provisioning, status.
|
|
- `ota` - firmware update over HTTP.
|
|
- Depends on: microSD, I2S mic, LiPo + charge IC (see `hardware/BOM.md`).
|
|
- Why this way: ESP32-S3 has WiFi + BLE 5 + PSRAM + USB in one cheap chip, so all three
|
|
sync paths and audio buffering fit on one board with off-the-shelf modules.
|
|
|
|
### server (self-hosted AI)
|
|
- Responsibility: ingest recordings, transcribe, summarise, store, and serve the open
|
|
API with exports.
|
|
- Location: `server/` (FastAPI).
|
|
- Pipeline: ingest (from cloud store or direct upload) -> store raw audio (object store)
|
|
-> transcribe (faster-whisper) -> summarise (Ollama LLM) -> index metadata (DB) ->
|
|
expose REST API + exports (audio, TXT, SRT, VTT, Markdown, JSON).
|
|
- Depends on: object storage (MinIO or local FS), a DB (SQLite to start, Postgres later),
|
|
faster-whisper, Ollama. All self-hostable.
|
|
- Why this way: keeps the device cheap and low-power (no on-device AI); all heavy compute
|
|
runs on hardware the user owns; every step swappable and open.
|
|
|
|
### app (Flutter)
|
|
- Responsibility: provision the device, browse the library, play audio, show transcripts
|
|
and summaries, export/share, manage settings.
|
|
- Location: `app/`.
|
|
- Depends on: device BLE + REST API (provisioning/transfer) and server REST API (library,
|
|
transcripts, summaries).
|
|
- Why this way: one codebase for Android + iOS. iOS restricts background BLE, so BLE is
|
|
used for control/provisioning and WiFi for bulk transfer, which matches the design.
|
|
|
|
### case (3D print)
|
|
- Responsibility: enclosure for the chosen board + battery + mic + button + USB + LED.
|
|
- Location: `case/` (OpenSCAD, parametric).
|
|
- Why this way: code-defined parametric model re-tunes to exact module dimensions and
|
|
stays fully open and diffable.
|
|
|
|
### hardware
|
|
- Responsibility: BOM, wiring/pinout, build notes. No custom PCB in v1.
|
|
- Location: `hardware/`.
|
|
|
|
### api
|
|
- Responsibility: the single source of truth for the open API (device + server).
|
|
- Location: `api/openapi.yaml`.
|
|
|
|
## Data and state
|
|
|
|
Recording metadata (sidecar JSON on device; row in server DB), canonical shape:
|
|
|
|
```json
|
|
{
|
|
"id": "rec_20260703T101500Z_ab12",
|
|
"device_id": "openscribe-abc123",
|
|
"started_at": "2026-07-03T10:15:00Z",
|
|
"duration_s": 372.5,
|
|
"sample_rate": 16000,
|
|
"channels": 1,
|
|
"codec": "wav_pcm_s16le",
|
|
"size_bytes": 11920000,
|
|
"sha256": "…",
|
|
"source": "device",
|
|
"sync_state": "local | uploaded | ingested | transcribed | summarised",
|
|
"transcript_ref": null,
|
|
"summary_ref": null
|
|
}
|
|
```
|
|
|
|
- On device: files on microSD (`/recordings/<id>.wav` + `<id>.json`); config + secrets in
|
|
NVS (never on the SD card in clear).
|
|
- In transit: audio + metadata JSON uploaded to the configured object store; server
|
|
ingests from there (or accepts direct upload).
|
|
- On server: audio + artefacts (transcript, summary, subtitle files) in the object store;
|
|
metadata + refs in the DB.
|
|
|
|
## External dependencies
|
|
|
|
- ESP32-S3 (Espressif), Arduino-ESP32, PlatformIO - mature, free, WiFi + BLE + PSRAM.
|
|
- faster-whisper (CTranslate2) - fast self-hosted STT, CPU or GPU.
|
|
- Ollama - self-hosted local LLM runtime for summaries.
|
|
- MinIO (or any S3-compatible / WebDAV target) - self-hosted object storage.
|
|
- FastAPI, Flutter - open, well supported.
|
|
All chosen to be self-hostable and open; no required proprietary SaaS.
|
|
|
|
## Constraints and trade-offs
|
|
|
|
- Audio default is WAV PCM 16 kHz mono for simplicity and quality; larger files, so WiFi
|
|
is the real transfer channel and Opus/ADPCM is a later size optimisation.
|
|
- No on-device transcription: keeps the device cheap/low-power; needs the server for AI.
|
|
- BLE bulk transfer is slow and iOS-restricted, so BLE only does control/provisioning and
|
|
hands transfers to WiFi.
|
|
- v1 uses off-the-shelf modules (no PCB): easier to build, bigger case than a Plaud.
|
|
- Security: device REST API and config writes must be authenticated (token in NVS);
|
|
independent uploads use scoped object-store credentials. Hardening tracked in TODO.
|