Laurence
054b2f6981
Implements the first working feature: the device records audio to the microSD
card, toggled by the button, with LED status and per-recording JSON metadata.
What changed:
- firmware/src/audio.{h,cpp}: mic capture via the core-bundled ESP_I2S. Dev board
uses I2S standard mode (INMP441/ICS-43434); XIAO uses PDM mode (onboard mic).
Presents 16-bit PCM mono to callers regardless of board.
- firmware/src/storage.{h,cpp}: microSD on a dedicated HSPI bus + a streaming
WavWriter that writes a 44-byte PCM header and patches RIFF/data sizes on close;
plus sidecar JSON metadata writer.
- firmware/src/recorder.{h,cpp}: idle/recording state machine; creates
/recordings/<id>.wav, pumps mic chunks in on update(), finalises + writes
<id>.json (Recording schema from api/openapi.yaml) on stop.
- firmware/src/ux.{h,cpp}: debounced button (short press toggles) + status LED
patterns (idle/recording/error), active-low aware.
- firmware/src/main.cpp: wires ux + recorder; loop toggles on button and drains
the mic while recording.
- firmware/include/audio_config.h: 16 kHz mono 16-bit, chunk size, rec dir.
- firmware/include/pins.h: added XIAO PDM mic + onboard SD pins, LED active-low flag.
- state/: TODO, ARCHITECTURE, NOTES updated for M1 and the deferred follow-ups.
Why:
- Recording is the foundation every later milestone (transfer, upload, transcription)
builds on. Kept dependency-free (only core-bundled ESP_I2S + SD) for simple CI builds.
Notes:
- Not compiled locally (no PlatformIO on the dev host) or hardware-verified; Forgejo
Actions CI builds both board profiles. Follow-ups tracked in state/TODO.md:
INMP441 16-bit level calibration, and real NTP timestamps (ids are uptime-based
until M2 brings WiFi/NTP).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6.3 KiB
6.3 KiB
Architecture
How the system is built and why. Update this when the structure changes; a change is not finished until this reflects it.
Overview
Four parts, connected by an open REST API and a shared recording data model:
[ Device: ESP32-S3 ] [ Self-hosted server ]
mic -> I2S -> ring buffer (PSRAM) FastAPI
-> encoder (WAV) -> microSD +-- ingest (from cloud store / upload)
button/LED/haptic UX +-- faster-whisper (transcribe)
power + charge detect +-- Ollama LLM (summarise)
| BLE (control/provision) +-- object store (MinIO/local) + DB
| WiFi REST API (LAN) `-- open REST API + exports
| WiFi uploader (on charge) --> cloud store -----------^
| |
v v
[ Flutter app: Android + iOS ] <---- open REST API (device + server)
Three sync paths, exactly as specified:
- BLE: control, status, and WiFi provisioning (small data). Portable/battery mode.
- WiFi to app: bulk recording transfer via the device REST API (fast).
- Independent WiFi upload: when on charge / hard-powered the device auto-joins WiFi and pushes recordings to generic cloud storage with no phone present.
Components
firmware (device)
- Responsibility: capture audio, store it, manage power/controls, expose control + data over BLE and WiFi, and upload autonomously when powered.
- Location:
firmware/(PlatformIO, Arduino-ESP32, target ESP32-S3). - Modules (M1 landed: audio, storage, recorder, ux; rest planned per milestone):
audio- I2S/PDM mic capture presenting 16-bit PCM mono (ESP_I2S). [M1]storage- microSD (FAT) streaming WAV writer + sidecar JSON metadata. [M1]recorder- session state machine (idle/recording), file naming, metadata. [M1]ux- button (short-press start/stop) + status LED (haptic later). [M1]power- battery read (ADC), charge/VBUS detect -> mode switch.config- NVS-stored settings (WiFi creds, upload target + keys, codec).net_wifi- WiFi manager (join, reconnect), mDNS.api_http- on-device REST server (seeapi/openapi.yaml).uploader- S3-compatible / WebDAV client; pushes audio + metadata when powered.ble- GATT: device info, battery, record control, WiFi provisioning, status.ota- firmware update over HTTP.
- Depends on: microSD, I2S mic, LiPo + charge IC (see
hardware/BOM.md). - Why this way: ESP32-S3 has WiFi + BLE 5 + PSRAM + USB in one cheap chip, so all three sync paths and audio buffering fit on one board with off-the-shelf modules.
server (self-hosted AI)
- Responsibility: ingest recordings, transcribe, summarise, store, and serve the open API with exports.
- Location:
server/(FastAPI). - Pipeline: ingest (from cloud store or direct upload) -> store raw audio (object store) -> transcribe (faster-whisper) -> summarise (Ollama LLM) -> index metadata (DB) -> expose REST API + exports (audio, TXT, SRT, VTT, Markdown, JSON).
- Depends on: object storage (MinIO or local FS), a DB (SQLite to start, Postgres later), faster-whisper, Ollama. All self-hostable.
- Why this way: keeps the device cheap and low-power (no on-device AI); all heavy compute runs on hardware the user owns; every step swappable and open.
app (Flutter)
- Responsibility: provision the device, browse the library, play audio, show transcripts and summaries, export/share, manage settings.
- Location:
app/. - Depends on: device BLE + REST API (provisioning/transfer) and server REST API (library, transcripts, summaries).
- Why this way: one codebase for Android + iOS. iOS restricts background BLE, so BLE is used for control/provisioning and WiFi for bulk transfer, which matches the design.
case (3D print)
- Responsibility: enclosure for the chosen board + battery + mic + button + USB + LED.
- Location:
case/(OpenSCAD, parametric). - Why this way: code-defined parametric model re-tunes to exact module dimensions and stays fully open and diffable.
hardware
- Responsibility: BOM, wiring/pinout, build notes. No custom PCB in v1.
- Location:
hardware/.
api
- Responsibility: the single source of truth for the open API (device + server).
- Location:
api/openapi.yaml.
Data and state
Recording metadata (sidecar JSON on device; row in server DB), canonical shape:
{
"id": "rec_20260703T101500Z_ab12",
"device_id": "openscribe-abc123",
"started_at": "2026-07-03T10:15:00Z",
"duration_s": 372.5,
"sample_rate": 16000,
"channels": 1,
"codec": "wav_pcm_s16le",
"size_bytes": 11920000,
"sha256": "…",
"source": "device",
"sync_state": "local | uploaded | ingested | transcribed | summarised",
"transcript_ref": null,
"summary_ref": null
}
- On device: files on microSD (
/recordings/<id>.wav+<id>.json); config + secrets in NVS (never on the SD card in clear). - In transit: audio + metadata JSON uploaded to the configured object store; server ingests from there (or accepts direct upload).
- On server: audio + artefacts (transcript, summary, subtitle files) in the object store; metadata + refs in the DB.
External dependencies
- ESP32-S3 (Espressif), Arduino-ESP32, PlatformIO - mature, free, WiFi + BLE + PSRAM.
- faster-whisper (CTranslate2) - fast self-hosted STT, CPU or GPU.
- Ollama - self-hosted local LLM runtime for summaries.
- MinIO (or any S3-compatible / WebDAV target) - self-hosted object storage.
- FastAPI, Flutter - open, well supported. All chosen to be self-hostable and open; no required proprietary SaaS.
Constraints and trade-offs
- Audio default is WAV PCM 16 kHz mono for simplicity and quality; larger files, so WiFi is the real transfer channel and Opus/ADPCM is a later size optimisation.
- No on-device transcription: keeps the device cheap/low-power; needs the server for AI.
- BLE bulk transfer is slow and iOS-restricted, so BLE only does control/provisioning and hands transfers to WiFi.
- v1 uses off-the-shelf modules (no PCB): easier to build, bigger case than a Plaud.
- Security: device REST API and config writes must be authenticated (token in NVS); independent uploads use scoped object-store credentials. Hardening tracked in TODO.