laurence/openscribe
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

scaffold: OpenScribe open-source self-hosted AI voice recorder
Some checks failed
ci / firmware (push) Failing after 2s
Details
ci / server (push) Failing after 45s
Details

Browse source 
Bootstrap of the project (M0). Sets up the monorepo, design docs, hardware BOM,
the open API contract, component skeletons, licensing and CI, following the
Default Workflow SOP.

What changed:
- CLAUDE.md + docs/: copied the Default Workflow so sessions load the SOP.
- state/: PROJECT, ARCHITECTURE, DECISIONS, TODO, NOTES filled in for OpenScribe.
  ARCHITECTURE captures the four-part design (firmware, server, app, case) and the
  three sync paths; DECISIONS records the hardware, AI-stack, storage, app and
  licensing choices; TODO lays out milestones M1-M9.
- hardware/BOM.md: two build options (compact XIAO ESP32-S3 Sense; dev ESP32-S3 +
  I2S mic + SD), wiring/pinout, indicative cost.
- api/openapi.yaml: the completely open API (device + server surfaces), including
  recording list/download/delete and exports (wav/ogg/txt/srt/vtt/md/json).
- firmware/: PlatformIO ESP32-S3 project, two board profiles, pin map, boot scaffold
  with module seams for M1-M4.
- server/: FastAPI skeleton mirroring the OpenAPI, config for self-hosted MinIO,
  faster-whisper and Ollama; stub routes browsable at /docs.
- app/, case/: Flutter app plan; parametric OpenSCAD enclosure.
- Licensing: GPL-3.0 (code), CERN-OHL-S-2.0 (hardware), CC-BY-SA-4.0 (case/docs),
  REUSE-style LICENSES/ with SPDX headers; LICENSING.md explains the split.
- CI: Forgejo Actions workflow builds firmware (both profiles) and lints/imports server.

Why:
- Everything self-hosted and openly licensed per the user's requirements: an open
  API, three sync paths (BLE control, WiFi transfer, independent WiFi upload on
  charge to generic cloud storage), and a full self-hosted transcription+summary stack.

Notes:
- No custom PCB in v1; off-the-shelf modules. Physical verification waits on parts.
- Component code is stubs at M0; features land milestone by milestone, each as its
  own branch/PR per the workflow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Laurence 2026-07-03 10:21:37 +01:00
commit 031074c9a9
36 changed files with 2922 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

134
state/ARCHITECTURE.md Normal file
View file

@ -0,0 +1,134 @@
# Architecture
> How the system is built and why. Update this when the structure changes; a change is
> not finished until this reflects it.
## Overview
Four parts, connected by an open REST API and a shared recording data model:
```
[ Device: ESP32-S3 ] [ Self-hosted server ]
mic -> I2S -> ring buffer (PSRAM) FastAPI
-> encoder (WAV) -> microSD +-- ingest (from cloud store / upload)
button/LED/haptic UX +-- faster-whisper (transcribe)
power + charge detect +-- Ollama LLM (summarise)
| BLE (control/provision) +-- object store (MinIO/local) + DB
| WiFi REST API (LAN) `-- open REST API + exports
| WiFi uploader (on charge) --> cloud store -----------^
| |
v v
[ Flutter app: Android + iOS ] <---- open REST API (device + server)
```
Three sync paths, exactly as specified:
1. BLE: control, status, and WiFi provisioning (small data). Portable/battery mode.
2. WiFi to app: bulk recording transfer via the device REST API (fast).
3. Independent WiFi upload: when on charge / hard-powered the device auto-joins WiFi and
pushes recordings to generic cloud storage with no phone present.
## Components
### firmware (device)
- Responsibility: capture audio, store it, manage power/controls, expose control + data
over BLE and WiFi, and upload autonomously when powered.
- Location: `firmware/` (PlatformIO, Arduino-ESP32, target ESP32-S3).
- Modules (planned):
- `audio` - I2S mic driver, DMA capture, PSRAM ring buffer, WAV encoder.
- `storage`- microSD (FAT) recording files + sidecar JSON metadata.
- `recorder` - session state machine (idle/recording), file naming, metadata.
- `ux` - button (start/stop, long-press pair), LED/haptic status.
- `power` - battery read (ADC), charge/VBUS detect -> mode switch.
- `config` - NVS-stored settings (WiFi creds, upload target + keys, codec).
- `net_wifi` - WiFi manager (join, reconnect), mDNS.
- `api_http` - on-device REST server (see `api/openapi.yaml`).
- `uploader` - S3-compatible / WebDAV client; pushes audio + metadata when powered.
- `ble` - GATT: device info, battery, record control, WiFi provisioning, status.
- `ota` - firmware update over HTTP.
- Depends on: microSD, I2S mic, LiPo + charge IC (see `hardware/BOM.md`).
- Why this way: ESP32-S3 has WiFi + BLE 5 + PSRAM + USB in one cheap chip, so all three
sync paths and audio buffering fit on one board with off-the-shelf modules.
### server (self-hosted AI)
- Responsibility: ingest recordings, transcribe, summarise, store, and serve the open
API with exports.
- Location: `server/` (FastAPI).
- Pipeline: ingest (from cloud store or direct upload) -> store raw audio (object store)
-> transcribe (faster-whisper) -> summarise (Ollama LLM) -> index metadata (DB) ->
expose REST API + exports (audio, TXT, SRT, VTT, Markdown, JSON).
- Depends on: object storage (MinIO or local FS), a DB (SQLite to start, Postgres later),
faster-whisper, Ollama. All self-hostable.
- Why this way: keeps the device cheap and low-power (no on-device AI); all heavy compute
runs on hardware the user owns; every step swappable and open.
### app (Flutter)
- Responsibility: provision the device, browse the library, play audio, show transcripts
and summaries, export/share, manage settings.
- Location: `app/`.
- Depends on: device BLE + REST API (provisioning/transfer) and server REST API (library,
transcripts, summaries).
- Why this way: one codebase for Android + iOS. iOS restricts background BLE, so BLE is
used for control/provisioning and WiFi for bulk transfer, which matches the design.
### case (3D print)
- Responsibility: enclosure for the chosen board + battery + mic + button + USB + LED.
- Location: `case/` (OpenSCAD, parametric).
- Why this way: code-defined parametric model re-tunes to exact module dimensions and
stays fully open and diffable.
### hardware
- Responsibility: BOM, wiring/pinout, build notes. No custom PCB in v1.
- Location: `hardware/`.
### api
- Responsibility: the single source of truth for the open API (device + server).
- Location: `api/openapi.yaml`.
## Data and state
Recording metadata (sidecar JSON on device; row in server DB), canonical shape:
```json
{
"id": "rec_20260703T101500Z_ab12",
"device_id": "openscribe-abc123",
"started_at": "2026-07-03T10:15:00Z",
"duration_s": 372.5,
"sample_rate": 16000,
"channels": 1,
"codec": "wav_pcm_s16le",
"size_bytes": 11920000,
"sha256": "…",
"source": "device",
"sync_state": "local | uploaded | ingested | transcribed | summarised",
"transcript_ref": null,
"summary_ref": null
}
```
- On device: files on microSD (`/recordings/<id>.wav` + `<id>.json`); config + secrets in
NVS (never on the SD card in clear).
- In transit: audio + metadata JSON uploaded to the configured object store; server
ingests from there (or accepts direct upload).
- On server: audio + artefacts (transcript, summary, subtitle files) in the object store;
metadata + refs in the DB.
## External dependencies
- ESP32-S3 (Espressif), Arduino-ESP32, PlatformIO - mature, free, WiFi + BLE + PSRAM.
- faster-whisper (CTranslate2) - fast self-hosted STT, CPU or GPU.
- Ollama - self-hosted local LLM runtime for summaries.
- MinIO (or any S3-compatible / WebDAV target) - self-hosted object storage.
- FastAPI, Flutter - open, well supported.
All chosen to be self-hostable and open; no required proprietary SaaS.
## Constraints and trade-offs
- Audio default is WAV PCM 16 kHz mono for simplicity and quality; larger files, so WiFi
is the real transfer channel and Opus/ADPCM is a later size optimisation.
- No on-device transcription: keeps the device cheap/low-power; needs the server for AI.
- BLE bulk transfer is slow and iOS-restricted, so BLE only does control/provisioning and
hands transfers to WiFi.
- v1 uses off-the-shelf modules (no PCB): easier to build, bigger case than a Plaud.
- Security: device REST API and config writes must be authenticated (token in NVS);
independent uploads use scoped object-store credentials. Hardening tracked in TODO.

85
state/DECISIONS.md Normal file
View file

@ -0,0 +1,85 @@
# Decisions
> A dated, append only log of decisions and their rationale. Newest at the top. Never
> rewrite past entries; if a decision is reversed, add a new entry that says so.
## 2026-07-03 - Licensing: copyleft, multi-part (REUSE-style)
- **Decision:** Code (firmware, server, app) under GPL-3.0-only; hardware design under
CERN-OHL-S-2.0; case models and documentation under CC-BY-SA-4.0. Licence texts live in
`LICENSES/`; top-level `LICENSE` is GPL-3.0 for forge detection; `LICENSING.md` explains
the split. Apache-2.0 text kept for a possible future permissive client SDK.
- **Context:** User asked for "as open source as possible".
- **Rationale:** Strong copyleft keeps derivatives open (the point of the project); CERN
and CC-BY-SA are the standard reciprocal licences for hardware and creative/docs.
- **Consequences:** Derivatives must stay open. If we later want wide third-party adoption
of a client library, that specific component can be relicensed permissive (Apache-2.0).
## 2026-07-03 - Self-hosted tools throughout
- **Decision:** Forge = Forgejo (git.discworld.casa); CI = Forgejo Actions; STT =
faster-whisper; summaries = Ollama (local LLM); object storage = MinIO (S3-compatible)
or local FS / WebDAV. No required proprietary SaaS anywhere.
- **Context:** User: "using selfhosted tools where possible".
- **Rationale:** Matches the own-your-data goal and keeps running costs at zero beyond the
user's own hardware.
- **Consequences:** User must run a server (NAS / mini-PC) for AI features; the device and
app work without it for plain recording + transfer.
## 2026-07-03 - Self-hosted AI stack in scope for v1
- **Decision:** Build the full pipeline: record -> transcribe (faster-whisper) ->
summarise (Ollama) -> export. AI runs on the server, not the device.
- **Context:** User chose "Full self-hosted AI stack" at the scope checkpoint.
- **Rationale:** Transcription + summary is Plaud's headline feature; server-side keeps the
device cheap and low-power while staying fully self-hosted.
- **Consequences:** Larger build; server is required for AI features. Device stays simple.
## 2026-07-03 - Independent upload target: generic cloud storage
- **Decision:** When on charge / hard-powered, the device uploads to configurable generic
storage: S3-compatible (default: self-hosted MinIO), with WebDAV/NAS as alternatives.
- **Context:** User chose "Generic cloud storage" for the independent WiFi path.
- **Rationale:** Decouples device from any bespoke always-on server; standard protocol;
self-hostable.
- **Consequences:** Server ingests from the store (watch/notify/poll). Object-store
credentials live in device NVS and must be scoped/rotatable.
## 2026-07-03 - Mobile app: Flutter (Android + iOS)
- **Decision:** One Flutter codebase targeting both platforms.
- **Context:** User chose Flutter (Android + iOS).
- **Rationale:** Single codebase, both stores.
- **Consequences:** iOS background BLE is restricted, so BLE = control/provisioning only;
WiFi handles bulk transfer (already the design).
## 2026-07-03 - Hardware: ESP32-S3 + I2S MEMS mic + microSD (off-the-shelf, no PCB)
- **Decision:** Target ESP32-S3 (PSRAM, WiFi + BLE 5, USB). Mic: I2S MEMS (INMP441
default, ICS-43434 upgrade). Storage: microSD. Power: LiPo + charge IC with charge/VBUS
detect. v1 uses modules on a carrier/protoboard; no custom PCB. Full list in
`hardware/BOM.md`.
- **Context:** User asked me to spec a BOM to buy; has ESP32 / Pico W / other boards.
- **Rationale:** ESP32-S3 does all three radios + audio buffering on one cheap chip; I2S
MEMS gives clean digital audio; microSD removes length limits. Pico W and classic ESP32
are viable fallbacks but weaker for audio/PSRAM.
- **Consequences:** Bigger enclosure than a commercial Plaud; a custom PCB is a later step.
## 2026-07-03 - Firmware toolchain: PlatformIO + Arduino-ESP32
- **Decision:** Build the firmware with PlatformIO using the Arduino-ESP32 framework.
- **Context:** Need approachable, reproducible builds and CI.
- **Rationale:** Lower barrier for contributors than raw ESP-IDF; good library support for
I2S, SD, BLE, HTTP; PlatformIO gives pinned, CI-friendly builds.
- **Consequences:** If we hit Arduino limits (fine-grained power, advanced BLE), we can
drop to ESP-IDF per-module or migrate; noted as a possible future change.
## 2026-07-03 - Project name and default audio format
- **Decision:** Name = "OpenScribe" (record + transcribe, open). Default recording format
= WAV PCM 16 kHz mono 16-bit; compressed codecs (ADPCM/Opus) are a later optimisation.
- **Context:** Project bootstrap.
- **Rationale:** Clear, descriptive, unencumbered name; WAV is simple and high quality for
speech and trivial to decode everywhere.
- **Consequences:** Larger files (~115 MB/hour) make WiFi the primary transfer path;
revisit with Opus for battery-mode transfer and storage.

47
state/NOTES.md Normal file
View file

@ -0,0 +1,47 @@
# Notes
> Working notes, gotchas, environment quirks, and dead ends to avoid. Free form. The
> point is to save a future session from rediscovering something the hard way.
## How to run / build / test
Per component (see each component's README for detail):
- Firmware (`firmware/`): PlatformIO.
- `pio run` - build for the ESP32-S3 target.
- `pio run -t upload` - flash over USB.
- `pio device monitor` - serial console.
- Server (`server/`): Python 3.11+.
- `python -m venv .venv && . .venv/Scripts/activate` (Windows) or `.venv/bin/activate`.
- `pip install -r requirements.txt`
- `uvicorn app.main:app --reload` - dev server (OpenAPI at `/docs`).
- App (`app/`): Flutter.
- `flutter pub get` then `flutter run`. (Full Flutter project is created in M7.)
- Case (`case/`): OpenSCAD. Open `case/openscribe_case.scad`; render/export STL.
## Environment
- Forge: https://git.discworld.casa/laurence/openscribe (Forgejo). Git auth via Windows
Credential Manager (user `laurence`); the Forgejo API accepts it via basic auth. Never
print or commit the token.
- Dev host: Windows 11, PowerShell primary + Git Bash. Project root `c:\temp\dev\openscribe`.
- Self-hosted services for the server (run on a NAS / mini-PC, not required for plain
recording): MinIO (S3), Ollama (LLM), and faster-whisper (pip install, downloads models
on first run).
## Gotchas
- ESP32-S3 board choice matters: pick a variant WITH PSRAM (e.g. N16R8, or XIAO ESP32-S3)
for audio ring buffers. Classic ESP32 / Pico W are fallbacks with less headroom.
- Windows + PlatformIO: install the CP210x/CH34x USB serial driver or the board will not
enumerate a COM port.
- WAV at 16 kHz mono 16-bit is ~115 MB/hour; transfer over WiFi, not BLE. Opus is a later
size win but costs CPU on the S3.
- iOS restricts background BLE: use BLE only for control/provisioning; do bulk transfer
over WiFi.
- Do not put object-store credentials or WiFi passwords on the microSD in clear; they live
in ESP32 NVS.
## Dead ends
- (none yet)

65
state/PROJECT.md Normal file
View file

@ -0,0 +1,65 @@
# Project: OpenScribe
> The anchor document. A session that reads only this, TODO.md and DECISIONS.md should
> understand what the project is and what to do next. Keep it current.
## Objective
An open-source, self-hosted AI voice recorder in the style of the Plaud Note / NotePin:
a small wearable device that records audio to local storage, syncs to a mobile app and
to a self-hosted server, and produces transcripts and summaries, with a completely open
API so the owner controls their data end to end. No proprietary cloud, no lock-in.
## Scope
- In scope:
- Firmware for an ESP32-S3 recorder (audio capture, storage, power, controls).
- Three sync paths: BLE control, WiFi bulk transfer to the app, and independent WiFi
upload to cloud storage when on charge / hard-powered (no phone present).
- A completely open, documented REST API (device and server) with an OpenAPI spec.
- A self-hosted AI stack: transcription (faster-whisper) and summarisation (local LLM
via Ollama), plus export in multiple formats.
- A Flutter mobile app (Android + iOS).
- A parametric 3D-printed case (OpenSCAD) plus a hardware BOM and wiring guide.
- Out of scope (for now):
- A custom PCB (v1 uses off-the-shelf modules on protoboard / carrier).
- Cloud SaaS hosting. Everything runs on hardware the user owns.
- Real-time on-device transcription (server does the AI; device just records + syncs).
- Speaker diarisation (deferred; noted as a later enhancement).
## Audience
Makers and privacy-minded users who want a Plaud-like capture-and-summarise workflow
they fully own: build the device from the BOM, print the case, run the server on a NAS
or mini-PC, install the app, keep every recording and transcript on their own kit.
## Description
The device records voice to microSD as WAV (compressed codecs optional later). On
battery it advertises over BLE for control and hands bulk transfers to WiFi. When placed
on charge or hard-powered it becomes autonomous: it joins configured WiFi, serves its
REST API on the LAN, and uploads new recordings to generic cloud storage (S3-compatible,
e.g. self-hosted MinIO, or WebDAV/NAS). A self-hosted server ingests recordings,
transcribes them with faster-whisper, summarises with a local LLM, and exposes an open
API with exports (audio, TXT, SRT/VTT, Markdown, JSON). The Flutter app provisions the
device, browses the library, plays audio and shows transcripts and summaries.
## Success criteria
- A person can build the device from `hardware/BOM.md`, flash `firmware/`, print
`case/`, run `server/`, install `app/`, and capture -> transcribe -> summarise -> export
a recording without any proprietary service.
- Every recording is retrievable and exportable through the open API.
- The device syncs three ways as specified (BLE, WiFi-to-app, independent WiFi upload).
- The whole stack is self-hostable and licensed for open reuse.
## Key facts
- Trunk branch: `main`
- Forge / remote: https://git.discworld.casa/laurence/openscribe
- Runtime / stack:
- Firmware: C++ (Arduino-ESP32) via PlatformIO, target ESP32-S3.
- Server: Python (FastAPI) + faster-whisper + Ollama + object storage (MinIO/local).
- App: Flutter (Dart), Android + iOS.
- Case: OpenSCAD. CI: Forgejo Actions (self-hosted runner).
- How to run it: see `state/NOTES.md` (per-component build/run commands).

35
state/PR_TEMPLATE.md Normal file
View file

@ -0,0 +1,35 @@
# <feature title>
> Fill this in when opening the PR. It is the summary a later documentation session
> reads to write the docs, so make it complete and self contained.
## Feature
<What was built, in plain terms.>
## What was achieved
<The outcome. What now works that did not before.>
## How to verify
<Steps or commands to confirm it works.>
## Tools used
<Languages, libraries, commands, services involved.>
## How it works
<Enough detail for a documentation session to start from this PR alone: the approach,
the key files, and how the pieces connect.>
## State updated
- [ ] `state/TODO.md`
- [ ] `state/DECISIONS.md` (if a decision was made)
- [ ] `state/ARCHITECTURE.md` (if the structure changed)
## Follow ups
<Anything deferred or worth doing next.>

45
state/TODO.md Normal file
View file

@ -0,0 +1,45 @@
# TODO
> The current state of play. Keep it honest and current; this is what the next session
> reads to know what to do. Move items between sections as they progress.
## In progress
- [ ] M0 Scaffold - repo, licences, state docs, BOM, OpenAPI, component skeletons, CI
(branch: bootstrap on `main`; this is the setup commit)
## Pending (roughly in build order)
- [ ] M1 Firmware recording core: I2S mic capture -> PSRAM ring buffer -> WAV on microSD,
button start/stop, LED status, sidecar JSON metadata. (branch `feature/fw-recorder-core`)
- [ ] M2 On-device WiFi + REST API: WiFi manager, mDNS, list/get/download(range)/delete
recordings, device status. Implements `api/openapi.yaml` device paths.
(branch `feature/fw-rest-api`)
- [ ] M3 Independent uploader: NVS config store, charge/VBUS detect -> powered mode,
S3-compatible + WebDAV upload of audio + metadata. (branch `feature/fw-uploader`)
- [ ] M4 BLE GATT: device info, battery, record control, WiFi provisioning, status
notifications; hand-off to WiFi for transfer. (branch `feature/fw-ble`)
- [ ] M5 Server ingest + transcription: FastAPI ingest from object store/upload, store,
faster-whisper transcription, DB, exports (audio/TXT/SRT/VTT/JSON). REST API.
(branch `feature/server-ingest-stt`)
- [ ] M6 Server summaries: Ollama LLM summary + action items, Markdown export.
(branch `feature/server-summary`)
- [ ] M7 Flutter app: BLE provisioning, device dashboard, library from server API,
playback, transcript/summary view, export/share, settings.
(branch `feature/app-mvp`)
- [ ] M8 3D case: parametric OpenSCAD enclosure fit to the BOM board + battery; print/fit.
(branch `feature/case-v1`)
- [ ] M9 Hardening + OTA: API auth, credential handling, OTA update, docs handoff.
(branch `feature/hardening-ota`)
## Done
- [x] Create Forgejo repo `laurence/openscribe` (public, main) - 2026-07-03
- [x] Decide hardware, sync model, AI stack, app platform, storage, licensing (see
DECISIONS.md) - 2026-07-03
## Blocked / waiting
- [ ] Physical build + on-device verification - waiting on parts from `hardware/BOM.md`.
Until parts arrive, firmware is developed against the ESP32-S3 target and validated
by build + (where possible) native/host unit tests, not on real hardware.