laurence/openscribe
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
openscribe/state/TODO.md
Laurence 51321aa7c5
Some checks failed
ci / firmware (pull_request) Failing after 1s
Details
ci / emulator (pull_request) Failing after 27s
Details
ci / openapi (pull_request) Failing after 33s
Details
ci / server (pull_request) Failing after 37s
Details
feat(server): pluggable AI providers - any open-standard or commercial AI 
Lets the owner point transcription and summarisation at any AI: an open-standard
endpoint (OpenAI-compatible / local faster-whisper / Ollama) or a commercial API
(OpenAI, Anthropic, Gemini). Config-driven, self-hostable, no lock-in.

What changed:
- server/app/providers/: provider layer.
  - base.py: Transcriber/Summariser protocols + shared summary prompt + tolerant JSON
    parser (uniform Summary shape across providers).
  - summary.py: OpenAICompatibleSummariser (any /chat/completions - OpenAI, Groq,
    OpenRouter, LocalAI, LM Studio, vLLM, Ollama /v1) and AnthropicSummariser (Claude
    via the official anthropic SDK; Messages API has no OpenAI-compatible endpoint).
  - transcription.py: OpenAICompatibleTranscriber (/audio/transcriptions - OpenAI,
    Groq, self-hosted whisper server) and LocalWhisperTranscriber (faster-whisper,
    execution wired in M5).
  - factory.py: builds the configured providers with per-provider defaults
    (anthropic -> claude-opus-4-8, openai_compatible -> gpt-4o-mini, ollama -> llama3.1).
- config.py + .env.example: transcription_provider / llm_provider selectors + base_url,
  key, model settings; local faster-whisper and Ollama kept as the self-hosted defaults.
- main.py: /health now reports the resolved provider names (no secrets).
- requirements.txt: httpx drives all HTTP providers; anthropic + faster-whisper are
  optional, only for their respective providers.
- docs/ai-providers.md: config recipes for OpenAI, Groq, Anthropic, Gemini, LocalAI,
  LM Studio, Ollama, self-hosted whisper.
- state/: DECISIONS, ARCHITECTURE, TODO updated.

Why:
- The user asked to connect the device to any open standard AI or commercial one; this
  is also the core differentiator vs Plaud's locked cloud.

Notes:
- Anthropic provider uses the official SDK and defaults to claude-opus-4-8 (per the
  claude-api guidance). AI deps are optional per chosen provider. Modules byte-compile
  cleanly; end-to-end wiring into the ingest pipeline lands with M5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 18:56:58 +01:00

4.5 KiB
Raw Blame History

TODO

The current state of play. Keep it honest and current; this is what the next session reads to know what to do. Move items between sections as they progress.

In progress

  • (nothing active - M3 is next)

Pending (roughly in build order)

  • M3 Independent uploader: charge/VBUS detect -> powered mode, S3-compatible + WebDAV upload of audio + metadata (config store already landed in M2). (branch feature/fw-uploader)
  • M4 BLE GATT: device info, battery, record control, WiFi provisioning, status notifications; hand-off to WiFi for transfer. (branch feature/fw-ble)
  • M5 Server ingest + transcription: FastAPI ingest from object store/upload, store, faster-whisper transcription, DB, exports (audio/TXT/SRT/VTT/JSON). REST API. (branch feature/server-ingest-stt)
  • M6 Server summaries: Ollama LLM summary + action items, Markdown export. (branch feature/server-summary)
  • M7 Flutter app: BLE provisioning, device dashboard, library from server API, playback, transcript/summary view, export/share, settings. (branch feature/app-mvp)
  • M8 3D case: parametric OpenSCAD enclosure fit to the BOM board + battery; print/fit. (branch feature/case-v1)
  • M9 Hardening + OTA: API auth, credential handling, OTA update, docs handoff. (branch feature/hardening-ota)

Pending - AI provider flexibility

  • Pluggable AI providers (server): transcription + LLM target any open-standard endpoint (OpenAI-compatible / local / Ollama) OR commercial (OpenAI, Anthropic, Gemini). Config-driven, no lock-in. Provider layer + HTTP + Anthropic providers done (PR #5). Local faster-whisper execution + calling the providers from the ingest pipeline complete in M5. See docs/ai-providers.md.

Pending - Plaud-derived backlog (see docs/plaud-comparison.md)

  • Speaker diarisation + labels (server); rename-speaker in the app.
  • Summary templating: built-in templates (meeting/action-items/to-do/mind-map/ decisions) + user templates; role-based multidimensional summaries.
  • Ask AI: chat grounded in a recording's transcript (RAG), via the chosen LLM provider.
  • Custom vocabulary / prompt hints for transcription + summary.
  • Multimodal attachments: text notes + images attached to a recording.
  • Mark-a-moment: device long-press during recording -> highlight timestamp, surfaced in the app and used to focus summaries.
  • Auto title + auto formatting of transcripts and summaries.
  • Library organisation: tags, folders, full-text search (server + app).
  • Share links for summaries (extends the export endpoint).
  • Optional hardware variant with a small OLED status display (InstantView equivalent).
  • Calendar integration (nice-to-have).

Done

  • Dev testing + emulation harness - native unit tests (wav/range), Wokwi emulator, cppcheck + OpenAPI validation in CI, docs/testing.md (PR #3, 2026-07-03).
  • M2 On-device WiFi + REST API - WiFi/SoftAP + mDNS + config(NVS) + device REST API (status, list, metadata, audio-with-Range, delete, record start/stop, config). (PR #2 feature/fw-rest-api, 2026-07-03). Not yet hardware-verified.
  • M1 Firmware recording core - mic capture to WAV on SD + button/LED + metadata (merged: PR #1 feature/fw-recorder-core, 2026-07-03). Not yet hardware-verified.
  • M0 Scaffold - repo, licences, state docs, BOM, OpenAPI, skeletons, CI (merged to main, 2026-07-03)
  • Create Forgejo repo laurence/openscribe (public, main) - 2026-07-03
  • Decide hardware, sync model, AI stack, app platform, storage, licensing (see DECISIONS.md) - 2026-07-03

Follow-ups (deferred, do when convenient)

  • Audio calibration: INMP441 emits 24-bit samples in a 32-bit slot. M1 reads in 16-bit mode (usable for speech, may be quiet). If levels are low on the dev build, capture 32-bit and right-shift into int16 in audio.cpp.
  • Real timestamps: ids/started_at are uptime-based (no RTC/NTP yet). Add NTP now that WiFi exists (M2), and switch ids and started_at to real UTC.
  • API auth is open when no token set; SoftAP has a fixed default passphrase. Harden both before any deployment (see NOTES "Security follow-ups").

Blocked / waiting

  • Physical build + on-device verification - waiting on parts from hardware/BOM.md. Until parts arrive, firmware is validated by CI build (Forgejo Actions) of both board profiles, not on real hardware.