Context
Omni was my graduation project at Hypersolid. It started from a practical problem: decisions and follow-up actions get lost because manual note-taking competes with active participation. Omni explores how an AI system can support meetings as a quiet second layer of information rather than as a distracting chatbot, without meeting data leaving the company network.
What I built
I engineered Omni's core components and owned the transcription pipeline, meeting-data orchestration, search and dashboard. Audio moves through one on-premise pipeline: audio intake → transcription → speaker labelling → PocketBase → LLM tasks → React interface.
- Transcription. Whisper large-v3-turbo transcribes segmented meeting audio with timestamps, and hallucination filters guard its output.
- Speakers. pyannote diarization and opt-in voice samples put names on segments, with a safe "Unknown speaker" fallback when confidence or mapping is insufficient.
- Coordination. Five Deno and Python services coordinate through PocketBase, which stores the data, pushes realtime updates to the interface and acts as the control plane for processing tasks. The services ship with Docker and Azure DevOps.
- LLM work. Summaries, action points, Q&A, incremental D3 mind maps and search across meetings run in the "Workhorse", a separate Ollama process on the same workstation. It is isolated from live transcription, so the meeting keeps going when LLM work falls behind. JSON output contracts keep its results predictable enough to render, audit and review.
- Interface. A fixed 2x2 layout with status feedback, so participants can scan it without interrupting the meeting.
- Evaluation and governance. I benchmarked latency and word error rate, compared Gemma, Phi and DeepSeek, tested the interface in realistic meeting sessions, and documented the governance it needs: opt-in voice samples, raw-audio minimization, a retention policy, role-based access and human correction.
Decisions and trade-offs
- PocketBase as the integrated backend coreover PostgreSQL with a separate API and message brokerFor an innovation prototype, fast iteration and low operational overhead mattered more than best-of-breed components, as long as auth, realtime updates and on-premise hosting were covered. Cloud-first options such as Firebase and Supabase were ruled out because meeting data would leave the internal network.cost · A broker-driven microservices setup scales better in the long run; this is a deliberate prototype simplification.
- A pull-based task queue for LLM workover synchronous request/response calls to the modelTranscription, speaker labeling and LLM inference share a single RTX 4090, so heavy LLM tasks would otherwise push up request latency and stall the interface. Queued tasks create natural backpressure while transcription keeps running; predictability mattered more than an always-immediate response.cost · LLM output can arrive later during workload peaks.
- Gemma as the default local LLMover Phi and DeepSeekThe interface parses the model's JSON, so schema reliability outweighed prose quality. In the evaluation Gemma followed the JSON schema 100% of the time, against 60% for Phi and 87% for DeepSeek.cost · Phi was faster (2.9s vs 3.8s average task latency) and DeepSeek scored higher on task quality.
- Speaker identification from opt-in voice samplesover plain diarization with generic speaker labelsNamed labels make action items assignable during a live meeting; generic labels are less useful for ownership and interpretation.cost · Voice samples are personal data, so they need opt-in, a removal path and stricter governance.
Results
| Quantity | typ | Unit |
|---|---|---|
| transcript latency | 0.93 | s |
| speaker-label latency | 1.12 | s |
| ASR word error rate | 9.1 | % |
| known-speaker identification | 87 | % |
| LLM schema adherence | 100 | % |
The transcription, latency and speaker figures come from 8 evaluation sessions with about 100 minutes of audio and 5 speakers. The model comparison behind the schema-adherence figure used a fixed meeting transcript.
- Kept speech-to-screen latency within the sub-2 s near-real-time design target, with transcription and speaker labelling running in parallel.
- Measured an average ASR word error rate of 9.1% over all 8 evaluation sessions (weighted; about 9 wrong, missing or extra words per 100 reference words): 8.3% in the 6 Dutch-dominant sessions and 11.1% in the 2 mixed Dutch/English sessions, where switching between the languages caused the increase.
- Named known speakers correctly in 87% of evaluated segments from people with a registered voice sample; 8% got a wrong name and 5% were shown as "Unknown speaker". People without a voice sample were shown as "Unknown speaker" every time. Overlapping speech was excluded from this evaluation.
- UX evaluation with five employees across the eight sessions found the interface non-intrusive, with a 4.2/5 average score and no disruptive UI behaviors observed.
- Identified clear next steps for production readiness: domain-specific glossary, better speaker correction, formal retention policy, role-based access control, and operational hardening.
What I learned
Omni taught me that AI product quality is not only about model quality. The most important engineering work was making the system reliable around the model: clear data contracts, safe fallbacks, transparent system status, auditability, and user control. For a workplace AI assistant, trust depends as much on what the system refuses to guess as on what it can generate.