F.R.I.D.A.Y
A voice-first desktop assistant that asks before it acts, and checks its own work.
- Role
- Agentic AI Engineer
- Timeline
- Apr 2026 – Present
- Team
- Solo
- Status
- Actively maintained
The problem
Most voice assistants answer questions but cannot act, and agents that do act often run whatever the model decides. In earlier versions of FRIDAY a single ambiguous command, “shut down the PC”, fired instantly and unconfirmed. An assistant that controls a real machine has to hold risky actions, prove they worked, survive a cloud outage, and keep its memory from becoming a liability.
What it does
Sentinel confirmation gate
Risky actions never run on the first ask. They are held, the user is told exactly what would happen, and they only proceed on an explicit “yes” on the next turn.
Independent Verifier
After a tool reports success, the Verifier re-checks real state (a running process, the volume level, a file on disk) before success is spoken. Results are VERIFIED, FAILED or INCONCLUSIVE, never forced.
Append-only audit trail
Every risky action that was held, what you answered (confirmed or cancelled) and what actually ran is logged as one JSON line. Message bodies, file contents and keys are stored as a length only, so the trail answers “what did FRIDAY do on my PC?” without becoming a leak.
Model router
Fast, strong, local and auto modes sit in front of NVIDIA NIM, Gemini, Ollama and LiteLLM endpoints. Local mode deliberately never escalates to the cloud.
Encrypted memory + Obsidian
Long-term memory is encrypted at rest with FAISS semantic recall, and syncs to a plain-markdown Obsidian vault you can read and edit. FRIDAY only rewrites its own marked block, so hand edits survive.
Full voice pipeline
WebRTC VAD, faster-whisper speech-to-text, ElevenLabs or edge-tts out, and openwakeword for hands-free mode, inside an Electron + React interface.
Deep Spotify control
Devices and play-on by fuzzy name, queue, shuffle, repeat, seek, per-device volume and playlist management, driven by voice.
Emotion & sentiment engine
Fuses text, voice and facial-expression signals into one smoothed user state, plus an appraisal model (mood, drives, memory) that shapes FRIDAY's tone and expressions.
3D face, Hindi/Hinglish speech
A lip-synced 3D face (three.js, 3 avatars) and Hindi/Hinglish speech, with control of Gmail, Calendar, Spotify and your phone or laptop via FCM.
Coding-agent orchestration
Drives 10 coding agents (Claude Code, Codex, Gemini CLI, Hermes) and a multi-specialist team mode (plan, parallel work, review, merge) that share a pre-scanned SKILL.md library.
Reverse engineering + self-healing
Recovers and rebuilds Electron, .NET and Android apps as clean projects (REA, jadx, ILSpy), and prepares tested fixes for its own errors.
How it flows
- Voice in
- Intent routing
- LLM tool call
- Sentinel gate
- Action
- Verifier
- Audit log
- Spoken result
Problems solved
An agent that ran everything it was told to
ProblemAcross earlier versions every LLM-decided action executed directly, including one ambiguous command that powered off the PC.
SolutionSentinel is a code-enforced gate, not a prompt instruction. Risky actions are held and need confirmation on the next turn.
Trusting a tool’s own “done”
ProblemA handler returning success does not mean the thing actually happened.
SolutionThe Verifier checks real state independently. If a check itself breaks it reports INCONCLUSIVE, so a bug in verification can never masquerade as a failed action.
Providers go down, and privacy matters
ProblemOne cloud model is a single point of failure, but silently falling back to the cloud defeats choosing local on purpose.
SolutionA tiered router with configurable chains. Local-only stays local-only unless you change the chain yourself.
Memory a human can inspect
ProblemOpaque memory blobs are hard to trust or fix.
SolutionMemory is written into an Obsidian-compatible vault using managed blocks, so it is browsable, editable, and never overwrites hand-written notes.
“What did it do on my PC?”
ProblemOnce the terminal is closed, nothing explains which risky actions were held, approved or actually run.
SolutionAn append-only audit trail records the hold, your answer and the outcome. Arguments are summarised and secrets are kept as a length only, and a logging failure can never break the action itself.