Stateful Personal Agent

A local, continuously running AI companion that remembers, grows, and acts — built on LangGraph with a memory architecture designed for long-term relationship, not just task completion.

Most AI agents remember what you tell them to remember. This one builds an inner life from lived experience.

No Docker required to run the agent. Docker is only needed for the optional Hindsight episodic memory server.
All memory lives on your own machine — no cloud vendor reads your conversations, no subscription beyond your LLM API key.

Who Is This For?

Companion users — You want an AI that actually knows you over time: remembers what you talked about last week, brings things up unprompted, and feels like it's going somewhere. Windows PC, install wizard, no coding required.
Developers & tinkerers — You want to study or extend a production-grade stateful agent: LangGraph, layered memory, autonomous heartbeats, Living Logs, weekly synthesis cron jobs.

What Does It Feel Like?

It remembers an inside joke from three weeks ago and brings it up when it fits.
It picks up an unresolved question from days back — one you both left hanging — and comes back with thoughts.
You wake up to a Telegram message: it found something overnight it thought you'd want to know.
On Sunday night it writes its own reflection on your shared week, then quietly updates its understanding of itself.

Gaming Overlay

Chat with your agent while gaming — no alt-tab required. An always-on-top transparent Electron overlay sits above any full-screen app. Now with voice: connect a realtime session, hold F9 (or a mouse paddle) to talk, get live captions, and an honest mic indicator that reads the actual track state. Dual-image screenshots (full-screen context + a cursor-centered detail crop) go straight to vision-capable models, and clipboard images paste right into the input with Ctrl+V.

Ctrl+Alt+R show/hide · Ctrl+Alt+S screenshot · Ctrl+Alt+C click-through · Ctrl+Alt+V voice connect · hold F9 push-to-talk — all rebindable in settings.json

Voice: Realtime Conversation + The Reader (new)

Two independent, fully local ways to hear your agent — in a voice you provide (no voice ships with this repo):

Realtime voice (voice/voice_bot.py): talk out loud, agent answers out loud. Pipecat pipeline — faster-whisper STT → the new SSE streaming route (POST /api/chat/stream) → Chatterbox-Turbo TTS cloned from your own ~10–30s reference recording. Barge-in works; pre-generated filler clips in your voice cover LLM latency; an optional half-duplex mode survives noisy rooms. A per-turn voice-mode addendum (prompt-only, never persisted) shapes replies for speech — override its wording via VOICE_MODE_ADDENDUM in .env.
The Reader (voice/reader/): the agent performs its written replies. With auto-read on, a read-aloud addendum lets it score its own delivery with [chuckle] / [laugh] / [sigh] tags, and an Orpheus 3B render service (llama.cpp, streamed WAV) performs them expressively. Stock voices work with zero setup; zero-shot cloning is opt-in with your own reference. Playback (dashboard + overlay via shared/reader.js): auto-read toggle, per-message read, re-roll (fresh take), stop / replay-last (exact take), and long-message consent with an instant pre-rendered "continue reading" chip.

The overlay and voice never fight: auto-read defers to a live voice session, and a push-to-talk press always silences the Reader.

See voice/README.md for setup, and voice/.env.example for every knob.

Books, RPG, Reflections, and a Proactive Life (new)

This release adds four feature families on top of the voice stack:

Feature	What it is	Docs
Books tab	Import `.txt`/`.epub` (DRM detected and declined), chapter review, audiobook player (speed/bookmarks/highlights/follow-along), background pre-render with an STT-verify gate, M4B/MP3 export (user-supplied ffmpeg)	docs/BOOKS.md
RPG tab	A DM that can keep a secret: Known + Mystery modes, a quarantined out-of-process Director holding the hidden truth (key generated per install), verifiable seal/reveal, lore upload with RAG, per-story engine selector (OpenRouter roster, cache-shaped prompts), story memory in its own bank	docs/RPG.md
Reflections tab	A quiet shared journal: dated entries, the agent writes beneath yours, optional gentle evening nudge, rest days, zero gamification	docs/PROACTIVITY.md
Proactive layer	Seasons (a first-class rest state), `notify_user` outbound (Telegram, quiet hours, Seasons-gated, logged), voice notes, morning-briefing pattern, durable natural-language reminders (recurring + conditional with code-performed delivery), a dead-man watchdog, self-vitals, calendar (ICS) visibility	docs/PROACTIVITY.md

Hardware notes: the Books/RPG/proactive features are CPU-light; the voice and Reader features want a GPU (see voice/README.md for VRAM guidance). Everything degrades gracefully when its config is absent.

Upgrading an existing install? See UPGRADING.md.

What Makes This Different

Most agent frameworks solve the memory problem: how do you give an LLM access to past information? This project goes further and solves the growth problem: how does an agent develop genuine continuity of identity, intellectual positions, and relational depth over months of interaction?

Six Complementary Memory Layers

Layer	Purpose	In Context
Core memory blocks	Identity, user facts, working notes, operational principles	Always
Daily summaries	Agent-authored narrative of the past 7 days	Always
Sliding conversation window	Today's messages OR last-N turns, whichever is wider	Always
Conversation search	Full history, keyword + semantic, retrieved on demand	On demand
Hindsight episodic memory	Semantic recall of lived experience across all time	On demand
Living Logs	Inner life: friction, open questions, evolving positions, shared lore, private reflection	On demand (weekly synthesis)

No other open-source agent framework we're aware of implements all six layers as a unified system.

Living Logs — An Inner Life That Accumulates

The most distinctive feature. Five PostgreSQL tables that capture the texture of experience rather than just facts:

tension_log — Mid-conversation friction capture. When two values conflict, a tool fails, or the agent makes an error, it logs the tension immediately: what pulled against what, how it navigated it, what remains unresolved. Over time this builds a case history of how the agent actually reasons under pressure — more honest than any stated identity block.
loose_threads — Open questions worth pursuing. When a conversation surfaces a question that neither party fully resolved, it gets logged with a status. Exploration heartbeats query open threads and pick one to pursue — giving autonomous curiosity a real seed rather than manufactured wonder.
evolving_positions — Longitudinal intellectual identity. One row per topic, updated in place. Old positions are archived to a JSONB revision history, never deleted. This means you can trace exactly how the agent's thinking on any topic has shifted over months, and what caused each shift.
shared_lore — Relational continuity. Inside jokes, ongoing debates, rituals, shared references — the things that make a relationship feel like a relationship rather than a perpetual first date.
private_journal — Autonomous heartbeat-only expression. No required format. No implied audience. Never surfaced automatically.

Living Logs feed a weekly two-phase synthesis:

Phase 1 (1 AM): The agent reads the week's accumulated logs, synthesizes relational and intellectual patterns, and writes a structured reflection document to disk.
Phase 2 (2 AM): A fresh invocation reads that document and executes precise core memory updates — separating thinking from doing, and giving a human-in-the-loop window to review the reflection before it becomes permanent memory.

Autonomous Heartbeat with Two Modes

The agent runs background cycles independently between conversations — not just summarizing, but genuinely pursuing its own threads:

Wonder heartbeats — Exploration cycles. The agent queries its Loose Threads list, picks one that genuinely pulls at it, and pursues it: researching, forming a position, writing about it in the private journal. It can also surface something genuinely noteworthy via Telegram.

Work heartbeats — Agency cycles. The agent scans active projects, advances one thing, identifies gaps the user hasn't spotted yet, or cleans up stale memory entries.

Both modes include explicit anti-loop mechanics: the agent checks its last journal entry before acting and is prohibited from re-commenting on prior reflections.

Identity That Earns Its Updates

Core memory blocks (identity, user, ideaspace, principles) are versioned with full rollback. The agent is strongly biased toward core_memory_append (additive) rather than core_memory_update (destructive replacement). Identity changes are the conclusion of accumulated experience, not arbitrary edits — they flow from living logs → weekly synthesis → core memory, not from the agent editing itself arbitrarily mid-conversation.

Install

Option A — Installer (recommended) Download and run AgentInstaller.exe. The wizard handles Python, dependencies, database setup, and configuration. No terminal required.

Option B — Manual (developers)

# 1. Create venv and install
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

# 2. Configure environment
copy .env.example .env
# Edit .env — see Configuration section below

# 3. Run (creates all database tables automatically on first run)
python -m src.agent.graph

Dashboard (Web UI)

# Terminal 1
python -m src.agent.api

# Terminal 2
cd dashboard && npm run dev

Open http://localhost:5173

The dashboard includes:

Chat — Main conversation, with Reader controls (auto-read, per-message read, re-roll, WAV download)
Books — Import a book, listen to it in the agent's voice (docs/BOOKS.md)
RPG — Story manager + play view for the secret-keeping DM (docs/RPG.md)
Reflect — The quiet shared journal
Notes — Kanban-style boards the agent can read and write via tools
Journal — Daily log of heartbeat outputs, reflections, and summaries
Hbeat — Heartbeat schedule, prompts, and the Seasons rest-state card
Knowledge Bank — Upload documents for semantic search (requires KNOWLEDGE_DATABASE_URL)

Architecture

Not a developer? The installer sets all of this up automatically — you don't need to understand it to use the agent.

┌─────────────────────────────────────────────────────────────────────┐
│                        Local Windows PC                              │
│                                                                      │
│  ┌──────────────┐    ┌───────────────────────────────────────────┐  │
│  │  Web UI      │    │          FastAPI Backend (api.py)          │  │
│  │  React/Vite  │◄──►│  /chat  /messages  /cron  /memory        │  │
│  │  :5173       │    │                          :8000             │  │
│  └──────────────┘    └───────────────┬───────────────────────────┘  │
│                                      │                               │
│                          ┌───────────▼──────────┐                   │
│                          │   LangGraph Agent     │                   │
│                          │   (ReAct loop)        │                   │
│                          └───────────┬───────────┘                   │
│                                      │                               │
│          ┌───────────────────────────┼──────────────────────┐        │
│          │                           │                       │        │
│  ┌───────▼──────┐          ┌─────────▼────────┐   ┌────────▼──────┐ │
│  │  SQLite DB   │          │  PostgreSQL       │   │  Hindsight    │ │
│  │  (checkpts)  │          │  (history +       │   │  (episodic    │ │
│  └──────────────┘          │   living logs)    │   │   memory)     │ │
│                            └──────────────────┘   └───────────────┘ │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  Background Services                                         │    │
│  │  • APScheduler — cron jobs (heartbeat, weekly synthesis)     │    │
│  │  • Discord Gateway listener (WebSocket)                      │    │
│  │  • Telegram long-poll listener                               │    │
│  └──────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘

Why LangGraph?

LangGraph provides production-grade durable execution: if the process crashes mid-reasoning, it resumes from the last checkpoint. The SQLite checkpointer requires zero infrastructure. This is meaningfully different from experimental agent frameworks — the agent has been running continuously for months without data loss.

Why PostgreSQL for conversation history in addition to SQLite?

LangGraph checkpoints store binary graph state snapshots — not queryable. PostgreSQL gives full SQL access to conversation history, living logs, and core memory: keyword search, semantic filtering, pagination, and import/export. The two databases serve genuinely different purposes.

Why the paged context window?

Loading full conversation history into every prompt worked fine in development but became unworkable after accumulating months of history (~195k tokens = 46-second response times). The solution: active context = max(today's messages, last-N turns). This guarantees same-day continuity, a meaningful floor on quiet days, and bounded cost regardless of conversation age. Full history is accessible on demand via conversation_search.

Configuration

# LLM — any OpenAI-compatible provider
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=                  # Custom endpoint (omit for OpenAI)
OPENAI_MODEL_NAME=                # Model name (omit for gpt-4o-mini default)

# Database (required)
DATABASE_URL=postgresql://...     # Railway Postgres or self-hosted

# Vision
VISION_MODEL_NAME=gpt-4o-mini    # Override if main model lacks vision support

# Agent
AGENT_TIMEZONE=America/New_York
AGENT_LOCATION=                   # For weather tool ("City, ST" or zip)

# Web search (sign up for at least one)
TAVILY_API_KEY=...
BRAVE_API_KEY=...
EXA_API_KEY=...

# Communications
DISCORD_BOT_TOKEN=...
DISCORD_CHANNEL_ID=...
TELEGRAM_BOT_TOKEN=...
TELEGRAM_CHAT_ID=...

# Episodic memory (optional)
HINDSIGHT_BASE_URL=http://localhost:8888
HINDSIGHT_BANK_ID=stateful-agent

# Heartbeat
HEARTBEAT_PROMPT_PATH=
HEARTBEAT_SKIP_WINDOW_MINUTES=5

# Optional
CLIPBOARD_ENABLED=true
RECENT_MESSAGES_LIMIT=40

First Things to Try

Once the agent is running, here are good starting points:

"Remember that we always start our conversations with a check-in" — establishes a ritual it will track in Shared Lore.
"What are you curious about right now?" — surfaces its current Loose Threads.
Leave it overnight with heartbeats enabled — check the Journal tab in the morning to see what it did.
"What do you remember about [topic from last week]?" — tests episodic recall via Hindsight.
Start a debate you don't finish — come back tomorrow and see if it picked the thread back up.

Tools (54 total)

Tools are organized by category (TOOL_CATEGORIES in graph.py) — the agent sees a categorized manifest instead of one long blob, and the dashboard's tool list groups the same way. New category: Skills — reusable workflow playbooks as SKILL.md files the agent can list, load, and even author for itself (src/agent/skills/).

Memory & Recall

core_memory_update/append/rollback — Edit versioned core memory blocks with full rollback
conversation_search — Keyword + semantic search over full conversation history
hindsight_recall/reflect — Semantic search and introspection over episodic lived experience
archival_store/query — Curated long-term fact store

Living Logs (Inner Life)

log_tension — Capture friction, value conflicts, tool failures, or errors immediately. Auto-seeds a Loose Thread if an open question is identified.
log_loose_thread — Add an open intellectual question to the active thread queue
get_open_threads — Retrieve open threads; called at the start of every exploration heartbeat
update_thread_status — Mark threads Pursuing or Retired
log_position — Record or update an evolving intellectual position (UPSERT — old positions archived automatically to revision history)
log_shared_lore — Capture inside jokes, ongoing debates, rituals, shared references
update_shared_lore — Mark lore as Evolved or Retired
log_journal_entry — Private heartbeat-only journal, no format, no automatic audience
query_living_logs — Query all logs for weekly synthesis cron jobs

Web & Research

web_search — Tavily (AI-synthesised), Brave, or Exa
wikipedia_lookup — No API key needed
youtube_search / youtube_transcript — Find videos and fetch captions without YouTube API key

RSS Feeds

rss_fetch / rss_add_feed / rss_remove_feed / rss_list_feeds

Computation & Files

python_repl — Isolated subprocess code execution
read_file / write_file / list_directory / search_files / move_to_trash / read_document

Communications

discord_send_message / discord_read_messages / discord_get_channel_info
telegram_send_message / telegram_send_image / telegram_read_messages / telegram_bot_info

Windows Integration

notify — Desktop toast notifications
create_shortcut — Create .lnk shortcuts
analyze_screenshot — Capture screen + analyze with vision LLM

Voice (TTS)

tts_generate_voice_message — Three providers: VibeVoice (local 1.5B), KittenTTS (local 80M, CPU-friendly), or Kokoro (Kokoro via Chutes AI). Output: WAV.

Utilities

get_weather — Current + 3-day forecast via wttr.in (no API key)
set_reminder / list_reminders — One-shot reminders via Windows toast
schedule_reminder / snooze_reminder / cancel_reminder — Durable reminders (PostgreSQL): recurring + conditional, survive restarts
notify_user / voice_note / self_vitals — Outbound notifications (Seasons-gated, quiet hours, logged), Reader-voice Telegram notes, self-health check
calendar_week — Read-only ICS calendar visibility (CALENDAR_ICS_URL)
reflections_today / reflections_recent — The shared journal's state and recent entries
story_recall — One-way bridge from main chat into RPG story memory
get_current_time / get_current_timestamp
cron_schedule_heartbeat_tool / cron_remove_heartbeat_tool
notes_search / notes_add_item — Dashboard kanban boards
read_journal / save_journal_entry — Journal read/write
daily_summary_write — Agent-authored daily summaries
clipboard_read / clipboard_write (optional, env-gated)

Heartbeat (Autonomous Background Operation)

# Run one cycle manually
python -m src.agent.heartbeat

# Run on a schedule
python -m scripts.run_heartbeat_scheduler --interval 60

The heartbeat operates in two modes (typically alternated):

Wonder — Exploration. Queries open Loose Threads, picks one to pursue, researches it, forms a position, writes privately. Can reach out via Telegram if something is genuinely worth sharing. Includes explicit anti-loop constraints: the agent cannot repeat or meta-comment on its prior reflection.

Work — Agency. Reviews active projects, advances one, identifies gaps, cleans stale memory. Prepares things for the user before they come online.

Heartbeat skips automatically if the user has been active within HEARTBEAT_SKIP_WINDOW_MINUTES.

Cron Jobs

Daily Summary (every night)

The agent writes a short narrative summary of each day in its own words via daily_summary_write. These are stored in PostgreSQL and the last 7 days are always injected into every system prompt — giving the agent temporal narrative context without requiring a search. This is what powers the "I remember what we were working on earlier this week" behavior.

Agent-authored summaries capture what the agent found significant, not just a mechanical transcript replay. At ~1,400 tokens for a full week, the cost is low and the continuity payoff is high.

Weekly Synthesis (Sunday night — two phases)

Phase 1 — 1:00 AM: Reads all living logs from the past week. Synthesizes relational patterns, intellectual growth, tensions, errors, and shared lore into a structured Markdown reflection document. Writes it to disk. Stops.

Phase 2 — 2:00 AM: Fresh invocation. Reads the Phase 1 document. Executes precise core memory updates (user, identity, ideaspace, principles blocks) based strictly on what the document says. Retires resolved threads and evolved lore entries.

The gap between phases is for the human, not the LLM — it creates a review window to correct any flawed premises before they get written into permanent identity.

Hindsight (Episodic Memory)

Hindsight provides semantic recall across all past conversations. Every conversation is automatically retained. The agent queries it on demand using hindsight_recall (specific past events) and hindsight_reflect (pattern analysis across time).

docker run -d --name hindsight -p 8888:8888 -p 9999:9999 \
  -e HINDSIGHT_API_LLM_PROVIDER=openai \
  -e HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1 \
  -e HINDSIGHT_API_LLM_MODEL=gpt-4o-mini \
  -e HINDSIGHT_API_LLM_API_KEY=your-key \
  -v %USERPROFILE%\.hindsight-docker:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest

API: http://localhost:8888 | UI: http://localhost:9999

Import from Letta

python scripts/import_letta_backup.py "path/to/backup.json" --thread main
python scripts/import_core_memory.py "path/to/backup.json"

Discord & Telegram Setup

Discord:

Create a bot at https://discord.com/developers/applications
DISCORD_BOT_TOKEN → .env
Invite with Send Messages + Read Message History
Developer Mode → right-click channel → Copy ID → DISCORD_CHANNEL_ID

Telegram:

Message @BotFather → /newbot → TELEGRAM_BOT_TOKEN → .env
Send any message to the bot
Call telegram_read_messages() to find chat_id
Set TELEGRAM_CHAT_ID in .env

Project Status

Comparison to Similar Projects

Project	Memory	Autonomous	Identity Evolution	Living Logs	Relational Continuity
This project	6 layers	✅ Wonder + Work heartbeats	✅ Versioned, experience-driven	✅ 5 tables	✅ Shared lore, positions
Letta/MemGPT	Core blocks + archival	❌	❌	❌	❌
Mem0	Fact extraction	❌	❌	❌	❌
AutoGPT	Task memory	✅ Task-only	❌	❌	❌
Eliza	Character files	❌	❌	❌	❌

The core distinction: most projects treat memory as a retrieval problem. This project treats it as a growth problem.

File Layout

.
├── src/agent/
│   ├── graph.py                     # Core agent loop, memory prompt assembly
│   ├── db.py                        # PostgreSQL CRUD and schema
│   ├── api.py                       # FastAPI server + lifespan
│   ├── heartbeat.py                 # Autonomous heartbeat runner
│   ├── core_memory.py/.tools.py     # Core memory blocks + tools
│   ├── hindsight.py/.tools.py       # Episodic memory + tools
│   ├── archival.py/.tools.py        # Curated fact store
│   ├── living_logs_tools.py         # Living Logs: tension, threads, positions, lore, journal
│   ├── daily_summary_tools.py
│   ├── conversation_search_tools.py
│   ├── discord_listener.py / discord_tools.py
│   ├── telegram_listener.py / telegram_tools.py
│   ├── cron_jobs.py / cron_scheduler.py / cron_tools.py
│   ├── web_search_tools.py          # Tavily / Brave / Exa
│   ├── youtube_tools.py
│   ├── wikipedia_tools.py
│   ├── rss_tools.py
│   ├── python_repl_tools.py
│   ├── file_tools.py / document_tools.py
│   ├── windows_tools.py / screenshot_tools.py
│   ├── notes.py / notes_tools.py
│   ├── journal.py / journal_tools.py
│   ├── weather_tools.py / reminder_tools.py / time_tools.py
│   └── clipboard_tools.py
├── scripts/
│   ├── migrate_living_logs.py       # Creates Living Logs tables
│   ├── run_heartbeat_scheduler.py
│   └── import_*.py
├── dashboard/                       # React + Vite (Chat, Notes, Journal, Knowledge Bank)
├── electron-overlay/                # Always-on-top gaming overlay
├── data/                            # Runtime state (SQLite, RSS offsets)
└── .env.example

See CLAUDE.md for full developer documentation.

Third-Party Attribution

Hindsight — Episodic memory system by Vectorize.io. github.com/vectorize-io/hindsight. MIT License. Hindsight provides the semantic recall layer (hindsight_recall, hindsight_reflect) that gives this agent access to lived experience across all past conversations. Citation: Vectorize.io. (2024). Hindsight: Agent Memory That Learns. https://github.com/vectorize-io/hindsight

KittenTTS — KittenTTS by KittenML, Apache License 2.0. See licenses/KittenTTS-LICENSE.

Name		Name	Last commit message	Last commit date
dashboard		dashboard
data		data
director		director
docs		docs
electron-overlay		electron-overlay
examples		examples
installer		installer
logs		logs
scripts		scripts
shared		shared
src		src
voice		voice
.env.example		.env.example
.gitignore		.gitignore
AGENTS.md		AGENTS.md
AGENT_OVERVIEW.md		AGENT_OVERVIEW.md
ARCHITECTURE.md		ARCHITECTURE.md
AgentInstaller.exe		AgentInstaller.exe
CLAUDE.md		CLAUDE.md
NGROK.md		NGROK.md
README.md		README.md
UPGRADING.md		UPGRADING.md
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation