Inside
Kreo

Contents

Kreo in one picture

Kreo is a personal assistant that lives on your Windows PC. It is a Python program that keeps running in the background. It has its own AI model, its own voice and its own memory.

Six layers, like a restaurant kitchen

The code is stacked in layers. Each layer only talks to the layer next to it. If you picture a small restaurant, every piece has an obvious job.

The golden rules

A handful of rules shape every file. Once you know them, most of the design stops looking strange.

Who does what: the model or Python?

This is the most important split in the whole project. The model is used for language. Python is used for everything that must be right.

The folder map

The repo root is the project root. The Python backend lives in kreo/ and the web client in frontend/.

1. 1
   CLAUDE.mdThe rules, and the traps found the hard way.
2. 2
   kreo/__main__.pyThe kreo command and how the agent starts.
3. 3
   kreo/api/app.pyHow every service is built and wired together.
4. 4
   kreo/core/agent/orchestrator.pyThe brain: 886 lines that handle every chat turn.
5. 5
   frontend/src/ChatContext.tsxHow the browser sends a message and shows the stream.

What happens when you run kreo run

The kreo command is a small launcher. Only run starts the agent. Every other subcommand talks to an agent that is already running, over HTTP.

scripts/kreo.cmd and kreo.ps1 run .venv\Scripts\python.exe -m kreo, and install-path.ps1 puts them on your user PATH so kreo works from any folder.

The Runtime: one box of shared services

Every service is created once, at start-up, and put into a single object called the Runtime. Any route that needs something takes it from there.

PAUSED and SLEEPING are waiting for the tray app (Stage 6) and the resource manager (Stage 7).

The model engine: llama-server

The AI model does not run inside Kreo's Python process. Kreo starts the official llama-server.exe as a child program and talks to it over HTTP, on 127.0.0.1:8766.

• Crashes. The ready-made llama-cpp-python wheels are AVX-512 builds. They crash this Ryzen 5 4600H.
• Security. Smart App Control is on, so unsigned DLLs without reputation are blocked. The official build is trusted.
• Memory. Killing a process always frees the GPU memory (VRAM). That matters when a game needs the GPU.

llama-server.exe -m models/Qwen3-4B-Q4_K_M.gguf
  -ngl 20  -c 4096  -t 6  --host 127.0.0.1 --port 8766
  -fa auto  -np 1  --jinja  --no-webui
  --reasoning-budget 640

Model states and the provider contract

The model can be switched off to free the GPU while the rest of Kreo keeps running. The ModelManager tracks where it is in its life.

thinking is passed on every call rather than stored on the provider, because one provider is shared by every request in flight.

A message's journey

This part is the heart of the book. We follow one message from your keyboard to Kreo's reply, through every file it touches.

A "yes" or "no" answering Kreo's own question skips step 1 entirely. Python recognises it first.

The same journey as a sequence diagram

Time runs downwards. Solid arrows are calls, dashed arrows are answers and events. The WebSocket carries small JSON events, in this order:

The docstring at the top of ws.py still leaves out sources and conversation_deleted, but both are sent.

Understanding the message

First, the model sorts your message into one of 19 intents. It fills in a form called ParsedIntent.

The switchboard: _perform

With the intent known and the permission checked, one method decides which piece of Python runs. It always returns an Outcome.

@dataclass
class Outcome:
    note: str | None              # what the model should know
    task_id: int | None           # task created or changed
    sources: list[dict]           # web sources to cite
    note_is_instruction: bool     # note is a sentence to say
    delete_conversations: list    # ids to delete at the very end

Framing the note

The Outcome's note is added to the prompt as one extra system message. How it is worded changes how a small model behaves, so there are three frames.

Writing the reply, and token budgets

A token is a piece of a word, roughly three or four characters. Every model call has a budget of tokens, and Kreo sets each one on purpose.

For streaming this retry lives in orchestrator.stream. For whole replies it lives in LlamaServerProvider.generate.

Runs: replies that survive a closed tab

A chat turn belongs to the agent, not to the browser tab that asked for it. That work unit is called a Run.

The toolbelt

Tools are ordinary Python modules in kreo/core/tools/. They do the exact work: dates, databases, maths and web requests.

Tasks and reminders

Every task, reminder and job is one row in the tasks table. You can create them by talking to Kreo, on the Tasks page, or with POST /api/tasks.

The API keeps started_at and completed_at in step with the status.

Weather, sunrise and location

Ask "Will it rain in Manchester tomorrow?" or "When does the sun set?" Every number in the answer is fetched or computed in Python.

Deleting conversations safely

Deleting is the one thing that cannot be undone, so it always takes two turns. The "yes" is read by Python, never by the model.

An exact yes-phrase, or up to 7 words starting with a yes-word and containing none of: but, not, except, instead, other, another, rather, unless, only, first, actually, wait, no. "Yes but only the old one" is not a yes.

1. 1
   Delete that conversation's generation_metrics
2. 2
   Detach its tasks (conversation_id → NULL). Tasks survive.
3. 3
   Delete its messages
4. 4
   Delete the conversations row, then write an audit row

Web search with citations

When you ask about something recent, Kreo searches DuckDuckGo, reads the top pages, and answers with numbered citations such as [1].

• is_public_http_url refuses localhost, private, loopback, link-local, reserved and multicast addresses. A web page cannot trick Kreo into reading your router or your own API.
• Each page download stops at 2 MB.
• Only readable text is kept, up to 1,800 characters a page.

Permission action web_search is AUTO by default. If nothing useful comes back, the note tells the model to say it could not find it.

Off the record

Kreo's answer to incognito mode. Flip the switch in the chat toolbar and the conversation is held in memory only. It is never written to the database.

The scheduler: reminders that fire on time

Kreo keeps working with no browser open. A background timer, APScheduler, checks for due reminders every 20 seconds. It never calls the AI model.

Both run with max_instances=1 and coalesce=True, so a slow run never piles up behind itself.

Permissions and the audit log

Every action with consequences has a permission level. The PermissionEngine looks it up before anything happens.

Changes made on the Permissions page are held in memory until restart. Saving them to the database is planned for Stage 12.

Resource awareness

The aim: Kreo gets out of the way when your PC is busy. Start a game and the model goes to sleep. Stop playing and it comes back.

How Kreo speaks

Kreo's voice comes out of the speakers of the machine running the agent. The browser never speaks; it only asks the agent to.

How Kreo listens

Listening also happens on the agent's machine. The client calls POST /api/speech/listen, and the agent records from its own microphone. The browser never records or uploads any audio.

When the client is on a different machine from the agent, it falls back to the browser's own recogniser. Chrome and Edge send that audio to the browser vendor, and the code says so.

The hands-free loop

The chat has two modes, text and voice, and you can switch in the middle of a conversation. Voice mode is a loop: listen, think, speak, listen again.

Messages sent by voice are stored with input_mode = "voice", and the chat shows a "spoken" mark under them.

The database

Everything Kreo remembers lives in one SQLite file, data/kreo.db. There are 21 tables. Eight are in use today; the rest are ready for later stages.

• Async. SQLAlchemy with aiosqlite, so a database call never blocks the chat stream.
• WAL mode and foreign keys on.
• Home-made migrations. No Alembic. On start-up, _add_missing_columns compares the models with the file and adds any new columns. Dropping or retyping a column would need a real migration.
• Full-text search. FTS5 indexes messages_fts and memories_fts, kept in step by triggers.
• Always UTC. SQLite stores no timezone, so everything is written in UTC and tagged on the way out by iso_utc().

Configuration and secrets

Every setting has a typed default in kreo/config/settings.py. Your own values go in config/kreo.json, which git ignores.

Logs and performance metrics

Kreo keeps two kinds of records about itself: logs (what happened) and metrics (how fast the model was).

Each row has latency, prompt and completion tokens, reasoning length, tokens per second, and whether it worked. The stream reports no token usage, so for streamed replies tokens are estimated as characters ÷ 4.

The web client

The browser app is React 19 and TypeScript, built with Vite. FastAPI serves the built files from frontend/dist, on the same port as the API.

• index.html is served with no-cache headers. The JS and CSS files have content hashes in their names, so they can be cached safely.
• /api/status reports which bundle is live. If the page is running an older one, StatusContext reloads it once.
• In development, npm run dev serves on port 5173 and proxies /api and /ws to 8765.

The eleven screens

The sidebar lists eleven pages in three groups. Two are honest placeholders for features that are not built yet.

ChatContext: the client's side

ChatContext.tsx (681 lines) owns the conversation in the browser: the messages, the socket, voice mode, dictation and off-the-record mode.

1. 1
   Trim the text. Give up if it is empty or a reply is already running.
2. 2
   Silence any speech and clear the box. Remember if voice mode is on.
3. 3
   Add two bubbles at once: yours, and an empty Kreo bubble marked pending.
4. 4
   Open a new WebSocket for this turn and send {type:"chat"}.
5. 5
   Each event updates the last bubble (see right).
6. 6
   On end: close the socket, reload the conversation list.

The terminal client

kreo chat is a second client, equal to the browser. It attaches to the running agent over the same /ws socket. If no agent is running, it says so and exits.

• Speaks the whole reply after end (up to 2,000 characters), not sentence by sentence.
• Never off the record.
• Three silent turns in a row switch voice mode back to text.
• Ignores sources and farewell.

Start with Windows, toasts and the tray

Kreo is meant to be part of Windows: it starts when you sign in, runs in the background, and taps you on the shoulder with a toast.

kreo root and the API

Each row: the file, its line count, its job in plain words, and the key names inside it.

The agent and its tools

Services and speech

Storage, config and the CLI

The web client's core files

Pages

Every endpoint on port 8765

All HTTP routes live under /api. Anything else serves the web client.

Where the project stands

Kreo is built stage by stage from initial-docs/stages.md. Each stage must leave a system that runs.

• Chat with a local Qwen3-4B, streaming, with visible reasoning
• Tasks and reminders created, edited, completed and deleted by conversation
• Reminders as toasts, chimes and spoken announcements, with quiet hours
• Web search with citations; offline sun times; Open-Meteo weather
• Voice in and out on the agent's machine, and a hands-free mode
• Off-the-record chats; safe two-step conversation deletion
• Permissions with an audit trail; start with Windows
• A web client with eleven screens, and a terminal client

Surprises in the code

Things that differ from what the comments say, or that catch newcomers out. Worth knowing before you change anything nearby.

Words you will meet

Agent

A program that runs by itself and acts for you. Here, the Python process started by kreo run.

LLM

Large language model: the AI that reads and writes text. Kreo uses Qwen3-4B.

GGUF

The file format llama.cpp loads models from.

Quantised (Q4_K_M)

Numbers stored in about 4 bits instead of 16, so the model fits a small GPU.

Token

A piece of a word, about 3–4 characters. Budgets are counted in tokens.

Context window

How many tokens the model can see at once: 4,096.

Reasoning / thinking

Private notes the model writes before answering.

Structured output

Forcing the model to reply with JSON that fits a schema.

Intent

The kind of request: chat, create a reminder, weather…

Orchestrator

The code that runs a chat turn from start to finish.

Note

Facts or a sentence Python hands the model to phrase.

Run

One chat turn, owned by the agent, with all its events.

VRAM

Memory on the graphics card. The GTX 1650 has 4 GB.

Running it yourself

The commands a developer uses every day, from the project root in PowerShell.

# run the tests (mock model, temporary database)
.\.venv\Scripts\python.exe -m pytest

# lint and format
.\.venv\Scripts\python.exe -m ruff check kreo tests benchmarks
.\.venv\Scripts\python.exe -m ruff format kreo tests benchmarks

# start the agent without the AI model
.\.venv\Scripts\python.exe -m kreo --no-model

# chat from the terminal
.\.venv\Scripts\python.exe -m kreo chat

# rebuild the web client
cd frontend; npm run build