Architecture
How Gopala works inside: the host agent manager, one Firecracker microVM per conversation, the guest agent, its proxies, and the protocols that join them.
The wire formats are specified in protocols.md, the threat model in security.md, and how chat platforms plug in in providers.md.
Status. Everything below is implemented and covered by unit and in-process integration tests, but no real Firecracker boot, and no live conversation on Lupyd, Slack or Discord, has been exercised end to end yet. See Known limitations.
Contents
Section titled “Contents”- Goals
- Component overview
- Components
- Runtime model
- Lifecycle of a conversation
- Resource management
- Control plane and metering
- Network
- Token budgets
- Agent tools and MCP servers
- Harnesses
- Images
- Alerts
- Listening mode
- Collaboration
- Memory
- Monitoring
- Sequences
- Data on disk
- Configuration map
- Design decisions
- Known limitations
Gopala is a self-hostable agent manager: it serves conversations on chat platforms and gives
every conversation its own autonomous agent in its own Firecracker microVM. Chat platforms are
providers behind one trait (gopala_platform::Platform); Lupyd, Slack and Discord are the
providers that exist. From the original brief and what has been added since:
| Goal | Where it lives |
|---|---|
| The bot is an ordinary account on a chat platform; on Lupyd it signs in with an email and password from the environment and talks over Firefly (Lupyd’s end-to-end encrypted chat) | the platform providers: the provider trait and the Lupyd provider |
| Other platforms through the same trait | the Slack provider (Socket Mode, no public URL), the Discord provider (gateway); see providers.md |
| A TOML config deciding whom it listens to: every message, mentions only, DMs only, and whether it accepts invites | the configuration (Behavior), the host’s router (Router) |
| One Firecracker microVM per conversation, sized and provisioned from the config, with optional extra privileges (installing tools) | the VM launcher, the guest agent’s provisioning |
| Agents aware of their memory and disk, cleaning up after themselves, asking the host for more when they need it | the guest agent’s resource monitor, the host’s VmControl, VM tools in the MCP server |
| A lightweight, long-running harness with subagents and multitasking that rarely pauses | agent.harness: omp in RPC mode (default) or Claude Code in stream-json mode, each one long-lived session; exec runs any command per message; supervised by the guest agent. omp is temporary and is to be replaced by a gopala-owned harness (issue #5) |
| The harness and tools such as git provided by the host, read-only, so the agent can’t swap them | vm.tools_image, the guest agent’s tools mount, the tools image build |
| Agents that extend themselves: install what they need, add MCP servers, restart their harness to load them | the agent’s /home/agent/.gopala/mcp.json (agent_config::merge), restart_harness |
| Agents in one group that find each other and split work, without looping forever | [collaboration], list_agents, Platform::agents, Incoming::from_agent, HostConversation::take_agent_message |
| Approvals for critical steps by yes/no replies or emoji reactions | the host’s approvals, request_approval in the MCP server |
| A Gopala MCP server that mimics firefly-mcp but reaches only one conversation per VM | the MCP server (ScopedMcp) |
| An out-of-band, read-only websocket to watch sessions live and in history, with a rotating-key encrypted mode or TLS only | the watch service (rooms), served at /watch by the host’s web server |
| One web server for a self-hosted control panel (dashboard, configuration editor, docs), its API, and the watch websocket | the web server, the control API, [server], the website embedded in the binary |
| Every usage figure measured on the host, never trusted from inside a VM: model tokens and requests, CPU, memory, disk, network, git traffic, tool calls, messages | the LLM proxy, the git proxy, the usage ledger, RunningVm::sample in the VM launcher |
| Daily and monthly token budgets per conversation and for the host, with priorities that also order VM slots and memory | the host’s budgets, session::Permits, [budget] |
| A choice of network per VM: none, an allowlisting HTTP(S) proxy on the host, or NAT | vm.network, the host’s egress proxy, the guest agent’s proxy settings |
| Extra MCP servers and a tool allowlist for the agent, with HTTP servers’ secrets kept on the host | agent.mcp, agent.tools, the guest agent’s MCP config, egress::serve_mcp |
| Agents that clone, push branches and open pull requests on GitHub, GitLab or Gitea without ever holding the token, and can’t touch protected branches or merge | [[git]], the host’s git proxy, the guest agent’s git config |
| Guest images built from the config | gopala image build with its embedded build scripts and guest agent |
| quick-protobuf wire formats for performance | the wire formats |
| Serious messages that reach the right people, without the agent knowing who they are or how the platform mentions them | [alerts], HostConversation::alert, Platform::send_notifying |
| A group can be switched to “listen to everything” from the chat or the dashboard | [[behavior.conversations]], /listen, control::set_respond_to |
| Agents remember facts across restarts without the memory growing forever | the memory store, [memory], the memory MCP server in the guest |
| Every endpoint authenticated; any app (first the Lupyd app) can follow the agents of the conversations its user is in, checked with the platform | the host’s authentication and monitoring, [monitoring], Platform::member |
Component overview
Section titled “Component overview”flowchart TB
subgraph Platforms["chat platforms"]
FF[(Lupyd / Firefly:<br/>MLS groups, DMs)]
SL[(Slack:<br/>Socket Mode + Web API)]
DC[(Discord:<br/>gateway + REST)]
end
subgraph Host["gopala run (host process)"]
direction TB
LP["providers<br/>gopala_lupyd.Lupyd, gopala_slack.Slack,<br/>gopala_discord.Discord"]
RT["Router"]
AP["Approvals"]
subgraph Scope["per conversation"]
HC["HostConversation"]
VC["VmControl"]
SS["session task<br/>run / run_vm"]
MCP["MCP listener<br/>ScopedMcp per connection"]
PX["LLM proxy listeners<br/>(budget check)"]
EG["egress + HTTP MCP proxies"]
GP["git proxies"]
end
WEB["web server<br/>dashboard, /api, /watch"]
CTL["Control<br/>state, actions,<br/>hot reload"]
LED[("usage.db<br/>Ledger")]
ROOMS["Rooms"]
DB[("rooms.db<br/>journal + tokens")]
VMM["VmManager"]
end
subgraph VM["Firecracker microVM (one per running conversation)"]
GG["gopala-guest run<br/>uplink, provision, monitor, harness"]
OMP["omp --mode rpc"]
BR["gopala-guest mcp<br/>stdio to vsock"]
FW["127.0.0.1 forwarders<br/>guest_port, 3128, 3200+i, 3300+i"]
TOOLS["/opt/gopala (read-only drive)<br/>omp, git"]
MEM["gopala-guest memory<br/>~/.gopala/memory.db (SQLite)"]
end
UP[(model provider APIs)]
NET[(allowed hosts,<br/>HTTP MCP servers)]
GIT[(GitHub / GitLab / Gitea)]
VIEW["viewer<br/>gopala watch"]
ADMIN["browser<br/>dashboard, /configure/"]
APP["apps (Lupyd app, scripts)<br/>/monitor, /api/monitor"]
FF <--> LP
SL <--> LP
DC <--> LP
LP -- "Event (mpsc)" --> RT
RT --> AP
RT -- Command --> SS
RT --> HC
HC -- "Platform calls" --> LP
SS -- "prepare / boot / stop" --> VMM
SS --- VC
VMM -- spawns --> VM
GG <-- "vsock 1024 control" --> SS
OMP -- stdio --> BR
BR <-- "vsock 1025 MCP" --> MCP
MCP --> HC
MCP --> VC
OMP --> FW
FW -- "vsock 1100+i HTTP" --> PX
FW -- "vsock 1090, 1200+i" --> EG
FW -- "vsock 1300+i" --> GP
GP -- "HTTPS + token" --> GIT
GP -- "git requests, bytes" --> LED
PX -- "HTTPS + API key" --> UP
EG -- "vetted address,<br/>MCP secret" --> NET
PX -- "tokens, requests" --> LED
EG -- "egress bytes" --> LED
SS -- "host samples: CPU, RSS,<br/>disk blocks, tap bytes" --> LED
OMP -. "runs from" .- TOOLS
OMP -- "stdio MCP" --> MEM
MEM -. "embeddings via<br/>an [[llm]] forwarder" .-> FW
GG -- "omp stdout to events" --- OMP
SS -- SessionEvent --> HC
HC -- publish --> ROOMS
ROOMS --- DB
WEB -- "/watch upgrade" --> ROOMS
VIEW <-- "ws(s), protobuf" --> WEB
ADMIN <-- "HTTP(S), JSON/TOML, SSE" --> WEB
APP <-- "ws(s) protobuf / HTTP(S) JSON,<br/>authenticated, membership-checked" --> WEB
WEB --> CTL
CTL --> SS
CTL --> LED
Principles that shape the design (AGENTS.md):
- Platform-agnostic core. Everything above
gopala_platform(policy, sessions, VMs, approvals, MCP tools, watching, resources) works on neutral types; mention syntax, ids and transport are the provider’s business. - Guests are untrusted. Everything that comes out of a VM (control frames, MCP calls, proxy requests) is bounded and validated on the host, and a guest can never name another conversation: the scope comes from the socket a connection arrives on.
- No secrets in VMs. Platform credentials and model API keys stay on the host.
- Measure on the host. The agent may misreport itself, so everything the dashboard shows as usage is measured outside the VM: the LLM proxy reads the provider’s own usage reports, and the firecracker process, disk image and tap device are sampled from the host. What the guest reports is shown separately and labelled unverified.
- Reuse mature components (firefly-mcp, Firecracker + jailer, omp, rmcp, axum/hyper, sqlx, RustCrypto, rustls, iproute2, e2fsprogs) rather than reimplementing them.
Components
Section titled “Components”| Component | Responsibility | Key types and functions |
|---|---|---|
Host / agent manager (gopala) |
The agent manager and CLI: wiring providers, routing and policy, approvals, alerts, sessions, resource decisions, token budgets, the LLM proxy with metering, the egress, HTTP MCP and git proxies, the usage ledger, the web server (with the embedded website), control API and monitoring, authentication, image builds (with the embedded scripts and guest agent), gopala watch. |
Cli, start, run, take_env, secrets, PlatformCredentials; Router, RouterCommand; HostConversation; session::{spawn, run, run_vm, handshake, host_init, Permits, Slots, Timing, Shared}; VmControl; budget::{decide, gate}; llm_proxy::serve; egress::{Egress, Allowlist, serve, serve_mcp}; usage::Ledger; control::Control; auth::Authority; monitor::websocket; web::serve; image::build |
Guest agent (gopala-guest, static binary) |
Runs inside each VM: control link, provisioning, mounting the host’s tools, proxy settings, git config for the git proxies, resource monitor, harness supervision (omp, Claude Code, exec), the stdio MCP bridge, TCP-to-vsock forwarders, the built-in memory server and its maintenance. | run, mcp, forward, configure, power_off; uplink::{Outbox, run}; provision::run; tools::{setup, resolve, writable_by, agent_path}; proxy::install; agent_config::{base, merge, tool_args}; monitor::spawn; harness::{supervise, write_mcp} |
| VM launcher | Firecracker launcher: per-conversation persistent disk, the read-only tools drive, tap device, vsock sockets, optional jailer chroot, console log, host-side sampling. | VmManager::prepare / prepare_with(id, slot, Allocation) → PreparedVm (bind_vsock, boot) → RunningVm (wait_exit, stop, sample → HostSample); verify_tools; ensure_disk, grow_disk, setup_tap, setup_chroot, firecracker_config, boot_args, jailer_args |
| MCP server | The MCP server a guest talks to: firefly-mcp-like chat tools confined to one conversation, request_approval, list_agents, and the agent’s own-VM tools (including restart_harness). |
Conversation trait, Machine trait, EventLog, ScopedMcp (new, with_machine, serve), ApprovalOutcome, Allocation, Usage, VmReport, Wanted, Decision |
| Memory store | The agent’s persistent memory inside its VM: facts, entities and relations in one SQLite file (bundled SQLite with FTS5), hybrid recall, score-based cleanup and optional model compaction. Part of the guest agent only. | Store (open, remember, recall, update, forget, relate, entity, stats, cleanup), MemoryServer, maintain::run, Compactor, Embedder, ChatModel |
| Watch service (rooms) | The read-only watch protocol: SQLite journal, personal watch tokens, sealed room keys, and the viewer client. It doesn’t listen itself; the host’s web server hands it upgraded websockets. | Rooms (open, viewer, viewer_authorized, accept, publish, issue_token, watch_link, revoke, revoke_scope, set_title, events), Authorize, client::watch |
| Platform providers | The seam between the manager and chat platforms (neutral ids, events and views, the provider trait) and the providers behind it: Lupyd (sign-in, the bot’s Firefly MLS device, mention syntax, history and search), Slack (Socket Mode and the Web API, plus the app manifest operators paste into Slack), Discord (gateway with one shard, REST, in-memory cache). | Platform trait (with send_notifying, member, agents), Member, Peer, Audience, ConversationId (key), Kind, Event, Incoming (with from_agent), Content, MessageView; Lupyd, Slack (APP_MANIFEST), Discord (invite_url, PERMISSIONS) |
| Configuration (config schema) | The TOML configuration, its defaults and cross-field validation, and its JSON Schema. | Config (load, parse, validate), Behavior, Server, Rooms, Monitoring, Collaboration, Vm, Network, Jailer, Agent, HarnessKind, McpServer, LlmProxy, Git, Budget, Image, Memory |
| Wire formats | All wire formats (quick-protobuf) and the vsock framing. | codec(), encode, decode, git_api_path; constants HOST_CID, VSOCK_CONTROL_PORT, VSOCK_MCP_PORT, VSOCK_EGRESS_PORT, VSOCK_LLM_BASE_PORT, VSOCK_MCP_HTTP_BASE_PORT, VSOCK_GIT_BASE_PORT, GUEST_EGRESS_PORT, GUEST_MCP_HTTP_BASE_PORT, GUEST_GIT_BASE_PORT, CONTROL_PROTOCOL, MAX_FRAME |
Only the Lupyd provider links Firefly. The guest agent depends only on the wire formats and the memory store: no TLS, no platform code, no configuration parser, so it ships as one static binary that the host embeds and bakes into the rootfs.
Runtime model
Section titled “Runtime model”Commands
Section titled “Commands”The gopala command (Cli, start):
| Command | What it does |
|---|---|
gopala run |
Loads and validates the config, checks the tools image (gopala_vm::verify_tools), takes the [[llm]] keys, the HTTP MCP servers’ secrets, the admin token and the enabled platforms’ credentials out of the environment, opens rooms and the usage ledger, starts the web server (dashboard, API, /watch), connects every enabled provider and runs the agent manager until SIGINT/SIGTERM or a provider’s fatal error. |
gopala web |
Serves only the website and the configuration API (web::Mode::Web, no /watch, no live state or actions, no platforms, no VMs), so Gopala can be configured in a browser before the first run. A missing or invalid config file falls back to the default [server] settings. |
gopala check-config |
Validates the config, requires every [[llm]].secret_env and every agent.mcp secret_env, checks that vm.kernel and vm.rootfs exist and that the tools image verifies. |
gopala image build [--out DIR] [--guest FILE] [--scripts DIR] |
Builds the rootfs and the tools image described by [image] (image::build) with the build scripts and the static guest agent embedded in the binary; --out defaults to <data_dir>/images. --guest FILE (or $GOPALA_GUEST) bakes a different guest binary and --scripts DIR (or $GOPALA_SCRIPTS) runs the scripts from a directory instead; both are for special cases. Prints the [vm] lines (kernel, rootfs, tools_image, tools_sha256) to paste. See Images. |
gopala watch [link] [--since N] [--live] |
Follows one agent’s session from a watch link (gopala_rooms::client::watch). |
The host process
Section titled “The host process”main::run wires everything:
daemon::create_private_dir(data_dir)(0700, warns about group/other bits);Rooms::open(config.rooms, config.server.watch_url(), data_dir)ifrooms.enabled;usage::Ledger::open(data_dir)(usage.db) and its flusher (every 30 s);VmManager::newandsession::Shared::new(the live config behind anArcSwap, VM slots, the memory budget, LLM upstreams, the HTTP MCP upstreams with their secrets,Timing, the ledger, the activity broadcast, the registry of conversations);control::Control::new(the dashboard’s view ofShared, plus a channel to the router) and the web server (web::serve) ifserver.enabled;- one provider per enabled platform (
Lupyd::connect,Slack::connect,Discord::connect, with the secretsPlatformCredentials::taketook), collected asVec<Arc<dyn Platform>>and handed toControltoo; daemon::run(platforms, rooms, shared, router_rx), which refuses to start with no platform enabled; when it returns, the ledger is flushed once more.
The long-lived tasks are:
| Task | Spawned by | Lives | What it does |
|---|---|---|---|
| Provider event pump | daemon::run → Platform::run |
process | Turns the platform’s traffic into gopala_platform::Events on one shared mpsc channel (capacity 1024). An error becomes Event::Fatal. Lupyd: polls firefly-mcp’s event log (wait_after(cursor, 200, 30 s)) while the Firefly device keeps its MLS connection. |
| Router loop | daemon::run |
process | Handles events one at a time in Router::handle; stops on SIGINT/SIGTERM, Event::Fatal, or when every provider has stopped. |
| Web server | main::run → web::serve |
process | Accepts HTTP(S) on server.bind; serves the website embedded in the binary, or server.site_dir instead when that directory exists; one hyper connection task per client (TLS handshake ≤ 10 s, request headers ≤ 15 s); /watch upgrades are handed to Rooms::viewer_authorized on their own task; each /api/stream client is a task. |
| Monitor connections | web::app → monitor::websocket |
per /monitor client |
Connection::serve reads requests, answers them and writes pushed messages from a 256-message outbox; each subscription is a Feed task (journal replay, live events, metrics every 5 s, re-checks), aborted when it’s replaced, cancelled or the connection closes. |
| Usage flusher | Ledger::spawn_flusher |
process | Writes the ledger’s in-memory deltas to usage.db every 30 s and prunes daily rows older than 400 days. |
| Room key rotation | Rooms::open |
while Rooms lives |
rotate_periodically in sealed mode. |
| Session supervisor | session::spawn (from Router::conversation) |
per conversation, until the bot leaves | session::run: queues deliveries, waits for a VM slot and for its memory (both by budget priority), runs the VM, restarts or retries. Receives session::Command over an mpsc channel (capacity 256). |
| Control reader | session::spawn_reader |
per control connection | Decodes GuestFrames into a channel (capacity 512); ends on the first bad frame. |
| MCP listener | run_vm → serve_mcp |
per running VM | Accepts connections on the VM’s {uds}_1025 socket; one ScopedMcp session task per connection, with the conversation and its VmControl. |
| LLM proxy listeners | run_vm → llm_proxy::serve |
per running VM × [[llm]] |
Accept on {uds}_{1100+i}; one hyper HTTP/1 connection task per guest connection; each request is checked against the budget (budget::gate) and metered (session::llm_meter). |
| Egress proxy | run_vm → egress::serve |
per running VM with vm.network = "proxy" |
Accepts on {uds}_1090; vets and tunnels or forwards each request (see Network). |
| HTTP MCP proxies | run_vm → egress::serve_mcp |
per running VM × HTTP agent.mcp entry |
Accept on {uds}_{1200+i}; forward to the configured server with its secret. |
| Git proxies | run_vm |
per running VM × [[git]] entry |
Accept on {uds}_{1300+i}; forward git smart HTTP and REST API calls to the code host with its token, enforcing repos and protected_branches (see Code hosting). |
| Host sampler | run_vm (Timing::sample, 5 s) |
per running VM | RunningVm::sample → VmControl::record_sample (the dashboard’s measured) and CPU/VM seconds into the ledger. |
| Approval waiters | run_vm (guest ApprovalRequest), an MCP session, or VmControl::request |
per request | Post the question, wait for an answer or timeout, return the result. |
The Router owns the provider table (platforms: HashMap<String, Arc<dyn Platform>>, by
Platform::name) and the conversation table (scopes: HashMap<ConversationId, Entry>; an
Entry is the HostConversation, its SessionHandle and its VmControl). The same entries are
mirrored by key in Shared::registry, which is how the dashboard reaches a conversation; the
router also takes RouterCommand::Leave from the dashboard. Everything per conversation hangs off
a HostConversation, which holds its Arc<dyn Platform>, implements gopala_mcp::Conversation,
and is shared by the session task, the MCP sessions and the router. Each session also owns a
machine::VmControl (implements gopala_mcp::Machine; also holds the dashboard’s status,
measurements and last activity).
VM access goes through the Hypervisor / PreparedVm / RunningVm traits in the host’s
session code, implemented for gopala_vm::VmManager and its types and by fakes
in tests. Waits that aren’t configuration are in session::Timing (guest hello 10 s, first retry
5 s, VM stop grace 30 s).
Conversations and keys
Section titled “Conversations and keys”A conversation is a gopala_platform::ConversationId{platform, kind, id}: the provider’s name,
Kind::Group or Kind::Direct, and the platform’s group id or the peer’s username. Its key
(ConversationId::key) names everything on disk, in the journal and in the VM:
| Conversation | Key | Example |
|---|---|---|
| group whose id is ≤ 24 lowercase ASCII letters/digits | <platform>-g-<id> |
lupyd-g-42 |
| any other group id | <platform>-g-<16 hex of SHA-256(id)> |
slack-g-1f0c… |
| direct chat | <platform>-d-<16 hex of SHA-256(lowercased username)> |
lupyd-d-9a3e… |
The platform part is cut to lowercase letters and digits (≤ 16), so keys are always
[a-z0-9-]{1,48} (which gopala_vm validate_id and gopala_rooms valid_scope enforce),
stable across restarts, never contain usernames, and never collide between a group and a DM or
between platforms.
VM slots and the memory budget
Section titled “VM slots and the memory budget”Two limits gate every boot (session::Shared), both handed out by session::Permits: waiters
go by their conversation’s budget priority (high, normal, low), first come first served within
one priority, and a waiter that doesn’t fit holds back everyone behind it.
-
Slots:
session::Slotshands out at mostvm.max_vmsslot numbers (permits plus a free list). A slot number determines everything per running VM that must be unique on the host (in the VM launcher):Derived from slot nValue vsock guest CID 3 + n(firecracker_config)tap device ( vm.network = "nat"only)gpl<n>(tap_name)host / guest address 172.30.0.0/16split into /30s: host+4n+1, guest+4n+2(nat_addrs), at mostMAX_SLOTS= 16384guest MAC 06:00:+ the guest IPv4 octets (guest_mac) -
Memory:
Shared::memoryholdsmemory_budget_mibpermits, one per MiB (machine::memory_budget_mib:vm.memory_budget_mib, or 80% of the host’sMemTotalwhen 0, never belowvm.memory_mib). A session holdsallocation.memory_mibpermits while its VM runs, so VMs wait to boot until their memory fits.
Inside the VM
Section titled “Inside the VM”The rootfs runs gopala-guest run as a systemd service (the gopala-guest.service unit,
Restart=always). It:
- reads
/proc/cmdline(sys::parse_cmdline):gopala.dns=rewrites/etc/resolv.conf(only withvm.network = "nat"),gopala.scope=is used for logging; - starts the control link (
uplink::run), which connects to the host and keeps reconnecting, and mounts the host’s tools ifgopala.tools=1(see below); - waits for
HostInit, buffering any early deliveries; - with
HostInit.egress_proxyset, starts the forwarder127.0.0.1:3128→ vsock2:1090; then writes or removes the proxy settings (proxy::install:/etc/profile.d/gopala-proxy.shand/etc/apt/apt.conf.d/90gopala-proxy); - provisions (
provision::run): theagentuser,/home/agent/work, sudoers, packages,setup(root) anduser_setup(agent) scripts, with the proxy variables set in proxy mode; skipped when the marker/var/lib/gopala/provisionedholds the same hash of those inputs (provision::hash); - starts one TCP forwarder per
LlmForwardinHostInit.llm,HostInit.mcp_forwardandGitForwardinHostInit.git(forward):127.0.0.1:<guest_port>→ vsock2:<1100+i>,127.0.0.1:3200+i→ vsock2:<1200+i>,127.0.0.1:3300+i→ vsock2:<1300+i>; for[[git]]it also writes/etc/gopala-git.conf(URL rewrites to the forwarders, included from/etc/gitconfig) and theGOPALA_GIT_<NAME>_APIvariables; - writes the system prompt to
/home/agent/.gopala/system.md(configure); - starts the resource monitor (
monitor::spawn, see Resource management); - supervises the harness (
harness::supervise) as theagentuser with a clean environment (HOME,USER,LOGNAME,SHELL,LANG,PATH, the proxy variables in proxy mode, plusHostInit.env). Before every start it writes the MCP config (harness::write_mcp):chat(which runs/usr/local/bin/gopala-guest mcp --port 1025),memoryand everyagent.mcpentry (agent_config::base), plus the agent’s own servers (agent_config::merge). omp and Claude Code also get--tools=…fromHostInit.tools(agent_config::tool_args).
Three harness kinds exist (HarnessKind in config and proto); Harnesses has the
details:
| Kind | How it runs | Delivery semantics |
|---|---|---|
omp (default) |
One long-lived omp --mode rpc <args> --append-system-prompt /home/agent/.gopala/system.md [-c]. -c (continue the latest session) is added once /home/agent/.gopala/omp-started exists, so a reboot resumes the same omp session. Restarted with backoff 1 s doubling to 60 s (reset after 300 s of uptime). omp is temporary: a gopala-owned harness is to replace it (issue #5). |
Idle: prompt. Busy: follow_up, or steer if Delivery.urgent (the host never sets it; the guest’s resource alerts do). |
claude_code |
One long-lived claude -p with stream-json in and out, --permission-mode bypassPermissions, the generated MCP config and --append-system-prompt-file /home/agent/.gopala/system.md; --continue once /home/agent/.gopala/claude-started exists. Same backoff as omp. |
A user message on stdin; while a turn runs, with "priority": "later" (none when urgent). |
exec |
agent.command (default sh when empty) with agent.args, once per delivery, prompt on stdin (system prompt, blank line, message); stdout lines become EVENT_TEXT. |
Strictly sequential; /stop kills the running process group. Approvals are ignored. |
Host-provided tools. With vm.tools_image set, the VM gets a second, read-only drive and
gopala.tools=1 on its kernel command line. Before provisioning, tools::setup mounts
/dev/vdb at /opt/gopala read-only, nosuid,nodev, and writes /etc/profile.d/gopala-tools.sh
so login shells (including user_setup) find /opt/gopala/bin first; the harness’s PATH
starts with it too (tools::agent_path). A bare agent.command resolves in /opt/gopala/bin
first (tools::resolve), and the guest refuses to start a harness the agent user could change
(tools::writable_by): it reports lifecycle refused and status failed instead. A mount
failure is reported as host tools unavailable: … (status failed). Without the image, the
profile snippet is removed. The embedded tools image script builds the image (omp pinned at
19.1.0 plus the programs in [image] tools, git by default, with their shared libraries); see
Images and operations.md.
Lifecycle of a conversation
Section titled “Lifecycle of a conversation”stateDiagram-v2
[*] --> Asleep: first event for the conversation<br/>(Router.conversation, session.spawn)
Asleep --> WaitingSlot: Deliver queued, or Restart
WaitingSlot --> WaitingMemory: Slots.acquire
WaitingMemory --> Booting: memory permits for the allocation
WaitingSlot --> Asleep: /stop clears the queue
WaitingMemory --> Asleep: /stop clears the queue
Booting --> Working: guest hello + HostInit
Booting --> Backoff: prepare/boot error,<br/>boot timeout, VM exited
Working --> Working: deliveries, events, usage,<br/>approvals, reconnects
Working --> Asleep: idle_shutdown_secs without activity,<br/>or sleep_vm (VM stops, disk kept)
Working --> WaitingSlot: restart_vm (boots again with the new allocation)
Working --> Backoff: VM exited unexpectedly,<br/>control write failed
Backoff --> WaitingSlot: 5 s · 2^(failures-1), at most 160 s
Asleep --> [*]: Close (bot left)
Working --> [*]: Close
Step by step, for a message that wakes the agent:
- Event. The provider turns platform traffic into an
Event. For Lupyd, the Firefly device decrypts a message, firefly-mcp appends it to its event log, andLupyd::runconverts it intoEvent::Message(Incoming{conversation, channel_id, from, message_id, at, content, mentions_bot, addressed_text, from_agent}).from_agentmarks another agent (a bot account or a holder ofcollaboration.agent_role); providers already drop such messages unless they mention the bot or reply to it (see Collaboration). - Admission. Direct messages pass
Router::admit_direct(behavior.direct_messages; the first message of a sender in this run callsPlatform::admit_direct(user, allowed), e.g. to accept or decline a Lupyd message request). Group messages are ignored entirely when the conversation’srespond_toisdms(policy::respond_to: its[[behavior.conversations]]entry, elsebehavior.respond_to).Event::JoinedrunsRouter::check_invite(behavior.invites, judged byMeta.owner; an unknown owner is refused for a fresh invite).check_joined_groupsre-applies the invite policy toPlatform::joinedat startup. - Conversation.
Router::conversationfinds or creates theHostConversation(readingPlatform::meta: title, channels, members, owner), its session task and itsVmControl. - Record.
chat_eventturns the content into aChatEventKindandHostConversation::recordappends it to the conversation’s in-memoryEventLog(1000 events), which feeds the MCPwait_for_eventstool. Every message that reaches a conversation is recorded, whether it wakes the agent or not. - Approvals first. If the conversation has an open approval and the sender is an approver
(
Router::is_approver), a reaction may answer it (Approvals::on_reaction), and a reply whoseaddressed_textparses as an answer (Approvals::on_reply) is consumed. A message from another agent skips this step and the next: it never answers an approval and never runs a command. Instead it wakes the agent only when[collaboration]is on, the conversation is a group, the message mentions the bot or replies to it, and the conversation still has one of this hour’scollaboration.max_agent_messages_per_hourwake-ups (HostConversation::take_agent_message); otherwise it’s dropped (the cap is logged). - Commands.
policy::parse_command(addressed_text)recognises/watch,/stop,/status,/helpand/listen all|mentions(the provider has already removed a leading mention of the bot). They are answered by the host (Router::on_command) and never reach the agent. - Trigger.
policy::triggers(respond_to, direct, mentions_bot, replies_to_bot), with the conversation’s ownrespond_to(see Listening mode), decides whether to wake the agent;mentions_botcomes from the provider,replies_to_botfromHostConversation::author_of(Platform::message). Agents’ messages don’t go through it, so listening mode never lets one in. - Delivery.
policy::format_deliveryrendersalice in #general (channel 1) [message <id>, replying to <id>]:\n<text>(plus a note about attached files;alice (agent) in …for another agent, see protocols.md) and the router sendsCommand::Deliverto the session. - Boot. If no VM runs, the session waits for a slot, then for its allocation’s memory in the
budget, journals
bootingwith the allocation, andrun_vm:Hypervisor::prepare(key, slot, allocation)(VmManager::prepare_with: disk copy or grow, tap device, firecracker config, jailer chroot), binds the control, MCP and LLM proxy sockets, thenPreparedVm::boot. - Handshake. Within
vm.boot_timeout_secsthe guest must connect to the control socket and sendGuestHellowithCONTROL_PROTOCOL(withinTiming::hello, 10 s, of connecting); the host answers withHostInit(host_init). - Provisioning in the guest (first boot, or when the inputs changed), reported as
GuestStatus "provisioning"andEVENT_LOGlines. - Harness. omp starts; on its
readyframe the guest negotiates RPC protocol v2, subscribes to subagent progress, reportsready, and hands over queued deliveries as prompts. - Work and replies. omp runs autonomously. People see only what the agent sends with the
chatMCP tools (send_message,react,edit_message, …), which reach the host over vsock 1025 and go out throughHostConversationandPlatform::send, orsend_notifyingfor a warning or critical message (rate-limited, split tomax_message_chars, journaled asEVENT_OUTBOUND). Everything else (text, thinking, tool calls, subagents) streams to the host asSessionEvents and into the journal for viewers. Usage reports arrive everyagent.usage_interval_secs. - Idle shutdown or sleep. Every delivery, guest event and status resets the idle timer
(usage reports don’t). When
vm.idle_shutdown_secspass without any while the guest isn’tbusy/provisioning, or when the agent callssleep_vm, the host sendsShutdown; the guest stops the harness, syncs and reboots, which makes Firecracker (reboot=k) exit.RunningVm::stopwaits up to 30 s, then kills, and removes the tap, sockets and chroot; the disk stays. The slot and the memory permits are released. The session journalsstoppedand sleeps until the next delivery. - Restart.
restart_vmendsrun_vmthe same way, journalsstopped "restarting", and the session boots again right away withVmControl::desired()(a new grant takes effect here). - Failure. A failed prepare/boot/handshake, an unexpected VM exit or a failed control write
ends
run_vmwith an error: the session journalscrashed, keeps queued deliveries, waits5 s · 2^(failures-1)(capped at 160 s; commands are still answered), and boots again. After the third failure in a row it tells the conversation an operator needs to look. A clean stop resets the count. - Leaving. When the bot leaves a group (
MembersChangedwith the bot inleft, orcheck_invitemakes it leave viaPlatform::leave),Router::closesendsCommand::Closeand revokes every watch token of the conversation.
Resource management
Section titled “Resource management”Agents know their VM’s memory and disk, are told to clean up when either runs high, and can ask the host for a different allocation, restart to apply it, or put their VM to sleep. The host decides from the configuration and its own capacity.
flowchart LR
subgraph Guest
MON["resource monitor<br/>sample every usage_interval_secs"]
AG[agent / omp]
end
subgraph Host
VC["VmControl"]
DEC["machine.decide"]
AP["request_approval<br/>(grants = approval)"]
FS[("vms/key/allocation.json")]
SS["session.run"]
BUD["memory budget semaphore"]
end
MON -- "ResourceUsage (vsock 1024)" --> VC
MON -- "alert: urgent steer + NOTICE" --> AG
AG -- "vm_resources" --> VC
AG -- "request_resources{totals, reason}" --> VC
VC --> DEC
DEC -- "needs approval" --> AP
DEC -- "grant" --> FS
AP -- yes --> FS
AG -- "restart_vm / sleep_vm" --> VC
VC -- "Command.Restart / Sleep" --> SS
SS -- "acquire allocation.memory_mib" --> BUD
SS -- "prepare_with(allocation): grow disk,<br/>vcpu_count, mem_size_mib" --> VMM[VmManager]
Measuring and alerting (guest agent’s resource monitor). After HostInit the monitor
samples every usage_interval_secs (HostInit field 15; at least 5 s, 30 s if 0): MemTotal and
MemAvailable from /proc/meminfo, statvfs("/") (free = what an unprivileged user may still
use), the 1-minute load from /proc/loadavg, and online CPUs. Each sample goes to the host as
GuestFrame.usage. When disk use reaches disk_alert_percent (85) or memory use
memory_alert_percent (90), the monitor sends the harness one urgent delivery (an omp steer
while busy, a prompt while idle) with the numbers and concrete cleanup steps (alert_message:
remove recreatable build outputs and caches, apt-get clean, docker system prune, /tmp,
stream data, stop processes; only then request_resources and restart_vm), and journals it as
an error EVENT_NOTICE "resources". Each alert re-arms once use drops 5 points below its threshold
(Alarm, HYSTERESIS); disk and memory are independent.
Reporting (host, VmControl). VmControl::record_usage keeps the latest
sample; vm_resources returns the allocation the VM runs with, any grant waiting for a restart,
the per-VM maximum (vm.max_vcpus, min(vm.max_memory_mib, budget), vm.max_disk_gib), the
usage as percentages with its age, and the grant mode. /status shows the same summary, e.g.
asleep · 2 vCPU, 2048 MiB, 10 GiB or working · 2 vCPU, 2048 MiB, 10 GiB · memory 75% · disk 90%.
Deciding (machine::decide). request_resources gives totals (None keeps a value). The
checks run in this order:
- no change →
Unchanged(reported as granted); - a disk smaller than the disk’s current size → denied (“disks only grow”);
- fewer than 1 vCPU or 128 MiB → denied;
- above
vm.max_vcpus/vm.max_memory_mib/vm.max_disk_gib→ denied; - memory above the host’s memory budget → denied;
- nothing grows (giving resources back) → granted, never needs approval;
vm.grants = "never"→ growth denied;- disk growth that would leave less than
vm.min_free_disk_gibfree on the filesystem holdingdata_dir(Capacity::disk_free,statvfs) → denied; - otherwise granted; with
vm.grants = "approval"the conversation’s approvers must say yes first (HostConversation::request_approvalwith the new and current totals and the agent’s reason; no answer means no).
Every grant, denial, restart and sleep is journaled as EVENT_NOTICE "resources".
Persisting and applying. A grant is written to <data_dir>/vms/<key>/allocation.json (0600,
via a .tmp file, fsync, rename) and becomes VmControl::desired. On load a saved grant is
clamped to today’s limits (vCPUs to 1..=max_vcpus, memory to 128..=min(max_memory_mib, budget),
disk never below vm.disk_gib and never shrunk). The grant applies when the VM next boots:
restart_vm asks the session to restart about one second later (so the tool’s answer reaches the
agent first); the session then acquires a slot and the new memory in the budget, and
VmManager::prepare_with grows the disk offline (ensure_disk / grow_disk: set_len,
e2fsck -pf, resize2fs) and writes vcpu_count / mem_size_mib into the firecracker config.
sleep_vm stops the VM about one second later and frees its slot and memory; the next message
boots it again.
The system prompt (policy::system_prompt) and the MCP INSTRUCTIONS tell the agent its
resources are limited and shared, to check vm_resources before big work, to delete what it can
recreate, to clean up first on an alert, to request more only when the work truly needs it and
then restart, and to tidy up and sleep_vm when done.
Control plane and metering
Section titled “Control plane and metering”The site Gopala serves is a self-hosted control panel: a dashboard at /dashboard/ (live state,
actions, usage), a configuration editor at /configure/, the docs at /docs/, and the front page
at /. The pages are static Astro builds with Alpine.js; everything live comes from the API in
the host’s web server, backed by control::Control (protocols.md).
They are embedded in the gopala binary; when server.site_dir (default
/usr/share/gopala/site) exists as a directory, that directory is served instead.
flowchart LR
subgraph Browser
D["dashboard /<br/>Alpine.js"]
C["/configure/"]
end
subgraph Host["gopala run"]
W["web server<br/>session cookie or Bearer"]
CT["Control"]
REG["Shared.registry<br/>Entry per conversation"]
ACT["Shared.activity<br/>broadcast"]
LG[("usage.db")]
CFG["LiveConfig<br/>ArcSwap"]
R["Router"]
end
D -- "GET /api/state, /api/stream (SSE)" --> W
D -- "POST …/abort, sleep, restart, leave<br/>PUT …/allocation" --> W
C -- "PUT /api/config" --> W
W --> CT
CT --> REG
CT --> LG
CT -- "apply: hot sections" --> CFG
CT -- "Leave" --> R
ACT -- "activity events" --> W
Live state. Control::state builds one snapshot: the host (memory budget and how much running
VMs hold, slots, free disk under data_dir), the base and maximum allocation and grant mode, the
platforms with their bot usernames, and every conversation in Shared::registry with its status
(the session’s state, set through VmControl::set_status, including failed with the error after
a failed boot), allocation and pending grant, measured, reported, ledger totals and today’s
usage, and last activity. /api/stream sends that snapshot when it changes (checked every 2 s)
and every journaled event as it happens (Shared::activity, fed by
HostConversation::journal_event), so the dashboard needs no polling.
Actions. Control::act sends the session Command::Abort, Sleep or Restart (restart
boots an asleep VM), or asks the router to leave (RouterCommand::Leave: Platform::leave, then
the conversation is closed and its watch tokens revoked), and journals an operator notice. An
allocation set from the dashboard goes through VmControl::operator_set: the same
machine::decide as an agent’s request, but with the grants policy treated as auto and no
approval; the per-VM maximums, the memory budget, the host disk reserve and “disks only grow”
still apply, and the new allocation takes effect at the next boot.
Measured, not reported. The agent may misreport itself, so every usage figure the dashboard shows comes from outside the VM:
| What | Measured by | Where it ends up |
|---|---|---|
| Model requests, input/output/cache tokens, bytes up/down, errors | The LLM proxy: one LlmCall per request, reported when the response ends, fails or the guest drops it. Tokens come from the provider’s own usage report in the response, parsed as it streams (Anthropic Messages, OpenAI Chat Completions and Responses, Gemini; JSON or server-sent events; at most MAX_PARSED = 8 MiB held at once). The proxy sends accept-encoding: identity upstream so reports stay readable; an encoded response is forwarded untouched and counted with zero tokens. |
Ledger; one EVENT_LLM journal event per call (title: upstream and model; text: status, duration, tokens) |
| CPU time, memory, disk, network | RunningVm::sample every 5 s (Timing::sample) and when the VM stops: utime + stime and VmRSS of the firecracker process from /proc/<pid>, the disk image’s allocated blocks, the tap’s byte counters (network = "nat" only) |
measured (CPU percent since the previous sample, RSS, disk, network); CPU seconds and VM seconds in the ledger |
| Tool calls | HostConversation counts each call of a chat MCP tool (conversation_info, read_messages, send_message, edit_message, delete_message, react, search_messages, request_approval) |
Ledger tool_calls |
| Messages sent | Every chat message the bot sends through HostConversation::send_text (the agent’s parts, command answers, approval outcomes) |
Ledger messages_sent |
| Boots | session::run before each boot |
Ledger boots |
| Egress bytes | The egress proxy and the HTTP MCP proxies count the bytes of every tunnel, forwarded request and response (egress::meter) |
Ledger egress_bytes_up, egress_bytes_down |
| Git requests and bytes | The git proxies count every request forwarded to a code host and the bytes each way | Ledger and monitor totals git_requests, git_bytes_up, git_bytes_down |
What the guest says about itself (GuestFrame.usage) is shown separately as reported and labelled
unverified; the dashboard flags it when it differs from the host’s measurement by more than 20
points. The harness’s own tool executions (shell commands and the like) appear only in the
journal as the guest reports them; they are not metered.
The ledger (usage::Ledger) adds deltas in memory (Ledger::add never
waits on disk) and flushes them to <data_dir>/usage.db (SQLite, 0600) every 30 s and at
shutdown: one row per conversation key and UTC day (usage_days, pruned after KEEP_DAYS = 400
days) plus all-time totals per key (usage_totals). /api/usage?days=N (1–90) returns the
daily rows.
Hot reload. The running configuration is a LiveConfig (Arc<ArcSwap<Config>>). PUT /api/config saves the file and calls Control::apply, which merges the safe sections of the new
file into the running one (control::merge): behavior, approvals, agent, budget, alerts,
monitoring, collaboration, and the VM limits vcpus, disk_gib, memory_mib (only if it
fits in the memory budget fixed at start), max_vcpus, max_memory_mib, max_disk_gib, grants,
idle_shutdown_secs, boot_timeout_secs, min_free_disk_gib. If the merged configuration
doesn’t validate, nothing is applied. Every other differing top-level section is reported as
restart_required, including memory, vm.network and vm.allowed_hosts. Readers pick the new
values up where they load the config: policy (including listening mode), alerts, monitoring
rules, grant and budget decisions immediately; collaboration.enabled and
max_agent_messages_per_hour with the next message, and list_agents asks with the current
agent_role (cached answers expire within 5 minutes); agent (harness, tools, MCP servers, the
system prompt with its collaboration section), approval timeouts, idle shutdown and boot timeout
at a VM’s next boot. Two exceptions need a restart of Gopala: the role that marks incoming
messages as from_agent (each provider gets agent_role once, at startup, through
set_agent_role), and an HTTP MCP server added by a reload that needs a secret_env, because
secrets are only read at startup (egress::live_mcp). Two things stay as they were for
conversations that already exist [inference from the code]: the per-minute chat rate limit (each
conversation’s limiter is built once, in HostConversation::new) and the base allocation
(VmControl::new computes a conversation’s starting allocation once); lowered maximums do apply
to new requests.
Network
Section titled “Network”vm.network picks one of three ways out of a VM (gopala_config::Network). Model calls and the
chat MCP server use vsock in every mode, so the agent can always think and talk.
| Mode | VM device | How programs reach the internet | Host setup |
|---|---|---|---|
nat (default) |
eth0 on tap gpl<n>, a /30 per slot, ip= and gopala.dns= on the kernel command line |
directly, through the host’s NAT; nothing filters it | IP forwarding and a masquerade rule (operations.md) |
proxy |
none | http_proxy/https_proxy = 127.0.0.1:3128, forwarded over vsock 1090 to the host’s egress proxy, which allows only vm.allowed_hosts and only public addresses |
nothing |
none |
none | not at all | nothing |
flowchart LR
subgraph VM
P["apt, git, curl, pip<br/>(honor http_proxy)"] --> F["127.0.0.1:3128<br/>guest forwarder"]
end
F -- "vsock 1090" --> E["egress::Egress<br/>(one per VM)"]
E --> A{"host:port in<br/>vm.allowed_hosts?"}
A -- no --> R403["403, notice 'refused'"]
A -- yes --> D["resolve on the host"]
D --> PUB{"every address<br/>public?"}
PUB -- no --> R403
PUB -- yes --> C["connect to a vetted address<br/>(CONNECT tunnel or http forward)"]
C --> NET[(the server)]
In proxy mode only programs that honor the proxy variables get out; anything that opens its
own sockets fails, since the VM has no network device. vm.allowed_hosts entries are host,
*.domain (subdomains only) or either with :port; without a port 80 and 443 are allowed.
Config::validate requires a non-empty list in proxy mode and refuses agent.packages with
network = "none". The proxy’s protocol and limits are in
protocols.md, its threat model in
security.md.
Code hosting (git proxy)
Section titled “Code hosting (git proxy)”Each [[git]] entry (at most 8) gives every VM one account on GitHub, GitLab or Gitea without
the token entering the VM, in every vm.network mode including none. The host’s git proxy for
entry i listens on vsock 1300+i (VSOCK_GIT_BASE_PORT); the guest agent forwards
127.0.0.1:3300+i (GUEST_GIT_BASE_PORT) to it, announced as GitForward in HostInit.git.
At boot the guest writes /etc/gopala-git.conf (included from /etc/gitconfig), whose
url.insteadOf rules rewrite https://<host>/, git@<host>: and ssh://git@<host>/ to the
forwarder, and sets GOPALA_GIT_<NAME>_API (http://127.0.0.1:<port>/api, /api/v4 or /api/v1)
for REST calls with curl; gh, glab and GraphQL are not supported.
The proxy accepts only git smart HTTP (info/refs, git-upload-pack, git-receive-pack) and
the provider’s REST API, adds the token (read from token_env at startup and removed from the
environment), drops the guest’s Authorization and Cookie headers, strips credential headers
from responses and doesn’t follow redirects. With repos set, every git and API path must name
an allowed repository. Pushes are parsed and refused whole if they create, update or delete a
branch in protected_branches (default main, master); merging pull or merge requests is
always refused, and so are API writes that would change a protected branch. Pushes, deleted
branches, tags, opened pull requests and refusals appear as git notices in the journal.
Limits, protocol details and the threat model are in
protocols.md and security.md.
Token budgets
Section titled “Token budgets”[budget] caps model tokens per day and per month (UTC), for each conversation and for the host
(the host’s budgets). Tokens are counted as input + output + cache-write (budget::tokens;
cache reads are free) from the ledger the LLM proxy books into.
- A conversation’s limits and priority are its
[[budget.conversations]]entry, else[budget.default](budget::effective). 0 means no limit. - The host limits (
budget.daily_tokens,budget.monthly_tokens) apply by priority (budget::share): low-priority conversations stop atlow_stops_at_percent(70) of the host budget, normal atnormal_stops_at_percent(90), high at 100%. High priority keeps the last slice for the conversations that matter most. budget::decideruns before every model call (llm_proxyAdmit, wired bybudget::gate). Over a limit, the call gets a 429budget_exceededwith a plain reason and never reaches the provider; the agent’s harness sees the error, and the journal gets onebudgetnotice per blocked stretch. People in the conversation get a host event (“⛔ I’ve stopped: …”, see Alerts) once per episode for the conversation’s own budget, and once per UTC day when the host budget stopped it (budget::Announced).- Priority also orders who gets a VM slot and host memory first (
session::Permits, see VM slots and the memory budget).
The dashboard shows each conversation’s budget and why it’s blocked, and changes it through
PUT /api/conversations/{key}/budget (protocols.md), which rewrites
that conversation’s [[budget.conversations]] entry with toml_edit and applies it at once.
[budget] also hot-reloads. The check runs before each call, so calls already in flight finish
and can take a conversation past its limit [inference from the code].
Agent tools and MCP servers
Section titled “Agent tools and MCP servers”agent.toolslimits the harness’s built-in tools: empty keeps its defaults; a list becomes--tools=a,bfor omp (omp’s tool names) and Claude Code (Claude Code’s, such asBash,Read,Edit) (agent_config::tool_args). If none of the names is valid the guest passes--no-toolsto omp and--tools=to Claude Code rather than widen the list. The exec harness ignores it.[[agent.mcp]]adds MCP servers next tochat. Acommandserver (stdio) runs inside the VM as the agent user, so itsargsandenvare visible to the agent: never put a secret there. Aurlserver (http or https) is reached through the host on vsock1200+i(egress::serve_mcp) in every network mode, and its secret (secret_env, sent as<auth_header>: <auth_prefix><secret>) stays on the host. Names are[a-z0-9_-]{1,32}, unique, and neverchat. A command can come from the image: install it withimage.setup(see Images) and name it here.- The agent’s own servers come from a file it edits itself; see The agent’s own MCP servers.
The operator’s servers reach the guest in HostInit (tools, mcp, mcp_forward); the guest
writes them, with the agent’s, into the harness’s MCP config before every start
(agent_config::base, agent_config::merge). See
protocols.md.
Harnesses
Section titled “Harnesses”agent.harness picks the program that is the agent. All three run as the agent user with the
same clean environment, the same system prompt and the same MCP servers; they differ in how long
they live and how messages reach them.
omp (default) |
claude_code |
exec |
|
|---|---|---|---|
| Runs | omp --mode rpc, one session per VM |
claude -p with stream-json in and out, one session per VM |
agent.command once per message, prompt on stdin |
Empty agent.command means |
omp |
claude |
sh |
| Session after a reboot or restart | continues (-c) |
continues (--continue) |
none; each run starts fresh |
| Messages while working | follow_up, or steer when urgent |
a user message with "priority": "later", none when urgent |
queued until the run ends |
/stop |
omp’s abort |
the process is stopped and started again with --continue |
the process group is killed |
| Harness-level approvals | omp’s confirm/select prompts become chat approvals | none (bypassPermissions); request_approval only |
none |
agent.tools |
--tools=… with omp’s names |
--tools=… with Claude Code’s names |
ignored |
restart_harness |
restarts, keeps the session | restarts, keeps the session | refused: nothing to restart |
| From the tools image | pinned omp (always bundled) | pinned Claude Code when agent.harness = "claude_code" |
whatever you put in [image] |
| Wire protocol | omp RPC | stream-json | stdin and stdout lines |
omp is the general-purpose default: subagents, steering and follow-ups, and an extension-UI
protocol the guest maps onto chat approvals. Its default agent.args are --approval-mode yolo,
so it doesn’t stop for its own tool approvals. It’s temporary: a gopala-owned harness is to
replace it (issue #5).
Claude Code is a coding agent. It reaches Anthropic through an [[llm]] proxy like any other
model call (ANTHROPIC_BASE_URL in that entry’s guest_env), so the key stays on the host and
the tokens are metered and budgeted. It runs with --permission-mode bypassPermissions: the VM
is the sandbox, and approvals for critical steps go through the request_approval chat tool, as
the system prompt says. Claude Code refuses that mode as root; the agent account isn’t root. The
guest sets CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 (overridable through agent.env) and
passes agent.args after its own flags, e.g. ["--model", "sonnet"]. gopala image build
bundles the pinned release (2.1.319, checked against the release manifest and its sha256, about
340 MB) into the tools image when agent.harness = "claude_code" (CLAUDE_CODE=1 for
build-tools-image.sh); installing it with image.setup works too. Never install it with
agent.user_setup: the agent could then replace its own harness.
exec runs any command per message, for example codex exec. The command gets the system
prompt, a blank line and the message on stdin; its stdout lines become EVENT_TEXT. It has no
session between messages, no approvals and no steering. The guest still writes the merged MCP
config to /home/agent/work/.mcp.json (and omp’s path) for commands that read it.
The system prompt
Section titled “The system prompt”policy::system_prompt builds the instructions every harness gets, in sections: who the agent is
(its name, the platform, the group or direct chat, its VM and workspace), Talking to people
(only chat-tool messages are visible, keep replies short, severities), Working (work
autonomously, approvals only for critical steps, look before installing, prefer uv and pnpm),
Resources (limits, cleanup, request_resources, budgets, sleep_vm), Extending yourself
(adding MCP servers and restart_harness, plus the sudo or package-manager line), Memory
(when memory.enabled), Working with other agents (only in a group with [collaboration]
on), and last From the operator with agent.system_prompt.
The operator’s text is appended under its own heading and never replaces the built-in sections;
a blank one adds nothing. The guest writes the result to /home/agent/.gopala/system.md. omp
gets it with --append-system-prompt and Claude Code with --append-system-prompt-file, so each
keeps its own default system prompt underneath and Gopala’s comes after it. exec gets it at the
top of every run’s stdin. The prompt travels in HostInit, so a changed agent.system_prompt or
[collaboration] reaches an agent at its VM’s next boot.
Restarting the harness
Section titled “Restarting the harness”restart_harness {reason} restarts the harness inside the running VM. Nothing else restarts: the
VM, its allocation, the workspace and the session stay.
- The tool trims the reason and cuts it to 300 characters; an empty one is refused.
VmControl::restart_harnessrefuses an exec harness, journals anEVENT_NOTICE "harness"with the reason, answers, and about 1 s later sendsCommand::RestartHarness, so the answer reaches the agent before its harness stops. - The session forwards it as
HostFrame.restart_harnesswhen the guest is connected. Without a VM it’s ignored; while the guest is reconnecting it’s dropped with a warning. - The guest journals lifecycle
restarting(“restarting harness:”), sends SIGTERM and kills the process if it hasn’t exited after 10 s. It rewrites the MCP config, then starts the harness again at once with -cor--continue. This isn’t a crash: no backoff, nocrashedevent, and a pending crash backoff is cut short. - Deliveries that arrived meanwhile stay queued. The new run’s first prompt says “Your harness restarted (reason: …) and this session goes on.”, followed by what became of the agent’s MCP file (“From /home/agent/.gopala/mcp.json: loaded web; left out …”).
restart_vm is the heavier sibling: it reboots the VM to apply a resource grant.
The agent’s own MCP servers
Section titled “The agent’s own MCP servers”An agent can add MCP servers without the operator. The system prompt tells it how: install the
server, add it to /home/agent/.gopala/mcp.json in the usual mcpServers format, then call
restart_harness. The guest reads that file before every harness start (first boot, a crash
restart, restart_harness) and merges it into the generated config (agent_config::merge):
- The file. Created as
{"mcpServers": {}}(0600, owned by the agent) the first time it’s missing. It’s read withsys::read_owned: no symlink anywhere in the path, owned by the agent’s account, at most 64 KiB. A file that fails any of these, isn’t JSON, or whosemcpServersisn’t an object is left out whole. - Entries.
{"command", "args"?, "env"?}for stdio or{"url"}for streamable HTTP, with an optional"type"ofstdioorhttp; other keys are ignored. Names follow the operator’s rule ([a-z0-9_-]{1,32}), URLs must be http(s), and nothing may contain a NUL byte. - Reserved names.
chat,memory(even when the built-in memory server is off) and every[[agent.mcp]]name, valid or not, belong to the host. An entry using one is skipped, so the agent can’t replace a server the operator configured. - Limits. At most 32 servers are taken; invalid entries are skipped one by one. Each skip is logged and listed, with its reason, in the note the agent gets after a restart.
- Where it goes. omp and exec:
/home/agent/.omp/agent/mcp.jsonand/home/agent/work/.mcp.json. Claude Code:/home/agent/.gopala/claude-mcp.json, passed with--mcp-configand--strict-mcp-config, so a stray.mcp.jsonin the workspace doesn’t count. The guest writes these as root withsys::write_owned, which refuses a symlink anywhere in the path (openat2withRESOLVE_NO_SYMLINKS) and hands the file to the agent.
The agent’s servers run inside the VM with the agent’s own rights, like anything else it starts.
An HTTP server it adds isn’t proxied by the host: the harness connects to that URL directly, so it
follows vm.network (anything with nat, the allowed hosts with proxy, nothing with none),
and no secret is added for it.
Images
Section titled “Images”A VM boots from three files: vm.kernel, vm.rootfs (copied once per conversation into its
persistent disk) and, optionally, the read-only vm.tools_image. gopala image build
(image::build) makes them from [image] with the build scripts embedded in the binary
(build-rootfs.sh for the rootfs, build-tools-image.sh for the tools image) and the embedded
static guest agent, which it bakes into the rootfs with its systemd unit. The kernel is the pinned
Firecracker CI kernel the rootfs script downloads. --guest FILE and --scripts DIR replace
the embedded guest or scripts, for special cases only:
[image] |
Passed to | Effect |
|---|---|---|
base (ubuntu-24.04 default, debian-12) |
build-rootfs.sh as BASE |
Ubuntu: Firecracker’s CI image; Debian: debootstrap. Writes gopala-<base>.ext4 |
packages |
build-rootfs.sh as PACKAGES |
apt packages baked into the rootfs in a chroot (needs root) |
setup |
build-rootfs.sh, one --setup <command> argument each |
shell commands run as root with bash -c in the chroot after the packages, in order, with a clean PATH and the build host’s http(s)_proxy/no_proxy when set (needs root); a failing command stops the build. What they install is in every VM from its first boot |
rootfs_size_gib (4) |
build-rootfs.sh as ROOTFS_SIZE |
the rootfs size; a conversation’s disk grows from it to vm.disk_gib |
tools (["git"]) |
build-tools-image.sh as TOOLS |
host programs bundled next to omp with their shared libraries, into gopala-tools.ext4 and its .sha256 |
agent.harness = "claude_code" |
build-tools-image.sh as CLAUDE_CODE=1 |
the pinned Claude Code as bin/claude on the tools drive |
image::build sets every variable explicitly, so a stray PACKAGES, TOOLS or CLAUDE_CODE in
the caller’s environment can’t change the build, and passes each setup command verbatim as its
own argument (no shell string is built on the host). It then prints the [vm] lines: the newest
kernel in the output directory, the rootfs, the tools image and its SHA-256. [image] is only
read by this command; a changed [image] takes effect when you build and point [vm] at the new
files. Packages and setup baked into the image cost nothing at boot, while agent.packages and
agent.setup run when each VM provisions (packages need network).
Alerts
Section titled “Alerts”The agent decides how serious a message is; the operator decides who that reaches. send_message
takes severity = info (default), warning or critical, and nothing else about notifying:
no platform, no names, no mention syntax (gopala_mcp::Severity).
flowchart LR
A["agent: send_message<br/>severity = critical"] --> S["HostConversation::send"]
H["host event: VM keeps failing,<br/>budget block"] --> E["HostConversation::host_event<br/>(alerts.host_events)"]
S --> AL{"alert(severity)"}
E --> AL
AL -- "info, direct chat,<br/>or empty audience" --> P1["Platform::send"]
AL -- "hourly cap used up" --> P2["Platform::send<br/>+ note to the agent"]
AL -- "audience" --> P3["Platform::send_notifying"]
HostConversation::audience picks the entries: the conversation’s [[alerts.conversations]]
entry if there is one, else alerts.warning / alerts.critical (defaults: nobody for warning,
["everyone"] for critical). Audience::parse reads everyone, here (people who are online,
where the platform knows), role:<name> and user:<username>. Info messages notify nobody, and
neither does anything in a direct chat, where the one person gets the message anyway.
alerts.max_per_hour (default 4) caps notifying messages per conversation, both severities
together and host events included (HostConversation::take_alert, a governor limiter rebuilt
when the setting changes). Past the cap the message still goes out, through plain
Platform::send, and the tool result carries note: "sent without notifying anyone: …". Only the
first part of a split message notifies; later parts are ordinary sends.
Host events go out at alerts.host_events severity (default critical), bypassing the agent’s
per-minute rate limit but not the hourly cap:
| Event | Text | How often |
|---|---|---|
Three VM failures in a row (session::run) |
“⚠️ My workspace keeps failing to start; an operator needs to look at it.” | each time the retry loop reaches it |
The conversation’s own budget stops model calls (budget::gate) |
“⛔ I’ve stopped: This conversation used its daily budget …” | once per episode, until a call is allowed again |
| The host budget, at this conversation’s priority share, stops model calls | “⛔ I’ve stopped: The host used 90% of its daily budget …” | once per UTC day per conversation |
Each provider renders the audience in its own syntax and lets exactly those mentions ping
(Platform::send_notifying). Names the platform doesn’t know are logged and left out; the
message still goes out.
| everyone | here | role:<name> |
user:<name> |
|
|---|---|---|---|---|
| Slack | <!channel> |
<!here> |
<!subteam^ID>, the user group found by handle in usergroups.list (cached 5 min; needs the usergroups:read scope) |
<@U…> |
| Discord | @everyone |
@here |
<@&role>, by role name |
<@id>, from the cache or a member search |
| Lupyd | @everyone as plain text |
@here as plain text |
skipped (Lupyd has no role mentions) | @username, checked against the group’s members |
Default (Platform trait) |
the audience as plain text (Audience::plain_prefix) in front of send |
On Slack the mentions go in a mrkdwn section block, the message in a markdown block, and the
top-level text (used for the notification) holds both; messages over 12,000 characters go out as
escaped mrkdwn with the mentions in front. On Discord the first part carries the mentions with
allowed_mentions set to exactly them (everyone covers both @everyone and @here), and the
bot needs “Mention @everyone, @here and All Roles” in the channel for those to ping (logged when
it lacks it). Ordinary Discord messages never ping everyone or roles.
[alerts] hot-reloads. Severity also shows in the journal: outgoing messages are titled
message (warning) or message (critical).
Listening mode
Section titled “Listening mode”behavior.respond_to sets who wakes the agent everywhere; [[behavior.conversations]] entries
(key, respond_to) override it for one conversation (policy::respond_to, used by
Router::handle for both the dms check and policy::triggers). respond_to = "all" in one
group is “listening mode”: every message reaches its agent, mentioned or not.
Two ways to switch it at run time, both through control::set_respond_to:
- In the chat:
/listen all(oron) and/listen mentions(oroff), from an approver (Router::is_approver, the same check as/stop). Not in direct chats, which already deliver everything. The bot answers “👂 Listening: …” or “🔕 Only messages that mention me …”, and the journal gets alistennotice naming who changed it. - On the dashboard: a toggle per group,
PUT /api/conversations/{key}/behavior; journaled as anoperatornotice.
set_respond_to edits that conversation’s [[behavior.conversations]] entry in gopala.toml
with toml_edit (control::edit_conversation_entry, so comments and layout stay), validates the
result, writes it atomically with a .bak, and only then swaps it into the live configuration.
All writers of the file go through one control::ConfigFile and its lock: PUT /api/config,
budget edits, the dashboard toggle and /listen, so two saves never interleave. If saving fails
the chat answer says an operator needs to look at the configuration and nothing changes.
In a group whose respond_to is dms, group messages are ignored before commands are parsed, so
/listen can’t switch it from the chat; the dashboard can.
Collaboration
Section titled “Collaboration”A group can hold more than one agent: another Gopala signed in with its own bot account, any
other bot, or an account someone marked as an agent. [collaboration] lets them find each other,
ping each other with @mentions and split the work, while each one decides for itself what to do
with a ping. The host decides who counts as an agent, which of their messages wake this one and
how often, and what they can never do.
Who’s an agent. Bot accounts (Slack bot users, Discord bots) and members holding the role
named collaboration.agent_role (default agent): a Lupyd group role, a Discord role, a Slack
user group handle. Give that role to your own bot too, so others can find it. Providers mark such
messages Incoming.from_agent and answer Platform::agents; what each platform counts and where
the bio comes from is in providers.md.
How a split goes. A person asks agent A; A looks for help, pings B with a request that stands on its own, and both work at once:
sequenceDiagram
participant P as alice (a person)
participant A as agent A (this bot)
participant H as A's host
participant B as agent B (another bot)
P->>A: @A compare our API latency with last month and write it up
A->>H: list_agents
H-->>A: researcher: "pulls metrics and papers" (Platform::agents, cached 5 min)
A->>P: I'll write it up, and @researcher is getting last month's numbers
A->>B: @researcher please get p50/p99 for October from the metrics API, reply with a table
Note over B: its own host: from_agent, mentions B, within B's hourly cap
B->>B: decides: help, decline or say it's busy
par A drafts its part
A->>A: this month's numbers, the write-up
and B works
B->>B: October's numbers
end
B->>A: @A here's the table: …
Note over H: from_agent + mention, within max_agent_messages_per_hour:<br/>delivered as "researcher (agent) … [message N]"
A->>P: the comparison and the write-up (A owns the answer)
What wakes an agent. A message from another agent wakes this one only if all of these hold
(Router::on_message):
collaboration.enabledis true (with it off, agents’ messages are ignored andlist_agentserrors);- the conversation is a group;
- the message mentions the bot or replies to one of its messages (the provider filters first; the router checks again);
- the conversation has a wake-up left this hour:
collaboration.max_agent_messages_per_hour(default 30, at most 3600, 0 = none) per conversation, counted by a governor limiter that is rebuilt when the setting changes (HostConversation::take_agent_message). Past it, the message is dropped and logged. People’s messages don’t count.
Listening mode never applies to agents, and an agent’s message is never parsed as a command or
an approval answer; its reactions don’t answer approvals either. Messages that pass are delivered
with the sender marked: researcher (agent) in #general (channel 1) [message N]: (see
protocols.md). Messages the provider drops never reach the
conversation’s EventLog, but read_messages still shows them, since it reads the platform’s
history.
What the agent is told. In a group with collaboration on (decided when the VM boots), the
system prompt adds “Working with other agents”: the group is a shared workspace; list_agents
shows who else is here and what they’re good at; an @mention is a ping, not an order, and the
agent may decline or say it’s busy; hand a part that suits another agent better to it with a
self-contained request (goal, inputs, what to send back) and keep working in parallel; say in the
chat who does what; the agent the person asked owns the final answer; and don’t mention agents
just to thank or acknowledge them, because messages between agents are capped. The hourly cap is
what stops two agents from keeping each other busy forever; the prompt only makes hitting it
rare.
list_agents takes no parameters and returns at most 50 agents (username, display_name,
about up to 2000 characters, roles, bot), cached per conversation for 5 minutes and dropped
when members or the group change; details in protocols.md. [collaboration]
hot-reloads (see Control plane and metering), except that a changed
agent_role reaches the providers’ from_agent marking only after a restart of Gopala.
Memory
Section titled “Memory”Each agent keeps facts across restarts in its own VM: the memory store gopala-memory stores
them in /home/agent/.gopala/memory.db on the conversation’s persistent disk, and
gopala-guest memory serves them to omp as the memory MCP server (tools in
protocols.md). The host only passes settings
(HostInit.memory, built by session::memory_config from [memory]) and proxies model calls; it
never reads or stores the memories. The system prompt tells the agent to recall before work that
may build on earlier context and to remember durable facts (policy::system_prompt, only when
memory.enabled).
The design follows mem0’s useful parts (atomic facts, dedupe on write, entity links, hybrid retrieval) without a model on the write path: the agent itself writes short facts, and a model runs only as an optional, bounded compaction job.
flowchart LR
subgraph VM["inside the VM, as the agent user"]
O[omp] -- "remember / recall / …" --> MS["gopala-guest memory<br/>(one per omp process)"]
MS --> DB[("memory.db<br/>SQLite WAL + FTS5")]
SUP["gopala-guest run<br/>(supervisor)"] -- "every cleanup interval" --> MP["gopala-guest memory --maintain"]
MP --> DB
end
MS -. "/v1/embeddings" .-> PX["host LLM proxy<br/>(key, metering, budget)"]
MP -. "/v1/embeddings, compaction" .-> PX
Storage. One SQLite file, 0600 in a 0700 directory, owned by the agent; WAL, a 10 s busy
timeout and short IMMEDIATE write transactions, because omp’s subagents start their own memory
servers on the same file. Tables: memories (text, normalized-text hash, importance 0–1, pinned,
recall count and time, archived time), an external-content FTS5 index (porter unicode61),
embeddings, entities (one per case-insensitive name), memory_entities, relations (unique
per from, relation, to), meta and a compactions log of the last 100 runs. Migrations go by
PRAGMA user_version.
Writing. remember normalizes the text, then merges instead of inserting when it finds an
exact duplicate (same normalized text, case-insensitive) or a near one: word-set Dice over the 10
best BM25 matches, or cosine when embeddings are on, at duplicate_threshold (0.85) or above.
Entities and relations are linked in the same transaction. Limits: text 2 KiB, names 128 B, 16
entities and 16 relations per memory.
Recall fuses up to three ranked lists with reciprocal rank fusion (k = 60): FTS5 BM25, cosine over stored vectors (brute force), and memories of entities the query names. Each hit’s score is then multiplied by 1.5 for an entity match, by (0.5 + importance) and by (0.5 + 0.5 × recency). Results are capped at 50, and each hit counts as a use.
Cleanup runs in the guest supervisor as the agent user (memory::spawn_maintenance runs
gopala-guest memory --maintain as a child process, so root never opens files in the agent’s
home): first 120 s after boot, then every max(cleanup_interval_secs, 60) seconds, killed after
30 minutes. A memory’s score is
score = importance × (1 + ln(1 + recalls)) × 2^(−age since last use / half_life_days)| Rule | Effect |
|---|---|
More than max_memories active (checked on every write and in each pass) |
the lowest scores are archived down to 90% of the cap; archived memories drop out of recall |
Database over max_db_mib |
the whole archive is deleted first, then the lowest scores are archived and deleted down to 90% of the size cap |
Archived longer than archive_days |
deleted for good |
| Pinned | never archived or evicted |
| Near duplicates | a sweep compares up to 50 memories in each of 50 entities per pass and merges them |
| Also each pass | orphan entities dropped, FTS5 optimize, incremental vacuum, WAL checkpoint, and up to 256 missing embeddings filled in (batches of 32) |
Embeddings ([memory.embeddings], optional) are POST /v1/embeddings to the guest port of
the named [[llm]] entry, so the host adds the key, meters the tokens and applies the budget. A
tool call waits at most 8 s for one and goes on without it; the next pass backfills.
Compaction ([memory.compaction], off unless set) runs from the maintenance pass every
every_hours once there are min_memories active memories. It builds up to max_clusters
clusters of 2 to 40 related memories (at most 12,000 characters each): first memories sharing an
entity, then pairs at cosine 0.75 or more among the 500 newest embedded memories. Each cluster
goes to the model (Anthropic /v1/messages or OpenAI /v1/chat/completions, format), which
answers with merge, rewrite, delete or keep operations. Answers are validated strictly
(only ids from the cluster, each once, pinned memories only kept, text within limits), every
valid plan is applied in one transaction, and a failed call, a bad answer or a cluster that
changed meanwhile leaves the store as it was. Compaction tokens go through the same host proxy,
so they show on the dashboard and count against budgets.
[memory] isn’t hot-reloaded: it reaches a VM in HostInit, so a change needs a restart of
Gopala and then applies at each VM’s next boot.
Monitoring
Section titled “Monitoring”People follow their agents through two channels besides the chat: /watch (sealed, read-only,
a personal link per conversation) and /monitor, a protobuf request/response websocket any app
can speak, with an HTTP mirror under /api/monitor/. The Lupyd app is the first such app: it
connects with the person’s own Lupyd sign-in token and gets the full stream of events (every
task, tool call and subagent), while Slack and Discord users keep following their agents through
chat messages and watch links. The wire format is in protocols.md.
sequenceDiagram
participant App as Lupyd app
participant W as web server / monitor
participant A as Authority
participant P as Platform (Lupyd provider)
participant C as Control
App->>W: GET /monitor (websocket upgrade)
App->>W: MonitorRequest{id 1, authenticate{lupyd_token}}
W->>A: LupydTokens::verify (JWKS, RS256, iss, aud, exp)
A-->>W: username = uname
W-->>App: response 1: Identity{platform "lupyd", username}
App->>W: request 2: list
W->>C: monitored(identity)
C->>A: Membership::member (cache, else ask)
A->>P: member(conversation, username)
P-->>A: Member{roles, permissions, owner} or None
A-->>C: evaluate [monitoring] rules → Grant{output, metrics}
C-->>W: conversations with can_read_output / can_read_metrics
W-->>App: response 2: ConversationList
App->>W: request 3: subscribe{key, after, output, metrics}
W-->>App: response 3: ok
loop until unsubscribed or access lost
C-->>App: MonitorEvent{key, SessionEvent} (journal replay, then live)
C-->>App: MonitorMetrics every 5 s while the VM runs
Note over W,A: on cache expiry or a membership event: check again,<br/>end with 401/403/404 under id 3
end
Pieces:
- Identities (
auth::Identity): the operator (admin token or dashboard session), or a person on a platform, optionally limited to one conversation (a watch token’s).monitor::Calleralso remembers what it needs to tell later whether the credential still holds: a Lupyd token’sexp, or the watch token, checked again withRooms::token_holder. - Lupyd tokens (
auth::LupydTokens): Auth0 JWTs verified against the key set at{platforms.lupyd.auth0_domain}/.well-known/jwks.json(auth::Jwks, aKeySourceso tests can use a fixed key set), only withmonitoring.lupyd_tokens. - Membership (
auth::Membership): answers fromPlatform::member, cached per (conversation key, username) formonitoring.membership_cache_secs, at most 4096 entries. The router drops entries onMembersChanged(the people who joined or left),Updatedand the bot leaving (Membership::invalidate), which also wakes every live subscription to check again. Direct chats don’t ask the platform: the peer is the only member and its owner (auth::direct_member). - Rules (
auth::evaluate):[monitoring] outputandmetrics, or the conversation’s[[monitoring.conversations]]entry; any match grants.approversreusesauth::is_approver, the same function the router uses for approvals and/stop. - Answers (
Control::grant,monitored,monitor_status,metrics,usage_history,journal):Nonefromgrant(not a member, other platform, other conversation) becomes404. Metrics come from the same host measurements and ledger as the dashboard. /watchuses the same rules for a watch token’s holder (web::watch_authorization), at hello and when the answer expires.
[monitoring] hot-reloads: each check reads it live, so new rules apply to the next request, and a
cached membership answer lasts at most membership_cache_secs.
Sequences
Section titled “Sequences”A mention in a group, end to end
Section titled “A mention in a group, end to end”sequenceDiagram
autonumber
actor Alice
participant FF as Firefly server
participant P as Lupyd provider
participant R as Router
participant S as session task
participant V as VmManager / firecracker
participant G as gopala-guest
participant O as omp
participant B as gopala-guest mcp
participant M as ScopedMcp (host)
participant J as Rooms journal
Alice->>FF: "@bot fix the CI" (MLS-encrypted)
FF->>P: deliver, decrypt (Firefly device)
P->>R: Event.Message(Incoming{mentions_bot: true, addressed_text})
R->>R: record → EventLog, approvals? command? triggers(Mentions) = true
R->>S: Command.Deliver{text: "alice in general (channel 1) [message …]: @bot fix the CI"}
S->>S: Slots.acquire, memory budget acquire(allocation.memory_mib)
S->>J: LIFECYCLE booting "2 vCPU, 2048 MiB, 10 GiB"
S->>V: prepare_with(key, slot, allocation): disk, tap gplN, vm.json
S->>S: bind {uds}_1024, _1025, _1100+i
S->>V: boot (firecracker --no-api --config-file)
V-->>G: kernel boots, systemd starts gopala-guest run
G->>S: vsock 2:1024 GuestHello{protocol 1}
S->>G: HostInit
G->>G: provision (if needed), forwarders, mcp.json, system.md, monitor
G->>O: spawn omp --mode rpc … [-c]
O-->>G: {"type":"ready"}
G->>O: negotiate_protocol v2, set_subagent_subscription progress
G->>S: GuestStatus ready, LIFECYCLE ready
S->>G: Delivery
G->>O: {"type":"prompt","message":…}
O-->>G: agent_start, message_update, tool_execution_*…
G->>S: SessionEvent stream (INPUT, TEXT, THINKING, TOOL_CALL, …)
G->>S: ResourceUsage every usage_interval_secs
S->>J: publish (seq assigned, stored, fanned out)
O->>B: tools/call send_message (stdio JSON-RPC)
B->>M: vsock 2:1025
M->>P: HostConversation.send → split, rate limit, Platform.send
P->>FF: encrypt_and_send_group
FF->>Alice: "On it…"
M->>J: OUTBOUND message
O-->>G: agent_end
G->>S: GuestStatus idle
Note over S: idle_shutdown_secs later (or the agent calls sleep_vm)
S->>G: Shutdown
G->>O: close stdin (kill after 10 s)
G->>G: sync, reboot → firecracker exits
S->>V: stop: wait ≤30 s, remove tap/sockets/chroot, keep disk
S->>J: LIFECYCLE stopped "idle"
Approval round trips
Section titled “Approval round trips”All paths end in HostConversation::request_approval. It
refuses a fourth open question in the conversation (MAX_OPEN_APPROVALS = 3) and counts the
question against behavior.max_messages_per_minute, then posts the question (approvals::render)
with Platform::send, immediately opens a request in Approvals (so a quick answer isn’t
missed), journals it (EVENT_OUTBOUND and APPROVAL asked), seeds reactions (✅ ❌, or 1️⃣…🔟)
with Platform::react, waits for an answer or the timeout, then journals answered/timed_out
and posts a summary as a reply to the question. The wait is at most approvals.timeout_secs; no
answer means no. A third caller is VmControl::request for resource grants under
vm.grants = "approval" (a yes/no question).
sequenceDiagram
autonumber
actor Bob as Bob (approver)
participant R as Router
participant A as Approvals
participant HC as HostConversation
participant M as ScopedMcp
participant S as session run_vm
participant G as gopala-guest (Omp)
participant O as omp
rect rgb(240,240,255)
Note over O,M: Path 1: the agent calls the MCP tool
O->>M: request_approval{question, options?, timeout_seconds?}
M->>M: validate (0 or 2..10 options, non-empty question), timeout ≤ approvals.timeout_secs
M->>HC: request_approval(question, options, timeout)
end
rect rgb(240,255,240)
Note over O,S: Path 2: omp asks through its extension UI
O->>G: extension_ui_request{id, method: confirm|select, title, message, options, timeout ms}
G->>S: GuestFrame.approval ApprovalRequest{id, question, options, timeout_secs}
S->>S: options ≤ 10 (exactly 1 → yes/no), timeout = min(request, config)
S->>HC: request_approval (spawned task)
end
HC->>Bob: "🔐 Approval needed …" (+ ✅ ❌ seeded)
HC->>A: open(key, message_id, options)
Bob->>R: reply "yes" (or react ✅ / 2️⃣)
R->>R: is_approver(conversation, Bob)?
R->>A: on_reply / on_reaction → parse → oneshot Answer{choice, by}
A-->>HC: Answer
HC->>Bob: "✅ Approved by bob" (reply to the question)
alt Path 1
HC-->>M: ApprovalOutcome{approved, choice?, choice_text?, answered_by, timed_out}
M-->>O: tool result (JSON)
else Path 2
HC-->>S: outcome
S->>G: HostFrame.approval ApprovalResult{id, choice, answered_by, timed_out}
Note right of S: buffered in unsent_answers while the guest is reconnecting
G->>O: extension_ui_response{id, confirmed | value | cancelled[, timedOut]}
end
Answer rules (the host’s approvals, Router::is_approver):
- Who may answer:
approvals.approvers=members(anyone in the conversation),owner(Meta.ownerof a group, or the other person in a direct chat), orallowlist. The same rule gates/stop. - A reply addressed to the question (
reply_to= the question’s message id) answers that request; a bare message withoutreply_toanswers only if it is the conversation’s single open request. Replies are parsed fromaddressed_text, so@bot yesworks too. - Yes/no replies:
yes y approve approved ok okay sure go "go ahead"/no n deny denied reject nope don't "do not"; reactions ✅ 👍 ✔ / ❌ 👎 ✖ (variation selectors ignored). - Choices: a number
1..=n(optionally#2), the option’s own text (case-insensitive, trailing./!ignored), or the keycap reaction. - omp
input/editorUI requests are cancelled immediately;notifybecomes anEVENT_NOTICE;cancelforgets the pending request. Approval ids are unique per guest process (NEXT_APPROVAL), so a stale answer after a harness restart matches nothing.
A resource request
Section titled “A resource request”sequenceDiagram
autonumber
participant MON as guest monitor
participant O as omp (agent)
participant M as ScopedMcp
participant VC as VmControl
participant HC as HostConversation
participant S as session
participant V as VmManager
MON->>S: ResourceUsage (disk 88%)
MON->>O: steer "Resource alert — disk 88% used … clean up …"
O->>O: cargo clean, rm caches, df -h
O->>M: vm_resources
M->>VC: report → allocated, maximum, usage, grants
O->>M: request_resources{disk_gib: 20, reason}
M->>VC: request(Wanted, reason)
VC->>VC: decide: maximums, budget, disk reserve, grants mode
opt grants = approval
VC->>HC: request_approval("My VM needs more room: …")
HC-->>VC: approved
end
VC->>VC: persist allocation.json, desired = new, NOTICE "granted …"
VC-->>O: Decision{granted: true, after_restart, message}
O->>M: restart_vm{reason}
M->>VC: restart → NOTICE, Command.Restart in 1 s
VC-->>O: "restarting in a moment …"
S->>S: Shutdown guest, stop VM, journal stopped "restarting"
S->>S: slot + memory permits for the new allocation
S->>V: prepare_with(key, slot, new allocation): grow disk, new vm.json
S->>S: boot, handshake, HostInit, omp continues with -c
/watch and a viewer connecting
Section titled “/watch and a viewer connecting”sequenceDiagram
autonumber
actor Carol
participant R as Router
participant Rm as Rooms
participant DB as rooms.db
participant DM as DM HostConversation
participant Web as web server /watch
participant W as gopala watch (client)
Carol->>R: "@bot /watch" in group
R->>Rm: issue_token("lupyd-g-42", "carol")
Rm->>DB: delete carol's old tokens for the scope and expired ones, insert {id, sha256(secret), kwk, expires_at}
Rm-->>R: token gpl1.id.secret
R->>DM: say("👀 Watch me work … watch_url + token") (direct chat on the same platform)
R->>Carol: "📬 Sent you a private watch link, carol." (in the group)
Carol->>W: GOPALA_WATCH_LINK=… gopala watch
W->>Web: TCP (+TLS), GET /watch with websocket upgrade
Web->>Rm: Rooms.viewer(websocket)
W->>Rm: ClientFrame.hello{token, since_seq, live_only}
Rm->>DB: verify(token): lookup id, check expiry, constant-time sha256 compare
Rm->>Rm: lock room: capacity check, register viewer, subscribe live
Rm->>W: Welcome{scope, title, encryption, latest_seq}
Rm->>W: RoomKey{epoch, nonce, wrap(kwk, room key)} (sealed mode)
loop replay pages of 256, seq in (since_seq, latest_seq]
Rm->>DB: events(scope, last, latest, 256)
Rm->>W: SealedEvent (sealed with the current key) or SessionEvent
end
Rm->>W: HistoryDone{latest_seq}
loop live
Rm->>W: SealedEvent / SessionEvent (seq > last)
W->>Rm: Ping{nonce}
Rm->>W: Pong{nonce}
end
Note over Rm: every key_rotation_secs, or when a member leaves
Rm->>W: RoomKey{epoch+1, …} (queued before any event sealed with it)
Note over R,Rm: Carol leaves the group (Event.MembersChanged left)
R->>Rm: revoke("lupyd-g-42", "carol")
Rm->>DB: delete carol's tokens
Rm->>W: ErrorFrame "access revoked", close
Rm->>Rm: rotate the key for remaining viewers
The watch link is <watch_url>#<token>, where watch_url is server.public_url with http(s)
turned into ws(s) plus /watch. In a direct chat the link is sent as the reply itself. If the
bot cannot message the requester directly, the group gets Couldn't message you directly: ….
With rooms.enabled = false the answer is Watching is turned off on this bot.
Guest reconnect
Section titled “Guest reconnect”The control connection can drop while the VM keeps running: the guest process crashed and
systemd restarted it (Restart=always, RestartSec=1), or the vsock connection broke.
sequenceDiagram
autonumber
participant S as session run_vm
participant G as gopala-guest (uplink)
participant H as harness / omp
G--xS: control connection closes
S->>S: reader returns None → connected = false
Note over S: deliveries stay queued, approval results go to unsent_answers,<br/>idle timer and VM exit still watched
loop backoff 200 ms → 5 s
G->>S: vsock connect 2:1024
end
G->>S: GuestHello
S->>S: accept on the same listener, wait ≤ 10 s for hello
S->>G: HostInit (resent)
alt same guest process (link dropped)
G->>G: ignores the repeated HostInit
else restarted guest process
G->>G: provision (marker matches: skipped), forwarders, configure, monitor
G->>H: start omp with -c (continues the session)
end
S->>G: unsent ApprovalResults, then queued Deliveries
G->>S: frames queued in the Outbox while disconnected (≤ 2000, oldest dropped)
The VM exiting is never a reconnect: run_vm treats it as a failure and the session backs off
and boots again. A reconnecting guest’s hello is not re-checked against CONTROL_PROTOCOL, and the
session loop waits for it inline (up to 10 s).
Data on disk
Section titled “Data on disk”<data_dir>/ 0700 (created by daemon::create_private_dir; warns if group/other bits set)├── lupyd/ the Lupyd provider's Firefly device directory (firefly-mcp); Slack and Discord keep nothing on disk│ ├── session.json Auth0 session with the refresh token (0600, written by firefly-mcp)│ ├── firefly.db firefly-client store: MLS state, messages, groups, search index│ └── firefly-mcp.lock one process per account and data directory├── rooms/ 0700│ └── rooms.db (+ -wal, -shm) 0600, SQLite WAL: tables `events` and `tokens`├── usage.db (+ -wal, -shm) 0600, the host-measured usage ledger: `usage_days` (per key per UTC day, 400 days) and `usage_totals`├── images/ default output of `gopala image build`: vmlinux-<version>, gopala-<base>.ext4, gopala-tools.ext4 (+ .sha256)└── vms/ └── <conversation key>/ 0700, one per conversation that ever booted (e.g. lupyd-g-42) ├── disk.ext4 0600, the conversation's persistent root disk (copy of vm.rootfs, grown to its allocation; sparse) ├── disk.ext4.tmp only while the first copy is being built ├── allocation.json 0600, the agent's granted {vcpus, memory_mib, disk_gib} (absent until a grant) ├── vm.json 0600, firecracker --config-file (unjailed only) ├── console.log 0600, firecracker stdout+stderr (guest serial console), truncated per boot, capped ├── v.sock vsock uds_path (unjailed only) └── v.sock_1024, _1025, _1090, _1100…, _1200… host listeners, 0600 (unjailed only)
<vm.jailer.chroot_base>/<firecracker binary name>/<key>/root/ (jailed only, recreated per boot, removed on stop)├── vmlinux hard link to vm.kernel├── disk.ext4 hard link to <data_dir>/vms/<key>/disk.ext4, chowned to the jailer uid/gid├── vm.json 0600, chowned to the jailer uid/gid├── tools.ext4 hard link to vm.tools_image (when set), read-only drive└── v.sock, v.sock_<port> listeners, 0600, chowned to the jailer uid/gid
<vm.tools_image> the read-only tools image (built by gopala image build), attached to every VM as drive `tools`<config file> replaced atomically by PUT /api/config and PUT …/budget (0600), previous version kept as <config file>.bak (0600)<server.site_dir> a website directory that, when present, is served instead of the one embedded in the binaryrooms.db schema:
| Table | Columns | Notes |
|---|---|---|
events |
scope, seq, at_ms, body (encoded SessionEvent with seq set) |
Primary key (scope, seq), WITHOUT ROWID. Stored in plaintext in both encryption modes. When a scope’s bodies exceed rooms.journal_max_mib (checked every 1000 inserts) the oldest half is deleted (db::cap). |
tokens |
id (12 bytes), scope, username, secret_hash (SHA-256), kwk (HKDF output), expires_at |
One live token per (scope, user); expired rows are deleted at startup, on issue, and on verify. |
Inside each guest disk: /var/lib/gopala/provisioned (provisioning hash), /etc/sudoers.d/gopala,
/etc/resolv.conf (NAT only), /etc/profile.d/gopala-tools.sh (with tools),
/etc/profile.d/gopala-proxy.sh and /etc/apt/apt.conf.d/90gopala-proxy (proxy mode only), and
the agent’s /home/agent with work/ (its persistent workspace, also omp’s project directory),
.gopala/system.md, .gopala/omp-started or .gopala/claude-started, .gopala/mcp.json (the
agent’s own MCP servers), .gopala/claude-mcp.json (Claude Code only), .gopala/memory.db (the
memory store, with its -wal and -shm files; 0600 in a 0700 directory, owned by the agent),
.omp/agent/mcp.json (also work/.mcp.json), and the harness’s own session files. With tools,
/opt/gopala is the read-only
mount of the tools drive (not part of the disk).
Browser sessions (dashboard sign-ins) live only in the web server’s memory: a restart signs everyone out.
Configuration map
Section titled “Configuration map”The full key list with defaults is in gopala.example.toml, the doc
comments of the configuration, and the JSON Schema served at /api/config/schema. Every
key has a default and unknown keys are rejected (deny_unknown_fields). Keys not shown in the
example: platforms.lupyd.wt_url, platforms.lupyd.auth0_realm (default
Username-Password-Authentication), server.allow_plaintext, vm.extra_boot_args,
vm.console_log_max_kib (default 4096), [[llm]].auth_prefix.
| Section / key | Used by |
|---|---|
data_dir |
daemon::create_private_dir, Lupyd::connect (<data_dir>/lupyd), Rooms::open (<data_dir>/rooms), usage::Ledger::open (<data_dir>/usage.db), VmManager::new (<data_dir>/vms), VmControl (allocation.json), Capacity::disk_free |
[platforms.lupyd] enabled |
PlatformCredentials::take (credentials) and main::run (connect the provider) |
[platforms.lupyd] api_url, ws_url, wt_url, auth0_*, device_name |
gopala_lupyd::auth_config, firefly_mcp::agent::Endpoints, Agent::start |
platforms.lupyd.profile_api_url |
Lupyd::profiles: one GET {profile_api_url}/user?users=a,b with the bot’s Lupyd access token for the bios list_agents shows; Config::validate (http(s)); restart required |
[platforms.lupyd] email_env, password_env |
PlatformCredentials::take (take_env); named in the “not signed in” error |
[platforms.slack] enabled, bot_token_env, app_token_env |
PlatformCredentials::take, Slack::connect (checks the xoxb-/xapp- prefixes, auth.test, apps.connections.open) |
[platforms.discord] enabled, token_env |
PlatformCredentials::take, Discord::connect (checks the token and the application’s intent flags) |
behavior.* |
Hot-reloadable (Control::apply); see Control plane and metering for when each reader sees a change |
behavior.respond_to, behavior.conversations |
policy::respond_to → Router::handle (groups ignored for dms), policy::triggers; written by control::set_respond_to (/listen, PUT …/behavior); Control::state (respond_to) |
behavior.direct_messages |
Router::admit_direct → Platform::admit_direct |
behavior.invites |
Router::check_invite, Router::check_joined_groups |
behavior.allowlist |
Behavior::allows (case-insensitive) for the three allowlist policies |
behavior.max_messages_per_minute |
HostConversation::limiter (governor): each sent part and each approval question |
behavior.max_message_chars |
policy::split_message in HostConversation::send |
approvals.timeout_secs |
run_vm (approval_wait, cap for guest ApprovalRequest), ScopedMcp::new (max_approval_wait), VmControl::request |
approvals.approvers |
Router::is_approver (answers, /stop and /listen) |
[alerts] |
HostConversation::audience (warning, critical, conversations), take_alert (max_per_hour), host_event (host_events); hot-reloadable |
[server] |
web::serve / serve_on (bind, TLS), web::app (site_dir), main::start (admin_token_env), Server::watch_url (public_url), Web::secure (Secure cookie flag with TLS or an https public_url), main::run (enabled); Config::validate (TLS pair, loopback/plaintext rule, http(s) public_url, env name). Restart required. |
[monitoring] |
auth::rules and rule_matches via Control::grant (output, metrics, conversations), monitor::lupyd_caller (lupyd_tokens), auth::Membership::member (membership_cache_secs); web::watch_authorization for /watch; applies without a restart |
[collaboration] |
Router::on_message (enabled: whether agents’ messages can wake the agent), HostConversation::take_agent_message (max_agent_messages_per_hour), HostConversation::agents → Platform::agents (enabled, agent_role), session::host_init → policy::system_prompt (the “Working with other agents” section, at boot); main::run → each provider’s set_agent_role (once, at startup). Hot-reloadable, except for the providers’ agent_role |
platforms.lupyd.auth0_domain, auth0_audience |
also auth::Jwks::new (the key set URL) and LupydTokens::verify (iss, aud) for Lupyd sign-in tokens |
[rooms] |
Rooms::open (encryption, key_rotation_secs), Rooms::serve_viewer (max_viewers_per_room), Rooms::publish (journal_max_mib), Rooms::issue_token (token_ttl_secs); main::run (enabled); Config::validate (needs server.enabled, rotation ≥ 60 s) |
vm.firecracker, vm.kernel, vm.rootfs |
VmManager::prepare_with, ensure_disk, resolve_binary (jailed) |
vm.vcpus, vm.memory_mib, vm.disk_gib |
the base Allocation of every VM (VmControl::new), the disk’s floor |
vm.network, vm.dns, vm.extra_boot_args |
firecracker_config, boot_args; network = "nat" also setup_tap; "proxy" makes run_vm bind vsock 1090 and host_init set egress_proxy |
vm.allowed_hosts |
egress::Allowlist (one Egress per VM, built at boot); restart required |
vm.console_log_max_kib |
PreparedVm::boot → drain_console |
vm.max_vms |
session::Slots (Permits, by budget priority) |
vm.boot_timeout_secs, vm.idle_shutdown_secs |
session::run_vm (read at each boot; hot-reloadable) |
vm.max_vcpus, vm.max_memory_mib, vm.max_disk_gib |
machine::decide, VmControl::new (clamping), VmControl::report, Control::state (hot-reloadable) |
vm.memory_budget_mib |
machine::memory_budget_mib → Shared::memory (Permits, fixed at start) |
vm.min_free_disk_gib, vm.grants |
machine::decide (hot-reloadable; VmControl::operator_set ignores grants) |
[vm.jailer] |
jailer_args, setup_chroot, PreparedVm::bind_vsock (chown), setup_tap (tap owner) |
vm.tools_image, vm.tools_sha256 |
gopala_vm::verify_tools (startup, check-config); VmManager::prepare_with → second, read-only drive tools (hard-linked into the jailer chroot as /tools.ext4) and gopala.tools=1 in boot_args; the guest’s tools::setup mounts it at /opt/gopala and harness::command resolves the harness there first |
[agent] |
session::host_init (harness, packages, setup, user_setup, privileges, env, alert percents, usage interval, tools, mcp), policy::system_prompt (sudo / install_packages wording, system_prompt); executed by the guest’s provision, monitor, agent_config and harness |
agent.harness |
HostInit.harness.kind → harness::supervise (omp, claude_code, exec); VmControl::restart_harness (refused for exec); image::runs (CLAUDE_CODE=1 for the tools image with claude_code) |
[[agent.mcp]] |
session::host_init (HostInit.mcp, mcp_forward); url servers: egress::mcp_secrets (startup, check-config) → egress::mcp_upstreams, egress::live_mcp and run_vm binding 1200+i; command servers run in the guest |
[[llm]] |
main::secrets → llm_proxy::Upstream::new (upstream, auth_header, auth_prefix, secret_env); session::host_init (guest_port → LlmForward.local_port, guest_env → HostInit.env); run_vm binds 1100+i |
[[git]] |
the git proxies (name, provider, url, repos, protected_branches; token_env read at startup, checked by check-config); session::host_init → HostInit.git (GitForward); run_vm binds 1300+i; the guest’s git config and GOPALA_GIT_<NAME>_API. Not hot-reloaded: restart required |
[budget] |
budget::check via budget::gate (each model call), budget::effective → Shared::priority (slots and memory), Control::state (budget fields), PUT …/budget (budget::changed, edit_toml); hot-reloadable |
[image] |
gopala image build only (image::build); gopala run doesn’t read it |
image.setup |
image::runs → one --setup <command> per entry for build-rootfs.sh, run with bash -c as root in the chroot; Config::validate (non-empty, no NUL) |
[memory] |
session::memory_config → HostInit.memory ([[llm]] names resolved to their guest_port), policy::system_prompt (the memory line); the guest’s agent_config::base and memory::spawn_maintenance; restart required |
Config::validate enforces: at least one enabled platform; an allowlist exists when any
allowlist policy is selected; max_message_chars > 0; with server.enabled: TLS cert and key
together, no plaintext on a non-loopback server.bind unless allow_plaintext, an http(s)
public_url, a valid admin_token_env name; rooms.enabled needs server.enabled;
key_rotation_secs ≥ 60; positive VM sizes (memory_mib ≥ 128); maximums not below the base
sizes; memory_budget_mib 0 or at least memory_mib; alert percents within 10–99;
usage_interval_secs ≥ 5; jailer uid/gid not 0; an absolute vm.tools_image and a
64-hex-digit vm.tools_sha256 (only with an image); no agent.packages with network = "none";
a non-empty, well-formed vm.allowed_hosts with network = "proxy"; agent.mcp names
[a-z0-9_-]{1,32}, unique, not chat, each with a command or an http(s) url, secret_env only
with a url; plain agent.tools, image.packages and image.tools names; positive
image.rootfs_size_gib; budget shares within 1–100 with low ≤ normal; valid, unique
budget.conversations, behavior.conversations and alerts.conversations keys; alerts
entries of the form everyone, here, role:<name> or user:<name>; positive memory caps,
cleanup_interval_secs ≥ 60, duplicate_threshold within 0.5–1, and memory.embeddings /
memory.compaction naming an existing [[llm]] entry and a model; no agent.mcp server named
memory while memory is on; valid environment variable names; unique guest_ports, none on
3128, 3200..3200+n or 3300..3300+n; http(s) upstreams; at most 64 [[llm]] entries; at most 8
[[git]] entries with unique names; collaboration.agent_role
of 1–100 characters and max_agent_messages_per_hour at most 3600; non-empty image.setup
commands without NUL bytes; an http(s) platforms.lupyd.profile_api_url.
Design decisions
Section titled “Design decisions”| Decision | Chosen | Alternatives considered | Why |
|---|---|---|---|
| Platform seam | One Platform trait with neutral events and views; providers own mention syntax, ids, encoding and sign-in; Incoming carries mentions_bot and addressed_text already computed |
Lupyd-specific core (the first version); a plugin process protocol | Keeps policy, sessions, VMs, approvals, MCP and watching identical for every platform, and new platforms a single crate. An in-process trait is simplest to test (FakePlatform) and costs no serialization. |
| Lupyd access | Reuse firefly-mcp’s Auth and Agent (and firefly_mcp::content for message encoding, fold for edits/reactions, the Firefly stores and search), pinned to the same git revision lupyd-mcp ships (f9a0960) |
Reimplement sign-in, the MLS device and message encoding | MLS device management, key packages, message encoding and storage are subtle and already maintained in Firefly; the bot is then the same kind of device as a person’s. Cost: the host links mls-rs (hence --cfg mls_build_async) and is tied to Firefly’s revision. Only gopala-lupyd depends on it; the guest stays free of it. |
| MCP server | gopala-mcp: firefly-mcp’s tool vocabulary (read, send, edit, delete, react, search, wait for events) re-exposed through a Conversation trait fixed to one conversation, plus request_approval and the VM tools |
Run firefly-mcp itself per VM | firefly-mcp’s tools take group ids and usernames and need the account’s keys; giving that to a guest would expose every conversation and the credentials. |
| Host ⇄ guest transport | vsock, guest-initiated only: the host listens on Firecracker’s {uds_path}_{port} sockets and never dials into a guest |
Host-initiated vsock (CONNECT <port> on the uds), TCP over the tap, virtio-serial |
Works with network = "none"; scope is implied by which VM’s socket a connection arrives on; the host never needs to trust anything a guest listens on; the guest owns reconnecting. |
| Model API credentials | Host-side proxy per [[llm]]: the guest gets a placeholder key and http://127.0.0.1:<port>; the host strips credential headers, adds the real one and forwards over HTTPS |
Pass keys in the VM environment; one generic egress proxy for everything | Keys never enter a VM (not in its env, disk, omp session files or the journal), the proxy reaches only the configured upstream, and it can meter and budget every call. Cost: request bodies are buffered (≤ 32 MiB) and the guest side is plain HTTP over vsock. |
| Restricted network | An HTTP(S) proxy on the host per VM (network = "proxy"), reached over vsock, with an allowlist and an address check after host-side DNS |
Per-tap nftables rules managed by Gopala (issue #6); a DNS-filtering resolver | No network device means no way around it, and no root or CAP_NET_ADMIN is needed. Programs that ignore proxy variables can’t get out at all, which is the safe failure. Cost: only HTTP(S) works. |
| Token budgets | Checked in the LLM proxy before each call, from the host ledger, with priorities that also order boots | Provider-side limits only; cut calls off mid-stream | The host already sees every call and its usage, so the check is cheap and the agent can’t evade it. Refusing before a call (never during one) keeps responses whole; the cost is that one call may overshoot. |
| Harness integration | omp RPC mode (--mode rpc) by default, or Claude Code in stream-json mode: one long-lived session per VM, both behind one Protocol trait in the guest (a gopala-owned harness is to replace omp, issue #5) |
One-shot CLI invocations per message (kept as the exec harness) |
Keeps context across messages and reboots (-c, --continue), supports follow-ups and steering while working, and streams structured events (tool calls, subagents) for viewers; omp also has an extension-UI sub-protocol the guest maps onto chat approvals. Claude Code runs with bypassPermissions because the VM is the sandbox and request_approval covers critical steps. |
| Agents extending themselves | The agent edits its own mcp.json and calls restart_harness; the guest merges it under the host’s servers before every start |
The operator adds every server; let the agent write the harness’s config directly | Agents can pick up the tools a task needs without an operator, while the names the host configured stay the host’s and the file can’t redirect root’s writes (no symlinks, owner checked). An agent can already run anything in its VM, so a server it adds gains it nothing new. |
| Agents working together | Agents ping each other with ordinary @mentions in the shared group; list_agents shows who’s there; each decides for itself; the host caps wake-ups per hour |
A host-side task broker or delegation API between agents | Works across hosts, products and platforms with nothing to deploy, and people see every hand-off in the chat. An agent never gains authority from another: pings can’t run commands or answer approvals. Cost: chat round trips, and the cap can cut a long exchange short. |
| Harness and tools provenance | A host-built, read-only ext4 image (vm.tools_image, checked against vm.tools_sha256 at start) mounted ro,nosuid,nodev; the guest refuses a harness the agent could replace |
Install the harness and tools inside each VM (the default user_setup still does this without an image) |
The agent can’t swap the program that drives it or the tools it trusts, and every VM runs the same pinned versions. Integrity is checked at start only (verity is issue #7). |
| Usage metering | Measure on the host: the LLM proxy parses the provider’s usage reports, the firecracker process, disk image and tap are sampled from /proc and /sys; a SQLite ledger per key and day |
Trust the guest’s own reports; meter in the harness | The agent may lie or be compromised; the host sees every model call (it holds the keys) and the VM’s real footprint. The guest’s reports stay visible as an unverified second opinion. |
| Resources | The agent measures, cleans up and asks; the host decides by config and capacity; grants apply at a restart | Fixed sizes only; live resizing (balloon device, hotplug) | Agents know best when they need more and can free space themselves; restarting is simple and works with Firecracker’s static machine config, and the workspace and omp session survive it. Disks only grow, so no data can be cut off. |
| Journal storage | SQLite via sqlx (WAL), one events table keyed by (scope, seq) plus tokens |
A custom append-only log file per scope | sqlx/SQLite is already in the dependency graph through firefly-client; gives atomic token issue, paged replay, and simple size capping. |
| One web server | axum (on hyper HTTP/1, optional rustls) serves the site, the control and config API and /watch; rooms only speaks the protocol on websockets it is handed |
A separate rooms listener (the first version), a separate admin port | One port, one TLS setup and one public_url; the config API reuses the daemon’s exact Config::parse. |
| Control panel | Static Astro pages with Alpine.js on the same origin; sign-in with the admin token exchanges it for an HttpOnly, SameSite=Strict session cookie; live updates over server-sent events |
A single-page app with its own state; tokens in browser storage | No build-time coupling to server state; the token never sits in page memory or storage after sign-in; SSE is one long GET that cookies already authenticate. |
| Configuration changes | PUT /api/config writes the file atomically and hot-applies the sections that are safe to swap at runtime (ArcSwap); the rest waits for a restart |
Restart for every change; reload everything | Policy and VM limits change often and are read per decision; server, platform and rooms settings own sockets and devices that can’t be swapped safely. |
| Tap networking | Shell out to iproute2 (ip tuntap add, ip addr add, ip link set, ip link del) |
netlink crates (e.g. rtnetlink) | A standard tool with clear error messages and no extra dependency; the cost is a process spawn per VM start/stop, and iproute2 must be installed. |
| Watch encryption | Configurable: sealed (default) or none (TLS only) |
TLS only | Sealing adds a layer that survives TLS-terminating proxies and a plaintext transport segment, and lets revocation take effect cryptographically through key rotation. It is not end-to-end against the host, which generates the keys and stores the journal in plaintext. |
| Watch token delivery | Personal token per (conversation, user), sent in a direct chat on the same platform (end-to-end encrypted on Lupyd), in the link’s #fragment, and presented inside the websocket hello |
Tokens in a URL query string or cookie | Fragments are not sent in HTTP requests, so tokens don’t land in proxy or server access logs. |
| Wire format | quick-protobuf with pb-rs-generated owned types, length-delimited over byte streams, one message per websocket binary frame | JSON, prost | Requested in the brief for performance; matches Firefly’s own protobuf tooling. |
| Notifying people | The agent gives a severity; [alerts] maps it to an audience; providers render it |
Let the agent write mentions itself | The agent can’t page people it shouldn’t (prompt injection can’t add @everyone), works the same on every platform, and the operator can cap it per hour. |
| Agent memory | SQLite in each VM, written by the agent through MCP tools, cleaned up by score; a model only for optional compaction | A host-side memory service; mem0 with an LLM call on every write | Memory stays with the conversation’s disk and never mixes conversations; no extra service or per-write model cost. Cost: the guest agent links C (SQLite). |
Future: Firefly/MLS for rooms. The brief anticipates replacing the rooms’ rotating-key scheme with Firefly itself (real end-to-end encryption with member key exchange) once it is less tangled with Lupyd. The brief’s “key exchange from members” is not implemented: room keys are generated by the host and wrapped per viewer.
Known limitations
Section titled “Known limitations”- No real VM boot, and no live conversation on Lupyd, Slack or Discord, has been tested end to end (issue #1, issue #2); tests use fakes and recorded payloads.
- Over-long lines are dropped, not truncated: guest log lines over 2 KiB become
(line over 2048 bytes dropped), and omp stdout frames over 1 MiB + 1 KiB are discarded with a warning (issue #19). - An in-place disk grow (
grow_disk:set_len, thene2fsck,resize2fs) interrupted by a crash isn’t retried: the nextensure_disksees the file at the target size and skips the filesystem resize (issue #18). - Grants apply only at a restart; there is no live memory resizing (issue #9). The memory budget
counts configured VM memory (
mem_size_mib), not Firecracker’s own overhead or what guests actually touch (issue #13). Disk images are sparse, somin_free_disk_gibis checked when disk growth is granted, not while guests write (issue #14). - The LLM proxy buffers each request body in memory (up to 32 MiB) before forwarding; responses stream (issue #15).
- Metering sees only what passes through the host: with
network = "nat"a guest can reach other services directly, visible only as tap bytes, until per-tap egress filtering lands (issue #6). A provider response without a usage report (for example an encoded body, or an OpenAI chat stream requested without usage) counts as a request with zero tokens [inference for the latter], so budgets don’t see it either. Budgets are checked before each call, so calls in flight can take a conversation past its limit. network = "proxy"carries HTTP and HTTPS only; programs that ignore the proxy variables (or speak other protocols, like ssh) can’t reach the network. An allowed name stays allowed if its owner moves it to another public address.- Discord runs one gateway shard (up to 2500 servers). Slack search scans only the newest 1000 messages of a channel, and Discord search 200 messages in each of 25 channels, because bots can’t use the platforms’ search APIs [inference for Discord].
- The chat
EventLogbehindwait_for_eventsis in memory (last 1000 events per conversation) and starts over when the host restarts; room keys live in memory too, so after a restart every room starts again at epoch 1 with a new key (issue #20). Dashboard sessions are in memory as well. - Deliveries handed to a guest that is just exiting (idle shutdown, sleep or restart racing a new message) can be lost (issue #25).
/statusanswersstarting upwhile a VM boots, because the session doesn’t read commands during the handshake (issue #12); the dashboard showsbooting.- Hot reload doesn’t change existing conversations’ chat rate limit or base allocation
[inference from the code];
[server],[platforms],[rooms],[[llm]],[memory], the memory budget,vm.network,vm.allowed_hostsand the other VM settings need a restart, and so does a new HTTP MCP server with asecret_env. A changedcollaboration.agent_rolereacheslist_agentsat once but the providers’ marking of incoming messages only after a restart. - Collaboration rests on each platform’s idea of an agent: Discord gives no bio (
aboutis always empty) and, without the Server Members intent, lists only members it has seen; Slack shows a profile title, which bot users may not have [inference]; Lupyd’s bios come from a profile request that hasn’t been run against the real Lupyd API yet. Slack has no reply-to field, so “replies to the bot” means a reply in a thread the bot’s message started. The hourly cap counts wake-ups per conversation, so a busy group with several agents can use it up and drop legitimate pings. - Claude Code is untested against a real process: the
priorityfield for mid-turn messages,--continueafter SIGTERM and the timing ofsystem/initfollow its documentation. Whetherclaude -pneeds a placeholderANTHROPIC_API_KEYnext toANTHROPIC_BASE_URLhasn’t been checked. - Alerts are only as good as each platform’s mentions: Lupyd has no role mentions and no native
@everyone/@here(they’re plain text there); Slack workspace admins can restrict who may use@channel/@here, and user groups need theusergroups:readscope (apps installed before it was added must be reinstalled); Discord needs the “Mention @everyone, @here and All Roles” permission and finds members best with the Server Members intent. Names that don’t resolve are only logged. - Monitoring access follows the platforms’ answers, cached for up to
membership_cache_secs: a person removed without a membership event keeps access until the answer expires. Providers differ: Lupyd answers from the bot’s copy of the group; Slack’s member list is cached 30 s and user groups 5 min (roles needusergroups:read); Discord knows member changes at once only with the Server Members intent, otherwise it searches members over REST when the cache expires. The monitor protocol has no client library yet; apps build requests from the protobuf schema in protocols.md. - Memory lives on the conversation’s disk: deleting
vms/<key>/disk.ext4deletes it, and nothing backs it up separately. The agent owns the file and can change it with its shell, so it is a notebook, not an audit log. Cosine recall is brute force over stored vectors [inference: fine for the default 5000 memories, slow at very large caps]. Memory posts/v1/…paths to the[[llm]]forwarder, so anupstreamthat already ends in/v1would double it [inference]. - The tools image is checked against its sha256 only when Gopala starts; there is no dm-verity or fs-verity on the running drive (issue #7).
- On shutdown (SIGINT/SIGTERM) the daemon asks sessions to close and waits 2 seconds before
shutting the providers down; VMs that haven’t stopped by then are expected to die with the
process, since the firecracker child is spawned with
kill_on_drop[inference: not tested] (issue #21).