Skip to content

Architecture

How Gopala works inside: the host agent manager, one Firecracker microVM per conversation, the guest agent, its proxies, and the protocols that join them.

The wire formats are specified in protocols.md, the threat model in security.md, and how chat platforms plug in in providers.md.

Status. Everything below is implemented and covered by unit and in-process integration tests, but no real Firecracker boot, and no live conversation on Lupyd, Slack or Discord, has been exercised end to end yet. See Known limitations.

Gopala is a self-hostable agent manager: it serves conversations on chat platforms and gives every conversation its own autonomous agent in its own Firecracker microVM. Chat platforms are providers behind one trait (gopala_platform::Platform); Lupyd, Slack and Discord are the providers that exist. From the original brief and what has been added since:

Goal Where it lives
The bot is an ordinary account on a chat platform; on Lupyd it signs in with an email and password from the environment and talks over Firefly (Lupyd’s end-to-end encrypted chat) the platform providers: the provider trait and the Lupyd provider
Other platforms through the same trait the Slack provider (Socket Mode, no public URL), the Discord provider (gateway); see providers.md
A TOML config deciding whom it listens to: every message, mentions only, DMs only, and whether it accepts invites the configuration (Behavior), the host’s router (Router)
One Firecracker microVM per conversation, sized and provisioned from the config, with optional extra privileges (installing tools) the VM launcher, the guest agent’s provisioning
Agents aware of their memory and disk, cleaning up after themselves, asking the host for more when they need it the guest agent’s resource monitor, the host’s VmControl, VM tools in the MCP server
A lightweight, long-running harness with subagents and multitasking that rarely pauses agent.harness: omp in RPC mode (default) or Claude Code in stream-json mode, each one long-lived session; exec runs any command per message; supervised by the guest agent. omp is temporary and is to be replaced by a gopala-owned harness (issue #5)
The harness and tools such as git provided by the host, read-only, so the agent can’t swap them vm.tools_image, the guest agent’s tools mount, the tools image build
Agents that extend themselves: install what they need, add MCP servers, restart their harness to load them the agent’s /home/agent/.gopala/mcp.json (agent_config::merge), restart_harness
Agents in one group that find each other and split work, without looping forever [collaboration], list_agents, Platform::agents, Incoming::from_agent, HostConversation::take_agent_message
Approvals for critical steps by yes/no replies or emoji reactions the host’s approvals, request_approval in the MCP server
A Gopala MCP server that mimics firefly-mcp but reaches only one conversation per VM the MCP server (ScopedMcp)
An out-of-band, read-only websocket to watch sessions live and in history, with a rotating-key encrypted mode or TLS only the watch service (rooms), served at /watch by the host’s web server
One web server for a self-hosted control panel (dashboard, configuration editor, docs), its API, and the watch websocket the web server, the control API, [server], the website embedded in the binary
Every usage figure measured on the host, never trusted from inside a VM: model tokens and requests, CPU, memory, disk, network, git traffic, tool calls, messages the LLM proxy, the git proxy, the usage ledger, RunningVm::sample in the VM launcher
Daily and monthly token budgets per conversation and for the host, with priorities that also order VM slots and memory the host’s budgets, session::Permits, [budget]
A choice of network per VM: none, an allowlisting HTTP(S) proxy on the host, or NAT vm.network, the host’s egress proxy, the guest agent’s proxy settings
Extra MCP servers and a tool allowlist for the agent, with HTTP servers’ secrets kept on the host agent.mcp, agent.tools, the guest agent’s MCP config, egress::serve_mcp
Agents that clone, push branches and open pull requests on GitHub, GitLab or Gitea without ever holding the token, and can’t touch protected branches or merge [[git]], the host’s git proxy, the guest agent’s git config
Guest images built from the config gopala image build with its embedded build scripts and guest agent
quick-protobuf wire formats for performance the wire formats
Serious messages that reach the right people, without the agent knowing who they are or how the platform mentions them [alerts], HostConversation::alert, Platform::send_notifying
A group can be switched to “listen to everything” from the chat or the dashboard [[behavior.conversations]], /listen, control::set_respond_to
Agents remember facts across restarts without the memory growing forever the memory store, [memory], the memory MCP server in the guest
Every endpoint authenticated; any app (first the Lupyd app) can follow the agents of the conversations its user is in, checked with the platform the host’s authentication and monitoring, [monitoring], Platform::member
flowchart TB
    subgraph Platforms["chat platforms"]
      FF[(Lupyd / Firefly:<br/>MLS groups, DMs)]
      SL[(Slack:<br/>Socket Mode + Web API)]
      DC[(Discord:<br/>gateway + REST)]
    end
    subgraph Host["gopala run (host process)"]
      direction TB
      LP["providers<br/>gopala_lupyd.Lupyd, gopala_slack.Slack,<br/>gopala_discord.Discord"]
      RT["Router"]
      AP["Approvals"]
      subgraph Scope["per conversation"]
        HC["HostConversation"]
        VC["VmControl"]
        SS["session task<br/>run / run_vm"]
        MCP["MCP listener<br/>ScopedMcp per connection"]
        PX["LLM proxy listeners<br/>(budget check)"]
        EG["egress + HTTP MCP proxies"]
        GP["git proxies"]
      end
      WEB["web server<br/>dashboard, /api, /watch"]
      CTL["Control<br/>state, actions,<br/>hot reload"]
      LED[("usage.db<br/>Ledger")]
      ROOMS["Rooms"]
      DB[("rooms.db<br/>journal + tokens")]
      VMM["VmManager"]
    end
    subgraph VM["Firecracker microVM (one per running conversation)"]
      GG["gopala-guest run<br/>uplink, provision, monitor, harness"]
      OMP["omp --mode rpc"]
      BR["gopala-guest mcp<br/>stdio to vsock"]
      FW["127.0.0.1 forwarders<br/>guest_port, 3128, 3200+i, 3300+i"]
      TOOLS["/opt/gopala (read-only drive)<br/>omp, git"]
      MEM["gopala-guest memory<br/>~/.gopala/memory.db (SQLite)"]
    end
    UP[(model provider APIs)]
    NET[(allowed hosts,<br/>HTTP MCP servers)]
    GIT[(GitHub / GitLab / Gitea)]
    VIEW["viewer<br/>gopala watch"]
    ADMIN["browser<br/>dashboard, /configure/"]
    APP["apps (Lupyd app, scripts)<br/>/monitor, /api/monitor"]

    FF <--> LP
    SL <--> LP
    DC <--> LP
    LP -- "Event (mpsc)" --> RT
    RT --> AP
    RT -- Command --> SS
    RT --> HC
    HC -- "Platform calls" --> LP
    SS -- "prepare / boot / stop" --> VMM
    SS --- VC
    VMM -- spawns --> VM
    GG <-- "vsock 1024 control" --> SS
    OMP -- stdio --> BR
    BR <-- "vsock 1025 MCP" --> MCP
    MCP --> HC
    MCP --> VC
    OMP --> FW
    FW -- "vsock 1100+i HTTP" --> PX
    FW -- "vsock 1090, 1200+i" --> EG
    FW -- "vsock 1300+i" --> GP
    GP -- "HTTPS + token" --> GIT
    GP -- "git requests, bytes" --> LED
    PX -- "HTTPS + API key" --> UP
    EG -- "vetted address,<br/>MCP secret" --> NET
    PX -- "tokens, requests" --> LED
    EG -- "egress bytes" --> LED
    SS -- "host samples: CPU, RSS,<br/>disk blocks, tap bytes" --> LED
    OMP -. "runs from" .- TOOLS
    OMP -- "stdio MCP" --> MEM
    MEM -. "embeddings via<br/>an [[llm]] forwarder" .-> FW
    GG -- "omp stdout to events" --- OMP
    SS -- SessionEvent --> HC
    HC -- publish --> ROOMS
    ROOMS --- DB
    WEB -- "/watch upgrade" --> ROOMS
    VIEW <-- "ws(s), protobuf" --> WEB
    ADMIN <-- "HTTP(S), JSON/TOML, SSE" --> WEB
    APP <-- "ws(s) protobuf / HTTP(S) JSON,<br/>authenticated, membership-checked" --> WEB
    WEB --> CTL
    CTL --> SS
    CTL --> LED

Principles that shape the design (AGENTS.md):

  • Platform-agnostic core. Everything above gopala_platform (policy, sessions, VMs, approvals, MCP tools, watching, resources) works on neutral types; mention syntax, ids and transport are the provider’s business.
  • Guests are untrusted. Everything that comes out of a VM (control frames, MCP calls, proxy requests) is bounded and validated on the host, and a guest can never name another conversation: the scope comes from the socket a connection arrives on.
  • No secrets in VMs. Platform credentials and model API keys stay on the host.
  • Measure on the host. The agent may misreport itself, so everything the dashboard shows as usage is measured outside the VM: the LLM proxy reads the provider’s own usage reports, and the firecracker process, disk image and tap device are sampled from the host. What the guest reports is shown separately and labelled unverified.
  • Reuse mature components (firefly-mcp, Firecracker + jailer, omp, rmcp, axum/hyper, sqlx, RustCrypto, rustls, iproute2, e2fsprogs) rather than reimplementing them.
Component Responsibility Key types and functions
Host / agent manager (gopala) The agent manager and CLI: wiring providers, routing and policy, approvals, alerts, sessions, resource decisions, token budgets, the LLM proxy with metering, the egress, HTTP MCP and git proxies, the usage ledger, the web server (with the embedded website), control API and monitoring, authentication, image builds (with the embedded scripts and guest agent), gopala watch. Cli, start, run, take_env, secrets, PlatformCredentials; Router, RouterCommand; HostConversation; session::{spawn, run, run_vm, handshake, host_init, Permits, Slots, Timing, Shared}; VmControl; budget::{decide, gate}; llm_proxy::serve; egress::{Egress, Allowlist, serve, serve_mcp}; usage::Ledger; control::Control; auth::Authority; monitor::websocket; web::serve; image::build
Guest agent (gopala-guest, static binary) Runs inside each VM: control link, provisioning, mounting the host’s tools, proxy settings, git config for the git proxies, resource monitor, harness supervision (omp, Claude Code, exec), the stdio MCP bridge, TCP-to-vsock forwarders, the built-in memory server and its maintenance. run, mcp, forward, configure, power_off; uplink::{Outbox, run}; provision::run; tools::{setup, resolve, writable_by, agent_path}; proxy::install; agent_config::{base, merge, tool_args}; monitor::spawn; harness::{supervise, write_mcp}
VM launcher Firecracker launcher: per-conversation persistent disk, the read-only tools drive, tap device, vsock sockets, optional jailer chroot, console log, host-side sampling. VmManager::prepare / prepare_with(id, slot, Allocation) → PreparedVm (bind_vsock, boot) → RunningVm (wait_exit, stop, sample → HostSample); verify_tools; ensure_disk, grow_disk, setup_tap, setup_chroot, firecracker_config, boot_args, jailer_args
MCP server The MCP server a guest talks to: firefly-mcp-like chat tools confined to one conversation, request_approval, list_agents, and the agent’s own-VM tools (including restart_harness). Conversation trait, Machine trait, EventLog, ScopedMcp (new, with_machine, serve), ApprovalOutcome, Allocation, Usage, VmReport, Wanted, Decision
Memory store The agent’s persistent memory inside its VM: facts, entities and relations in one SQLite file (bundled SQLite with FTS5), hybrid recall, score-based cleanup and optional model compaction. Part of the guest agent only. Store (open, remember, recall, update, forget, relate, entity, stats, cleanup), MemoryServer, maintain::run, Compactor, Embedder, ChatModel
Watch service (rooms) The read-only watch protocol: SQLite journal, personal watch tokens, sealed room keys, and the viewer client. It doesn’t listen itself; the host’s web server hands it upgraded websockets. Rooms (open, viewer, viewer_authorized, accept, publish, issue_token, watch_link, revoke, revoke_scope, set_title, events), Authorize, client::watch
Platform providers The seam between the manager and chat platforms (neutral ids, events and views, the provider trait) and the providers behind it: Lupyd (sign-in, the bot’s Firefly MLS device, mention syntax, history and search), Slack (Socket Mode and the Web API, plus the app manifest operators paste into Slack), Discord (gateway with one shard, REST, in-memory cache). Platform trait (with send_notifying, member, agents), Member, Peer, Audience, ConversationId (key), Kind, Event, Incoming (with from_agent), Content, MessageView; Lupyd, Slack (APP_MANIFEST), Discord (invite_url, PERMISSIONS)
Configuration (config schema) The TOML configuration, its defaults and cross-field validation, and its JSON Schema. Config (load, parse, validate), Behavior, Server, Rooms, Monitoring, Collaboration, Vm, Network, Jailer, Agent, HarnessKind, McpServer, LlmProxy, Git, Budget, Image, Memory
Wire formats All wire formats (quick-protobuf) and the vsock framing. codec(), encode, decode, git_api_path; constants HOST_CID, VSOCK_CONTROL_PORT, VSOCK_MCP_PORT, VSOCK_EGRESS_PORT, VSOCK_LLM_BASE_PORT, VSOCK_MCP_HTTP_BASE_PORT, VSOCK_GIT_BASE_PORT, GUEST_EGRESS_PORT, GUEST_MCP_HTTP_BASE_PORT, GUEST_GIT_BASE_PORT, CONTROL_PROTOCOL, MAX_FRAME

Only the Lupyd provider links Firefly. The guest agent depends only on the wire formats and the memory store: no TLS, no platform code, no configuration parser, so it ships as one static binary that the host embeds and bakes into the rootfs.

The gopala command (Cli, start):

Command What it does
gopala run Loads and validates the config, checks the tools image (gopala_vm::verify_tools), takes the [[llm]] keys, the HTTP MCP servers’ secrets, the admin token and the enabled platforms’ credentials out of the environment, opens rooms and the usage ledger, starts the web server (dashboard, API, /watch), connects every enabled provider and runs the agent manager until SIGINT/SIGTERM or a provider’s fatal error.
gopala web Serves only the website and the configuration API (web::Mode::Web, no /watch, no live state or actions, no platforms, no VMs), so Gopala can be configured in a browser before the first run. A missing or invalid config file falls back to the default [server] settings.
gopala check-config Validates the config, requires every [[llm]].secret_env and every agent.mcp secret_env, checks that vm.kernel and vm.rootfs exist and that the tools image verifies.
gopala image build [--out DIR] [--guest FILE] [--scripts DIR] Builds the rootfs and the tools image described by [image] (image::build) with the build scripts and the static guest agent embedded in the binary; --out defaults to <data_dir>/images. --guest FILE (or $GOPALA_GUEST) bakes a different guest binary and --scripts DIR (or $GOPALA_SCRIPTS) runs the scripts from a directory instead; both are for special cases. Prints the [vm] lines (kernel, rootfs, tools_image, tools_sha256) to paste. See Images.
gopala watch [link] [--since N] [--live] Follows one agent’s session from a watch link (gopala_rooms::client::watch).

main::run wires everything:

  1. daemon::create_private_dir(data_dir) (0700, warns about group/other bits);
  2. Rooms::open(config.rooms, config.server.watch_url(), data_dir) if rooms.enabled;
  3. usage::Ledger::open(data_dir) (usage.db) and its flusher (every 30 s);
  4. VmManager::new and session::Shared::new (the live config behind an ArcSwap, VM slots, the memory budget, LLM upstreams, the HTTP MCP upstreams with their secrets, Timing, the ledger, the activity broadcast, the registry of conversations);
  5. control::Control::new (the dashboard’s view of Shared, plus a channel to the router) and the web server (web::serve) if server.enabled;
  6. one provider per enabled platform (Lupyd::connect, Slack::connect, Discord::connect, with the secrets PlatformCredentials::take took), collected as Vec<Arc<dyn Platform>> and handed to Control too;
  7. daemon::run(platforms, rooms, shared, router_rx), which refuses to start with no platform enabled; when it returns, the ledger is flushed once more.

The long-lived tasks are:

Task Spawned by Lives What it does
Provider event pump daemon::run → Platform::run process Turns the platform’s traffic into gopala_platform::Events on one shared mpsc channel (capacity 1024). An error becomes Event::Fatal. Lupyd: polls firefly-mcp’s event log (wait_after(cursor, 200, 30 s)) while the Firefly device keeps its MLS connection.
Router loop daemon::run process Handles events one at a time in Router::handle; stops on SIGINT/SIGTERM, Event::Fatal, or when every provider has stopped.
Web server main::run → web::serve process Accepts HTTP(S) on server.bind; serves the website embedded in the binary, or server.site_dir instead when that directory exists; one hyper connection task per client (TLS handshake ≤ 10 s, request headers ≤ 15 s); /watch upgrades are handed to Rooms::viewer_authorized on their own task; each /api/stream client is a task.
Monitor connections web::app → monitor::websocket per /monitor client Connection::serve reads requests, answers them and writes pushed messages from a 256-message outbox; each subscription is a Feed task (journal replay, live events, metrics every 5 s, re-checks), aborted when it’s replaced, cancelled or the connection closes.
Usage flusher Ledger::spawn_flusher process Writes the ledger’s in-memory deltas to usage.db every 30 s and prunes daily rows older than 400 days.
Room key rotation Rooms::open while Rooms lives rotate_periodically in sealed mode.
Session supervisor session::spawn (from Router::conversation) per conversation, until the bot leaves session::run: queues deliveries, waits for a VM slot and for its memory (both by budget priority), runs the VM, restarts or retries. Receives session::Command over an mpsc channel (capacity 256).
Control reader session::spawn_reader per control connection Decodes GuestFrames into a channel (capacity 512); ends on the first bad frame.
MCP listener run_vm → serve_mcp per running VM Accepts connections on the VM’s {uds}_1025 socket; one ScopedMcp session task per connection, with the conversation and its VmControl.
LLM proxy listeners run_vm → llm_proxy::serve per running VM × [[llm]] Accept on {uds}_{1100+i}; one hyper HTTP/1 connection task per guest connection; each request is checked against the budget (budget::gate) and metered (session::llm_meter).
Egress proxy run_vm → egress::serve per running VM with vm.network = "proxy" Accepts on {uds}_1090; vets and tunnels or forwards each request (see Network).
HTTP MCP proxies run_vm → egress::serve_mcp per running VM × HTTP agent.mcp entry Accept on {uds}_{1200+i}; forward to the configured server with its secret.
Git proxies run_vm per running VM × [[git]] entry Accept on {uds}_{1300+i}; forward git smart HTTP and REST API calls to the code host with its token, enforcing repos and protected_branches (see Code hosting).
Host sampler run_vm (Timing::sample, 5 s) per running VM RunningVm::sample → VmControl::record_sample (the dashboard’s measured) and CPU/VM seconds into the ledger.
Approval waiters run_vm (guest ApprovalRequest), an MCP session, or VmControl::request per request Post the question, wait for an answer or timeout, return the result.

The Router owns the provider table (platforms: HashMap<String, Arc<dyn Platform>>, by Platform::name) and the conversation table (scopes: HashMap<ConversationId, Entry>; an Entry is the HostConversation, its SessionHandle and its VmControl). The same entries are mirrored by key in Shared::registry, which is how the dashboard reaches a conversation; the router also takes RouterCommand::Leave from the dashboard. Everything per conversation hangs off a HostConversation, which holds its Arc<dyn Platform>, implements gopala_mcp::Conversation, and is shared by the session task, the MCP sessions and the router. Each session also owns a machine::VmControl (implements gopala_mcp::Machine; also holds the dashboard’s status, measurements and last activity).

VM access goes through the Hypervisor / PreparedVm / RunningVm traits in the host’s session code, implemented for gopala_vm::VmManager and its types and by fakes in tests. Waits that aren’t configuration are in session::Timing (guest hello 10 s, first retry 5 s, VM stop grace 30 s).

A conversation is a gopala_platform::ConversationId{platform, kind, id}: the provider’s name, Kind::Group or Kind::Direct, and the platform’s group id or the peer’s username. Its key (ConversationId::key) names everything on disk, in the journal and in the VM:

Conversation Key Example
group whose id is ≤ 24 lowercase ASCII letters/digits <platform>-g-<id> lupyd-g-42
any other group id <platform>-g-<16 hex of SHA-256(id)> slack-g-1f0c…
direct chat <platform>-d-<16 hex of SHA-256(lowercased username)> lupyd-d-9a3e…

The platform part is cut to lowercase letters and digits (≤ 16), so keys are always [a-z0-9-]{1,48} (which gopala_vm validate_id and gopala_rooms valid_scope enforce), stable across restarts, never contain usernames, and never collide between a group and a DM or between platforms.

Two limits gate every boot (session::Shared), both handed out by session::Permits: waiters go by their conversation’s budget priority (high, normal, low), first come first served within one priority, and a waiter that doesn’t fit holds back everyone behind it.

  • Slots: session::Slots hands out at most vm.max_vms slot numbers (permits plus a free list). A slot number determines everything per running VM that must be unique on the host (in the VM launcher):

    Derived from slot n Value
    vsock guest CID 3 + n (firecracker_config)
    tap device (vm.network = "nat" only) gpl<n> (tap_name)
    host / guest address 172.30.0.0/16 split into /30s: host +4n+1, guest +4n+2 (nat_addrs), at most MAX_SLOTS = 16384
    guest MAC 06:00: + the guest IPv4 octets (guest_mac)
  • Memory: Shared::memory holds memory_budget_mib permits, one per MiB (machine::memory_budget_mib: vm.memory_budget_mib, or 80% of the host’s MemTotal when 0, never below vm.memory_mib). A session holds allocation.memory_mib permits while its VM runs, so VMs wait to boot until their memory fits.

The rootfs runs gopala-guest run as a systemd service (the gopala-guest.service unit, Restart=always). It:

  1. reads /proc/cmdline (sys::parse_cmdline): gopala.dns= rewrites /etc/resolv.conf (only with vm.network = "nat"), gopala.scope= is used for logging;
  2. starts the control link (uplink::run), which connects to the host and keeps reconnecting, and mounts the host’s tools if gopala.tools=1 (see below);
  3. waits for HostInit, buffering any early deliveries;
  4. with HostInit.egress_proxy set, starts the forwarder 127.0.0.1:3128 → vsock 2:1090; then writes or removes the proxy settings (proxy::install: /etc/profile.d/gopala-proxy.sh and /etc/apt/apt.conf.d/90gopala-proxy);
  5. provisions (provision::run): the agent user, /home/agent/work, sudoers, packages, setup (root) and user_setup (agent) scripts, with the proxy variables set in proxy mode; skipped when the marker /var/lib/gopala/provisioned holds the same hash of those inputs (provision::hash);
  6. starts one TCP forwarder per LlmForward in HostInit.llm, HostInit.mcp_forward and GitForward in HostInit.git (forward): 127.0.0.1:<guest_port> → vsock 2:<1100+i>, 127.0.0.1:3200+i → vsock 2:<1200+i>, 127.0.0.1:3300+i → vsock 2:<1300+i>; for [[git]] it also writes /etc/gopala-git.conf (URL rewrites to the forwarders, included from /etc/gitconfig) and the GOPALA_GIT_<NAME>_API variables;
  7. writes the system prompt to /home/agent/.gopala/system.md (configure);
  8. starts the resource monitor (monitor::spawn, see Resource management);
  9. supervises the harness (harness::supervise) as the agent user with a clean environment (HOME, USER, LOGNAME, SHELL, LANG, PATH, the proxy variables in proxy mode, plus HostInit.env). Before every start it writes the MCP config (harness::write_mcp): chat (which runs /usr/local/bin/gopala-guest mcp --port 1025), memory and every agent.mcp entry (agent_config::base), plus the agent’s own servers (agent_config::merge). omp and Claude Code also get --tools=… from HostInit.tools (agent_config::tool_args).

Three harness kinds exist (HarnessKind in config and proto); Harnesses has the details:

Kind How it runs Delivery semantics
omp (default) One long-lived omp --mode rpc <args> --append-system-prompt /home/agent/.gopala/system.md [-c]. -c (continue the latest session) is added once /home/agent/.gopala/omp-started exists, so a reboot resumes the same omp session. Restarted with backoff 1 s doubling to 60 s (reset after 300 s of uptime). omp is temporary: a gopala-owned harness is to replace it (issue #5). Idle: prompt. Busy: follow_up, or steer if Delivery.urgent (the host never sets it; the guest’s resource alerts do).
claude_code One long-lived claude -p with stream-json in and out, --permission-mode bypassPermissions, the generated MCP config and --append-system-prompt-file /home/agent/.gopala/system.md; --continue once /home/agent/.gopala/claude-started exists. Same backoff as omp. A user message on stdin; while a turn runs, with "priority": "later" (none when urgent).
exec agent.command (default sh when empty) with agent.args, once per delivery, prompt on stdin (system prompt, blank line, message); stdout lines become EVENT_TEXT. Strictly sequential; /stop kills the running process group. Approvals are ignored.

Host-provided tools. With vm.tools_image set, the VM gets a second, read-only drive and gopala.tools=1 on its kernel command line. Before provisioning, tools::setup mounts /dev/vdb at /opt/gopala read-only, nosuid,nodev, and writes /etc/profile.d/gopala-tools.sh so login shells (including user_setup) find /opt/gopala/bin first; the harness’s PATH starts with it too (tools::agent_path). A bare agent.command resolves in /opt/gopala/bin first (tools::resolve), and the guest refuses to start a harness the agent user could change (tools::writable_by): it reports lifecycle refused and status failed instead. A mount failure is reported as host tools unavailable: … (status failed). Without the image, the profile snippet is removed. The embedded tools image script builds the image (omp pinned at 19.1.0 plus the programs in [image] tools, git by default, with their shared libraries); see Images and operations.md.

stateDiagram-v2
    [*] --> Asleep: first event for the conversation<br/>(Router.conversation, session.spawn)
    Asleep --> WaitingSlot: Deliver queued, or Restart
    WaitingSlot --> WaitingMemory: Slots.acquire
    WaitingMemory --> Booting: memory permits for the allocation
    WaitingSlot --> Asleep: /stop clears the queue
    WaitingMemory --> Asleep: /stop clears the queue
    Booting --> Working: guest hello + HostInit
    Booting --> Backoff: prepare/boot error,<br/>boot timeout, VM exited
    Working --> Working: deliveries, events, usage,<br/>approvals, reconnects
    Working --> Asleep: idle_shutdown_secs without activity,<br/>or sleep_vm (VM stops, disk kept)
    Working --> WaitingSlot: restart_vm (boots again with the new allocation)
    Working --> Backoff: VM exited unexpectedly,<br/>control write failed
    Backoff --> WaitingSlot: 5 s · 2^(failures-1), at most 160 s
    Asleep --> [*]: Close (bot left)
    Working --> [*]: Close

Step by step, for a message that wakes the agent:

  1. Event. The provider turns platform traffic into an Event. For Lupyd, the Firefly device decrypts a message, firefly-mcp appends it to its event log, and Lupyd::run converts it into Event::Message(Incoming{conversation, channel_id, from, message_id, at, content, mentions_bot, addressed_text, from_agent}). from_agent marks another agent (a bot account or a holder of collaboration.agent_role); providers already drop such messages unless they mention the bot or reply to it (see Collaboration).
  2. Admission. Direct messages pass Router::admit_direct (behavior.direct_messages; the first message of a sender in this run calls Platform::admit_direct(user, allowed), e.g. to accept or decline a Lupyd message request). Group messages are ignored entirely when the conversation’s respond_to is dms (policy::respond_to: its [[behavior.conversations]] entry, else behavior.respond_to). Event::Joined runs Router::check_invite (behavior.invites, judged by Meta.owner; an unknown owner is refused for a fresh invite). check_joined_groups re-applies the invite policy to Platform::joined at startup.
  3. Conversation. Router::conversation finds or creates the HostConversation (reading Platform::meta: title, channels, members, owner), its session task and its VmControl.
  4. Record. chat_event turns the content into a ChatEventKind and HostConversation::record appends it to the conversation’s in-memory EventLog (1000 events), which feeds the MCP wait_for_events tool. Every message that reaches a conversation is recorded, whether it wakes the agent or not.
  5. Approvals first. If the conversation has an open approval and the sender is an approver (Router::is_approver), a reaction may answer it (Approvals::on_reaction), and a reply whose addressed_text parses as an answer (Approvals::on_reply) is consumed. A message from another agent skips this step and the next: it never answers an approval and never runs a command. Instead it wakes the agent only when [collaboration] is on, the conversation is a group, the message mentions the bot or replies to it, and the conversation still has one of this hour’s collaboration.max_agent_messages_per_hour wake-ups (HostConversation::take_agent_message); otherwise it’s dropped (the cap is logged).
  6. Commands. policy::parse_command(addressed_text) recognises /watch, /stop, /status, /help and /listen all|mentions (the provider has already removed a leading mention of the bot). They are answered by the host (Router::on_command) and never reach the agent.
  7. Trigger. policy::triggers(respond_to, direct, mentions_bot, replies_to_bot), with the conversation’s own respond_to (see Listening mode), decides whether to wake the agent; mentions_bot comes from the provider, replies_to_bot from HostConversation::author_of (Platform::message). Agents’ messages don’t go through it, so listening mode never lets one in.
  8. Delivery. policy::format_delivery renders alice in #general (channel 1) [message <id>, replying to <id>]:\n<text> (plus a note about attached files; alice (agent) in … for another agent, see protocols.md) and the router sends Command::Deliver to the session.
  9. Boot. If no VM runs, the session waits for a slot, then for its allocation’s memory in the budget, journals booting with the allocation, and run_vm: Hypervisor::prepare(key, slot, allocation) (VmManager::prepare_with: disk copy or grow, tap device, firecracker config, jailer chroot), binds the control, MCP and LLM proxy sockets, then PreparedVm::boot.
  10. Handshake. Within vm.boot_timeout_secs the guest must connect to the control socket and send GuestHello with CONTROL_PROTOCOL (within Timing::hello, 10 s, of connecting); the host answers with HostInit (host_init).
  11. Provisioning in the guest (first boot, or when the inputs changed), reported as GuestStatus "provisioning" and EVENT_LOG lines.
  12. Harness. omp starts; on its ready frame the guest negotiates RPC protocol v2, subscribes to subagent progress, reports ready, and hands over queued deliveries as prompts.
  13. Work and replies. omp runs autonomously. People see only what the agent sends with the chat MCP tools (send_message, react, edit_message, …), which reach the host over vsock 1025 and go out through HostConversation and Platform::send, or send_notifying for a warning or critical message (rate-limited, split to max_message_chars, journaled as EVENT_OUTBOUND). Everything else (text, thinking, tool calls, subagents) streams to the host as SessionEvents and into the journal for viewers. Usage reports arrive every agent.usage_interval_secs.
  14. Idle shutdown or sleep. Every delivery, guest event and status resets the idle timer (usage reports don’t). When vm.idle_shutdown_secs pass without any while the guest isn’t busy/provisioning, or when the agent calls sleep_vm, the host sends Shutdown; the guest stops the harness, syncs and reboots, which makes Firecracker (reboot=k) exit. RunningVm::stop waits up to 30 s, then kills, and removes the tap, sockets and chroot; the disk stays. The slot and the memory permits are released. The session journals stopped and sleeps until the next delivery.
  15. Restart. restart_vm ends run_vm the same way, journals stopped "restarting", and the session boots again right away with VmControl::desired() (a new grant takes effect here).
  16. Failure. A failed prepare/boot/handshake, an unexpected VM exit or a failed control write ends run_vm with an error: the session journals crashed, keeps queued deliveries, waits 5 s · 2^(failures-1) (capped at 160 s; commands are still answered), and boots again. After the third failure in a row it tells the conversation an operator needs to look. A clean stop resets the count.
  17. Leaving. When the bot leaves a group (MembersChanged with the bot in left, or check_invite makes it leave via Platform::leave), Router::close sends Command::Close and revokes every watch token of the conversation.

Agents know their VM’s memory and disk, are told to clean up when either runs high, and can ask the host for a different allocation, restart to apply it, or put their VM to sleep. The host decides from the configuration and its own capacity.

flowchart LR
    subgraph Guest
      MON["resource monitor<br/>sample every usage_interval_secs"]
      AG[agent / omp]
    end
    subgraph Host
      VC["VmControl"]
      DEC["machine.decide"]
      AP["request_approval<br/>(grants = approval)"]
      FS[("vms/key/allocation.json")]
      SS["session.run"]
      BUD["memory budget semaphore"]
    end
    MON -- "ResourceUsage (vsock 1024)" --> VC
    MON -- "alert: urgent steer + NOTICE" --> AG
    AG -- "vm_resources" --> VC
    AG -- "request_resources{totals, reason}" --> VC
    VC --> DEC
    DEC -- "needs approval" --> AP
    DEC -- "grant" --> FS
    AP -- yes --> FS
    AG -- "restart_vm / sleep_vm" --> VC
    VC -- "Command.Restart / Sleep" --> SS
    SS -- "acquire allocation.memory_mib" --> BUD
    SS -- "prepare_with(allocation): grow disk,<br/>vcpu_count, mem_size_mib" --> VMM[VmManager]

Measuring and alerting (guest agent’s resource monitor). After HostInit the monitor samples every usage_interval_secs (HostInit field 15; at least 5 s, 30 s if 0): MemTotal and MemAvailable from /proc/meminfo, statvfs("/") (free = what an unprivileged user may still use), the 1-minute load from /proc/loadavg, and online CPUs. Each sample goes to the host as GuestFrame.usage. When disk use reaches disk_alert_percent (85) or memory use memory_alert_percent (90), the monitor sends the harness one urgent delivery (an omp steer while busy, a prompt while idle) with the numbers and concrete cleanup steps (alert_message: remove recreatable build outputs and caches, apt-get clean, docker system prune, /tmp, stream data, stop processes; only then request_resources and restart_vm), and journals it as an error EVENT_NOTICE "resources". Each alert re-arms once use drops 5 points below its threshold (Alarm, HYSTERESIS); disk and memory are independent.

Reporting (host, VmControl). VmControl::record_usage keeps the latest sample; vm_resources returns the allocation the VM runs with, any grant waiting for a restart, the per-VM maximum (vm.max_vcpus, min(vm.max_memory_mib, budget), vm.max_disk_gib), the usage as percentages with its age, and the grant mode. /status shows the same summary, e.g. asleep · 2 vCPU, 2048 MiB, 10 GiB or working · 2 vCPU, 2048 MiB, 10 GiB · memory 75% · disk 90%.

Deciding (machine::decide). request_resources gives totals (None keeps a value). The checks run in this order:

  1. no change → Unchanged (reported as granted);
  2. a disk smaller than the disk’s current size → denied (“disks only grow”);
  3. fewer than 1 vCPU or 128 MiB → denied;
  4. above vm.max_vcpus / vm.max_memory_mib / vm.max_disk_gib → denied;
  5. memory above the host’s memory budget → denied;
  6. nothing grows (giving resources back) → granted, never needs approval;
  7. vm.grants = "never" → growth denied;
  8. disk growth that would leave less than vm.min_free_disk_gib free on the filesystem holding data_dir (Capacity::disk_free, statvfs) → denied;
  9. otherwise granted; with vm.grants = "approval" the conversation’s approvers must say yes first (HostConversation::request_approval with the new and current totals and the agent’s reason; no answer means no).

Every grant, denial, restart and sleep is journaled as EVENT_NOTICE "resources".

Persisting and applying. A grant is written to <data_dir>/vms/<key>/allocation.json (0600, via a .tmp file, fsync, rename) and becomes VmControl::desired. On load a saved grant is clamped to today’s limits (vCPUs to 1..=max_vcpus, memory to 128..=min(max_memory_mib, budget), disk never below vm.disk_gib and never shrunk). The grant applies when the VM next boots: restart_vm asks the session to restart about one second later (so the tool’s answer reaches the agent first); the session then acquires a slot and the new memory in the budget, and VmManager::prepare_with grows the disk offline (ensure_disk / grow_disk: set_len, e2fsck -pf, resize2fs) and writes vcpu_count / mem_size_mib into the firecracker config. sleep_vm stops the VM about one second later and frees its slot and memory; the next message boots it again.

The system prompt (policy::system_prompt) and the MCP INSTRUCTIONS tell the agent its resources are limited and shared, to check vm_resources before big work, to delete what it can recreate, to clean up first on an alert, to request more only when the work truly needs it and then restart, and to tidy up and sleep_vm when done.

The site Gopala serves is a self-hosted control panel: a dashboard at /dashboard/ (live state, actions, usage), a configuration editor at /configure/, the docs at /docs/, and the front page at /. The pages are static Astro builds with Alpine.js; everything live comes from the API in the host’s web server, backed by control::Control (protocols.md). They are embedded in the gopala binary; when server.site_dir (default /usr/share/gopala/site) exists as a directory, that directory is served instead.

flowchart LR
    subgraph Browser
      D["dashboard /<br/>Alpine.js"]
      C["/configure/"]
    end
    subgraph Host["gopala run"]
      W["web server<br/>session cookie or Bearer"]
      CT["Control"]
      REG["Shared.registry<br/>Entry per conversation"]
      ACT["Shared.activity<br/>broadcast"]
      LG[("usage.db")]
      CFG["LiveConfig<br/>ArcSwap"]
      R["Router"]
    end
    D -- "GET /api/state, /api/stream (SSE)" --> W
    D -- "POST …/abort, sleep, restart, leave<br/>PUT …/allocation" --> W
    C -- "PUT /api/config" --> W
    W --> CT
    CT --> REG
    CT --> LG
    CT -- "apply: hot sections" --> CFG
    CT -- "Leave" --> R
    ACT -- "activity events" --> W

Live state. Control::state builds one snapshot: the host (memory budget and how much running VMs hold, slots, free disk under data_dir), the base and maximum allocation and grant mode, the platforms with their bot usernames, and every conversation in Shared::registry with its status (the session’s state, set through VmControl::set_status, including failed with the error after a failed boot), allocation and pending grant, measured, reported, ledger totals and today’s usage, and last activity. /api/stream sends that snapshot when it changes (checked every 2 s) and every journaled event as it happens (Shared::activity, fed by HostConversation::journal_event), so the dashboard needs no polling.

Actions. Control::act sends the session Command::Abort, Sleep or Restart (restart boots an asleep VM), or asks the router to leave (RouterCommand::Leave: Platform::leave, then the conversation is closed and its watch tokens revoked), and journals an operator notice. An allocation set from the dashboard goes through VmControl::operator_set: the same machine::decide as an agent’s request, but with the grants policy treated as auto and no approval; the per-VM maximums, the memory budget, the host disk reserve and “disks only grow” still apply, and the new allocation takes effect at the next boot.

Measured, not reported. The agent may misreport itself, so every usage figure the dashboard shows comes from outside the VM:

What Measured by Where it ends up
Model requests, input/output/cache tokens, bytes up/down, errors The LLM proxy: one LlmCall per request, reported when the response ends, fails or the guest drops it. Tokens come from the provider’s own usage report in the response, parsed as it streams (Anthropic Messages, OpenAI Chat Completions and Responses, Gemini; JSON or server-sent events; at most MAX_PARSED = 8 MiB held at once). The proxy sends accept-encoding: identity upstream so reports stay readable; an encoded response is forwarded untouched and counted with zero tokens. Ledger; one EVENT_LLM journal event per call (title: upstream and model; text: status, duration, tokens)
CPU time, memory, disk, network RunningVm::sample every 5 s (Timing::sample) and when the VM stops: utime + stime and VmRSS of the firecracker process from /proc/<pid>, the disk image’s allocated blocks, the tap’s byte counters (network = "nat" only) measured (CPU percent since the previous sample, RSS, disk, network); CPU seconds and VM seconds in the ledger
Tool calls HostConversation counts each call of a chat MCP tool (conversation_info, read_messages, send_message, edit_message, delete_message, react, search_messages, request_approval) Ledger tool_calls
Messages sent Every chat message the bot sends through HostConversation::send_text (the agent’s parts, command answers, approval outcomes) Ledger messages_sent
Boots session::run before each boot Ledger boots
Egress bytes The egress proxy and the HTTP MCP proxies count the bytes of every tunnel, forwarded request and response (egress::meter) Ledger egress_bytes_up, egress_bytes_down
Git requests and bytes The git proxies count every request forwarded to a code host and the bytes each way Ledger and monitor totals git_requests, git_bytes_up, git_bytes_down

What the guest says about itself (GuestFrame.usage) is shown separately as reported and labelled unverified; the dashboard flags it when it differs from the host’s measurement by more than 20 points. The harness’s own tool executions (shell commands and the like) appear only in the journal as the guest reports them; they are not metered.

The ledger (usage::Ledger) adds deltas in memory (Ledger::add never waits on disk) and flushes them to <data_dir>/usage.db (SQLite, 0600) every 30 s and at shutdown: one row per conversation key and UTC day (usage_days, pruned after KEEP_DAYS = 400 days) plus all-time totals per key (usage_totals). /api/usage?days=N (1–90) returns the daily rows.

Hot reload. The running configuration is a LiveConfig (Arc<ArcSwap<Config>>). PUT /api/config saves the file and calls Control::apply, which merges the safe sections of the new file into the running one (control::merge): behavior, approvals, agent, budget, alerts, monitoring, collaboration, and the VM limits vcpus, disk_gib, memory_mib (only if it fits in the memory budget fixed at start), max_vcpus, max_memory_mib, max_disk_gib, grants, idle_shutdown_secs, boot_timeout_secs, min_free_disk_gib. If the merged configuration doesn’t validate, nothing is applied. Every other differing top-level section is reported as restart_required, including memory, vm.network and vm.allowed_hosts. Readers pick the new values up where they load the config: policy (including listening mode), alerts, monitoring rules, grant and budget decisions immediately; collaboration.enabled and max_agent_messages_per_hour with the next message, and list_agents asks with the current agent_role (cached answers expire within 5 minutes); agent (harness, tools, MCP servers, the system prompt with its collaboration section), approval timeouts, idle shutdown and boot timeout at a VM’s next boot. Two exceptions need a restart of Gopala: the role that marks incoming messages as from_agent (each provider gets agent_role once, at startup, through set_agent_role), and an HTTP MCP server added by a reload that needs a secret_env, because secrets are only read at startup (egress::live_mcp). Two things stay as they were for conversations that already exist [inference from the code]: the per-minute chat rate limit (each conversation’s limiter is built once, in HostConversation::new) and the base allocation (VmControl::new computes a conversation’s starting allocation once); lowered maximums do apply to new requests.

vm.network picks one of three ways out of a VM (gopala_config::Network). Model calls and the chat MCP server use vsock in every mode, so the agent can always think and talk.

Mode VM device How programs reach the internet Host setup
nat (default) eth0 on tap gpl<n>, a /30 per slot, ip= and gopala.dns= on the kernel command line directly, through the host’s NAT; nothing filters it IP forwarding and a masquerade rule (operations.md)
proxy none http_proxy/https_proxy = 127.0.0.1:3128, forwarded over vsock 1090 to the host’s egress proxy, which allows only vm.allowed_hosts and only public addresses nothing
none none not at all nothing
flowchart LR
    subgraph VM
      P["apt, git, curl, pip<br/>(honor http_proxy)"] --> F["127.0.0.1:3128<br/>guest forwarder"]
    end
    F -- "vsock 1090" --> E["egress::Egress<br/>(one per VM)"]
    E --> A{"host:port in<br/>vm.allowed_hosts?"}
    A -- no --> R403["403, notice 'refused'"]
    A -- yes --> D["resolve on the host"]
    D --> PUB{"every address<br/>public?"}
    PUB -- no --> R403
    PUB -- yes --> C["connect to a vetted address<br/>(CONNECT tunnel or http forward)"]
    C --> NET[(the server)]

In proxy mode only programs that honor the proxy variables get out; anything that opens its own sockets fails, since the VM has no network device. vm.allowed_hosts entries are host, *.domain (subdomains only) or either with :port; without a port 80 and 443 are allowed. Config::validate requires a non-empty list in proxy mode and refuses agent.packages with network = "none". The proxy’s protocol and limits are in protocols.md, its threat model in security.md.

Each [[git]] entry (at most 8) gives every VM one account on GitHub, GitLab or Gitea without the token entering the VM, in every vm.network mode including none. The host’s git proxy for entry i listens on vsock 1300+i (VSOCK_GIT_BASE_PORT); the guest agent forwards 127.0.0.1:3300+i (GUEST_GIT_BASE_PORT) to it, announced as GitForward in HostInit.git. At boot the guest writes /etc/gopala-git.conf (included from /etc/gitconfig), whose url.insteadOf rules rewrite https://<host>/, git@<host>: and ssh://git@<host>/ to the forwarder, and sets GOPALA_GIT_<NAME>_API (http://127.0.0.1:<port>/api, /api/v4 or /api/v1) for REST calls with curl; gh, glab and GraphQL are not supported.

The proxy accepts only git smart HTTP (info/refs, git-upload-pack, git-receive-pack) and the provider’s REST API, adds the token (read from token_env at startup and removed from the environment), drops the guest’s Authorization and Cookie headers, strips credential headers from responses and doesn’t follow redirects. With repos set, every git and API path must name an allowed repository. Pushes are parsed and refused whole if they create, update or delete a branch in protected_branches (default main, master); merging pull or merge requests is always refused, and so are API writes that would change a protected branch. Pushes, deleted branches, tags, opened pull requests and refusals appear as git notices in the journal. Limits, protocol details and the threat model are in protocols.md and security.md.

[budget] caps model tokens per day and per month (UTC), for each conversation and for the host (the host’s budgets). Tokens are counted as input + output + cache-write (budget::tokens; cache reads are free) from the ledger the LLM proxy books into.

  • A conversation’s limits and priority are its [[budget.conversations]] entry, else [budget.default] (budget::effective). 0 means no limit.
  • The host limits (budget.daily_tokens, budget.monthly_tokens) apply by priority (budget::share): low-priority conversations stop at low_stops_at_percent (70) of the host budget, normal at normal_stops_at_percent (90), high at 100%. High priority keeps the last slice for the conversations that matter most.
  • budget::decide runs before every model call (llm_proxy Admit, wired by budget::gate). Over a limit, the call gets a 429 budget_exceeded with a plain reason and never reaches the provider; the agent’s harness sees the error, and the journal gets one budget notice per blocked stretch. People in the conversation get a host event (“⛔ I’ve stopped: …”, see Alerts) once per episode for the conversation’s own budget, and once per UTC day when the host budget stopped it (budget::Announced).
  • Priority also orders who gets a VM slot and host memory first (session::Permits, see VM slots and the memory budget).

The dashboard shows each conversation’s budget and why it’s blocked, and changes it through PUT /api/conversations/{key}/budget (protocols.md), which rewrites that conversation’s [[budget.conversations]] entry with toml_edit and applies it at once. [budget] also hot-reloads. The check runs before each call, so calls already in flight finish and can take a conversation past its limit [inference from the code].

  • agent.tools limits the harness’s built-in tools: empty keeps its defaults; a list becomes --tools=a,b for omp (omp’s tool names) and Claude Code (Claude Code’s, such as Bash, Read, Edit) (agent_config::tool_args). If none of the names is valid the guest passes --no-tools to omp and --tools= to Claude Code rather than widen the list. The exec harness ignores it.
  • [[agent.mcp]] adds MCP servers next to chat. A command server (stdio) runs inside the VM as the agent user, so its args and env are visible to the agent: never put a secret there. A url server (http or https) is reached through the host on vsock 1200+i (egress::serve_mcp) in every network mode, and its secret (secret_env, sent as <auth_header>: <auth_prefix><secret>) stays on the host. Names are [a-z0-9_-]{1,32}, unique, and never chat. A command can come from the image: install it with image.setup (see Images) and name it here.
  • The agent’s own servers come from a file it edits itself; see The agent’s own MCP servers.

The operator’s servers reach the guest in HostInit (tools, mcp, mcp_forward); the guest writes them, with the agent’s, into the harness’s MCP config before every start (agent_config::base, agent_config::merge). See protocols.md.

agent.harness picks the program that is the agent. All three run as the agent user with the same clean environment, the same system prompt and the same MCP servers; they differ in how long they live and how messages reach them.

omp (default) claude_code exec
Runs omp --mode rpc, one session per VM claude -p with stream-json in and out, one session per VM agent.command once per message, prompt on stdin
Empty agent.command means omp claude sh
Session after a reboot or restart continues (-c) continues (--continue) none; each run starts fresh
Messages while working follow_up, or steer when urgent a user message with "priority": "later", none when urgent queued until the run ends
/stop omp’s abort the process is stopped and started again with --continue the process group is killed
Harness-level approvals omp’s confirm/select prompts become chat approvals none (bypassPermissions); request_approval only none
agent.tools --tools=… with omp’s names --tools=… with Claude Code’s names ignored
restart_harness restarts, keeps the session restarts, keeps the session refused: nothing to restart
From the tools image pinned omp (always bundled) pinned Claude Code when agent.harness = "claude_code" whatever you put in [image]
Wire protocol omp RPC stream-json stdin and stdout lines

omp is the general-purpose default: subagents, steering and follow-ups, and an extension-UI protocol the guest maps onto chat approvals. Its default agent.args are --approval-mode yolo, so it doesn’t stop for its own tool approvals. It’s temporary: a gopala-owned harness is to replace it (issue #5).

Claude Code is a coding agent. It reaches Anthropic through an [[llm]] proxy like any other model call (ANTHROPIC_BASE_URL in that entry’s guest_env), so the key stays on the host and the tokens are metered and budgeted. It runs with --permission-mode bypassPermissions: the VM is the sandbox, and approvals for critical steps go through the request_approval chat tool, as the system prompt says. Claude Code refuses that mode as root; the agent account isn’t root. The guest sets CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 (overridable through agent.env) and passes agent.args after its own flags, e.g. ["--model", "sonnet"]. gopala image build bundles the pinned release (2.1.319, checked against the release manifest and its sha256, about 340 MB) into the tools image when agent.harness = "claude_code" (CLAUDE_CODE=1 for build-tools-image.sh); installing it with image.setup works too. Never install it with agent.user_setup: the agent could then replace its own harness.

exec runs any command per message, for example codex exec. The command gets the system prompt, a blank line and the message on stdin; its stdout lines become EVENT_TEXT. It has no session between messages, no approvals and no steering. The guest still writes the merged MCP config to /home/agent/work/.mcp.json (and omp’s path) for commands that read it.

policy::system_prompt builds the instructions every harness gets, in sections: who the agent is (its name, the platform, the group or direct chat, its VM and workspace), Talking to people (only chat-tool messages are visible, keep replies short, severities), Working (work autonomously, approvals only for critical steps, look before installing, prefer uv and pnpm), Resources (limits, cleanup, request_resources, budgets, sleep_vm), Extending yourself (adding MCP servers and restart_harness, plus the sudo or package-manager line), Memory (when memory.enabled), Working with other agents (only in a group with [collaboration] on), and last From the operator with agent.system_prompt.

The operator’s text is appended under its own heading and never replaces the built-in sections; a blank one adds nothing. The guest writes the result to /home/agent/.gopala/system.md. omp gets it with --append-system-prompt and Claude Code with --append-system-prompt-file, so each keeps its own default system prompt underneath and Gopala’s comes after it. exec gets it at the top of every run’s stdin. The prompt travels in HostInit, so a changed agent.system_prompt or [collaboration] reaches an agent at its VM’s next boot.

restart_harness {reason} restarts the harness inside the running VM. Nothing else restarts: the VM, its allocation, the workspace and the session stay.

  1. The tool trims the reason and cuts it to 300 characters; an empty one is refused. VmControl::restart_harness refuses an exec harness, journals an EVENT_NOTICE "harness" with the reason, answers, and about 1 s later sends Command::RestartHarness, so the answer reaches the agent before its harness stops.
  2. The session forwards it as HostFrame.restart_harness when the guest is connected. Without a VM it’s ignored; while the guest is reconnecting it’s dropped with a warning.
  3. The guest journals lifecycle restarting (“restarting harness: ”), sends SIGTERM and kills the process if it hasn’t exited after 10 s. It rewrites the MCP config, then starts the harness again at once with -c or --continue. This isn’t a crash: no backoff, no crashed event, and a pending crash backoff is cut short.
  4. Deliveries that arrived meanwhile stay queued. The new run’s first prompt says “Your harness restarted (reason: …) and this session goes on.”, followed by what became of the agent’s MCP file (“From /home/agent/.gopala/mcp.json: loaded web; left out …”).

restart_vm is the heavier sibling: it reboots the VM to apply a resource grant.

An agent can add MCP servers without the operator. The system prompt tells it how: install the server, add it to /home/agent/.gopala/mcp.json in the usual mcpServers format, then call restart_harness. The guest reads that file before every harness start (first boot, a crash restart, restart_harness) and merges it into the generated config (agent_config::merge):

  • The file. Created as {"mcpServers": {}} (0600, owned by the agent) the first time it’s missing. It’s read with sys::read_owned: no symlink anywhere in the path, owned by the agent’s account, at most 64 KiB. A file that fails any of these, isn’t JSON, or whose mcpServers isn’t an object is left out whole.
  • Entries. {"command", "args"?, "env"?} for stdio or {"url"} for streamable HTTP, with an optional "type" of stdio or http; other keys are ignored. Names follow the operator’s rule ([a-z0-9_-]{1,32}), URLs must be http(s), and nothing may contain a NUL byte.
  • Reserved names. chat, memory (even when the built-in memory server is off) and every [[agent.mcp]] name, valid or not, belong to the host. An entry using one is skipped, so the agent can’t replace a server the operator configured.
  • Limits. At most 32 servers are taken; invalid entries are skipped one by one. Each skip is logged and listed, with its reason, in the note the agent gets after a restart.
  • Where it goes. omp and exec: /home/agent/.omp/agent/mcp.json and /home/agent/work/.mcp.json. Claude Code: /home/agent/.gopala/claude-mcp.json, passed with --mcp-config and --strict-mcp-config, so a stray .mcp.json in the workspace doesn’t count. The guest writes these as root with sys::write_owned, which refuses a symlink anywhere in the path (openat2 with RESOLVE_NO_SYMLINKS) and hands the file to the agent.

The agent’s servers run inside the VM with the agent’s own rights, like anything else it starts. An HTTP server it adds isn’t proxied by the host: the harness connects to that URL directly, so it follows vm.network (anything with nat, the allowed hosts with proxy, nothing with none), and no secret is added for it.

A VM boots from three files: vm.kernel, vm.rootfs (copied once per conversation into its persistent disk) and, optionally, the read-only vm.tools_image. gopala image build (image::build) makes them from [image] with the build scripts embedded in the binary (build-rootfs.sh for the rootfs, build-tools-image.sh for the tools image) and the embedded static guest agent, which it bakes into the rootfs with its systemd unit. The kernel is the pinned Firecracker CI kernel the rootfs script downloads. --guest FILE and --scripts DIR replace the embedded guest or scripts, for special cases only:

[image] Passed to Effect
base (ubuntu-24.04 default, debian-12) build-rootfs.sh as BASE Ubuntu: Firecracker’s CI image; Debian: debootstrap. Writes gopala-<base>.ext4
packages build-rootfs.sh as PACKAGES apt packages baked into the rootfs in a chroot (needs root)
setup build-rootfs.sh, one --setup <command> argument each shell commands run as root with bash -c in the chroot after the packages, in order, with a clean PATH and the build host’s http(s)_proxy/no_proxy when set (needs root); a failing command stops the build. What they install is in every VM from its first boot
rootfs_size_gib (4) build-rootfs.sh as ROOTFS_SIZE the rootfs size; a conversation’s disk grows from it to vm.disk_gib
tools (["git"]) build-tools-image.sh as TOOLS host programs bundled next to omp with their shared libraries, into gopala-tools.ext4 and its .sha256
agent.harness = "claude_code" build-tools-image.sh as CLAUDE_CODE=1 the pinned Claude Code as bin/claude on the tools drive

image::build sets every variable explicitly, so a stray PACKAGES, TOOLS or CLAUDE_CODE in the caller’s environment can’t change the build, and passes each setup command verbatim as its own argument (no shell string is built on the host). It then prints the [vm] lines: the newest kernel in the output directory, the rootfs, the tools image and its SHA-256. [image] is only read by this command; a changed [image] takes effect when you build and point [vm] at the new files. Packages and setup baked into the image cost nothing at boot, while agent.packages and agent.setup run when each VM provisions (packages need network).

The agent decides how serious a message is; the operator decides who that reaches. send_message takes severity = info (default), warning or critical, and nothing else about notifying: no platform, no names, no mention syntax (gopala_mcp::Severity).

flowchart LR
    A["agent: send_message<br/>severity = critical"] --> S["HostConversation::send"]
    H["host event: VM keeps failing,<br/>budget block"] --> E["HostConversation::host_event<br/>(alerts.host_events)"]
    S --> AL{"alert(severity)"}
    E --> AL
    AL -- "info, direct chat,<br/>or empty audience" --> P1["Platform::send"]
    AL -- "hourly cap used up" --> P2["Platform::send<br/>+ note to the agent"]
    AL -- "audience" --> P3["Platform::send_notifying"]

HostConversation::audience picks the entries: the conversation’s [[alerts.conversations]] entry if there is one, else alerts.warning / alerts.critical (defaults: nobody for warning, ["everyone"] for critical). Audience::parse reads everyone, here (people who are online, where the platform knows), role:<name> and user:<username>. Info messages notify nobody, and neither does anything in a direct chat, where the one person gets the message anyway.

alerts.max_per_hour (default 4) caps notifying messages per conversation, both severities together and host events included (HostConversation::take_alert, a governor limiter rebuilt when the setting changes). Past the cap the message still goes out, through plain Platform::send, and the tool result carries note: "sent without notifying anyone: …". Only the first part of a split message notifies; later parts are ordinary sends.

Host events go out at alerts.host_events severity (default critical), bypassing the agent’s per-minute rate limit but not the hourly cap:

Event Text How often
Three VM failures in a row (session::run) “⚠️ My workspace keeps failing to start; an operator needs to look at it.” each time the retry loop reaches it
The conversation’s own budget stops model calls (budget::gate) “⛔ I’ve stopped: This conversation used its daily budget …” once per episode, until a call is allowed again
The host budget, at this conversation’s priority share, stops model calls “⛔ I’ve stopped: The host used 90% of its daily budget …” once per UTC day per conversation

Each provider renders the audience in its own syntax and lets exactly those mentions ping (Platform::send_notifying). Names the platform doesn’t know are logged and left out; the message still goes out.

everyone here role:<name> user:<name>
Slack <!channel> <!here> <!subteam^ID>, the user group found by handle in usergroups.list (cached 5 min; needs the usergroups:read scope) <@U…>
Discord @everyone @here <@&role>, by role name <@id>, from the cache or a member search
Lupyd @everyone as plain text @here as plain text skipped (Lupyd has no role mentions) @username, checked against the group’s members
Default (Platform trait) the audience as plain text (Audience::plain_prefix) in front of send

On Slack the mentions go in a mrkdwn section block, the message in a markdown block, and the top-level text (used for the notification) holds both; messages over 12,000 characters go out as escaped mrkdwn with the mentions in front. On Discord the first part carries the mentions with allowed_mentions set to exactly them (everyone covers both @everyone and @here), and the bot needs “Mention @everyone, @here and All Roles” in the channel for those to ping (logged when it lacks it). Ordinary Discord messages never ping everyone or roles.

[alerts] hot-reloads. Severity also shows in the journal: outgoing messages are titled message (warning) or message (critical).

behavior.respond_to sets who wakes the agent everywhere; [[behavior.conversations]] entries (key, respond_to) override it for one conversation (policy::respond_to, used by Router::handle for both the dms check and policy::triggers). respond_to = "all" in one group is “listening mode”: every message reaches its agent, mentioned or not.

Two ways to switch it at run time, both through control::set_respond_to:

  • In the chat: /listen all (or on) and /listen mentions (or off), from an approver (Router::is_approver, the same check as /stop). Not in direct chats, which already deliver everything. The bot answers “👂 Listening: …” or “🔕 Only messages that mention me …”, and the journal gets a listen notice naming who changed it.
  • On the dashboard: a toggle per group, PUT /api/conversations/{key}/behavior; journaled as an operator notice.

set_respond_to edits that conversation’s [[behavior.conversations]] entry in gopala.toml with toml_edit (control::edit_conversation_entry, so comments and layout stay), validates the result, writes it atomically with a .bak, and only then swaps it into the live configuration. All writers of the file go through one control::ConfigFile and its lock: PUT /api/config, budget edits, the dashboard toggle and /listen, so two saves never interleave. If saving fails the chat answer says an operator needs to look at the configuration and nothing changes.

In a group whose respond_to is dms, group messages are ignored before commands are parsed, so /listen can’t switch it from the chat; the dashboard can.

A group can hold more than one agent: another Gopala signed in with its own bot account, any other bot, or an account someone marked as an agent. [collaboration] lets them find each other, ping each other with @mentions and split the work, while each one decides for itself what to do with a ping. The host decides who counts as an agent, which of their messages wake this one and how often, and what they can never do.

Who’s an agent. Bot accounts (Slack bot users, Discord bots) and members holding the role named collaboration.agent_role (default agent): a Lupyd group role, a Discord role, a Slack user group handle. Give that role to your own bot too, so others can find it. Providers mark such messages Incoming.from_agent and answer Platform::agents; what each platform counts and where the bio comes from is in providers.md.

How a split goes. A person asks agent A; A looks for help, pings B with a request that stands on its own, and both work at once:

sequenceDiagram
    participant P as alice (a person)
    participant A as agent A (this bot)
    participant H as A's host
    participant B as agent B (another bot)
    P->>A: @A compare our API latency with last month and write it up
    A->>H: list_agents
    H-->>A: researcher: "pulls metrics and papers" (Platform::agents, cached 5 min)
    A->>P: I'll write it up, and @researcher is getting last month's numbers
    A->>B: @researcher please get p50/p99 for October from the metrics API, reply with a table
    Note over B: its own host: from_agent, mentions B, within B's hourly cap
    B->>B: decides: help, decline or say it's busy
    par A drafts its part
        A->>A: this month's numbers, the write-up
    and B works
        B->>B: October's numbers
    end
    B->>A: @A here's the table: …
    Note over H: from_agent + mention, within max_agent_messages_per_hour:<br/>delivered as "researcher (agent) … [message N]"
    A->>P: the comparison and the write-up (A owns the answer)

What wakes an agent. A message from another agent wakes this one only if all of these hold (Router::on_message):

  • collaboration.enabled is true (with it off, agents’ messages are ignored and list_agents errors);
  • the conversation is a group;
  • the message mentions the bot or replies to one of its messages (the provider filters first; the router checks again);
  • the conversation has a wake-up left this hour: collaboration.max_agent_messages_per_hour (default 30, at most 3600, 0 = none) per conversation, counted by a governor limiter that is rebuilt when the setting changes (HostConversation::take_agent_message). Past it, the message is dropped and logged. People’s messages don’t count.

Listening mode never applies to agents, and an agent’s message is never parsed as a command or an approval answer; its reactions don’t answer approvals either. Messages that pass are delivered with the sender marked: researcher (agent) in #general (channel 1) [message N]: (see protocols.md). Messages the provider drops never reach the conversation’s EventLog, but read_messages still shows them, since it reads the platform’s history.

What the agent is told. In a group with collaboration on (decided when the VM boots), the system prompt adds “Working with other agents”: the group is a shared workspace; list_agents shows who else is here and what they’re good at; an @mention is a ping, not an order, and the agent may decline or say it’s busy; hand a part that suits another agent better to it with a self-contained request (goal, inputs, what to send back) and keep working in parallel; say in the chat who does what; the agent the person asked owns the final answer; and don’t mention agents just to thank or acknowledge them, because messages between agents are capped. The hourly cap is what stops two agents from keeping each other busy forever; the prompt only makes hitting it rare.

list_agents takes no parameters and returns at most 50 agents (username, display_name, about up to 2000 characters, roles, bot), cached per conversation for 5 minutes and dropped when members or the group change; details in protocols.md. [collaboration] hot-reloads (see Control plane and metering), except that a changed agent_role reaches the providers’ from_agent marking only after a restart of Gopala.

Each agent keeps facts across restarts in its own VM: the memory store gopala-memory stores them in /home/agent/.gopala/memory.db on the conversation’s persistent disk, and gopala-guest memory serves them to omp as the memory MCP server (tools in protocols.md). The host only passes settings (HostInit.memory, built by session::memory_config from [memory]) and proxies model calls; it never reads or stores the memories. The system prompt tells the agent to recall before work that may build on earlier context and to remember durable facts (policy::system_prompt, only when memory.enabled).

The design follows mem0’s useful parts (atomic facts, dedupe on write, entity links, hybrid retrieval) without a model on the write path: the agent itself writes short facts, and a model runs only as an optional, bounded compaction job.

flowchart LR
    subgraph VM["inside the VM, as the agent user"]
      O[omp] -- "remember / recall / …" --> MS["gopala-guest memory<br/>(one per omp process)"]
      MS --> DB[("memory.db<br/>SQLite WAL + FTS5")]
      SUP["gopala-guest run<br/>(supervisor)"] -- "every cleanup interval" --> MP["gopala-guest memory --maintain"]
      MP --> DB
    end
    MS -. "/v1/embeddings" .-> PX["host LLM proxy<br/>(key, metering, budget)"]
    MP -. "/v1/embeddings, compaction" .-> PX

Storage. One SQLite file, 0600 in a 0700 directory, owned by the agent; WAL, a 10 s busy timeout and short IMMEDIATE write transactions, because omp’s subagents start their own memory servers on the same file. Tables: memories (text, normalized-text hash, importance 0–1, pinned, recall count and time, archived time), an external-content FTS5 index (porter unicode61), embeddings, entities (one per case-insensitive name), memory_entities, relations (unique per from, relation, to), meta and a compactions log of the last 100 runs. Migrations go by PRAGMA user_version.

Writing. remember normalizes the text, then merges instead of inserting when it finds an exact duplicate (same normalized text, case-insensitive) or a near one: word-set Dice over the 10 best BM25 matches, or cosine when embeddings are on, at duplicate_threshold (0.85) or above. Entities and relations are linked in the same transaction. Limits: text 2 KiB, names 128 B, 16 entities and 16 relations per memory.

Recall fuses up to three ranked lists with reciprocal rank fusion (k = 60): FTS5 BM25, cosine over stored vectors (brute force), and memories of entities the query names. Each hit’s score is then multiplied by 1.5 for an entity match, by (0.5 + importance) and by (0.5 + 0.5 × recency). Results are capped at 50, and each hit counts as a use.

Cleanup runs in the guest supervisor as the agent user (memory::spawn_maintenance runs gopala-guest memory --maintain as a child process, so root never opens files in the agent’s home): first 120 s after boot, then every max(cleanup_interval_secs, 60) seconds, killed after 30 minutes. A memory’s score is

score = importance × (1 + ln(1 + recalls)) × 2^(−age since last use / half_life_days)
Rule Effect
More than max_memories active (checked on every write and in each pass) the lowest scores are archived down to 90% of the cap; archived memories drop out of recall
Database over max_db_mib the whole archive is deleted first, then the lowest scores are archived and deleted down to 90% of the size cap
Archived longer than archive_days deleted for good
Pinned never archived or evicted
Near duplicates a sweep compares up to 50 memories in each of 50 entities per pass and merges them
Also each pass orphan entities dropped, FTS5 optimize, incremental vacuum, WAL checkpoint, and up to 256 missing embeddings filled in (batches of 32)

Embeddings ([memory.embeddings], optional) are POST /v1/embeddings to the guest port of the named [[llm]] entry, so the host adds the key, meters the tokens and applies the budget. A tool call waits at most 8 s for one and goes on without it; the next pass backfills.

Compaction ([memory.compaction], off unless set) runs from the maintenance pass every every_hours once there are min_memories active memories. It builds up to max_clusters clusters of 2 to 40 related memories (at most 12,000 characters each): first memories sharing an entity, then pairs at cosine 0.75 or more among the 500 newest embedded memories. Each cluster goes to the model (Anthropic /v1/messages or OpenAI /v1/chat/completions, format), which answers with merge, rewrite, delete or keep operations. Answers are validated strictly (only ids from the cluster, each once, pinned memories only kept, text within limits), every valid plan is applied in one transaction, and a failed call, a bad answer or a cluster that changed meanwhile leaves the store as it was. Compaction tokens go through the same host proxy, so they show on the dashboard and count against budgets.

[memory] isn’t hot-reloaded: it reaches a VM in HostInit, so a change needs a restart of Gopala and then applies at each VM’s next boot.

People follow their agents through two channels besides the chat: /watch (sealed, read-only, a personal link per conversation) and /monitor, a protobuf request/response websocket any app can speak, with an HTTP mirror under /api/monitor/. The Lupyd app is the first such app: it connects with the person’s own Lupyd sign-in token and gets the full stream of events (every task, tool call and subagent), while Slack and Discord users keep following their agents through chat messages and watch links. The wire format is in protocols.md.

sequenceDiagram
    participant App as Lupyd app
    participant W as web server / monitor
    participant A as Authority
    participant P as Platform (Lupyd provider)
    participant C as Control
    App->>W: GET /monitor (websocket upgrade)
    App->>W: MonitorRequest{id 1, authenticate{lupyd_token}}
    W->>A: LupydTokens::verify (JWKS, RS256, iss, aud, exp)
    A-->>W: username = uname
    W-->>App: response 1: Identity{platform "lupyd", username}
    App->>W: request 2: list
    W->>C: monitored(identity)
    C->>A: Membership::member (cache, else ask)
    A->>P: member(conversation, username)
    P-->>A: Member{roles, permissions, owner} or None
    A-->>C: evaluate [monitoring] rules → Grant{output, metrics}
    C-->>W: conversations with can_read_output / can_read_metrics
    W-->>App: response 2: ConversationList
    App->>W: request 3: subscribe{key, after, output, metrics}
    W-->>App: response 3: ok
    loop until unsubscribed or access lost
      C-->>App: MonitorEvent{key, SessionEvent} (journal replay, then live)
      C-->>App: MonitorMetrics every 5 s while the VM runs
      Note over W,A: on cache expiry or a membership event: check again,<br/>end with 401/403/404 under id 3
    end

Pieces:

  • Identities (auth::Identity): the operator (admin token or dashboard session), or a person on a platform, optionally limited to one conversation (a watch token’s). monitor::Caller also remembers what it needs to tell later whether the credential still holds: a Lupyd token’s exp, or the watch token, checked again with Rooms::token_holder.
  • Lupyd tokens (auth::LupydTokens): Auth0 JWTs verified against the key set at {platforms.lupyd.auth0_domain}/.well-known/jwks.json (auth::Jwks, a KeySource so tests can use a fixed key set), only with monitoring.lupyd_tokens.
  • Membership (auth::Membership): answers from Platform::member, cached per (conversation key, username) for monitoring.membership_cache_secs, at most 4096 entries. The router drops entries on MembersChanged (the people who joined or left), Updated and the bot leaving (Membership::invalidate), which also wakes every live subscription to check again. Direct chats don’t ask the platform: the peer is the only member and its owner (auth::direct_member).
  • Rules (auth::evaluate): [monitoring] output and metrics, or the conversation’s [[monitoring.conversations]] entry; any match grants. approvers reuses auth::is_approver, the same function the router uses for approvals and /stop.
  • Answers (Control::grant, monitored, monitor_status, metrics, usage_history, journal): None from grant (not a member, other platform, other conversation) becomes 404. Metrics come from the same host measurements and ledger as the dashboard.
  • /watch uses the same rules for a watch token’s holder (web::watch_authorization), at hello and when the answer expires.

[monitoring] hot-reloads: each check reads it live, so new rules apply to the next request, and a cached membership answer lasts at most membership_cache_secs.

sequenceDiagram
    autonumber
    actor Alice
    participant FF as Firefly server
    participant P as Lupyd provider
    participant R as Router
    participant S as session task
    participant V as VmManager / firecracker
    participant G as gopala-guest
    participant O as omp
    participant B as gopala-guest mcp
    participant M as ScopedMcp (host)
    participant J as Rooms journal

    Alice->>FF: "@bot fix the CI" (MLS-encrypted)
    FF->>P: deliver, decrypt (Firefly device)
    P->>R: Event.Message(Incoming{mentions_bot: true, addressed_text})
    R->>R: record → EventLog, approvals? command? triggers(Mentions) = true
    R->>S: Command.Deliver{text: "alice in general (channel 1) [message …]: @bot fix the CI"}
    S->>S: Slots.acquire, memory budget acquire(allocation.memory_mib)
    S->>J: LIFECYCLE booting "2 vCPU, 2048 MiB, 10 GiB"
    S->>V: prepare_with(key, slot, allocation): disk, tap gplN, vm.json
    S->>S: bind {uds}_1024, _1025, _1100+i
    S->>V: boot (firecracker --no-api --config-file)
    V-->>G: kernel boots, systemd starts gopala-guest run
    G->>S: vsock 2:1024 GuestHello{protocol 1}
    S->>G: HostInit
    G->>G: provision (if needed), forwarders, mcp.json, system.md, monitor
    G->>O: spawn omp --mode rpc … [-c]
    O-->>G: {"type":"ready"}
    G->>O: negotiate_protocol v2, set_subagent_subscription progress
    G->>S: GuestStatus ready, LIFECYCLE ready
    S->>G: Delivery
    G->>O: {"type":"prompt","message":…}
    O-->>G: agent_start, message_update, tool_execution_*…
    G->>S: SessionEvent stream (INPUT, TEXT, THINKING, TOOL_CALL, …)
    G->>S: ResourceUsage every usage_interval_secs
    S->>J: publish (seq assigned, stored, fanned out)
    O->>B: tools/call send_message (stdio JSON-RPC)
    B->>M: vsock 2:1025
    M->>P: HostConversation.send → split, rate limit, Platform.send
    P->>FF: encrypt_and_send_group
    FF->>Alice: "On it…"
    M->>J: OUTBOUND message
    O-->>G: agent_end
    G->>S: GuestStatus idle
    Note over S: idle_shutdown_secs later (or the agent calls sleep_vm)
    S->>G: Shutdown
    G->>O: close stdin (kill after 10 s)
    G->>G: sync, reboot → firecracker exits
    S->>V: stop: wait ≤30 s, remove tap/sockets/chroot, keep disk
    S->>J: LIFECYCLE stopped "idle"

All paths end in HostConversation::request_approval. It refuses a fourth open question in the conversation (MAX_OPEN_APPROVALS = 3) and counts the question against behavior.max_messages_per_minute, then posts the question (approvals::render) with Platform::send, immediately opens a request in Approvals (so a quick answer isn’t missed), journals it (EVENT_OUTBOUND and APPROVAL asked), seeds reactions (✅ ❌, or 1️⃣…🔟) with Platform::react, waits for an answer or the timeout, then journals answered/timed_out and posts a summary as a reply to the question. The wait is at most approvals.timeout_secs; no answer means no. A third caller is VmControl::request for resource grants under vm.grants = "approval" (a yes/no question).

sequenceDiagram
    autonumber
    actor Bob as Bob (approver)
    participant R as Router
    participant A as Approvals
    participant HC as HostConversation
    participant M as ScopedMcp
    participant S as session run_vm
    participant G as gopala-guest (Omp)
    participant O as omp

    rect rgb(240,240,255)
    Note over O,M: Path 1: the agent calls the MCP tool
    O->>M: request_approval{question, options?, timeout_seconds?}
    M->>M: validate (0 or 2..10 options, non-empty question), timeout ≤ approvals.timeout_secs
    M->>HC: request_approval(question, options, timeout)
    end
    rect rgb(240,255,240)
    Note over O,S: Path 2: omp asks through its extension UI
    O->>G: extension_ui_request{id, method: confirm|select, title, message, options, timeout ms}
    G->>S: GuestFrame.approval ApprovalRequest{id, question, options, timeout_secs}
    S->>S: options ≤ 10 (exactly 1 → yes/no), timeout = min(request, config)
    S->>HC: request_approval (spawned task)
    end
    HC->>Bob: "🔐 Approval needed …" (+ ✅ ❌ seeded)
    HC->>A: open(key, message_id, options)
    Bob->>R: reply "yes" (or react ✅ / 2️⃣)
    R->>R: is_approver(conversation, Bob)?
    R->>A: on_reply / on_reaction → parse → oneshot Answer{choice, by}
    A-->>HC: Answer
    HC->>Bob: "✅ Approved by bob" (reply to the question)
    alt Path 1
      HC-->>M: ApprovalOutcome{approved, choice?, choice_text?, answered_by, timed_out}
      M-->>O: tool result (JSON)
    else Path 2
      HC-->>S: outcome
      S->>G: HostFrame.approval ApprovalResult{id, choice, answered_by, timed_out}
      Note right of S: buffered in unsent_answers while the guest is reconnecting
      G->>O: extension_ui_response{id, confirmed | value | cancelled[, timedOut]}
    end

Answer rules (the host’s approvals, Router::is_approver):

  • Who may answer: approvals.approvers = members (anyone in the conversation), owner (Meta.owner of a group, or the other person in a direct chat), or allowlist. The same rule gates /stop.
  • A reply addressed to the question (reply_to = the question’s message id) answers that request; a bare message without reply_to answers only if it is the conversation’s single open request. Replies are parsed from addressed_text, so @bot yes works too.
  • Yes/no replies: yes y approve approved ok okay sure go "go ahead" / no n deny denied reject nope don't "do not"; reactions ✅ 👍 ✔ / ❌ 👎 ✖ (variation selectors ignored).
  • Choices: a number 1..=n (optionally #2), the option’s own text (case-insensitive, trailing ./! ignored), or the keycap reaction.
  • omp input/editor UI requests are cancelled immediately; notify becomes an EVENT_NOTICE; cancel forgets the pending request. Approval ids are unique per guest process (NEXT_APPROVAL), so a stale answer after a harness restart matches nothing.
sequenceDiagram
    autonumber
    participant MON as guest monitor
    participant O as omp (agent)
    participant M as ScopedMcp
    participant VC as VmControl
    participant HC as HostConversation
    participant S as session
    participant V as VmManager

    MON->>S: ResourceUsage (disk 88%)
    MON->>O: steer "Resource alert — disk 88% used … clean up …"
    O->>O: cargo clean, rm caches, df -h
    O->>M: vm_resources
    M->>VC: report → allocated, maximum, usage, grants
    O->>M: request_resources{disk_gib: 20, reason}
    M->>VC: request(Wanted, reason)
    VC->>VC: decide: maximums, budget, disk reserve, grants mode
    opt grants = approval
      VC->>HC: request_approval("My VM needs more room: …")
      HC-->>VC: approved
    end
    VC->>VC: persist allocation.json, desired = new, NOTICE "granted …"
    VC-->>O: Decision{granted: true, after_restart, message}
    O->>M: restart_vm{reason}
    M->>VC: restart → NOTICE, Command.Restart in 1 s
    VC-->>O: "restarting in a moment …"
    S->>S: Shutdown guest, stop VM, journal stopped "restarting"
    S->>S: slot + memory permits for the new allocation
    S->>V: prepare_with(key, slot, new allocation): grow disk, new vm.json
    S->>S: boot, handshake, HostInit, omp continues with -c
sequenceDiagram
    autonumber
    actor Carol
    participant R as Router
    participant Rm as Rooms
    participant DB as rooms.db
    participant DM as DM HostConversation
    participant Web as web server /watch
    participant W as gopala watch (client)

    Carol->>R: "@bot /watch" in group
    R->>Rm: issue_token("lupyd-g-42", "carol")
    Rm->>DB: delete carol's old tokens for the scope and expired ones, insert {id, sha256(secret), kwk, expires_at}
    Rm-->>R: token gpl1.id.secret
    R->>DM: say("👀 Watch me work … watch_url + token")  (direct chat on the same platform)
    R->>Carol: "📬 Sent you a private watch link, carol." (in the group)
    Carol->>W: GOPALA_WATCH_LINK=… gopala watch
    W->>Web: TCP (+TLS), GET /watch with websocket upgrade
    Web->>Rm: Rooms.viewer(websocket)
    W->>Rm: ClientFrame.hello{token, since_seq, live_only}
    Rm->>DB: verify(token): lookup id, check expiry, constant-time sha256 compare
    Rm->>Rm: lock room: capacity check, register viewer, subscribe live
    Rm->>W: Welcome{scope, title, encryption, latest_seq}
    Rm->>W: RoomKey{epoch, nonce, wrap(kwk, room key)}  (sealed mode)
    loop replay pages of 256, seq in (since_seq, latest_seq]
      Rm->>DB: events(scope, last, latest, 256)
      Rm->>W: SealedEvent (sealed with the current key) or SessionEvent
    end
    Rm->>W: HistoryDone{latest_seq}
    loop live
      Rm->>W: SealedEvent / SessionEvent (seq > last)
      W->>Rm: Ping{nonce}
      Rm->>W: Pong{nonce}
    end
    Note over Rm: every key_rotation_secs, or when a member leaves
    Rm->>W: RoomKey{epoch+1, …}  (queued before any event sealed with it)
    Note over R,Rm: Carol leaves the group (Event.MembersChanged left)
    R->>Rm: revoke("lupyd-g-42", "carol")
    Rm->>DB: delete carol's tokens
    Rm->>W: ErrorFrame "access revoked", close
    Rm->>Rm: rotate the key for remaining viewers

The watch link is <watch_url>#<token>, where watch_url is server.public_url with http(s) turned into ws(s) plus /watch. In a direct chat the link is sent as the reply itself. If the bot cannot message the requester directly, the group gets Couldn't message you directly: …. With rooms.enabled = false the answer is Watching is turned off on this bot.

The control connection can drop while the VM keeps running: the guest process crashed and systemd restarted it (Restart=always, RestartSec=1), or the vsock connection broke.

sequenceDiagram
    autonumber
    participant S as session run_vm
    participant G as gopala-guest (uplink)
    participant H as harness / omp

    G--xS: control connection closes
    S->>S: reader returns None → connected = false
    Note over S: deliveries stay queued, approval results go to unsent_answers,<br/>idle timer and VM exit still watched
    loop backoff 200 ms → 5 s
      G->>S: vsock connect 2:1024
    end
    G->>S: GuestHello
    S->>S: accept on the same listener, wait ≤ 10 s for hello
    S->>G: HostInit (resent)
    alt same guest process (link dropped)
      G->>G: ignores the repeated HostInit
    else restarted guest process
      G->>G: provision (marker matches: skipped), forwarders, configure, monitor
      G->>H: start omp with -c (continues the session)
    end
    S->>G: unsent ApprovalResults, then queued Deliveries
    G->>S: frames queued in the Outbox while disconnected (≤ 2000, oldest dropped)

The VM exiting is never a reconnect: run_vm treats it as a failure and the session backs off and boots again. A reconnecting guest’s hello is not re-checked against CONTROL_PROTOCOL, and the session loop waits for it inline (up to 10 s).

<data_dir>/ 0700 (created by daemon::create_private_dir; warns if group/other bits set)
├── lupyd/ the Lupyd provider's Firefly device directory (firefly-mcp); Slack and Discord keep nothing on disk
│ ├── session.json Auth0 session with the refresh token (0600, written by firefly-mcp)
│ ├── firefly.db firefly-client store: MLS state, messages, groups, search index
│ └── firefly-mcp.lock one process per account and data directory
├── rooms/ 0700
│ └── rooms.db (+ -wal, -shm) 0600, SQLite WAL: tables `events` and `tokens`
├── usage.db (+ -wal, -shm) 0600, the host-measured usage ledger: `usage_days` (per key per UTC day, 400 days) and `usage_totals`
├── images/ default output of `gopala image build`: vmlinux-<version>, gopala-<base>.ext4, gopala-tools.ext4 (+ .sha256)
└── vms/
└── <conversation key>/ 0700, one per conversation that ever booted (e.g. lupyd-g-42)
├── disk.ext4 0600, the conversation's persistent root disk (copy of vm.rootfs, grown to its allocation; sparse)
├── disk.ext4.tmp only while the first copy is being built
├── allocation.json 0600, the agent's granted {vcpus, memory_mib, disk_gib} (absent until a grant)
├── vm.json 0600, firecracker --config-file (unjailed only)
├── console.log 0600, firecracker stdout+stderr (guest serial console), truncated per boot, capped
├── v.sock vsock uds_path (unjailed only)
└── v.sock_1024, _1025, _1090, _1100…, _1200… host listeners, 0600 (unjailed only)
<vm.jailer.chroot_base>/<firecracker binary name>/<key>/root/ (jailed only, recreated per boot, removed on stop)
├── vmlinux hard link to vm.kernel
├── disk.ext4 hard link to <data_dir>/vms/<key>/disk.ext4, chowned to the jailer uid/gid
├── vm.json 0600, chowned to the jailer uid/gid
├── tools.ext4 hard link to vm.tools_image (when set), read-only drive
└── v.sock, v.sock_<port> listeners, 0600, chowned to the jailer uid/gid
<vm.tools_image> the read-only tools image (built by gopala image build), attached to every VM as drive `tools`
<config file> replaced atomically by PUT /api/config and PUT …/budget (0600), previous version kept as <config file>.bak (0600)
<server.site_dir> a website directory that, when present, is served instead of the one embedded in the binary

rooms.db schema:

Table Columns Notes
events scope, seq, at_ms, body (encoded SessionEvent with seq set) Primary key (scope, seq), WITHOUT ROWID. Stored in plaintext in both encryption modes. When a scope’s bodies exceed rooms.journal_max_mib (checked every 1000 inserts) the oldest half is deleted (db::cap).
tokens id (12 bytes), scope, username, secret_hash (SHA-256), kwk (HKDF output), expires_at One live token per (scope, user); expired rows are deleted at startup, on issue, and on verify.

Inside each guest disk: /var/lib/gopala/provisioned (provisioning hash), /etc/sudoers.d/gopala, /etc/resolv.conf (NAT only), /etc/profile.d/gopala-tools.sh (with tools), /etc/profile.d/gopala-proxy.sh and /etc/apt/apt.conf.d/90gopala-proxy (proxy mode only), and the agent’s /home/agent with work/ (its persistent workspace, also omp’s project directory), .gopala/system.md, .gopala/omp-started or .gopala/claude-started, .gopala/mcp.json (the agent’s own MCP servers), .gopala/claude-mcp.json (Claude Code only), .gopala/memory.db (the memory store, with its -wal and -shm files; 0600 in a 0700 directory, owned by the agent), .omp/agent/mcp.json (also work/.mcp.json), and the harness’s own session files. With tools, /opt/gopala is the read-only mount of the tools drive (not part of the disk).

Browser sessions (dashboard sign-ins) live only in the web server’s memory: a restart signs everyone out.

The full key list with defaults is in gopala.example.toml, the doc comments of the configuration, and the JSON Schema served at /api/config/schema. Every key has a default and unknown keys are rejected (deny_unknown_fields). Keys not shown in the example: platforms.lupyd.wt_url, platforms.lupyd.auth0_realm (default Username-Password-Authentication), server.allow_plaintext, vm.extra_boot_args, vm.console_log_max_kib (default 4096), [[llm]].auth_prefix.

Section / key Used by
data_dir daemon::create_private_dir, Lupyd::connect (<data_dir>/lupyd), Rooms::open (<data_dir>/rooms), usage::Ledger::open (<data_dir>/usage.db), VmManager::new (<data_dir>/vms), VmControl (allocation.json), Capacity::disk_free
[platforms.lupyd] enabled PlatformCredentials::take (credentials) and main::run (connect the provider)
[platforms.lupyd] api_url, ws_url, wt_url, auth0_*, device_name gopala_lupyd::auth_config, firefly_mcp::agent::Endpoints, Agent::start
platforms.lupyd.profile_api_url Lupyd::profiles: one GET {profile_api_url}/user?users=a,b with the bot’s Lupyd access token for the bios list_agents shows; Config::validate (http(s)); restart required
[platforms.lupyd] email_env, password_env PlatformCredentials::take (take_env); named in the “not signed in” error
[platforms.slack] enabled, bot_token_env, app_token_env PlatformCredentials::take, Slack::connect (checks the xoxb-/xapp- prefixes, auth.test, apps.connections.open)
[platforms.discord] enabled, token_env PlatformCredentials::take, Discord::connect (checks the token and the application’s intent flags)
behavior.* Hot-reloadable (Control::apply); see Control plane and metering for when each reader sees a change
behavior.respond_to, behavior.conversations policy::respond_to → Router::handle (groups ignored for dms), policy::triggers; written by control::set_respond_to (/listen, PUT …/behavior); Control::state (respond_to)
behavior.direct_messages Router::admit_direct → Platform::admit_direct
behavior.invites Router::check_invite, Router::check_joined_groups
behavior.allowlist Behavior::allows (case-insensitive) for the three allowlist policies
behavior.max_messages_per_minute HostConversation::limiter (governor): each sent part and each approval question
behavior.max_message_chars policy::split_message in HostConversation::send
approvals.timeout_secs run_vm (approval_wait, cap for guest ApprovalRequest), ScopedMcp::new (max_approval_wait), VmControl::request
approvals.approvers Router::is_approver (answers, /stop and /listen)
[alerts] HostConversation::audience (warning, critical, conversations), take_alert (max_per_hour), host_event (host_events); hot-reloadable
[server] web::serve / serve_on (bind, TLS), web::app (site_dir), main::start (admin_token_env), Server::watch_url (public_url), Web::secure (Secure cookie flag with TLS or an https public_url), main::run (enabled); Config::validate (TLS pair, loopback/plaintext rule, http(s) public_url, env name). Restart required.
[monitoring] auth::rules and rule_matches via Control::grant (output, metrics, conversations), monitor::lupyd_caller (lupyd_tokens), auth::Membership::member (membership_cache_secs); web::watch_authorization for /watch; applies without a restart
[collaboration] Router::on_message (enabled: whether agents’ messages can wake the agent), HostConversation::take_agent_message (max_agent_messages_per_hour), HostConversation::agents → Platform::agents (enabled, agent_role), session::host_init → policy::system_prompt (the “Working with other agents” section, at boot); main::run → each provider’s set_agent_role (once, at startup). Hot-reloadable, except for the providers’ agent_role
platforms.lupyd.auth0_domain, auth0_audience also auth::Jwks::new (the key set URL) and LupydTokens::verify (iss, aud) for Lupyd sign-in tokens
[rooms] Rooms::open (encryption, key_rotation_secs), Rooms::serve_viewer (max_viewers_per_room), Rooms::publish (journal_max_mib), Rooms::issue_token (token_ttl_secs); main::run (enabled); Config::validate (needs server.enabled, rotation ≥ 60 s)
vm.firecracker, vm.kernel, vm.rootfs VmManager::prepare_with, ensure_disk, resolve_binary (jailed)
vm.vcpus, vm.memory_mib, vm.disk_gib the base Allocation of every VM (VmControl::new), the disk’s floor
vm.network, vm.dns, vm.extra_boot_args firecracker_config, boot_args; network = "nat" also setup_tap; "proxy" makes run_vm bind vsock 1090 and host_init set egress_proxy
vm.allowed_hosts egress::Allowlist (one Egress per VM, built at boot); restart required
vm.console_log_max_kib PreparedVm::boot → drain_console
vm.max_vms session::Slots (Permits, by budget priority)
vm.boot_timeout_secs, vm.idle_shutdown_secs session::run_vm (read at each boot; hot-reloadable)
vm.max_vcpus, vm.max_memory_mib, vm.max_disk_gib machine::decide, VmControl::new (clamping), VmControl::report, Control::state (hot-reloadable)
vm.memory_budget_mib machine::memory_budget_mib → Shared::memory (Permits, fixed at start)
vm.min_free_disk_gib, vm.grants machine::decide (hot-reloadable; VmControl::operator_set ignores grants)
[vm.jailer] jailer_args, setup_chroot, PreparedVm::bind_vsock (chown), setup_tap (tap owner)
vm.tools_image, vm.tools_sha256 gopala_vm::verify_tools (startup, check-config); VmManager::prepare_with → second, read-only drive tools (hard-linked into the jailer chroot as /tools.ext4) and gopala.tools=1 in boot_args; the guest’s tools::setup mounts it at /opt/gopala and harness::command resolves the harness there first
[agent] session::host_init (harness, packages, setup, user_setup, privileges, env, alert percents, usage interval, tools, mcp), policy::system_prompt (sudo / install_packages wording, system_prompt); executed by the guest’s provision, monitor, agent_config and harness
agent.harness HostInit.harness.kind → harness::supervise (omp, claude_code, exec); VmControl::restart_harness (refused for exec); image::runs (CLAUDE_CODE=1 for the tools image with claude_code)
[[agent.mcp]] session::host_init (HostInit.mcp, mcp_forward); url servers: egress::mcp_secrets (startup, check-config) → egress::mcp_upstreams, egress::live_mcp and run_vm binding 1200+i; command servers run in the guest
[[llm]] main::secrets → llm_proxy::Upstream::new (upstream, auth_header, auth_prefix, secret_env); session::host_init (guest_port → LlmForward.local_port, guest_env → HostInit.env); run_vm binds 1100+i
[[git]] the git proxies (name, provider, url, repos, protected_branches; token_env read at startup, checked by check-config); session::host_init → HostInit.git (GitForward); run_vm binds 1300+i; the guest’s git config and GOPALA_GIT_<NAME>_API. Not hot-reloaded: restart required
[budget] budget::check via budget::gate (each model call), budget::effective → Shared::priority (slots and memory), Control::state (budget fields), PUT …/budget (budget::changed, edit_toml); hot-reloadable
[image] gopala image build only (image::build); gopala run doesn’t read it
image.setup image::runs → one --setup <command> per entry for build-rootfs.sh, run with bash -c as root in the chroot; Config::validate (non-empty, no NUL)
[memory] session::memory_config → HostInit.memory ([[llm]] names resolved to their guest_port), policy::system_prompt (the memory line); the guest’s agent_config::base and memory::spawn_maintenance; restart required

Config::validate enforces: at least one enabled platform; an allowlist exists when any allowlist policy is selected; max_message_chars > 0; with server.enabled: TLS cert and key together, no plaintext on a non-loopback server.bind unless allow_plaintext, an http(s) public_url, a valid admin_token_env name; rooms.enabled needs server.enabled; key_rotation_secs ≥ 60; positive VM sizes (memory_mib ≥ 128); maximums not below the base sizes; memory_budget_mib 0 or at least memory_mib; alert percents within 10–99; usage_interval_secs ≥ 5; jailer uid/gid not 0; an absolute vm.tools_image and a 64-hex-digit vm.tools_sha256 (only with an image); no agent.packages with network = "none"; a non-empty, well-formed vm.allowed_hosts with network = "proxy"; agent.mcp names [a-z0-9_-]{1,32}, unique, not chat, each with a command or an http(s) url, secret_env only with a url; plain agent.tools, image.packages and image.tools names; positive image.rootfs_size_gib; budget shares within 1–100 with low ≤ normal; valid, unique budget.conversations, behavior.conversations and alerts.conversations keys; alerts entries of the form everyone, here, role:<name> or user:<name>; positive memory caps, cleanup_interval_secs ≥ 60, duplicate_threshold within 0.5–1, and memory.embeddings / memory.compaction naming an existing [[llm]] entry and a model; no agent.mcp server named memory while memory is on; valid environment variable names; unique guest_ports, none on 3128, 3200..3200+n or 3300..3300+n; http(s) upstreams; at most 64 [[llm]] entries; at most 8 [[git]] entries with unique names; collaboration.agent_role of 1–100 characters and max_agent_messages_per_hour at most 3600; non-empty image.setup commands without NUL bytes; an http(s) platforms.lupyd.profile_api_url.

Decision Chosen Alternatives considered Why
Platform seam One Platform trait with neutral events and views; providers own mention syntax, ids, encoding and sign-in; Incoming carries mentions_bot and addressed_text already computed Lupyd-specific core (the first version); a plugin process protocol Keeps policy, sessions, VMs, approvals, MCP and watching identical for every platform, and new platforms a single crate. An in-process trait is simplest to test (FakePlatform) and costs no serialization.
Lupyd access Reuse firefly-mcp’s Auth and Agent (and firefly_mcp::content for message encoding, fold for edits/reactions, the Firefly stores and search), pinned to the same git revision lupyd-mcp ships (f9a0960) Reimplement sign-in, the MLS device and message encoding MLS device management, key packages, message encoding and storage are subtle and already maintained in Firefly; the bot is then the same kind of device as a person’s. Cost: the host links mls-rs (hence --cfg mls_build_async) and is tied to Firefly’s revision. Only gopala-lupyd depends on it; the guest stays free of it.
MCP server gopala-mcp: firefly-mcp’s tool vocabulary (read, send, edit, delete, react, search, wait for events) re-exposed through a Conversation trait fixed to one conversation, plus request_approval and the VM tools Run firefly-mcp itself per VM firefly-mcp’s tools take group ids and usernames and need the account’s keys; giving that to a guest would expose every conversation and the credentials.
Host ⇄ guest transport vsock, guest-initiated only: the host listens on Firecracker’s {uds_path}_{port} sockets and never dials into a guest Host-initiated vsock (CONNECT <port> on the uds), TCP over the tap, virtio-serial Works with network = "none"; scope is implied by which VM’s socket a connection arrives on; the host never needs to trust anything a guest listens on; the guest owns reconnecting.
Model API credentials Host-side proxy per [[llm]]: the guest gets a placeholder key and http://127.0.0.1:<port>; the host strips credential headers, adds the real one and forwards over HTTPS Pass keys in the VM environment; one generic egress proxy for everything Keys never enter a VM (not in its env, disk, omp session files or the journal), the proxy reaches only the configured upstream, and it can meter and budget every call. Cost: request bodies are buffered (≤ 32 MiB) and the guest side is plain HTTP over vsock.
Restricted network An HTTP(S) proxy on the host per VM (network = "proxy"), reached over vsock, with an allowlist and an address check after host-side DNS Per-tap nftables rules managed by Gopala (issue #6); a DNS-filtering resolver No network device means no way around it, and no root or CAP_NET_ADMIN is needed. Programs that ignore proxy variables can’t get out at all, which is the safe failure. Cost: only HTTP(S) works.
Token budgets Checked in the LLM proxy before each call, from the host ledger, with priorities that also order boots Provider-side limits only; cut calls off mid-stream The host already sees every call and its usage, so the check is cheap and the agent can’t evade it. Refusing before a call (never during one) keeps responses whole; the cost is that one call may overshoot.
Harness integration omp RPC mode (--mode rpc) by default, or Claude Code in stream-json mode: one long-lived session per VM, both behind one Protocol trait in the guest (a gopala-owned harness is to replace omp, issue #5) One-shot CLI invocations per message (kept as the exec harness) Keeps context across messages and reboots (-c, --continue), supports follow-ups and steering while working, and streams structured events (tool calls, subagents) for viewers; omp also has an extension-UI sub-protocol the guest maps onto chat approvals. Claude Code runs with bypassPermissions because the VM is the sandbox and request_approval covers critical steps.
Agents extending themselves The agent edits its own mcp.json and calls restart_harness; the guest merges it under the host’s servers before every start The operator adds every server; let the agent write the harness’s config directly Agents can pick up the tools a task needs without an operator, while the names the host configured stay the host’s and the file can’t redirect root’s writes (no symlinks, owner checked). An agent can already run anything in its VM, so a server it adds gains it nothing new.
Agents working together Agents ping each other with ordinary @mentions in the shared group; list_agents shows who’s there; each decides for itself; the host caps wake-ups per hour A host-side task broker or delegation API between agents Works across hosts, products and platforms with nothing to deploy, and people see every hand-off in the chat. An agent never gains authority from another: pings can’t run commands or answer approvals. Cost: chat round trips, and the cap can cut a long exchange short.
Harness and tools provenance A host-built, read-only ext4 image (vm.tools_image, checked against vm.tools_sha256 at start) mounted ro,nosuid,nodev; the guest refuses a harness the agent could replace Install the harness and tools inside each VM (the default user_setup still does this without an image) The agent can’t swap the program that drives it or the tools it trusts, and every VM runs the same pinned versions. Integrity is checked at start only (verity is issue #7).
Usage metering Measure on the host: the LLM proxy parses the provider’s usage reports, the firecracker process, disk image and tap are sampled from /proc and /sys; a SQLite ledger per key and day Trust the guest’s own reports; meter in the harness The agent may lie or be compromised; the host sees every model call (it holds the keys) and the VM’s real footprint. The guest’s reports stay visible as an unverified second opinion.
Resources The agent measures, cleans up and asks; the host decides by config and capacity; grants apply at a restart Fixed sizes only; live resizing (balloon device, hotplug) Agents know best when they need more and can free space themselves; restarting is simple and works with Firecracker’s static machine config, and the workspace and omp session survive it. Disks only grow, so no data can be cut off.
Journal storage SQLite via sqlx (WAL), one events table keyed by (scope, seq) plus tokens A custom append-only log file per scope sqlx/SQLite is already in the dependency graph through firefly-client; gives atomic token issue, paged replay, and simple size capping.
One web server axum (on hyper HTTP/1, optional rustls) serves the site, the control and config API and /watch; rooms only speaks the protocol on websockets it is handed A separate rooms listener (the first version), a separate admin port One port, one TLS setup and one public_url; the config API reuses the daemon’s exact Config::parse.
Control panel Static Astro pages with Alpine.js on the same origin; sign-in with the admin token exchanges it for an HttpOnly, SameSite=Strict session cookie; live updates over server-sent events A single-page app with its own state; tokens in browser storage No build-time coupling to server state; the token never sits in page memory or storage after sign-in; SSE is one long GET that cookies already authenticate.
Configuration changes PUT /api/config writes the file atomically and hot-applies the sections that are safe to swap at runtime (ArcSwap); the rest waits for a restart Restart for every change; reload everything Policy and VM limits change often and are read per decision; server, platform and rooms settings own sockets and devices that can’t be swapped safely.
Tap networking Shell out to iproute2 (ip tuntap add, ip addr add, ip link set, ip link del) netlink crates (e.g. rtnetlink) A standard tool with clear error messages and no extra dependency; the cost is a process spawn per VM start/stop, and iproute2 must be installed.
Watch encryption Configurable: sealed (default) or none (TLS only) TLS only Sealing adds a layer that survives TLS-terminating proxies and a plaintext transport segment, and lets revocation take effect cryptographically through key rotation. It is not end-to-end against the host, which generates the keys and stores the journal in plaintext.
Watch token delivery Personal token per (conversation, user), sent in a direct chat on the same platform (end-to-end encrypted on Lupyd), in the link’s #fragment, and presented inside the websocket hello Tokens in a URL query string or cookie Fragments are not sent in HTTP requests, so tokens don’t land in proxy or server access logs.
Wire format quick-protobuf with pb-rs-generated owned types, length-delimited over byte streams, one message per websocket binary frame JSON, prost Requested in the brief for performance; matches Firefly’s own protobuf tooling.
Notifying people The agent gives a severity; [alerts] maps it to an audience; providers render it Let the agent write mentions itself The agent can’t page people it shouldn’t (prompt injection can’t add @everyone), works the same on every platform, and the operator can cap it per hour.
Agent memory SQLite in each VM, written by the agent through MCP tools, cleaned up by score; a model only for optional compaction A host-side memory service; mem0 with an LLM call on every write Memory stays with the conversation’s disk and never mixes conversations; no extra service or per-write model cost. Cost: the guest agent links C (SQLite).

Future: Firefly/MLS for rooms. The brief anticipates replacing the rooms’ rotating-key scheme with Firefly itself (real end-to-end encryption with member key exchange) once it is less tangled with Lupyd. The brief’s “key exchange from members” is not implemented: room keys are generated by the host and wrapped per viewer.

  • No real VM boot, and no live conversation on Lupyd, Slack or Discord, has been tested end to end (issue #1, issue #2); tests use fakes and recorded payloads.
  • Over-long lines are dropped, not truncated: guest log lines over 2 KiB become (line over 2048 bytes dropped), and omp stdout frames over 1 MiB + 1 KiB are discarded with a warning (issue #19).
  • An in-place disk grow (grow_disk: set_len, then e2fsck, resize2fs) interrupted by a crash isn’t retried: the next ensure_disk sees the file at the target size and skips the filesystem resize (issue #18).
  • Grants apply only at a restart; there is no live memory resizing (issue #9). The memory budget counts configured VM memory (mem_size_mib), not Firecracker’s own overhead or what guests actually touch (issue #13). Disk images are sparse, so min_free_disk_gib is checked when disk growth is granted, not while guests write (issue #14).
  • The LLM proxy buffers each request body in memory (up to 32 MiB) before forwarding; responses stream (issue #15).
  • Metering sees only what passes through the host: with network = "nat" a guest can reach other services directly, visible only as tap bytes, until per-tap egress filtering lands (issue #6). A provider response without a usage report (for example an encoded body, or an OpenAI chat stream requested without usage) counts as a request with zero tokens [inference for the latter], so budgets don’t see it either. Budgets are checked before each call, so calls in flight can take a conversation past its limit.
  • network = "proxy" carries HTTP and HTTPS only; programs that ignore the proxy variables (or speak other protocols, like ssh) can’t reach the network. An allowed name stays allowed if its owner moves it to another public address.
  • Discord runs one gateway shard (up to 2500 servers). Slack search scans only the newest 1000 messages of a channel, and Discord search 200 messages in each of 25 channels, because bots can’t use the platforms’ search APIs [inference for Discord].
  • The chat EventLog behind wait_for_events is in memory (last 1000 events per conversation) and starts over when the host restarts; room keys live in memory too, so after a restart every room starts again at epoch 1 with a new key (issue #20). Dashboard sessions are in memory as well.
  • Deliveries handed to a guest that is just exiting (idle shutdown, sleep or restart racing a new message) can be lost (issue #25).
  • /status answers starting up while a VM boots, because the session doesn’t read commands during the handshake (issue #12); the dashboard shows booting.
  • Hot reload doesn’t change existing conversations’ chat rate limit or base allocation [inference from the code]; [server], [platforms], [rooms], [[llm]], [memory], the memory budget, vm.network, vm.allowed_hosts and the other VM settings need a restart, and so does a new HTTP MCP server with a secret_env. A changed collaboration.agent_role reaches list_agents at once but the providers’ marking of incoming messages only after a restart.
  • Collaboration rests on each platform’s idea of an agent: Discord gives no bio (about is always empty) and, without the Server Members intent, lists only members it has seen; Slack shows a profile title, which bot users may not have [inference]; Lupyd’s bios come from a profile request that hasn’t been run against the real Lupyd API yet. Slack has no reply-to field, so “replies to the bot” means a reply in a thread the bot’s message started. The hourly cap counts wake-ups per conversation, so a busy group with several agents can use it up and drop legitimate pings.
  • Claude Code is untested against a real process: the priority field for mid-turn messages, --continue after SIGTERM and the timing of system/init follow its documentation. Whether claude -p needs a placeholder ANTHROPIC_API_KEY next to ANTHROPIC_BASE_URL hasn’t been checked.
  • Alerts are only as good as each platform’s mentions: Lupyd has no role mentions and no native @everyone/@here (they’re plain text there); Slack workspace admins can restrict who may use @channel/@here, and user groups need the usergroups:read scope (apps installed before it was added must be reinstalled); Discord needs the “Mention @everyone, @here and All Roles” permission and finds members best with the Server Members intent. Names that don’t resolve are only logged.
  • Monitoring access follows the platforms’ answers, cached for up to membership_cache_secs: a person removed without a membership event keeps access until the answer expires. Providers differ: Lupyd answers from the bot’s copy of the group; Slack’s member list is cached 30 s and user groups 5 min (roles need usergroups:read); Discord knows member changes at once only with the Server Members intent, otherwise it searches members over REST when the cache expires. The monitor protocol has no client library yet; apps build requests from the protobuf schema in protocols.md.
  • Memory lives on the conversation’s disk: deleting vms/<key>/disk.ext4 deletes it, and nothing backs it up separately. The agent owns the file and can change it with its shell, so it is a notebook, not an audit log. Cosine recall is brute force over stored vectors [inference: fine for the default 5000 memories, slow at very large caps]. Memory posts /v1/… paths to the [[llm]] forwarder, so an upstream that already ends in /v1 would double it [inference].
  • The tools image is checked against its sha256 only when Gopala starts; there is no dm-verity or fs-verity on the running drive (issue #7).
  • On shutdown (SIGINT/SIGTERM) the daemon asks sessions to close and waits 2 seconds before shutting the providers down; VMs that haven’t stopped by then are expected to die with the process, since the firecracker child is spawned with kill_on_drop [inference: not tested] (issue #21).