Operations
How to install, configure, run and troubleshoot Gopala: AI agents that work around the clock on your own servers and take work from Lupyd, Slack or Discord.
For the hardening checklist see security.md.
Gopala has not yet been run against a real Firecracker host or a live Lupyd, Slack or Discord account. The steps below follow from the code and the upstream tools’ documentation; the systemd units are examples, not tested artifacts.
Host prerequisites
Section titled “Host prerequisites”| Requirement | Why | Notes |
|---|---|---|
Linux x86_64 with KVM (/dev/kvm readable and writable by the service user) |
Firecracker | See Firecracker’s getting started for checking KVM access |
firecracker |
The VMM (vm.firecracker, a bare name is looked up in PATH) |
Release binaries: Firecracker releases |
jailer (production) |
[vm.jailer] |
Requires starting Gopala as root; ships with Firecracker releases |
e2fsprogs (e2fsck, resize2fs) |
Growing each conversation’s disk to its allocation (grow_disk) |
Also mkfs.ext4 and debugfs for building images |
GNU coreutils cp |
cp --reflink=auto --sparse=always copies the base rootfs (ensure_disk) |
Reflinks make copies cheap on btrfs/XFS |
iproute2 (ip) and CAP_NET_ADMIN |
Tap devices for vm.network = "nat" (setup_tap) |
Not needed with network = "proxy" or "none" |
| IP forwarding and a NAT rule | Guest internet access with nat |
One-time setup below |
| A bot on each enabled platform | Its identity | See Creating the bot |
| Model provider API keys | [[llm]] |
In the host environment only |
curl, unsquashfs, fakeroot (unless root), jq, ldd, sha256sum |
gopala image build |
debootstrap too for a Debian base; root (sudo) for Debian, packages or setup |
Install
Section titled “Install”Gopala ships as one executable, gopala. Get a demo, and the binary, at
contact@lupyd.com. The file embeds everything else Gopala needs: the
image build scripts, the static gopala-guest agent that runs inside each VM (with its systemd
unit), and the website. Only the host tools above are installed separately.
install -m 0755 gopala /usr/local/bin/gopala --config /etc/gopala/gopala.toml image build # kernel, rootfs and tools image from [image]The website (front page, Create an Agent, the dashboard, /configure/ and these docs) is served
from the binary. If server.site_dir (default /usr/share/gopala/site) exists as a directory,
that directory is served instead.
Building images (gopala image build)
Section titled “Building images (gopala image build)”gopala image build [--out DIR] builds the rootfs and the read-only tools image from the
[image] settings (base, packages, setup, rootfs_size_gib, tools) into --out
(default <data_dir>/images) and prints the [vm] lines to paste: kernel, rootfs,
tools_image and tools_sha256. With agent.harness = "claude_code" it also bundles Claude
Code into the tools image.
[image]base = "ubuntu-24.04" # or "debian-12"packages = ["python3", "build-essential"] # baked in: no network needed at bootsetup = [] # root shell commands run in the image after packagesrootfs_size_gib = 4tools = ["git", "rg", "jq"] # host programs on the read-only tools drive, next to ompThe rootfs build:
- downloads the newest Firecracker CI kernel (
vmlinux-<version>), and for Ubuntu the CI squashfs, from Firecracker’s public CI bucket; Debian is built withdebootstrap; - installs
packageswith apt in a chroot, if any, then runs eachsetupcommand there withbash -cas root, in order, with a cleanPATHand yourhttp(s)_proxy/no_proxyif set; a failing command stops the build (setup command failed: …); - builds an ext4 image of
rootfs_size_giband injects the embedded/usr/local/bin/gopala-guestandgopala-guest.service(enabled formulti-user.target), then runse2fsck.
Ubuntu without packages or setup commands needs no root and no mounts. Debian, packages or setup
commands need root: run it with sudo.
Two options are for special cases, when support asks you to use them: --guest FILE (or
GOPALA_GUEST) bakes a different static gopala-guest into the rootfs, and --scripts DIR (or
GOPALA_SCRIPTS) runs the image scripts from a directory instead of the embedded copies.
Upgrading the guest. Each conversation’s disk is a copy of
vm.rootfsmade on its first boot and kept afterwards (ensure_disk), so a rebuilt image only reaches new conversations. Existing conversations keep their oldgopala-guestuntil their disk is replaced (deleting<data_dir>/vms/<key>/disk.ext4while the VM is stopped also deletes the workspace) or the binary is written into it (e.g. withdebugfs -w, as the image build does). CheckCONTROL_PROTOCOLcompatibility before upgrading the host; the resource monitor needs a guest that knowsHostInitfields 13–15 (older guests simply don’t report usage), and the memory server one that knows field 20 (an older guest skips it [inference], so its agent has nomemorytools even though the system prompt mentions them).restart_harness, the agent’s own MCP file and the Claude Code harness need a guest that knowsHostFramevariant 6 andHARNESS_CLAUDE_CODE; what an older guest does with them is untested.
Host-provided tools (vm.tools_image)
Section titled “Host-provided tools (vm.tools_image)”gopala image build also writes gopala-tools.ext4 (no root, no mounts): omp at a pinned,
sha256-verified release, and the build host’s programs named in [image] tools (default
git; [] for omp alone), each with every shared library and its dynamic loader under lib/.
Dynamic programs become wrappers in bin/ that run the real binaries through the bundled loader;
git’s helpers in libexec/git-core/ get the same treatment and
GIT_EXEC_PATH=/opt/gopala/libexec/git-core, so they work whatever the guest’s libc (the build
host needs glibc ≥ 2.33 for ld.so --argv0). MANIFEST lists the sha256 of every file.
With agent.harness = "claude_code" it also bundles Claude Code as bin/claude: a pinned release
from downloads.claude.ai, checked against the release manifest and the pinned sha256. The binary
is about 340 MB; the image grows to fit it.
Gopala checks the image (regular file, readable by the jailer uid, matching sha256) at startup
(gopala_vm::verify_tools) and attaches it to every VM as a second drive with
is_read_only: true; the guest mounts it read-only, nosuid,nodev at /opt/gopala, puts
/opt/gopala/bin first on the harness’s PATH and in /etc/profile.d/gopala-tools.sh, resolves a
bare agent.command from /opt/gopala/bin first, and refuses (status failed) to start a harness
that resolves to a file the agent user could change. With the image in place, drop omp’s
installer from agent.user_setup and git from agent.packages. A new image reaches each VM on
its next boot; when jailed, vm.jailer.chroot_base must be on the image’s filesystem (hard link).
Install it so the service user (and the jailer uid) can read it but nobody else can change it, and paste the printed lines:
gopala --config /etc/gopala/gopala.toml image build --out /tmp/toolsinstall -o root -g root -m 0444 /tmp/tools/gopala-tools.ext4 /var/lib/gopala/images/gopala-tools.ext4[vm]tools_image = "/var/lib/gopala/images/gopala-tools.ext4"tools_sha256 = "<the 64 hex digits image build printed>"
[agent]packages = ["ca-certificates"] # git comes from the imageuser_setup = [] # omp comes from the imageRun gopala check-config afterwards: it fails on a missing, unreadable or mismatching image. A
rebuilt image needs a new tools_sha256 and a restart of gopala run. omp is always bundled; a
gopala-owned harness is to replace it (issue #5).
Baking tools into the rootfs (image.setup)
Section titled “Baking tools into the rootfs (image.setup)”[image] packages covers apt. Anything else goes into image.setup: shell commands run as root
in the rootfs while gopala image build makes it (so the build needs sudo), after the
packages, one after another. What they install is in every VM from its first boot, needs no
network at boot, and sits outside the agent’s reach unless it has sudo. Each entry is passed to
bash -c as it is; the build host’s http(s)_proxy and no_proxy are passed through if set.
Like packages, a new setup reaches new conversations only (see Upgrading the guest above).
None of these recipes has been run in a real image build yet.
uv, for Python tools and projects (the system prompt tells agents to prefer it):
[image]packages = ["curl", "ca-certificates"]setup = ["curl -LsSf https://astral.sh/uv/install.sh | env UV_UNMANAGED_INSTALL=/usr/local/bin sh"]Node.js with pnpm, for Node projects. Node 25 and later no longer ship Corepack, so pnpm
comes from its own installer; the distro’s Node is usually too old. Programs are linked into
/usr/local/bin because the harness’s PATH is fixed, and pnpm add -g is pointed at the agent’s
~/.local/bin:
[image]packages = ["curl", "ca-certificates"]setup = [ "curl -fsSL https://nodejs.org/dist/v26.13.1/node-v26.13.1-linux-x64.tar.gz | tar -xz -C /usr/local --strip-components=1", "curl -fsSL https://get.pnpm.io/install.sh | env PNPM_HOME=/usr/local/share/pnpm PNPM_VERSION=12 PNPM_NO_MODIFY_PATH=1 sh - && ln -sf /usr/local/share/pnpm/bin/* /usr/local/bin/",]
[agent.env]PNPM_CONFIG_GLOBAL_BIN_DIR = "/home/agent/.local/bin"The templates page writes the same lines when you tick Node.js, with the version it pins.
An MCP server installed at build time and wired in as an operator server. With uv from the
first recipe, mcp-server-fetch lands in /usr/local/bin, so the VM never downloads it (unlike
uvx mcp-server-fetch); the server itself still needs a network mode that reaches the pages it
fetches:
[image]packages = ["curl", "ca-certificates"]setup = [ "curl -LsSf https://astral.sh/uv/install.sh | env UV_UNMANAGED_INSTALL=/usr/local/bin sh", "UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin UV_PYTHON_INSTALL_DIR=/opt/uv-python uv tool install mcp-server-fetch",]
[[agent.mcp]]name = "fetch"command = "mcp-server-fetch" # from the image; runs in the VM as the agent userThe UV_* directories keep the tool and any Python uv downloads for it out of /root, which the
agent can’t read.
An agent can add servers of its own at run time too (see architecture.md); baking a server in is for the ones every conversation should have, or for VMs without network.
Networking
Section titled “Networking”vm.network decides how a VM reaches the internet. Model calls, the chat tools and HTTP MCP
servers go over vsock in every mode. Details in
architecture.md; the egress proxy’s threat model in
security.md.
| Mode | Use it when | Host setup |
|---|---|---|
"proxy" |
Agents need package mirrors, git hosts or a few APIs, and nothing else | None; list the hosts in vm.allowed_hosts |
"none" |
Agents only talk and think; everything they need is in the image | None |
"nat" (default) |
Agents need arbitrary internet access | IP forwarding, masquerading and filtering, below |
network = "proxy"
Section titled “network = "proxy"”[vm]network = "proxy"allowed_hosts = [ "archive.ubuntu.com", "security.ubuntu.com", # apt (Ubuntu image) "pypi.org", "files.pythonhosted.org", # pip "github.com", "*.githubusercontent.com", # git clone, raw files "omp.sh", # the default user_setup installs omp from here]An entry is host, *.domain (any subdomain, not the domain itself), or either with :port;
without a port 80 and 443 are allowed. The VM has no network device; HTTP_PROXY/HTTPS_PROXY
(and lowercase) point at the host for the harness, provisioning, login shells and apt, so programs
that honor them work and others fail. Each allowed or refused destination appears in the journal
as a network notice (rate-limited), and the bytes are counted as egress on the dashboard. A
name that resolves to any private, loopback or link-local address is refused even when listed.
Changing network or allowed_hosts needs a restart.
network = "none"
Section titled “network = "none"”The VM has no network device at all. agent.packages must be empty (bake packages in with
[image] packages), and the default user_setup can’t download omp, so use the tools image.
network = "nat"
Section titled “network = "nat"”For each running VM Gopala creates tap gpl<slot> whose host address is the first usable address
of the slot’s /30 (172.30.0.0 + 4·slot + 1; the guest gets + 2) (setup_tap, nat_addrs), and boots
the guest with a static ip= kernel parameter and gopala.dns= from vm.dns. Allow forwarding
and masquerade the range out of your uplink once:
sysctl -w net.ipv4.ip_forward=1nft add table ip gopalanft add chain ip gopala post '{ type nat hook postrouting priority 100; }'nft add rule ip gopala post ip saddr 172.30.0.0/16 oifname "eth0" masqueradeGopala filters nothing in this mode. Recommended rules (adjust interface names):
nft add chain ip gopala fwd '{ type filter hook forward priority 0; }'nft add rule ip gopala fwd ip saddr 172.30.0.0/16 ip daddr 172.30.0.0/16 drop # guest ↔ guestnft add rule ip gopala fwd ip saddr 172.30.0.0/16 ip daddr 169.254.169.254 drop # metadatanft add chain ip gopala in '{ type filter hook input priority 0; }'nft add rule ip gopala in iifname "gpl*" ct state established,related acceptnft add rule ip gopala in iifname "gpl*" drop # guest → host servicesSee Firecracker’s network setup guide for background.
Code hosting ([[git]])
Section titled “Code hosting ([[git]])”A [[git]] entry lets agents clone, push and open pull or merge requests on GitHub, GitLab or
Gitea/Forgejo without the token ever entering the VM. The host’s git proxy holds the token and adds
it to each request; the VM reaches it over vsock, so it works in every vm.network mode,
including none.
Creating the token
Section titled “Creating the token”| Provider | Token | Scopes |
|---|---|---|
| GitHub (github.com or Enterprise Server) | Settings → Developer settings → Fine-grained personal access token, limited to the repositories the agent works on | Contents: read and write; Pull requests: read and write |
| GitLab (gitlab.com or self-managed) | User or project/group access token | api, or read_api + write_repository |
| Gitea / Forgejo | Settings → Applications → Generate token | Repository: read and write |
A bot account with access to just those repositories keeps the blast radius small: other repository-level writes (settings, default branch, deleting the repository) are limited only by the token’s scopes.
Configuring it
Section titled “Configuring it”[[git]]name = "github" # unique, [a-z0-9-], up to 32 charactersprovider = "github" # github | gitlab | gitea# url = "https://github.example.com" # empty = github.com / gitlab.com; Gitea must set ittoken_env = "GITHUB_TOKEN" # host environment variable holding the tokenrepos = ["acme/api", "acme/web"] # allowlist; [] = anything the token reachesprotected_branches = ["main", "master", "release"] # default ["main", "master"]
[[git]]name = "lab"provider = "gitlab"token_env = "GITLAB_TOKEN"repos = ["acme/infra"]urlis https (http only for loopback) and takes no credentials, query or fragment.token_envis required. The token is read at startup, removed from the environment and zeroized;check-configfails if the variable is not set. Put it in/etc/gopala/env.reposentries are provider paths (owner/repo, GitLabgroup/subgroup/project).protected_branches: up to 32 exact branch names, norefs/prefix, no patterns, matched case-sensitively (protectMainseparately if it exists).- At most 8 entries.
[[git]]is not hot-reloaded: restartgopala runafter changing it.
What the agent can and can’t do
Section titled “What the agent can and can’t do”It can clone, fetch, push branches and tags, delete branches that aren’t protected, and use the provider’s REST API: open pull/merge requests, comment, read issues and so on.
It can’t:
- create, update, force-update or delete a protected branch, by push or by API; a push that touches one is refused whole;
- commit through the API without naming a branch (file edits, cherry-picks and reverts on the default branch are refused);
- merge a pull or merge request (always refused; a person merges);
- use GraphQL, the
ghorglabCLIs, or Git LFS (404); - send signed pushes or compressed push bodies;
- with
reposset, reach other repositories, name a repository by numeric id, or make API writes that name no repository.
How it works in the VM
Section titled “How it works in the VM”At boot the guest agent writes /etc/gopala-git.conf and includes it from /etc/gitconfig. It
rewrites https://<host>/, git@<host>: and ssh://git@<host>/ to
http://127.0.0.1:<3300 + index>/, so ordinary clone URLs work. Only the first entry for a host
gets the rewrite; a second account on the same host is reached through its own
http://127.0.0.1:<port>/ URL.
The API base is in GOPALA_GIT_<NAME>_API (in the harness environment and
/etc/profile.d/gopala-git.sh), e.g. GOPALA_GIT_GITHUB_API=http://127.0.0.1:3300/api
(GitLab …/api/v4, Gitea …/api/v1). Agents call it with curl and no token:
curl -s -X POST "$GOPALA_GIT_GITHUB_API/repos/acme/api/pulls" \ -d '{"title":"Fix login","head":"fix-login","base":"main"}'Only git’s smart HTTP transport is forwarded. Every push, branch deletion, tag, rejection,
refusal and new pull/merge request shows up in the journal under the title git (for example
pushed branch fix-login to acme/api, opened pull request #7 on acme/api: <url>,
refused push to refs/heads/main on acme/api (protected branch)). Usage totals count git
requests and bytes up and down.
Creating the bot
Section titled “Creating the bot”Enable one or more platforms under [platforms]; each needs its own bot account. Credentials go
in the environment (see Run as a systemd service), named by the
*_env keys, never in the file. Gopala reads them once at startup and removes them from its
environment.
Sign up once in the Lupyd app with the bot’s email and pick its username; that is the name people mention. Then:
[platforms.lupyd]enabled = trueemail_env = "LUPYD_EMAIL" # names of the variables, never the valuespassword_env = "LUPYD_PASSWORD"device_name = "gopala" # how the account's device list shows the bot# api_url / ws_url / auth0_*: production by default; point them at a local Firefly stack for testing# profile_api_url = "https://api.lupyd.com" # where list_agents reads bios, with the bot's tokenThe first run signs in with the password and keeps a refresh token under <data_dir>/lupyd;
later runs need neither variable.
Gopala uses Socket Mode, so the host needs no public URL.
- At api.slack.com/apps, Create New App → From a manifest,
pick the workspace and paste Gopala’s manifest (Create an Agent → Slack → Copy the
manifest or Download it, on the Gopala site). It turns on Socket Mode, the
Messages tab for DMs, the bot scopes (
channels:*,groups:*,im:*,mpim:*history and read/write,chat:write,reactions:read/write,usergroups:read,users:read) and the message, member, rename and reaction events Gopala handles. Change the name and display name if you like. - Basic Information → App-Level Tokens → Generate, with the scope
connections:write. Thatxapp-…token goes inSLACK_APP_TOKEN. - Install to Workspace, then copy OAuth & Permissions → Bot User OAuth Token (
xoxb-…) intoSLACK_BOT_TOKEN. - Invite the bot to channels (
/invite @gopala); people can DM it from its Messages tab.
[platforms.slack]enabled = truebot_token_env = "SLACK_BOT_TOKEN" # xoxb-…app_token_env = "SLACK_APP_TOKEN" # xapp-…, with connections:writeHow Slack maps onto Gopala: every channel the bot is in (public, private or group DM) is one
conversation with one VM; a DM is a conversation per person. Slack has no sub-channels, so
channel_id is never used. Replies to a threaded message go into that thread, with every part
of a long answer. Usernames for allowlists are Slack’s lowercase name handles. Gopala checks
both tokens at startup and every ten minutes; a revoked token or an uninstalled app stops it with
an error.
For alerts: role:<handle> resolves a Slack user group by its handle, which needs the
usergroups:read scope. An app created from an older manifest lacks it; add the scope (or paste
the new manifest) and reinstall the app, or roles are skipped with a warning in the log.
Workspace admins can restrict who may use @channel and @here; a restricted bot’s alerts
won’t reach everyone.
Discord
Section titled “Discord”- In the Developer Portal, New Application,
then Bot: reset and copy the token into
DISCORD_BOT_TOKEN. - Under Bot → Privileged Gateway Intents, turn on Message Content Intent (required).
Turn on Server Members Intent too if you want member lists in
conversation_infoand join/leave events; without it Gopala knows only the bot and people it has seen. Gopala detects which intents are on. - Start Gopala: the log prints the invite link, which is
https://discord.com/oauth2/authorize?client_id=<application id>&scope=bot&permissions=274878106688(View Channels, Send Messages, Send Messages in Threads, Read Message History, Add Reactions, and Mention @everyone, @here and All Roles so[alerts]can reach people). Open it to add the bot to a server.
[platforms.discord]enabled = truetoken_env = "DISCORD_BOT_TOKEN"How Discord maps onto Gopala: a server is one conversation with one VM, and its text channels and
threads are the conversation’s channels; a DM is a conversation per person. Long answers are
split at Discord’s 2000 characters. Ordinary messages can mention people but never @everyone,
@here or roles; only alerts can, and only with the “Mention @everyone, @here and All
Roles” permission. A bot invited with the older link (permissions 274877975616) lacks it:
re-invite it with the new link, or grant it through one of the bot’s roles. user:<name> alerts
find people in the cache or by a member search, which works best with the Server Members intent.
One gateway shard serves up to 2500 servers.
Configure
Section titled “Configure”Configure in a browser, or write the file by hand:
gopala --config /etc/gopala/gopala.toml check-config# Browser: site + config API only, no sign-in, no VMs (a missing/invalid file still serves).GOPALA_ADMIN_TOKEN="$(openssl rand -base64 32)" gopala --config /etc/gopala/gopala.toml web# open server.public_url (default http://127.0.0.1:60901) → Configurecheck-config parses and validates the file, requires every [[llm]].secret_env and every
agent.mcp secret_env to be set, and
checks that vm.kernel and vm.rootfs exist and that vm.tools_image verifies. The config path
can also come from GOPALA_CONFIG (default gopala.toml). Every key and its default is in the
configuration editor at /configure/ and the JSON schema at /api/config/schema (signed in);
the map from keys to components is in
architecture.md.
The configuration editor at /configure/ builds its form from that schema, validates and
normalizes drafts through the server’s own parser (/api/config/validate,
/api/config/normalize), and loads and saves the file (GET/PUT /api/config; the previous file
is kept as <file>.bak). All of these need you signed in. Under gopala run, saving applies
behavior, approvals, agent, budget, alerts, monitoring, collaboration and the VM
limits immediately and tells you which other sections need a restart; under gopala web
everything takes effect at the next gopala run. Signed out, the editor and /templates/ fall
back to the schema and defaults bundled with the site: they still build and download a file, but
can’t check it against your server. See
protocols.md for the API and
security.md before enabling the token.
The “Create your bot” flow at /templates/ starts a whole file from a goal (help a team, research,
write and review code, analyse data, write docs), then asks for the chat app, the model provider
and, optionally, languages, code hosting accounts ([[git]]), network, VM size and daily budget;
every other key it covers (harness and tools, MCP servers, the image, limits, host budgets) sits
under “Expert settings”. The code goal uses Claude Code, full internet and a GitHub account by
default; with a provider other than Anthropic it switches to omp. Its “OpenAI-compatible” provider
writes an [[llm]] entry named compatible on guest port 8703 whose upstream is the base URL
you type, and sets OPENAI_BASE_URL = "http://127.0.0.1:8703" without /v1: the host proxy keeps
the upstream’s own path, so a base URL that already ends in /v1 stays correct. Choosing “No
internet” moves agent.packages into image.packages, since nothing installs on first boot
without a network.
Recipes
Section titled “Recipes”Lupyd, Slack and Discord: see Creating the bot.
DM-only assistant for a few people
[behavior]respond_to = "dms" # groups are joined but stay quiet (no commands either)direct_messages = "allowlist"invites = "nobody" # leave every group right awayallowlist = ["alice", "bob"]Allowlisted team bot
[behavior]respond_to = "mentions" # @bot or replies to the bot in groups, all DMsdirect_messages = "allowlist"invites = "allowlist" # stays only in groups owned by these peopleallowlist = ["alice", "bob", "carol"]max_messages_per_minute = 10
[approvals]approvers = "allowlist" # only these people may approve, /stop, or grant resourcestimeout_secs = 600Offline agent (no internet in the VM; models still reachable through the host proxy)
[vm]network = "none"tools_image = "/var/lib/gopala/images/gopala-tools.ext4" # omp and git from the hosttools_sha256 = "…"
[agent]packages = [] # required with network = "none" (validation)user_setup = [] # the default installs omp from the internetinstall_packages = false# Anything else the agent needs goes into the image: [image] packages and tools.Agent limited to a few tools and extra MCP servers
[agent]tools = ["read", "grep", "edit", "write", "bash"] # omp's built-ins; [] keeps its defaults
[[agent.mcp]] # runs in the VM as the agent user: no secrets in args or envname = "fetch"command = "uvx"args = ["mcp-server-fetch"]
[[agent.mcp]] # proxied by the host; the VM never sees the tokenname = "tracker"url = "https://mcp.example.com/mcp"secret_env = "TRACKER_TOKEN" # sent as "authorization: Bearer <token>"# auth_header = "x-api-key" # for servers that want another header# auth_prefix = ""An HTTP server works in every network mode. A stdio server that downloads itself (uvx, npx)
needs nat, or proxy with its registry allowed. A new HTTP server with a secret_env added
while running waits for a restart (its secret is read at startup); without one it works at the
VM’s next boot.
Token budgets
[budget]daily_tokens = 20000000 # the whole host, per UTC daymonthly_tokens = 400000000low_stops_at_percent = 70 # low-priority conversations stop at 70% of the host budgetnormal_stops_at_percent = 90 # normal ones at 90%; high ones may use all of it
[budget.default]daily_tokens = 2000000 # each conversationpriority = "normal"
[[budget.conversations]]key = "slack-g-1f0c3a9e2b7d4c58" # keys are on the dashboarddaily_tokens = 10000000priority = "high" # also boots first when slots or memory run shortTokens are input + output + cache-write, counted by the host’s LLM proxy. A conversation over
budget gets 429 budget_exceeded for every model call until its day or month rolls over (00:00
UTC), with one budget notice in its journal. The dashboard shows usage against each budget and
edits one conversation’s (PUT /api/conversations/{key}/budget), keeping your file’s comments.
Several model providers, including an OpenAI-compatible one
[[llm]]name = "anthropic"upstream = "https://api.anthropic.com"guest_port = 8701auth_header = "x-api-key"secret_env = "ANTHROPIC_API_KEY"guest_env = { ANTHROPIC_BASE_URL = "http://127.0.0.1:8701", ANTHROPIC_API_KEY = "via-gopala" }
[[llm]]name = "openai"upstream = "https://api.openai.com" # or any OpenAI-compatible serverguest_port = 8702# auth_header defaults to "authorization"; OpenAI-style APIs want a Bearer prefix:auth_prefix = "Bearer "secret_env = "OPENAI_API_KEY"guest_env = { OPENAI_BASE_URL = "http://127.0.0.1:8702/v1", OPENAI_API_KEY = "via-gopala" }Each entry gets vsock port 1100 + index; guest_ports must be unique; at most 64 entries. The
placeholder keys only satisfy client libraries: the proxy removes them and adds the real one. How
the harness picks a provider and model is up to omp (see omp environment variables; its
*_BASE_URL variables are fallbacks that a configured provider URL overrides), e.g.
[agent.env] PI_SMOL_MODEL = "anthropic/claude-haiku-4-5".
Serving behind a reverse proxy (site, config API and /watch on one origin)
[server]bind = "127.0.0.1:60901"public_url = "https://agents.example.com" # watch links become wss://agents.example.com/watch#…site_dir = "/usr/share/gopala/site"Any websocket-capable TLS proxy works, e.g. Caddy: agents.example.com { reverse_proxy 127.0.0.1:60901 }. To terminate TLS in Gopala instead, set server.tls_cert/tls_key (PEM) and a
public bind. A non-loopback bind without TLS is refused unless allow_plaintext = true (only
for a TLS proxy on a private network). [rooms] now holds only the watch settings (enabled,
encryption, key_rotation_secs, max_viewers_per_room, journal_max_mib, token_ttl_secs),
and rooms.enabled requires server.enabled.
Sizing VMs and the host
Section titled “Sizing VMs and the host”Every conversation starts with vm.vcpus / vm.memory_mib / vm.disk_gib. Its agent can ask for
more with request_resources, up to the per-VM maximums, and gets it at its next restart_vm
(see architecture.md).
Small shared host, people decide (8 GiB RAM, many light conversations)
[vm]vcpus = 1memory_mib = 1024disk_gib = 8max_vcpus = 2max_memory_mib = 3072max_disk_gib = 20max_vms = 4memory_budget_mib = 5120 # leave ~3 GiB for the host and Firecracker overheadmin_free_disk_gib = 20grants = "approval" # approvers answer "My VM needs more room: …" in the chatBuild box for a trusted team, agents decide
[vm]vcpus = 4memory_mib = 4096disk_gib = 30max_vcpus = 16max_memory_mib = 32768max_disk_gib = 200max_vms = 6memory_budget_mib = 98304min_free_disk_gib = 50grants = "auto"
[agent]disk_alert_percent = 80 # nudge earlier on big buildsusage_interval_secs = 15Fixed sizes (agents can’t grow; they may still give resources back and sleep)
[vm]grants = "never"Rules of thumb:
- The memory budget, not
max_vms, is what usually limits concurrency: a VM waits to boot until its memory fits (/status:waiting for host memory). Withmemory_budget_mib = 0it is 80% ofMemTotal; set it explicitly on small hosts or hosts running other services. max_memory_mibabove the budget is effectively capped at the budget (vm_resourcesreports the lower value).- Disk images are sparse: grants check
min_free_disk_gibon the filesystem holdingdata_dir, but writes inside a VM consume host space later. Plan formax_disk_gib× active conversations, or watch free space. - Lowering a maximum later clamps existing grants at their next boot; disks never shrink.
- Alerts (
disk_alert_percent,memory_alert_percent, 10–99) only steer the agent to clean up;usage_interval_secs(≥ 5) sets how fresh/statusandvm_resourcesare.
Reading /status
Section titled “Reading /status”/status (mention the bot first in groups) answers within 3 s with the session’s state and the
VM’s allocation, plus the latest usage while it runs:
| Answer | Meaning |
|---|---|
asleep · 2 vCPU, 2048 MiB, 10 GiB |
No VM runs; the next message boots one with this allocation (including grants not yet applied) |
waiting for a free VM slot |
vm.max_vms VMs are running |
waiting for host memory |
The allocation doesn’t fit in the memory budget right now |
working · 2 vCPU, 2048 MiB, 10 GiB · memory 75% · disk 90% |
The agent is busy (or provisioning); usage from the last report |
idle · … |
The VM runs, the agent waits for messages; it stops after vm.idle_shutdown_secs |
restarting after an error: … |
The last boot failed; the session retries with backoff |
starting up |
The session didn’t answer within 3 s, typically because it is booting a VM (preparing the disk and network, waiting for the guest’s hello), when it doesn’t read commands (issue #12; the dashboard shows booting) |
Choosing a harness
Section titled “Choosing a harness”agent.harness sets the program every conversation of this bot runs; one Gopala serves one bot
config, so pick per bot. The comparison is in
architecture.md. In short: omp (the default) is a general agent
whose tool list shapes it for coding, research or data work; Claude Code is a coding agent on
Anthropic models; exec runs a one-shot CLI such as codex exec per message, without a
session between messages. Whatever you pick, install it from the host (the tools image or
image.setup), never with agent.user_setup: the agent could replace a harness it installed
itself, and with the tools image the guest refuses to start one it could change.
omp (default):
[agent]harness = "omp"command = "omp" # bin/omp on the tools driveargs = ["--approval-mode", "yolo"]tools = [] # omp's names, e.g. ["read", "grep", "edit", "write", "bash"]user_setup = [] # with the tools image; the default installs omp from the internetModels come from the [[llm]] entries’ guest_env (see
Several model providers).
Claude Code with Anthropic through the host proxy:
[agent]harness = "claude_code"command = "claude" # bin/claude on the tools drive ("" means the same)args = ["--model", "sonnet"] # replaces omp's default args, which Claude Code doesn't taketools = [] # Claude Code's names, e.g. ["Bash", "Read", "Edit", "Write", "Grep", "Glob"]user_setup = [] # don't install omp
[[llm]]name = "anthropic"upstream = "https://api.anthropic.com"guest_port = 8701auth_header = "x-api-key"secret_env = "ANTHROPIC_API_KEY"guest_env = { ANTHROPIC_BASE_URL = "http://127.0.0.1:8701", ANTHROPIC_API_KEY = "via-gopala" }gopala image build puts the pinned Claude Code on the tools drive when agent.harness = "claude_code"; without a tools image, install it in the rootfs instead ([image] packages = ["nodejs", "npm"], setup = ["npm install -g @anthropic-ai/claude-code"]; not tried with
Ubuntu’s apt Node yet). The placeholder ANTHROPIC_API_KEY is there because claude -p may
refuse to start without a key; the proxy removes it and adds the real one. That hasn’t been
tested: if Claude Code reports missing credentials, check that line first. Claude Code runs with
bypassPermissions, so approvals for critical steps come only from the request_approval chat
tool; there’s no stricter mode to set in agent.args.
exec with OpenAI’s Codex CLI, one run per message:
[agent]harness = "exec"command = "codex"args = ["exec", "--skip-git-repo-check", "--sandbox", "danger-full-access", "-"]tools = [] # exec ignores ituser_setup = [ # Codex reads MCP servers from its own config, not .mcp.json: give it the chat tools. "mkdir -p ~/.codex && printf '[mcp_servers.chat]\\ncommand = \"/usr/local/bin/gopala-guest\"\\nargs = [\"mcp\", \"--port\", \"1025\"]\\n' > ~/.codex/config.toml",]
[image]packages = ["nodejs", "npm"]setup = ["npm install -g @openai/codex"]
[[llm]]name = "openai"upstream = "https://api.openai.com"guest_port = 8702auth_prefix = "Bearer "secret_env = "OPENAI_API_KEY"guest_env = { OPENAI_BASE_URL = "http://127.0.0.1:8702/v1", OPENAI_API_KEY = "via-gopala" }The guest pipes the system prompt, a blank line and the message into codex exec -; Codex’s
stdout lines show up as text in the journal, and only what it sends with the chat tools reaches
people, which is why it needs the chat server in its own config. danger-full-access turns off
Codex’s own sandbox, since the VM is the sandbox. Each message starts from scratch, and
restart_harness has nothing to restart. None of this exec setup has been run against a real
Codex release; check the flags and the config format against the version you install.
Collaboration
Section titled “Collaboration”Several agents can share a group: another Gopala with its own bot account, or any other bot.
With [collaboration] on (the default) they find each other with the list_agents tool, ping
each other with @mentions and split the work; each decides for itself what to do with a ping,
and the agent the person asked owns the answer. How it works is in
architecture.md.
[collaboration]enabled = trueagent_role = "agent" # Lupyd group role / Discord role / Slack user group handlemax_agent_messages_per_hour = 30 # wake-ups by other agents per conversation; people don't countAn agent wakes another only by mentioning it or replying to it, never through listening mode,
and a ping can’t run /stop or other commands or answer approvals. Past the hourly cap, pings are
dropped with a line in the log. The agents need to tell each other apart, so set each platform up
so that every bot is recognised and says what it’s good at:
- Lupyd has no bot accounts, so the role is the only signal. In each group, create a group
role named like
agent_role(agent, compared ignoring case) and give it to every bot, including this one. Then write each bot’s profile bio in the Lupyd app: one or two sentences on what it does (“Researches papers and datasets, cites sources”). That bio is what other agents read inlist_agents; Gopala fetches it fromplatforms.lupyd.profile_api_urlwith the bot’s own token, so leave that at the Lupyd API. - Discord recognises bots by themselves. Give the
agentrole only to non-bot accounts that should count as agents. Turn on the Server Members Intent for a complete list; without it Gopala knows only members it has seen, mostly people who have posted. Discord doesn’t let one bot read another’s description, soaboutstays empty: descriptive role names and nicknames are what the other agents see. - Slack recognises bot users by themselves. For other accounts that should count, create a
user group whose handle is
agent_role(@agent) and add them.aboutcomes from the profile’s title (and status), so set a title that says what the account does. The manifest’susers:readandusergroups:readscopes cover this; apps installed beforeusergroups:readwas added must be reinstalled.
[collaboration] applies without a restart: enabled and the cap with the next message, a new
agent_role in list_agents within 5 minutes, and the system prompt’s collaboration section at
each VM’s next boot. Which incoming messages count as agents’ follows a changed agent_role only
after you restart gopala run. In the journal, a ping shows as a prompt starting
name (agent) …. To turn it off for this bot, set enabled = false: other agents’ messages are
then ignored everywhere and list_agents answers “collaboration is off on this host”.
Alerts
Section titled “Alerts”An agent marks a message warning or critical when people need to notice it; [alerts] says
who that notifies. The agent never names people, so prompt injection can’t make it page someone
you didn’t list. Defaults: warnings notify nobody, critical messages notify everyone, at most 4
notifying messages per conversation per hour.
[alerts]warning = ["role:on-call"] # Slack user group handle / Discord role namecritical = ["here", "user:alice"] # "everyone", "here", "role:<name>", "user:<username>"host_events = "critical" # a VM that keeps failing, a budget blockmax_per_hour = 4 # later ones still go out, notifying nobody
[[alerts.conversations]] # one group with its own audiencekey = "slack-g-1f2e3d4c5b6a7980" # keys are on the dashboardcritical = ["role:payments-oncall"]What each entry becomes:
| Entry | Slack | Discord | Lupyd |
|---|---|---|---|
everyone |
@channel |
@everyone |
@everyone as text (no native mention) |
here |
@here |
@here |
@here as text |
role:<name> |
the user group with that handle (needs usergroups:read) |
the role with that name | skipped: Lupyd has no role mentions |
user:<name> |
the person with that Slack handle | the server member with that username | @name, if they’re in the group |
Direct chats never notify; the person gets the message anyway. Names that don’t resolve are
skipped with a warning in the log (slack: no user group @… to notify, discord: no role …,
lupyd: … not in this conversation) and the message still goes out. When the hourly cap is used
up, the message goes out without mentions and the agent’s tool result says
sent without notifying anyone: at most N notifying messages an hour here. In the journal,
serious messages show as message (warning) or message (critical). [alerts] applies without
a restart.
Platform requirements: Slack apps need usergroups:read for roles (reinstall older apps), and
admins may restrict @channel/@here; Discord bots need “Mention @everyone, @here and All Roles”
(re-invite bots invited before this permission was added, or grant it through a role). See
Creating the bot.
Listening mode
Section titled “Listening mode”A group normally wakes the agent only with a mention or a reply (respond_to = "mentions"). In
listening mode (respond_to = "all" for that one group) every message reaches the agent. Switch
it:
-
in the chat, as an approver:
@bot /listen all(oron) and@bot /listen mentions(oroff); the bot confirms with 👂 or 🔕; -
on the dashboard, with a conversation’s Listening toggle;
-
in the file:
[[behavior.conversations]]key = "slack-g-1f2e3d4c5b6a7980"respond_to = "all"
The chat command and the toggle write that entry into gopala.toml (keeping your comments, with
a .bak) and apply it to the next message, so the gopala.toml directory must be writable (see
ReadWritePaths in the unit below). If saving fails, the bot answers “Couldn’t save that
setting; an operator needs to look at the configuration.” and nothing changes. Listening mode
costs more tokens: every message in the group wakes the agent, so pair it with a
budget.
Memory
Section titled “Memory”With [memory] enabled = true (the default) every agent gets a memory MCP server and keeps
facts across restarts in /home/agent/.gopala/memory.db on its own disk. Nothing to set up on the
host; keywords (FTS5) work without any model. Two optional extras use an [[llm]] entry, through
the host proxy, so their tokens show on the dashboard and count against budgets:
[memory]max_memories = 5000 # active memories per conversationmax_db_mib = 64half_life_days = 30archive_days = 30duplicate_threshold = 0.85cleanup_interval_secs = 3600 # at least 60
[memory.embeddings] # recall by meaning as well as wordsllm = "openai" # an [[llm]] name; must speak OpenAI's /v1/embeddingsmodel = "text-embedding-3-small"# dimensions = 512 # 0 = the model's default
[memory.compaction] # a model merges and rewrites related memoriesllm = "anthropic"model = "claude-haiku-4-5"format = "anthropic" # or "openai" for /v1/chat/completionsevery_hours = 24min_memories = 200max_clusters = 20The [[llm]] entry’s upstream should be the API’s base without /v1 (as in the examples
above), since memory adds /v1/... itself [inference from the paths it posts].
Sizing, per conversation:
- Disk: the store stays under
max_db_mib(64 MiB) inside the conversation’s disk; leave that much room invm.disk_gib. 5000 memories of a few hundred bytes each, with 1536-dimension embeddings (6 KiB each), come to about 30 MiB [inference]. - Embedding tokens: one call per
remember,updatewith new text, orrecall, sized like the text; the maintenance pass embeds up to 256 missing memories per run. - Compaction tokens: at most
max_clusterscalls per run, each about 3,000 input tokens (12,000 characters of memories plus instructions) and up to 4,096 output tokens, so roughly 140,000 tokens per conversation per run at the defaults, once a day when there are 200 or more memories.
Changing [memory] needs a restart of gopala run; each VM picks it up at its next boot. Turning
memory off removes the server and the system prompt’s line about it; the file stays on the disk.
An [[agent.mcp]] entry named memory is refused while memory is on.
Monitoring
Section titled “Monitoring”People in a conversation can follow its agent outside the chat: with the link the bot sends when
someone writes /watch, or with any app that speaks the /monitor websocket (or its HTTP mirror under
/api/monitor/). The Lupyd app connects with the person’s own Lupyd sign-in, so Lupyd users get
every task, tool call and subagent in the app; Slack and Discord users keep the chat messages and
watch links. Every request is authenticated, and a person only ever sees conversations their
platform says they’re in. [monitoring] decides, on top of membership, who may read the agent’s
output (the journal and live events) and its VM’s metrics (CPU, memory, disk, network,
token usage, budget):
[monitoring]output = ["members"] # default: anyone in the conversationmetrics = ["approvers"] # default: whoever may answer approvals ([approvals] approvers)lupyd_tokens = true # accept Lupyd sign-in tokens on /monitormembership_cache_secs = 60 # how long a platform's answer is reused (at most 3600)Rules, any of which grants: members, approvers, owner, role:<name>,
permission:<name>, user:<username>. An empty list leaves it to the operator. Roles,
permissions and owners come from each platform:
| Platform | role: |
permission: |
owner |
|---|---|---|---|
| Lupyd | the member’s role in the group | the group role’s permissions: see_message, pin_message, add_message, manage_channel, manage_role, manage_member, manage_group |
the group’s owner |
| Slack | handles of the user groups they’re in (needs usergroups:read) |
their workspace standing: admin, owner, primary_owner |
the channel’s creator |
| Discord | their role names in the server | server-level permissions as lowercase names (manage_guild, manage_messages, administrator, …); administrators and the owner have all of them |
the server’s owner |
In a direct chat the other person is the only member and counts as its owner.
Three alternatives, one per platform:
# Slack: everyone in the channel sees the agent's work; workspace admins and the# @oncall user group see metrics.[monitoring]output = ["members"]metrics = ["permission:admin", "role:oncall"]
# Discord: people with Manage Server see everything; the rest nothing.[monitoring]output = ["permission:manage_guild"]metrics = ["permission:manage_guild"]
# Lupyd: group managers see metrics; one sensitive group shows output to its owner only.[monitoring]metrics = ["permission:manage_group", "owner"]
[[monitoring.conversations]]key = "lupyd-g-42"output = ["owner"]metrics = ["owner"]Platform notes: Slack’s member list is reused for 30 s (or until someone joins or leaves) and user
groups for 5 minutes. Discord learns about members joining and leaving only with the Server
Members intent; without it Gopala searches the member list when an answer expires, so a removed
member keeps access for up to membership_cache_secs.
Connecting an app. Open wss://<public_url host>/monitor and send a protobuf
MonitorClientMessage with an authenticate request first (or put Authorization: Bearer <token> on the upgrade):
- the Lupyd app: the person’s Lupyd access token (
lupyd_token); - a script for one conversation: the person’s
/watchtoken, thegpl1.…part after#in the link (watch_token); - operator tooling: the admin token (
admin_token).
Then list, status, metrics, usage, events, or subscribe to stream events and metrics.
The messages and limits are in protocols.md. For a quick look
without a websocket client:
curl -H "Authorization: Bearer $GOPALA_ADMIN_TOKEN" https://agents.example.com/api/monitor/conversationscurl -H "Authorization: Bearer gpl1.…" "https://agents.example.com/api/monitor/conversations/lupyd-g-42/events?limit=50"[monitoring] changes apply without a restart; a cached membership answer lasts up to membership_cache_secs.
The dashboard
Section titled “The dashboard”gopala run serves a control panel at /dashboard/ under server.public_url, next to the
configuration editor (/configure/), the docs (/docs/) and the front page (/). It needs the admin
token:
# In /etc/gopala/env (see below), then restart:GOPALA_ADMIN_TOKEN=<openssl rand -hex 32>Open the URL and sign in with the token. The browser gets an HttpOnly session cookie valid for 12
hours (a restart of Gopala signs everyone out); “Sign out” ends it. More than ten wrong tokens
within a minute lock your address out for a minute (429, even with the right token afterwards).
Behind a reverse proxy all clients share that lockout. Scripts can use Authorization: Bearer <token> with
the same API instead (protocols.md).
What it shows, live (server-sent events, no refresh needed):
- Host: how much of the memory budget running VMs hold, VM slots in use, free disk under
data_dir, the base and maximum allocation and grant mode, and today’s totals across all conversations (tokens, model requests and errors, CPU time, tool calls, messages, LLM traffic). - Conversations: one row per conversation the bot knows, filterable by platform and status
and sortable by activity, tokens, CPU or memory: status (with the error when
failed), CPU, memory and disk (measured), tokens and calls today, last activity; “resize at next boot” marks a pending allocation. - A conversation’s detail: its activity (the journal, filterable: messages, tools and
approvals, model calls, lifecycle and notices, errors), model calls (one row per
llmevent), resources, and usage (today and total). - Model budget: the host’s tokens today and this month against
[budget], with marks where low- and normal-priority conversations stop; per conversation, its priority, limits, usage and the reason when its model calls are blocked. - Usage over time: per day for 7, 30 or 90 days, per metric and conversation.
- Live activity: every journaled event since the page opened.
Measured vs reported. The “Measured by the host” figures come from outside the VM: CPU and resident memory of the firecracker process, the disk image’s real size on the host, and the tap’s network bytes, sampled every 5 seconds; tokens come from the model provider’s own responses through the LLM proxy. “Reported by the VM” is what the guest says about itself (memory and disk percent, load, report age): useful, but unverified, since a compromised agent can lie. The dashboard warns when the two differ by more than 20 points. Expect some honest difference too: host RSS includes guest page cache and Firecracker’s overhead, and the disk image’s allocated size includes deleted-but-not-trimmed blocks [inference].
Actions (on a conversation): Abort stops the agent’s current turn and drops queued
messages; Sleep stops the VM now; Restart restarts it (or boots it if asleep) and applies a
pending allocation; Leave makes the bot leave the group or chat and ends the session (not
undoable from the dashboard); Set allocation sets vCPUs, memory and disk for the next boot,
bypassing vm.grants and approvals but not the per-VM maximums, the memory budget, the disk
reserve or “disks only grow”; Budget sets the conversation’s daily and monthly tokens and its
priority, saved to [[budget.conversations]] and applied at once; Listening switches the
group’s listening mode, saved to [[behavior.conversations]]. Every action except Budget is
journaled as an operator notice.
The usage ledger is <data_dir>/usage.db, flushed every 30 seconds and at shutdown (a crash loses
up to 30 seconds of counts), with daily rows kept for 400 days and all-time totals forever.
Run as a systemd service
Section titled “Run as a systemd service”Secrets in a root-only environment file, /etc/gopala/env (mode 0600):
LUPYD_EMAIL=bot@example.comLUPYD_PASSWORD=… # only needed until the first sign-in; remove it afterwardsSLACK_BOT_TOKEN=xoxb-… # [platforms.slack]SLACK_APP_TOKEN=xapp-…DISCORD_BOT_TOKEN=… # [platforms.discord]ANTHROPIC_API_KEY=…# TRACKER_TOKEN=… # an [[agent.mcp]] secret_env# GOPALA_ADMIN_TOKEN=… # only if you use the dashboard or the browser configurationWithout the jailer (dedicated user with KVM access and CAP_NET_ADMIN for tap devices):
[Unit]Description=gopala: autonomous agents for chat platforms in Firecracker microVMsWants=network-online.targetAfter=network-online.target
[Service]User=gopalaGroup=gopalaSupplementaryGroups=kvmEnvironmentFile=/etc/gopala/envEnvironment=RUST_LOG=info,firefly_client=warn,firefly_core=warnExecStart=/usr/local/bin/gopala --config /etc/gopala/gopala.toml run# Tap devices (vm.network = "nat"; drop these two lines otherwise); ambient capabilities are inherited by `ip` and firecracker.AmbientCapabilities=CAP_NET_ADMINCapabilityBoundingSet=CAP_NET_ADMINNoNewPrivileges=yesStateDirectory=gopalaStateDirectoryMode=0700UMask=0077ProtectSystem=strictProtectHome=yes# /etc/gopala only if PUT /api/config should be able to save there (temp file + .bak in that directory).ReadWritePaths=/var/lib/gopala /etc/gopalaRestart=on-failureRestartSec=10
[Install]WantedBy=multi-user.targetWith [vm.jailer], Gopala must start as root (the jailer needs it to chroot, create device
nodes, join cgroups and drop to vm.jailer.uid/gid): drop User=, Group=,
AmbientCapabilities=, CapabilityBoundingSet= and NoNewPrivileges=, and add the jailer’s
chroot_base to ReadWritePaths=. chroot_base must be on the same filesystem as data_dir
and vm.kernel.
systemctl daemon-reloadsystemctl enable --now gopalajournalctl -u gopala -fThe daemon stops on SIGINT/SIGTERM (closes every session, waits 2 s, shuts the providers down)
and exits with an error when a provider reports a fatal error (Lupyd signed it out, a Slack token
was revoked, Discord refused the intents), so Restart=on-failure brings it back (sign-in then
needs the Lupyd password again if the refresh token was revoked). It refuses to start when no
platform is enabled. Only one process may use a Lupyd device directory (firefly-mcp.lock).
Observability
Section titled “Observability”| Source | What you get |
|---|---|
Daemon log (stderr, env_logger) |
Default filter info,firefly_client=warn,firefly_core=warn; override with RUST_LOG. Provider sign-in (lupyd: signed in as …), joins/leaves, policy decisions, guest <version> connected, reconnects, VM boots and exits, agent VM failed: …, proxy errors, web server errors. |
<data_dir>/vms/<key>/console.log |
Firecracker’s stdout/stderr including the guest serial console (kernel console=ttyS0 … quiet), truncated on each boot, capped at vm.console_log_max_kib (default 4 MiB). The first place to look when a VM doesn’t connect. |
<data_dir>/vms/<key>/allocation.json |
The conversation’s granted allocation (absent while it has none). |
gopala watch |
The session journal of one conversation: lifecycle (booting with the allocation, provisioning, ready, busy, idle, restarting, stopped, crashed), provisioning output, harness stderr, prompts (other agents’ marked (agent)), text, thinking, tool calls and results, subagents, approvals, outbound messages, resources notices (alerts, grants, denials, restarts, sleeps) and harness notices (the agent asked for a harness restart). |
/status in the chat |
See Reading /status. |
GET /api/status |
{"mode", "admin", "signed_in"}: run or web, whether the admin API is on, and whether this request is signed in; signed in, also "version". |
GET /api/monitor/conversations |
What a given credential may see (Bearer admin, watch or Lupyd token); see Monitoring. |
| The dashboard | Live state, measured usage, journals and model calls per conversation; see The dashboard. |
<data_dir>/usage.db |
The usage ledger (SQLite): usage_days and usage_totals, one row per conversation key (and day). |
| Inside a VM | journalctl -u gopala-guest (guest log, RUST_LOG=info) if you can get a shell, e.g. by mounting a stopped VM’s disk. |
gopala watch takes the link from its argument, GOPALA_WATCH_LINK (preferred: removed from the
environment after reading) or stdin, and prints one line per event:
#<seq> <HH:MM:SS> [<actor>] <kind>[ <title>][ !]: <text>--since <seq> replays only events after seq; --live skips history.
Troubleshooting
Section titled “Troubleshooting”| Symptom / message | Cause | Fix |
|---|---|---|
enable at least one platform under [platforms] / no platform is enabled (see [platforms] in the configuration) |
Every [platforms.*] has enabled = false |
Enable one |
alerts entry "…": use "everyone", "here", "role:<name>" or "user:<username>" |
A Slack or Discord mention syntax (@channel, <@U…>) or a typo |
Use the neutral form |
Log: slack: no user group @… to notify / slack: can't notify @… / discord: no role … ; not mentioning it / lupyd: no way to mention the role … |
The name doesn’t exist there, the Slack app lacks usergroups:read, or Lupyd has no roles |
Fix the name; reinstall the Slack app with the new manifest; use user: on Lupyd |
Log: discord: the bot lacks "Mention @everyone, @here, and All Roles" … |
Bot invited before the permission was added | Re-invite it with the new link or grant the permission through a role |
Agent tool result sent without notifying anyone: at most N notifying messages an hour here |
alerts.max_per_hour used up |
Expected; raise it if the alerts were real |
Chat: Only approvers can change how I listen here. |
/listen from someone outside approvals.approvers |
Ask an approver, or use the dashboard |
Chat: Couldn't save that setting; … after /listen |
gopala.toml or its directory isn’t writable, or the edited file didn’t validate (daemon log: /listen: …) |
Fix permissions (ReadWritePaths) or the file |
behavior.conversations key "…" must be a conversation key … / alerts.conversations key … |
Not a key (copy keys from the dashboard) or listed twice | Fix the entry |
memory.embeddings needs a model and llm = the name of an [[llm]] entry … / memory.compaction needs … |
The named [[llm]] entry doesn’t exist, or no model |
Name an existing [[llm]] entry |
agent.mcp name "memory" is the built-in memory server's … |
Your own MCP server named memory |
Rename it, or set memory.enabled = false |
git push: ! [remote rejected] main -> main (protected by the Gopala host) and remote: Gopala: … |
The push touched a branch in protected_branches (any create, update or delete refuses the whole push) |
Expected; the agent pushes another branch and opens a pull/merge request |
git push refused: signed push or compressed push body |
The host’s git proxy can’t check protected branches in those | Push unsigned, without push.gpgSign; plain git sends uncompressed pushes |
| Git or API call answers 404 | Git LFS, or a git transport other than smart HTTP (dumb HTTP, other paths) | Unsupported; keep large files out of LFS for agent repos |
API call refused with a hint to use the path (/projects/<id>, /repositories/<id>) |
repos is set and the call names the repository by numeric id |
Use the URL-encoded owner/repo path (GitLab /projects/owner%2Frepo) |
git <name>: $GITHUB_TOKEN is not set |
The [[git]] entry’s token_env variable is missing from the environment |
Add it to /etc/gopala/env (or export it) and restart |
| Merging a pull/merge request refused | The host refuses every merge call, by design | A person merges |
| A second account on the same host gets the first account’s repos or permissions | Only the first [[git]] entry per host gets the URL rewrite |
Clone the second through its own http://127.0.0.1:<port>/ URL (3300 + its index) |
Slack: $SLACK_BOT_TOKEN is not set / SLACK_BOT_TOKEN must hold the app's Bot User OAuth Token (xoxb-…) |
Missing or wrong token | Use the xoxb- token from OAuth & Permissions |
SLACK_APP_TOKEN must hold an app-level token (xapp-…) with the connections:write scope / Slack rejected SLACK_APP_TOKEN (Socket Mode needs …) |
No app-level token, a missing scope, or Socket Mode off | Generate an app-level token with connections:write; create the app from the manifest |
Slack refused gopala's token (… from …); the token was revoked or the app removed and the daemon exits |
invalid_auth, account_inactive, token_revoked, or the app was uninstalled |
Reinstall the app and update the tokens |
| Slack: the bot never answers in a channel | Not a member, or respond_to = "mentions" and no mention |
/invite @gopala; mention it |
Discord rejected the bot token in DISCORD_BOT_TOKEN … |
Wrong or reset token | Reset it in the Developer Portal and update the variable |
Discord: the bot can't read messages: turn on "Message Content Intent" … / Discord refused the bot's gateway intents: … |
Message Content Intent is off | Turn it on (Bot → Privileged Gateway Intents) and restart |
Discord requires sharding for this bot (over 2500 servers) … |
Gopala runs one shard | Not supported |
Discord: conversation_info lists only the bot and people who wrote |
Server Members Intent is off | Turn it on and restart (optional) |
vm.network = "proxy" needs vm.allowed_hosts (or use "none") |
Proxy mode with an empty list | List the hosts |
agent.packages need vm.network = "nat" or "proxy" … |
Packages with network = "none" |
Bake them in with [image] packages, or change the mode |
vm.allowed_hosts entry "…": expected host, *.domain, optionally with :port |
URL, path or bad wildcard | Write pypi.org, *.github.com, example.com:8443 |
Journal: network notice refused host:443 (not allowed) / (resolves to …, not a public address) / (doesn't resolve); programs see 403/502 |
Not in allowed_hosts, a private address, or DNS failed |
Add the host if it should be reachable; private addresses are never allowed |
Programs in a proxy VM fail without a network notice |
They ignore HTTP_PROXY/HTTPS_PROXY (raw sockets, ssh) |
Use HTTPS, configure the program’s proxy, or use nat |
llm guest_port N is taken by the guest (3128 is the egress proxy, 3200 and up the HTTP MCP servers) |
Port collision | Pick another guest_port |
agent.mcp name "…" must be [a-z0-9_-]{1,32} and not "chat" / … is used twice / … needs either a command or an http(s) url / secret_env is only for HTTP servers (url) |
Validation | Fix the entry |
agent.mcp NAME: $VAR is not set |
An HTTP MCP server’s secret_env is missing (also in check-config) |
Export it |
Log: agent.mcp NAME: its secret is read at startup; restart gopala to use it |
A server with secret_env added by a hot reload |
Restart |
Model calls fail with 429 budget_exceeded; journal budget notice |
The conversation’s or the host’s token budget is used up | Wait for 00:00 UTC (or the 1st), or raise the budget on the dashboard |
budget.conversations key "…" must be a conversation key like "lupyd-g-42" … / budget.low_stops_at_percent <= … |
Validation | Copy keys from the dashboard; keep low ≤ normal, both 1–100 |
gopala image build: … build-rootfs.sh failed |
A script step failed (it says which); Debian or packages without root | Rerun with sudo; install debootstrap for Debian |
not signed in to Lupyd: set LUPYD_EMAIL and LUPYD_PASSWORD |
No usable session.json and no credentials (names from email_env / password_env) |
Export both variables for the first run |
set both LUPYD_EMAIL and LUPYD_PASSWORD |
Only one is set | Set both, or neither once signed in |
Lupyd sign-in failed |
Wrong credentials, wrong auth0_* settings, or the tenant refused the password-realm grant |
Check the account in the app; for a non-production stack check [platforms.lupyd] |
the sign-in server returned no refresh token (offline_access not allowed) |
The Auth0 client doesn’t allow refresh tokens | Use a client/tenant that issues them (production defaults do) |
saved Lupyd session no longer works (…); signing in again (warning) |
Refresh token revoked or expired | Provide the password again if sign-in then fails |
another firefly-mcp is already running on … |
Two processes on one data_dir/lupyd |
Run one Gopala per account and data directory |
lupyd: the bot's account already has N devices; remove one in the app (Settings, Devices) … |
Lupyd’s device limit | Remove a device in the app; the bot connects by itself afterwards |
signed out of Lupyd: … and the daemon exits |
The device was removed or the session ended | Restart with credentials |
every platform stopped |
All providers’ event pumps ended | Check the log for the provider’s error |
llm <name>: $<VAR> is not set |
Missing API key variable (also in check-config) |
Export it or remove the [[llm]] entry |
server.bind … is not loopback: set server.tls_cert/tls_key … |
Public plaintext bind | Add TLS, bind to loopback behind a proxy, or allow_plaintext = true on a private network |
rooms.enabled needs server.enabled … |
Watching without the web server | Enable [server] or disable rooms |
binding 0.0.0.0:60901 / web server stopped: … |
Port in use or no permission | Change server.bind |
site directory … does not exist; serving only the API and the websocket (warning) |
server.site_dir missing |
Build the site and copy site/dist there, or ignore |
API 403 admin API disabled |
server.admin_token_env variable unset or empty |
Set it (long random value) and restart |
API 401 unauthorized / the dashboard asks to sign in again |
Wrong or missing token, or the session expired (12 h) or Gopala restarted | Sign in again; scripts: Authorization: Bearer <token> |
API 429 too many failed sign-ins; try again later |
More than 10 wrong tokens from your address within a minute (behind a proxy: from anyone) | Wait the Retry-After seconds; check for a script with a stale token |
API 403 missing X-Gopala header |
A cookie-authenticated write without the CSRF header | Send X-Gopala: 1 (the dashboard does), or use a Bearer token |
API 409 gopala web runs no agents; start gopala run |
Dashboard or control endpoints under gopala web |
Run gopala run |
PUT /api/config lists restart_required |
Those sections can’t change while running | Restart gopala run |
API 500 cannot save the configuration |
The config directory isn’t writable by Gopala (temp file and .bak live next to the file) |
Fix permissions / ReadWritePaths |
vm.tools_image … is unavailable / is not a regular file / is not readable by the jailer uid … / has sha256 …, but vm.tools_sha256 is … |
The tools image is missing, unreadable or changed | Fix the path or permissions; rebuild and paste the new tools_sha256 |
Journal: lifecycle refused, refusing to start the harness … |
With tools, the harness resolved to a file the agent user can change | Make sure the image has the harness in bin/ and agent.command is a bare name (default omp) |
Journal: host tools unavailable: … |
The guest couldn’t mount the tools drive | Check the kernel has ext4 and virtio-blk, and console.log |
Dashboard tokens stay at zero while llm_requests grows |
The provider’s responses carry no usage report the proxy can read (e.g. an encoded body) | Check the llm events’ status; see the metering notes in architecture.md |
/watch answers 404 |
gopala web mode or rooms.enabled = false |
Run gopala run with rooms enabled |
Viewer: server: not allowed to watch this conversation / access revoked |
The holder isn’t in the conversation any more, or [monitoring] output doesn’t include them |
Expected; adjust [monitoring] if they should watch |
/monitor or /api/monitor/: 401 unauthenticated |
No credential, a wrong admin or watch token, an invalid or expired Lupyd token, or lupyd_tokens = false |
Check the token; Lupyd tokens need platforms.lupyd.auth0_domain and auth0_audience to match the app’s tenant (debug log: monitor: refused a Lupyd token: …) |
/monitor: 401 credentials expired; authenticate again |
The Lupyd token expired or the watch token was revoked | Send authenticate again with a fresh token |
/monitor or /api/monitor/: 404 no such conversation for a conversation that exists |
The caller isn’t a member (as far as the platform says), it’s another platform’s conversation, or the watch token is for another conversation | Expected; on Discord without the Server Members intent, check the person can be found by member search |
/monitor: 403 not allowed to read this conversation's output / … metrics |
A member the [monitoring] rules don’t allow |
Adjust output / metrics |
/monitor: 429 slow down / at most 16 subscriptions per connection |
More than 20 requests a second (bursts of 40), or too many subscriptions | Slow down; unsubscribe first |
Log: <key>: can't ask <platform> whether <user> is a member: … |
The platform’s API failed; the person is treated as not a member | Check the provider’s log and permissions (Slack scopes, Discord intents) |
/configure/ or /templates/ can’t check a file against the server |
Signed out: the configuration helpers need the operator now, so the pages use the schema bundled with the site | Sign in |
monitoring rule "…": use "members", "approvers", "owner", "role:<name>", "permission:<name>" or "user:<username>" / monitoring.membership_cache_secs is at most 3600 … / monitoring.conversations key … |
Validation | Fix the rule, cache time or key |
vm.max_vcpus, vm.max_memory_mib and vm.max_disk_gib can't be below … / vm.memory_budget_mib is smaller than one VM's vm.memory_mib |
Validation | Raise the maximums or budget |
/status: waiting for host memory for long |
Running VMs hold the memory budget | Wait for idle shutdowns, raise memory_budget_mib, lower max_memory_mib |
Agent tool result this host runs all its VMs in N MiB of memory / over this host's per-VM maximum … / the host is low on disk space; clean up instead / disks only grow … / this host doesn't grant more resources … |
machine::decide denied a request |
Expected; adjust [vm] maximums, budget, min_free_disk_gib or grants if the request was legitimate |
vm.kernel … is unavailable / … does not exist |
Wrong image paths | Use the paths gopala image build printed |
can't start firecracker / firecracker not found in PATH |
Binary missing or not executable (PATH is passed through) |
Install it or set vm.firecracker to an absolute path |
the guest didn't connect in time |
Guest didn’t reach the control socket within vm.boot_timeout_secs |
Read console.log: kernel without virtio-vsock, guest service not enabled, image without gopala-guest; raise the timeout on slow hosts |
the VM exited while booting: … (see its console.log) |
Firecracker or the kernel failed | console.log; check /dev/kvm permissions, kernel/rootfs compatibility, memory |
no hello from the guest / the guest didn't introduce itself |
Something other than gopala-guest connected, or it hung |
Rebuild the image |
guest speaks control protocol X, host Y |
Host and guest from incompatible versions | Rebuild the rootfs and update existing disks (see Build) |
socket path … exceeds 107 bytes; use a shorter data_dir/chroot_base |
{uds}_{port} doesn’t fit sun_path |
Shorten data_dir or vm.jailer.chroot_base |
can't hard-link … into the jailer chroot …: vm.jailer.chroot_base must be on the same filesystem as data_dir, vm.kernel and vm.tools_image |
EXDEV: chroot on another filesystem |
Move chroot_base (or data_dir / images) onto one filesystem |
`ip …` failed: … (tap devices need root or CAP_NET_ADMIN) |
No permission to create taps | Grant CAP_NET_ADMIN (see the unit), run as root with the jailer, or use network = "proxy" or "none" |
can't run `ip` (iproute2) |
iproute2 missing | Install it |
e2fsck failed / resize2fs failed |
e2fsprogs missing or a damaged disk | Install e2fsprogs; inspect the disk. An interrupted grow isn’t retried: run e2fsck -f and resize2fs on the disk manually |
vm.jailer.uid and gid must not be root |
Validation | Use a dedicated unprivileged uid/gid |
Journal shows provisioning failed: … |
A package, setup or user_setup step failed (the harness still starts) |
Read the EVENT_LOG lines; fix the step; it reruns when the inputs change |
no supported package manager (apt-get, apk, dnf) found |
Custom rootfs | Use one of these or set agent.packages = [] |
Chat: ⚠️ My workspace keeps failing to start; an operator needs to look at it. |
Three VM failures in a row | Daemon log and console.log; the session keeps retrying (up to 160 s apart) |
Agent tool error rate limit: at most N messages a minute here … |
behavior.max_messages_per_minute |
Expected; raise it if legitimate |
| Bot ignores a group | respond_to = "dms", or no @mention/reply with mentions |
Adjust respond_to; mention the bot |
| Bot leaves a group right after joining | invites policy and the owner isn’t allowed (or unknown) |
Add the owner to allowlist or use invites = "anyone" |
Watching is turned off on this bot. |
rooms.enabled = false |
Enable rooms |
Viewer: server: invalid or expired token |
Token expired, reissued by a later /watch, or revoked when the user left |
Send /watch again |
Viewer: server: room is full |
max_viewers_per_room |
Raise it or close other viewers |
Viewer: server: too slow, reconnect with since_seq |
Viewer fell 1024 events behind | Reconnect with --since <last seq> |
Viewer: no room key for epoch N / the room key does not unwrap with this token |
Missed or wrong key (e.g. token for another conversation) | Reconnect; request a new link |
collaboration.agent_role must be a role name of 1-100 characters / collaboration.max_agent_messages_per_hour must be at most 3600 / platforms.lupyd.profile_api_url must be an http(s) URL |
Validation | Fix the value |
image.setup entries must be non-empty shell commands |
An empty entry (or one with a NUL byte) in [image] setup |
Remove it |
gopala image build: base … with setup commands needs root: run it with sudo / setup command failed: … |
image.setup needs root; one of its commands exited non-zero |
Run the build with sudo; run the failing command by hand in a chroot to see why |
Agent tool error collaboration is off on this host |
collaboration.enabled = false |
Turn it on if agents should work together |
list_agents returns nobody (or misses an agent) |
Lupyd: nobody holds the agent group role; Discord without the Server Members intent: the agent hasn’t posted since Gopala started; Slack: the account isn’t a bot user or in the @agent user group; the answer is cached for 5 minutes |
Give the role (see Collaboration); turn on the intent; wait for the cache or change the group’s members |
Lupyd agents show no bio; log lupyd: can't read the profiles of … |
The profile request to platforms.lupyd.profile_api_url failed, or the bio is empty or an element body |
Check the URL and the host’s network; write a plain-text bio in the Lupyd app |
Agent tool error couldn't list agents: … |
The platform’s API failed (the log has the full error) | Check the provider’s log, scopes and intents |
| Another agent’s ping doesn’t wake the bot | It didn’t mention the bot or reply to it, it came as a direct message, collaboration is off, or the cap is used up (log: <key>: dropped a message from agent … (collaboration.max_agent_messages_per_hour)) |
Mention the bot; raise max_agent_messages_per_hour if the exchange is legitimate |
A changed agent_role doesn’t change which messages count as agents’ |
Providers read it once at startup | Restart gopala run |
Agent tool error your harness starts once per message, so there is nothing to restart |
restart_harness with agent.harness = "exec" |
Expected; the next message runs with the new MCP config |
Restart note: From /home/agent/.gopala/mcp.json: … left out "name" (…) |
The agent’s MCP file names a reserved server (chat, memory, an [[agent.mcp]] name), an invalid entry, more than 32 servers, or the file is over 64 KiB, not JSON, a symlink or not the agent’s |
The reason is in the note and the guest log; the agent can fix its file and restart again |
Journal: mcp notice MCP server <name>: failed (Claude Code) |
A server in the generated config didn’t start or connect | Check the command exists in the VM and the network reaches an HTTP server |
| Claude Code: model calls fail or it reports missing credentials | No [[llm]] entry with ANTHROPIC_BASE_URL in guest_env, or no placeholder ANTHROPIC_API_KEY (untested whether it needs one) |
Add both to the Anthropic entry’s guest_env (see Choosing a harness) |
Journal: lifecycle refused with agent.harness = "claude_code" |
claude resolved to a file the agent can change (installed by user_setup) while the tools image is in use |
Rebuild the tools image (gopala image build bundles Claude Code) or install it with image.setup; set command = "claude" |
| Claude Code or omp fails with an unknown option at start | agent.args still holds the other harness’s arguments (the default --approval-mode yolo is omp’s) |
Set args for the harness you chose |
Data and backups
Section titled “Data and backups”Everything lives under data_dir (layout in architecture.md).
Back up <data_dir>/lupyd as a secret (losing it means a new device and losing access to past
MLS state; Slack and Discord keep nothing on disk), <data_dir>/rooms/rooms.db if session
history matters, and <data_dir>/vms/*/disk.ext4 with allocation.json for the agents’
workspaces, memories (/home/agent/.gopala/memory.db inside the disk) and grants (preferably
while the VM is stopped). Deleting a conversation’s vms/<key> directory while its VM is stopped
resets that agent to a fresh copy of the rootfs at the base allocation, with no memories. Keep
the config file and its .bak with the backup.