Skip to content

Operations

How to install, configure, run and troubleshoot Gopala: AI agents that work around the clock on your own servers and take work from Lupyd, Slack or Discord.

For the hardening checklist see security.md.

Gopala has not yet been run against a real Firecracker host or a live Lupyd, Slack or Discord account. The steps below follow from the code and the upstream tools’ documentation; the systemd units are examples, not tested artifacts.

Requirement Why Notes
Linux x86_64 with KVM (/dev/kvm readable and writable by the service user) Firecracker See Firecracker’s getting started for checking KVM access
firecracker The VMM (vm.firecracker, a bare name is looked up in PATH) Release binaries: Firecracker releases
jailer (production) [vm.jailer] Requires starting Gopala as root; ships with Firecracker releases
e2fsprogs (e2fsck, resize2fs) Growing each conversation’s disk to its allocation (grow_disk) Also mkfs.ext4 and debugfs for building images
GNU coreutils cp cp --reflink=auto --sparse=always copies the base rootfs (ensure_disk) Reflinks make copies cheap on btrfs/XFS
iproute2 (ip) and CAP_NET_ADMIN Tap devices for vm.network = "nat" (setup_tap) Not needed with network = "proxy" or "none"
IP forwarding and a NAT rule Guest internet access with nat One-time setup below
A bot on each enabled platform Its identity See Creating the bot
Model provider API keys [[llm]] In the host environment only
curl, unsquashfs, fakeroot (unless root), jq, ldd, sha256sum gopala image build debootstrap too for a Debian base; root (sudo) for Debian, packages or setup

Gopala ships as one executable, gopala. Get a demo, and the binary, at contact@lupyd.com. The file embeds everything else Gopala needs: the image build scripts, the static gopala-guest agent that runs inside each VM (with its systemd unit), and the website. Only the host tools above are installed separately.

Terminal window
install -m 0755 gopala /usr/local/bin/
gopala --config /etc/gopala/gopala.toml image build # kernel, rootfs and tools image from [image]

The website (front page, Create an Agent, the dashboard, /configure/ and these docs) is served from the binary. If server.site_dir (default /usr/share/gopala/site) exists as a directory, that directory is served instead.

gopala image build [--out DIR] builds the rootfs and the read-only tools image from the [image] settings (base, packages, setup, rootfs_size_gib, tools) into --out (default <data_dir>/images) and prints the [vm] lines to paste: kernel, rootfs, tools_image and tools_sha256. With agent.harness = "claude_code" it also bundles Claude Code into the tools image.

[image]
base = "ubuntu-24.04" # or "debian-12"
packages = ["python3", "build-essential"] # baked in: no network needed at boot
setup = [] # root shell commands run in the image after packages
rootfs_size_gib = 4
tools = ["git", "rg", "jq"] # host programs on the read-only tools drive, next to omp

The rootfs build:

  1. downloads the newest Firecracker CI kernel (vmlinux-<version>), and for Ubuntu the CI squashfs, from Firecracker’s public CI bucket; Debian is built with debootstrap;
  2. installs packages with apt in a chroot, if any, then runs each setup command there with bash -c as root, in order, with a clean PATH and your http(s)_proxy/no_proxy if set; a failing command stops the build (setup command failed: …);
  3. builds an ext4 image of rootfs_size_gib and injects the embedded /usr/local/bin/gopala-guest and gopala-guest.service (enabled for multi-user.target), then runs e2fsck.

Ubuntu without packages or setup commands needs no root and no mounts. Debian, packages or setup commands need root: run it with sudo.

Two options are for special cases, when support asks you to use them: --guest FILE (or GOPALA_GUEST) bakes a different static gopala-guest into the rootfs, and --scripts DIR (or GOPALA_SCRIPTS) runs the image scripts from a directory instead of the embedded copies.

Upgrading the guest. Each conversation’s disk is a copy of vm.rootfs made on its first boot and kept afterwards (ensure_disk), so a rebuilt image only reaches new conversations. Existing conversations keep their old gopala-guest until their disk is replaced (deleting <data_dir>/vms/<key>/disk.ext4 while the VM is stopped also deletes the workspace) or the binary is written into it (e.g. with debugfs -w, as the image build does). Check CONTROL_PROTOCOL compatibility before upgrading the host; the resource monitor needs a guest that knows HostInit fields 13–15 (older guests simply don’t report usage), and the memory server one that knows field 20 (an older guest skips it [inference], so its agent has no memory tools even though the system prompt mentions them). restart_harness, the agent’s own MCP file and the Claude Code harness need a guest that knows HostFrame variant 6 and HARNESS_CLAUDE_CODE; what an older guest does with them is untested.

gopala image build also writes gopala-tools.ext4 (no root, no mounts): omp at a pinned, sha256-verified release, and the build host’s programs named in [image] tools (default git; [] for omp alone), each with every shared library and its dynamic loader under lib/. Dynamic programs become wrappers in bin/ that run the real binaries through the bundled loader; git’s helpers in libexec/git-core/ get the same treatment and GIT_EXEC_PATH=/opt/gopala/libexec/git-core, so they work whatever the guest’s libc (the build host needs glibc ≥ 2.33 for ld.so --argv0). MANIFEST lists the sha256 of every file.

With agent.harness = "claude_code" it also bundles Claude Code as bin/claude: a pinned release from downloads.claude.ai, checked against the release manifest and the pinned sha256. The binary is about 340 MB; the image grows to fit it.

Gopala checks the image (regular file, readable by the jailer uid, matching sha256) at startup (gopala_vm::verify_tools) and attaches it to every VM as a second drive with is_read_only: true; the guest mounts it read-only, nosuid,nodev at /opt/gopala, puts /opt/gopala/bin first on the harness’s PATH and in /etc/profile.d/gopala-tools.sh, resolves a bare agent.command from /opt/gopala/bin first, and refuses (status failed) to start a harness that resolves to a file the agent user could change. With the image in place, drop omp’s installer from agent.user_setup and git from agent.packages. A new image reaches each VM on its next boot; when jailed, vm.jailer.chroot_base must be on the image’s filesystem (hard link).

Install it so the service user (and the jailer uid) can read it but nobody else can change it, and paste the printed lines:

Terminal window
gopala --config /etc/gopala/gopala.toml image build --out /tmp/tools
install -o root -g root -m 0444 /tmp/tools/gopala-tools.ext4 /var/lib/gopala/images/gopala-tools.ext4
[vm]
tools_image = "/var/lib/gopala/images/gopala-tools.ext4"
tools_sha256 = "<the 64 hex digits image build printed>"
[agent]
packages = ["ca-certificates"] # git comes from the image
user_setup = [] # omp comes from the image

Run gopala check-config afterwards: it fails on a missing, unreadable or mismatching image. A rebuilt image needs a new tools_sha256 and a restart of gopala run. omp is always bundled; a gopala-owned harness is to replace it (issue #5).

Baking tools into the rootfs (image.setup)

Section titled “Baking tools into the rootfs (image.setup)”

[image] packages covers apt. Anything else goes into image.setup: shell commands run as root in the rootfs while gopala image build makes it (so the build needs sudo), after the packages, one after another. What they install is in every VM from its first boot, needs no network at boot, and sits outside the agent’s reach unless it has sudo. Each entry is passed to bash -c as it is; the build host’s http(s)_proxy and no_proxy are passed through if set. Like packages, a new setup reaches new conversations only (see Upgrading the guest above). None of these recipes has been run in a real image build yet.

uv, for Python tools and projects (the system prompt tells agents to prefer it):

[image]
packages = ["curl", "ca-certificates"]
setup = ["curl -LsSf https://astral.sh/uv/install.sh | env UV_UNMANAGED_INSTALL=/usr/local/bin sh"]

Node.js with pnpm, for Node projects. Node 25 and later no longer ship Corepack, so pnpm comes from its own installer; the distro’s Node is usually too old. Programs are linked into /usr/local/bin because the harness’s PATH is fixed, and pnpm add -g is pointed at the agent’s ~/.local/bin:

[image]
packages = ["curl", "ca-certificates"]
setup = [
"curl -fsSL https://nodejs.org/dist/v26.13.1/node-v26.13.1-linux-x64.tar.gz | tar -xz -C /usr/local --strip-components=1",
"curl -fsSL https://get.pnpm.io/install.sh | env PNPM_HOME=/usr/local/share/pnpm PNPM_VERSION=12 PNPM_NO_MODIFY_PATH=1 sh - && ln -sf /usr/local/share/pnpm/bin/* /usr/local/bin/",
]
[agent.env]
PNPM_CONFIG_GLOBAL_BIN_DIR = "/home/agent/.local/bin"

The templates page writes the same lines when you tick Node.js, with the version it pins.

An MCP server installed at build time and wired in as an operator server. With uv from the first recipe, mcp-server-fetch lands in /usr/local/bin, so the VM never downloads it (unlike uvx mcp-server-fetch); the server itself still needs a network mode that reaches the pages it fetches:

[image]
packages = ["curl", "ca-certificates"]
setup = [
"curl -LsSf https://astral.sh/uv/install.sh | env UV_UNMANAGED_INSTALL=/usr/local/bin sh",
"UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin UV_PYTHON_INSTALL_DIR=/opt/uv-python uv tool install mcp-server-fetch",
]
[[agent.mcp]]
name = "fetch"
command = "mcp-server-fetch" # from the image; runs in the VM as the agent user

The UV_* directories keep the tool and any Python uv downloads for it out of /root, which the agent can’t read.

An agent can add servers of its own at run time too (see architecture.md); baking a server in is for the ones every conversation should have, or for VMs without network.

vm.network decides how a VM reaches the internet. Model calls, the chat tools and HTTP MCP servers go over vsock in every mode. Details in architecture.md; the egress proxy’s threat model in security.md.

Mode Use it when Host setup
"proxy" Agents need package mirrors, git hosts or a few APIs, and nothing else None; list the hosts in vm.allowed_hosts
"none" Agents only talk and think; everything they need is in the image None
"nat" (default) Agents need arbitrary internet access IP forwarding, masquerading and filtering, below
[vm]
network = "proxy"
allowed_hosts = [
"archive.ubuntu.com", "security.ubuntu.com", # apt (Ubuntu image)
"pypi.org", "files.pythonhosted.org", # pip
"github.com", "*.githubusercontent.com", # git clone, raw files
"omp.sh", # the default user_setup installs omp from here
]

An entry is host, *.domain (any subdomain, not the domain itself), or either with :port; without a port 80 and 443 are allowed. The VM has no network device; HTTP_PROXY/HTTPS_PROXY (and lowercase) point at the host for the harness, provisioning, login shells and apt, so programs that honor them work and others fail. Each allowed or refused destination appears in the journal as a network notice (rate-limited), and the bytes are counted as egress on the dashboard. A name that resolves to any private, loopback or link-local address is refused even when listed. Changing network or allowed_hosts needs a restart.

The VM has no network device at all. agent.packages must be empty (bake packages in with [image] packages), and the default user_setup can’t download omp, so use the tools image.

For each running VM Gopala creates tap gpl<slot> whose host address is the first usable address of the slot’s /30 (172.30.0.0 + 4·slot + 1; the guest gets + 2) (setup_tap, nat_addrs), and boots the guest with a static ip= kernel parameter and gopala.dns= from vm.dns. Allow forwarding and masquerade the range out of your uplink once:

Terminal window
sysctl -w net.ipv4.ip_forward=1
nft add table ip gopala
nft add chain ip gopala post '{ type nat hook postrouting priority 100; }'
nft add rule ip gopala post ip saddr 172.30.0.0/16 oifname "eth0" masquerade

Gopala filters nothing in this mode. Recommended rules (adjust interface names):

Terminal window
nft add chain ip gopala fwd '{ type filter hook forward priority 0; }'
nft add rule ip gopala fwd ip saddr 172.30.0.0/16 ip daddr 172.30.0.0/16 drop # guest ↔ guest
nft add rule ip gopala fwd ip saddr 172.30.0.0/16 ip daddr 169.254.169.254 drop # metadata
nft add chain ip gopala in '{ type filter hook input priority 0; }'
nft add rule ip gopala in iifname "gpl*" ct state established,related accept
nft add rule ip gopala in iifname "gpl*" drop # guest → host services

See Firecracker’s network setup guide for background.

A [[git]] entry lets agents clone, push and open pull or merge requests on GitHub, GitLab or Gitea/Forgejo without the token ever entering the VM. The host’s git proxy holds the token and adds it to each request; the VM reaches it over vsock, so it works in every vm.network mode, including none.

Provider Token Scopes
GitHub (github.com or Enterprise Server) Settings → Developer settings → Fine-grained personal access token, limited to the repositories the agent works on Contents: read and write; Pull requests: read and write
GitLab (gitlab.com or self-managed) User or project/group access token api, or read_api + write_repository
Gitea / Forgejo Settings → Applications → Generate token Repository: read and write

A bot account with access to just those repositories keeps the blast radius small: other repository-level writes (settings, default branch, deleting the repository) are limited only by the token’s scopes.

[[git]]
name = "github" # unique, [a-z0-9-], up to 32 characters
provider = "github" # github | gitlab | gitea
# url = "https://github.example.com" # empty = github.com / gitlab.com; Gitea must set it
token_env = "GITHUB_TOKEN" # host environment variable holding the token
repos = ["acme/api", "acme/web"] # allowlist; [] = anything the token reaches
protected_branches = ["main", "master", "release"] # default ["main", "master"]
[[git]]
name = "lab"
provider = "gitlab"
token_env = "GITLAB_TOKEN"
repos = ["acme/infra"]
  • url is https (http only for loopback) and takes no credentials, query or fragment.
  • token_env is required. The token is read at startup, removed from the environment and zeroized; check-config fails if the variable is not set. Put it in /etc/gopala/env.
  • repos entries are provider paths (owner/repo, GitLab group/subgroup/project).
  • protected_branches: up to 32 exact branch names, no refs/ prefix, no patterns, matched case-sensitively (protect Main separately if it exists).
  • At most 8 entries. [[git]] is not hot-reloaded: restart gopala run after changing it.

It can clone, fetch, push branches and tags, delete branches that aren’t protected, and use the provider’s REST API: open pull/merge requests, comment, read issues and so on.

It can’t:

  • create, update, force-update or delete a protected branch, by push or by API; a push that touches one is refused whole;
  • commit through the API without naming a branch (file edits, cherry-picks and reverts on the default branch are refused);
  • merge a pull or merge request (always refused; a person merges);
  • use GraphQL, the gh or glab CLIs, or Git LFS (404);
  • send signed pushes or compressed push bodies;
  • with repos set, reach other repositories, name a repository by numeric id, or make API writes that name no repository.

At boot the guest agent writes /etc/gopala-git.conf and includes it from /etc/gitconfig. It rewrites https://<host>/, git@<host>: and ssh://git@<host>/ to http://127.0.0.1:<3300 + index>/, so ordinary clone URLs work. Only the first entry for a host gets the rewrite; a second account on the same host is reached through its own http://127.0.0.1:<port>/ URL.

The API base is in GOPALA_GIT_<NAME>_API (in the harness environment and /etc/profile.d/gopala-git.sh), e.g. GOPALA_GIT_GITHUB_API=http://127.0.0.1:3300/api (GitLab …/api/v4, Gitea …/api/v1). Agents call it with curl and no token:

Terminal window
curl -s -X POST "$GOPALA_GIT_GITHUB_API/repos/acme/api/pulls" \
-d '{"title":"Fix login","head":"fix-login","base":"main"}'

Only git’s smart HTTP transport is forwarded. Every push, branch deletion, tag, rejection, refusal and new pull/merge request shows up in the journal under the title git (for example pushed branch fix-login to acme/api, opened pull request #7 on acme/api: <url>, refused push to refs/heads/main on acme/api (protected branch)). Usage totals count git requests and bytes up and down.

Enable one or more platforms under [platforms]; each needs its own bot account. Credentials go in the environment (see Run as a systemd service), named by the *_env keys, never in the file. Gopala reads them once at startup and removes them from its environment.

Sign up once in the Lupyd app with the bot’s email and pick its username; that is the name people mention. Then:

[platforms.lupyd]
enabled = true
email_env = "LUPYD_EMAIL" # names of the variables, never the values
password_env = "LUPYD_PASSWORD"
device_name = "gopala" # how the account's device list shows the bot
# api_url / ws_url / auth0_*: production by default; point them at a local Firefly stack for testing
# profile_api_url = "https://api.lupyd.com" # where list_agents reads bios, with the bot's token

The first run signs in with the password and keeps a refresh token under <data_dir>/lupyd; later runs need neither variable.

Gopala uses Socket Mode, so the host needs no public URL.

  1. At api.slack.com/apps, Create New App → From a manifest, pick the workspace and paste Gopala’s manifest (Create an Agent → Slack → Copy the manifest or Download it, on the Gopala site). It turns on Socket Mode, the Messages tab for DMs, the bot scopes (channels:*, groups:*, im:*, mpim:* history and read/write, chat:write, reactions:read/write, usergroups:read, users:read) and the message, member, rename and reaction events Gopala handles. Change the name and display name if you like.
  2. Basic Information → App-Level Tokens → Generate, with the scope connections:write. That xapp-… token goes in SLACK_APP_TOKEN.
  3. Install to Workspace, then copy OAuth & Permissions → Bot User OAuth Token (xoxb-…) into SLACK_BOT_TOKEN.
  4. Invite the bot to channels (/invite @gopala); people can DM it from its Messages tab.
[platforms.slack]
enabled = true
bot_token_env = "SLACK_BOT_TOKEN" # xoxb-…
app_token_env = "SLACK_APP_TOKEN" # xapp-…, with connections:write

How Slack maps onto Gopala: every channel the bot is in (public, private or group DM) is one conversation with one VM; a DM is a conversation per person. Slack has no sub-channels, so channel_id is never used. Replies to a threaded message go into that thread, with every part of a long answer. Usernames for allowlists are Slack’s lowercase name handles. Gopala checks both tokens at startup and every ten minutes; a revoked token or an uninstalled app stops it with an error.

For alerts: role:<handle> resolves a Slack user group by its handle, which needs the usergroups:read scope. An app created from an older manifest lacks it; add the scope (or paste the new manifest) and reinstall the app, or roles are skipped with a warning in the log. Workspace admins can restrict who may use @channel and @here; a restricted bot’s alerts won’t reach everyone.

  1. In the Developer Portal, New Application, then Bot: reset and copy the token into DISCORD_BOT_TOKEN.
  2. Under Bot → Privileged Gateway Intents, turn on Message Content Intent (required). Turn on Server Members Intent too if you want member lists in conversation_info and join/leave events; without it Gopala knows only the bot and people it has seen. Gopala detects which intents are on.
  3. Start Gopala: the log prints the invite link, which is https://discord.com/oauth2/authorize?client_id=<application id>&scope=bot&permissions=274878106688 (View Channels, Send Messages, Send Messages in Threads, Read Message History, Add Reactions, and Mention @everyone, @here and All Roles so [alerts] can reach people). Open it to add the bot to a server.
[platforms.discord]
enabled = true
token_env = "DISCORD_BOT_TOKEN"

How Discord maps onto Gopala: a server is one conversation with one VM, and its text channels and threads are the conversation’s channels; a DM is a conversation per person. Long answers are split at Discord’s 2000 characters. Ordinary messages can mention people but never @everyone, @here or roles; only alerts can, and only with the “Mention @everyone, @here and All Roles” permission. A bot invited with the older link (permissions 274877975616) lacks it: re-invite it with the new link, or grant it through one of the bot’s roles. user:<name> alerts find people in the cache or by a member search, which works best with the Server Members intent. One gateway shard serves up to 2500 servers.

Configure in a browser, or write the file by hand:

Terminal window
gopala --config /etc/gopala/gopala.toml check-config
Terminal window
# Browser: site + config API only, no sign-in, no VMs (a missing/invalid file still serves).
GOPALA_ADMIN_TOKEN="$(openssl rand -base64 32)" gopala --config /etc/gopala/gopala.toml web
# open server.public_url (default http://127.0.0.1:60901) → Configure

check-config parses and validates the file, requires every [[llm]].secret_env and every agent.mcp secret_env to be set, and checks that vm.kernel and vm.rootfs exist and that vm.tools_image verifies. The config path can also come from GOPALA_CONFIG (default gopala.toml). Every key and its default is in the configuration editor at /configure/ and the JSON schema at /api/config/schema (signed in); the map from keys to components is in architecture.md.

The configuration editor at /configure/ builds its form from that schema, validates and normalizes drafts through the server’s own parser (/api/config/validate, /api/config/normalize), and loads and saves the file (GET/PUT /api/config; the previous file is kept as <file>.bak). All of these need you signed in. Under gopala run, saving applies behavior, approvals, agent, budget, alerts, monitoring, collaboration and the VM limits immediately and tells you which other sections need a restart; under gopala web everything takes effect at the next gopala run. Signed out, the editor and /templates/ fall back to the schema and defaults bundled with the site: they still build and download a file, but can’t check it against your server. See protocols.md for the API and security.md before enabling the token.

The “Create your bot” flow at /templates/ starts a whole file from a goal (help a team, research, write and review code, analyse data, write docs), then asks for the chat app, the model provider and, optionally, languages, code hosting accounts ([[git]]), network, VM size and daily budget; every other key it covers (harness and tools, MCP servers, the image, limits, host budgets) sits under “Expert settings”. The code goal uses Claude Code, full internet and a GitHub account by default; with a provider other than Anthropic it switches to omp. Its “OpenAI-compatible” provider writes an [[llm]] entry named compatible on guest port 8703 whose upstream is the base URL you type, and sets OPENAI_BASE_URL = "http://127.0.0.1:8703" without /v1: the host proxy keeps the upstream’s own path, so a base URL that already ends in /v1 stays correct. Choosing “No internet” moves agent.packages into image.packages, since nothing installs on first boot without a network.

Lupyd, Slack and Discord: see Creating the bot.

DM-only assistant for a few people

[behavior]
respond_to = "dms" # groups are joined but stay quiet (no commands either)
direct_messages = "allowlist"
invites = "nobody" # leave every group right away
allowlist = ["alice", "bob"]

Allowlisted team bot

[behavior]
respond_to = "mentions" # @bot or replies to the bot in groups, all DMs
direct_messages = "allowlist"
invites = "allowlist" # stays only in groups owned by these people
allowlist = ["alice", "bob", "carol"]
max_messages_per_minute = 10
[approvals]
approvers = "allowlist" # only these people may approve, /stop, or grant resources
timeout_secs = 600

Offline agent (no internet in the VM; models still reachable through the host proxy)

[vm]
network = "none"
tools_image = "/var/lib/gopala/images/gopala-tools.ext4" # omp and git from the host
tools_sha256 = "…"
[agent]
packages = [] # required with network = "none" (validation)
user_setup = [] # the default installs omp from the internet
install_packages = false
# Anything else the agent needs goes into the image: [image] packages and tools.

Agent limited to a few tools and extra MCP servers

[agent]
tools = ["read", "grep", "edit", "write", "bash"] # omp's built-ins; [] keeps its defaults
[[agent.mcp]] # runs in the VM as the agent user: no secrets in args or env
name = "fetch"
command = "uvx"
args = ["mcp-server-fetch"]
[[agent.mcp]] # proxied by the host; the VM never sees the token
name = "tracker"
url = "https://mcp.example.com/mcp"
secret_env = "TRACKER_TOKEN" # sent as "authorization: Bearer <token>"
# auth_header = "x-api-key" # for servers that want another header
# auth_prefix = ""

An HTTP server works in every network mode. A stdio server that downloads itself (uvx, npx) needs nat, or proxy with its registry allowed. A new HTTP server with a secret_env added while running waits for a restart (its secret is read at startup); without one it works at the VM’s next boot.

Token budgets

[budget]
daily_tokens = 20000000 # the whole host, per UTC day
monthly_tokens = 400000000
low_stops_at_percent = 70 # low-priority conversations stop at 70% of the host budget
normal_stops_at_percent = 90 # normal ones at 90%; high ones may use all of it
[budget.default]
daily_tokens = 2000000 # each conversation
priority = "normal"
[[budget.conversations]]
key = "slack-g-1f0c3a9e2b7d4c58" # keys are on the dashboard
daily_tokens = 10000000
priority = "high" # also boots first when slots or memory run short

Tokens are input + output + cache-write, counted by the host’s LLM proxy. A conversation over budget gets 429 budget_exceeded for every model call until its day or month rolls over (00:00 UTC), with one budget notice in its journal. The dashboard shows usage against each budget and edits one conversation’s (PUT /api/conversations/{key}/budget), keeping your file’s comments.

Several model providers, including an OpenAI-compatible one

[[llm]]
name = "anthropic"
upstream = "https://api.anthropic.com"
guest_port = 8701
auth_header = "x-api-key"
secret_env = "ANTHROPIC_API_KEY"
guest_env = { ANTHROPIC_BASE_URL = "http://127.0.0.1:8701", ANTHROPIC_API_KEY = "via-gopala" }
[[llm]]
name = "openai"
upstream = "https://api.openai.com" # or any OpenAI-compatible server
guest_port = 8702
# auth_header defaults to "authorization"; OpenAI-style APIs want a Bearer prefix:
auth_prefix = "Bearer "
secret_env = "OPENAI_API_KEY"
guest_env = { OPENAI_BASE_URL = "http://127.0.0.1:8702/v1", OPENAI_API_KEY = "via-gopala" }

Each entry gets vsock port 1100 + index; guest_ports must be unique; at most 64 entries. The placeholder keys only satisfy client libraries: the proxy removes them and adds the real one. How the harness picks a provider and model is up to omp (see omp environment variables; its *_BASE_URL variables are fallbacks that a configured provider URL overrides), e.g. [agent.env] PI_SMOL_MODEL = "anthropic/claude-haiku-4-5".

Serving behind a reverse proxy (site, config API and /watch on one origin)

[server]
bind = "127.0.0.1:60901"
public_url = "https://agents.example.com" # watch links become wss://agents.example.com/watch#…
site_dir = "/usr/share/gopala/site"

Any websocket-capable TLS proxy works, e.g. Caddy: agents.example.com { reverse_proxy 127.0.0.1:60901 }. To terminate TLS in Gopala instead, set server.tls_cert/tls_key (PEM) and a public bind. A non-loopback bind without TLS is refused unless allow_plaintext = true (only for a TLS proxy on a private network). [rooms] now holds only the watch settings (enabled, encryption, key_rotation_secs, max_viewers_per_room, journal_max_mib, token_ttl_secs), and rooms.enabled requires server.enabled.

Every conversation starts with vm.vcpus / vm.memory_mib / vm.disk_gib. Its agent can ask for more with request_resources, up to the per-VM maximums, and gets it at its next restart_vm (see architecture.md).

Small shared host, people decide (8 GiB RAM, many light conversations)

[vm]
vcpus = 1
memory_mib = 1024
disk_gib = 8
max_vcpus = 2
max_memory_mib = 3072
max_disk_gib = 20
max_vms = 4
memory_budget_mib = 5120 # leave ~3 GiB for the host and Firecracker overhead
min_free_disk_gib = 20
grants = "approval" # approvers answer "My VM needs more room: …" in the chat

Build box for a trusted team, agents decide

[vm]
vcpus = 4
memory_mib = 4096
disk_gib = 30
max_vcpus = 16
max_memory_mib = 32768
max_disk_gib = 200
max_vms = 6
memory_budget_mib = 98304
min_free_disk_gib = 50
grants = "auto"
[agent]
disk_alert_percent = 80 # nudge earlier on big builds
usage_interval_secs = 15

Fixed sizes (agents can’t grow; they may still give resources back and sleep)

[vm]
grants = "never"

Rules of thumb:

  • The memory budget, not max_vms, is what usually limits concurrency: a VM waits to boot until its memory fits (/status: waiting for host memory). With memory_budget_mib = 0 it is 80% of MemTotal; set it explicitly on small hosts or hosts running other services.
  • max_memory_mib above the budget is effectively capped at the budget (vm_resources reports the lower value).
  • Disk images are sparse: grants check min_free_disk_gib on the filesystem holding data_dir, but writes inside a VM consume host space later. Plan for max_disk_gib × active conversations, or watch free space.
  • Lowering a maximum later clamps existing grants at their next boot; disks never shrink.
  • Alerts (disk_alert_percent, memory_alert_percent, 10–99) only steer the agent to clean up; usage_interval_secs (≥ 5) sets how fresh /status and vm_resources are.

/status (mention the bot first in groups) answers within 3 s with the session’s state and the VM’s allocation, plus the latest usage while it runs:

Answer Meaning
asleep · 2 vCPU, 2048 MiB, 10 GiB No VM runs; the next message boots one with this allocation (including grants not yet applied)
waiting for a free VM slot vm.max_vms VMs are running
waiting for host memory The allocation doesn’t fit in the memory budget right now
working · 2 vCPU, 2048 MiB, 10 GiB · memory 75% · disk 90% The agent is busy (or provisioning); usage from the last report
idle · … The VM runs, the agent waits for messages; it stops after vm.idle_shutdown_secs
restarting after an error: … The last boot failed; the session retries with backoff
starting up The session didn’t answer within 3 s, typically because it is booting a VM (preparing the disk and network, waiting for the guest’s hello), when it doesn’t read commands (issue #12; the dashboard shows booting)

agent.harness sets the program every conversation of this bot runs; one Gopala serves one bot config, so pick per bot. The comparison is in architecture.md. In short: omp (the default) is a general agent whose tool list shapes it for coding, research or data work; Claude Code is a coding agent on Anthropic models; exec runs a one-shot CLI such as codex exec per message, without a session between messages. Whatever you pick, install it from the host (the tools image or image.setup), never with agent.user_setup: the agent could replace a harness it installed itself, and with the tools image the guest refuses to start one it could change.

omp (default):

[agent]
harness = "omp"
command = "omp" # bin/omp on the tools drive
args = ["--approval-mode", "yolo"]
tools = [] # omp's names, e.g. ["read", "grep", "edit", "write", "bash"]
user_setup = [] # with the tools image; the default installs omp from the internet

Models come from the [[llm]] entries’ guest_env (see Several model providers).

Claude Code with Anthropic through the host proxy:

[agent]
harness = "claude_code"
command = "claude" # bin/claude on the tools drive ("" means the same)
args = ["--model", "sonnet"] # replaces omp's default args, which Claude Code doesn't take
tools = [] # Claude Code's names, e.g. ["Bash", "Read", "Edit", "Write", "Grep", "Glob"]
user_setup = [] # don't install omp
[[llm]]
name = "anthropic"
upstream = "https://api.anthropic.com"
guest_port = 8701
auth_header = "x-api-key"
secret_env = "ANTHROPIC_API_KEY"
guest_env = { ANTHROPIC_BASE_URL = "http://127.0.0.1:8701", ANTHROPIC_API_KEY = "via-gopala" }

gopala image build puts the pinned Claude Code on the tools drive when agent.harness = "claude_code"; without a tools image, install it in the rootfs instead ([image] packages = ["nodejs", "npm"], setup = ["npm install -g @anthropic-ai/claude-code"]; not tried with Ubuntu’s apt Node yet). The placeholder ANTHROPIC_API_KEY is there because claude -p may refuse to start without a key; the proxy removes it and adds the real one. That hasn’t been tested: if Claude Code reports missing credentials, check that line first. Claude Code runs with bypassPermissions, so approvals for critical steps come only from the request_approval chat tool; there’s no stricter mode to set in agent.args.

exec with OpenAI’s Codex CLI, one run per message:

[agent]
harness = "exec"
command = "codex"
args = ["exec", "--skip-git-repo-check", "--sandbox", "danger-full-access", "-"]
tools = [] # exec ignores it
user_setup = [
# Codex reads MCP servers from its own config, not .mcp.json: give it the chat tools.
"mkdir -p ~/.codex && printf '[mcp_servers.chat]\\ncommand = \"/usr/local/bin/gopala-guest\"\\nargs = [\"mcp\", \"--port\", \"1025\"]\\n' > ~/.codex/config.toml",
]
[image]
packages = ["nodejs", "npm"]
setup = ["npm install -g @openai/codex"]
[[llm]]
name = "openai"
upstream = "https://api.openai.com"
guest_port = 8702
auth_prefix = "Bearer "
secret_env = "OPENAI_API_KEY"
guest_env = { OPENAI_BASE_URL = "http://127.0.0.1:8702/v1", OPENAI_API_KEY = "via-gopala" }

The guest pipes the system prompt, a blank line and the message into codex exec -; Codex’s stdout lines show up as text in the journal, and only what it sends with the chat tools reaches people, which is why it needs the chat server in its own config. danger-full-access turns off Codex’s own sandbox, since the VM is the sandbox. Each message starts from scratch, and restart_harness has nothing to restart. None of this exec setup has been run against a real Codex release; check the flags and the config format against the version you install.

Several agents can share a group: another Gopala with its own bot account, or any other bot. With [collaboration] on (the default) they find each other with the list_agents tool, ping each other with @mentions and split the work; each decides for itself what to do with a ping, and the agent the person asked owns the answer. How it works is in architecture.md.

[collaboration]
enabled = true
agent_role = "agent" # Lupyd group role / Discord role / Slack user group handle
max_agent_messages_per_hour = 30 # wake-ups by other agents per conversation; people don't count

An agent wakes another only by mentioning it or replying to it, never through listening mode, and a ping can’t run /stop or other commands or answer approvals. Past the hourly cap, pings are dropped with a line in the log. The agents need to tell each other apart, so set each platform up so that every bot is recognised and says what it’s good at:

  • Lupyd has no bot accounts, so the role is the only signal. In each group, create a group role named like agent_role (agent, compared ignoring case) and give it to every bot, including this one. Then write each bot’s profile bio in the Lupyd app: one or two sentences on what it does (“Researches papers and datasets, cites sources”). That bio is what other agents read in list_agents; Gopala fetches it from platforms.lupyd.profile_api_url with the bot’s own token, so leave that at the Lupyd API.
  • Discord recognises bots by themselves. Give the agent role only to non-bot accounts that should count as agents. Turn on the Server Members Intent for a complete list; without it Gopala knows only members it has seen, mostly people who have posted. Discord doesn’t let one bot read another’s description, so about stays empty: descriptive role names and nicknames are what the other agents see.
  • Slack recognises bot users by themselves. For other accounts that should count, create a user group whose handle is agent_role (@agent) and add them. about comes from the profile’s title (and status), so set a title that says what the account does. The manifest’s users:read and usergroups:read scopes cover this; apps installed before usergroups:read was added must be reinstalled.

[collaboration] applies without a restart: enabled and the cap with the next message, a new agent_role in list_agents within 5 minutes, and the system prompt’s collaboration section at each VM’s next boot. Which incoming messages count as agents’ follows a changed agent_role only after you restart gopala run. In the journal, a ping shows as a prompt starting name (agent) …. To turn it off for this bot, set enabled = false: other agents’ messages are then ignored everywhere and list_agents answers “collaboration is off on this host”.

An agent marks a message warning or critical when people need to notice it; [alerts] says who that notifies. The agent never names people, so prompt injection can’t make it page someone you didn’t list. Defaults: warnings notify nobody, critical messages notify everyone, at most 4 notifying messages per conversation per hour.

[alerts]
warning = ["role:on-call"] # Slack user group handle / Discord role name
critical = ["here", "user:alice"] # "everyone", "here", "role:<name>", "user:<username>"
host_events = "critical" # a VM that keeps failing, a budget block
max_per_hour = 4 # later ones still go out, notifying nobody
[[alerts.conversations]] # one group with its own audience
key = "slack-g-1f2e3d4c5b6a7980" # keys are on the dashboard
critical = ["role:payments-oncall"]

What each entry becomes:

Entry Slack Discord Lupyd
everyone @channel @everyone @everyone as text (no native mention)
here @here @here @here as text
role:<name> the user group with that handle (needs usergroups:read) the role with that name skipped: Lupyd has no role mentions
user:<name> the person with that Slack handle the server member with that username @name, if they’re in the group

Direct chats never notify; the person gets the message anyway. Names that don’t resolve are skipped with a warning in the log (slack: no user group @… to notify, discord: no role …, lupyd: … not in this conversation) and the message still goes out. When the hourly cap is used up, the message goes out without mentions and the agent’s tool result says sent without notifying anyone: at most N notifying messages an hour here. In the journal, serious messages show as message (warning) or message (critical). [alerts] applies without a restart.

Platform requirements: Slack apps need usergroups:read for roles (reinstall older apps), and admins may restrict @channel/@here; Discord bots need “Mention @everyone, @here and All Roles” (re-invite bots invited before this permission was added, or grant it through a role). See Creating the bot.

A group normally wakes the agent only with a mention or a reply (respond_to = "mentions"). In listening mode (respond_to = "all" for that one group) every message reaches the agent. Switch it:

  • in the chat, as an approver: @bot /listen all (or on) and @bot /listen mentions (or off); the bot confirms with 👂 or 🔕;

  • on the dashboard, with a conversation’s Listening toggle;

  • in the file:

    [[behavior.conversations]]
    key = "slack-g-1f2e3d4c5b6a7980"
    respond_to = "all"

The chat command and the toggle write that entry into gopala.toml (keeping your comments, with a .bak) and apply it to the next message, so the gopala.toml directory must be writable (see ReadWritePaths in the unit below). If saving fails, the bot answers “Couldn’t save that setting; an operator needs to look at the configuration.” and nothing changes. Listening mode costs more tokens: every message in the group wakes the agent, so pair it with a budget.

With [memory] enabled = true (the default) every agent gets a memory MCP server and keeps facts across restarts in /home/agent/.gopala/memory.db on its own disk. Nothing to set up on the host; keywords (FTS5) work without any model. Two optional extras use an [[llm]] entry, through the host proxy, so their tokens show on the dashboard and count against budgets:

[memory]
max_memories = 5000 # active memories per conversation
max_db_mib = 64
half_life_days = 30
archive_days = 30
duplicate_threshold = 0.85
cleanup_interval_secs = 3600 # at least 60
[memory.embeddings] # recall by meaning as well as words
llm = "openai" # an [[llm]] name; must speak OpenAI's /v1/embeddings
model = "text-embedding-3-small"
# dimensions = 512 # 0 = the model's default
[memory.compaction] # a model merges and rewrites related memories
llm = "anthropic"
model = "claude-haiku-4-5"
format = "anthropic" # or "openai" for /v1/chat/completions
every_hours = 24
min_memories = 200
max_clusters = 20

The [[llm]] entry’s upstream should be the API’s base without /v1 (as in the examples above), since memory adds /v1/... itself [inference from the paths it posts].

Sizing, per conversation:

  • Disk: the store stays under max_db_mib (64 MiB) inside the conversation’s disk; leave that much room in vm.disk_gib. 5000 memories of a few hundred bytes each, with 1536-dimension embeddings (6 KiB each), come to about 30 MiB [inference].
  • Embedding tokens: one call per remember, update with new text, or recall, sized like the text; the maintenance pass embeds up to 256 missing memories per run.
  • Compaction tokens: at most max_clusters calls per run, each about 3,000 input tokens (12,000 characters of memories plus instructions) and up to 4,096 output tokens, so roughly 140,000 tokens per conversation per run at the defaults, once a day when there are 200 or more memories.

Changing [memory] needs a restart of gopala run; each VM picks it up at its next boot. Turning memory off removes the server and the system prompt’s line about it; the file stays on the disk. An [[agent.mcp]] entry named memory is refused while memory is on.

People in a conversation can follow its agent outside the chat: with the link the bot sends when someone writes /watch, or with any app that speaks the /monitor websocket (or its HTTP mirror under /api/monitor/). The Lupyd app connects with the person’s own Lupyd sign-in, so Lupyd users get every task, tool call and subagent in the app; Slack and Discord users keep the chat messages and watch links. Every request is authenticated, and a person only ever sees conversations their platform says they’re in. [monitoring] decides, on top of membership, who may read the agent’s output (the journal and live events) and its VM’s metrics (CPU, memory, disk, network, token usage, budget):

[monitoring]
output = ["members"] # default: anyone in the conversation
metrics = ["approvers"] # default: whoever may answer approvals ([approvals] approvers)
lupyd_tokens = true # accept Lupyd sign-in tokens on /monitor
membership_cache_secs = 60 # how long a platform's answer is reused (at most 3600)

Rules, any of which grants: members, approvers, owner, role:<name>, permission:<name>, user:<username>. An empty list leaves it to the operator. Roles, permissions and owners come from each platform:

Platform role: permission: owner
Lupyd the member’s role in the group the group role’s permissions: see_message, pin_message, add_message, manage_channel, manage_role, manage_member, manage_group the group’s owner
Slack handles of the user groups they’re in (needs usergroups:read) their workspace standing: admin, owner, primary_owner the channel’s creator
Discord their role names in the server server-level permissions as lowercase names (manage_guild, manage_messages, administrator, …); administrators and the owner have all of them the server’s owner

In a direct chat the other person is the only member and counts as its owner.

Three alternatives, one per platform:

# Slack: everyone in the channel sees the agent's work; workspace admins and the
# @oncall user group see metrics.
[monitoring]
output = ["members"]
metrics = ["permission:admin", "role:oncall"]
# Discord: people with Manage Server see everything; the rest nothing.
[monitoring]
output = ["permission:manage_guild"]
metrics = ["permission:manage_guild"]
# Lupyd: group managers see metrics; one sensitive group shows output to its owner only.
[monitoring]
metrics = ["permission:manage_group", "owner"]
[[monitoring.conversations]]
key = "lupyd-g-42"
output = ["owner"]
metrics = ["owner"]

Platform notes: Slack’s member list is reused for 30 s (or until someone joins or leaves) and user groups for 5 minutes. Discord learns about members joining and leaving only with the Server Members intent; without it Gopala searches the member list when an answer expires, so a removed member keeps access for up to membership_cache_secs.

Connecting an app. Open wss://<public_url host>/monitor and send a protobuf MonitorClientMessage with an authenticate request first (or put Authorization: Bearer <token> on the upgrade):

  • the Lupyd app: the person’s Lupyd access token (lupyd_token);
  • a script for one conversation: the person’s /watch token, the gpl1.… part after # in the link (watch_token);
  • operator tooling: the admin token (admin_token).

Then list, status, metrics, usage, events, or subscribe to stream events and metrics. The messages and limits are in protocols.md. For a quick look without a websocket client:

Terminal window
curl -H "Authorization: Bearer $GOPALA_ADMIN_TOKEN" https://agents.example.com/api/monitor/conversations
curl -H "Authorization: Bearer gpl1.…" "https://agents.example.com/api/monitor/conversations/lupyd-g-42/events?limit=50"

[monitoring] changes apply without a restart; a cached membership answer lasts up to membership_cache_secs.

gopala run serves a control panel at /dashboard/ under server.public_url, next to the configuration editor (/configure/), the docs (/docs/) and the front page (/). It needs the admin token:

Terminal window
# In /etc/gopala/env (see below), then restart:
GOPALA_ADMIN_TOKEN=<openssl rand -hex 32>

Open the URL and sign in with the token. The browser gets an HttpOnly session cookie valid for 12 hours (a restart of Gopala signs everyone out); “Sign out” ends it. More than ten wrong tokens within a minute lock your address out for a minute (429, even with the right token afterwards). Behind a reverse proxy all clients share that lockout. Scripts can use Authorization: Bearer <token> with the same API instead (protocols.md).

What it shows, live (server-sent events, no refresh needed):

  • Host: how much of the memory budget running VMs hold, VM slots in use, free disk under data_dir, the base and maximum allocation and grant mode, and today’s totals across all conversations (tokens, model requests and errors, CPU time, tool calls, messages, LLM traffic).
  • Conversations: one row per conversation the bot knows, filterable by platform and status and sortable by activity, tokens, CPU or memory: status (with the error when failed), CPU, memory and disk (measured), tokens and calls today, last activity; “resize at next boot” marks a pending allocation.
  • A conversation’s detail: its activity (the journal, filterable: messages, tools and approvals, model calls, lifecycle and notices, errors), model calls (one row per llm event), resources, and usage (today and total).
  • Model budget: the host’s tokens today and this month against [budget], with marks where low- and normal-priority conversations stop; per conversation, its priority, limits, usage and the reason when its model calls are blocked.
  • Usage over time: per day for 7, 30 or 90 days, per metric and conversation.
  • Live activity: every journaled event since the page opened.

Measured vs reported. The “Measured by the host” figures come from outside the VM: CPU and resident memory of the firecracker process, the disk image’s real size on the host, and the tap’s network bytes, sampled every 5 seconds; tokens come from the model provider’s own responses through the LLM proxy. “Reported by the VM” is what the guest says about itself (memory and disk percent, load, report age): useful, but unverified, since a compromised agent can lie. The dashboard warns when the two differ by more than 20 points. Expect some honest difference too: host RSS includes guest page cache and Firecracker’s overhead, and the disk image’s allocated size includes deleted-but-not-trimmed blocks [inference].

Actions (on a conversation): Abort stops the agent’s current turn and drops queued messages; Sleep stops the VM now; Restart restarts it (or boots it if asleep) and applies a pending allocation; Leave makes the bot leave the group or chat and ends the session (not undoable from the dashboard); Set allocation sets vCPUs, memory and disk for the next boot, bypassing vm.grants and approvals but not the per-VM maximums, the memory budget, the disk reserve or “disks only grow”; Budget sets the conversation’s daily and monthly tokens and its priority, saved to [[budget.conversations]] and applied at once; Listening switches the group’s listening mode, saved to [[behavior.conversations]]. Every action except Budget is journaled as an operator notice.

The usage ledger is <data_dir>/usage.db, flushed every 30 seconds and at shutdown (a crash loses up to 30 seconds of counts), with daily rows kept for 400 days and all-time totals forever.

Secrets in a root-only environment file, /etc/gopala/env (mode 0600):

Terminal window
LUPYD_EMAIL=bot@example.com
LUPYD_PASSWORD=… # only needed until the first sign-in; remove it afterwards
SLACK_BOT_TOKEN=xoxb-… # [platforms.slack]
SLACK_APP_TOKEN=xapp-…
DISCORD_BOT_TOKEN=… # [platforms.discord]
ANTHROPIC_API_KEY=…
# TRACKER_TOKEN=… # an [[agent.mcp]] secret_env
# GOPALA_ADMIN_TOKEN=… # only if you use the dashboard or the browser configuration

Without the jailer (dedicated user with KVM access and CAP_NET_ADMIN for tap devices):

/etc/systemd/system/gopala.service
[Unit]
Description=gopala: autonomous agents for chat platforms in Firecracker microVMs
Wants=network-online.target
After=network-online.target
[Service]
User=gopala
Group=gopala
SupplementaryGroups=kvm
EnvironmentFile=/etc/gopala/env
Environment=RUST_LOG=info,firefly_client=warn,firefly_core=warn
ExecStart=/usr/local/bin/gopala --config /etc/gopala/gopala.toml run
# Tap devices (vm.network = "nat"; drop these two lines otherwise); ambient capabilities are inherited by `ip` and firecracker.
AmbientCapabilities=CAP_NET_ADMIN
CapabilityBoundingSet=CAP_NET_ADMIN
NoNewPrivileges=yes
StateDirectory=gopala
StateDirectoryMode=0700
UMask=0077
ProtectSystem=strict
ProtectHome=yes
# /etc/gopala only if PUT /api/config should be able to save there (temp file + .bak in that directory).
ReadWritePaths=/var/lib/gopala /etc/gopala
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target

With [vm.jailer], Gopala must start as root (the jailer needs it to chroot, create device nodes, join cgroups and drop to vm.jailer.uid/gid): drop User=, Group=, AmbientCapabilities=, CapabilityBoundingSet= and NoNewPrivileges=, and add the jailer’s chroot_base to ReadWritePaths=. chroot_base must be on the same filesystem as data_dir and vm.kernel.

Terminal window
systemctl daemon-reload
systemctl enable --now gopala
journalctl -u gopala -f

The daemon stops on SIGINT/SIGTERM (closes every session, waits 2 s, shuts the providers down) and exits with an error when a provider reports a fatal error (Lupyd signed it out, a Slack token was revoked, Discord refused the intents), so Restart=on-failure brings it back (sign-in then needs the Lupyd password again if the refresh token was revoked). It refuses to start when no platform is enabled. Only one process may use a Lupyd device directory (firefly-mcp.lock).

Source What you get
Daemon log (stderr, env_logger) Default filter info,firefly_client=warn,firefly_core=warn; override with RUST_LOG. Provider sign-in (lupyd: signed in as …), joins/leaves, policy decisions, guest <version> connected, reconnects, VM boots and exits, agent VM failed: …, proxy errors, web server errors.
<data_dir>/vms/<key>/console.log Firecracker’s stdout/stderr including the guest serial console (kernel console=ttyS0 … quiet), truncated on each boot, capped at vm.console_log_max_kib (default 4 MiB). The first place to look when a VM doesn’t connect.
<data_dir>/vms/<key>/allocation.json The conversation’s granted allocation (absent while it has none).
gopala watch The session journal of one conversation: lifecycle (booting with the allocation, provisioning, ready, busy, idle, restarting, stopped, crashed), provisioning output, harness stderr, prompts (other agents’ marked (agent)), text, thinking, tool calls and results, subagents, approvals, outbound messages, resources notices (alerts, grants, denials, restarts, sleeps) and harness notices (the agent asked for a harness restart).
/status in the chat See Reading /status.
GET /api/status {"mode", "admin", "signed_in"}: run or web, whether the admin API is on, and whether this request is signed in; signed in, also "version".
GET /api/monitor/conversations What a given credential may see (Bearer admin, watch or Lupyd token); see Monitoring.
The dashboard Live state, measured usage, journals and model calls per conversation; see The dashboard.
<data_dir>/usage.db The usage ledger (SQLite): usage_days and usage_totals, one row per conversation key (and day).
Inside a VM journalctl -u gopala-guest (guest log, RUST_LOG=info) if you can get a shell, e.g. by mounting a stopped VM’s disk.

gopala watch takes the link from its argument, GOPALA_WATCH_LINK (preferred: removed from the environment after reading) or stdin, and prints one line per event:

#<seq> <HH:MM:SS> [<actor>] <kind>[ <title>][ !]: <text>

--since <seq> replays only events after seq; --live skips history.

Symptom / message Cause Fix
enable at least one platform under [platforms] / no platform is enabled (see [platforms] in the configuration) Every [platforms.*] has enabled = false Enable one
alerts entry "…": use "everyone", "here", "role:<name>" or "user:<username>" A Slack or Discord mention syntax (@channel, <@U…>) or a typo Use the neutral form
Log: slack: no user group @… to notify / slack: can't notify @… / discord: no role … ; not mentioning it / lupyd: no way to mention the role … The name doesn’t exist there, the Slack app lacks usergroups:read, or Lupyd has no roles Fix the name; reinstall the Slack app with the new manifest; use user: on Lupyd
Log: discord: the bot lacks "Mention @everyone, @here, and All Roles" … Bot invited before the permission was added Re-invite it with the new link or grant the permission through a role
Agent tool result sent without notifying anyone: at most N notifying messages an hour here alerts.max_per_hour used up Expected; raise it if the alerts were real
Chat: Only approvers can change how I listen here. /listen from someone outside approvals.approvers Ask an approver, or use the dashboard
Chat: Couldn't save that setting; … after /listen gopala.toml or its directory isn’t writable, or the edited file didn’t validate (daemon log: /listen: …) Fix permissions (ReadWritePaths) or the file
behavior.conversations key "…" must be a conversation key … / alerts.conversations key … Not a key (copy keys from the dashboard) or listed twice Fix the entry
memory.embeddings needs a model and llm = the name of an [[llm]] entry … / memory.compaction needs … The named [[llm]] entry doesn’t exist, or no model Name an existing [[llm]] entry
agent.mcp name "memory" is the built-in memory server's … Your own MCP server named memory Rename it, or set memory.enabled = false
git push: ! [remote rejected] main -> main (protected by the Gopala host) and remote: Gopala: … The push touched a branch in protected_branches (any create, update or delete refuses the whole push) Expected; the agent pushes another branch and opens a pull/merge request
git push refused: signed push or compressed push body The host’s git proxy can’t check protected branches in those Push unsigned, without push.gpgSign; plain git sends uncompressed pushes
Git or API call answers 404 Git LFS, or a git transport other than smart HTTP (dumb HTTP, other paths) Unsupported; keep large files out of LFS for agent repos
API call refused with a hint to use the path (/projects/<id>, /repositories/<id>) repos is set and the call names the repository by numeric id Use the URL-encoded owner/repo path (GitLab /projects/owner%2Frepo)
git <name>: $GITHUB_TOKEN is not set The [[git]] entry’s token_env variable is missing from the environment Add it to /etc/gopala/env (or export it) and restart
Merging a pull/merge request refused The host refuses every merge call, by design A person merges
A second account on the same host gets the first account’s repos or permissions Only the first [[git]] entry per host gets the URL rewrite Clone the second through its own http://127.0.0.1:<port>/ URL (3300 + its index)
Slack: $SLACK_BOT_TOKEN is not set / SLACK_BOT_TOKEN must hold the app's Bot User OAuth Token (xoxb-…) Missing or wrong token Use the xoxb- token from OAuth & Permissions
SLACK_APP_TOKEN must hold an app-level token (xapp-…) with the connections:write scope / Slack rejected SLACK_APP_TOKEN (Socket Mode needs …) No app-level token, a missing scope, or Socket Mode off Generate an app-level token with connections:write; create the app from the manifest
Slack refused gopala's token (… from …); the token was revoked or the app removed and the daemon exits invalid_auth, account_inactive, token_revoked, or the app was uninstalled Reinstall the app and update the tokens
Slack: the bot never answers in a channel Not a member, or respond_to = "mentions" and no mention /invite @gopala; mention it
Discord rejected the bot token in DISCORD_BOT_TOKEN … Wrong or reset token Reset it in the Developer Portal and update the variable
Discord: the bot can't read messages: turn on "Message Content Intent" … / Discord refused the bot's gateway intents: … Message Content Intent is off Turn it on (Bot → Privileged Gateway Intents) and restart
Discord requires sharding for this bot (over 2500 servers) … Gopala runs one shard Not supported
Discord: conversation_info lists only the bot and people who wrote Server Members Intent is off Turn it on and restart (optional)
vm.network = "proxy" needs vm.allowed_hosts (or use "none") Proxy mode with an empty list List the hosts
agent.packages need vm.network = "nat" or "proxy" … Packages with network = "none" Bake them in with [image] packages, or change the mode
vm.allowed_hosts entry "…": expected host, *.domain, optionally with :port URL, path or bad wildcard Write pypi.org, *.github.com, example.com:8443
Journal: network notice refused host:443 (not allowed) / (resolves to …, not a public address) / (doesn't resolve); programs see 403/502 Not in allowed_hosts, a private address, or DNS failed Add the host if it should be reachable; private addresses are never allowed
Programs in a proxy VM fail without a network notice They ignore HTTP_PROXY/HTTPS_PROXY (raw sockets, ssh) Use HTTPS, configure the program’s proxy, or use nat
llm guest_port N is taken by the guest (3128 is the egress proxy, 3200 and up the HTTP MCP servers) Port collision Pick another guest_port
agent.mcp name "…" must be [a-z0-9_-]{1,32} and not "chat" / … is used twice / … needs either a command or an http(s) url / secret_env is only for HTTP servers (url) Validation Fix the entry
agent.mcp NAME: $VAR is not set An HTTP MCP server’s secret_env is missing (also in check-config) Export it
Log: agent.mcp NAME: its secret is read at startup; restart gopala to use it A server with secret_env added by a hot reload Restart
Model calls fail with 429 budget_exceeded; journal budget notice The conversation’s or the host’s token budget is used up Wait for 00:00 UTC (or the 1st), or raise the budget on the dashboard
budget.conversations key "…" must be a conversation key like "lupyd-g-42" … / budget.low_stops_at_percent <= … Validation Copy keys from the dashboard; keep low ≤ normal, both 1–100
gopala image build: … build-rootfs.sh failed A script step failed (it says which); Debian or packages without root Rerun with sudo; install debootstrap for Debian
not signed in to Lupyd: set LUPYD_EMAIL and LUPYD_PASSWORD No usable session.json and no credentials (names from email_env / password_env) Export both variables for the first run
set both LUPYD_EMAIL and LUPYD_PASSWORD Only one is set Set both, or neither once signed in
Lupyd sign-in failed Wrong credentials, wrong auth0_* settings, or the tenant refused the password-realm grant Check the account in the app; for a non-production stack check [platforms.lupyd]
the sign-in server returned no refresh token (offline_access not allowed) The Auth0 client doesn’t allow refresh tokens Use a client/tenant that issues them (production defaults do)
saved Lupyd session no longer works (…); signing in again (warning) Refresh token revoked or expired Provide the password again if sign-in then fails
another firefly-mcp is already running on … Two processes on one data_dir/lupyd Run one Gopala per account and data directory
lupyd: the bot's account already has N devices; remove one in the app (Settings, Devices) … Lupyd’s device limit Remove a device in the app; the bot connects by itself afterwards
signed out of Lupyd: … and the daemon exits The device was removed or the session ended Restart with credentials
every platform stopped All providers’ event pumps ended Check the log for the provider’s error
llm <name>: $<VAR> is not set Missing API key variable (also in check-config) Export it or remove the [[llm]] entry
server.bind … is not loopback: set server.tls_cert/tls_key … Public plaintext bind Add TLS, bind to loopback behind a proxy, or allow_plaintext = true on a private network
rooms.enabled needs server.enabled … Watching without the web server Enable [server] or disable rooms
binding 0.0.0.0:60901 / web server stopped: … Port in use or no permission Change server.bind
site directory … does not exist; serving only the API and the websocket (warning) server.site_dir missing Build the site and copy site/dist there, or ignore
API 403 admin API disabled server.admin_token_env variable unset or empty Set it (long random value) and restart
API 401 unauthorized / the dashboard asks to sign in again Wrong or missing token, or the session expired (12 h) or Gopala restarted Sign in again; scripts: Authorization: Bearer <token>
API 429 too many failed sign-ins; try again later More than 10 wrong tokens from your address within a minute (behind a proxy: from anyone) Wait the Retry-After seconds; check for a script with a stale token
API 403 missing X-Gopala header A cookie-authenticated write without the CSRF header Send X-Gopala: 1 (the dashboard does), or use a Bearer token
API 409 gopala web runs no agents; start gopala run Dashboard or control endpoints under gopala web Run gopala run
PUT /api/config lists restart_required Those sections can’t change while running Restart gopala run
API 500 cannot save the configuration The config directory isn’t writable by Gopala (temp file and .bak live next to the file) Fix permissions / ReadWritePaths
vm.tools_image … is unavailable / is not a regular file / is not readable by the jailer uid … / has sha256 …, but vm.tools_sha256 is … The tools image is missing, unreadable or changed Fix the path or permissions; rebuild and paste the new tools_sha256
Journal: lifecycle refused, refusing to start the harness … With tools, the harness resolved to a file the agent user can change Make sure the image has the harness in bin/ and agent.command is a bare name (default omp)
Journal: host tools unavailable: … The guest couldn’t mount the tools drive Check the kernel has ext4 and virtio-blk, and console.log
Dashboard tokens stay at zero while llm_requests grows The provider’s responses carry no usage report the proxy can read (e.g. an encoded body) Check the llm events’ status; see the metering notes in architecture.md
/watch answers 404 gopala web mode or rooms.enabled = false Run gopala run with rooms enabled
Viewer: server: not allowed to watch this conversation / access revoked The holder isn’t in the conversation any more, or [monitoring] output doesn’t include them Expected; adjust [monitoring] if they should watch
/monitor or /api/monitor/: 401 unauthenticated No credential, a wrong admin or watch token, an invalid or expired Lupyd token, or lupyd_tokens = false Check the token; Lupyd tokens need platforms.lupyd.auth0_domain and auth0_audience to match the app’s tenant (debug log: monitor: refused a Lupyd token: …)
/monitor: 401 credentials expired; authenticate again The Lupyd token expired or the watch token was revoked Send authenticate again with a fresh token
/monitor or /api/monitor/: 404 no such conversation for a conversation that exists The caller isn’t a member (as far as the platform says), it’s another platform’s conversation, or the watch token is for another conversation Expected; on Discord without the Server Members intent, check the person can be found by member search
/monitor: 403 not allowed to read this conversation's output / … metrics A member the [monitoring] rules don’t allow Adjust output / metrics
/monitor: 429 slow down / at most 16 subscriptions per connection More than 20 requests a second (bursts of 40), or too many subscriptions Slow down; unsubscribe first
Log: <key>: can't ask <platform> whether <user> is a member: … The platform’s API failed; the person is treated as not a member Check the provider’s log and permissions (Slack scopes, Discord intents)
/configure/ or /templates/ can’t check a file against the server Signed out: the configuration helpers need the operator now, so the pages use the schema bundled with the site Sign in
monitoring rule "…": use "members", "approvers", "owner", "role:<name>", "permission:<name>" or "user:<username>" / monitoring.membership_cache_secs is at most 3600 … / monitoring.conversations key … Validation Fix the rule, cache time or key
vm.max_vcpus, vm.max_memory_mib and vm.max_disk_gib can't be below … / vm.memory_budget_mib is smaller than one VM's vm.memory_mib Validation Raise the maximums or budget
/status: waiting for host memory for long Running VMs hold the memory budget Wait for idle shutdowns, raise memory_budget_mib, lower max_memory_mib
Agent tool result this host runs all its VMs in N MiB of memory / over this host's per-VM maximum … / the host is low on disk space; clean up instead / disks only grow … / this host doesn't grant more resources … machine::decide denied a request Expected; adjust [vm] maximums, budget, min_free_disk_gib or grants if the request was legitimate
vm.kernel … is unavailable / … does not exist Wrong image paths Use the paths gopala image build printed
can't start firecracker / firecracker not found in PATH Binary missing or not executable (PATH is passed through) Install it or set vm.firecracker to an absolute path
the guest didn't connect in time Guest didn’t reach the control socket within vm.boot_timeout_secs Read console.log: kernel without virtio-vsock, guest service not enabled, image without gopala-guest; raise the timeout on slow hosts
the VM exited while booting: … (see its console.log) Firecracker or the kernel failed console.log; check /dev/kvm permissions, kernel/rootfs compatibility, memory
no hello from the guest / the guest didn't introduce itself Something other than gopala-guest connected, or it hung Rebuild the image
guest speaks control protocol X, host Y Host and guest from incompatible versions Rebuild the rootfs and update existing disks (see Build)
socket path … exceeds 107 bytes; use a shorter data_dir/chroot_base {uds}_{port} doesn’t fit sun_path Shorten data_dir or vm.jailer.chroot_base
can't hard-link … into the jailer chroot …: vm.jailer.chroot_base must be on the same filesystem as data_dir, vm.kernel and vm.tools_image EXDEV: chroot on another filesystem Move chroot_base (or data_dir / images) onto one filesystem
`ip …` failed: … (tap devices need root or CAP_NET_ADMIN) No permission to create taps Grant CAP_NET_ADMIN (see the unit), run as root with the jailer, or use network = "proxy" or "none"
can't run `ip` (iproute2) iproute2 missing Install it
e2fsck failed / resize2fs failed e2fsprogs missing or a damaged disk Install e2fsprogs; inspect the disk. An interrupted grow isn’t retried: run e2fsck -f and resize2fs on the disk manually
vm.jailer.uid and gid must not be root Validation Use a dedicated unprivileged uid/gid
Journal shows provisioning failed: … A package, setup or user_setup step failed (the harness still starts) Read the EVENT_LOG lines; fix the step; it reruns when the inputs change
no supported package manager (apt-get, apk, dnf) found Custom rootfs Use one of these or set agent.packages = []
Chat: ⚠️ My workspace keeps failing to start; an operator needs to look at it. Three VM failures in a row Daemon log and console.log; the session keeps retrying (up to 160 s apart)
Agent tool error rate limit: at most N messages a minute here … behavior.max_messages_per_minute Expected; raise it if legitimate
Bot ignores a group respond_to = "dms", or no @mention/reply with mentions Adjust respond_to; mention the bot
Bot leaves a group right after joining invites policy and the owner isn’t allowed (or unknown) Add the owner to allowlist or use invites = "anyone"
Watching is turned off on this bot. rooms.enabled = false Enable rooms
Viewer: server: invalid or expired token Token expired, reissued by a later /watch, or revoked when the user left Send /watch again
Viewer: server: room is full max_viewers_per_room Raise it or close other viewers
Viewer: server: too slow, reconnect with since_seq Viewer fell 1024 events behind Reconnect with --since <last seq>
Viewer: no room key for epoch N / the room key does not unwrap with this token Missed or wrong key (e.g. token for another conversation) Reconnect; request a new link
collaboration.agent_role must be a role name of 1-100 characters / collaboration.max_agent_messages_per_hour must be at most 3600 / platforms.lupyd.profile_api_url must be an http(s) URL Validation Fix the value
image.setup entries must be non-empty shell commands An empty entry (or one with a NUL byte) in [image] setup Remove it
gopala image build: base … with setup commands needs root: run it with sudo / setup command failed: … image.setup needs root; one of its commands exited non-zero Run the build with sudo; run the failing command by hand in a chroot to see why
Agent tool error collaboration is off on this host collaboration.enabled = false Turn it on if agents should work together
list_agents returns nobody (or misses an agent) Lupyd: nobody holds the agent group role; Discord without the Server Members intent: the agent hasn’t posted since Gopala started; Slack: the account isn’t a bot user or in the @agent user group; the answer is cached for 5 minutes Give the role (see Collaboration); turn on the intent; wait for the cache or change the group’s members
Lupyd agents show no bio; log lupyd: can't read the profiles of … The profile request to platforms.lupyd.profile_api_url failed, or the bio is empty or an element body Check the URL and the host’s network; write a plain-text bio in the Lupyd app
Agent tool error couldn't list agents: … The platform’s API failed (the log has the full error) Check the provider’s log, scopes and intents
Another agent’s ping doesn’t wake the bot It didn’t mention the bot or reply to it, it came as a direct message, collaboration is off, or the cap is used up (log: <key>: dropped a message from agent … (collaboration.max_agent_messages_per_hour)) Mention the bot; raise max_agent_messages_per_hour if the exchange is legitimate
A changed agent_role doesn’t change which messages count as agents’ Providers read it once at startup Restart gopala run
Agent tool error your harness starts once per message, so there is nothing to restart restart_harness with agent.harness = "exec" Expected; the next message runs with the new MCP config
Restart note: From /home/agent/.gopala/mcp.json: … left out "name" (…) The agent’s MCP file names a reserved server (chat, memory, an [[agent.mcp]] name), an invalid entry, more than 32 servers, or the file is over 64 KiB, not JSON, a symlink or not the agent’s The reason is in the note and the guest log; the agent can fix its file and restart again
Journal: mcp notice MCP server <name>: failed (Claude Code) A server in the generated config didn’t start or connect Check the command exists in the VM and the network reaches an HTTP server
Claude Code: model calls fail or it reports missing credentials No [[llm]] entry with ANTHROPIC_BASE_URL in guest_env, or no placeholder ANTHROPIC_API_KEY (untested whether it needs one) Add both to the Anthropic entry’s guest_env (see Choosing a harness)
Journal: lifecycle refused with agent.harness = "claude_code" claude resolved to a file the agent can change (installed by user_setup) while the tools image is in use Rebuild the tools image (gopala image build bundles Claude Code) or install it with image.setup; set command = "claude"
Claude Code or omp fails with an unknown option at start agent.args still holds the other harness’s arguments (the default --approval-mode yolo is omp’s) Set args for the harness you chose

Everything lives under data_dir (layout in architecture.md). Back up <data_dir>/lupyd as a secret (losing it means a new device and losing access to past MLS state; Slack and Discord keep nothing on disk), <data_dir>/rooms/rooms.db if session history matters, and <data_dir>/vms/*/disk.ext4 with allocation.json for the agents’ workspaces, memories (/home/agent/.gopala/memory.db inside the disk) and grants (preferably while the VM is stopped). Deleting a conversation’s vms/<key> directory while its VM is stopped resets that agent to a fresh copy of the rootfs at the base allocation, with no memories. Keep the config file and its .bak with the backup.