Running Kiro Crew on Amazon Lightsail: it was the CPU that gave out, not the memory
Two fixed vCPUs, load average 58, SSH dead at banner exchange, and a watchdog quietly killing the gateway. The diagnosis, the fixes that did not work, and nine build-time potholes.
Summary
I ran Kiro Crew on a single Amazon Lightsail instance, keeping cron jobs and a Slack bot alive 24/7.
The build itself took half a day. Getting from there to "stays up reliably" took a lot longer,
and that part is where all the interesting failures lived.
The instance is long since torn down, so what follows is a record of the period I ran it.
The build itself took half a day. Getting from there to "stays up reliably" took a lot longer,
and that part is where all the interesting failures lived.
The instance is long since torn down, so what follows is a record of the period I ran it.
The environment was Amazon Linux 2023 in ap-northeast-1, kirocrew 0.1.3, kiro-cli 2.16.2, gh 2.96.0.
Every number below was measured on that setup.
Every number below was measured on that setup.
First, a caveat: kirocrew cloud exists
Let me get the deflating part out of the way. Kiro Crew officially supports running on AWS.
1
2
3
4
5
6
kirocrew cloud doctor # check prerequisites and AWS reachability
kirocrew cloud iam-policy # print a least-privilege IAM policy
kirocrew cloud launch # provision, configure, open the dashboard
kirocrew cloud connect # connect to the dashboard over an SSM tunnel
kirocrew cloud stop|start # pause and resume billing
kirocrew cloud destroy # delete the instance and all AWS resourcesA single
The connection goes through an SSM tunnel, so you need neither a static IP nor open ports 22/443.
launch handles EC2 provisioning, configuration, and dashboard access.The connection goes through an SSM tunnel, so you need neither a static IP nor open ports 22/443.
The only real reasons I had for picking Lightsail instead were predictable flat monthly cost, and not
wanting to touch EC2/SSM. Choosing to hand-roll it means walking into every pothole described below.
Read on with that in mind.
wanting to touch EC2/SSM. Choosing to hand-roll it means walking into every pothole described below.
Read on with that in mind.
Problem 1: the CPU saturates long before the memory does
Sizing discussions tend to be framed as "how many GB of RAM do I need," and that is how I framed it too.
What actually mattered was the vCPU count.
What actually mattered was the vCPU count.
Kiro Crew keeps an embedding model (
llama-cpp-python. The official documentation states plainly that "Embeddings are always on ...
there is no way to disable" it. You cannot turn it off. So if you budget based on
"the Kiro CLI process is about 160MB RSS," you will always come up short.
qwen3-embedding-0.6b.gguf) permanently loaded in-process viallama-cpp-python. The official documentation states plainly that "Embeddings are always on ...
there is no way to disable" it. You cannot turn it off. So if you budget based on
"the Kiro CLI process is about 160MB RSS," you will always come up short.
Measurements taken 2026-08-08 on the $24 (4GB) plan:
- With only the gateway resident and no agent running, kirocrew's total RSS was about 1.46GB
- System-wide: 2.4GB used, 1.0GB free
So the $12 (2GB) plan is essentially full at idle. That much is still a memory story.
Here is where it turns. Every Lightsail plan from $5 to $24 shipped the same 2 vCPUs.
Paying more does not buy you cores. And embedding inference (
is CPU-bound, so once MCP servers and Playwright startup pile on top of it, two cores saturate immediately.
Paying more does not buy you cores. And embedding inference (
_infer_loop in embeddings.py)is CPU-bound, so once MCP servers and Playwright startup pile on top of it, two cores saturate immediately.
What that looked like in practice: memory at 44 percent with 2.1GB free — plenty of headroom —
while load average climbed to 50 and SSH stopped letting me in.
while load average climbed to 50 and SSH stopped letting me in.
1
ssh: connect to host ... Connection timed out during banner exchangeThe nasty part is that the TCP connection to port 22 still succeeds. A connectivity test shows
nothing wrong at all. It is an easy way to burn an hour suspecting the firewall.
nothing wrong at all. It is an easy way to burn an hour suspecting the firewall.
Problem 2: loop-stall, or what "it restarts by itself" actually means
Under sustained load, the gateway would die with
over and over. The logs give no reason. From the outside it just looks like the thing restarts on a whim.
Main process exited, code=exited, status=1/FAILURE, and systemd would bring it back ten seconds later,over and over. The logs give no reason. From the outside it just looks like the thing restarts on a whim.
The culprit turned out to be Kiro Crew's own watchdog.
responding for a set period — 25 seconds by default. This is not a crash; it is a deliberate
self-protection mechanism. Because systemd has
dashboard/loop_watchdog.py fires faulthandler and terminates the process when the event loop stopsresponding for a set period — 25 seconds by default. This is not a crash; it is a deliberate
self-protection mechanism. Because systemd has
Restart=on-failure, you observe it as a restart.Dumps land in
~/.kiro/crew/logs/crash-dumps/loopstall-*.txt.And that default of 25 seconds is far too tight for 2 vCPUs.
Running an agent once was enough to kill it before a single command executed. The audit log stayed empty.
Running an agent once was enough to kill it before a single command executed. The audit log stayed empty.
The fix is raising the threshold
The real fix is raising
Kiro Crew's own setting description says "Raise it on a slow machine," so doing this on
underpowered hardware is an anticipated use, not a hack.
dashboard.loop_stall_exit_after_secs.Kiro Crew's own setting description says "Raise it on a slow machine," so doing this on
underpowered hardware is an anticipated use, not a hack.
1
2
3
4
5
6
7
8
9
10
11
cp ~/.kiro/crew/config.json ~/.kiro/crew/config.json.bak
python3 - <<'EOF'
import json, pathlib
p = pathlib.Path.home() / ".kiro/crew/config.json"
d = json.loads(p.read_text())
d["dashboard"]["loop_stall_exit_after_secs"] = 180
p.write_text(json.dumps(d, ensure_ascii=False, indent=2) + "\n")
EOF
sudo systemctl restart kirocrewThe value is clamped to 10-300 by
It is only read at startup, so a restart is required after editing.
LOOP_STALL_EXIT_AFTER_MIN / _MAX in config/loader.py.It is only read at startup, so a restart is required after editing.
The description also warns that "the desktop app's liveness probe kills at around 20 seconds first,
so values above 20 only take effect in headless setups." Lightsail is a headless systemd deployment,
there is no such probe, and the caveat does not apply.
so values above 20 only take effect in headless setups." Lightsail is a headless systemd deployment,
there is no such probe, and the caveat does not apply.
Here is the effect of going from 25 to 90 seconds. That was the only change — I touched neither
CPUQuota nor playwright-mcp.| threshold 25s | threshold 90s | |
|---|---|---|
| Restarts | killed mid-run by loop-stall | zero in normal operation |
| Commands executed | 0 (audit log empty) | ran to completion |
| Load average | 58 | 0.21 |
| New dumps | with real stacks (a true stall) | header only |
90 seconds still was not enough, so 180
It looked stable, and then it recurred at 07:47 JST on 2026-08-10:
load average 48.48, memory 75 percent,
load average 48.48, memory 75 percent,
NRestarts at 1.The diagnosis from here was the most interesting part of the whole exercise.
Start with the first line of the dump.
1
Timeout (0:01:30)!It burned the full 90 seconds before being shot. That single line tells you the threshold is still
too low. Conversely, a number smaller than your configured value would point at a different cause.
Reading that one line decides your direction, so always start at the top of the dump.
too low. Conversely, a number smaller than your configured value would point at a different cause.
Reading that one line decides your direction, so always start at the top of the dump.
Next, read the stacks.
| Thread | Where it stopped |
|---|---|
| MainThread (event loop) | asyncio/futures.py:406 _chain_future → _base.py:419 add_done_callback → threading.py:272 __enter__ (waiting to acquire a lock) |
| Worker | apps/dev_mode.py:198 _stat_sentinel → threading.py:275 __exit__ (releasing that same lock) |
| Another thread | embeddings.py:1038 _infer_loop (embedding inference hogging the CPU) |
Seeing MainThread parked on a lock acquisition makes you reach for the deadlock hypothesis.
But when the side releasing the lock (
in the dump, nothing is actually stuck. It will progress if given time; it is not progressing because
embedding inference took the CPU and nothing else gets scheduled.
But when the side releasing the lock (
__exit__) and the side acquiring it (__enter__) both appearin the dump, nothing is actually stuck. It will progress if given time; it is not progressing because
embedding inference took the CPU and nothing else gets scheduled.
Which means chasing a deadlock through the code here would have been wasted effort.
The fix is the threshold. "A thread waiting on a lock does not imply a deadlock" is worth
remembering on its own as a crash-dump reading habit.
The fix is the threshold. "A thread waiting on a lock does not imply a deadlock" is worth
remembering on its own as a crash-dump reading habit.
So I raised it from 90 to 180 seconds — half the 300-second ceiling.
A 1-minute average of 48 with a 5-minute average of 31 describes a sustained spike, not something
that settles in tens of seconds.
A 1-minute average of 48 with a 5-minute average of 31 describes a sustained spike, not something
that settles in tens of seconds.
There is also a reason not to overshoot. A longer threshold delays recovery from a genuine deadlock
by exactly as much. 180 seconds is the compromise: it lets multi-minute CPU spikes pass unnoticed,
while a true wedge still recovers within three minutes.
If it recurs, the path is 240, then 300 — but 300 is the ceiling, and once you are there the only
remaining lever is the load side (fewer resident MCP servers, moving cron to a separate instance).
by exactly as much. 180 seconds is the compromise: it lets multi-minute CPU spikes pass unnoticed,
while a true wedge still recovers within three minutes.
If it recurs, the path is 240, then 300 — but 300 is the ceiling, and once you are there the only
remaining lever is the load side (fewer resident MCP servers, moving cron to a separate instance).
Things that looked promising and did not work
Worth writing down the failures too. They may be the more useful half.
CPUQuota and CPUWeight do not stop loop-stall
The reasoning was that throttling CPU via a systemd drop-in would leave the OS some breathing room
even during a runaway.
even during a runaway.
1
2
CPUWeight=50 # share under contention. default 100, lowered to 50 to yield to sshd and friends
CPUQuota=180% # hard cap. at most 1.8 of the 2 coresloop-stall continued regardless.
The watchdog fires because, inside the process, the event loop loses to the embedding inference thread.
A cgroup ceiling does not address that. Squeeze too hard and you make stalls more likely, not less.
The watchdog fires because, inside the process, the event loop loses to the embedding inference thread.
A cgroup ceiling does not address that. Squeeze too hard and you make stalls more likely, not less.
CPUQuota=180% did not protect SSH either
The assumption that leaving 0.2 of a core for the OS would keep sshd alive also failed.
With that setting in place, load average still reached 58 and SSH still timed out at banner exchange.
Under that load, 0.2 of a core is not enough for sshd to complete a handshake.
With that setting in place, load average still reached 58 and SSH still timed out at banner exchange.
Under that load, 0.2 of a core is not enough for sshd to complete a handshake.
CPUWeight=50 only kicks in under contention and costs nothing at rest, so it earns its keep.CPUQuota ended up as something to loosen or remove if restarts increase.Pinning cores with AllowedCPUs is not recommended either
AllowedCPUs=0 dedicates the second core to the OS and reliably preserves SSH access.But since loop-stall is caused by contention inside the process, halving the cores available to it
intensifies the fight between the event loop and embedding inference and makes stalls more frequent.
You end up with the worst outcome: you can log in, but it still falls over.
Nine other potholes
These are independent of the two problems above — just things I hit while building it out.
1. Playwright MCP will not start on Node.js 18
The
so the MCP server fails to start with
nodejs package on AL2023 gives you 18. Kiro Crew's playwright-mcp requires Node 20 or newer,so the MCP server fails to start with
Playwright requires Node.js 20 or higher.1
2
sudo dnf install -y --allowerasing nodejs22 nodejs22-npm git jq
sudo alternatives --set node /usr/bin/node-22nodejs22 fights nodejs over /usr/bin/node, hence --allowerasing.And with both installed, they share the same priority (100) and 18 wins, so you have to be explicit
with
alternatives --set.The annoying part: none of this shows up in
kirocrew doctor. You only find it via the MCP probe.2. Vector Memory reports an error (missing libllama.so)
Vector Memory: runtime ❌ vendored runtime failed to load is a Linux packaging defect in kirocrew 0.1.3.The bundled
_vendor/llama_cpp_libs/linux_x86_64/ contains libggml*.so.0 but not libllama.so.The macOS
.dylib and Windows .dll are both present, so Linux is the only platform missing a piece.I fixed it by pulling just the missing files from the official prebuilt wheel of the same version
(PyPI only carries an sdist, so you need
Verify the ggml SHA-256 hashes match first, then copy
The extension must be
(PyPI only carries an sdist, so you need
--extra-index-url pointing at the prebuilt index).Verify the ggml SHA-256 hashes match first, then copy
libllama.so and libmtmd.so across.The extension must be
.so; .so.0 will not be picked up.This fix lives inside the venv, so every update wipes it out and you have to reapply it.
3. sudo kirocrew service install breaks everything
The installer is designed to generate a unit that runs as the user who invoked it, escalating to sudo
internally only for the parts that need root (placing the unit and calling
internally only for the parts that need root (placing the unit and calling
systemctl).Run it with
config, and embedding model all live under
sudo and you get a unit with User=root and HOME=/root, while the credentials,config, and embedding model all live under
/home/ec2-user/.kiro. It breaks like this:kiro-cli not found in PATH, so agent sessions never spawn (the unit's PATH becomes/root/.local/bin)kirocrew tokenreturns HTTP 403 (the gateway and the CLI are reading different config directories)
Note also that
The correct answer is to not use sudo.
sudo kirocrew itself yields command not found, because secure_path does not include~/.local/bin. That is corroborating evidence that sudo was never the intended path.The correct answer is to not use sudo.
4. Slack Socket Mode returns invalid_auth
~/.kiro/crew/.env is parsed by Kiro Crew itself, not by a shell, and its parser does not strip quotes.1
SLACK_APP_TOKEN="xapp-..." # the double quotes get sent to Slack as part of the tokenWith that,
apps.connections.open returns invalid_auth and Socket Mode never connects.What makes it hard to diagnose is that a curl run after
(because the shell does strip the quotes). The symptom reads as "the token is valid but only the
gateway fails." When in doubt, have Kiro Crew itself load the value and print its length and prefix.
source-ing the same file works fine(because the shell does strip the quotes). The symptom reads as "the token is valid but only the
gateway fails." When in doubt, have Kiro Crew itself load the value and print its length and prefix.
More confusingly, the separate
(the shell sources that one, so quoting is required if a value contains spaces).
Two files both called
~/.env used by your own shell scripts should be quoted(the shell sources that one, so quoting is required if a value contains spaces).
Two files both called
.env, with opposite conventions — worth keeping them clearly separated in your head.5. kirocrew chat hangs when a tool runs
The sandbox's
(
prewarm_backend() is only called on the gateway startup path(
mcp_gateway/gatewayd.py and slack/gateway.py).| Path | Tool execution | Reason |
|---|---|---|
| Gateway (resident service, cron, Slack, dashboard) | works | prewarmed at startup |
CLI such as kirocrew chat / kirocrew doctor | denied | no prewarm, hits a cold cache |
Ask a tool to run from the CLI and you get
All 24/7 traffic goes through the gateway, so production is unaffected, but local smoke tests have to
use
sandbox.wrap_argv → denied, with no response ever coming back.All 24/7 traffic goes through the gateway, so production is unaffected, but local smoke tests have to
use
kirocrew cron trigger rather than kirocrew chat.The
The backend itself — an
MCP Tools: ❌ Sandbox backend unavailable line from kirocrew doctor has the same cause.The backend itself — an
unshare-based namespace approach — is healthy. Installing bubblewrap changes nothing.6. kirocrew update does not work
That subcommand is really a
On an installer-based install (
It breaks nothing, and it also updates nothing.
fetch plus reset --hard plus rebuild against the git tree atKIROCREW_PROJECT_DIR, meant only for setups running from a git clone.On an installer-based install (
cli.sh) it prints KIROCREW_PROJECT_DIR not set and exits 1.It breaks nothing, and it also updates nothing.
The correct update procedure is to rerun the installer.
The data home
credentials, and history survive. Only the venv side is touched, so there is no need to redo
reapplied every time.
The data home
~/.kiro/crew is kept separate from the venv at ~/.kiro/crew-venv, so config,credentials, and history survive. Only the venv side is touched, so there is no need to redo
service install. As noted above, though, the libllama.so fix lives in the venv and has to bereapplied every time.
7. You will not notice abnormal exits
A unit with
restarts it.
between is silently lost.
Restart=on-failure never enters the failed state when it crashes — systemd justrestarts it.
OnFailure= only fires on a transition into failed, so nothing notifies you untilStartLimitBurst (three times in five minutes) is exhausted. Every watchdog-triggered restart inbetween is silently lost.
Use
ExecStopPost= instead. It runs on every exit of the main process, so the script checks$SERVICE_RESULT from systemd and exits quietly when the value is success (a clean stop).8. False loop-stall alerts
I walked into this one myself. Kiro Crew writes a header-only dump (155 bytes) on every single startup.
So if you decide "a dump exists, therefore loop-stall," every ordinary restart triggers an alert.
faulthandler only appends full thread stacks when the event loop genuinely froze, so the real test is
whether the file contains
So if you decide "a dump exists, therefore loop-stall," every ordinary restart triggers an alert.
faulthandler only appends full thread stacks when the event loop genuinely froze, so the real test is
whether the file contains
^Thread 0x.Also, the timestamp in the filename is the process start time, not the crash time.
The file is opened at startup and appended to later, so it is entirely normal for a notification
citing
Use mtime for the actual time of occurrence.
The file is opened at startup and appended to later, so it is entirely normal for a notification
citing
loopstall-20260809T103401Z.txt (started Aug 9 19:34 JST) to refer to a crash at Aug 10 07:47.Use mtime for the actual time of occurrence.
9. New Relic silently forwards no logs
The installer drops an empty
The log config loader cannot parse it, spews
and forwards nothing at all. Renaming the file to disable it fixes things.
/etc/newrelic-infra/logging.d/discovered.yml.The log config loader cannot parse it, spews
failed to load log configs every few seconds,and forwards nothing at all. Renaming the file to disable it fixes things.
1
2
3
sudo mv /etc/newrelic-infra/logging.d/discovered.yml \
/etc/newrelic-infra/logging.d/discovered.yml.disabled
sudo systemctl restart newrelic-infraWhile you are there: the default forwarding targets are only
with Kiro Crew itself absent, so you need to add a config that reads the systemd journal directly.
On the dashboard I put load average in the top row, since with 2 vCPUs fixed the CPU is what runs out first.
cloud-init.log and newrelic-cli.log,with Kiro Crew itself absent, so you need to add a config that reads the systemd journal directly.
On the dashboard I put load average in the top row, since with 2 vCPUs fixed the CPU is what runs out first.
Takeaway
If you run Kiro Crew on Lightsail, raising
It was the one change whose effect I could measure directly: load 58 to 0.21, zero commands executed
to a full run. Everything else was a footnote by comparison.
loop_stall_exit_after_secs is effectively mandatory.It was the one change whose effect I could measure directly: load 58 to 0.21, zero commands executed
to a full run. Everything else was a footnote by comparison.
And if all of the above sounds like a lot of work, just use
kirocrew cloud.Posting Kiro Crew's execution log to Backlog automatically with hooks
A separate topic. Every time Kiro Crew finished a task, a summary of what it did was posted as a comment
on a Backlog issue.
on a Backlog issue.
Asking for it in the prompt did not hold up
The first version simply said "leave a comment on Backlog when you are done" in the prompt.
Sometimes it was ignored. Log collection should not depend on the agent's judgment, so I moved it to
kiro-cli's
Sometimes it was ignored. Log collection should not depend on the agent's judgment, so I moved it to
kiro-cli's
stop hook, where it fires mechanically.Do not post from postToolUse
The obvious move is to post from
You end up mass-producing comments until the log is unreadable.
postToolUse, but that fires on every tool call.You end up mass-producing comments until the log is unreadable.
The trick was splitting it in two.
| File | Event | Role |
|---|---|---|
mark-activity-post.sh | postToolUse | just leaves a marker saying "this session used a tool" |
backlog-log-stop.sh | stop | posts to Backlog if the marker is present |
The post happens only at
That way only sessions with real work get recorded, and pure conversation with no tool use does not.
stop — once, on completion. postToolUse does nothing but drop a marker.That way only sessions with real work get recorded, and pure conversation with no tool use does not.
For reference, any executable
starts. The event is determined by the filename suffix (
You can also declare it with an
over the filename.
*.sh under ~/.kiro/hooks/ is picked up automatically when the gatewaystarts. The event is determined by the filename suffix (
-stop.sh for stop, -post.sh for postToolUse).You can also declare it with an
# event: header within the first five lines, which takes precedenceover the filename.
A failing hook must not stop the agent
The hook always exits 0 and records the reason in its own log at
so a Backlog outage or a network blip never took the main process down with it.
The other rules I settled on:
~/.kiro/crew/backlog-log-hook.log,so a Backlog outage or a network blip never took the main process down with it.
The other rules I settled on:
- Do not post empty responses (an interrupted run is meaningless as a log entry)
- The stop hook deletes the per-session marker, so markers never accumulate
- Truncate long responses (3000 characters by default)
- Pass the body on stdin rather than as an argument, so newlines, quotes, and symbols are handled safely
Use bee for Backlog, not MCP
The hook can just invoke bee mechanically, but you also have to decide how the agent itself touches
Backlog. Left alone it will try to add
Backlog. Left alone it will try to add
backlog-mcp-server, or hit the API directly with curl.The Backlog MCP server requires Docker. On a Lightsail box fixed at 2 vCPUs, the resident cost of one
container competed with embedding inference and pushed things toward the loop-stall described in the
first half of this article. So everything went through bee, Backlog's official CLI.
container competed with embedding inference and pushed things toward the loop-stall described in the
first half of this article. So everything went through bee, Backlog's official CLI.
Permanent rules belong in learn, not steering
I got this wrong once before finding the right place for the rule.
The agent config's
against the per-session working directory (
Putting a file in
you want applied globally.
resources uses a relative path, file://.kiro/steering/**/*.md, which resolvesagainst the per-session working directory (
~/workplace/kirocrew-workspace/<session>/).Putting a file in
~/.kiro/steering/ means it never gets read. That location is not suited to rulesyou want applied globally.
kirocrew learn add also takes --negative, so you can state what not to do.1
2
3
4
kirocrew learn add \
'Always use the bee CLI to work with Backlog. Credentials are already configured in ~/.beerc.' \
--category tool \
--negative 'Do not add or start a Backlog MCP server such as backlog-mcp-server. It requires Docker, which strains the CPU on a Lightsail box fixed at 2 vCPUs and induces gateway restarts via loop-stall.'Whatever goes into learn is stored in the data home at
The agent config's
Rules I could not afford to lose went into learn, and that is where the split settled.
~/.kiro/crew/memory.db, so it survives updates.The agent config's
prompt points at a file inside the venv, and that one is lost on every update.Rules I could not afford to lose went into learn, and that is where the split settled.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article