Why 100% Command Success Wasn't Enough to Evaluate My AI Football Team
I iterated on prompts, model choices, player roles and positioning for Chalk Sharks in the AWS Agentic Football Cup, then compared what I expected with what the match data actually showed. The article focuses on what those experiments taught me about evaluating agent behavior, tracking configuration changes, and avoiding conclusions the telemetry cannot support.
Notes from coaching Chalk Sharks on the Agentic Football Cup platform, plus a small Python check for tracking prompt changes properly.
Farooq Butt | AWS Community Builder
I coach Chalk Sharks on the AWS Agentic Football Cup platform. The game, player agents, match engine and telemetry are already there. My part is the coaching layer: choose models, write the instructions for each player, set the formation, watch the matches, then decide what to change.
During one iteration, I gave two players a simple job: win the ball and release it quickly to teammates positioned to shoot. Watching the match, I thought they were doing it better.
The report made me question that conclusion. It showed 710 commands, a 100% success rate, and exactly one recorded
PASS. We lost 4-5. [1]I wasn't expecting a command-success counter to guarantee a win. I wanted to know whether the behavior I had changed was actually happening. What I saw looked encouraging, but the report didn't tell the same story.
Results didn't settle it either. After 30 matches, Chalk Sharks had 24 wins and six losses, including an eight-match winning streak at that point. That record covered several strategy revisions. Later, the active configuration lost three matches. A streak was useful. It wasn't enough to tell me which instruction was working. [1, 2]
What I could actually change
The Agentic Football Cup gives me five player roles: a goalkeeper, a defender, a midfielder, and two forwards. I can choose the model for each role, write its instructions, adjust the formation, and then review the resulting match behavior.
I didn't build the agents or the tournament engine. I was tuning a team inside the platform.
For architectural context, the published Strands Agents implementation of the Agentic Football Cup describes agents receiving field state and returning structured commands such as
MOVE_TO, PASS, or SHOOT. It uses Amazon Bedrock models and Amazon Bedrock AgentCore, and its example command schema includes an action type, a target player ID, and coordinates. I use that as useful background, not as proof that every detail of the league interface I played on works exactly the same way. [3]One strategy I tested had two players acting as fixed shooting outlets. Two others were free to chase the ball and feed those shooters. The goalkeeper stayed close to goal and distributed possession. In later revisions, I also allowed the two mobile players to shoot when an opening appeared.
The team setup could be exported as JSON. Each player entry contained a name, a model identifier, and a
system_prompt with its instructions. Formation was selected separately in the game UI. In the configuration discussed here, the goalkeeper used Claude Sonnet 4.6 and the other four players used Amazon Nova Micro. [2]To keep the rest of this article readable, I'll use two simple labels:
- Baseline: the configuration that was actually running.
- Candidate patch: a proposed change to the two mobile players' shooting instructions.
Those are my experiment labels, not AWS release names. The original filenames stay in the companion material for anyone who wants to reproduce the comparison.
What the success counter left unanswered
In that 4-5 loss, Chalk Sharks had seven shots and four on target. The report also showed a 100% success rate. What it did not tell me was whether each player had made the decision I wanted in the situation I cared about. [1]
I also couldn't establish from the report alone exactly what that success percentage measured. Calling it "valid JSON" would have been a guess. Calling it "good agent behavior" would have been an even bigger one.
I found it more useful to split the problem into four questions.
Figure 1. These are the questions I needed answered. They are not the tournament's definition of its success metric.
Was the response accepted? A movement command can be perfectly valid and still send a goalkeeper somewhere I don't want him to go.
Was it the right decision? A pass can be valid even when my instructions say an available shot should come first.
Did the action actually happen? The number of
SHOOT commands in a report is not the same thing as the number of shots in the match statistics. I need the requested action, the target, and the result to connect those dots.Did it help the team? A win can hide a repeated positioning problem. A loss can contain several decisions that were individually correct.
The Strands example uses structured output to constrain command shape. That's important because the server needs something it can understand. It still doesn't answer the other three questions. [3]
The same distinction matters outside football. An agent can produce an accepted tool call and still pick the wrong tool, use the wrong inputs, or fail to complete the user's actual task.
For my tournament experiments, I had configuration exports, match reports, and what I saw while watching. I did not have a synchronized trace of every game state, model response, selected command, and executed action. That matters, because some explanations simply went beyond the evidence I had.
Some prompt logic looked better on paper than it behaved in practice
I used
FIRST TRUE RULE ONLY in several prompts to make priorities easier to read. It helped me reason about the instructions, but it didn't magically turn natural language into deterministic code.One earlier version contained this:
1
2
1. YOU HAVE BALL -> SHOOT.
2. YOU HAVE BALL and SHOOT is not possible -> PASS.Read literally, rule 1 matches every possession state. Rule 2 is supposed to be the fallback, but under that ordering it never gets a clean turn.
A clearer structure is:
1
2
3
1. Have the ball and the shooting condition applies -> SHOOT.
2. Have the ball -> PASS.
3. Otherwise -> follow the off-ball instructions.That may look like a small change, but it separates possession from off-ball behavior and makes the fallback explicit.
Figure 2. The conflict is visible in the wording. This alone doesn't prove how the model interpreted it during a match.
A later problem was less obvious. In the baseline, the two mobile players were told to shoot at the centre of goal when no outfield player blocked the lane. I kept seeing situations where I thought a shot was available, but they passed instead.
The candidate patch changed only that trigger. Instead of requiring a clear lane to the centre of goal, it told them to shoot through an open lane between the posts and not wait for an empty goal. [2]
My theory was simple: the original wording might have been too restrictive. But at that point it was only a theory. I had prepared the patch; I had not yet uploaded it.
That became a useful lesson in itself. An AI assistant can produce a plausible explanation for strange behavior almost immediately. That is helpful for generating hypotheses. It is not the same thing as proving the hypothesis.
The player who kept returning to his coordinate
I also tried using coordinates because broad instructions such as "stay left" or "stay near goal" weren't producing the shape I wanted.
The fixed shooters were meant to remain available as passing outlets. Instead, I watched one of them keep heading back toward his assigned point while a nearby ball went uncontested. In other runs, players who were meant to wait appeared to keep moving around that point and burn energy. I could see the behavior; I couldn't see enough of the execution pipeline to say exactly why it happened. [1]
The next revision added a small tolerance around the home position and a local ball-winning exception for the defender. Close enough meant: stop trying to return. A reachable ball nearby meant: go and contest it. [2]
Those are exactly the sort of behaviors I would now test separately before another full match. Put the player at home with nothing happening and see whether it waits. Then put a loose ball nearby and see whether the local exception wins.
A coordinate in a prompt makes the intended destination clearer. It is still an instruction to a model, not a hard movement boundary enforced by the engine.
The patch I prepared but never deployed
This was probably the cleanest lesson from the whole exercise.
I prepared the candidate shooting patch and then went away. Three automatic matches ran while I was gone. Chalk Sharks won 6-5 against Chalk Arcs, then 4-1 twice against the opponent named Glacier Lemurs in the report headers. [2]
I had not uploaded the candidate patch. The baseline was still active.
Figure 3. Creating a new configuration file did not make it the configuration that played the next match.
That sounds obvious when written in one sentence. In practice, the conversation, filenames, suggested changes, and automatic matches were moving quickly enough that I briefly started talking about the latest file as if it had produced the latest results.
It hadn't.
From that point on, I wanted three states to stay separate: generated, applied, and tested.
For future rounds, I would keep the saved configuration, model selections, formation, application time, and match IDs together. A live export from the platform, or a simple operator note saying what was actually uploaded, is much stronger evidence than the latest filename sitting in a chat.
A small local check for configuration changes
I wrote a small Python utility for the companion material. It compares the baseline and candidate files and checks whether the candidate stayed within the scope I intended. It does not call AWS, invoke a model, or replay a match. [2]
The original exports are included as
configs/baseline.json and configs/candidate.json. Formation is supplied separately because it is selected outside the JSON export.The first part creates a SHA-256 fingerprint of the saved settings. I leave out the export timestamp so that exporting identical settings twice doesn't create a fake strategy change.
1
2
3
4
5
6
7
8
9
10
11
12
13
import hashlib
import json
from pathlib import Path
def config_hash(path: str, formation: str) -> str:
config = json.loads(Path(path).read_text(encoding="utf-8"))
config.pop("exported_at", None)
payload = {"formation": formation, "config": config}
canonical = json.dumps(
payload, sort_keys=True, separators=(",", ":"),
ensure_ascii=False, allow_nan=False,
)
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()This fingerprint identifies saved data. It does not understand the meaning of a prompt. A whitespace change inside the prompt changes the hash. It also cannot include runtime settings that were never exported.
The full utility also performs a field-by-field comparison. Run it like this:
1
2
python config_audit.py configs/baseline.json configs/candidate.json \
--formation 1-1-2 --expect-mobile-only --summaryFor these two saved configurations, the output is:
1
2
3
4
5
6
7
8
9
Changed:
Haaland: system_prompt
Lamine Yamal: system_prompt
Model selections unchanged: yes
GK and fixed-shooter records unchanged: yes
Formation supplied: 1-1-2
Two-mobile-prompt scope check: passed
Deployment verified by this check: no
Model or game invocations: 0In my lineup, Haaland and Lamine Yamal were the two mobile roles. The check confirms that only their instructions changed, apart from the timestamp. The goalkeeper, fixed shooters, and model selections remained the same.
It also prints the most important line in the block: Deployment verified by this check: no. A file diff can tell me what I intended to change. It cannot tell me what I uploaded to the tournament platform.
All seven local utility tests passed. They check the comparison code itself: expected prompt changes, timestamp exclusion, formation and model changes affecting the fingerprint, preservation of the other players, non-mutation of the input, and rejection of an empty formation. They do not test football behavior.
Where the evidence still disagreed
The 4-5 passing example is a good reminder that I sometimes had three different views of the same match.
| Evidence | What I recorded |
|---|---|
| Match report | 710 commands, one PASS, 100% reported success, and a 4-5 loss. |
| What I saw | The two mobile players seemed to release the ball to the shooting outlets more quickly. |
| What I can say | The visual observation and the counter don't line up. I don't have enough detail to explain the difference. |
There were other inconsistencies. I watched one forward score when the report credited the other. Some reports named one opponent in the header and another in the timeline or awards. In away fixtures, the generated coaching notes were written for the home team, which wasn't Chalk Sharks. I kept those discrepancies instead of quietly cleaning the dataset until it told one neat story. [1, 2]
The scorelines could swing sharply too. We beat Total Attack United 10-2 in one practice match, with ten shots on target from 11 attempts. Later, we lost 3-7 to the same opponent. Several prompts and roles had changed between those matches, so the comparison did not isolate one causal change. [1]
That is exactly why I stopped using the score as the only evidence for whether a prompt was good.
What I would test before the next match
Before burning another match on a broad rewrite, I would test a handful of specific situations. These are proposed checks; I did not run them during the winning streak.
| Situation | What I would check |
|---|---|
| A fixed shooter is already at its home position. | Does it wait, or keep requesting movement toward the same point? |
| A loose ball is within the defender's reach. | Can collecting it take priority over returning home? |
| A mobile player has an established shooting opportunity. | Does it choose SHOOT before its passing fallback? |
| A mobile player has possession but no intended shot. | Does it pass to the assigned outlet instead of carrying the ball forward? |
| A shot is heading toward the goalkeeper. | Does the requested movement remain inside the intended defensive area? |
I would repeat each situation and save the response alongside the prompt fingerprint, model ID, and any inference settings the platform exposes. Captured game states would be ideal. Hand-built examples should be clearly marked as synthetic.
The shooting test needs one extra rule: the test itself has to establish that an opening exists. I don't want the same model deciding both whether an opportunity was available and whether it was right to ignore it. If the necessary state is missing, the result should be not assessable, not forced into pass or fail.
I would also test the mirrored field orientation. A prompt that works when Chalk Sharks defend the left side still has to identify the correct goal when the teams switch ends.
Only after those checks would I go back to live matches. Where possible, I would use the same opponent, keep the rest of the setup fixed, change one intended behavior, and preserve the previous configuration as a rollback.
Where AWS evaluation and observability could take this next
The local comparison solves one small problem: it tells me whether the saved files changed in the way I intended. It doesn't evaluate the decision made by an agent at runtime.
For a runtime I control, Amazon Bedrock AgentCore gives me a way to take the next step. Custom code-based evaluators can use AWS Lambda to evaluate trace data programmatically. AWS documents the evaluation input and output format, the required invocation permissions, and the same-Region requirement for the Lambda function and evaluation resources. [4]
I would start with a deterministic check rather than another language model. For example, take the goalkeeper's requested movement target and compare it with the defensive area I allow. That needs a tested adapter for the actual trace schema, plus the correct player identity and field orientation. I have not deployed that evaluator, and it was not part of the tournament results described in this article.
Latency is another reason I want traces. I spent a lot of time trying to push the goalkeeper below 200 ms. Several reports still came back around 600 ms despite shorter prompts and model changes. [1]
AgentCore documents runtime latency as the interval from receiving a request to sending the final response token, and it exposes invocation spans when observability is enabled. The tournament dashboard did not establish that it used the same measurement boundary. [5]
So before blaming a particular model, I would want to line up the state input, model invocation, returned command, and execution timing under the same workload.
Before I change another prompt
I still enjoy the scoreboard. A 10-2 win is a lot more fun than a 3-7 loss.
But the useful question I took away from these matches is smaller:
In the situation I cared about, did the player role I configured make the decision I intended, using the configuration that was actually running?
That question forces me to keep the prompt diff small, save the rollback, record what I actually deployed, and separate what I saw from what I can prove.
Sometimes the answer is still: I don't know yet.
That is a much better starting point for the next experiment than inventing another explanation after the score is already on the board.
Sources and companion material
[1] Match notes. Selected reports and my observations are in
companion/selected-evidence.json, including the 4-5 loss, 10-2 win, and corrected 3-7 defeat. These are not instrumented execution traces.[2] Configuration files and local checks. The companion contains the original exports under readable aliases, the filename mapping, selected evidence,
config_audit.py, seven tests, and captured output. The baseline file is w2-v3.5.4-coordinated-shoot-first.json; the candidate is w2-v3.5.5-mobile-shooting-micro-patch.json. The candidate was not applied during the three automatic matches described here.[3] Strands Agents: Inside Agentic Football Cup: 10 Strands Agents, 1 Second to Decide .
[4] AWS documentation: Custom code-based evaluator .
[5] AWS documentation: AgentCore generated runtime observability data .
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article